Sunday, June 30, 2019

Using Azure DevOps to collaborate Data Scientists and App Developers


What is your most favorite IDE? A developer can mention various of IDE’s. How about data scientist?


Normally, the Developers develop the code and data scientists train develop and train the model. Tools such as VSCode come in handy as easy to install and use. Many have many preferences so the chances are that it can be a choice of the developer or the development team. Usually the trained model is handed over to the app developer for integrating it and build the final application. There are times where the mismatches in compatibility can cost both the app developer and the model developer. The resulting friction between app developers and data scientists to identify and fix the root cause can be a slow, frustrating, and expensive process
We often here organizations including managers continuously talking about Artificial Intelligence. People like to find solutions that are integrated with AI. So as the developers have a development lifecycle, the data scientists follow a data science lifecycle. 

The lifecycle includes processes such as,

Data Ingestion --> Data Preparation --> Model Development --> Model Deployment

There can be many iterations of this lifecycle as there can be requirements for changing the data labels, removing anomalies, changes upon user feedback and timely decision changes and many more. 
Application Development Lifecycle for a developer includes building app, testing, maintaining, and continuously adopt to changes in user requirements where the models of data scientists also can be a part of. There shall be a good understanding between the application developers and model developers as mismatches can lead to errors in the integrated systems. I have seen many issues with even traditional SSRS implementations to web applications. When the model changes, app fails. 

Azure DevOps and Azure Machine Learning are combined to overcome this issue with a solution suggested by Microsoft as shown in the image below. 

Source: Azure Blog

Both the app developers and model developers need to use Git as the repository for managing the development artifacts. 

If we simplify this process to understand, the model training code developed by the data scientist will trigger the Azure DevOps CI/CD Pipeline as available on the right top of the diagram above. There the developer can execute multiple steps such as unit tests, integration tests, training and etc.. The changes made by the app developer will trigger integration tests whenever a code change is pushed to the repository. There can be triggers set on data lake as well to execute different commands. The Model Store will register the new models and trigger releases for human approval followed by many other necessary tests as shown in the bottom right. These procedures will help both the developers coordinate and work well within the changes required as well. 
The detailed article on Azure Blog can be found here.

26 comments:

  1. This implies the labyrinth has no out of reach segments, no roundabout ways, no open territories.ExcelR Data Science Courses

    ReplyDelete
    Replies
    1. Great Article
      Cloud Computing Projects


      Networking Projects

      Final Year Projects for CSE


      JavaScript Training in Chennai

      JavaScript Training in Chennai

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
  2. I just loved your article on the beginners guide to starting a blog.If somebody take this blog article seriously in their life, he/she can earn his living by doing blogging.thank you for thizs article. best devops online training

    ReplyDelete
  3. Very informative post! There is a lot of information here that can help any business get started with a successful social networking campaign. Admond Lee

    ReplyDelete
  4. I admire this article for the well-researched content and excellent wording. I got so involved in this material that I couldn’t stop reading. I am impressed with your work and skill. Thank you so much. anaconda install tensorflow

    ReplyDelete
  5. I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

    Digital marketing course

    ReplyDelete
  6. Thanks for the informative article. This is one of the best resources I have found in quite some time. Nicely written and great info. I really cannot thank you enough for sharing.

    AWS Online Training
    AWS Certification Training
    AWS Certification Course Online
    AWS Training
    AWS Online Course

    ReplyDelete
  7. I recently found many useful information in your website especially this blog page. Among the lots of comments on your articles. Thanks for sharing. apps

    ReplyDelete
  8. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.

    data science course

    ReplyDelete
  9. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data Science Course

    ReplyDelete
  10. This is very significant, and yet necessary towards just click this unique backlink: northshore connect to success

    ReplyDelete
  11. Wow, cool post. I'd like to write like this too - taking time and real hard work to make a great article... but I put things off too much and never seem to get started. Thanks though. Tableau Data Blending

    ReplyDelete
  12. I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

    Correlation vs Covariance

    ReplyDelete
  13. Thanks for the valuable information and insights you have so provided here... buy instagram likes cheap paypal

    ReplyDelete
  14. I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.

    data analytics courses hyderabad
    data science training
    business analytics course

    ReplyDelete
  15. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!


    business analytics course
    data science course in hyderabad
    data analytics courses hyderabad

    ReplyDelete
  16. I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!
    data science training in guwahati

    ReplyDelete
  17. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression

    ReplyDelete
  18. It is the kind of information I have been trying to find. Thank you for writing this information. It has proved utmost beneficial for me. fa├ža o download do Appvn apk 2020

    ReplyDelete
  19. I really like your take on the issue. I now have a clear idea on what this matter is all about.. acesse aqui para baixar Lucky Patcher 2020

    ReplyDelete
  20. I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  21. That is really nice to hear. thank you for the update and good luck. blackmart alpha 0 49 b 93 apk

    ReplyDelete
  22. This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value of providing a quality resource for free. visit site

    ReplyDelete
  23. Very interesting blog. Many blogs I see these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome.You can also check my articles as well.

    Data Science In Banglore With Placements
    Data Science Course In Bangalore
    Data Science Training In Bangalore
    Best Data Science Courses In Bangalore
    Data Science Institute In Bangalore

    Thank you..

    ReplyDelete
  24. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  25. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete