Hi, I'm one of the project creators. Continuous Machine Learning (CML) is an open source project to help ML projects use CI/CD with Github Actions and Gitlab CI (https://github.com/iterative/cml).
CML automatically generates human-readable reports with metrics and data viz in every pull/merge request, and helps you use storage and GPU/CPU resources from cloud services. CML addresses three hurdles for making ML compatible with CI:
1. In ML, pass/fail tests aren’t enough. Understanding model performance might require data visualizations and detailed metric reports. CML automatically generates custom reports after every CI run with visual elements like tables and graphs. You can even get a Tensorboard.dev link as part of your report.
2. Dataset changes need to trigger feedback just like source code. CML works with DVC so dataset changes trigger automatic training and testing.
3.Hardware for ML is an ecosystem in itself. We’ve developed use cases with CML and Docker Machine to automatically provision and deploy cloud compute instances (CPU & GPU) for model training.
Our philosophy is that ML projects- and MLOps practices- should be built on top of traditional software tools and CI systems, and not as a separate platform. Our goal is to extend DevOps’ wins from software development to ML. Check out our project site (https://cml.dev) and repo, and please let us know what you think!
This looks really amazing. Do you have any plans to support on-prem GPUs? If we had Gitlab runners with GPUs in them would this project pick it up and use those runners for training/analysis?
Hey! Disclaimer - I'm one of the DVC maintainers :) Super excited for the team on this release!
For the last two years we have seen over and over again how our users take DVC and use it inside Gitlab, Github, etc. This product was born partially as a result of these discussions, partially as an initial visions for the ML tools ecosystem - Hashicorp-like.
Having A software engineering background I really hope that integrating ML workflow into engineering tools will be the future of this space. And with CML and other tools (e.g. https://github.blog/2020-06-17-using-github-actions-for-mlop...) we see this happening.
If you want to show the results in the Merge Request itself like we do for things like test results and security scans please let us know. Open an issue and if there is no response my Twitter handle is @sytses
2. If training fails, you'll be notified that your run failed in the GitHub Action dashboard (or GitLab CI/CD dashboard). See here for some real life examples of failure ;) : https://github.com/iterative/cml_cloud_case/actions
3. CML reports are markdown documents, so you can write any kind of text to them. If your metrics are output in a file `metrics.txt`, you can have your runner execute `cat metrics.txt >> report.md` and then have CML pass on the report to GitHub/GitLab. Likewise, any graphing library is supported because you can add standard image files (.png, .jpg) to the report. So custom metrics and custom graphs. We like DVC for managing and plotting metrics, but we're biased because we also maintain it.
4. Yep, GitHub Actions is pretty powerful and flexible. Works with whatever external services you can connect to your Action!
I think taking the approach of "help[ing] your team make informed, data-driven decisions" through generating reports is valuable here. In my opinion, it goes too far if we start continuously deploying ML code like it's a SWE project. To take an example in the case of autonomous vehicles, pushing continuous updates to perception modules without thoroughly exploring the ramifications of an update could be potentially catastrophic.
Obviously we can't predict every error by thinking hard, but datasets will never serve as a full representation of what models might experience in the real world. Continuous deployment to an ML model could affect undefined behavior in unpredictable ways.
Definitely- automatic model deployment would only work for a subset of applications. We think a lot more about having models available as candidates for deployment, which can then be inspected by domain experts on a team.
One of our motivations for building visual reports that appear like comments in a pull request is giving teams metrics & info to discuss when deciding if merge is right. That way, the automated part is training and testing, but the decision making is human (i.e., data scientists whose skills are better used interpreting models & data than running repetitive training scripts).
Agreed! I think, it's similar to software engineering. It's not that often that large websites are being deployed to prod without someone approving/QA-ing it manually first, right?
CI/CD systems in this case help automating this as much as possible, but do not completely replace decision making process, I would say.
ML is a relatively young field, and decades behind Software Engineering in terms of best practices for running production systems. CI/CD massively improved the innovation cycle time and quality of production software, and I believe it is key for building robust production ML systems as well. CML looks like a really easy to use product for bringing CI/CD to ML projects.
Thanks for the kind words! DevOps has had profound results and we think culturally and technically, it's the right moment to push for those practices in ML.
Out of curiosity, what kind of research? I'm very interested in getting DVC and Git more broadly into academic research circles, but finding a lot of barriers in my home field.
Longtime DVC user here - this is going to be so helpful. We use DVC for all of our model and data versioning, but what's been missing is the ability to cleanly integrate that into our CI workflow. Looks like that's solved now! The cml.yaml syntax also looks quite nice, very easy to follow. Looking forward to trying this out.
CML looks really awesome. I've been on one of your online meetups. Are you planning to host more in the future? It would be great to learn real production use cases!
Yes, we're aiming to have another at the end of July! If you have a particular use case you're interested in, we'd love to know. We might be able to develop some materials around it.
rhythmvertigo|5 years ago
CML automatically generates human-readable reports with metrics and data viz in every pull/merge request, and helps you use storage and GPU/CPU resources from cloud services. CML addresses three hurdles for making ML compatible with CI:
1. In ML, pass/fail tests aren’t enough. Understanding model performance might require data visualizations and detailed metric reports. CML automatically generates custom reports after every CI run with visual elements like tables and graphs. You can even get a Tensorboard.dev link as part of your report.
2. Dataset changes need to trigger feedback just like source code. CML works with DVC so dataset changes trigger automatic training and testing.
3.Hardware for ML is an ecosystem in itself. We’ve developed use cases with CML and Docker Machine to automatically provision and deploy cloud compute instances (CPU & GPU) for model training.
Our philosophy is that ML projects- and MLOps practices- should be built on top of traditional software tools and CI systems, and not as a separate platform. Our goal is to extend DevOps’ wins from software development to ML. Check out our project site (https://cml.dev) and repo, and please let us know what you think!
doppenhe|5 years ago
Here is the missing part for a total e2e solution: https://github.com/marketplace/actions/algorithmia-ci-cd
{disclaimer, we built this Github action}
calebkaiser|5 years ago
gravypod|5 years ago
ishcheklein|5 years ago
For the last two years we have seen over and over again how our users take DVC and use it inside Gitlab, Github, etc. This product was born partially as a result of these discussions, partially as an initial visions for the ML tools ecosystem - Hashicorp-like.
Having A software engineering background I really hope that integrating ML workflow into engineering tools will be the future of this space. And with CML and other tools (e.g. https://github.blog/2020-06-17-using-github-actions-for-mlop...) we see this happening.
sytse|5 years ago
I've added integrating CML closer into GitLab to our direction page with https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...
If you want to show the results in the Merge Request itself like we do for things like test results and security scans please let us know. Open an issue and if there is no response my Twitter handle is @sytses
toisanji|5 years ago
1) can we see examples of generated reports?
2) what happens if training fails?
3) what kind of metrics can it graph? can we have it track our custom metrics?
4) can we connect with external services like with webhooks,slack, and other integrations.
5) is this a docker technology, or how does it deal with images and dependencies?
Great work!
rhythmvertigo|5 years ago
1. Yes! Let me link some reports and example repos:
- A basic classification problem with scikit learn: https://github.com/iterative/cml_base_case/pull/2
- CML with DVC & Vega-Lite graphs: https://github.com/iterative/cml_dvc_case/pull/4
- Neural style transfer with EC2 GPU: https://github.com/iterative/cml_cloud_case/pull/2
2. If training fails, you'll be notified that your run failed in the GitHub Action dashboard (or GitLab CI/CD dashboard). See here for some real life examples of failure ;) : https://github.com/iterative/cml_cloud_case/actions
3. CML reports are markdown documents, so you can write any kind of text to them. If your metrics are output in a file `metrics.txt`, you can have your runner execute `cat metrics.txt >> report.md` and then have CML pass on the report to GitHub/GitLab. Likewise, any graphing library is supported because you can add standard image files (.png, .jpg) to the report. So custom metrics and custom graphs. We like DVC for managing and plotting metrics, but we're biased because we also maintain it.
4. Yep, GitHub Actions is pretty powerful and flexible. Works with whatever external services you can connect to your Action!
5. It's not strictly a Docker technology. We use Docker images preinstalled with the CML library in our examples, but you can just install the library with npm in your own image. https://github.com/iterative/cml#using-your-own-docker-image
Let me know if there's anything else I can tell you about
bigfoot675|5 years ago
Obviously we can't predict every error by thinking hard, but datasets will never serve as a full representation of what models might experience in the real world. Continuous deployment to an ML model could affect undefined behavior in unpredictable ways.
rhythmvertigo|5 years ago
One of our motivations for building visual reports that appear like comments in a pull request is giving teams metrics & info to discuss when deciding if merge is right. That way, the automated part is training and testing, but the decision making is human (i.e., data scientists whose skills are better used interpreting models & data than running repetitive training scripts).
ishcheklein|5 years ago
CI/CD systems in this case help automating this as much as possible, but do not completely replace decision making process, I would say.
tknaup|5 years ago
obowersa|5 years ago
The CD Foundation has a SIG around MLOps which is pretty active and has some awesome folk participating.
For anyone who's interested in this space, there's some more detail here: https://cd.foundation/blog/2020/02/11/announcing-the-cd-foun...
rhythmvertigo|5 years ago
tnachen|5 years ago
rhythmvertigo|5 years ago
ayanb9440|5 years ago
rhythmvertigo|5 years ago
rkaplan|5 years ago
rhythmvertigo|5 years ago
m0sth8|5 years ago
m0sth8|5 years ago
rhythmvertigo|5 years ago
skuda|5 years ago
[deleted]