[rhythmvertigo]

Hi, I'm one of the project creators. Continuous Machine Learning (CML) is an open source project to help ML projects use CI/CD with Github Actions and Gitlab CI (https://github.com/iterative/cml).

CML automatically generates human-readable reports with metrics and data viz in every pull/merge request, and helps you use storage and GPU/CPU resources from cloud services. CML addresses three hurdles for making ML compatible with CI:

1. In ML, pass/fail tests aren’t enough. Understanding model performance might require data visualizations and detailed metric reports. CML automatically generates custom reports after every CI run with visual elements like tables and graphs. You can even get a Tensorboard.dev link as part of your report.

2. Dataset changes need to trigger feedback just like source code. CML works with DVC so dataset changes trigger automatic training and testing.

3.Hardware for ML is an ecosystem in itself. We’ve developed use cases with CML and Docker Machine to automatically provision and deploy cloud compute instances (CPU & GPU) for model training.

Our philosophy is that ML projects- and MLOps practices- should be built on top of traditional software tools and CI systems, and not as a separate platform. Our goal is to extend DevOps’ wins from software development to ML. Check out our project site (https://cml.dev) and repo, and please let us know what you think!

[doppenhe]

Deployment, inference and management can participate in this as well!

Here is the missing part for a total e2e solution: https://github.com/marketplace/actions/algorithmia-ci-cd

{disclaimer, we built this Github action}

[davidortega]

Hi doppenhe, we have that part already implemented using cml-send-github-check and dvc metrics diff. You can compare the metric that you prefer with dvc and then just set the status of the github check uploading your full report. Of course, you can also fail the workflow as your Github action does, but I think is more useful to see it as a report in the check.

disclaimer: I'm work with CML

[rhythmvertigo]

coooool! going to try this out :)

[calebkaiser]

This is really cool. We've been recommending DVC to our users for a long time, and this looks like a natural step forward for the Iterative ecosystem.

[rhythmvertigo]

Yeah, we were seeing a lot of users organically asking about CI/CD. DVC had to come first to address some of the data management issues, which seemed like the biggest hurdle to CI. But we're excited to add this now.

[gravypod]

This looks really amazing. Do you have any plans to support on-prem GPUs? If we had Gitlab runners with GPUs in them would this project pick it up and use those runners for training/analysis?

[davidortega]

You can also deploy runners with GPU on premise using CML docker image with GPU already supported having to install only the ndivia drivers and nvidia-docker in your machine.

docker run --name myrunner -d -e RUNNER_IDLE_TIMEOUT=1800 -e RUNNER_LABELS=cml -e RUNNER_REPO=$my_repo_url -e repo_token=$my_repo_token dvcorg/cml-gpu-py3-cloud-runner

It works for Gitlab and Github. Just only point your url and repo token

[rhythmvertigo]

Yep, it will work with your GPUs! You just set up self-hosted runners as usual https://docs.gitlab.com/runner/

GitHub & GitLab have both made it quite easy to use your own resources as runners. I recently met someone who was doing Actions with a Jetson Nano on their dresser :)