I wrote a blog post about this a couple weeks ago [1], and I think as a result of writing it in the midst of the week-long annoyance that I had at having to quickly move my active projects over to GH Actions, I may have been a bit too negative in my overall tone.
I do like that this post expresses the gratitude for many years of free OSS CI that Travis did provide, and I especially want to echo that and thank the original founders and engineers who helped grow Travis CI and the entire idea of 'every project gets CI' that is now prevalent (in most professional realms), just like the previous generation embraced the idea of 'every project is in a VCS'.
It's too bad what's happened, I think, but Travis does have a great legacy.
I think in general we are tuned to have a stronger reaction to unexpected setbacks. The people who have no aversion to them end up dying by a thousand cuts, as a long series of predictable but ignored scenarios play out one after another. In the most delusional cases, blaming it all on bad luck instead of lack of foresight.
Not every setback is a pattern, but if you have no histamine response at all then you will miss a lot of preventable issues.
We (radareorg/radare2 team) use Travis CI specifically to build and run tests on ARM64 (ARMv8 or AArch64), PowerPC64, and SystemZ (s390)[1]. No other CI service offers those. I wish there were also MIPS, SPARC, and RISC-V.
I work with an important open source project with large full-time professional and competent teams working on it... that had no idea .org as shutting down until a couple of weeks ago. Their messaging wasn't enough.
Mving may require significant effort for products with sophisticated configurations.
Also, while I don't imply either to move or to stay, Travis still does the job as it did before¹, so, unless pricing or specific features are a requirement, the technical incentive to move may (may) not be high enough.
¹=I did notice one change though, and it's that the list of the builds slowed down at some point; their solution: reduce the entries listed while loading (!!).
Gitlab is a pioneer here. The did start with the 2000 minutes free of shared runners, for private and public repos. I really think we should all thank them for pushing ahead on innovation.
Gitlab invented this, github copied.
Same as Snapchat invented, Instagram copied them.
I don't know about the timeline between when Gitlab CI started vs Travis, and gitlab certainly made the innovation of integrating it into the code host, but certainly Travis and Circle CI were popular before Gitlab was.
This makes me wonder what Travis' endgame here is. When Github and Gitlab, the two sites that account for the vast majority of OSS development, offer free CI, who's their target market? Enterprise? They already use Jenkins or Bamboo (or Github Enterprise or Gitlab self hosted).
Probably cut costs and hope enough of the revenue sticks around until they’re profitable. GitHub Actions is a relative newcomer to the party, but they probably have the better pricing structure here, charging by the minute.
I moved to github actions (GHA) for an R package, and found these benefits of GHA over travisCI:
1. It's much faster. I think this is because it does not have to build up an environment from scratch, but instead caches some elements.
2. It is more reliable. Those early steps that travisCI takes in building up an environment seem to be error-prone, leading to failures long before it got around to checking my code. Usually, these were bad-signature or failure-to-download problems with basic libraries, and they tended to go away after a few days or a week, but in the meantime, my "badges" made it look as though my code had failed tests, when it was more that the tests were failing of their own accord.
And the disadvantages of the switch? None that I've noticed, so far. Perhaps the bloom will go off this rose as well, but that's life in the software world.
(Paraphrasing) "Therefore, we find ourselves looking to other open-source friendly solutions, the greatest of which currently seems to be the one by Microsoft."
What I would do is find a way to decouple your test running system from the lifecycle tooling. Develop a custom solution that run CI pipelines in k8s and collates results and passes them upstream to the lifecycle tooling whether it's GitHub Actions or Jenkins or whatever. By doing this you free your pipelines from lock-in and can utilize cheap compute (DigitalOcean, GKE pre-emptibles, etc).
Take a look at Tekton CI/CD for a good primitive that you can build on.
Just having your CI steps run scripts that install/build/deploy/??? already goes a long way to decoupling from your CI solution, and can be a lot faster than spinning up a custom image for your CI build. Custom images don't necessairly do much for fixing the last 10% either - I'll typically want CI-specific:
1. Build matricies. I was running windows builds on appveyor and linux builds on travis for a long time (faster builds / iteration / feedback.) Regression testing against multiple rustc versions can be done from within a script, but doing so via CI-configuration makes for easier to read CI results that flag individual builds as failed instead of an entire script that you have to go log diving into.
2. Custom runners. Be it custom hardware or licensing-burdened custom software, I frequently need to own the actual CI hardware, and using the native solutions for CI runners is typically easier than merely configuring whatever custom infrastructure I might come up with.
3. Images. Using prewarmed cloud-ci-specific images is often faster to spin up.
Buildkite is BYO infrastructure. It's worked really well for us, as we have a fairly idiosyncratic build process/test suite that out of the box solutions like GH Actions or CircleCI don't really cater to.
As long as you are looking for “free and someone else maintains it” you are going to be hopping services regularly to stay on something that has VC money to burn for goodwill.
If you host it yourself you either have a maintenance headache or a shortage of features.
Depends what your time is worth really; I use buildkite with runners on AWS for work, and it doesn’t suck.
GitLab CI is nice, if you're comfortable with GitHub as your primary remote, it's easy enough to set up a secondary remote (and have most git actions duplex to both).
We’ve been running Gitlab on-prem for around 6 years. We run self-hosted Gitlab CI runners on an on-prem Kubernetes cluster. We’re a team of around 20 and collectively execute around 1000 Gitlab CI jobs every day across numerous client projects for web, mobile, cloud deployments, etc. It works amazingly well.
I know this isn't super helpful, but I built my own bors-style CI bot over a few weekends, and I'm very satisfied with it. It looks for PR comments with a specific keyword, then pulls the branch, tests it, and pushes a merge commit automatically.
Taking this approach has two big upsides. First, the bot is just a binary running on a cheap VPS; so I know that it'll be fast and that I can ssh in at any time to debug if necessary. Second, if there's a feature that I want, I can just add it, rather than twiddling my thumbs. For example, I noticed that I was manually deleting my merged PR branches every time, so I added a few lines to the bot and now it deletes them automatically.
The obvious downsides of this approach are: 1) Implementing even an MVP of such a bot can consume significant time and energy, which you may not have a lot of; 2) If there's a bug, it's your fault and now you need to spend more time and energy tracking it down; and 3) Your bot might have a security vulnerability that exposes secrets, allows injection attacks, etc.
I'd love to see tooling/frameworks that make it easier to create custom CI systems. I think startups (and perhaps larger companies as well) could benefit a lot from building a CI in-house, since it allows you to optimize for your own specific needs. I see a lot of parallels to code linters, where spending a bit of time writing custom lint rules or static analysis tools can have a large payoff.
I also moved from Travis CI to GitHub Actions and was pleased with the results. It wasn't mentioned in the article, but GitHub Actions is free for both public and private repos. To my knowledge, Travis CI was never free for private repos.
There are limits on GHA to the number of minutes for private repos when not using self-hosted runners (currently 2_000 minutes per month allocated per org/user when using "Github Free")
I am/was using Travis for my open source projects. I started noticing my builds taking forever. I did some checking around and decided its just easier to move to github actions rather than figuring things out.
We are also moving away from Travis for openfaas, k3sup, arkade and inlets. That's about 50 repos and it's unfortunately taking the community away from delivering actual value.
[+] [-] geerlingguy|5 years ago|reply
I do like that this post expresses the gratitude for many years of free OSS CI that Travis did provide, and I especially want to echo that and thank the original founders and engineers who helped grow Travis CI and the entire idea of 'every project gets CI' that is now prevalent (in most professional realms), just like the previous generation embraced the idea of 'every project is in a VCS'.
It's too bad what's happened, I think, but Travis does have a great legacy.
[1] https://www.jeffgeerling.com/blog/2020/travis-cis-new-pricin...
[+] [-] hinkley|5 years ago|reply
Not every setback is a pattern, but if you have no histamine response at all then you will miss a lot of preventable issues.
[+] [-] cube00|5 years ago|reply
We really need to improve the "annoying authentication" so this doesn't become a reason to continue building silos with a single vendor.
[+] [-] Spivak|5 years ago|reply
[+] [-] jniedrauer|5 years ago|reply
[+] [-] xvilka|5 years ago|reply
[1] https://travis-ci.com/github/radareorg/radare2/builds/202966...
[+] [-] chrisseaton|5 years ago|reply
[+] [-] cortesoft|5 years ago|reply
[+] [-] pizza234|5 years ago|reply
Also, while I don't imply either to move or to stay, Travis still does the job as it did before¹, so, unless pricing or specific features are a requirement, the technical incentive to move may (may) not be high enough.
¹=I did notice one change though, and it's that the list of the builds slowed down at some point; their solution: reduce the entries listed while loading (!!).
[+] [-] kimsant|5 years ago|reply
Gitlab invented this, github copied. Same as Snapchat invented, Instagram copied them.
[+] [-] Macha|5 years ago|reply
[+] [-] hendry|5 years ago|reply
Have we learned nothing?
[+] [-] scrollaway|5 years ago|reply
What will you have us do, in the mean time? Will you pay to run my open source projects' CI servers?
And if you do, what exactly makes you better than "another Microsoft monopoly"?
[+] [-] joshxyz|5 years ago|reply
[+] [-] Karunamon|5 years ago|reply
[+] [-] philihp|5 years ago|reply
[+] [-] bluenose69|5 years ago|reply
And the disadvantages of the switch? None that I've noticed, so far. Perhaps the bloom will go off this rose as well, but that's life in the software world.
[+] [-] tpoacher|5 years ago|reply
Honestly, we live in weird times.
[+] [-] 0x1F8B|5 years ago|reply
Take a look at Tekton CI/CD for a good primitive that you can build on.
[+] [-] MaulingMonkey|5 years ago|reply
1. Build matricies. I was running windows builds on appveyor and linux builds on travis for a long time (faster builds / iteration / feedback.) Regression testing against multiple rustc versions can be done from within a script, but doing so via CI-configuration makes for easier to read CI results that flag individual builds as failed instead of an entire script that you have to go log diving into.
2. Custom runners. Be it custom hardware or licensing-burdened custom software, I frequently need to own the actual CI hardware, and using the native solutions for CI runners is typically easier than merely configuring whatever custom infrastructure I might come up with.
3. Images. Using prewarmed cloud-ci-specific images is often faster to spin up.
[+] [-] toomanybeersies|5 years ago|reply
[+] [-] koeng|5 years ago|reply
[+] [-] danielheath|5 years ago|reply
As long as you are looking for “free and someone else maintains it” you are going to be hopping services regularly to stay on something that has VC money to burn for goodwill.
If you host it yourself you either have a maintenance headache or a shortage of features.
Depends what your time is worth really; I use buildkite with runners on AWS for work, and it doesn’t suck.
For personal stuff it’s harder to pick.
[+] [-] jjgreen|5 years ago|reply
[+] [-] bottled_poe|5 years ago|reply
[+] [-] nemo1618|5 years ago|reply
Taking this approach has two big upsides. First, the bot is just a binary running on a cheap VPS; so I know that it'll be fast and that I can ssh in at any time to debug if necessary. Second, if there's a feature that I want, I can just add it, rather than twiddling my thumbs. For example, I noticed that I was manually deleting my merged PR branches every time, so I added a few lines to the bot and now it deletes them automatically.
The obvious downsides of this approach are: 1) Implementing even an MVP of such a bot can consume significant time and energy, which you may not have a lot of; 2) If there's a bug, it's your fault and now you need to spend more time and energy tracking it down; and 3) Your bot might have a security vulnerability that exposes secrets, allows injection attacks, etc.
I'd love to see tooling/frameworks that make it easier to create custom CI systems. I think startups (and perhaps larger companies as well) could benefit a lot from building a CI in-house, since it allows you to optimize for your own specific needs. I see a lot of parallels to code linters, where spending a bit of time writing custom lint rules or static analysis tools can have a large payoff.
[+] [-] remram|5 years ago|reply
[+] [-] cghendrix|5 years ago|reply
[+] [-] koirapoika|5 years ago|reply
[+] [-] roylez|5 years ago|reply
[+] [-] tcbasche|5 years ago|reply
[+] [-] q3k|5 years ago|reply
[+] [-] hgs3|5 years ago|reply
[+] [-] codys|5 years ago|reply
[+] [-] systematical|5 years ago|reply
It will be a slow but easy migration.
[+] [-] alexellisuk|5 years ago|reply
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] jjshsb|5 years ago|reply