> How often have you or a coworker pushed an empty commit to "re-trigger" the CI?
Zero, because unless my dog configured the CI, I’d just click a button to re-run it on the same PR/commit/branch and not “make an empty commit”.
End to end testing of large systems with hundreds or thousands of man-years of code in them will be slow because there is a large surface area to test and testing in individual modules doesn’t give the same kind of end to end coverage that testing everything together does. So while pre-commit would be good, I doubt it will replace server CI for all but the smallest repositories.
Agreed, but for instance, I am working on a project hosted on AWS where the trigger build action is not allowed for my user. Empty commit is my only option.
I've seen issues before, where Jenkins agents "go bad" (Gradle cache corruption, Docker filesystem corruption, etc) – pressing the "re-run" button in the UI usually runs the job again on the same Jenkins agent, and good chance it fails again in the exact same way. What I've discovered, is if I restart the job twice, in quick succession, the first re-run will go to the same agent (and will likely fail with the same problem), the second to a different one (hopefully one without that issue). Easiest way to restart it twice, is `git commit --allow-empty -m rerun && git push`, wait until the job starts, then repeat. Maybe there is some way to do the same thing in the UI, but pushing a couple of empty commits is quick and I know it works.
I always have to be careful responding to things like this because I get sucked into the trap of thinking “we” means myself, and it just needs to mean “most of us” for whatever they’re talking about to be worrisome or even true.
I do worry that a lot of people are missing the first principles where CI is concerned, but a few of us still remember. Whether that’s enough is always a question.
> First, CI rarely knows what has changed, so everything needs to be rebuilt and retested
CI is the impartial judge in your system. That is the ultimate goal and much of the value. Don’t believe me when I say that your check in broke something. Believe the robot. It does not think, it does not interpret errors (which is why intermittent failures poison the well). It just knows that whatever you did was not good.
We make it stupid simple because the more ways you try to interpret things, the higher the likelihood the system will either miss an error or report imaginary ones. And in this case I am very sorry Mr Lasso, but it is in fact the hope that kills. Optimism makes us try things, things others would give up on. That’s what makes a piece of software good. Better. But too much and you start ignoring very real problems and make optimism into a liability. I’ve seen it over and over again, take a build with intermittent failures and people will check in broken code and assume it’s the intermittent problem and not their change.
Ideally, when you start breaking builds you’ve found your limits. But the build has to break when I break the code, and it needs to do it repeatably, because if I can’t reproduce the error I risk telling myself stories, and foisting off debugging to other people, which is a big no no.
What if it was possible to have a local build system that would run tests etc and provide a cryptographically secure hash proving that the tests passed, the build was good, etc then a “centralized” CI system just verified that before permitting merge/deploy.
There’s a lot of hand waving there but it doesn’t seem wildly implausible.
Local use of containers or something like nixos could provide sufficient protection against the “works on my machine but not in production” problem.
I fail to see the point being made in the article. CI done right is the best thing that ever happened to me in terms of collaborative productivity.
Of course there are many pitfalls. Like with any tool it becomes trivial to use it wrong. Integrating less than once per day and deferring all testing and linting to the CI process is an anti-pattern and the issue is not the CI process but how the team chooses to use it.
For example I worked in a team of 10 and we were doing multiple production deploys per day. This was made possible by a great CI workflow. Everyone was running tests precommit and focusing on keeping the pipeline green. Yes I've seen the opposite as well, but that usually is a symptom of issues with the team not the general concept of CI.
Not saying that there isn't room for improvement but all the real-time collaboration features I used on low-code platforms feel like a step back.
> deferring all testing and linting to the CI process is an anti-pattern
I'm confused as to why this is an anti-pattern? My understanding is that the CI pipeline should run unit tests and linting for every commit. But at the same time, developers should run their tests before pushing code.
I think the point is about far shorter feedback loops than possible with today's pervasive tech.
> something even more continuous than continuous integration. It might look like real-time integration of your changes, collaborative editing, or smarter build systems that produce near-live feedback
It seems like a core argument is the pre-commit tests that runs as commit hooks on the developers computer.
I have worked in a place where they did that, and I think the cons heavily outweighed the pros. I can not push incomplete work to a remote, I can not separate feature development and chores (eg. linting) because I _need_ to fix stuff in order to commit an push, etc.
> Continuous Integration is a model thats tightly tied to how we run version control.
I would say that a pre-commit testing world is much tighter. CI, as many know it, as a passive observer. When stuff fails, it will let you know. You can still merge.
One thing that would be nice, however, would be the ability to run the entire pipeline locally. For GitHub actions, it indeed seems like there are viable options to do that.
> One thing that would be nice, however, would be the ability to run the entire pipeline locally. For GitHub actions, it indeed seems like there are viable options to do that.
I prefer not having to replicate locally how GitHub runs GitHub actions, but rather just make my GitHub actions run something that I know I can run locally. So all the complicated stuff is contained in the script that I can run locally, and GitHub actions is just a dumb “runner” of this script.
For my local script I prefer using Nix, since it’s the best way I know to guarantee a consistent environment between different machines.
> One thing that would be nice, however, would be the ability to run the entire pipeline locally.
This cost me many hours of waiting for the Gitlab CI runner when debugging non-trivial pipelines, when the issue was something that did not have to do with the script steps inside of the jobs but rather how the Gitlab runner handled things.
I've found gitlab-ci-local [1] which actually does run the Gitlab pipeline locally, although I had to write some boilerplaye scripts to set up all the necessary 'CI_FOO_SOMETHING' environment variables before running the tool. (Which sometimes came back to bite me because the issue was actually in the content of some of those environment variables). It's still a very good tool.
git has cli flags that will happily let you bypass pre commit/push checks so it's not even a reliable way to know that tests actually got run. Central CI gives you a traceable test history of every change and keeps us all honest.
We use nektos act to run pipelines locally. Works OK, but everytime something fails locally you are left wondering if its act or your pipeline, and some features (like reusable workflows) are not implemented yet.
Maybe the right thing is to just write the workflows in something else, and have the github workflow file be a single call to your script. But
-It would be nice to be able to use github actions others has made (or libraries/abstractions in general)
- I don't see how to easily get parallell execution.
- I love github environments, how do I pass down all my environments and their variables?
"One thing that would be nice, however, would be the ability to run the entire pipeline locally."
I'm working on something to do just this. Although I've redefined what "local" means in this context. I'm still using remote servers but everything is happening pre-commit from the terminal. If you are interested check out this demo and let me know if you have any feedback https://brisktest.com/demos
I do feel like if we’re going the precommit route we need to go much deeper into things like Bazel. Loads of companies are using CI to get a huge test suite to run with a lot of parralelsim for a reason!
Though for me it’s also like… so many people do unspeakable horrors to their machine setups that I like there being a consistent runner in CI.
But “CI is to run tests” in the world of CD is a bit of a simplification anyways.
It sounds like a problem with the policy, not the tool. Pre-commit hooks is a useful tool but there should be a way to skip them if you really need it.
Per-commit hooks save my time by not allowing to push code which guaranteed to break later in CI.
> But all technology evolves, and git is already showing its weaknesses. Fork-and-pull isn't the final form of development workflows.
I kind of get what this is saying, but technology evolution doesn't have to mean completely replacing said technology with something else.
I think that's one weird thing about the software field, whereby we keep moving to these shiny new things that we think are better than the tools of yesteryear, yet in the end there is only a marginal gain in productivity.
Fork and pull is an incredibly productive and powerful workflow. CI is incredibly, incredibly useful. If these things were not the case, then neither of these would be even discussed by this article. There is a reason for their success - and it's not because GitHub is the most ubiquitous code hosting service out there. Git is _actually_ pretty great. CI is _actually_ very useful and has secured codebases for decades at this point.
So if one were to proclaim the "End of CI" I really need to see a viable alternative that addresses the same problems as CI and significantly improves upon it. An incremental improvement is not enough to shift and rewrite everything - there needs to be a significant jump in ability, productivity, security, or something else in order for me (and I imagine many others) to consider it.
> I think that's one weird thing about the software field, whereby we keep moving to these shiny new things that we think are better than the tools of yesteryear, yet in the end there is only a marginal gain in productivity.
"Thought leaders" need to he constantly talking up the next new thing so that they can stay ahead of the herd on socia media.
Developers get bored, or worry that their career is stagnating, if they're not using the new shiny. Particularly if they pay attention to the thought leaders, or they're stuck building unglamorous crud apps.
And the software industry generally has a poor collective memory of tools and practices and experience from even the recent past. Contrast with more mature engineering disciplines, or architecture, medicine, etc. I'm not sure why this is.
> I think that's one weird thing about the software field, whereby we keep moving to these shiny new things that we think are better than the tools of yesteryear, yet in the end there is only a marginal gain in productivity.
Agreed. But I'd take it a step further and say:
...yet in the end no one is any happier. Not engineers. Not management. Not leadership. And most importantly not the users of the product.
We keep building and delivering more. But how often is that better? Either as an end to end experience or simply fewer bugs? Most of us - who are at some point users of products we didn't build - have resigned ourselves to that fact (read: it's been normalized) that an uncomfortable amount of friction is a given. That's sad.
What the author should have made more clear is that we’re starting to see technology that eliminates the “local developer environment” completely, and we’re also starting to see collaborative real-time code editors. Taken together, we can imagine a world where a dev points their browser at codebase hosted somewhere and just starts editing. Each edit is a change event that the system reacts to by running compilers, then unit tests, etc. and provides near real-time feedback that the code works/doesn’t work, right in the editor. In this world the developer does not push their changes, the system does that for them, while they’re working. The CI server disappears from view completely.
> I kind of get what this is saying, but technology evolution doesn't have to mean completely replacing said technology with something else.
Technology and in general historical trends are rarely broken, they change but there is no gaps in sense, I don't remember who said it, someone in defense R&D, if recall correctly.
"Fork-and-pull isn't the final form of development workflows."
Actually, like the wheel, I think it is the final form. When artificially sentient machines are writing code for us and someone or some thing pollutes or corrupts a critical model; a human will need so step in and fork where things were working to fix it.
What's that old saying? Don't remove a fence unless you really know why it's there.
CI isn't just a remote compute scheduler. Its also the place that contains tooling, tooling which repeated across a dev team creates n * jobs replicas of that tooling - which run on a local machine would be fairly wasteful. Its also centralized because CI is one of the biggest vectors an organization runs; it contains credentials for outside services, credentials for deployment (which, if you're SOX-bound cannot exist on a dev machine), and if you're cryptographically signing binaries and doing reproducible builds it's got those keys too. n * developer laptops makes that vector a planet sized blot. A lot of the policies I imagine dealing with that would be limiting developer access to their machines, limiting tooling to only approved or internal tooling.
Some of this is good, and some of it I feel makes some fundamental misunderstandings. The good: "tight feedback loops are really important". Absolutely. Getting feedback about a change you (as a developer) are making is key, and the longer that takes the more time is spent waiting which is annoying and concentration breaking. This is, in my experience, usually tied to the implementation of tests (read: not bloated and useless) and not really a core tenant of CI.
However, things like:
[...] git is already showing its weaknesses. Fork-and-pull isn't the final form of development workflows.
Show that the author is perhaps conflating GitHub with Git.
I disagree with the POV of the article. That said, I do work at Spacelift[0], a specialized CI/CD for Infra as Code (so you can either take this with a grain of salt, or more trust, due to the increased exposure).
> First, CI rarely knows what has changed, so everything needs to be rebuilt and retested.
Yes, this is an issue with some CI/CD systems, an issue you can solve however. See our Push Policies[1] based on Open Policy Agent. Test-case caching is also sometimes available (available in i.e. Go).
> Running tests pre-commit is one answer.
Running tests locally for a quick feedback loop - sure, that's fairly mainstream though (something you can use our local preview[2] for as well in the case of IaC). Running tests locally before directly deploying to prod - that would be a compliance and security nightmare.
The author presents what looks to me like a very "move fast, break things" attitude, that doesn't really work in a whole lot of cases.
If your CI/CD is slowing you down, make it faster. Easier said than done, I know, but a lot of people don't even think about optimizing their CI/CD, which you often can do, by being smarter about what you run and/or parallelizing extensively and/or caching[3].
This take doesn’t make any sense. What is the author suggest as a replacement, hooking your IDE up to prod via FTP and editing PHP files directly on the live server? Sure I’ve done this, as a solo developer on smaller projects, but this doesn’t scale even to a team of two or three, and it goes without saying I’m not relying on tests in this scenario and I have basically no ability to roll back.
“CI is frustrating!!” I hear ya, but this article does nothing to clarify a viable alternative.
A poor title; I'd say "The current CI is inadequate, we need to begin with a new CI".
Why is the current CI slow and resource-heavy?
The language(s) are inadequate: every function can affect another function or data; every function / method may have arbitrary side effects. Even if not so, the language does not export the dependency graph.
The build system is inadequate: because in the source code everything potentially affects everything else, a lot more needs to be rebuild to be sure than would be strictly necessary for the logic of the change. Even if not so, the build system does not export the dependency graph and the scope of the actual changes.
The tests end up inadequate: even if they are written competently, they cannot trust any info about the scope of the changes, so they need to test way more than logically required to be sure, or, worse, they actually need to re-test everything. Also, due to the way other systems (databases, etc) are represented in the test setup, they are hard to run in parallel.
Microservices were invented partly as a way to fight this: they make very clear bounds between independent parts of the system, and strive to keep the scope of every part small. Their small scale makes the problems with the CI tolerable.
What we need is better tools, better static analysis and compartmentalization of the code, more narrow waists [1] in our systems that allow to isolate parts and test them separately.
- trunk-based workflow. Small commits. No feature branches as a rule to be occasionally broken
- unit tests move left - run pre-commit (not necessarily run in CI). Emphasis placed on them as a refactoring tool.
- a critical, small suite of e2e and integration tests that block the pipeline before publication (fast feedback)
- latest build publication being constantly redeployed to production, even if no changes have taken place to exercise the deployment routines
- a larger suite of e2e and integration tests being constantly run against production, to give feedback when something isn't quite right, but it's not a disaster (slow, non-blocking feedback).
In summary, emphasise getting code into production, minimise blocking tests to critical ones, test in production & notify when features are broken.
Why?
- Engineers spend too much time in test environments that give the illusion of the real thing. They lose touch with production as the Dev cycle increases in circumference.
- Enabling tighter feedback cycles by accepting that some features are important and some are not helps put the cost of change into perspective for the entire product team.
- Engineers get used to working in and around production on a daily basis. Production operations and observation of systems (& users) are emphasised - protective bubbles for writing code are minimised.
You're not trying to maximise code output, you're trying to maximise the velocity of safe change, and you do that by understanding the environment (production) through your intelligence (observability of systems and user behaviour), so that you can employ your force (changes, code, features) rapidly and effectively, whilst maintaining the ability to quickly deal with unexpected problems (defects) along the way.
Disclaimer: might not be possible for your specific theatre of war for any number of reasons.
Why bother running a test pipeline at all? Why not have two production environments running in red/green, and always just run your e2e tests then flip.
I like CI tools that run locally on a developer machine and run end to end tests with all Microservices locally. The secret is integrating early and often.
That's why I wrote mazzle and platform-up.
Mazzle is a run server that is configured as a Graphviz file (.dot) and defines an end to end environment. Here's an example graph file of a pipeline:
It's largely a prototype of an idea. It's infrastructure and pipelines as code but incorporates every tool you use from terraform, chef, ansible, puppet, Kubernetes, packer and shell scripts. My example code spins up a Consul and Kubernetes cluster with hashicorp vault and a Debian repository server, configured SSH keys on every worker, bastion and Java application. And Prometheus exporter and grafana. I haven't got around to adding ELK yet. But it didn't take long to do all these things due to Mazzle meaning it's very easy to test complicated changes together.
Platform up is a pattern for local development that tests your Microservices locally all together. You use vagrant-lxc and ansible together to deploy everything locally. So you can test your secret management locally and deployment process. If your ELK stack is ansible driven you can even run your ELK stack locally as I did on a client project.
https://GitHub.com/samsquire/platform-up
First of all, it should be a fundamental principle of practical software engineering to be able to execute every test on the developer's machine.
Software development that relies on "special" environments is prone to break down sooner rather than later. If you cannot execute your tests on a fresh machine after half an hour of setup or so, something is fundamentally broken in your toolset.
In turn, this requirement means that in a well-designed development environment your integration frequency is only limited by the product of integration points (e.g., your branch and "develop", or your branch and n other ongoing branches) and the time it takes to run the tests.
"Most" and "everybody" on HN often means "we at the few megacorps around".
Let me assume instead that most software projects happen in the long tail of small companies, low LOC numbers, low developer head counts (even one or zero - consultants working only when needed.) In that world I saw deployments run when tests pass on one PC, deployed from that PC. In the case of scripting languages even with a git pull on the server and a restart. That works surprisingly well until the team gets larger. Then customers often resist the extra costs and burden of CI. Then some of them cave in after something bad happens.
I think these tiny or zero teams benefit even more from CI. As people come and go the CI process remains and people can use it without any local setup. Use GitHub actions to avoid main thing yet another thing (assuming that’s where the source code lives)
Many integrations between CIs and git frontends (as in showing green checkmarks under your pull request) only work properly if you push. Rebuilding manually in the CI will run the build, but it will no longer be recognized as related to a PR
Maybe this will happen, but I really doubt it will come anytime soon, especially since the original tweet mentions that local, laptop-based testing is enough, which is laughingly ridiculous in terms of resources, time and coverage for anyone who's worked on large, real-life, many-year, bloated projects.
: which is pretty much every project that survived the POC phase
I feel as though this article comes from a place of very subjective personal experience, perhaps one which is painful. I've seen people avoid git because merge conflicts are hard to get their head around, afraid of the terminal because GUIs are what they're used to. CI/CD... CX, when done right, allows for automating a lot of work and allowing you to become much more efficient while documenting build, integration, and delivery workflows in code. If Mr Rickard doesn't wish to see Jenkins in 2025 (three years away, so not exactly far in the future), how does he expect to be building & deploying 100s of services?
Not seeing jenkins does not mean CI/CD does not exist. User experience of CI/CD might be something similar to what you get in vercel or netlify where build, deploy and hosting is taken care of without any complicated yaml based pipeline definition, configuration management and learning curve.
"I, for one, hope that we aren't using Jenkins in 2025"
Or at least, if we are, it's a far more robust and capable system that's actually really designed for dealing with all the different things that people need to do while building and testing software.
But surely most IDEs these days make it simple to run all unit tests locally? My usual M.O. is to manually run the tests locally that I believe will demonstrate my changes are "working" and rely on the full suite of CI unit tests to check I haven't broken anything else. What other options are there?
Fundamentally, you can't trust what worked on a developer's machine and you need a centralized system (the CI) to validate your build. So if the author is right, this will likely comes with the development taking place closer to the CI than the CI taking place closer to the development.
[+] [-] alkonaut|3 years ago|reply
Zero, because unless my dog configured the CI, I’d just click a button to re-run it on the same PR/commit/branch and not “make an empty commit”.
End to end testing of large systems with hundreds or thousands of man-years of code in them will be slow because there is a large surface area to test and testing in individual modules doesn’t give the same kind of end to end coverage that testing everything together does. So while pre-commit would be good, I doubt it will replace server CI for all but the smallest repositories.
[+] [-] gusbremm|3 years ago|reply
[+] [-] skissane|3 years ago|reply
[+] [-] adrianN|3 years ago|reply
[+] [-] oldjavacoder|3 years ago|reply
[deleted]
[+] [-] hinkley|3 years ago|reply
I do worry that a lot of people are missing the first principles where CI is concerned, but a few of us still remember. Whether that’s enough is always a question.
> First, CI rarely knows what has changed, so everything needs to be rebuilt and retested
CI is the impartial judge in your system. That is the ultimate goal and much of the value. Don’t believe me when I say that your check in broke something. Believe the robot. It does not think, it does not interpret errors (which is why intermittent failures poison the well). It just knows that whatever you did was not good.
We make it stupid simple because the more ways you try to interpret things, the higher the likelihood the system will either miss an error or report imaginary ones. And in this case I am very sorry Mr Lasso, but it is in fact the hope that kills. Optimism makes us try things, things others would give up on. That’s what makes a piece of software good. Better. But too much and you start ignoring very real problems and make optimism into a liability. I’ve seen it over and over again, take a build with intermittent failures and people will check in broken code and assume it’s the intermittent problem and not their change.
Ideally, when you start breaking builds you’ve found your limits. But the build has to break when I break the code, and it needs to do it repeatably, because if I can’t reproduce the error I risk telling myself stories, and foisting off debugging to other people, which is a big no no.
[+] [-] lukeasrodgers|3 years ago|reply
[+] [-] auxym|3 years ago|reply
If it doesnt build on CI, you broke it, fix it before merging.
Though, I'm still sour from my only experience in "tech" where "works on my machine" meant devs got to demand I fix the CI somehow.
[+] [-] cntainer|3 years ago|reply
Of course there are many pitfalls. Like with any tool it becomes trivial to use it wrong. Integrating less than once per day and deferring all testing and linting to the CI process is an anti-pattern and the issue is not the CI process but how the team chooses to use it.
For example I worked in a team of 10 and we were doing multiple production deploys per day. This was made possible by a great CI workflow. Everyone was running tests precommit and focusing on keeping the pipeline green. Yes I've seen the opposite as well, but that usually is a symptom of issues with the team not the general concept of CI.
Not saying that there isn't room for improvement but all the real-time collaboration features I used on low-code platforms feel like a step back.
[+] [-] adalarmed|3 years ago|reply
I'm confused as to why this is an anti-pattern? My understanding is that the CI pipeline should run unit tests and linting for every commit. But at the same time, developers should run their tests before pushing code.
[+] [-] benjiweber|3 years ago|reply
> something even more continuous than continuous integration. It might look like real-time integration of your changes, collaborative editing, or smarter build systems that produce near-live feedback
e.g. https://medium.com/darklang/how-dark-deploys-code-in-50ms-77...
[+] [-] madsbuch|3 years ago|reply
I have worked in a place where they did that, and I think the cons heavily outweighed the pros. I can not push incomplete work to a remote, I can not separate feature development and chores (eg. linting) because I _need_ to fix stuff in order to commit an push, etc.
> Continuous Integration is a model thats tightly tied to how we run version control.
I would say that a pre-commit testing world is much tighter. CI, as many know it, as a passive observer. When stuff fails, it will let you know. You can still merge.
One thing that would be nice, however, would be the ability to run the entire pipeline locally. For GitHub actions, it indeed seems like there are viable options to do that.
[+] [-] runeks|3 years ago|reply
I prefer not having to replicate locally how GitHub runs GitHub actions, but rather just make my GitHub actions run something that I know I can run locally. So all the complicated stuff is contained in the script that I can run locally, and GitHub actions is just a dumb “runner” of this script.
For my local script I prefer using Nix, since it’s the best way I know to guarantee a consistent environment between different machines.
[+] [-] lvncelot|3 years ago|reply
This cost me many hours of waiting for the Gitlab CI runner when debugging non-trivial pipelines, when the issue was something that did not have to do with the script steps inside of the jobs but rather how the Gitlab runner handled things.
I've found gitlab-ci-local [1] which actually does run the Gitlab pipeline locally, although I had to write some boilerplaye scripts to set up all the necessary 'CI_FOO_SOMETHING' environment variables before running the tool. (Which sometimes came back to bite me because the issue was actually in the content of some of those environment variables). It's still a very good tool.
[1] https://github.com/firecow/gitlab-ci-local
[+] [-] __alexs|3 years ago|reply
[+] [-] mpweiher|3 years ago|reply
Also hate those.
What I do instead is have the tests run as part of the build. If the tests fail, the build fails.
Definitely a good motivator for having fast tests, which is essential. The ones for my projects tend to run in about a second or two.
Also gives you way greater confidence in your tests.
Try it!
https://blog.metaobject.com/2020/05/mpwtest-reducing-test-fr...
https://blog.metaobject.com/2020/06/mpwtest-only-tests-frame...
[+] [-] Epa095|3 years ago|reply
Maybe the right thing is to just write the workflows in something else, and have the github workflow file be a single call to your script. But
-It would be nice to be able to use github actions others has made (or libraries/abstractions in general)
- I don't see how to easily get parallell execution.
- I love github environments, how do I pass down all my environments and their variables?
[+] [-] reillyse|3 years ago|reply
I'm working on something to do just this. Although I've redefined what "local" means in this context. I'm still using remote servers but everything is happening pre-commit from the terminal. If you are interested check out this demo and let me know if you have any feedback https://brisktest.com/demos
[+] [-] rtpg|3 years ago|reply
Though for me it’s also like… so many people do unspeakable horrors to their machine setups that I like there being a consistent runner in CI.
But “CI is to run tests” in the world of CD is a bit of a simplification anyways.
[+] [-] citrin_ru|3 years ago|reply
Per-commit hooks save my time by not allowing to push code which guaranteed to break later in CI.
[+] [-] wereHamster|3 years ago|reply
[+] [-] ownagefool|3 years ago|reply
What if the client is compromised? Are we throwing away reviews? Should the reviewer re-run all the tests?
[+] [-] swagonomixxx|3 years ago|reply
I kind of get what this is saying, but technology evolution doesn't have to mean completely replacing said technology with something else.
I think that's one weird thing about the software field, whereby we keep moving to these shiny new things that we think are better than the tools of yesteryear, yet in the end there is only a marginal gain in productivity.
Fork and pull is an incredibly productive and powerful workflow. CI is incredibly, incredibly useful. If these things were not the case, then neither of these would be even discussed by this article. There is a reason for their success - and it's not because GitHub is the most ubiquitous code hosting service out there. Git is _actually_ pretty great. CI is _actually_ very useful and has secured codebases for decades at this point.
So if one were to proclaim the "End of CI" I really need to see a viable alternative that addresses the same problems as CI and significantly improves upon it. An incremental improvement is not enough to shift and rewrite everything - there needs to be a significant jump in ability, productivity, security, or something else in order for me (and I imagine many others) to consider it.
[+] [-] andyjohnson0|3 years ago|reply
"Thought leaders" need to he constantly talking up the next new thing so that they can stay ahead of the herd on socia media.
Developers get bored, or worry that their career is stagnating, if they're not using the new shiny. Particularly if they pay attention to the thought leaders, or they're stuck building unglamorous crud apps.
And the software industry generally has a poor collective memory of tools and practices and experience from even the recent past. Contrast with more mature engineering disciplines, or architecture, medicine, etc. I'm not sure why this is.
[+] [-] chiefalchemist|3 years ago|reply
Agreed. But I'd take it a step further and say:
...yet in the end no one is any happier. Not engineers. Not management. Not leadership. And most importantly not the users of the product.
We keep building and delivering more. But how often is that better? Either as an end to end experience or simply fewer bugs? Most of us - who are at some point users of products we didn't build - have resigned ourselves to that fact (read: it's been normalized) that an uncomfortable amount of friction is a given. That's sad.
[+] [-] jt2190|3 years ago|reply
[+] [-] neatze|3 years ago|reply
Technology and in general historical trends are rarely broken, they change but there is no gaps in sense, I don't remember who said it, someone in defense R&D, if recall correctly.
[+] [-] oldjavacoder|3 years ago|reply
Actually, like the wheel, I think it is the final form. When artificially sentient machines are writing code for us and someone or some thing pollutes or corrupts a critical model; a human will need so step in and fork where things were working to fix it.
[+] [-] kodah|3 years ago|reply
CI isn't just a remote compute scheduler. Its also the place that contains tooling, tooling which repeated across a dev team creates n * jobs replicas of that tooling - which run on a local machine would be fairly wasteful. Its also centralized because CI is one of the biggest vectors an organization runs; it contains credentials for outside services, credentials for deployment (which, if you're SOX-bound cannot exist on a dev machine), and if you're cryptographically signing binaries and doing reproducible builds it's got those keys too. n * developer laptops makes that vector a planet sized blot. A lot of the policies I imagine dealing with that would be limiting developer access to their machines, limiting tooling to only approved or internal tooling.
It'd be a nightmare.
[+] [-] swyx|3 years ago|reply
[+] [-] tsujp|3 years ago|reply
However, things like:
Show that the author is perhaps conflating GitHub with Git.[+] [-] cube2222|3 years ago|reply
> First, CI rarely knows what has changed, so everything needs to be rebuilt and retested.
Yes, this is an issue with some CI/CD systems, an issue you can solve however. See our Push Policies[1] based on Open Policy Agent. Test-case caching is also sometimes available (available in i.e. Go).
> Running tests pre-commit is one answer.
Running tests locally for a quick feedback loop - sure, that's fairly mainstream though (something you can use our local preview[2] for as well in the case of IaC). Running tests locally before directly deploying to prod - that would be a compliance and security nightmare.
The author presents what looks to me like a very "move fast, break things" attitude, that doesn't really work in a whole lot of cases.
If your CI/CD is slowing you down, make it faster. Easier said than done, I know, but a lot of people don't even think about optimizing their CI/CD, which you often can do, by being smarter about what you run and/or parallelizing extensively and/or caching[3].
[0]:https://spacelift.io
[1]:https://docs.spacelift.io/concepts/policy/git-push-policy
[2]:https://docs.spacelift.io/concepts/stack/stack-settings#enab...
[3]:https://github.com/actions/cache
[+] [-] sequoia|3 years ago|reply
“CI is frustrating!!” I hear ya, but this article does nothing to clarify a viable alternative.
[+] [-] nine_k|3 years ago|reply
Why is the current CI slow and resource-heavy?
The language(s) are inadequate: every function can affect another function or data; every function / method may have arbitrary side effects. Even if not so, the language does not export the dependency graph.
The build system is inadequate: because in the source code everything potentially affects everything else, a lot more needs to be rebuild to be sure than would be strictly necessary for the logic of the change. Even if not so, the build system does not export the dependency graph and the scope of the actual changes.
The tests end up inadequate: even if they are written competently, they cannot trust any info about the scope of the changes, so they need to test way more than logically required to be sure, or, worse, they actually need to re-test everything. Also, due to the way other systems (databases, etc) are represented in the test setup, they are hard to run in parallel.
Microservices were invented partly as a way to fight this: they make very clear bounds between independent parts of the system, and strive to keep the scope of every part small. Their small scale makes the problems with the CI tolerable.
What we need is better tools, better static analysis and compartmentalization of the code, more narrow waists [1] in our systems that allow to isolate parts and test them separately.
[1]: https://www.swyx.io/narrow-waists
[+] [-] beaker52|3 years ago|reply
- trunk-based workflow. Small commits. No feature branches as a rule to be occasionally broken
- unit tests move left - run pre-commit (not necessarily run in CI). Emphasis placed on them as a refactoring tool.
- a critical, small suite of e2e and integration tests that block the pipeline before publication (fast feedback)
- latest build publication being constantly redeployed to production, even if no changes have taken place to exercise the deployment routines
- a larger suite of e2e and integration tests being constantly run against production, to give feedback when something isn't quite right, but it's not a disaster (slow, non-blocking feedback).
In summary, emphasise getting code into production, minimise blocking tests to critical ones, test in production & notify when features are broken.
Why?
- Engineers spend too much time in test environments that give the illusion of the real thing. They lose touch with production as the Dev cycle increases in circumference.
- Enabling tighter feedback cycles by accepting that some features are important and some are not helps put the cost of change into perspective for the entire product team.
- Engineers get used to working in and around production on a daily basis. Production operations and observation of systems (& users) are emphasised - protective bubbles for writing code are minimised.
You're not trying to maximise code output, you're trying to maximise the velocity of safe change, and you do that by understanding the environment (production) through your intelligence (observability of systems and user behaviour), so that you can employ your force (changes, code, features) rapidly and effectively, whilst maintaining the ability to quickly deal with unexpected problems (defects) along the way.
Disclaimer: might not be possible for your specific theatre of war for any number of reasons.
[+] [-] karma_fountain|3 years ago|reply
[+] [-] samsquire|3 years ago|reply
That's why I wrote mazzle and platform-up.
Mazzle is a run server that is configured as a Graphviz file (.dot) and defines an end to end environment. Here's an example graph file of a pipeline:
https://github.com/samsquire/mazzle/blob/master/docs/archite...
It's largely a prototype of an idea. It's infrastructure and pipelines as code but incorporates every tool you use from terraform, chef, ansible, puppet, Kubernetes, packer and shell scripts. My example code spins up a Consul and Kubernetes cluster with hashicorp vault and a Debian repository server, configured SSH keys on every worker, bastion and Java application. And Prometheus exporter and grafana. I haven't got around to adding ELK yet. But it didn't take long to do all these things due to Mazzle meaning it's very easy to test complicated changes together.
https://devops-pipeline.com - Mazzle
Platform up is a pattern for local development that tests your Microservices locally all together. You use vagrant-lxc and ansible together to deploy everything locally. So you can test your secret management locally and deployment process. If your ELK stack is ansible driven you can even run your ELK stack locally as I did on a client project. https://GitHub.com/samsquire/platform-up
[+] [-] choeger|3 years ago|reply
Software development that relies on "special" environments is prone to break down sooner rather than later. If you cannot execute your tests on a fresh machine after half an hour of setup or so, something is fundamentally broken in your toolset.
In turn, this requirement means that in a well-designed development environment your integration frequency is only limited by the product of integration points (e.g., your branch and "develop", or your branch and n other ongoing branches) and the time it takes to run the tests.
[+] [-] pmontra|3 years ago|reply
Let me assume instead that most software projects happen in the long tail of small companies, low LOC numbers, low developer head counts (even one or zero - consultants working only when needed.) In that world I saw deployments run when tests pass on one PC, deployed from that PC. In the case of scripting languages even with a git pull on the server and a restart. That works surprisingly well until the team gets larger. Then customers often resist the extra costs and burden of CI. Then some of them cave in after something bad happens.
[+] [-] pharmakom|3 years ago|reply
[+] [-] Scea91|3 years ago|reply
Build can be triggered manually in every CI system I know. Why would I push empty commits?
[+] [-] infensus|3 years ago|reply
[+] [-] BozeWolf|3 years ago|reply
[+] [-] bananaowl|3 years ago|reply
[+] [-] xcambar|3 years ago|reply
: which is pretty much every project that survived the POC phase
[+] [-] imdsm|3 years ago|reply
[+] [-] debarshri|3 years ago|reply
[+] [-] AdrienBrault|3 years ago|reply
[+] [-] wizofaus|3 years ago|reply
Or at least, if we are, it's a far more robust and capable system that's actually really designed for dealing with all the different things that people need to do while building and testing software.
But surely most IDEs these days make it simple to run all unit tests locally? My usual M.O. is to manually run the tests locally that I believe will demonstrate my changes are "working" and rely on the full suite of CI unit tests to check I haven't broken anything else. What other options are there?
[+] [-] NewEntryHN|3 years ago|reply