I've been a build/release engineer for quite a few years now, most spent using Jenkins.
It can be quite a heavyweight beast, especially since it's written in Java (so heavy in terms of resource usage) and since it's old (so lots of legacy in terms of architecture).
Still, in terms of costs and scalability (up to the level where you have thousands of developers and you probably want to roll your own), I think it is one of the best tools out there for its role as a prettier cron/queuing system.
A few tips:
* if you can afford it, stop ASAP using the UI for configurating jobs; look into Pipelines or the JobDSL, based on your needs
* once you have more than a few jobs, stop running anything on the master node, for increased stability
* use the minimum number of plugins and tie as few things to Jenkins as possible; script as many things and put the scripts into version control
* backup the global configuration; you can try to version control it but it's tricky; at least back it up periodically and most of the time it is enough
Going off of this I wouldn't even use job dsl or pipelines as they allow you to execute arbitrary groovy. I would consider emulating Travis ci or Amazon pipelines where it is all declarative not imperitive. This way you can re-impliment and consolidate without doing a Turing complete parse of tons of custom groovy scripts
I have come up with a slogan for Jenkins: "Jenkins - The butler that does not hesitate to do a disservice".
It is currently the most brittle piece of infrastructure we run, with jobs spiralling out of control and slaves crashing with cryptic exceptions. Add to that a horrible plugin ecosystem (tried to install the Slack plugin, it crashed all builds and our webhook endpoint). Add to that groovy (I don't feel like becoming an archeologist).
Gosh, can't someone just build a lightweight ci/cd system ontop of e.g. k8s jobs?
My workplace has primarily switched all net new ci/cd workflow from Jenkins to Drone (https://github.com/drone/drone - Drone is a Continuous Delivery system built on container technology.). Having managed masters with 20,000 + configured jobs I agree that Jenkins is a very sore spot for our entire pipeline. We’ve had greater success with increasing the number of masters (one per team or application) and using the kubernetes plugin for slaves. However, most of our pipeline complaints have been mitigated by getting teams to migrate from Jenkins to Drone.
While I don’t think Drone currently supports k8s job, it is very lightweight in comparison to Jenkins. Most of our developers would agree that it isn’t a perfect product, but it has reduced friction points by a large amount.
Allow me to shill for Concourse[0]. It's oriented around 'tasks' composed into 'pipelines'. Each task either fails or succeeds with the yaml pipeline definition providing hooks into either state to proc other tasks. A task is a yaml definition that invokes a bash script. Each task runs inside a preconfigured docker container for full reproducibility.
It's declarative and everything needed to bring up a pipeline can be stored in git alongside the software it supports. For an example pipeline definition, check out the one for bosh-gcscli[1] and its accompanying tasks[2].
Now, there is the downside that upkeep is a hassle because it requires buying into the BOSH ecosystem to deploy Concourse. That pays for itself by being explicitly anti-brittle, with support for migrations and canary rollouts builtin.
Disclaimer: I worked with Concourse at my previous position but currently have no financial stake in its success.
That matches my experience. Jenkins is built out of a spiderweb of plugins and that makes it very fragile. Though, I did find I could mitigate the problem.
* only use Jenkins stable releases
* snapshot the system before updating or installing anything
* make regular backups of your Jenkins server
* never install a plugin unless it's actually neccessary
* always update Jenkins before installing any new plugins as otherwise you may end up with an untested mishmash of old and new plugins
I like having control over my CI, but it does seem a lot harder than it should be.
I replaced it with saltstack. I had so many things not working I thought it was me just being dumb or not spending enough time learning to use it. A few salt states on the end of a git hook did everything, auto deployment, testing, staging, monitoring. I am probably missing something about the awsome power of jenkins but know firms still running my saltstack CI/CD systems years after I left. Weird.
It’s really nice, but expensive if you exceed the limits for the free version (3 agents and 100 build configurations). Luckily we are a fair way off that so can go along fine without having to pay yet.
On setting up Jenkins Job Builder using a 'seed job'. This setup allows you to add a .jenkins directory to you project, define a yaml based jenkins job inside that directory, push to github, and Jenkins will create the job you defined - NO web UI required.
In other words you make Jenkins behave like Travis.
Set this up with your first job being a job to backup Jenkins to S3 and things aren't as bad to manage.
I'm currently looking into how JJB could be used for simple scheduled jobs. But isn't the setup described in the article basically the same what you can do with Jenkinsfile?
If I may suggest my own alternative to Jenkins: builds.sr.ht. It runs build manifests submitted through an API and does not support job configuration itself; you must use an external integration to automate your builds. Build manifests look something like this:
If anyone wants to give it a try, please get in touch (email in my profile) and I'll hook you up with an account on my hosted instance. I could use more people to kick the wheels and help me find the pain points.
I am spearheading an effort to release a CloudFormation template that will launch Jenkins clusters into your AWS account. It represents everything learned while working with Jenkins for over 6 years including 2.5 years of Jenkins at Lyft and 1 year at DoorDash. If you would like to collaborate, please contact [email protected]. The project was written with an eye towards making the launching of a Jenkins cluster very very easy. It's designed with Docker workloads in mind. Some of the gems include autoscaling, instrumentation & metrics, automatic Let's Encrypt certificate registration, GitHub webhook integration and retries during pipeline slave disconnects which makes spot instances viable even when they abruptly go away... something that saves you 75%!! on your AWS instance costs.
Man, I'm having this problem right now at work and it's been a nightmare. Admittedly the problem is like 90% cultural.
People (on my team) don't understand pipeline and they absolutely don't understand Groovy. Before, they would just write a job with a shell script that e.g. ran valgrind, etc. Lots of small repos with similar steps, so they'd write one job and apply it to 20 different repos. It worked pretty well.
That's still an option with pipelines, but it feels much more discouraged (to e.g. write a shared pipeline job that runs a multi-line shell script).
It's also just a bit of an organizational nightmare to see 200+ jobs on the main screen (as opposed to 20 jobs that did the same thing for 200 repos).
Unfortunately, it also seems like declarative pipeline is limited enough that we end up writing a lot of Groovy. I sorta get why Groovy became the official scripting language, but it's like pulling teeth getting people to learn even basic Groovy.
I realize, again, that a lot of this is a cultural problem. Mostly just trying to give a counterpoint that pipelines don't always look as clean as the article implies, especially for orgs with a lot of repos (where committing the same 'test.sh' file to each repo doesn't make a lot of sense).
”It's also just a bit of an organizational nightmare to see 200+ jobs on the main screen”
I never look at the main screen. You can create tabs that segment those 200 jobs. If your job names are consistent (as they should), you can do the segmenting by regular expression (I’m not sure whether creating tabs requires a plug-in)
As to doing (almost) the same thing for different projects: you can POST XML job definitions to the web interface, so if you have the rights to create jobs and to run code on your local system, you can script job creation. In my experience, that’s the way to keep job definitions consistent. To figure out what the XML to POST should look like, grab it from your browser by GETting job/jobname/config.xml (https://support.cloudbees.com/hc/en-us/articles/218353308-Ho...)
Yes, that duplicates lots of stuff in the Jenkins job definitions, but you shouldn’t treat that as source, but as the output of your job creation scripts.
Rather than run monolithic Jenkins with a kagillion jobs in it that will easily bog down, you could consider running distributed Jenkins in which each team, or even each microservice, gets its own Jenkins Cluster with a small number of slaves and a small number of jobs. See my earlier post in this thread where I posted a YouTube video of the method I use for doing this using a CloudFormation template.
Also, when you say that your developers don't understand groovy, you could teach them how to ding make targets inside of their docker containers. That's the pattern we use at DoorDash... very minimal Jenkins scripting which exists to ding make targets inside Docker images.
Why not make it easy to reuse functions to do the steps the groovy is doing and build all of the common steps in a reusable way? Eg what the article is suggesting
Jenkins and any other CI/CD tool is a prime candidate for immutable infrastructure; it’s the one thing that nobody cares about too much until it no longer works, and it is relatively easy to get going with scripts or Docker
[+] [-] oblio|8 years ago|reply
It can be quite a heavyweight beast, especially since it's written in Java (so heavy in terms of resource usage) and since it's old (so lots of legacy in terms of architecture).
Still, in terms of costs and scalability (up to the level where you have thousands of developers and you probably want to roll your own), I think it is one of the best tools out there for its role as a prettier cron/queuing system.
A few tips:
* if you can afford it, stop ASAP using the UI for configurating jobs; look into Pipelines or the JobDSL, based on your needs
* once you have more than a few jobs, stop running anything on the master node, for increased stability
* use the minimum number of plugins and tie as few things to Jenkins as possible; script as many things and put the scripts into version control
* backup the global configuration; you can try to version control it but it's tricky; at least back it up periodically and most of the time it is enough
That's it for now, for more details, $200/h :p
[+] [-] grogenaut|8 years ago|reply
[+] [-] mikewhy|8 years ago|reply
[+] [-] correlation|8 years ago|reply
It is currently the most brittle piece of infrastructure we run, with jobs spiralling out of control and slaves crashing with cryptic exceptions. Add to that a horrible plugin ecosystem (tried to install the Slack plugin, it crashed all builds and our webhook endpoint). Add to that groovy (I don't feel like becoming an archeologist).
Gosh, can't someone just build a lightweight ci/cd system ontop of e.g. k8s jobs?
Or should we just give up and do managed ci/cd?
[+] [-] CodeAlong|8 years ago|reply
While I don’t think Drone currently supports k8s job, it is very lightweight in comparison to Jenkins. Most of our developers would agree that it isn’t a perfect product, but it has reduced friction points by a large amount.
[+] [-] Everlag|8 years ago|reply
It's declarative and everything needed to bring up a pipeline can be stored in git alongside the software it supports. For an example pipeline definition, check out the one for bosh-gcscli[1] and its accompanying tasks[2].
Now, there is the downside that upkeep is a hassle because it requires buying into the BOSH ecosystem to deploy Concourse. That pays for itself by being explicitly anti-brittle, with support for migrations and canary rollouts builtin.
Disclaimer: I worked with Concourse at my previous position but currently have no financial stake in its success.
[0] https://concourse.ci/ [1] https://github.com/cloudfoundry/bosh-gcscli/blob/c908176/ci/... [2] https://github.com/cloudfoundry/bosh-gcscli/tree/30316d6/ci/...
[+] [-] mikewhy|8 years ago|reply
[+] [-] slavik81|8 years ago|reply
* only use Jenkins stable releases
* snapshot the system before updating or installing anything
* make regular backups of your Jenkins server
* never install a plugin unless it's actually neccessary
* always update Jenkins before installing any new plugins as otherwise you may end up with an untested mishmash of old and new plugins
I like having control over my CI, but it does seem a lot harder than it should be.
[+] [-] mianos|8 years ago|reply
[+] [-] stephen_g|8 years ago|reply
It’s really nice, but expensive if you exceed the limits for the free version (3 agents and 100 build configurations). Luckily we are a fair way off that so can go along fine without having to pay yet.
[+] [-] chrisanthropic|8 years ago|reply
On setting up Jenkins Job Builder using a 'seed job'. This setup allows you to add a .jenkins directory to you project, define a yaml based jenkins job inside that directory, push to github, and Jenkins will create the job you defined - NO web UI required.
In other words you make Jenkins behave like Travis.
Set this up with your first job being a job to backup Jenkins to S3 and things aren't as bad to manage.
[+] [-] finspin|8 years ago|reply
[+] [-] ddevault|8 years ago|reply
https://git.sr.ht/~sircmpwn/wlroots/tree/.build.yml
More complex example:
https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/.build.yml
Builds run with full virtualization (KVM) on any number of distributed (optional trust) build runners. The whole thing is open source:
https://git.sr.ht/~sircmpwn/builds.sr.ht
It's also self-hosting, it builds and deploys itself and the rest of the *.sr.ht network:
https://builds.sr.ht/job/602
If anyone wants to give it a try, please get in touch (email in my profile) and I'll hook you up with an account on my hosted instance. I could use more people to kick the wheels and help me find the pain points.
[+] [-] jonthepirate|8 years ago|reply
[+] [-] mooreds|8 years ago|reply
[+] [-] BlanketApple|8 years ago|reply
People (on my team) don't understand pipeline and they absolutely don't understand Groovy. Before, they would just write a job with a shell script that e.g. ran valgrind, etc. Lots of small repos with similar steps, so they'd write one job and apply it to 20 different repos. It worked pretty well.
That's still an option with pipelines, but it feels much more discouraged (to e.g. write a shared pipeline job that runs a multi-line shell script).
It's also just a bit of an organizational nightmare to see 200+ jobs on the main screen (as opposed to 20 jobs that did the same thing for 200 repos).
Unfortunately, it also seems like declarative pipeline is limited enough that we end up writing a lot of Groovy. I sorta get why Groovy became the official scripting language, but it's like pulling teeth getting people to learn even basic Groovy.
I realize, again, that a lot of this is a cultural problem. Mostly just trying to give a counterpoint that pipelines don't always look as clean as the article implies, especially for orgs with a lot of repos (where committing the same 'test.sh' file to each repo doesn't make a lot of sense).
[+] [-] Someone|8 years ago|reply
I never look at the main screen. You can create tabs that segment those 200 jobs. If your job names are consistent (as they should), you can do the segmenting by regular expression (I’m not sure whether creating tabs requires a plug-in)
As to doing (almost) the same thing for different projects: you can POST XML job definitions to the web interface, so if you have the rights to create jobs and to run code on your local system, you can script job creation. In my experience, that’s the way to keep job definitions consistent. To figure out what the XML to POST should look like, grab it from your browser by GETting job/jobname/config.xml (https://support.cloudbees.com/hc/en-us/articles/218353308-Ho...)
Yes, that duplicates lots of stuff in the Jenkins job definitions, but you shouldn’t treat that as source, but as the output of your job creation scripts.
[+] [-] jonthepirate|8 years ago|reply
Also, when you say that your developers don't understand groovy, you could teach them how to ding make targets inside of their docker containers. That's the pattern we use at DoorDash... very minimal Jenkins scripting which exists to ding make targets inside Docker images.
[+] [-] grogenaut|8 years ago|reply
[+] [-] nunez|8 years ago|reply
[+] [-] emmelaich|8 years ago|reply
https://jenkins.io/projects/blueocean/
[+] [-] tokenizerrr|8 years ago|reply
[+] [-] draw_down|8 years ago|reply
[deleted]