> Unlike some other CI tools, pipelines in Vespene are easily and graphically configured, and there is no custom DSL (“Domain Specific Language”) to learn and debug.
Does this mean that the CI config cannot be checked into some sort of version control? That was one of my biggest issues with Jenkins. Someone would change something somewhere and suddenly my builds would fail, and even if I track down what setting is causing the problem, I have no idea why it was changed, who changed it, what will happen if I put it back, or even what it's previous value is.
Additionally, it made duplicating a given workflow an extremely tedious, manual process.
> When organizations start to have too many projects, they need an easy way to share values and code snippets between those projects. To support this we have Jinja2 templating of build scripts
I would implore everyone implementing CI/CD to do the opposite. If you have a small project, maybe use a build script to get it off the ground quickly. If you have a large organization with complex, disparate tech projects, do not use build scripts.
Use CM or orchestration tools. Use pre-rolled containers/images. Use configuration options for existing solutions/plugins/deployment tools. Write only open source, composable, extensions/modules/plugins, with yaml/json/xml configs to control them.
The open source bit is actually crucial. By making all your build/test/deploy plugins open source, you force yourself not to include proprietary information, which is always a dark pattern. Hard-coding IPs/hostnames, credentials, product names, regions, etc just makes your work less composable/reusable. Open source all your pipeline code except for your configs. Then everyone in your business can find everyone else's work simply by going to GitHub and looking up a tool. Because they won't be scripts, they can be picked up and used immediately by any team, and not have to be forked by every team that needs a modified build script.
To do all this properly requires discipline and training and research and time, which big companies have. The point is not to be fast, but to be reliable, to prevent muda from sapping your productivity.
Compiling C code with puppet or whatever seems a little weird to me, to be honest :)
If you want to keep your data externally, there are some good options for this. I'd read the plugin docs about variable plugins, that can easily source variables for things like etcd or Consul.
A script for a given repo could be nothing more than "make go", and that actually is a pretty good practice, to keep that in source control.
Still, you need something to build your code, and to have a good place to see the status.
Ultimately, code still needs to be built, and orgs do want to avoid images they cannot easily recreate. That's the role of a build system and making sure the process to create those artifacts is in source control.
I should also mention that your build script CAN be sourced from source control.
In your .vespene file, just say "script: <path>.
However, the variables in that file can be evaluated by Jinja2 variables, so for instance your security team could set up a snippet everybody could use or your feature flags could be defined from there.
Another cool feature of Vespene is in each project build root all those variables appear as vespene.json files, so it can be a good tool to use to launch all kinds of automation from a web console where you want to record results.
Id strongly recommend adding the ability to implement pipelines as code -- ie: you check in the CI/CD configuration into the repo (or another repo, containing CI/CD configurations). Having to configure the pipelines manually is nasty, and often builds are caused by misconfigured of the pipeline. having this checked it allows easy versioning.
How does this compare to something like Buildbot (https://buildbot.net/) - I share a lot of your frustrations with Jenkins and this is where we landed.
This looks really great to me, and I can't wait to try it. I've been using buildbot for extremely light-weight CI/CD server that basically just triggers Ansible scripts on git changes. Not at all the way buildbot was designed to work, but very quick and easy to manage.
This looks like a tool that could be a lot closer to what I want to do and designed to do it that way.
Does every machine running a worker have to be a full node with Django? Workers in a complex CI setup can often be e.g. raspberry pi machines, iOS emulators, Windows machines without admin access deep inside a firewall that make outgoing connections but not incoming, etc.
So, you might consider a protocol for builders to report their results (e.g. over SSH) without being a full node, if that doesn't exist. Looks good!
Any node typically would run the webserver because it's not that heavy, but doesn't technically have to. It could just run one worker process. Right now the workers do need database access and I suspect that will stay that way for the short term.
Windows needs to happen, but I'm not sure when it happens. (Also a good list discussion).
I'm a bit curious about your raspberry pi use case - maybe a good topic for the forum? http://talk.vespene.io/
How do you manage normalizing webhook events from each source control provider, i.e. GitHub, GitLab, BitBucket, etc?
I have been interested in building a general purpose CI/CD tool but this part seems like the most "grunt work". Is there an open source library that standardizes webhooks? Or would you be interested in publishing that portion in a library?
Cool project! It strikes me that Django might not be the ideal for this project, have you thought about that? This logic [1] looks shoehorned into the framework.
While you're welcome to that opinion, Django has been freaking awesome. This whole thing got written in 3 months, not even working full time. I wrote pipelines in like a day.
It seems like you're saying I should have used a many-to-many there rather than 7 FKs. Probably, but shrug... it can be fixed later if it ever becomes a problem.
This looks cool. Unfortunately it is Common Clause licensed and therefore not Open Source. My organization would never consider a Commons Clause licensed tool.
The Anti-Commons Clause. I agree, this is a non-starter. No sane company would introduce themselves to the liabilities this presents and no sane open source project would utilize a nonfree CI.
I'd argue the definition of what is open source is up for debate, but I had some reasons for this and thought about it for weeks.
I hope people know my history and how I've run projects in the past.
But what this does for me is keep me from releasing anything open core. With ansible, ansible was open source but the GUI was not, and we had to hold something back.
The traditional alternatives to OSS are to make a support business, which somewhat encourages making buggy or hard to install software, or a consulting business (IMHO, same) and all of those detract from building a product, collectively, with some of your favorite people on the internet.
So basically Vespene is going to be like a pseudo-foundation.
The structure for this is described here - http://docs.vespene.io/partnership.html - which keeps consulting and support totally free for small shops, and encouraging larger ones that could potentially make millions form Vespene to give back a very small amount.
What normally happens is large companies close six figure contracts supporting open source, and they don't give back a dime, and my goal here is to essentially make a developer's salary off of this project by having those who make more give back a very small amount.
This seems pretty fair to me, but I also recognize that some people don't agree with it.
That all being said, it's not a "no-commercial" clause in any way, and still free for small consultancies, so I'm not anticipating any major problems.
Most of the time, software gets installed in a place of business for that business.
It's not something I've considered lightly, but this keeps MORE software open, and for a small one-person shop, I think that's more noble than trying to hold some code back and not release it at all.
Some people just want 100% free, gimme gimme, etc - and fine - you're entitled to pick one of those things. That's ok.
I personally view this a little more pragmatically, under the view of fairness, and also have taken steps (in the CLA, etc), to guarantee that trust is never going to be abused.
For instance, the license reverts to pure Apache if Vespene were ever to change hands.
On the downside, I can't get screwed over by IBM or Amazon while I'm trying to crank out free software for everyone. That seems fair too, seeing the time investment running an open source community and working on the code takes.
There's a fair amount of that available in http://docs.vespene.io/importing.html ... if you have more ideas can you stop by https://talk.vespene.io/c/ideas ? ... I don't want to make the syntax more complex than the YAML that is there now, but I think we can add other fields if there is something you want or a capability that might be missing.
[+] [-] markovbot|7 years ago|reply
Does this mean that the CI config cannot be checked into some sort of version control? That was one of my biggest issues with Jenkins. Someone would change something somewhere and suddenly my builds would fail, and even if I track down what setting is causing the problem, I have no idea why it was changed, who changed it, what will happen if I put it back, or even what it's previous value is.
Additionally, it made duplicating a given workflow an extremely tedious, manual process.
[+] [-] pytester|7 years ago|reply
[+] [-] attractiveape|7 years ago|reply
[+] [-] dhuramas|7 years ago|reply
That seems to take care of who changed, what changed, etc.
[+] [-] _wmd|7 years ago|reply
[+] [-] peterwwillis|7 years ago|reply
I would implore everyone implementing CI/CD to do the opposite. If you have a small project, maybe use a build script to get it off the ground quickly. If you have a large organization with complex, disparate tech projects, do not use build scripts.
Use CM or orchestration tools. Use pre-rolled containers/images. Use configuration options for existing solutions/plugins/deployment tools. Write only open source, composable, extensions/modules/plugins, with yaml/json/xml configs to control them.
The open source bit is actually crucial. By making all your build/test/deploy plugins open source, you force yourself not to include proprietary information, which is always a dark pattern. Hard-coding IPs/hostnames, credentials, product names, regions, etc just makes your work less composable/reusable. Open source all your pipeline code except for your configs. Then everyone in your business can find everyone else's work simply by going to GitHub and looking up a tool. Because they won't be scripts, they can be picked up and used immediately by any team, and not have to be forked by every team that needs a modified build script.
To do all this properly requires discipline and training and research and time, which big companies have. The point is not to be fast, but to be reliable, to prevent muda from sapping your productivity.
[+] [-] mpdehaan2|7 years ago|reply
If you want to keep your data externally, there are some good options for this. I'd read the plugin docs about variable plugins, that can easily source variables for things like etcd or Consul.
A script for a given repo could be nothing more than "make go", and that actually is a pretty good practice, to keep that in source control.
Still, you need something to build your code, and to have a good place to see the status.
Ultimately, code still needs to be built, and orgs do want to avoid images they cannot easily recreate. That's the role of a build system and making sure the process to create those artifacts is in source control.
I should also mention that your build script CAN be sourced from source control.
In your .vespene file, just say "script: <path>.
However, the variables in that file can be evaluated by Jinja2 variables, so for instance your security team could set up a snippet everybody could use or your feature flags could be defined from there.
Another cool feature of Vespene is in each project build root all those variables appear as vespene.json files, so it can be a good tool to use to launch all kinds of automation from a web console where you want to record results.
[+] [-] mpdehaan2|7 years ago|reply
I thought I'd share and if you have any questions, let me know!
[+] [-] df24g3t36|7 years ago|reply
[+] [-] tfinch|7 years ago|reply
(BTW thanks for ansible, it's awesome :) )
[+] [-] jweir|7 years ago|reply
[+] [-] rlopezcc|7 years ago|reply
[+] [-] unknown|7 years ago|reply
[deleted]
[+] [-] morenoh149|7 years ago|reply
[+] [-] ianamartin|7 years ago|reply
This looks like a tool that could be a lot closer to what I want to do and designed to do it that way.
Also, is the repo being moved somewhere? https://github.com/vespene/vespene is 404.
[+] [-] cbcoutinho|7 years ago|reply
https://github.com/vespene-io/vespene
[+] [-] mpdehaan2|7 years ago|reply
Still hard to keep myself from typing that :)
[+] [-] p1necone|7 years ago|reply
[+] [-] cjbprime|7 years ago|reply
So, you might consider a protocol for builders to report their results (e.g. over SSH) without being a full node, if that doesn't exist. Looks good!
[+] [-] mpdehaan2|7 years ago|reply
Any node typically would run the webserver because it's not that heavy, but doesn't technically have to. It could just run one worker process. Right now the workers do need database access and I suspect that will stay that way for the short term.
Windows needs to happen, but I'm not sure when it happens. (Also a good list discussion).
I'm a bit curious about your raspberry pi use case - maybe a good topic for the forum? http://talk.vespene.io/
[+] [-] caleblloyd|7 years ago|reply
I have been interested in building a general purpose CI/CD tool but this part seems like the most "grunt work". Is there an open source library that standardizes webhooks? Or would you be interested in publishing that portion in a library?
[+] [-] dsumenkovic|7 years ago|reply
[+] [-] michaelmcmillan|7 years ago|reply
[1] https://github.com/vespene-io/vespene/blob/master/vespene/mo...
[+] [-] mpdehaan2|7 years ago|reply
It seems like you're saying I should have used a many-to-many there rather than 7 FKs. Probably, but shrug... it can be fixed later if it ever becomes a problem.
[+] [-] unknown|7 years ago|reply
[deleted]
[+] [-] ensignavenger|7 years ago|reply
[+] [-] Sir_Cmpwn|7 years ago|reply
[+] [-] mpdehaan2|7 years ago|reply
I hope people know my history and how I've run projects in the past.
But what this does for me is keep me from releasing anything open core. With ansible, ansible was open source but the GUI was not, and we had to hold something back.
The traditional alternatives to OSS are to make a support business, which somewhat encourages making buggy or hard to install software, or a consulting business (IMHO, same) and all of those detract from building a product, collectively, with some of your favorite people on the internet.
So basically Vespene is going to be like a pseudo-foundation.
The structure for this is described here - http://docs.vespene.io/partnership.html - which keeps consulting and support totally free for small shops, and encouraging larger ones that could potentially make millions form Vespene to give back a very small amount.
What normally happens is large companies close six figure contracts supporting open source, and they don't give back a dime, and my goal here is to essentially make a developer's salary off of this project by having those who make more give back a very small amount.
This seems pretty fair to me, but I also recognize that some people don't agree with it.
That all being said, it's not a "no-commercial" clause in any way, and still free for small consultancies, so I'm not anticipating any major problems.
Most of the time, software gets installed in a place of business for that business.
For some of my previous thoughts on this, I'd refer folks to https://medium.com/@michaeldehaan/why-open-source-needs-new-... and later my current thinking https://medium.com/@michaeldehaan/going-with-the-commons-cla...
It's not something I've considered lightly, but this keeps MORE software open, and for a small one-person shop, I think that's more noble than trying to hold some code back and not release it at all.
Some people just want 100% free, gimme gimme, etc - and fine - you're entitled to pick one of those things. That's ok.
I personally view this a little more pragmatically, under the view of fairness, and also have taken steps (in the CLA, etc), to guarantee that trust is never going to be abused.
For instance, the license reverts to pure Apache if Vespene were ever to change hands.
On the downside, I can't get screwed over by IBM or Amazon while I'm trying to crank out free software for everyone. That seems fair too, seeing the time investment running an open source community and working on the code takes.
[+] [-] cagenut|7 years ago|reply
[+] [-] mpdehaan2|7 years ago|reply
[+] [-] bribri|7 years ago|reply
[+] [-] mpdehaan2|7 years ago|reply