I'm not all that surprised. A friend saw a phishing email that was imitating them because they lacked a DMARC record. Sent them explicit instructions on how to fix it by adding a DMARC policy and all they did was create a p=none record that doesn't prevent direct imitation. That's definitely the first step, but eventually you need to turn it up to p=quarantine for it to do you any good and it's been a while (several weeks). Shouldn't have needed a random user to point it out in the first place.
I just don't have a tremendous amount of confidence that they take their infrastructure seriously at this point.
I still fail to see the heavily opinionated appeal of CircleCI over running a dockerized Jenkins instance (and agents) in AWS. (Or GitHub Actions or any other managed CI environment)
We get all the customization I want and it scales just fine.
But I am also still annoyed that when CircleCI announced their templates, they did not offer you the ability to have private templates (or something along those lines, it left a bad taste in my mouth and we moved to Jenkins a couple months later)
There are _a lot_ of companies/teams that don't need/want to manage the complexity that comes with setting up and configuring generic CI environments. They want to get going as fast as possible and never have to think about what's going on behind the scenes - that's especially true in some specific areas. For example mobile apps. 90% of people aren't too happy about managing all the complexity that comes with having to juggle Xcode versions or Android emulators etc in a scalable way.
..at least that's my experience and why I founded an extremely opinionated CI service for mobile apps years ago :)
I used CircleCI a long time ago, just before and after 2.0, and thought it was fine.
These days GitHub Actions is awesome and full featured and does everything I want though without getting in the way. I doubt I'd use anything besides GHA unless I wanted to decouple from GitHub as a dependency.
Even then, I wouldn't be surprised if someone else hasn't already come up with an open source or local runner for the GHA yaml files as a stopgap.
I wonder how much it's going to take before people realize that maybe a single server somewhere in the office running Jenkins isn't that bad of an idea after all. Unless you're Google, "scale" will inherently not be a problem, and risks of operator error can be reduced by scheduling maintenance at times where an accidental outage won't impact your business.
I agree with the sentiment that people should evaluate whether or not they need an external service to run their builds.
That said, there are a number of reasons to not use the Jenkins server in the office:
1) Someone on staff needs to maintain it.
2) A single hardware failure can cause significant downtime.
3) Your office internet service may have limited bandwidth and be a bottleneck for your build or artifact deployment
4) Having your server on-site may be considered a security risk.
I'm not saying that a server in the office is a bad idea, I'm just saying that each business needs to consider the advantages and disadvantages. I'm sure there are those who could get by just fine with a server in the office.
Folks are likely going to say they can't do this "at scale". However, I've been in both the "small shop with Jenkins running on a Dell blade" and also in the "mega-corp with lots of devs everywhere using all kinds of CI pipelines". I think there is merit in what you're saying, but I suggest an addendum.
In both cases, have seen the "CI is down" issues crop up.
There seems to be a middle ground that I have yet to fully do, but have seen folks do and heard it works well:
Make it so folks can deploy their CI pipeline anywhere. In my case, I would create Docker containers that can run the CI pipeline, pipe the output/result to anywhere you want and have it be able to run anywhere.
The goal is that you could deploy it on a VPS somewhere, or on a Dell blade in your closet, or even run it on a dev's machine. The idea is that you want to reliably be able to run the CI steps anywhere without jank while following the full process the same as if it was running on your dedicated job runner solution. Redundancy, if you will.
I think the idea isn't necessarily "Jenkins in your company closet is better" and more of "how easily can you setup your CI pipeline to get your work done?"
I personally really dislike remote building. It's so much faster to do everything locally, even more if you can do it on your own machine (machine which is likely to be very powerful, close to the state-of-the art performance, as the tech geeks we are).
CircleCI used to be an absolutely awesome way to set up CI. The default images just worked for the vast majority of cases. Sure, it always took a bit of fiddling with YAML, but IMO was way ahead of Jenkins et al when it came out.
Then they reworked their YAML format to enable pipelines (and make everything way more complicated, and break all our existing flows). Then they reworked their images to make them more efficient or something, but now there is almost never an image that has everything I need, so I have to build my own image in docker (with yet another CircleCI job). Now that Github Actions exists, we have been slowly migrating everything off CircleCI. Too bad they lost the simplicity that brought us there in the first place.
I wish it was only monthly. We have a Slack alert whenever there is reported downtime and it goes off at least once a week. Though granted it's seldom as business impacting as this current outage.
The problem CircleCI faces is that most hosted VCS now support CI/CD tools, as do most enterprise clouds. These will all have better integration with most peoples systems because they'll already be using the VCS or the public cloud (and if you're not using the public cloud you'd likely favor Jenkins / Concourse / etc over a cloud CI solution). So CircleCI's relevance is constantly being eaten at. The last thing they need it to damage their own reputation with these constant outages. I really hope CircleCI can turn around as it's great to see some competition but at this point in time I'm not feeling to optimistic about their long term future.
I'm switching to X and will be happy until either,
a) X gets large enough to matter and also has scale issues.
b) X gets acquired by Y and sends a "what a great journey we're so excited that X will have Y's resources" email and then inevitably becomes just another forgotten tool under Y
to all those saying "why you don't spin a GitLab CI instance" - we are a small team, we want to focus on shipping code that adds value to our customers, not maintaining something that has been largely commoditised.
oxff|3 years ago
Sounds like they haven't got the first clue about what is causing it.
brightball|3 years ago
I just don't have a tremendous amount of confidence that they take their infrastructure seriously at this point.
ismayilzadan|3 years ago
nerdjon|3 years ago
We get all the customization I want and it scales just fine.
But I am also still annoyed that when CircleCI announced their templates, they did not offer you the ability to have private templates (or something along those lines, it left a bad taste in my mouth and we moved to Jenkins a couple months later)
tauntz|3 years ago
..at least that's my experience and why I founded an extremely opinionated CI service for mobile apps years ago :)
tedmiston|3 years ago
These days GitHub Actions is awesome and full featured and does everything I want though without getting in the way. I doubt I'd use anything besides GHA unless I wanted to decouple from GitHub as a dependency.
Even then, I wouldn't be surprised if someone else hasn't already come up with an open source or local runner for the GHA yaml files as a stopgap.
Nextgrid|3 years ago
justin_oaks|3 years ago
That said, there are a number of reasons to not use the Jenkins server in the office:
1) Someone on staff needs to maintain it.
2) A single hardware failure can cause significant downtime.
3) Your office internet service may have limited bandwidth and be a bottleneck for your build or artifact deployment
4) Having your server on-site may be considered a security risk.
I'm not saying that a server in the office is a bad idea, I'm just saying that each business needs to consider the advantages and disadvantages. I'm sure there are those who could get by just fine with a server in the office.
softfalcon|3 years ago
In both cases, have seen the "CI is down" issues crop up.
There seems to be a middle ground that I have yet to fully do, but have seen folks do and heard it works well:
Make it so folks can deploy their CI pipeline anywhere. In my case, I would create Docker containers that can run the CI pipeline, pipe the output/result to anywhere you want and have it be able to run anywhere.
The goal is that you could deploy it on a VPS somewhere, or on a Dell blade in your closet, or even run it on a dev's machine. The idea is that you want to reliably be able to run the CI steps anywhere without jank while following the full process the same as if it was running on your dedicated job runner solution. Redundancy, if you will.
I think the idea isn't necessarily "Jenkins in your company closet is better" and more of "how easily can you setup your CI pipeline to get your work done?"
xcambar|3 years ago
It is significantly cheaper and more efficient to pay a third party.
XCSme|3 years ago
xyst|3 years ago
aftbit|3 years ago
Then they reworked their YAML format to enable pipelines (and make everything way more complicated, and break all our existing flows). Then they reworked their images to make them more efficient or something, but now there is almost never an image that has everything I need, so I have to build my own image in docker (with yet another CircleCI job). Now that Github Actions exists, we have been slowly migrating everything off CircleCI. Too bad they lost the simplicity that brought us there in the first place.
proxysna|3 years ago
hnlmorg|3 years ago
The problem CircleCI faces is that most hosted VCS now support CI/CD tools, as do most enterprise clouds. These will all have better integration with most peoples systems because they'll already be using the VCS or the public cloud (and if you're not using the public cloud you'd likely favor Jenkins / Concourse / etc over a cloud CI solution). So CircleCI's relevance is constantly being eaten at. The last thing they need it to damage their own reputation with these constant outages. I really hope CircleCI can turn around as it's great to see some competition but at this point in time I'm not feeling to optimistic about their long term future.
kldx|3 years ago
frays|3 years ago
carimura|3 years ago
a) X gets large enough to matter and also has scale issues.
b) X gets acquired by Y and sends a "what a great journey we're so excited that X will have Y's resources" email and then inevitably becomes just another forgotten tool under Y
/s sorta
kawsper|3 years ago
bpicolo|3 years ago
mr337|3 years ago
Been on BK for about 6mo now and still loving it.
keithpitt|3 years ago
thinkindie|3 years ago
PrimeDirective|3 years ago
folkrav|3 years ago
What do you mean by "these types of services"?
shdh|3 years ago
GitLab CI #1 in my opinion.