Rob Zuber, CTO of CircleCI here. We take security very seriously and are taking a deep look at the issues Kevin raised.
We'll save additional commenting until we have gathered more information. In the meantime, our security policy and steps for reporting issues are here: https://circleci.com/security/ and we'd like to ask the community to please use our outlined methods for reporting potential security issues so we can keep CircleCI as safe as possible for everyone.
OP here. Thanks for responding promptly. To be clear, letting third party JS run in a trusted environment like a dashboard is an industry wide problem. If we assume CircleCI is the only bad actor we're kind of missing the point of the exercise.
CircleCI is a notable example, due to the fact it hosts source code and secrets for so many different companies and loads so many third party scripts in its dashboard context. But they're not unique in this regard. I hope everyone reading this is reconsidering the scripts that have access to their dashboard and pushing for changes at their company.
If CircleCI was immediately vulnerable to injection via the scripts above I would have used the private disclosure route and I encourage others to as well. But I don't know that there is a "vulnerability" here so much as a discussion that we need to have about what and how much third party code we let run in a trusted context. I wrote a little more in the thread below, https://news.ycombinator.com/item?id=15442988.
I'm not sure private disclosure is appropriate here.
This isn't the same as a browser vulnerability where some kid could hack a load of people before they patched. One of these supposedly trusted companies could attack your customers... But they could already. They know their scripts get loaded into all sorts of inappropriate environments.
Getting people notified, getting credentials changed. That's the paramount concern. You can rub PR lotion into an in-depth audit later on.
Is there any way to limit what access CircleCI has to my GitHub account/orgs? GitHub says you have "Full control of private repositories," but it seems like all you need is read access + a push webhook.
What serious business hosts their proprietary, mission critical code on someone else's computers? Isn't that like Coke or KFC putting their original recipe on Google Drive?
Edit: Okay, I get it. People like convenience over security. I'll stick to self-hosting a Drone-CI instance and only deploying binaries.
At GitLab we separated the infrastructure between about.gitlab.com (our marketing site, lots of third party javascript) and gitlab.com (our application, zero third party javascript). We recently deprecated the Piwik instance we ran in house for gitlab.com because it slowed the page load time. Please let me know if there is anything we can do better, these things are complex.
Google's policies are a big part of this problem. Ad code and tracking code ought to be in an iframe, where the code can't snoop on the surrounding page context. Google insists that their ad code must not be in an iframe. That gives the lesser players an opening to insist that they, too, should't be constrained.
A few big site operators should push back against Google on this. The New York Times and CBS, for example. They use Google Publisher Tags, but no Google ads, so they don't really need Google on their pages.
Most ad code does run in iframes and the industry standard is to use "friendly iframes" which offer security but also access to the parent window. And even if you use iframes, something has to create that iframe initially and publisher devs are not going to do that manually, which is why there are top level script tags. All tag managers and adservers today default to using iframes for 3rd party tags.
> Send an email any time an API access token is created. Add a setting to allow org-wide disabling of API token creation.
I wish more apps supported this.
Also on my list is a "Don't ever let me disable 2FA" setting. I'm more worried about malicoius resets than I am about ever losing my 2FA device and backup codes.
> Add an option to delete old logs. If you have ever dumped env vars to the log file, an attacker can export these.
I surprised that secrets make it into the logs at all. I would have expected them to filter anything that's in an env secret from the log output. Pretty sure Travis does this.
> Enable subresource integrity, or serve JS from each of these companies from the CircleCI domain.
That's not much of an option for this type of thing as the third parties would then have a PITA time dealing with upgrades (so they wouldn't support it). Using <script src="hxxps://tracker.example.com/path/to/script.js"></script> (rather than a fixed version) allows for live upgrades as they're in control of the resource that is loaded.
And you can't load the script from your domain as at the end of the day it's got to get instructions and upload data to the third party. At best you'd be loading a shim that does the same thing as the script tag.
Wouldn't this be possible to circumvent if the user is allowed to switch TOTP devices? Are you saying you'd like a way to irrevocably tie this to a single TOTP secret? And you're not worried that someone could steal your TOTP secret and you'd be 100% powerless to stop them?
I feel calling all 8 of these companies "analytics" is a little unfair.
Pusher is a SaaS product for websockets and push notifications. I think it's reasonable that a company like CircleCI might outsource this instead of maintaining the infrastructure for it in-house.
Launch Darkly is a feature flagging service. While I feel the value is much less than Pusher (and I'd be inclined to just build an in-house solution), I think it's still a relatively reasonable tool to use to create a better user experience.
Intercom when used for customer communication is also something that I feel I'd want as a user, although this could perhaps only be loaded when requested by the user.
Why does this matter? Most 3rd party services are "reasonable" in some setting. When you build a highly-sensitive application, the bar for reasonableness changes.
So, startups and companies hire for standard, coglike (CTCI) CS skills, yet they outsource basic functionality.
There should be a corollary to NIH Syndrome[1] that takes into account 3rd party APIs. The way I see it, it's not about reinventing the wheel, it's about having your house in order.
I feel like narrowing down what any of those companies could do to what they are doing today irresponsible. And analytics is one of the could-do's that many third party tools do offer or add on later to increase monetization or perceived value to customers.
The important point is that there are now 8 attack surfaces instead of 1. Whether they're analytics or pad-string doesn't really matter.
Thinking about mitigation here... It appears that some of the included scripts have crossorigin="anonymous" script tags. Wouldn't this prevent authenticated access to to the circleCI domain, aka preventing the creation of api tokens or access to the API using the logged in browser context for any script loaded off the circleci domain?
Also, not that they do so, but would Access-Control-Allow-Origin set to something other than * prevent 3rd party requests to the API for scripts loaded from 3rd party domains.
Also curious if anyone has written a JS library that patches XMLHttpRequest.prototype to audit exfiltration of data in the DOM.
I use CircleCI because it allows 1 free private repo. Personally though I do not like Circle too much, it has a lot of cool features but the UI loads so slowly because they have so much stuff going on. They use the UI skeletons technique to make the load time appear to be faster but it is still very apparent how slow the load time is.
I have hit numerous bugs with their website as well where stuff doesn't load, builds kick off into infinity, when I transferred a repo everything broke, ack!
Furthermore, I am frustrated with their pricing, they go from free to $50/mo for the next upgrade tier, that seems like a crazy jump to me. I would gladly pay $10/mo or so for another container or 2x parallelization. The issue is being a single developer I rarely would save any time and the $50/mo is just too steep.
Finally, when I tried to build a justification case for my manager to pay the $50/mo, I wanted to use data from their CircleCI Insights which shows how long your builds are queued on average and some other important data points. But you cannot access these insights from a free account. Seems like that info should be available and prominent to help people understand the cost-savings they might receive by upgrading. I emailed support and asked for a one-time data point for that statistic to build the case as I was considering upgrading and they said sorry, nothing we can do for you. Is their goal to make money and have happy customers? If so they aren't doing a great job of it.
Overall a lot of frustrations with the platform and this just adds more fuel to the fire.
Why not Shippable? Free for 150 builds/month for private projects, brand new UI created with a focus on usability and perf, integrates with GitHub and Bitbucket, and we're adding metrics very shortly.
Loading third party JS is increasingly common for a lot of sites, and I tend to raise it when doing security reviews, for this reason, you're trusting the security of those 3rd parties.
There are some defenses that can be put in place. The first one is kind of awkward in many cases which is to host the JS on your own domain. There's still the risk of course that it will go off and get additional code from the 3rd party source to execute, but that can be reviewed for.
I noticed that Galaxy - Meteor Development Group’s hosting solution for Meteor apps does similar. I checked the console and saw multiple analytics, even some that seem to monitor your visual usage/screenshots etc, which would seemingly capture screens that have portions of environment variables / settings etc. So basically potentially a lower level marketing employee who has an infected laptop and is reviewing UX patterns etc could expose sensitive data on their MacBook Air that could potentially be infected. They might not expose that data. But I had the same thoughts as this article points out.
I'm trying to understand why "CircleCI browser context has full access to the CircleCI API, which is hosted on the same domain".
Doesn't API have seperate authentication credentials (API key/tokens) than the web UI (i.e. cookies)? I understand these loaded 3rd party scripts can scrape what's rendered on the UI, but making calls to the API??? I wonder how that's possible in CircleCI case.
Wouldn't similar vulnerability be present in most SAAS platforms? All of them include tons of analytics scripts on their website. If you are a logged in user, your session presumably includes a token (encrypted cookie?) to use the SAAS API.
Therefor in theory other scripts loaded on the page could grab the token and make authenticated API calls as well? Although I guess this is mitigated by verifying script integrity when loading scripts from CDN (e.g. integrity attribute of script tag)?
> Why don't you include a POC? I have one that demonstrates this attack, but I don't want to show script kiddies how. If you can't figure out how to construct it by reading the above description and the network traffic, you don't deserve to know how.
I feel this is a bad way to approach it and a bad attitude in general. Showing people how this can be exploited (after responsible disclosure) is a great opportunity to spread awareness of potentially their own issues in the same space and create points of discussion. If a user feels compelled to do so, they can seek out the info. But to show disdain to the reader for their lack of knowledge comes off as unprofessional in spirit.
As someone who has been a web developer since the 90's, this topic brings me back to ~2005 when Google Analytics launched. I thought it was absolutely insane that people would just stick someone else's JavaScript in their pages (and especially on e-commerce sites!).
Today it's perfectly normal. I don't have exact numbers but at least 50% of all sites pull in third party code. It really is bizarre to me that this became the norm.
I think these kind of issues will become more and more important, both from a security standpoint, and from a privacy standpoint as well. GPDR might apply to api access from 3rd party tools (not just analytics btw) - it’ll be up to them to prove they don’t, and I’m curious to see how it’ll be regulated.
I’ve tried to explore the privacy aspects here [1] (like many, we completely separated libraries for the public site and libraries for the apps)
I think we need a framework when thinking about which companies to pick when adding 3rd party features to our products - and probably a system to trust such 3rd party with their data processing.
I’d be interested to know how companies make these decisions today...
Marketing and sales people usually ask to have tons of third party scripts loaded everywhere and devs don't have power to turn them down. They need analytics for their work. It's a bad idea from security standpoint for sure.
Also, Github and Bitbucket should allow much more granular ACLs such as single repo, single repo shallow clone only. Ideally they would also support some form of steganographic tagging of repo contents as well, in case of a breach.
[+] [-] z00b|8 years ago|reply
We'll save additional commenting until we have gathered more information. In the meantime, our security policy and steps for reporting issues are here: https://circleci.com/security/ and we'd like to ask the community to please use our outlined methods for reporting potential security issues so we can keep CircleCI as safe as possible for everyone.
[+] [-] kevinburke|8 years ago|reply
CircleCI is a notable example, due to the fact it hosts source code and secrets for so many different companies and loads so many third party scripts in its dashboard context. But they're not unique in this regard. I hope everyone reading this is reconsidering the scripts that have access to their dashboard and pushing for changes at their company.
If CircleCI was immediately vulnerable to injection via the scripts above I would have used the private disclosure route and I encourage others to as well. But I don't know that there is a "vulnerability" here so much as a discussion that we need to have about what and how much third party code we let run in a trusted context. I wrote a little more in the thread below, https://news.ycombinator.com/item?id=15442988.
[+] [-] oliwarner|8 years ago|reply
This isn't the same as a browser vulnerability where some kid could hack a load of people before they patched. One of these supposedly trusted companies could attack your customers... But they could already. They know their scripts get loaded into all sorts of inappropriate environments.
Getting people notified, getting credentials changed. That's the paramount concern. You can rub PR lotion into an in-depth audit later on.
[+] [-] bsimpson|8 years ago|reply
[+] [-] caust1c|8 years ago|reply
[+] [-] _Codemonkeyism|8 years ago|reply
[deleted]
[+] [-] atomi|8 years ago|reply
Edit: Okay, I get it. People like convenience over security. I'll stick to self-hosting a Drone-CI instance and only deploying binaries.
[+] [-] unknown|8 years ago|reply
[deleted]
[+] [-] sytse|8 years ago|reply
[+] [-] tptacek|8 years ago|reply
[+] [-] SkyPuncher|8 years ago|reply
[+] [-] unknown|8 years ago|reply
[deleted]
[+] [-] pg_is_a_butt|8 years ago|reply
[deleted]
[+] [-] hungerstrike|8 years ago|reply
Since you're asking though, here's one thing you can do better: Stop spamming HN with your irrelevant nonsense.
[+] [-] BoorishBears|8 years ago|reply
[+] [-] eropple|8 years ago|reply
You can not opportunistically spam nearly every thread where GitLab might be tangentially related.
I don't use GitLab largely because of the way you post on HN, though the software seems reasonably well-written. It's very frustrating.
[+] [-] Animats|8 years ago|reply
A few big site operators should push back against Google on this. The New York Times and CBS, for example. They use Google Publisher Tags, but no Google ads, so they don't really need Google on their pages.
[+] [-] manigandham|8 years ago|reply
[+] [-] icebraining|8 years ago|reply
[+] [-] unknown|8 years ago|reply
[deleted]
[+] [-] unknown|8 years ago|reply
[deleted]
[+] [-] koolba|8 years ago|reply
I wish more apps supported this.
Also on my list is a "Don't ever let me disable 2FA" setting. I'm more worried about malicoius resets than I am about ever losing my 2FA device and backup codes.
> Add an option to delete old logs. If you have ever dumped env vars to the log file, an attacker can export these.
I surprised that secrets make it into the logs at all. I would have expected them to filter anything that's in an env secret from the log output. Pretty sure Travis does this.
> Enable subresource integrity, or serve JS from each of these companies from the CircleCI domain.
That's not much of an option for this type of thing as the third parties would then have a PITA time dealing with upgrades (so they wouldn't support it). Using <script src="hxxps://tracker.example.com/path/to/script.js"></script> (rather than a fixed version) allows for live upgrades as they're in control of the resource that is loaded.
And you can't load the script from your domain as at the end of the day it's got to get instructions and upload data to the third party. At best you'd be loading a shim that does the same thing as the script tag.
[+] [-] orginal__idear|8 years ago|reply
I like this idea. I'd only want to use this though if my Google 2FA backup codes are backed up via Authy or a similar app.
I don't trust codes I've printed out and have always lost.
[+] [-] rbranson|8 years ago|reply
Wouldn't this be possible to circumvent if the user is allowed to switch TOTP devices? Are you saying you'd like a way to irrevocably tie this to a single TOTP secret? And you're not worried that someone could steal your TOTP secret and you'd be 100% powerless to stop them?
[+] [-] dom0|8 years ago|reply
It does.
[+] [-] danpalmer|8 years ago|reply
Pusher is a SaaS product for websockets and push notifications. I think it's reasonable that a company like CircleCI might outsource this instead of maintaining the infrastructure for it in-house.
Launch Darkly is a feature flagging service. While I feel the value is much less than Pusher (and I'd be inclined to just build an in-house solution), I think it's still a relatively reasonable tool to use to create a better user experience.
Intercom when used for customer communication is also something that I feel I'd want as a user, although this could perhaps only be loaded when requested by the user.
[+] [-] tptacek|8 years ago|reply
[+] [-] rhizome|8 years ago|reply
There should be a corollary to NIH Syndrome[1] that takes into account 3rd party APIs. The way I see it, it's not about reinventing the wheel, it's about having your house in order.
1. https://en.wikipedia.org/wiki/Not_invented_here
[+] [-] ballenf|8 years ago|reply
The important point is that there are now 8 attack surfaces instead of 1. Whether they're analytics or pad-string doesn't really matter.
[+] [-] epanastasi|8 years ago|reply
Also, not that they do so, but would Access-Control-Allow-Origin set to something other than * prevent 3rd party requests to the API for scripts loaded from 3rd party domains.
Also curious if anyone has written a JS library that patches XMLHttpRequest.prototype to audit exfiltration of data in the DOM.
[+] [-] noway421|8 years ago|reply
[+] [-] nexfitter|8 years ago|reply
I have hit numerous bugs with their website as well where stuff doesn't load, builds kick off into infinity, when I transferred a repo everything broke, ack!
Furthermore, I am frustrated with their pricing, they go from free to $50/mo for the next upgrade tier, that seems like a crazy jump to me. I would gladly pay $10/mo or so for another container or 2x parallelization. The issue is being a single developer I rarely would save any time and the $50/mo is just too steep.
Finally, when I tried to build a justification case for my manager to pay the $50/mo, I wanted to use data from their CircleCI Insights which shows how long your builds are queued on average and some other important data points. But you cannot access these insights from a free account. Seems like that info should be available and prominent to help people understand the cost-savings they might receive by upgrading. I emailed support and asked for a one-time data point for that statistic to build the case as I was considering upgrading and they said sorry, nothing we can do for you. Is their goal to make money and have happy customers? If so they aren't doing a great job of it.
Overall a lot of frustrations with the platform and this just adds more fuel to the fire.
[+] [-] daxorid|8 years ago|reply
[+] [-] manishas|8 years ago|reply
Upgrade to an additional container is just $25. \
[+] [-] raesene6|8 years ago|reply
There are some defenses that can be put in place. The first one is kind of awkward in many cases which is to host the JS on your own domain. There's still the risk of course that it will go off and get additional code from the 3rd party source to execute, but that can be reviewed for.
The other option is to use sub-resource integrity (https://developer.mozilla.org/en-US/docs/Web/Security/Subres...) to ensure that only scripts you've reviewed are used.
Of course you need then to make sure you're notified before the 3rd party makes changes that would break the signature.
[+] [-] sbr464|8 years ago|reply
[+] [-] alpb|8 years ago|reply
Doesn't API have seperate authentication credentials (API key/tokens) than the web UI (i.e. cookies)? I understand these loaded 3rd party scripts can scrape what's rendered on the UI, but making calls to the API??? I wonder how that's possible in CircleCI case.
[+] [-] richardknop|8 years ago|reply
Therefor in theory other scripts loaded on the page could grab the token and make authenticated API calls as well? Although I guess this is mitigated by verifying script integrity when loading scripts from CDN (e.g. integrity attribute of script tag)?
[+] [-] Posibyte|8 years ago|reply
I feel this is a bad way to approach it and a bad attitude in general. Showing people how this can be exploited (after responsible disclosure) is a great opportunity to spread awareness of potentially their own issues in the same space and create points of discussion. If a user feels compelled to do so, they can seek out the info. But to show disdain to the reader for their lack of knowledge comes off as unprofessional in spirit.
[+] [-] jeffnappi|8 years ago|reply
Today it's perfectly normal. I don't have exact numbers but at least 50% of all sites pull in third party code. It really is bizarre to me that this became the norm.
[+] [-] CaliforniaKarl|8 years ago|reply
>Why don't you include a POC? I have one that demonstrates this attack, but I don't want to show script kiddies how.
That leads me to this question: Did the author reach out to CircleCI to report this issue, before publishing their blog post?
[+] [-] kevinburke|8 years ago|reply
[+] [-] nathan_f77|8 years ago|reply
Or is that a browser extension that the OP forgot about?
[+] [-] motdiem|8 years ago|reply
I’ve tried to explore the privacy aspects here [1] (like many, we completely separated libraries for the public site and libraries for the apps)
I think we need a framework when thinking about which companies to pick when adding 3rd party features to our products - and probably a system to trust such 3rd party with their data processing.
I’d be interested to know how companies make these decisions today...
[1] https://blog.getkumbu.com/posts/building-a-privacy-respectfu...
[+] [-] Sir_Cmpwn|8 years ago|reply
[+] [-] richardknop|8 years ago|reply
[+] [-] coding123|8 years ago|reply
[+] [-] ivanfon|8 years ago|reply
[+] [-] matthewcford|8 years ago|reply
[+] [-] grandalf|8 years ago|reply