There are countless SaaS applications asking for full-repo access to Github (all the source code, with write access).
- Productboard
- Bugsnag
- Sentry
- Skylight
- Percy
- CodeTree
- Databox
There are heaps of others, these are just some on top the of my mind. A ticking supply chain attack waiting to happen, since these companies make themselves into alluring hacking targets.
Most of them need access only to issues (a few need read access to code or recent commits, almost none need write).
Solution:
- Let customers give granular access (only issues, only read to source code, etc) when the integration is setup. This is possible with Github's APIs.
- Try to use push instead of pull where possible, i.e. provide a CLI tool to use with Github actions or use Github's webhooks.
> There are countless SaaS applications asking for full-repo access to Github (all the source code, with write access).
Sentry does not request write access to source code. It requests read/write access to issues and read access to source code. You can also see this on the documentation for the GitHub enterprise integration which lists the exact permissions required: https://docs.sentry.io/product/integrations/source-code-mgmt...
For a while now I've been worried about this -- either integrations asking for full read/write access for a service that might seem to need only read.
Or maybe worse, integrations asking for access to anything my account has access to, when I only want to grant it to one repo or organization, or only to public repos and not private ones.
Whenever I've reached out to inquire/complain about this, I've been told that github does not give them granular enough auth settings to ask for less than this.
Is this true? I don't know. When I've tried looking at the relevant github docs myself, i quickly get confused.
Does anyone understand the github auth architecture -- does it need to be fixed to allow more granular access, or are integrations just not using it properly? Like... who should I be complaining to?
> Additionally, we recommend disconnecting Heroku from your GitHub repositories.
I have respect for the Heroku/Salesforce Security team for willing to ask users to perform this action. Many companies would be too worried about losing customers or having users not reconnect it afterwards.
My thoughts are with the team working on responding to this incident on Easter Friday.
I can confirm that ~2h ago my integration was working but now it's not, so it seems they automatically disconnected everyone. Not sure if "they" is Github or Heroku here, but my master branch is not autodeploying anymore and trying to reconnect results into an error.
We're a small org with a github connected to heroku. All of our repos were cloned between April 8 and April 15 with the majority of them having no activity for several years. The audit logs don't show this, you can only see this information in the traffic graphs (/graphs/traffic). If you're seeing cloning of repos that you haven't touched in a while, you've likely been compromised.
For anyone not on a pro plan: I believe you can upgrade and still see the past two weeks of data. I cloned a few of my private repos last night to see how that affects my security logs and no logs appeared. I later upgraded to pro and visited /<username>/<repo>/graphs/traffic and can see the clone counts from before I upgraded. I also can see visitor counts from about a week ago. These clones still don't appear in the security logs though.
Are you on an enterprise plan or a personal one? I know personal plans have limited logs, but I'd hope the enterprise ones would show clones (in reality, they both probably should). Kind of defeats the purpose of an audit log if it doesn't.
Ughh here I was at 1:30AM after a hard week, checking the news one last time before going to bed, and see this. I hate this dang industry and regret ever becoming a tech lead. I know the Heroku engineers have it worse and all, but just venting.
News on HN are usually not relevant to my work and I actively avoid reading anything work related on my off time. I’d find some other activity to relax and wind down if reading HN has an impact on you. Burnout is no joke - you need yourself rested for yourself and your team. Take care of yourself.
I'm not a security expert, but if you're reading this and wondering what to do, a good start could be to just assume your repo was accessed, and so to run a tool like gitleaks against your repo. If it detects anything sensitive, I'd see about revoking/deleting those secrets right away.
In general, it's good practice not to check anything sensitive into source code for precisely this reason (if your code is compromised you don't want your secrets to be as well). So it'd also be good practice to add something like gitleaks into your CI/CD pipeline for the future.
I do remember hooking up Heroku to Github for auto-deployments and thinking to myself something along the lines of, "why does Heroku need ALL of this access?"
It'd be great if Github could allow read/write permission grants on a per-repo basis. Maybe they do already!.. in which case I'd much rather have and setup that granular detail than have a token that goes across all my public/private repos...
Edit: I do see in my Github's integration page that the Heroku connection was used within the past week... but it doesn't show how exactly it was used. Until Github can provide specific details, is it safe to assume that all repos, public and private, could have been cloned?
Disappointed that the GitHub security log doesn’t show access for personal accounts. Would be rather nice if they temporarily made that available for a short period of time so we can see if any of our repositories have been cloned/downloaded.
The attacker got _write_ access to all of these repositories as well? That's extremely worrying. I hope github or someone will be able to track down if any code changes were made.
A forced push could introduce vulnerability into code in a years old commit that is hard to detect. I guess it will look dodgy when your next push is rejected.
If I want to revoke all Heroku's access to Github, is it the "Heroku Dashboard" I'm looking for under "Authrorized Oauth apps"? I revoked that one, but not sure if that is everything.
Credentials and other secrets, like API keys, should never be hard-coded in the source code repo. Use some sort of secrets management or configuration for that kind of stuff.
In this case, can we confirm that Heroku environment variables were not accessed? Because if they were, even not storing secrets in the source code wouldn’t have prevented a breach.
If Heroku could confirm environment variables were safe I’d have a much better sleep tonight.
at work thats fine, we use vault or secrets manager, but neither of those are really suitable for self hosting for toy apps. What do you tell someone using github's free tier and aws/gcp's free tier?
I’ve been using Render since January, after switching from Heroku for a project of mine with about 10,000 direct users at present. The project is a Rails app that imports hundreds of thousands of rows of data from XLSX files and spits out PDFs on demand from those imported spreadsheets.
I was worried that Heroku would end up costing me a small fortune as demand scaled. Plus the platform seemed to have stagnated.
I contemplated switching to AWS, but didn’t want to deal with the extra hassle of it. By chance, I saw someone mention Render on here, checked it out, and couldn’t be happier.
It’s a bit harder to get up and running with Render than Heroku, but orders of magnitude easier than with AWS. And once you’re operational, it’s a cinch.
I have always wondered if putting confidential info in even private git repo was a good idea, although it seems to be a common practice? I feel like that question has been answered, for me anyway.
Instead of connecting to a Github repo, you can use https://github.com/heroku/heroku-builds. It allows to create a build locally and then deploy it to Heroku. From what I've read so far, this approach has not been impacted and it should still be possible to do deployments like that.
I just started a Discord server: https://discord.gg/K9ecetqn Please join if you're impacted by this incident, or interested in these topics in general.
I'd like to discuss mitigations around this and similar incidents with other HN:ers:
- Knowledge sharing: resources, how-tos, tips
- Discussing prevention, mitigation, etc
- Moral support and venting
If there's already such a forum (I assume there is), please send me an invite :)
[+] [-] sandstrom|3 years ago|reply
There are countless SaaS applications asking for full-repo access to Github (all the source code, with write access).
- Productboard
- Bugsnag
- Sentry
- Skylight
- Percy
- CodeTree
- Databox
There are heaps of others, these are just some on top the of my mind. A ticking supply chain attack waiting to happen, since these companies make themselves into alluring hacking targets.
Most of them need access only to issues (a few need read access to code or recent commits, almost none need write).
Solution:
- Let customers give granular access (only issues, only read to source code, etc) when the integration is setup. This is possible with Github's APIs.
- Try to use push instead of pull where possible, i.e. provide a CLI tool to use with Github actions or use Github's webhooks.
[+] [-] the_mitsuhiko|3 years ago|reply
Sentry does not request write access to source code. It requests read/write access to issues and read access to source code. You can also see this on the documentation for the GitHub enterprise integration which lists the exact permissions required: https://docs.sentry.io/product/integrations/source-code-mgmt...
[+] [-] jrochkind1|3 years ago|reply
Or maybe worse, integrations asking for access to anything my account has access to, when I only want to grant it to one repo or organization, or only to public repos and not private ones.
Whenever I've reached out to inquire/complain about this, I've been told that github does not give them granular enough auth settings to ask for less than this.
Is this true? I don't know. When I've tried looking at the relevant github docs myself, i quickly get confused.
Does anyone understand the github auth architecture -- does it need to be fixed to allow more granular access, or are integrations just not using it properly? Like... who should I be complaining to?
[+] [-] frays|3 years ago|reply
I have respect for the Heroku/Salesforce Security team for willing to ask users to perform this action. Many companies would be too worried about losing customers or having users not reconnect it afterwards.
My thoughts are with the team working on responding to this incident on Easter Friday.
[+] [-] ryantgtg|3 years ago|reply
[+] [-] czbond|3 years ago|reply
[+] [-] franciscop|3 years ago|reply
[+] [-] SnowHill9902|3 years ago|reply
[+] [-] anon3949494|3 years ago|reply
[+] [-] mariusz331|3 years ago|reply
[+] [-] from_endor|3 years ago|reply
[+] [-] Ozzie_osman|3 years ago|reply
[+] [-] mepiethree|3 years ago|reply
[+] [-] notimetorelax|3 years ago|reply
[+] [-] aaronbrethorst|3 years ago|reply
[+] [-] rosndo|3 years ago|reply
[+] [-] ketzo|3 years ago|reply
[+] [-] Ozzie_osman|3 years ago|reply
In general, it's good practice not to check anything sensitive into source code for precisely this reason (if your code is compromised you don't want your secrets to be as well). So it'd also be good practice to add something like gitleaks into your CI/CD pipeline for the future.
[+] [-] TheSpiciestDev|3 years ago|reply
It'd be great if Github could allow read/write permission grants on a per-repo basis. Maybe they do already!.. in which case I'd much rather have and setup that granular detail than have a token that goes across all my public/private repos...
Edit: I do see in my Github's integration page that the Heroku connection was used within the past week... but it doesn't show how exactly it was used. Until Github can provide specific details, is it safe to assume that all repos, public and private, could have been cloned?
[+] [-] pineconewarrior|3 years ago|reply
They totally do. Shopify's Github integration works this way, and it is fantastic!
[+] [-] samwillis|3 years ago|reply
[+] [-] fxtentacle|3 years ago|reply
https://github.blog/2022-04-15-security-alert-stolen-oauth-u...
[+] [-] jsmeaton|3 years ago|reply
[+] [-] duxup|3 years ago|reply
Yesterday I got a notification that someone tried logging into that Gmail account. The password was hard coded in the code…
[+] [-] r-s|3 years ago|reply
[+] [-] kadoban|3 years ago|reply
[+] [-] quickthrower2|3 years ago|reply
[+] [-] nu11ptr|3 years ago|reply
[+] [-] samcheng|3 years ago|reply
Credentials and other secrets, like API keys, should never be hard-coded in the source code repo. Use some sort of secrets management or configuration for that kind of stuff.
[+] [-] ryanSrich|3 years ago|reply
If Heroku could confirm environment variables were safe I’d have a much better sleep tonight.
[+] [-] maccard|3 years ago|reply
[+] [-] ozgune|3 years ago|reply
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] buf|3 years ago|reply
I've been eyeing it recently and I'm thinking about launching my next project with it. Does anyone have any takeaways from using Render vs Heroku?
[+] [-] aaronbrethorst|3 years ago|reply
I was worried that Heroku would end up costing me a small fortune as demand scaled. Plus the platform seemed to have stagnated.
I contemplated switching to AWS, but didn’t want to deal with the extra hassle of it. By chance, I saw someone mention Render on here, checked it out, and couldn’t be happier.
It’s a bit harder to get up and running with Render than Heroku, but orders of magnitude easier than with AWS. And once you’re operational, it’s a cinch.
And way, way, way cheaper.
[+] [-] leetrout|3 years ago|reply
[+] [-] fourstar|3 years ago|reply
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] dangoor|3 years ago|reply
Travis-CI was also compromised here and that may actually affect more people than the Heroku side of this.
[+] [-] dangoor|3 years ago|reply
[+] [-] jrochkind1|3 years ago|reply
[+] [-] nijave|3 years ago|reply
- it's stored locally (so any developer machine access could compromise secrets)
- it's stored on the build/CI machines (that might run untrusted code in the form of dependencies--possibly from unrelated repos)
- it can end up in build artifacts
[+] [-] uallo|3 years ago|reply
[+] [-] debarshri|3 years ago|reply
[+] [-] worker_thread|3 years ago|reply
[+] [-] andrelaszlo|3 years ago|reply
I'd like to discuss mitigations around this and similar incidents with other HN:ers:
- Knowledge sharing: resources, how-tos, tips - Discussing prevention, mitigation, etc - Moral support and venting
If there's already such a forum (I assume there is), please send me an invite :)