The fact that this key was apparently not stored in an HSM, and that GH employees had access to this private key (allowing them to accidentally push it) means that effectively all communication with GH since the founding of the company has to be considered compromised. This basically means that, depending on your level of paranoia, you will have to review all code that has ever interacted with Github repositories, and any code pushed or pulled from private repositories can no longer be considered private.
Github's customers trust GH/MS with their code, which, for most businesses, is a high value asset. It wouldn't surprise me if this all results in a massive lawsuit. Not just against GH as a company, but also to those involved (like the CISO). Also, how on earth was it never discovered during an audit that the SSH private key was plain text and accessible? How has GH ever been able to become ISO certified last year [0], when they didn't even place their keys in a HSM?
Obviously, as a paying customer I am quite angry with GH right now. So I might be overreacting when I write this, but IMO the responsible persons (not talking about the poor dev that pushed the key, but the managers, CISO, auditors, etc.) should be fined, and/or lose their license.
SSH uses ephemeral keys. It's not enough to have the private key and listen to the bytes on the wire, you have to actively MITM the connection. A github employee who has access to the private key and enough network admin privileges to MITM your connection already has access to the disk platters your data is saved on.
Regarding the secrecy of the data you host at github, you should operate under the assumption that a sizeable number of github employees will have access to it. You should assume that it's all sitting unencrypted on several different disk platters replicated at several different geographically separated locations. Because it is.
One of the things that you give up when you host your private data on the cloud is controlling who and under what circumstances people can view or modify your private data. If the content of the data is enough to sink you/your company without remedy you should not store it on the cloud.
> How has GH ever been able to become ISO certified last year?
ISO/IEC 27001:2013 doesn’t say you have to store private keys in HSMs? It just requires you to have a standards compliant ISMS that implements all Annex A controls and all of the clauses. Annex A and the clauses don’t specifically mandate this.
If you can convince an auditor that you have controls in-place that meet the standard for protecting cryptographic media you basically meet the standard. The controls can be a wide variety of options and don’t specifically mandate technical implementation details for many, many things.
You shouldn’t rely on ISO/IEC 27001:2013 as attestation for technical implementation details here. Just because your auditor would red flag you doesn’t mean all auditors would. The standard is only as effective as the weakest, cheapest auditor, and there are perverse incentives that make auditors financially incentivized to certify companies due to recurring revenue.
> [...] depending on your level of paranoia, you will have to review all code that has ever interacted with Github repositories [...]
Not to diminish the problems of having a large entity like Github handle a private key like that, but if that was your level of paranoia, you probably should have used commit signatures all along and not relied on Github to do that job for you.
Git provides the ability for authors to sign their commits with their own private key. To ensure the integrity of code in a repository, this method should be relied on rather than whatever hosting provider(s) have a copy of the repository.
Requiring all commits to be signed by trusted keys avoids the risks associated with someone tampering with a repository hosted on GitHub if they are able to get access to it, although it doesn't protect against code being leaked.
> The fact that this key was apparently not stored in an HSM, and that GH employees had access to this private key (allowing them to accidentally push it) means that effectively all communication with GH since the founding of the company has to be considered compromised.
For a host key? Like I get that being able to impersonate Github isn't great as far as state level actors having the ability to do this but you do know the actual transport layer keys are ephemeral and aren't derived at all from the host key, right?
> How has GH ever been able to become ISO certified last year [0], when they didn't even place their keys in a HSM?
ISO 27001 certification does not require you to put keys into an HSM. The standard requires you to have controls in place, be aware of your risks and to maintain a risk register. But in no way does the standard require HSM's.
The standard would even be OK with storing this on a floppy drive if the risks surrounding that were identified and mitigated (or accepted).
It's easy to say "should have used an HSM" (or, in truth, many HSMs), but I can appreciate the technical challenges of acutually doing that at their scale. It would not be a trivial project. There's a ton of operational concerns here, including figuring out how you would go about rotating the key on all those HSMs in an emergency.
> the responsible persons (not talking about the poor dev that pushed the key, but the managers, CISO, auditors, etc.) should be fined, and/or lose their license.
By no means do I want to see the dev get fined or blackballed from the industry.
But if there is any 1 person responsible, it’s the person who did it. The reason why the dev shouldn’t be fined/blackballed is because it’s not just 1 person’s fault. I mean, fining or booting his manager out of the industry? Really?
1) Nation state level actors can probably insert or compromise high level staff, or multiple high level staff, at any given company, and perform MITM attacks fairly easily. And some could compel turning over your code or secrets more directly anyway. Not worth worrying about this scenario: nobody working on anything truly sensitive should be using any externally hosted (or even externally networked) repositories.
2) It is much more difficult for other actors to do a MITM attack, and if they did, they'd probably have access to your code more directly.
3) Your code actually isn't worth much to anybody else. Imagine someone launching a complete clone of HN or any other site. Who cares? Nobody. What makes your code valuable is that you have it, and that you have a network and relationship with your customers. If somebody stole my company's codebases, I'd feel sorry for them, that they are going to waste any time wading through these useless piles of shit. The only potential problems are if secrets or major vulnerabilities are exposed and provide a path for exploit (like ability to access servers, services, exposing potential ransomware attacks).
Your 100% right to hold critical infrastructure to higher standards.
Putting Solarwinds aside, how many companies could to grind to a halt via this 3rd party.
> The fact that this key was apparently not stored in an HSM, and that GH employees had access to this private key (allowing them to accidentally push it) means that effectively all communication with GH since the founding of the company has to be considered compromised.
I think this suggests we need more information from github. For instance GH employees may not always have had live access to this key, this could have happened as part of an operation that gave temporary access to an employee only recently. Or it could have been stored plaintext on multiple employees' home computers since creation.
Looking from the outside, for many of these companies: GitHub, OpenAI, Cloudflare, Facebook, and so on, it seems that they torture their hires with ridiculously code challenges. Spend a lot of time on elegant engineering blogs. Write about how many Phds work at their locations, can't stop talking about how selective of the 0.01% Developers they are... But then, internally, everything seems more or less tied up with Shoestrings and Rube Goldberg machines.
It is so incredibly rare for public-facing service keys to be stored on an HSM that I don't think anyone could reasonably have expected this to be the case?
I don't think there will be any lawsuits. The user agreement precludes that. I don't even know how anybody could be angry about this - your code's on somebody else's computer and if you didn't know that that's a huge risk, you do now.
Please before replacing your local fingerprint with the new one, double check it is the expected value. This is an opportune time for man-in-the-middle attackers to strike, knowing everyone has to replace their stored signatures, and that some will be lazy about it with a blind "ssh-keygen -R github.com" command.
Github had one RSA ssh host key, the most widely supported key format.
It was trusted to clone code into the infrastructure of hundreds of thousands of organizations. People pin it everywhere.
With this key someone could infect the infrastructure of fintech companies and steal billions of dollars. I know this well because I run a security consulting company focusing mostly on that industry. Mainly this is possible because almost no companies check for git commit signing, and Github does not enforce it anyway, and I digress.
This key held enough power over value that some might have used violence to acquire it.
With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right?
Nope. They admit they just stuck it in a git repo in plain text where any engineer could have copied it to a compromised workstation, or been bribed for it, for who knows how many years. Worse, it was not even some separate employee only intranet git repo, but their own regular public production infra and someone had the power to accidentally make it public.
I have no words for this level of gross negligence.
Maybe, just maybe, centralizing all trust and security for most of the worlds software development to a proprietary for-profit company with an abysmal security reputation was not the best plan.
Start protecting yourself from a potential MITM better, mark the key as revoked. Hopefully distributions & OpenSSH upstream can start shipping this by default.
```
# You might need to insert this in a slightly different place
cat >>/etc/ssh/ssh_config <<EOF
Host *
RevokedHostKeys /etc/ssh/ssh_revoked_hosts
EOF
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
> ... out of an abundance of caution, we replaced our RSA SSH host key used to secure Git operations for GitHub.com
Yeah, that's not an "abundance of caution." That's the bare minimum response at that point. What's the "not cautious approach?" Make the repo private and go on your merry way?
In an ideal world this could be avoided by using SSH certificates - an assertion by a trusted party that the key presented by an SSH server is legitimate. The signer of those certificates could be in an HSM, ensuring that the private keys can never be leaked, and the certificates could be sufficiently short-lived that there'd be no need to deal with revocation.
Unfortunately there's no way to make this easy. To associate a CA key with a specific domain you need to add something like:
@cert-authority github.com (key)
to known_hosts, and this (as far as I know) can't currently be managed in the same way that keys are generally added to known_hosts on first use when you ssh to something. So instead we're left with every github.com SSH frontend using the same private keys, and those can't be in an HSM because it simply wouldn't scale to the required load. Which means the private key exists in a form where this kind of thing can occur.
Why did I, a paying GitHub customer, discover this when trying to do "git pull" on a repo I own?
My immediate assumption wasn't that GitHub had changed keys. I thought that my computer was having some sort of problem. Only after searching around for a bit did I find out that yes, GitHub changed their private key.
Couldn't they have e-mailed all of their users and/or customers, to tell us what had happened?
Info to everyone using Windows and as confused as I was:
• When you run a git command and git tells you about the mismatching key, it will prompt you to enter `y` or `n` but not actually let you do so.
• You need to run `plink github.com` to get the same prompt. You can then enter `y` to accept it.
• You should only accept it if it says the new fingerprint is d5:2c:63:d9:bc:75:9d:de:b1:4e:36:28:9f:7a:9c:39. If it is any different, you may be targeted by an attacker.
• The new key is stored in the file `C:\Users\<your name>\.ssh\known_hosts`. The new key at the time of this writing is `github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=`.
• In the future, if the key ever changes again, you can look it up on GitHub’s SSH key fingerprints page [0].
$ ssh -o VerifyHostKeyDNS=yes github.com
The authenticity of host 'github.com (140.82.121.3)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
No matching host key fingerprint found in DNS.
the last command combines all keys in the subdirectory into the one properly-formatted binary file. So, you can add more keys into the subdirectory later (but you do have to remember to rerun it -- personally I saved it into a one-line script at ~/.ssh/revoked_host_keys.sh so I don't forget)
For anyone who removed the old key and accepted the new one without comparing it to the published list[1], you can review the fingerprint you accepted with this command:
ssh-keygen -l -F github.com
It works for IP addresses as well, which are also published[2].
So... uh... why was their private key in a form that could be stored in a Git repo? In the old days, we used to put private keys in hardware co-processors that would never reveal an un-encrypted private key. I guess we don't do that anymore.
Imagine the amount of ssh connections that github must handle every second, I don't think it's so easy to hold the private key in a HSM while maintaining the necessary performance and availability.
The authenticity of host 'github.com (140.82.121.3)' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
So, first thing of course, is that the fingerprint does not match the one in the blogpost which is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s
Second problem, is that ip 140.82.121.3 seems to be reported as HIGH RISK[0]
So basically, how should I proceed? I am not security expert but I would say I am not an illiterate on this, and I have no idea. I guess the majority of users would you accept the new key, but is this the right move? I would need to do it if I want to do some work, that is for sure
You’re looking at the fingerprint of the ECDSA key; only the RSA key was replaced (and only the new RSA key’s fingerprint is in the blogpost). Check https://docs.github.com/en/authentication/keeping-your-accou... for the full list: the key you’re seeing is listed, so you should be fine.
I'm still having this problem. Here is my known_hosts file:
github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=
I get this error each time I interact with Github:
The authenticity of host 'github.com (140.82.112.3)' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
When I type 'yes' I get this added to my host file:
楧桴扵挮浯ㄬ〴㠮⸲ㄱ⸲″捥獤ⵡ桳㉡渭獩灴㔲‶䅁䅁㉅橖䡚桎塌潎呙瑉浢穬䡤祁呎䅙䅁䥁浢穬䡤祁呎䅙䅁䉂䕂䭭䕓橎䕑穥浏歸䵚㝹灯杋䙷㥂歮㕴剙奲橍畎㕇㡎男杒㙧䱃扲㕯䅷呤礯瘶洰噋唰眲地㉚䉙⬯含潰正㵧
Warning: Permanently added the RSA host key for IP address '140.82.121.3' to the list of known hosts.
Hi gassius! You've successfully authenticated, but GitHub does not provide shell access.
Then, when trying a git pull, I got this:
Warning: the RSA host key for 'github.com' differs from the key for the IP address '140.82.121.4'
Offending key for IP in ~/.ssh/known_hosts:63
Matching host key in ~/.ssh/known_hosts:64
So basically, the Offending key is the one I added manually as per blogpost?
Ok, I am in Europe, and this seems like an issue of global distribution network or something, but this is not great AT ALL, either the blogpost information is not complete or something fishy is going on
UPDATE: The replies makes clear what I was seeing those errors and make sense. Thanks
EDIT: Formatting and Acknowledge of the situation per replies
Their instructions could stand to be more thorough. If you have hashed hostnames in your known_hosts, you don't just have to
ssh-keygen -R github.com
you also have to
ssh-keygen -R 140.82.121.3
which is the IP address for that host. Otherwise you'll get something like
Warning: the RSA host key for 'github.com' differs from the key for the IP address '140.82.121.3'
Offending key for IP in /home/tom/.ssh/known_hosts:77
Matching host key in /home/tom/.ssh/known_hosts:102
You're absolutely right. It's absurd to frame it this way.
Do they expect people to think "Wow, Github leaked a key, but even without knowing if anyone snagged it, they're still replacing it. Wow, they go above and beyond."
[+] [-] LeonM|3 years ago|reply
Github's customers trust GH/MS with their code, which, for most businesses, is a high value asset. It wouldn't surprise me if this all results in a massive lawsuit. Not just against GH as a company, but also to those involved (like the CISO). Also, how on earth was it never discovered during an audit that the SSH private key was plain text and accessible? How has GH ever been able to become ISO certified last year [0], when they didn't even place their keys in a HSM?
Obviously, as a paying customer I am quite angry with GH right now. So I might be overreacting when I write this, but IMO the responsible persons (not talking about the poor dev that pushed the key, but the managers, CISO, auditors, etc.) should be fined, and/or lose their license.
[0] https://github.blog/2022-05-16-github-achieves-iso-iec-27001...
[+] [-] nwallin|3 years ago|reply
Regarding the secrecy of the data you host at github, you should operate under the assumption that a sizeable number of github employees will have access to it. You should assume that it's all sitting unencrypted on several different disk platters replicated at several different geographically separated locations. Because it is.
One of the things that you give up when you host your private data on the cloud is controlling who and under what circumstances people can view or modify your private data. If the content of the data is enough to sink you/your company without remedy you should not store it on the cloud.
[+] [-] Shank|3 years ago|reply
ISO/IEC 27001:2013 doesn’t say you have to store private keys in HSMs? It just requires you to have a standards compliant ISMS that implements all Annex A controls and all of the clauses. Annex A and the clauses don’t specifically mandate this.
If you can convince an auditor that you have controls in-place that meet the standard for protecting cryptographic media you basically meet the standard. The controls can be a wide variety of options and don’t specifically mandate technical implementation details for many, many things.
You shouldn’t rely on ISO/IEC 27001:2013 as attestation for technical implementation details here. Just because your auditor would red flag you doesn’t mean all auditors would. The standard is only as effective as the weakest, cheapest auditor, and there are perverse incentives that make auditors financially incentivized to certify companies due to recurring revenue.
[+] [-] lxgr|3 years ago|reply
Not to diminish the problems of having a large entity like Github handle a private key like that, but if that was your level of paranoia, you probably should have used commit signatures all along and not relied on Github to do that job for you.
[+] [-] peterkelly|3 years ago|reply
Requiring all commits to be signed by trusted keys avoids the risks associated with someone tampering with a repository hosted on GitHub if they are able to get access to it, although it doesn't protect against code being leaked.
See here for details: https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work
[+] [-] Veliladon|3 years ago|reply
For a host key? Like I get that being able to impersonate Github isn't great as far as state level actors having the ability to do this but you do know the actual transport layer keys are ephemeral and aren't derived at all from the host key, right?
[+] [-] megous|3 years ago|reply
Number of people just not blindly using TOFU with github over ssh must be quite low.
Who here went here https://docs.github.com/en/authentication/keeping-your-accou... and added the keys manually to known_hosts before using github for the first time?
[+] [-] killerstorm|3 years ago|reply
Do you realize that the code just sits on GitHub servers even if it's private?
If you have any degree of paranoia, why do you put your code into GitHub?!?!
Like, if you work on code which
[+] [-] sgt|3 years ago|reply
ISO 27001 certification does not require you to put keys into an HSM. The standard requires you to have controls in place, be aware of your risks and to maintain a risk register. But in no way does the standard require HSM's.
The standard would even be OK with storing this on a floppy drive if the risks surrounding that were identified and mitigated (or accepted).
[+] [-] j16sdiz|3 years ago|reply
In fact, this is not a supported option in openssh.
[+] [-] tashian|3 years ago|reply
[+] [-] nonethewiser|3 years ago|reply
By no means do I want to see the dev get fined or blackballed from the industry.
But if there is any 1 person responsible, it’s the person who did it. The reason why the dev shouldn’t be fined/blackballed is because it’s not just 1 person’s fault. I mean, fining or booting his manager out of the industry? Really?
[+] [-] grumple|3 years ago|reply
1) Nation state level actors can probably insert or compromise high level staff, or multiple high level staff, at any given company, and perform MITM attacks fairly easily. And some could compel turning over your code or secrets more directly anyway. Not worth worrying about this scenario: nobody working on anything truly sensitive should be using any externally hosted (or even externally networked) repositories.
2) It is much more difficult for other actors to do a MITM attack, and if they did, they'd probably have access to your code more directly.
3) Your code actually isn't worth much to anybody else. Imagine someone launching a complete clone of HN or any other site. Who cares? Nobody. What makes your code valuable is that you have it, and that you have a network and relationship with your customers. If somebody stole my company's codebases, I'd feel sorry for them, that they are going to waste any time wading through these useless piles of shit. The only potential problems are if secrets or major vulnerabilities are exposed and provide a path for exploit (like ability to access servers, services, exposing potential ransomware attacks).
[+] [-] amrb|3 years ago|reply
[+] [-] nebulous1|3 years ago|reply
I think this suggests we need more information from github. For instance GH employees may not always have had live access to this key, this could have happened as part of an operation that gave temporary access to an employee only recently. Or it could have been stored plaintext on multiple employees' home computers since creation.
When was the leaked key created anyway?
[+] [-] belter|3 years ago|reply
[+] [-] peanut-walrus|3 years ago|reply
[+] [-] j45|3 years ago|reply
The cloud is always the convenience of someone else’s computer over some amount of security.
[+] [-] steponlego|3 years ago|reply
[+] [-] brightball|3 years ago|reply
[+] [-] djbusby|3 years ago|reply
[+] [-] yetanotherjosh|3 years ago|reply
[+] [-] lrvick|3 years ago|reply
It was trusted to clone code into the infrastructure of hundreds of thousands of organizations. People pin it everywhere.
With this key someone could infect the infrastructure of fintech companies and steal billions of dollars. I know this well because I run a security consulting company focusing mostly on that industry. Mainly this is possible because almost no companies check for git commit signing, and Github does not enforce it anyway, and I digress.
This key held enough power over value that some might have used violence to acquire it.
With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right?
Nope. They admit they just stuck it in a git repo in plain text where any engineer could have copied it to a compromised workstation, or been bribed for it, for who knows how many years. Worse, it was not even some separate employee only intranet git repo, but their own regular public production infra and someone had the power to accidentally make it public.
I have no words for this level of gross negligence.
Maybe, just maybe, centralizing all trust and security for most of the worlds software development to a proprietary for-profit company with an abysmal security reputation was not the best plan.
I will just leave this here: https://sfconservancy.org/blog/2022/jun/30/give-up-github-la...
[+] [-] paxys|3 years ago|reply
- When exactly was the key leaked? Yesterday? A month ago?
- How long was it visible for? And no, "briefly" doesn't cut it.
- Did any internet traffic reach the page while it was exposed? We know you log everything, so it is a yes or no answer.
If any of these answers were pretty, I imagine they would have explicitly included them in the post.
[+] [-] robbat2|3 years ago|reply
(sorry, the comments are mangling this, clean version at https://gist.github.com/robbat2/b456f09b7799f4dafe24115095b8...)
``` # You might need to insert this in a slightly different place cat >>/etc/ssh/ssh_config <<EOF Host * RevokedHostKeys /etc/ssh/ssh_revoked_hosts EOF
cat >>/etc/ssh/ssh_revoked_hosts <<EOF # https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-k... ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== EOF ```
[+] [-] fjni|3 years ago|reply
> ... out of an abundance of caution, we replaced our RSA SSH host key used to secure Git operations for GitHub.com
Yeah, that's not an "abundance of caution." That's the bare minimum response at that point. What's the "not cautious approach?" Make the repo private and go on your merry way?
[+] [-] nessex|3 years ago|reply
[1] https://github.blog/changelog/2022-01-18-githubs-ssh-host-ke...
[+] [-] mjg59|3 years ago|reply
Unfortunately there's no way to make this easy. To associate a CA key with a specific domain you need to add something like:
@cert-authority github.com (key)
to known_hosts, and this (as far as I know) can't currently be managed in the same way that keys are generally added to known_hosts on first use when you ssh to something. So instead we're left with every github.com SSH frontend using the same private keys, and those can't be in an HSM because it simply wouldn't scale to the required load. Which means the private key exists in a form where this kind of thing can occur.
[+] [-] reuven|3 years ago|reply
My immediate assumption wasn't that GitHub had changed keys. I thought that my computer was having some sort of problem. Only after searching around for a bit did I find out that yes, GitHub changed their private key.
Couldn't they have e-mailed all of their users and/or customers, to tell us what had happened?
[+] [-] Timwi|3 years ago|reply
• When you run a git command and git tells you about the mismatching key, it will prompt you to enter `y` or `n` but not actually let you do so.
• You need to run `plink github.com` to get the same prompt. You can then enter `y` to accept it.
• You should only accept it if it says the new fingerprint is d5:2c:63:d9:bc:75:9d:de:b1:4e:36:28:9f:7a:9c:39. If it is any different, you may be targeted by an attacker.
• The new key is stored in the file `C:\Users\<your name>\.ssh\known_hosts`. The new key at the time of this writing is `github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=`.
• In the future, if the key ever changes again, you can look it up on GitHub’s SSH key fingerprints page [0].
[0] https://docs.github.com/en/authentication/keeping-your-accou...
[+] [-] herewulf|3 years ago|reply
Seriously? How that happened is deeply concerning.
And why weren't the other keys exposed?
[+] [-] acatton|3 years ago|reply
What a security nightmare…
[+] [-] barryfam|3 years ago|reply
[+] [-] foresto|3 years ago|reply
[1] https://docs.github.com/en/authentication/keeping-your-accou...
[2] https://docs.github.com/en/authentication/keeping-your-accou...
[+] [-] retrocryptid|3 years ago|reply
[+] [-] 0x0|3 years ago|reply
[+] [-] gassius|3 years ago|reply
When I tried to pull from my repo I got the warning message, right
I removed the old keys with ssh-keygen -R github.com
Then, trying with `ssh -T [email protected]` I see this
The authenticity of host 'github.com (140.82.121.3)' can't be established. ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
So, first thing of course, is that the fingerprint does not match the one in the blogpost which is SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s
Second problem, is that ip 140.82.121.3 seems to be reported as HIGH RISK[0]
So basically, how should I proceed? I am not security expert but I would say I am not an illiterate on this, and I have no idea. I guess the majority of users would you accept the new key, but is this the right move? I would need to do it if I want to do some work, that is for sure
EDIT: Formatting
[0]https://www.ipqualityscore.com/free-ip-lookup-proxy-vpn-test...
[+] [-] nneonneo|3 years ago|reply
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] mdrain18|2 years ago|reply
I get this error each time I interact with Github: The authenticity of host 'github.com (140.82.112.3)' can't be established. ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
When I type 'yes' I get this added to my host file: 楧桴扵挮浯ㄬ〴㠮⸲ㄱ⸲″捥獤ⵡ桳㉡渭獩灴㔲‶䅁䅁㉅橖䡚桎塌潎呙瑉浢穬䡤祁呎䅙䅁䥁浢穬䡤祁呎䅙䅁䉂䕂䭭䕓橎䕑穥浏歸䵚㝹灯杋䙷㥂歮㕴剙奲橍畎㕇㡎男杒㙧䱃扲㕯䅷呤礯瘶洰噋唰眲地㉚䉙⬯含潰正㵧
[+] [-] gassius|3 years ago|reply
I manually added the new RSA SSH public key entry to my known_hosts file (like they say in the blogpost)
Then ran ssh -T [email protected] and got
Warning: Permanently added the RSA host key for IP address '140.82.121.3' to the list of known hosts. Hi gassius! You've successfully authenticated, but GitHub does not provide shell access.
Then, when trying a git pull, I got this:
Warning: the RSA host key for 'github.com' differs from the key for the IP address '140.82.121.4' Offending key for IP in ~/.ssh/known_hosts:63 Matching host key in ~/.ssh/known_hosts:64
So basically, the Offending key is the one I added manually as per blogpost?
Ok, I am in Europe, and this seems like an issue of global distribution network or something, but this is not great AT ALL, either the blogpost information is not complete or something fishy is going on
UPDATE: The replies makes clear what I was seeing those errors and make sense. Thanks
EDIT: Formatting and Acknowledge of the situation per replies
[+] [-] jnsaff2|3 years ago|reply
Not in the blog post but the blog post points to my link which is the official documentation.
[+] [-] DangitBobby|3 years ago|reply
1. https://docs.github.com/en/authentication/keeping-your-accou...
[+] [-] tome|3 years ago|reply
[+] [-] dskloet|3 years ago|reply
> GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
What the... That's not "an abundance of caution". That's the only possible course of action.
[+] [-] awill|3 years ago|reply
Do they expect people to think "Wow, Github leaked a key, but even without knowing if anyone snagged it, they're still replacing it. Wow, they go above and beyond."
It's so ridiculous.
[+] [-] irjustin|3 years ago|reply
Only _reasonable_ course of action. Possible is to do nothing =)
[+] [-] ars|3 years ago|reply
That's the "caution" part.
[+] [-] franky47|3 years ago|reply
https://docs.github.com/en/authentication/connecting-to-gith...
(it would actually be best to update it to the Ed25519 public key, as it's the recommended algorithm when generating a new SSH key).
[+] [-] pquerna|3 years ago|reply
https://lwn.net/Articles/637156/
I'm honestly not familiar with anyone actually using host-key rotation?