[tjk_]

Bitbucket dev here. Firstly, thanks for the post, this topic is too often skimmed and as someone who worked directly on the host key rotation here, it was shocking to find how many devs had zero idea what "that prompt from git on a new computer" was.

Secondly, I just wanted to say that there is an interesting side-effect to everything covered in that blog that effectively leads to a gap in most people's mental models (or maybe just mine). Because of the way these systems were defined/the way people interact with them, there are several conundrums when a large centralized host needs to deprecate an old key.

If a key were suspected of being compromised, it seems obvious that you could just stop serving the old key and tell all of your users to be prepared to re-TOFU. However, since _most_ devs blindly type "yes" when interacting with a large git host, this effectively primes the pumps for any bad actor that have MITM control but have not actually stolen the keys. This gives them a relatively long (if not infinitely long) window of time where a user will not be surprised to see "the prompt" and blindly accept trust of a host that could be controlled by the bad actor. If a key was completely successfully compromised by a bad actor, and said bad actor had MITM control of the victim network, then requests with the _old_ key would never actually reach the correct host and just quietly continue working (assuming the bad actor was savy enough to setup a remote system to properly behave as a git host without a prior copy of the target repository).

Damned if you do, damned if you don't, so always carefully TOFU.

Also,not sure why the author linked off to a Bitbucket Cloud blog post for the SSH keys, they are documented here[1] along with our recommended best practices WRT TOFU.

1 - https://support.atlassian.com/bitbucket-cloud/docs/configure...

[eichin]

We actually ship a bunch of vetted pubkeys in our developer tools package (in ssh_known_hosts2) . We tried to get a little bit out of the loop and tell our devs that they could use the instructions at https://bitbucket.org/blog/ssh-host-key-changes (confirm it doesn't cert fail, then read the curl https://bitbucket.org/site/ssh bit and see that it also checks certs, so we can leverage "trusted https cert" to "trusted ssh key"... much better than TOFU...

Didn't work, because there's no trailing newline on the output of site/ssh. So even if it works, it corrupts the next addition.

[js2]

Change from this:

  curl https://bitbucket.org/site/ssh >> ~/.ssh/known_hosts

To this:

  (curl https://bitbucket.org/site/ssh; echo) >> ~/.ssh/known_hosts

[tjk_]

Thanks for pointing this out, I'll put in a PR to fix that up today.

[ahoka]

Umm, why don’t you just use git over HTTPS?

[c7DJTLrn]

Thanks. I linked to the blog post because it was one of the first results in the search engine.

How are you guys tackling host key rotation? Do you do it periodically, or on compromise? How do you protect such an important set of keys?

[MaxBarraclough]

I don't work for them, but: BitBucket changed their SSH keys earlier this year due to possible compromise of their secret keys. [0] I don't think they change them periodically.

[0] https://bitbucket.org/blog/ssh-host-key-changes

edit I see eichin also linked to the same page

[tedunangst]

sshd also supports online key rotation where it tells clients about new keys, but I think it requires the new key be installed on the server, so it ends up getting compromised at the same time as the existing key.