[praash]

I stopped worrying after I began protecting all keys with a passphrase.

[doug713705]

Then the access of your git repos is protected by a single factor, the private key, since the private key is already in the wild.

Copying a private key on a removable storage or to another device than the device that generated it is never a good idea.

[zikduruqe]

I protect mine with GPG for SSH authentication.

[Ferret7446]

The only use of the passphrase is to give you time to rotate out the key after it's been compromised. It's not meant to be your main line of defense