[lstoll]

We use an AWS KMS asymmetric key for the CA keys, they're cheap and avoids exposing the private key material in an any way.

For signing SSH certificates, we run a small service (prototype code dump at https://github.com/pardot/sshsigner) that uses this key to sign short lived certificates. Auth to the service is via OIDC issued ID tokens.

On the client side we have a custom SSH agent that uses an ephemeral in-memory private key. The agent manages the OIDC web flow and calling out to the service for signing on demand. This lets us keep the cert duration small and scoped, and allows us to force re-auth for sudo etc. via the web flow.

We also do a similar thing for host keys, IAM auth the instances and sign certificates.

Altogether works well, provides a nice user experience, and keeps long-lived/leakable creds out of out environment.