What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking.
But secrets are next to useless if they are:
- not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys)
- not regularly rotated (at the very least when a person that had access to them leaves the company)
And rotation is hard: a lot of systems still don't support multiple keys, so rotation has to be very carefully tied to some form of blue-green deployment, which is often not possible.
So much practice around secrets and identity is oriented around assuming secrets are expensive. Like, people will set up a database and create ‘the database user’, with its one password which now needs to be used by all the systems that access that database.
To maximize security you need to treat secrets as being cheap. In fact they need to be disposable.
In theory any time a system needs access to that database you could have a process that creates an entirely new user just for that session, with a random password - and, bonus, only the permissions that that particular client needs.
If that secret gets compromised, no biggie - you can destroy that user account and the credential will never work again.
Secret rotation is a related approach, but again - if your rotation frequency is ‘quarterly’ rather than ‘hourly’, is there a good reason for that, or have you again just assumed secrets are expensive?
>And rotation is hard: a lot of systems still don't support multiple keys, so rotation has to be very carefully tied to some form of blue-green deployment
Oh just today I found out the previous team stored all secrets in git in multiple repositories, and now I need to rotate them all. They encrypted them eventually with sops but did not rotate them so you can find everything in git history. The only solution I can come up with so far is create new credentials and gradually migrate everyone to new secrets, and then delete the old ones. And there's still a chance we can forget something and something will break. I wish there was a button "rotate secrets"...
I find it frustrating that people still write programs without thinking about secrets rotation. It's kind of understandable that older stuff didn't think about it, since there was certainly a time when people treated secrets, especially things like API keys, as effectively indefinite, but that just doesn't make sense, especially not now. Same for any ephemeral keymatter, or other cryptographic keymatter: there should be a way to have multiple, with at least one "primary" secret and at least one secondary secret. The way I often do this is as simple as having an environment variable/flag/whatever for two separate values and try them in order. Rotating secrets used to salt or encrypt database entries is definitely a bigger pain in the ass, but if it's a big pain in the ass to design a system for it when you're building out a feature, you'd better bet it will be a pain in the ass when you've got a huge security breach and are faced with the task to do it not only in short order, but in concert with many other tasks related to the ongoing fire.
There are definitely other things to be said about the state of secret tokens and secret management, but this one is honestly not that hard to do significantly better at. Rotating secrets should not involve downtime, it should be something you can do out of precaution...
You're exactly right about those two problems, and they actually go together quite well. If a user who previously had access to a secret suddenly has that access revoked, you should rotate that secret. Technically that's only really necessary if the user ever actually saw that secret value, but most secret managers don't expose that information.
At Doppler, we're currently in the process of building out our rotation engine[0]. It allows you to automatically rotate secrets with third party providers (including your database) on a schedule you define. We also provide access logs to see exactly who saw what and when. The ultimate goal is to automatically rotate your secrets as users' access changes. And if you get notification of a breach, you can rotate everything with a single click.
The last part is the key thing. I'm in a continuous running battle with our information security risk team over this - they believe that secret rotation is a secrets management platform feature (and we use Vault so that box is ticked) but the reality is that secret rotation that isn't tied to restarts / reloads of said secrets within distributed application estates is just another way to have an incident.
This is particularly true if you've backed into secrets management as a practice and retrofitted existing platforms; but even if you've designed net new, it's an abstraction-breaker to get reloads of credentials happening all the way down the stack.
Even a blue/green deployment isn't a panacea if there're databases or other shared states that depend on the credential cutover.
From my point of view, the better approach is not to share secrets at all where possible and drive everything from service identity directly. e.g. short-lived credentials like AWS IAM or short-lived PKI proofs like SPIFFE.
The "better way" for me has been managed identities in Azure. These are Azure Active Directory service principals, which are similar to Group-Managed Service Accounts (gMSA) in Active Directory. They're essentially a platform-managed identity with automatically rotating secrets that the developers never need to know, and sysops don't have to deal with. They can be assigned to VMs or various PaaS services, and then service-to-service access is via standard Azure RBAC, with auditing and everything.
There is nothing to rotate, nothing to put into a "key vault", nothing to worry about leaking out, nothing that could accidentally get pasted into Git, and nothing on that developers' laptop that he left at the airport. You literally say: "I want service A to have read access to service B", and you're done.
Visual Studio automatically injects the developer's personal identity into processes, so that they can access cloud resources during local development, so that's covered too. (For Linux developers, the Azure Identity SDK can pick up the user credentials via the "az" cli.)
"Making fire with stones to avoid a proprietary lighter." -- /user?id=pjmlp
I'm having the same success with Google IAM, and I agree it completely solves the problem, except for one thing; It only works with compatible services, within your company boundary, within the same cloud provider whereas API keys or oauth flows work everywhere.
The fact that we don't have a well adopted standard for password management, critically including password rotation and key sharing, by now is both surprising and infuriating.
Are there any widely available, published threat models for secret management of this sort? It seems to me that if you want to frame the discussion in a useful way, you'd start there. Then you can point out where current practice works, where it doesn't, and how your solution works better.
HashiCorp vault is a pretty good answer. It has plugins to auto rotate a number of different connections, works with both static and dynamicly created users, whole host of auditing optinons.
hbrn|3 years ago
But secrets are next to useless if they are:
- not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys)
- not regularly rotated (at the very least when a person that had access to them leaves the company)
And rotation is hard: a lot of systems still don't support multiple keys, so rotation has to be very carefully tied to some form of blue-green deployment, which is often not possible.
jameshart|3 years ago
To maximize security you need to treat secrets as being cheap. In fact they need to be disposable.
In theory any time a system needs access to that database you could have a process that creates an entirely new user just for that session, with a random password - and, bonus, only the permissions that that particular client needs.
If that secret gets compromised, no biggie - you can destroy that user account and the credential will never work again.
Secret rotation is a related approach, but again - if your rotation frequency is ‘quarterly’ rather than ‘hourly’, is there a good reason for that, or have you again just assumed secrets are expensive?
kgeist|3 years ago
Oh just today I found out the previous team stored all secrets in git in multiple repositories, and now I need to rotate them all. They encrypted them eventually with sops but did not rotate them so you can find everything in git history. The only solution I can come up with so far is create new credentials and gradually migrate everyone to new secrets, and then delete the old ones. And there's still a chance we can forget something and something will break. I wish there was a button "rotate secrets"...
jchw|3 years ago
There are definitely other things to be said about the state of secret tokens and secret management, but this one is honestly not that hard to do significantly better at. Rotating secrets should not involve downtime, it should be something you can do out of precaution...
tompic823|3 years ago
At Doppler, we're currently in the process of building out our rotation engine[0]. It allows you to automatically rotate secrets with third party providers (including your database) on a schedule you define. We also provide access logs to see exactly who saw what and when. The ultimate goal is to automatically rotate your secrets as users' access changes. And if you get notification of a breach, you can rotate everything with a single click.
[0] https://docs.doppler.com/docs/secrets-rotation
NovemberWhiskey|3 years ago
This is particularly true if you've backed into secrets management as a practice and retrofitted existing platforms; but even if you've designed net new, it's an abstraction-breaker to get reloads of credentials happening all the way down the stack.
Even a blue/green deployment isn't a panacea if there're databases or other shared states that depend on the credential cutover.
From my point of view, the better approach is not to share secrets at all where possible and drive everything from service identity directly. e.g. short-lived credentials like AWS IAM or short-lived PKI proofs like SPIFFE.
jiggawatts|3 years ago
There is nothing to rotate, nothing to put into a "key vault", nothing to worry about leaking out, nothing that could accidentally get pasted into Git, and nothing on that developers' laptop that he left at the airport. You literally say: "I want service A to have read access to service B", and you're done.
Visual Studio automatically injects the developer's personal identity into processes, so that they can access cloud resources during local development, so that's covered too. (For Linux developers, the Azure Identity SDK can pick up the user credentials via the "az" cli.)
"Making fire with stones to avoid a proprietary lighter." -- /user?id=pjmlp
fvdessen|3 years ago
nixpulvis|3 years ago
javajosh|3 years ago
msolberg|3 years ago
yodon|3 years ago