First reaction: Holy crap! They finally turned ParameterStore into a proper product!
Second reaction: Holy shit that's expensive [for what it does].
ParameterStore is free (minus the KMS component). The only value-add is secret rotation and that's not something that most of the time makes sense to use. [Edit: I'm not advocating for no rotation; see replies]
Edit: Had more time to think about it. Someone enlighten me: What's the difference between writing a rotation lambda for this new product, vs. writing a rotation lambda for ParameterStore that you then cron? The pricing really doesn't make sense.
My biggest criticism of Parameter Store is actually that it's free. Let me explain:
Because it's free they limit the requests per seconds you can make to the Parameter Store. That's especially noticeable when doing requests for all parameters of a given path, as the limit is way lower there than for requesting (a bunch of) individual parameters. In the past that caused serious problems for us when using Parameter Store for AWS Lambda functions during a deploy of new versions of functions, as suddenly there was a spike in the number of requests to Parameter Store as all AWS Lambda containers got replaced.
They of course set such limits because it's free, so I'd gladly pay for getting increased limits.
> The only value-add is secret rotation and that's not something that most of the time makes sense to use.
From a security perspective, you should be rotating secrets somewhere between annually and every 90 days, depending on your business/security/compliance requirements and the nature of the data secured by the secret.
The pricing is a bit surprising, but I am sure that if you tried to build the full solution you would end up needing a few more pieces than just the rotation lambda and cron.
Would have been nice if there was a 5 secret free tier though. I feel like the sticker shock will dissuade small users when everybody really should be doing secret rotation.
After Amazon EC2 Systems Manager Parameter Store got some much needed love last year (adding KMS encryption and versioning), I wrote a small utility for populating environment variables using a specific Parameter Store prefix: https://github.com/nlindblad/confidential
It can either write the environment variables out to a Docker compatible environment file or it can be used as a wrapper to run any executable/script with the environment pre-populated. The README contains various examples of those use cases.
The primary motivation was to have proper secrets management for my hobby projects that didn't rely on Ansible Vault, configuration files in S3 or having to use the AWS SDK in each project.
Wrt rotation theres also a cultural component of it. Systems grow in complexity and importance. Forcing periodic rotation, With corresponding automagic client support, ensures thats its even possible when you need it. Compare STS tokens to IAM credentials.
This looks like an AWS equivalent of the Amazon-internal secret management tool called Odin. Which is very nice because Odin was pretty much universally loved from what I saw.
and boom your shell script or whatever was very easily using secrets.
The fact that Amazon systems bootstrapped EC2 instances meant they could easily enable this org-wide. The odin back-end then owned all the lifecycles around those secrets including what kinds of hosts they would go to, if/how they would rotate, etc etc. It has its annoyances including the fact you have to use http to localhost which can get saturated if you're not doing things right, but overall it really made secrets-management a non-issue.
After leaving Amazon you really go on to understand how good the internal tools are at Amazon and how they probably have the best tools in the industry.
Odin works well, it's just a hassle to share credentials with non-developers who don't have a dev desktop and therefore can't access Odin. It's an issue I've encountered several times recently with no clear solution, it seems.
Odin works really well but but only because it integrates so tightly with with the rest of the internal tooling. Without Apollo (another internal tool) to manage and orchestrate the deployment of packages that need Odin's secrets, it's a lot clumsier to use.
For those of us living in the service-development stone ages, is the idea that a secret-manager service replaces any number of ad-hoc local secret-storage and configuration mechanisms with a single robust mechanism that takes only a single root credential to retrieve all the individual secrets that your service needs?
You do still have to figure out a way to securely provide the root credential to your service so that it can fetch the secrets from the secret manager, correct? Otherwise this would be magic of a kind I think is impossible.
If my questions aren't too far off in the weeds, then this service sounds like a personal password manager but for a service rather than a person, though I'm sure AWS's service has finer-grained controls than just the all-or-nothing master passphrase. Similar risks apply: an attacker obtaining the master passphrase is a major issue, losing the master passphrase is devastating (though recoverable here because you probably didn't lose your personal AWS login credentials), and unavailability of the password database is catastrophic. But the usability benefits of having everything in one secure place, behind a service managed by experts, should outweigh those risks.
I have more questions about the credential-rotation feature, but this is enough for now.
The big missing piece is roles. No service uses a root access key directly. Instead, there's a webserver role with access to a relevant secrets group but no access to data warehouse secrets, for example.
Access keys can be provisioned and downloaded straight onto the box from the service. Sure, a compromise is bad, but only exposes the secrets that would be available on the pwned box regardless.
It's a non issue to grant access to a particular group of secrets to a particular server role, assuming the requesting server has an IAM instance profile attached to it. The IAM Policy just describes what secrets the role can access, and then the server simply makes the api call to get the secret.
Having your personal AWS keys compromised should also be a non-issue because of MFA. My keys that I keep in OSX Keychain grant zero permissions, except the ability to assume a more privileged role, but that requires MFA.
FWIW, we're using Parameter Store for secrets and it works great. Given that I just finished that set up just weeks ago, I'm in no rush to jump on the Secrets Manager wagon based on what I'm seeing.
The only problem with both services is the 4k character limit. We have some big cert chains and such that exceed that, so we had to build the tools such that they could put/get a chunked secret. Wasn't too bad though.
At $0.40 per secret per month and $0.05 per 10,000 requests this is much more expensive than the practically free SSM Parameter Store product, even if you factor in the auto-rotating bits.
It'd be cheaper than running Vault with a backing Consul cluster which also provides rotation and other features.
There is a point where Vault is more cost effective, but I believe it'd require a ton of requests and secrets to justify min 6 machines of at least t2.micro that also need to managed and secured.
Agree the pricing is way off, but at least you can use an object/table inside a secret to serve as a group of secrets. eg, not just key:"value", but key:{key:"value", key:"value", ...}
Besides the secrets rotation how is this different from EC2 Parameter Store? I’m genuinely curious and will move away from parameter store if this provides some benefits.
At my work, we have a vault service, but it takes 3 servers for vault plus another 3 for consul and they are not very cheap servers (m4.larges). Our rough calculations, using a 3x3 vault/consul architecture, estimated you'd need over 1100-1300 secrets to make it worth implementing vault (not including development/maintenance cost which could be significant. and if you have multiple, independent environments, it gets worse.
The real issue for some of the people at my work is the vendor lock-in versus vault.
FTFU - Valut service.... You need more than one server, it'll need to have 100% uptime too, backed up and available in multiple regions. No security team want to own this infrastructure.
I'm still trying to figure out how different this is from their "parameter store" offering in AWS Systems Manager. The main thing I guess is you get more control over key rotation.
I am curious as to how it handles those race conditions where a connection is made with the older credentials just after the time the rds master key rotates, or a connection is made with the newer credentials just before the time the rds client key rotates. Short of using two credentials accounts ...
As others have noted, the pricing is really the odd thing here. AWS seems to be moving to value based pricing rather than cost based pricing for some of its niche products.
I used to think that Cloudwatch metrics were very expensive at $.50 per custom metric per month, but this seems waaayyyy cheaper to store.
How is this different than KMS? “Key Management Service” is practically synonymous with the name of this new product, so how exactly do the two differ/interact?
A key difference is in kms you never know the actual secret (kms stores this on your behalf) . This is for use cases where you need the secret in application code, like db creds.
Looks like a cool product I noticed they dont include a revoke workflow though, the trickiest bit of key rotation
Yeah I'm thinking about starting to use it for personal projects but don't want to bother with these 2 dollar bills. I'm pretty sure they can benefit from me playing with it first and then advocating for it at work
Oh, I misread it as a secret for Twitter (which, I guess, makes no sense, because tweets are generally public).. and I was wondering how the password got updated on the Twitter side. So I guess, naturally, secrets can only be rotated for AWS services that support it.
There is a lot of commentary about the use of vault as an alternative, number of secrets needed, etc. I think the inclusion of Secrets Manager is a great addition for AWS and will definitely help people get better control over their secrets, however, vault contains richer functionality than just secrets key/value storage.
It can provision users to backends like SSH, databases, cloud providers, and such. Use is audited, can be revoked, and has a TTL associated.
Additionally, vault contains a full "crypto-in-a-box" implementation that allows for sign/verify, hmac/verify, encrypt/decrypt, random number generation, and other functions.
So I applaud AWS for doing this and hope the will continue developing KMS/HSM/Parameter Store/Secret Store/??? in the future and innovating, but evaluating Secret Store vs. Vault simply on price may be a short sighted comparison.
Disclaimers: Employer is an AWS customer using vault
I was hoping to use this for RDS parameters (to get automatic rotation) and leave everything else in ParameterStore.
Unfortunately — although I'm sure it's built on-top of ParameterStore internally — I just checked and you can't see SecretsManager secrets in ParameterStore, so an application would need to read from both and merge them, or switch entirely to SecretsManager to take advantage of the automatic rotation.
[+] [-] scrollaway|8 years ago|reply
Second reaction: Holy shit that's expensive [for what it does].
ParameterStore is free (minus the KMS component). The only value-add is secret rotation and that's not something that most of the time makes sense to use. [Edit: I'm not advocating for no rotation; see replies]
Edit: Had more time to think about it. Someone enlighten me: What's the difference between writing a rotation lambda for this new product, vs. writing a rotation lambda for ParameterStore that you then cron? The pricing really doesn't make sense.
[+] [-] Dunedan|8 years ago|reply
Because it's free they limit the requests per seconds you can make to the Parameter Store. That's especially noticeable when doing requests for all parameters of a given path, as the limit is way lower there than for requesting (a bunch of) individual parameters. In the past that caused serious problems for us when using Parameter Store for AWS Lambda functions during a deploy of new versions of functions, as suddenly there was a spike in the number of requests to Parameter Store as all AWS Lambda containers got replaced.
They of course set such limits because it's free, so I'd gladly pay for getting increased limits.
[+] [-] toomuchtodo|8 years ago|reply
From a security perspective, you should be rotating secrets somewhere between annually and every 90 days, depending on your business/security/compliance requirements and the nature of the data secured by the secret.
[+] [-] talawahdotnet|8 years ago|reply
Would have been nice if there was a 5 secret free tier though. I feel like the sticker shock will dissuade small users when everybody really should be doing secret rotation.
[+] [-] niklaslindblad|8 years ago|reply
It can either write the environment variables out to a Docker compatible environment file or it can be used as a wrapper to run any executable/script with the environment pre-populated. The README contains various examples of those use cases.
The primary motivation was to have proper secrets management for my hobby projects that didn't rely on Ansible Vault, configuration files in S3 or having to use the AWS SDK in each project.
[+] [-] donavanm|8 years ago|reply
[+] [-] Florin_Andrei|8 years ago|reply
[+] [-] staticassertion|8 years ago|reply
[+] [-] jcolella|8 years ago|reply
[+] [-] januszeal|8 years ago|reply
This attitude is why company after company keeps leaking customer data.
[+] [-] whateveracct|8 years ago|reply
[+] [-] ryanianian|8 years ago|reply
The killer-app of odin imho was/is the on-machine http server that let all manner of applications very easily get credentials.
You could just do something like
and boom your shell script or whatever was very easily using secrets.The fact that Amazon systems bootstrapped EC2 instances meant they could easily enable this org-wide. The odin back-end then owned all the lifecycles around those secrets including what kinds of hosts they would go to, if/how they would rotate, etc etc. It has its annoyances including the fact you have to use http to localhost which can get saturated if you're not doing things right, but overall it really made secrets-management a non-issue.
[+] [-] ronnier|8 years ago|reply
[+] [-] valar_m|8 years ago|reply
[+] [-] qxi|8 years ago|reply
[+] [-] jcims|8 years ago|reply
[+] [-] josefdlange|8 years ago|reply
[+] [-] sowbug|8 years ago|reply
You do still have to figure out a way to securely provide the root credential to your service so that it can fetch the secrets from the secret manager, correct? Otherwise this would be magic of a kind I think is impossible.
If my questions aren't too far off in the weeds, then this service sounds like a personal password manager but for a service rather than a person, though I'm sure AWS's service has finer-grained controls than just the all-or-nothing master passphrase. Similar risks apply: an attacker obtaining the master passphrase is a major issue, losing the master passphrase is devastating (though recoverable here because you probably didn't lose your personal AWS login credentials), and unavailability of the password database is catastrophic. But the usability benefits of having everything in one secure place, behind a service managed by experts, should outweigh those risks.
I have more questions about the credential-rotation feature, but this is enough for now.
[+] [-] lmartel|8 years ago|reply
Access keys can be provisioned and downloaded straight onto the box from the service. Sure, a compromise is bad, but only exposes the secrets that would be available on the pwned box regardless.
[+] [-] jeremykoerber|8 years ago|reply
Having your personal AWS keys compromised should also be a non-issue because of MFA. My keys that I keep in OSX Keychain grant zero permissions, except the ability to assume a more privileged role, but that requires MFA.
FWIW, we're using Parameter Store for secrets and it works great. Given that I just finished that set up just weeks ago, I'm in no rush to jump on the Secrets Manager wagon based on what I'm seeing.
The only problem with both services is the 4k character limit. We have some big cert chains and such that exceed that, so we had to build the tools such that they could put/get a chunked secret. Wasn't too bad though.
[+] [-] kondro|8 years ago|reply
[+] [-] 0xCMP|8 years ago|reply
There is a point where Vault is more cost effective, but I believe it'd require a ton of requests and secrets to justify min 6 machines of at least t2.micro that also need to managed and secured.
[+] [-] o_____________o|8 years ago|reply
[+] [-] bpicolo|8 years ago|reply
[+] [-] ranman|8 years ago|reply
[+] [-] epberry|8 years ago|reply
[+] [-] 013a|8 years ago|reply
WOWZER. I get having a managed solution is great, but you don't have to store many secrets before running your own Vault server makes sense.
[+] [-] crazypyro|8 years ago|reply
The real issue for some of the people at my work is the vendor lock-in versus vault.
[+] [-] ctvo|8 years ago|reply
[+] [-] seanieb|8 years ago|reply
[+] [-] bithavoc|8 years ago|reply
[+] [-] ryanSrich|8 years ago|reply
[+] [-] weej|8 years ago|reply
[+] [-] marvinpinto|8 years ago|reply
[+] [-] Thaxll|8 years ago|reply
[+] [-] netvisao|8 years ago|reply
[+] [-] jtwaleson|8 years ago|reply
I used to think that Cloudwatch metrics were very expensive at $.50 per custom metric per month, but this seems waaayyyy cheaper to store.
[+] [-] chatmasta|8 years ago|reply
[+] [-] cavisne|8 years ago|reply
Looks like a cool product I noticed they dont include a revoke workflow though, the trickiest bit of key rotation
[+] [-] talawahdotnet|8 years ago|reply
That would get smaller users to start using it instead of parameter store and eventually realize the value of automated and audited secret rotation
[+] [-] kungito|8 years ago|reply
[+] [-] pageandrew|8 years ago|reply
If so, thats pretty damn cool.
[+] [-] rconti|8 years ago|reply
[+] [-] ranman|8 years ago|reply
[+] [-] ppierald|8 years ago|reply
It can provision users to backends like SSH, databases, cloud providers, and such. Use is audited, can be revoked, and has a TTL associated.
Additionally, vault contains a full "crypto-in-a-box" implementation that allows for sign/verify, hmac/verify, encrypt/decrypt, random number generation, and other functions.
So I applaud AWS for doing this and hope the will continue developing KMS/HSM/Parameter Store/Secret Store/??? in the future and innovating, but evaluating Secret Store vs. Vault simply on price may be a short sighted comparison.
Disclaimers: Employer is an AWS customer using vault
[+] [-] bestbear|8 years ago|reply
[+] [-] spullara|8 years ago|reply
[+] [-] drodgers|8 years ago|reply
Unfortunately — although I'm sure it's built on-top of ParameterStore internally — I just checked and you can't see SecretsManager secrets in ParameterStore, so an application would need to read from both and merge them, or switch entirely to SecretsManager to take advantage of the automatic rotation.
[+] [-] 0xCMP|8 years ago|reply
[+] [-] outworlder|8 years ago|reply
But not with this price per secret...