There are many solutions out there that attempt to solve the secrets distribution problem. Vault is the newest one.
None of them solve the fundamental problem of the secret zero: provisioning the first secret on an untrusted system the automated way is hard, and without that secret zero, there is no way to authenticate the new system.
With Vault, you still need to figure out how to push secret zero (here a client authentication token). If you can solve that, why not push the secret you need to push directly, and bypass Vault entirely?
It allows you to have a non-sensitive single factor within things like configuration management, and have the second factor come from a machine-local location (instance ID, MAC address, TSM module, etc.). The idea is that another team out of band sets the 2nd factor that developers and config management never get to see. The result is you have full automation without secret zero issues.
As an additional security parameter, you can bind the two factors together to a single CIDR block, so you can have an additional factor restricting access from that machine's IP.
The URL itself above goes into a lot more detail, but our beta users are successfully automated in an elastic environment this way.
We've had the same issues at CoinJar, and we used SSH agent forwarding to solve it. This is how it can work:
Every new instance of application server is provisioned automatically by a trusted server (which holds the key to a credentials server). During orchestration/deployment, the application server has a temporary permission to fetch the secrets it needed, through SSH agent forwarding. Once the deployment is done, the session will end and the application server can never read new secrets until the next deployment.
This way we don't really need a solution like Vault. SSH is mature enough to provide authentication, and encryption is easy once you've figured out how to distribute keys automatically.
Why not do it at the network level? In some infrastructures this is really easy. For example in AWS you can set up all your servers in a VPC. Then put Vault on a machine on a specific subnetwork, and firewall it off such that only the machines from your VPC may access it. You have now created a walled garden where anything inside your VPC has access to the secrets without holding an authentication token, while anything outside does not.
Personally, I think this is your only line of defense anyways. Someone with access to your private network will likely be able to do some harm, and if they get even a non-root user on your server, it's game over, since you are just one local exploit away from them gaining root. Remember, once they gain root on one box, it's just a matter of time before they replace /usr/bin/sudo and /usr/bin/login with a script that emails them your password and/or steals your ssh-agent socket.
Don't ever pass passwords on the command line. On linux, other users can see your command line (ps aux), and it usually ends up in a shell's history file.
Absolutely true! We support values coming from the command-line, stdin, and files: https://vaultproject.io/docs/commands/read-write.html So you should use what you feel is most secure given the data you're entering.
Honestly, I dislike the approach. I don't want to tell my mysql <hello, please allow the user foo to authenticate itself with the password bar> and my webapp <hello webapp, please use the user foo and the password bar for the mysql over there>.
I want to tell the system <look, I have a user foo, and the webapp over there needs access to the mysql over here with that user. Make it so.>
I don't need to know the password my application uses to access the database. I don't WANT that password. No human should ever see this password. And heck, if the system wants to, just swithc passwords every 60 minutes. I don't want to care.
When I want to run something on a server and I don't want it to end in the .history file, I always run vi and execute the commands inside it ( <esc>:!command ).
It always shows in the process list; but for short-lived commands the opportunity window is small for an attacker.
It's a tricky question, one that we've spent a lot of time thinking about at my company (Conjur -- shameless plug, one of our use cases is secrets management). Obviously doing this on a server that others can access/pwn is a bad idea, but that's a very rare use case. Most secrets are being added via the command line from an admin's single user laptop.
In the end, we've settled on what seems to be the same solution as Vault: support other sources (stdin, files, etc.), but allow the command line.
That said, I think you're right that we should look into adding a warning for the CLI tools and placing more emphasis on this fact in our documentation.
How many people these days share their linux machine with anyone else? If you're the only user, first point doesn't matter. History file is a concern, but can be mitigated e.g. setting HISTIGNORE
This is true on shared systems. But for example as I am setting up a brand new server, I might do this from there since I know I'm the only user. The second half of your complaint is solved (at least in bash) by adding a space character before the beginning of the command.
Edit: am I wrong? If so, please explain, I really am curious.
> "Vault is the sixth such tool we've built. We've also built Vagrant, Packer, Serf, Consul, and Terraform. "
Sigh.
This is the first time that I actually feel overwhelmed. That nagging feeling that I'm doing something in some way solely because of habits. And that someone somewhere has created an awesome tool that would have allowed me to go home two hours earlier.
The amount of new tools cropping in the last couple of years alone is the 'culprit'.
These are awesome times, of course. Still, a part of my brain misses the time where you could ftp crap away to a server (even if automated by a Capistrano tool or similar) and spend the rest of the time arguing about the merits of puppet vs chef.
Obligatory quote from that XKCD about automating things. Remember, most of these tools simply automate what you already do. If you don't spend much time performing certain things, you don't need the tool. The cargo cult of "I should use Packer to create my Vagrant boxes, so I can have dev/prod parity" only works to a point. Instead, look at where you spend lots of time, and then look for tools that automate that task.
Personally, I try to be pretty conservative with tools that I start relying on. I know by painful experience that some tools get abandoned, some change drastically, making it impossible to upgrade, and some just turn out to be very buggy and horrible once you start putting them through their paces. In lots of cases, it's better to be critical than permissive when it comes to your infrastructure (and that's what these tools are aimed at). In almost all cases, your edge is in knowing the tools you use, and knowing about other tools. It's not in knowing all and using everything under the sun.
Don't feel that way. Very few tools like this stand the test of time.
I avoid using new tools in production. I wait a few years first to see if they last - after all I'm going to have to support the system for years, I don't want the tools to be the weak spot.
We were already using consul as our service discovery engine and as a data store for dynamic configurations via consul template. We feel Vault will help us manage secrets on our infrastructure and inside containers.
Neat! I'll have to see how you ended up using Vault since your project uses a lot of the same tech my personal webrtc server project uses. I'm also using ansible, docker, consul, registrator.
I'm not terribly familiar with blackbox, but based on the blackbox Readme it seems that this critique from Vault's website [1] addresses why HashiCorp thinks Vault is a better solution.
> Chef, Puppet, etc. all solve this in a similar way: single-key encrypted storage. Chef has encrypted data bags, Puppet has encrypted Hiera, an so on. The encrypted data is always one secret (a password, a key, etc.) away from being decrypted, and this secret is generally not well protected since in an elastic environment, every server needs to somehow get this secret to decrypt the data. Additionally, access to the encrypted data isn't always logged, so if there is an intrusion, it isn't clear what data has been accessed and by who.
Any reason why AWS dynamic credentials use IAM users instead of STS and role assumption? That seems to be a somewhat more widely accepted way to do dynamic credentials / role proxying.
What's coming next from hashicorp? The one infrastructure piece for which good open-source solutions still lack is ACL with solid auditing and variety of permissions approval models.
I haven't finished reading the docs yet, but this part[1] caught my attention:
> When a backend is unmounted, all of its secrets are revoked and its data is deleted.
If a backend is similar to a filesystem as mentioned in the docs, that's quite a surprise. If a backend is like a filesystem, the secrets (files) should become unavailable when unmounted, but should become available again after remounting the backend (filesystem).
Are the secrets actually _deleted_ (ie, can't be recovered again), or am I misunderstanding?
I'm responding a little late here, but hopefully you'll still see this.
Yes, when you unmount a backend, all the data is deleted. Vault mostly behaves like a filesystem, but not in this scenario. The raw data is not preserved. If you want to preserve the raw data, you can remount (move the mount point).
We didn't finish it for 0.1, but we already built in secret enumeration into the core. We plan to expose this at some point. If you want to unmount and preserve your secrets, you can always enumerate them and back them up elsewhere (unencrypted). This isn't yet available.
Looks interesting. There is definitely a need for something secure to manage secrets easily & efficiently.
I'm not super familiar with Shamir's secret sharing. The post says that the master key is split up across a number of people. One thing that I am confused about is if Vault goes down (possibly in the middle of the night), does it require a quorum of humans to boot it back up?
The solution to this problem is to run Vault in high availability mode[1]. This will have a set of stand-bys that can take over if the leader goes down. The idea is to unseal all the stand bys, and if the leader goes down, the standbys take over, and the sealed one can be unsealed at some other future point.
Security-sensitive things have been an interesting UX challenge for us, but this compromise comes at an expense of UX for a much better security promise. As one of the other comments says around this comment: without this feature there would be no real security for the stored data.
Shamir's secret sharing is actually fairly simple.
You want to have a secret shared by n people, any m of whom can decode it. You create a polynomial of degree m, and give each of those people the value of said polynomial at a different point. The secret will be the value of the polynomial at yet another point.
From the value of the polynomial at any m points you can figure out the polynomial and the secret. But the value at any m-1 points does not let you predict its value anywhere else.
(Detail. You would make the polynomial be over a finite field instead of the real numbers. But the math is all the same.)
Haven't tried it yet, but most likely yes. But if you actually need vault and do spread the key to more than 2 people, you likely have more than one vault server running at a time.
To the best of my knowledge it comes from contracting/support and mostly their Vagrant Cloud (now Atlas) features that are business/enterprise only, which they charge money for. For example private hosting of Vagrant box images is a paid feature. The costs for that are per-user/per-month.
Neat! I've been working on something similar for about the past six months, although my approach is a little more minimalistic, but also based on shamir's secret sharing. I was planning to release a preview the next couple of days. If any one is interested https://github.com/duse-io/api#how-it-works
//edit My solution focuses more on the sharing aspect between people, but can just as well be used like Vault
This is really awesome! I've been passively looking for something like Vault for a while, can't wait to start integrating.
I just finished tossing together a Chef cookbook to get Vault installed/configured up to the init/unseal step. Here's the link in case its useful to anyone: https://github.com/hexedpackets/chef-vault
> Vault is one giant leap forward for practical security in a cloud environment.
My biggest security concern with a cloud (or VPS) provider is that they can very easily read any keys or unencrypted data from memory. Still a worry when renting a dedicated box, but it's a damn site easier when it's virtualised.
I don't see Vault helping me with this... but then I'm probably both paranoid and missing the point.
There must be something in the air regarding tools for managing secrets. First Keywiz, then Credstash, a little python utility for managing secrets using AWS KMS https://github.com/LuminalOSS/credstash
The policies look like they'd be a bear to manage since they are path- and not role-based. In order to implement least privilege with multiple actors you'd have to be really careful with your paths.
Example:
2 apps.
App 1 needs secrets A B and D. App 2 need secrets A B and C.
So we need to set up our paths in a way that App 1 can get A and B and C, but not D. App 2 needs C but not D. Now when you want to modify secret access from your apps you have to rethink how your paths are set up.
I haven't been to HashiCorp's website basically since they released Vagrant. I use vagrant all the time though.
A few reactions. First: amazing design. Awesome documentation. Second: wow they have an awesome little suite of high-impact devops tools. I spent all day learning about them.
The unified style of websites they have organized into "intro" and "documentation" is awesome and easy to really digest what's going on quickly. All of the tools seem powerful, well thought out, and complementary.
Vagrant's been awesome to me and I'm excited to start using some of their tools like Packer, Consul, and Terraform in production!
Slightly OT, but is there such a tool (to store secrets securely and accessibly) for personal use that runs well with Linux and can be easily moved around (portable).
[+] [-] jvehent|11 years ago|reply
With Vault, you still need to figure out how to push secret zero (here a client authentication token). If you can solve that, why not push the secret you need to push directly, and bypass Vault entirely?
[+] [-] mitchellh|11 years ago|reply
It allows you to have a non-sensitive single factor within things like configuration management, and have the second factor come from a machine-local location (instance ID, MAC address, TSM module, etc.). The idea is that another team out of band sets the 2nd factor that developers and config management never get to see. The result is you have full automation without secret zero issues.
As an additional security parameter, you can bind the two factors together to a single CIDR block, so you can have an additional factor restricting access from that machine's IP.
The URL itself above goes into a lot more detail, but our beta users are successfully automated in an elastic environment this way.
[+] [-] zhoutong|11 years ago|reply
Every new instance of application server is provisioned automatically by a trusted server (which holds the key to a credentials server). During orchestration/deployment, the application server has a temporary permission to fetch the secrets it needed, through SSH agent forwarding. Once the deployment is done, the session will end and the application server can never read new secrets until the next deployment.
This way we don't really need a solution like Vault. SSH is mature enough to provide authentication, and encryption is easy once you've figured out how to distribute keys automatically.
[+] [-] IgorPartola|11 years ago|reply
Personally, I think this is your only line of defense anyways. Someone with access to your private network will likely be able to do some harm, and if they get even a non-root user on your server, it's game over, since you are just one local exploit away from them gaining root. Remember, once they gain root on one box, it's just a matter of time before they replace /usr/bin/sudo and /usr/bin/login with a script that emails them your password and/or steals your ssh-agent socket.
[+] [-] perlgeek|11 years ago|reply
But, the first example made me flinch:
Don't ever pass passwords on the command line. On linux, other users can see your command line (ps aux), and it usually ends up in a shell's history file.[+] [-] mitchellh|11 years ago|reply
[+] [-] jolan|11 years ago|reply
jolan 4950 0.0 0.0 46164 8820 pts/12 S+ 16:06 0:00 mysql -uroot -px xxxxxxxx
Probably still a race condition and can be viewed before it is erased but at least it would be something.
[+] [-] Freaky|11 years ago|reply
On FreeBSD I like doing:
/boot/loader.conf: mac_seeotheruids_load="YES"
/etc/sysctl.conf: security.mac.seeotheruids.enabled="1"
Which restricts users to seeing only their own processes. You can make exceptions for a specific group or users with the same primary group.
I still don't put anything sensitive in a command line, of course.
[+] [-] odiroot|11 years ago|reply
[+] [-] tetha|11 years ago|reply
I want to tell the system <look, I have a user foo, and the webapp over there needs access to the mysql over here with that user. Make it so.>
I don't need to know the password my application uses to access the database. I don't WANT that password. No human should ever see this password. And heck, if the system wants to, just swithc passwords every 60 minutes. I don't want to care.
[+] [-] eb0la|11 years ago|reply
When I want to run something on a server and I don't want it to end in the .history file, I always run vi and execute the commands inside it ( <esc>:!command ).
It always shows in the process list; but for short-lived commands the opportunity window is small for an attacker.
[+] [-] dmgbrn|11 years ago|reply
It's a tricky question, one that we've spent a lot of time thinking about at my company (Conjur -- shameless plug, one of our use cases is secrets management). Obviously doing this on a server that others can access/pwn is a bad idea, but that's a very rare use case. Most secrets are being added via the command line from an admin's single user laptop.
In the end, we've settled on what seems to be the same solution as Vault: support other sources (stdin, files, etc.), but allow the command line.
That said, I think you're right that we should look into adding a warning for the CLI tools and placing more emphasis on this fact in our documentation.
[+] [-] ams6110|11 years ago|reply
[+] [-] IgorPartola|11 years ago|reply
Edit: am I wrong? If so, please explain, I really am curious.
[+] [-] outworlder|11 years ago|reply
Sigh.
This is the first time that I actually feel overwhelmed. That nagging feeling that I'm doing something in some way solely because of habits. And that someone somewhere has created an awesome tool that would have allowed me to go home two hours earlier.
The amount of new tools cropping in the last couple of years alone is the 'culprit'.
These are awesome times, of course. Still, a part of my brain misses the time where you could ftp crap away to a server (even if automated by a Capistrano tool or similar) and spend the rest of the time arguing about the merits of puppet vs chef.
[+] [-] IgorPartola|11 years ago|reply
Personally, I try to be pretty conservative with tools that I start relying on. I know by painful experience that some tools get abandoned, some change drastically, making it impossible to upgrade, and some just turn out to be very buggy and horrible once you start putting them through their paces. In lots of cases, it's better to be critical than permissive when it comes to your infrastructure (and that's what these tools are aimed at). In almost all cases, your edge is in knowing the tools you use, and knowing about other tools. It's not in knowing all and using everything under the sun.
[+] [-] ars|11 years ago|reply
I avoid using new tools in production. I wait a few years first to see if they last - after all I'm going to have to support the system for years, I don't want the tools to be the weak spot.
[+] [-] steveb|11 years ago|reply
https://github.com/CiscoCloud/microservices-infrastructure
We were already using consul as our service discovery engine and as a data store for dynamic configurations via consul template. We feel Vault will help us manage secrets on our infrastructure and inside containers.
[+] [-] jgrowl|11 years ago|reply
Excuse the mess, I'm still working on it: https://github.com/jgrowl/livehq
[+] [-] rmetzler|11 years ago|reply
[1] https://square.github.io/keywhiz/
[+] [-] hbbio|11 years ago|reply
[+] [-] dorfsmay|11 years ago|reply
https://github.com/StackExchange/blackbox
[+] [-] metavida|11 years ago|reply
> Chef, Puppet, etc. all solve this in a similar way: single-key encrypted storage. Chef has encrypted data bags, Puppet has encrypted Hiera, an so on. The encrypted data is always one secret (a password, a key, etc.) away from being decrypted, and this secret is generally not well protected since in an elastic environment, every server needs to somehow get this secret to decrypt the data. Additionally, access to the encrypted data isn't always logged, so if there is an intrusion, it isn't clear what data has been accessed and by who.
[1]: https://vaultproject.io/intro/vs/chef-puppet-etc.html
[+] [-] mentat|11 years ago|reply
[+] [-] setheron|11 years ago|reply
[+] [-] ukd1|11 years ago|reply
[+] [-] malandrew|11 years ago|reply
There are a bunch of good ideas in this RFC, https://tools.ietf.org/html/rfc2904
Something like a host-your-own Amazon IAM would be pretty dope. Add support for 2fa and especially u2f and you've got yourself a winner.
[+] [-] dserodio|11 years ago|reply
> When a backend is unmounted, all of its secrets are revoked and its data is deleted.
If a backend is similar to a filesystem as mentioned in the docs, that's quite a surprise. If a backend is like a filesystem, the secrets (files) should become unavailable when unmounted, but should become available again after remounting the backend (filesystem).
Are the secrets actually _deleted_ (ie, can't be recovered again), or am I misunderstanding?
[1] http://vaultproject.io/intro/getting-started/secret-backends...
[+] [-] mitchellh|11 years ago|reply
Yes, when you unmount a backend, all the data is deleted. Vault mostly behaves like a filesystem, but not in this scenario. The raw data is not preserved. If you want to preserve the raw data, you can remount (move the mount point).
We didn't finish it for 0.1, but we already built in secret enumeration into the core. We plan to expose this at some point. If you want to unmount and preserve your secrets, you can always enumerate them and back them up elsewhere (unencrypted). This isn't yet available.
[+] [-] carllerche|11 years ago|reply
I'm not super familiar with Shamir's secret sharing. The post says that the master key is split up across a number of people. One thing that I am confused about is if Vault goes down (possibly in the middle of the night), does it require a quorum of humans to boot it back up?
[+] [-] mitchellh|11 years ago|reply
The solution to this problem is to run Vault in high availability mode[1]. This will have a set of stand-bys that can take over if the leader goes down. The idea is to unseal all the stand bys, and if the leader goes down, the standbys take over, and the sealed one can be unsealed at some other future point.
Security-sensitive things have been an interesting UX challenge for us, but this compromise comes at an expense of UX for a much better security promise. As one of the other comments says around this comment: without this feature there would be no real security for the stored data.
[1]: http://vaultproject.io/docs/concepts/ha.html
[+] [-] btilly|11 years ago|reply
You want to have a secret shared by n people, any m of whom can decode it. You create a polynomial of degree m, and give each of those people the value of said polynomial at a different point. The secret will be the value of the polynomial at yet another point.
From the value of the polynomial at any m points you can figure out the polynomial and the secret. But the value at any m-1 points does not let you predict its value anywhere else.
(Detail. You would make the polynomial be over a finite field instead of the real numbers. But the math is all the same.)
[+] [-] limaoscarjuliet|11 years ago|reply
This is akin to disk encryption where you have to type password on boot, but then data is accessible later.
[+] [-] viraptor|11 years ago|reply
[+] [-] MCRed|11 years ago|reply
Love all the projects. Are they contracting out features to companies? Selling support contracts?
Very curious-- I think if people, especially individuals, can make a living from open source projects, that's great for everybody.
But do you have to be "famous" to do it?
[+] [-] druiid|11 years ago|reply
[+] [-] ankushio|11 years ago|reply
[+] [-] amenghra|11 years ago|reply
[+] [-] flowerpot|11 years ago|reply
//edit My solution focuses more on the sharing aspect between people, but can just as well be used like Vault
[+] [-] hexedpackets|11 years ago|reply
I just finished tossing together a Chef cookbook to get Vault installed/configured up to the init/unseal step. Here's the link in case its useful to anyone: https://github.com/hexedpackets/chef-vault
[+] [-] BuildTheRobots|11 years ago|reply
My biggest security concern with a cloud (or VPS) provider is that they can very easily read any keys or unencrypted data from memory. Still a worry when renting a dedicated box, but it's a damn site easier when it's virtualised.
I don't see Vault helping me with this... but then I'm probably both paranoid and missing the point.
[+] [-] AndrewWright|11 years ago|reply
[+] [-] dustinrcollins|11 years ago|reply
Example: 2 apps. App 1 needs secrets A B and D. App 2 need secrets A B and C.
So we need to set up our paths in a way that App 1 can get A and B and C, but not D. App 2 needs C but not D. Now when you want to modify secret access from your apps you have to rethink how your paths are set up.
When you're instead assigning permissions to roles this is a lot easier. An example: http://blog.conjur.net/what-is-a-devops-secrets-server
[+] [-] LukeHoersten|11 years ago|reply
A few reactions. First: amazing design. Awesome documentation. Second: wow they have an awesome little suite of high-impact devops tools. I spent all day learning about them.
The unified style of websites they have organized into "intro" and "documentation" is awesome and easy to really digest what's going on quickly. All of the tools seem powerful, well thought out, and complementary.
Vagrant's been awesome to me and I'm excited to start using some of their tools like Packer, Consul, and Terraform in production!
[+] [-] christop|11 years ago|reply
The security page seems to be a generic one about reporting vulnerabilities: http://vaultproject.io/security.html
Edit: Found it under the threat model section here: https://vaultproject.io/docs/internals/security.html
[+] [-] weitzj|11 years ago|reply
[+] [-] andrewmwatson|11 years ago|reply
[+] [-] saintfiends|11 years ago|reply
[+] [-] vruiz|11 years ago|reply
[+] [-] notum|11 years ago|reply