Why does the title say "Zero Trust", when the article explains that this only works as long as every involved component of the Cloudflare MitM keylogger and its CA can be trusted?
If hosts keys are worthless because you do not know in advance what key the proxy will have.. than this scheme is back to trusting servers merely because they are in Cloudflare address space, no?
Every zero trust architecture ends up trusting an unbounded set of machines. Like most marketing terms, it’s probably easier to assume it does the inverse of what it claims.
My mental model:
With 1 trust (the default) any trusted machine with credentials is provided access and therefore gets one unit of access. With 2-trust, we’d need at least units of trust, so two machines. Equivalently, each credential-bearing machine is half trusted (think ssh bastion hosts or 2FA / mobikeys for 2 trust).
This generalizes to 1/N, so for zero trust, we place 1/0 = infinite units of trust in every machine that has a credential. In other words, if we provision any one machine for access, we necessarily provision an unbounded number of other machines for the same level of access.
As snarky as this math is, I’ve yet to see a more accurate formulation of what zero trust architectures actually provide.
Zero Trust just means you stop inherently trusting your private network and verify every user/device/request regardless. If you opt in to using Cloudflare to do this then it requires running Cloudflare software.
> Why does the title say "Zero Trust", when the article explains that this only works as long as every involved component of the Cloudflare MitM keylogger and its CA can be trusted?
The truth-yness of "zero trust" really depends on who's trusting who.
I'm a fan of SSH certificates and cannot understand why anyone would set up certificate authentication with an external third-party CA. When I'm selling people on SSH CA's, the first thing I usually have to convince them of is that I'm not saying they should trust some third party. You know where all your servers are. External CAs exist to solve the counterparty introduction problem, which is a problem SSH servers do not have.
> I'm a fan of SSH certificates and cannot understand why anyone would set up certificate authentication with an external third-party CA.
I think the sales pitch for these sorts of service is: "Get an SSH-like experience, but it integrates with your corporate single-sign-on system, has activity logs that can't be deleted even if you're root on the target, sorts out your every-ephemeral-cloud-instance-has-a-different-fingerprint issues, and we'll sort out all the reverse-tunnelling-through-NAT and bastion-host-for-virtual-private-cloud stuff too"
Big businesses pursuing SOC2 compliance love this sort of thing.
I'm with you, I imagine it's mostly people just drawing parallels, they can figure out how to get a web certificate so think SSH is the same thing.
The second order problem I've found is when you dig in there are plenty of people who ask for certs but when push comes to shove really want functionality where when user access is cancelled all active sessions get torn down immediatly as well.
I really enjoyed my time with Vault's ssh-ca (back when it had a sane license) but have now grown up and believe that any ssh access is an antipattern. For context, I'm also one of those "immutable OS or GTFO" chaps because in my experience the next thing that happens after some rando ssh-es into a machine is they launch vi or apt-get or whatever and now it's a snowflake with zero auditing of the actions taken to it
I don't mean to detract from this, because short-lived creds are always better, but for my money I hope I never have sshd running on any machine again
Not generally. In one particular class of deployments allowing ssh access to root enabled accounts without auditing may be.. but this is an exceptionally narrowed definition.
> I hope I never have sshd running on any machine again
Sounds great for production and ridiculous for development and testing.
Principle of least privilege trivially prevents updating system packages. Like if you don't want people using apt, don't give people root on your servers?
The whole MITM just makes me deeply uncomfortable, it's introducing a single point of trust with the keys to the kingdom. If I want to log what someone is doing, I do it server side e.g. some kind of rsyslog. That way I can leverage existing log anomaly detection systems to pick up and isolate the server if we detect any bad behaviour.
this just moves the trusted component from the SSH key to Cloudflare, and you still must trust something implicitly. except now it's a company that has agency and a will of its own instead of just some files on a filesystem.
I wrote a system that did this >5 years ago (luckily was able to open source it before the startup went under[0]). The bastion would record ssh sessions in asciicast v2 format and store those for later playback directly from a control panel. The main issue that still isn't solved by a solution like this is user management on the remote (ssh server) side. In a more recent implementation, integration with LDAP made the most sense and allows for separation of user and login credentials. A single integrated solution is likely the holy grail in this space.
I would think even a simple "sorry, this change does not align with the project's goals" -> closed would help the submitter (and others) have some clarity versus the PR limbo it's currently in
That aside, thanks so much for pointing this out: it looks like good fun, especially the Asciicast support!
I didn’t understand the marketing term “zero trust” and I still don’t.
In practice, I get it - a network zone shouldn’t require a lower authn/z bar on the implicit assumption that admission to that zone must have required a higher bar.
But all these systems are built on trust, and if it isn’t based on network zoning, it’s based on something else. Maybe that other thing is better, maybe not. But it exists and it needs to be understood.
An actual zero trust system is the proverbial unpowered computer in a bunker.
It means there is zero trust of a device/service/user on your network until they have been fully authenticated. It is about having zero trust in something just because it is inside your network perimeter.
The something else is specifically user/service identity. Not machine identity, not IP address. It is somewhat silly to have a buzzword that means "no, actually authenticate users" but here we are.
Yeah, it’s not a great name. Twenty years ago we called it “end to end authentication” and I think that’s better because it focuses on the most important aspect, but it probably doesn’t sound as cool for marketing purposes.
I also like how that makes it easier to understand how variation is normal: for example, authentication comes in various flavors and that’s okay whereas some of that zero trust vendors will try to claim that something is or isn’t ZT based on feature gaps in their competitors’ and it’s just so tedious to play that game.
Instead of stealing your password/keypair, the baddies will now have to spoof your authentication with cloudflare. If thats just a password, you gained nothing. If you have 2FA set up for that, you could equally use that for SSH directly, using a ssh key on a physical FIDO stick. OpenSSH already has native support for that (ecdsa-sk and ed25519-sk key formats).
So... don't trust long lived ssh keys, but trust Cloudflare's CA. Why? What has Cloudflare done to earn trust?
If that alone weren't reason enough to dismiss this, the article has marketing BS throughout. For instance, "SSH access to a server often comes with elevated privileges". Ummm... Every authentication system ever has whatever privileges that come with that authentication system. This is the kind of bull you say / write when you want to snow someone who doesn't know any better. To those of us who do understand this, this is almost AI level bullshit.
The same is true of their supposed selling points:
> Author fine-grained policy to govern who can SSH to your servers and through which SSH user(s) they can log in as.
That's exactly what ssh does. You set up precisely which authentication methods you accept, you set up keys for exactly that purpose, and you set up individual accounts. Do Cloudflare really think we're setting up a single user account and giving access to lots of different people, and we need them to save us? (now that I think about it, I bet some people do this, but this is still a ridiculous selling point)
> Monitor infrastructure access with Access and SSH command logs
So they're MITM all of our connections? We're supposed to trust them, even though they have a long history of not only working with scammers and malicious actors, but protecting them?
I suppose there's a sucker born every minute, so Cloudflare will undoubtedly sell some people on this silliness, but to me it just looks like yet another way that Cloudflare wants to recentralize the Internet around them. If they had their way, then in a few years, were they to go down, a majority of the Internet would literally stop working. That should scare everyone.
I just wanted to offer my congratulations on the acquisition. I don't know any details about your specific one, but I have been around enough to know that it's still worth celebrating o/
* This has nothing to do with zero-trust. If you already require pubkey auth to every connection made to a server regardless of origin, that's already meeting the definition of zero trust.
* What this actually gives you is a solution to the problem of centrally revoking long-lived keys by not having any and instead using certificate auth. Now the CA is the only long-lived key.
* This is a reasonable thing large orgs should probably do. There is no reason the CA should be an external third-party like Cloudflare, however.
* This also integrates with existing SSO providers so human users can be granted short-lived session certs based on whatever you use to authenticate them to the SSO provider. Also reasonable, also no reason this should be offered as a service from Cloudflare as opposed to something you can self-host like Kerberos.
* This also provides ssh command logging by proxying the session and capturing all commands as they get relayed. Arguably not a bad idea in principle, but a log collector like rsyslogd sending to an aggregator accomplishes the same thing in practice, and again, I would think you'd want to self-host a proxy if you choose to go that route, not rent it from Cloudflare.
All in all, good things a lot of orgs should do, but they should probably actually do. I get the "well, it's hard" angle, but you're usually looking at large, well-funded orgs when you're talking things like SOC and FedRamp compliance. If you want to be a bank or whatever, yeah, that's hard. It's supposed to be. As I understand it, at least part of the spirit of SOC and FedRamp and the like is your organization has processes, plans, procedures, and personnel in place with the expertise and care to take security seriously, not "we have no idea what any of this means, why it matters, and don't have the time, but we pay a subscription fee to Cloudflare and they say they take care of it."
hoopdev here. Zero trust for SSH is just table stakes these days. Real challenge is getting devs to actually adopt better practices without the tooling getting in their way.
Found in practice that certs > keys but you need to think beyond just SSH. Most teams have a mix of SSH, K8s, DBs etc. Using separate tools for each just creates more headache.
Haven't tried Boundary but Teleport/hoop/Tailscale all handle the mixed protocol issue decently. Main difference is hoop focuses more on protocol-level DLP and automated reviews vs pure network access. Horses for courses though, they're all valid approaches.
Key is picking something devs will actually use vs work around. Nothing worse than a "secure" solution that drives people to create workarounds.
Cloudflare has been offering SSH CA-based authentication for more than 2 years [1], I wrote a guide back in feb '23 [2]. The announcement is more about offering new features, such as more granular user control.
I get that HN does not like Cloudflare and does not like the term “Zero Trust”, but geez these comments are repetitive. Can anyone compare to Tailscale SSH? Are they basically offering an (even more) enterprise version of Tailscale’s product line?
Hah. I did pretty much all the same stuff in my previous company.
One thing that we did a bit better: we used AWS SSM to provision our SSH-CA certificates onto the running AWS EC2 instances during the first connection.
It would be even better if AWS allowed to use SSH CA certs as keys, but alas...
I too would love "native" support for SSH CAs in EC2. What I ended up doing is adding a line to every EC2 userdata script that would rewrite the /home/ec2-user/.ssh/authorized_keys file to treat the provided EC2 keypair as a CA instead of a regular pubkey.
Using CAs and signed certificates in SSH is definitely the way.
If anyone wants to play around with that, without the risk of locking themselves out of a server, I built a little "playground" awhile back whihc is a series of Docker containers that can SSH to each other. Give it a try at https://github.com/dmuth/ssh-principal-and-ca-playground
(I haven't touched the project in awhile, so if there are any issues, please open an Issue and I'll gladly look at it!)
Is there anything similar ("central point of SSH access/keys management" ) that is not Cloudflare ? I know about Tailscale and it's SSH but recently it introduced so much latency (even tho they say it's P2P between A and B) that is unusable.
Ideally something self hosted but not hard requirement
> the SSH certificates issued by the Cloudflare CA include a field called ValidPrinciples
Having implemented similar systems before, I was interested to read this post. Then I see this. Now I have to find out if that really is the field, if this was ChatGPT spellcheck, or something else entirely.
[+] [-] edelbitter|1 year ago|reply
[+] [-] hedora|1 year ago|reply
My mental model:
With 1 trust (the default) any trusted machine with credentials is provided access and therefore gets one unit of access. With 2-trust, we’d need at least units of trust, so two machines. Equivalently, each credential-bearing machine is half trusted (think ssh bastion hosts or 2FA / mobikeys for 2 trust).
This generalizes to 1/N, so for zero trust, we place 1/0 = infinite units of trust in every machine that has a credential. In other words, if we provision any one machine for access, we necessarily provision an unbounded number of other machines for the same level of access.
As snarky as this math is, I’ve yet to see a more accurate formulation of what zero trust architectures actually provide.
YMmV.
[+] [-] varenc|1 year ago|reply
Zero Trust just means you stop inherently trusting your private network and verify every user/device/request regardless. If you opt in to using Cloudflare to do this then it requires running Cloudflare software.
[+] [-] fs111|1 year ago|reply
[+] [-] ozim|1 year ago|reply
TLS having trusted CA cert publisher is not context of “Zero Trust”.
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] znpy|1 year ago|reply
The truth-yness of "zero trust" really depends on who's trusting who.
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] pjc50|1 year ago|reply
Would you like to explain what you mean by this?
[+] [-] tptacek|1 year ago|reply
[+] [-] michaelt|1 year ago|reply
I think the sales pitch for these sorts of service is: "Get an SSH-like experience, but it integrates with your corporate single-sign-on system, has activity logs that can't be deleted even if you're root on the target, sorts out your every-ephemeral-cloud-instance-has-a-different-fingerprint issues, and we'll sort out all the reverse-tunnelling-through-NAT and bastion-host-for-virtual-private-cloud stuff too"
Big businesses pursuing SOC2 compliance love this sort of thing.
[+] [-] kevin_nisbet|1 year ago|reply
The second order problem I've found is when you dig in there are plenty of people who ask for certs but when push comes to shove really want functionality where when user access is cancelled all active sessions get torn down immediatly as well.
[+] [-] xyst|1 year ago|reply
[+] [-] mdaniel|1 year ago|reply
I don't mean to detract from this, because short-lived creds are always better, but for my money I hope I never have sshd running on any machine again
[+] [-] akira2501|1 year ago|reply
Not generally. In one particular class of deployments allowing ssh access to root enabled accounts without auditing may be.. but this is an exceptionally narrowed definition.
> I hope I never have sshd running on any machine again
Sounds great for production and ridiculous for development and testing.
[+] [-] advael|1 year ago|reply
[+] [-] blueflow|1 year ago|reply
[+] [-] ashconnor|1 year ago|reply
Disclaimer: I work at Hashicorp.
[+] [-] ozim|1 year ago|reply
Stuff I work on is write heavy so spawning dozens of app copies doesn’t make sense if I just hog the db with Erie locks.
[+] [-] namxam|1 year ago|reply
[+] [-] riddley|1 year ago|reply
[+] [-] TechnicalVault|1 year ago|reply
[+] [-] naikrovek|1 year ago|reply
this just moves the trusted component from the SSH key to Cloudflare, and you still must trust something implicitly. except now it's a company that has agency and a will of its own instead of just some files on a filesystem.
I'll stick to forced key rotation, thanks.
[+] [-] antoniomika|1 year ago|reply
[0] https://github.com/notion/bastion
[+] [-] mdaniel|1 year ago|reply
I would think even a simple "sorry, this change does not align with the project's goals" -> closed would help the submitter (and others) have some clarity versus the PR limbo it's currently in
That aside, thanks so much for pointing this out: it looks like good fun, especially the Asciicast support!
[+] [-] shermantanktop|1 year ago|reply
In practice, I get it - a network zone shouldn’t require a lower authn/z bar on the implicit assumption that admission to that zone must have required a higher bar.
But all these systems are built on trust, and if it isn’t based on network zoning, it’s based on something else. Maybe that other thing is better, maybe not. But it exists and it needs to be understood.
An actual zero trust system is the proverbial unpowered computer in a bunker.
[+] [-] athorax|1 year ago|reply
[+] [-] wmf|1 year ago|reply
[+] [-] ngneer|1 year ago|reply
[+] [-] acdha|1 year ago|reply
I also like how that makes it easier to understand how variation is normal: for example, authentication comes in various flavors and that’s okay whereas some of that zero trust vendors will try to claim that something is or isn’t ZT based on feature gaps in their competitors’ and it’s just so tedious to play that game.
[+] [-] blueflow|1 year ago|reply
The gain here is minimal.
[+] [-] keepamovin|1 year ago|reply
[+] [-] udev4096|1 year ago|reply
[+] [-] knallfrosch|1 year ago|reply
[+] [-] ChoHag|1 year ago|reply
[deleted]
[+] [-] johnklos|1 year ago|reply
If that alone weren't reason enough to dismiss this, the article has marketing BS throughout. For instance, "SSH access to a server often comes with elevated privileges". Ummm... Every authentication system ever has whatever privileges that come with that authentication system. This is the kind of bull you say / write when you want to snow someone who doesn't know any better. To those of us who do understand this, this is almost AI level bullshit.
The same is true of their supposed selling points:
> Author fine-grained policy to govern who can SSH to your servers and through which SSH user(s) they can log in as.
That's exactly what ssh does. You set up precisely which authentication methods you accept, you set up keys for exactly that purpose, and you set up individual accounts. Do Cloudflare really think we're setting up a single user account and giving access to lots of different people, and we need them to save us? (now that I think about it, I bet some people do this, but this is still a ridiculous selling point)
> Monitor infrastructure access with Access and SSH command logs
So they're MITM all of our connections? We're supposed to trust them, even though they have a long history of not only working with scammers and malicious actors, but protecting them?
I suppose there's a sucker born every minute, so Cloudflare will undoubtedly sell some people on this silliness, but to me it just looks like yet another way that Cloudflare wants to recentralize the Internet around them. If they had their way, then in a few years, were they to go down, a majority of the Internet would literally stop working. That should scare everyone.
[+] [-] EthanHeilman|1 year ago|reply
We (BastionZero) recently got bought by Cloudflare and it is exciting bringing our SSH ideas to Cloudflare.
[+] [-] lenova|1 year ago|reply
[+] [-] mdaniel|1 year ago|reply
[+] [-] WesolyKubeczek|1 year ago|reply
[+] [-] nonameiguess|1 year ago|reply
* This has nothing to do with zero-trust. If you already require pubkey auth to every connection made to a server regardless of origin, that's already meeting the definition of zero trust.
* What this actually gives you is a solution to the problem of centrally revoking long-lived keys by not having any and instead using certificate auth. Now the CA is the only long-lived key.
* This is a reasonable thing large orgs should probably do. There is no reason the CA should be an external third-party like Cloudflare, however.
* This also integrates with existing SSO providers so human users can be granted short-lived session certs based on whatever you use to authenticate them to the SSO provider. Also reasonable, also no reason this should be offered as a service from Cloudflare as opposed to something you can self-host like Kerberos.
* This also provides ssh command logging by proxying the session and capturing all commands as they get relayed. Arguably not a bad idea in principle, but a log collector like rsyslogd sending to an aggregator accomplishes the same thing in practice, and again, I would think you'd want to self-host a proxy if you choose to go that route, not rent it from Cloudflare.
All in all, good things a lot of orgs should do, but they should probably actually do. I get the "well, it's hard" angle, but you're usually looking at large, well-funded orgs when you're talking things like SOC and FedRamp compliance. If you want to be a bank or whatever, yeah, that's hard. It's supposed to be. As I understand it, at least part of the spirit of SOC and FedRamp and the like is your organization has processes, plans, procedures, and personnel in place with the expertise and care to take security seriously, not "we have no idea what any of this means, why it matters, and don't have the time, but we pay a subscription fee to Cloudflare and they say they take care of it."
[+] [-] andriosr|1 year ago|reply
Found in practice that certs > keys but you need to think beyond just SSH. Most teams have a mix of SSH, K8s, DBs etc. Using separate tools for each just creates more headache.
Haven't tried Boundary but Teleport/hoop/Tailscale all handle the mixed protocol issue decently. Main difference is hoop focuses more on protocol-level DLP and automated reviews vs pure network access. Horses for courses though, they're all valid approaches.
Key is picking something devs will actually use vs work around. Nothing worse than a "secure" solution that drives people to create workarounds.
[+] [-] curben|1 year ago|reply
[1]:https://web.archive.org/web/20210418143636/https://developer...
[2]: https://mdleom.com/blog/2023/02/13/ssh-certificate-cloudflar...
[+] [-] singhrac|1 year ago|reply
[+] [-] cyberax|1 year ago|reply
One thing that we did a bit better: we used AWS SSM to provision our SSH-CA certificates onto the running AWS EC2 instances during the first connection.
It would be even better if AWS allowed to use SSH CA certs as keys, but alas...
[+] [-] pugz|1 year ago|reply
I too would love "native" support for SSH CAs in EC2. What I ended up doing is adding a line to every EC2 userdata script that would rewrite the /home/ec2-user/.ssh/authorized_keys file to treat the provided EC2 keypair as a CA instead of a regular pubkey.
[+] [-] dmuth|1 year ago|reply
If anyone wants to play around with that, without the risk of locking themselves out of a server, I built a little "playground" awhile back whihc is a series of Docker containers that can SSH to each other. Give it a try at https://github.com/dmuth/ssh-principal-and-ca-playground
(I haven't touched the project in awhile, so if there are any issues, please open an Issue and I'll gladly look at it!)
[+] [-] arianvanp|1 year ago|reply
Would be nice if they can replace TOFU access with SSH CA as well. Ideally based on device posture of the server (e.g. TPM2 attestation)
[+] [-] amar0c|1 year ago|reply
Ideally something self hosted but not hard requirement
[+] [-] nanis|1 year ago|reply
Having implemented similar systems before, I was interested to read this post. Then I see this. Now I have to find out if that really is the field, if this was ChatGPT spellcheck, or something else entirely.