Always nice to see a vendor respond quickly and adequately. Even better when they go above and beyond, as seems to be indicated here by the timeline posted and the assessment of what fixes were put in place by the researcher(s). Good job Tailscale.
Very well written. Also quite worrying given they're supposed to be a security company and these kinds of issues are well known.
Then again it does seem like the entire universe applies "eh probably nobody will try and hack it" to services listening on local TCP interfaces.
They certainly don't care about multi-user machines, though I suppose there are so many local root exploits these days you're basically trusting your users anyway in that situation.
> In theory, there is no path for a malicious Tailscale control plane to remotely execute code on your machine, unless you happen to run network services that are designed to allow it, like an SSH server with Tailscale-backed authentication.
Now I feel less crazy for not using Tailscale SSH for similar reasons.
I'd like to see a security evaluation of Tailscale, on a per feature basis.
I'd like to see tailscaled run with far fewer privileges.
Is there a Tailscale alternative that just does Wireguard + NAT traversal and doesn't try to do key management?
Yep. Same boat. Absolutely zero interest in granting them ssh authZ; transport wrapping is all I want to outsource. Just deliver my bits and I pay you, tyvm. My suspicions have been proven correct here.
Unfortunately reading about this remote RCE vector has me wondering whether I can use the product at all without all this bloat (taildrop, ssh, etc) affecting me. Going to have my team look at zerotier this week, I’ve heard a few ok things.
> I'd like to see tailscaled run with far fewer privileges.
Yeah - I have a dislike for services running as root when it's not necessary, and then getting users to escalate to root to interact with them routinely.
One thing I was thinking about was trying to identify the Linux capabilities which let tailscaled run, and then look at if it's feasible to adjust the default systemd unit to run it as a non root user. Closely followed by then trying to harden up the service with as many of the recommendations as possible in "systemd-analyze security".
Despite there being a pretty good range of restrictions available, it seems to be pretty rare that service definitions actually come locked down... Might be something for the tailscale team to look at in future?
I use a cheap public VPS and Wireguard and it works over my ISP connection at home. My servers run here at home but are only publicly visible at my VPS public ISP address. Is that the same as what you're asking for with 'NAT traversal'?
If so, the config is straightforward for techies, just Wireguard config, it routes into my home server and I use Apache Reverse Proxy to route to the backend services.
Yeah the whole "run local http server as control panel" is iffy for non security centric stuff, let alone VPN software.
I guess it is because it's easy ? But now even windows can make unix sockets, that seems like reasonably easy and secure solution for "talk with some daemon portably"
> Is there a Tailscale alternative that just does Wireguard + NAT traversal and doesn't try to do key management?
I really wish there was a NAT traversal protocol or library that wasn't overly complex and focused on the 90% cases. It would help not just tailscale's but anyone building p2p tech.
Do they have enough logs to reach out to people that were affected? As far as vulnerabilities go, this set is one is one of the worst ones I've seen this decade, and they seem rather straightforward.
Would be nice to get a blog post from them that goes a bit into impact, not just a report that tells you to update. It's nice that they responded quickly, but I feel like this shouldn't have happened in the first place for a network security company and it makes the Windows client feel like a bit of an afterthought. Looks like they have a PR open to switch it to named pipes, I hope that is properly reviewed by someone that knows Windows APIs before it's merged.
Related: note also that tailscale's tailnet 100.* subnet is som form of CGNAT public ip block. I think Tailscale thought long and hard about this, and landed on it because it was a path of lesser resistance to break fewer things. And if you squint they fit the stated purpose.
But even if browsers now implement PNA the tailnet itself is public address space, so that vector still exists. I wonder if browsers (and eventually standards) will be pressured to treat those blocks as private.
Geez, yeah, I was going to say that it's clearly bonkers that DNS rebinding can trick the browser into communicating with loopback addresses, until I got to the part of the article that explained how there actually is a mitigation for that. Hopefully Firefox fixes that soon, because I shudder to think how many applications are vulnerable.
> If you run non-HTTPS web services on your Tailnet, and those services are unauthenticated or rely on Tailscale for authentication, implement an allowlist of expected HTTP Host headers to prevent malicious Javascript from accessing these services.
In my opinion, this should be done not only for non-HTTPS services, but for all services: the "default" virtual host (used where there is no Host header, or when it has an unexpected value) should have nothing except a static 4xx error page. This not only avoids DNS rebinding attacks, but also avoids automated attacks in which the attacker doesn't know the correct hostname for the service (mostly automated scans for vulnerable PHP scripts and similar).
In addition, it's basically required if you're using a reverse proxy service, eg. Cloudflare or Akamai. CF websites have been found via Shodan or Censys because site info is left open via https ://[ip]:443.
I don't see a writeup of how this was fixed. Merely checking the Host header is insufficient -- the vulnerability would still be wide open to anyone who can open TCP sockets to localhost.
Windows has APIs (named pipes, DCOM (eww) and such) that allow authenticated local access to services. Unixes have unix sockets.
The concept of DNS rebinding and DNS records pointing to a private/localhost IP address is particularly interesting and I remember when I first came across it in the wild. It's not exactly re-binding in the classic attack sense described in the article: some US sportsbooks make you download a geolocation service that verifies your location in order to place bets. The sportsbook's front end communicates with it through a DNS record pointing back to 127.0.0.1, and opens up a WebSocket to talk to the service. I imagine the WebSocket is used to bypass the same-origin policy but perhaps someone more knowledgeable can speak to that.
The client app is not indicating that 1.32.3 for Windows is available yet but the download link on the site has been updated.
Tailscale client downloads are extremely slow at the moment, so I suggest you distribute one copy manually around your tailnet rather than bogging down their servers even more.
The Windows client caches the current version for a while, so may not yet have v1.32.3 available on your device.
In that case, you can still pull the latest release from http://pkgs.tailscale.com/stable.
I've read the description several times and find it hard to follow:
..an attacker-controlled website visited by the node..rebinds DNS for the peer API to an attacker-controlled DNS server making peer API requests in the client,
including accessing the node’s Tailscale environment variables
I enjoyed the explanation very much. Even though I don't use mesh VPNs (yet), the architectural discussion of the vulnerability entailed numerous useful bits of background on browser and network infrastructure. Commendable work!
[+] [-] ghuntley|3 years ago|reply
ps. she's looking an employer rn // hire her!
[+] [-] dang|3 years ago|reply
[+] [-] kbenson|3 years ago|reply
[+] [-] IshKebab|3 years ago|reply
Then again it does seem like the entire universe applies "eh probably nobody will try and hack it" to services listening on local TCP interfaces.
They certainly don't care about multi-user machines, though I suppose there are so many local root exploits these days you're basically trusting your users anyway in that situation.
[+] [-] beanjuiceII|3 years ago|reply
[+] [-] erdaniels|3 years ago|reply
[+] [-] freedinosaur|3 years ago|reply
Now I feel less crazy for not using Tailscale SSH for similar reasons.
I'd like to see a security evaluation of Tailscale, on a per feature basis.
I'd like to see tailscaled run with far fewer privileges.
Is there a Tailscale alternative that just does Wireguard + NAT traversal and doesn't try to do key management?
[+] [-] infotogivenm|3 years ago|reply
Unfortunately reading about this remote RCE vector has me wondering whether I can use the product at all without all this bloat (taildrop, ssh, etc) affecting me. Going to have my team look at zerotier this week, I’ve heard a few ok things.
[+] [-] g_p|3 years ago|reply
Yeah - I have a dislike for services running as root when it's not necessary, and then getting users to escalate to root to interact with them routinely.
One thing I was thinking about was trying to identify the Linux capabilities which let tailscaled run, and then look at if it's feasible to adjust the default systemd unit to run it as a non root user. Closely followed by then trying to harden up the service with as many of the recommendations as possible in "systemd-analyze security".
Despite there being a pretty good range of restrictions available, it seems to be pretty rare that service definitions actually come locked down... Might be something for the tailscale team to look at in future?
[+] [-] wankle|3 years ago|reply
If so, the config is straightforward for techies, just Wireguard config, it routes into my home server and I use Apache Reverse Proxy to route to the backend services.
[+] [-] skrowl|3 years ago|reply
More info on how it works: https://www.jordanwhited.com/posts/wireguard-endpoint-discov...
[+] [-] adql|3 years ago|reply
I guess it is because it's easy ? But now even windows can make unix sockets, that seems like reasonably easy and secure solution for "talk with some daemon portably"
[+] [-] klabb3|3 years ago|reply
I really wish there was a NAT traversal protocol or library that wasn't overly complex and focused on the 90% cases. It would help not just tailscale's but anyone building p2p tech.
[+] [-] jarym|3 years ago|reply
Focus is more on device identity/posture, DNS + remote access rather than straight VPN like Tailscale & co
[+] [-] mikestaub|3 years ago|reply
[+] [-] ehutch79|3 years ago|reply
[+] [-] meibo|3 years ago|reply
Would be nice to get a blog post from them that goes a bit into impact, not just a report that tells you to update. It's nice that they responded quickly, but I feel like this shouldn't have happened in the first place for a network security company and it makes the Windows client feel like a bit of an afterthought. Looks like they have a PR open to switch it to named pipes, I hope that is properly reviewed by someone that knows Windows APIs before it's merged.
[+] [-] mcoliver|3 years ago|reply
[+] [-] ripley12|3 years ago|reply
> Am I affected?
> Yes. Your tailnet has at least one Windows node running a version of Tailscale prior to v1.32.3.
[+] [-] arkadiyt|3 years ago|reply
Original comment:
> Do they have enough logs to reach out to people that were affected?
It happens on the client, there are no server logs that Tailscale could check
[+] [-] Thaxll|3 years ago|reply
It's not that bad actually.
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] semi-extrinsic|3 years ago|reply
Does anyone know why? It seems like an obviously good thing to have.
https://wicg.github.io/private-network-access/
[+] [-] klabb3|3 years ago|reply
But even if browsers now implement PNA the tailnet itself is public address space, so that vector still exists. I wonder if browsers (and eventually standards) will be pressured to treat those blocks as private.
[+] [-] modeless|3 years ago|reply
[+] [-] btdmaster|3 years ago|reply
[+] [-] angry_octet|3 years ago|reply
[+] [-] cesarb|3 years ago|reply
In my opinion, this should be done not only for non-HTTPS services, but for all services: the "default" virtual host (used where there is no Host header, or when it has an unexpected value) should have nothing except a static 4xx error page. This not only avoids DNS rebinding attacks, but also avoids automated attacks in which the attacker doesn't know the correct hostname for the service (mostly automated scans for vulnerable PHP scripts and similar).
[+] [-] judge2020|3 years ago|reply
[+] [-] ev1|3 years ago|reply
[+] [-] amluto|3 years ago|reply
Windows has APIs (named pipes, DCOM (eww) and such) that allow authenticated local access to services. Unixes have unix sockets.
[+] [-] dang|3 years ago|reply
[+] [-] Arnavion|3 years ago|reply
[+] [-] woodrow|3 years ago|reply
[+] [-] snake_plissken|3 years ago|reply
[+] [-] jakedata|3 years ago|reply
Tailscale client downloads are extremely slow at the moment, so I suggest you distribute one copy manually around your tailnet rather than bogging down their servers even more.
[+] [-] mayakacz|3 years ago|reply
The Windows client caches the current version for a while, so may not yet have v1.32.3 available on your device. In that case, you can still pull the latest release from http://pkgs.tailscale.com/stable.
[+] [-] bradfitz|3 years ago|reply
[+] [-] cassianoleal|3 years ago|reply
[+] [-] 0xbadcafebee|3 years ago|reply
[+] [-] fulafel|3 years ago|reply
https://en.wikipedia.org/wiki/Attack_tree
[+] [-] mrjin|3 years ago|reply
[+] [-] loo|3 years ago|reply
[+] [-] zvrba|3 years ago|reply
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] say_it_as_it_is|3 years ago|reply
..an attacker-controlled website visited by the node..rebinds DNS for the peer API to an attacker-controlled DNS server making peer API requests in the client, including accessing the node’s Tailscale environment variables
[+] [-] erosenbe0|3 years ago|reply
[+] [-] radicaldreamer|3 years ago|reply
[+] [-] chaz6|3 years ago|reply