It's worth noting that System Transparency is a multi-year effort to bring transparency to running computer systems. We are aiming for what we call transparent servers. Just like there's open source software and open source hardware we think there should be open source running systems.
That's the gist of it.
If you think this is interesting I can highly recommend you check out Sigsum - our transparency log design for signed checksums. We've been developing it for a few years and will most likely toggle it version 1 this spring. Here's its threat model:
Sigsum is designed to be secure against a powerful attacker that controls:
- The signer’s secret key and infrastructure
- The log’s secret key and infrastructure
- A threshold of so-called witnesses that cosign the log
Another project that started at Mullvad VPN and is now its own company is Tillitis. Its first product is an open source hardware USB device with unconditional measured boot and key derivation inspired by DICE. Everything from source code to Verilog and KiCad files are on GitHub. Enjoy!
Cheers, Fredrik Stromberg
(Disclosure: I cofounded Mullvad VPN, invented System Transparency, co-designed Sigsum, co-designed TKey, and cofounded Tillitis)
I love the concept. I created and ran a PXE/netbooted full OS on ramdisk[1] for my old companies servers for years. We were in the high performance computing and storage space. Stateless machines have so many advantages over stateful.
That said, solving a trusted boot problem was not something I could tackle alone. I didn't have a sense for how much/little I could trust the machine/bios/firmware. None of the tooling I considered (hashing firmware/boot data/etc.) seemed secure without a whole additional infrastructure.
These are great updates. I couldn't be happier with mullvad. The VPN space is saturated with a lot of VPNs constantly advertising with borderline false claims (a VPN won't stop advertisers from targeting you for example) and adding unrelated features (like an anti-virus). But mullvad is off to the side providing a high quality, truly private, VPN service at a great price.
Wow, I had no idea "diskless infrastructure" was even a thing. Easy to imagine in theory, but this is the first time I'm hearing about it in practice, and it makes total sense in this case.
It makes me curious if there are any other real-world use cases for diskless. Are there any customers who would benefit from such a configuration from major cloud providers? E.g. a diskless EC2 instance type that ran off of a RAM disk?
> Wow, I had no idea "diskless infrastructure" was even a thing.
OVPN[1] (Swedish jurisdiction) have been diskless[2] since day one, and this has been tested and proven in court[3]:
"To summarize the verdict, the Rights Alliance and their security experts have not been able prove any weaknesses in OVPN's systems that could mean that logs are stored. OVPN therefore wins the information injunction as our statements and evidence regarding our no log VPN policy have not been disproven. The movie companies also need to pay OVPN's legal fees which amounts to 108 000 SEK (roughly $12300 at current exchange rate)."
Netbooted machines were not that uncommon in the past. I implemented and ran a large scale nfsroot build farm around 2010. The company was in the business of high end NFS storage so maximum dogfooding was a part of the goal. Prior to that at another company we had most of the infra (including stuff like corporate email and web servers) and also all the engineering user desktops on nfsroot. Seems a bit crazy in a hindsight but it got the company through the startup years and shipping several generations of products.
It used to be pretty common (in the late 1980s) to run Sun workstations without hard disks, all booting off an NFS server. But then disks got cheap... I always set up a boot server hosting a bunch of BSD/Linux images so I could install an OS remotely (to a local disk) without running around with a CD or USB stick.
Been a thing for a long time. I first ran into it back in the Novell days with RPL netbooting, and it's moved through BOOTP to PXE booting. Biggest benefits pitched are usually "lower cost" and "centralized management".
As an example, X Terminals all worked this way. Much of the old Sun Microsystems "The Network Is The Computer" pitch was having low-end, diskless SPARC machines (e.g. SLC, ELC) netbooting and mounting disk and doing heavy lift compute via NFS & X11 from large SPARC servers and storage arrays.
You can look at the Linux Terminal Server Project (ltsp.org) for some reasonably current ideas of what someone might do with this.
Various flavours of diskless booting have been there since decades.
Simple example - thin client that boots off readonly NFS and just mounts user dir when they log in. Or one step further and have image with remote desktop software and nothing else.
More complex one - storage server that boots over network so you don't have to manage any OS install on disks and can use full capacity of drives for storage
I was maintaining a few public terminals for Internet access in local hackerspace many moons ago - back when the cheap broadband and wifi at every corner weren't at all common. Terminals were diskless Pentium 100 (or even slower) PCs that PXE booted off the terminal server Xen VM running Xfce via Xvfb. Terminals were basically oversized I/O controllers taking care of keyboard/mouse inputs and driving video output.
Nowadays you could use similar approach to run cluster for in-memory compute tasks or similar. PXE boot identical OS to bunch of servers and have them compute sth. If you need to repurpose them for something else - reboot them into a different PXE boot image. In case of VPN providers the motivation is probably to prevent (permanent) logging of sensitive information.
GCE has been offering diskless instances since always, even if customers did not realize it. They explicitly describe some instance types as "diskless" these days.
Talos Linux is a Linux designed for Kubernetes which runs from memory. I'm not sure if it could run diskless, because I expect kubernetes workloads to require some local disk.
I created a system that booted 12k+ diskless blades via PXE and running Ubuntu (it was built to scale to 30k+, but we never got there).
This generally works well, but I'd say there are about 0-20 blades that crash a day due to some sort of memory corruption issues.
Due to the fact that I was operating remotely from the hardware, I never really got a chance to resolve it... also... just a simple reboot would fix it (and the blades booted in ~60 seconds, so it wasn't a huge issue).
So, on large enough scale... this can be an issue to consider.
Is that caused or exacerbated by being diskless, though? Or is it just inevitable that 12k+ machines are going to have a certain rate of memory errors regardless?
> Running the system in RAM does not prevent the possibility of logging. It does however minimise the risk of accidentally storing something that can later be retrieved.
Mullvad offers flat rate $5 (no matter 1 month or 12 months or 120 months) and never have any sales so I'm surprised to see these[1] prepaid amazon cards ARE offering discounts: 12mo @ $4.75/mo & 6mo @ $4.83/mo esp. when these are /physical/ code-card purchases
They do provide a nominal discount if you pay with Bitcoin, though I assumed that had something to do with the lack of payment processor fees so it doesn't necessarily explain why the Amazon prices are different.
They don't have any other rate for you and me as individuals, but I'm sure they offer them slightly cheaper for resellers like Mozilla and Malwarebytes.
Those resellers then charge about the same price as Mullvad, but get to keep a piece of that as profit.
kfreds|3 years ago
That's the gist of it.
If you think this is interesting I can highly recommend you check out Sigsum - our transparency log design for signed checksums. We've been developing it for a few years and will most likely toggle it version 1 this spring. Here's its threat model:
Sigsum is designed to be secure against a powerful attacker that controls:
- The signer’s secret key and infrastructure - The log’s secret key and infrastructure - A threshold of so-called witnesses that cosign the log
Another project that started at Mullvad VPN and is now its own company is Tillitis. Its first product is an open source hardware USB device with unconditional measured boot and key derivation inspired by DICE. Everything from source code to Verilog and KiCad files are on GitHub. Enjoy!
Cheers, Fredrik Stromberg
(Disclosure: I cofounded Mullvad VPN, invented System Transparency, co-designed Sigsum, co-designed TKey, and cofounded Tillitis)
hpcjoe|3 years ago
That said, solving a trusted boot problem was not something I could tackle alone. I didn't have a sense for how much/little I could trust the machine/bios/firmware. None of the tooling I considered (hashing firmware/boot data/etc.) seemed secure without a whole additional infrastructure.
I'm thrilled to see this implemented though.
[1] modern version here: https://github.com/joelandman/nyble
morsecodist|3 years ago
netfortius|3 years ago
kfreds|3 years ago
crazygringo|3 years ago
It makes me curious if there are any other real-world use cases for diskless. Are there any customers who would benefit from such a configuration from major cloud providers? E.g. a diskless EC2 instance type that ran off of a RAM disk?
traceroute66|3 years ago
OVPN[1] (Swedish jurisdiction) have been diskless[2] since day one, and this has been tested and proven in court[3]:
"To summarize the verdict, the Rights Alliance and their security experts have not been able prove any weaknesses in OVPN's systems that could mean that logs are stored. OVPN therefore wins the information injunction as our statements and evidence regarding our no log VPN policy have not been disproven. The movie companies also need to pay OVPN's legal fees which amounts to 108 000 SEK (roughly $12300 at current exchange rate)."
[1]https://www.ovpn.com [2]https://www.ovpn.com/en/security [3]https://www.ovpn.com/en/blog/ovpn-wins-court-order
donio|3 years ago
mprovost|3 years ago
kjs3|3 years ago
As an example, X Terminals all worked this way. Much of the old Sun Microsystems "The Network Is The Computer" pitch was having low-end, diskless SPARC machines (e.g. SLC, ELC) netbooting and mounting disk and doing heavy lift compute via NFS & X11 from large SPARC servers and storage arrays.
You can look at the Linux Terminal Server Project (ltsp.org) for some reasonably current ideas of what someone might do with this.
ilyt|3 years ago
Simple example - thin client that boots off readonly NFS and just mounts user dir when they log in. Or one step further and have image with remote desktop software and nothing else.
More complex one - storage server that boots over network so you don't have to manage any OS install on disks and can use full capacity of drives for storage
jskrablin|3 years ago
Nowadays you could use similar approach to run cluster for in-memory compute tasks or similar. PXE boot identical OS to bunch of servers and have them compute sth. If you need to repurpose them for something else - reboot them into a different PXE boot image. In case of VPN providers the motivation is probably to prevent (permanent) logging of sensitive information.
jeffbee|3 years ago
vbezhenar|3 years ago
dang|3 years ago
Submitted URL was https://mullvad.net/en/blog/2022/12/30/review-of-2022/, which is a list, and that item looks like the most interesting thing on this list. (see https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... for why we do this sometimes)
latchkey|3 years ago
This generally works well, but I'd say there are about 0-20 blades that crash a day due to some sort of memory corruption issues.
Due to the fact that I was operating remotely from the hardware, I never really got a chance to resolve it... also... just a simple reboot would fix it (and the blades booted in ~60 seconds, so it wasn't a huge issue).
So, on large enough scale... this can be an issue to consider.
yjftsjthsd-h|3 years ago
siliconc0w|3 years ago
ignoramous|3 years ago
I don't know what the threat model is, but if it involves nation states confiscating servers, then diskless is of limited help: https://en.wikipedia.org/wiki/Cold_boot_attack
> If the computer is powered off, moved or confiscated, there is no data to retrieve.
Oh wait...
Mave83|3 years ago
It provides so many benefits and eases the server management greatly.
generalizations|3 years ago
Can you elaborate on this? I would have thought that needing local storage cache and the potential for network latency would make PXE untenable.
zppln|3 years ago
Semaphor|3 years ago
l2silver|3 years ago
patrakov|3 years ago
dang|3 years ago
RVRX|3 years ago
[1] https://www.amazon.com/Mullvad-VPN-Devices-Protect-Security/...
andrewmunsell|3 years ago
input_sh|3 years ago
Those resellers then charge about the same price as Mullvad, but get to keep a piece of that as profit.
ronsor|3 years ago
warinukraine|3 years ago
However, what makes them great and unique is that they're ideologically motivated, so of course they're not selling shares.