[saynay]

On the other hand, I had the misfortune of having a hardware failure on one of my Hetzner servers. They got a replacement harddrive in fairly quickly, but still complete data loss on that server, so I had to rebuild it from scratch.

This was extra painful, because I wasn't using one of the OS that is blessed by Hetzner, so it requires a remote install. Remote installs require a system that can run their Java web plugin, and that have a stable and fast enough connection to not time out. The only way I have reliably gotten them to work is by having an ancient Linux VM that was also running in Hetzner, and had the oldest Firefox version I could find that still supported Java in the browser.

My fault for trying to use what they provide in a way that is outside their intended use, and props to them for letting me do it anyway.

[jwr]

That can happen with any server, physical or virtual, at any time, and one should be prepared for it.

I learned a long time ago that servers should be an output of your declarative server management configuration, not something that is the source of any configuration state. In other words, you should have a system where you can recreate all your servers at any time.

In your case, I would indeed consider starting with one of the OS base installs that they provide. Much as I dislike the Linux distribution I'm using now, it is quite popular, so I can treat it as a common denominator that my ansible can start from.

[kro]

They allow netbooting to a recovery OS from which the disks can be provisioned via an ssh session too, for custom setups. Likely there are cases that require the remote "keyboard", but I wanted to mention that.