Takeover.sh – Wipe and reinstall a running Linux system via SSH without reboot

[gizmo]

Pretty cool, although I'm pretty sure I would never use something like this.

What has saved my skin on a number of occasions is the ability to boot remote servers into rescue mode and chroot into the broken system. That way you can use package managers, all your diagnostic tools, and everything else the boot image doesn't provide.

Basically you just mount the different partitions and then chroot just swaps /proc /sys /dev of the rescue image with the real ones, and BAM you're back in business.

For details see threads like: http://superuser.com/questions/111152/whats-the-proper-way-t...

I know that for many of you this isn't rocket surgery, but for those who don't know you have to google for "chroot" when you boot into a rescue image and discover you can't do anything, you might just remember this post.

[shocks]

Haha yes, this is the default install procedure for Gentoo! :D

[smudgymcscmudge]

This is one of those things you want to have practiced before it's critical.

[rijoja]

I like the term rocket surgery, I will steal it from you for sure.

[HeadlessChild]

I work at an IT Helpdesk and we build a custom GRML image which we write to our pen drives. We use it, for example, to quickly live boot Debian/Ubuntu PC's and run a script that mounts all necessary file systems (including pseudo file systems). This makes it really easy to quickly chroot into a broken system right at a users desk.

On our Linux installations we have separated most of the directories into different LVM partitions. Such as '/home', '/usr', '/var' and of course '/' (the root). Just last week I helped a user live boot his system and ran a fsck on his '/home' partition. We had "frozen" the system and hard reset his computer which led to a couple of corrupt blocks and inodes.

[lisivka]

I recommend to copy a rescue disk iso image or/and network installation image to boot partition and add it to GRUB menu. This way it is possible to boot into working environment without risk to damage root partition.

https://wiki.archlinux.org/index.php/Multiboot_USB_drive

[predakanga]

For anyone interested in adding this to their toolkit, I would suggest reading this StackOverflow answer: http://unix.stackexchange.com/a/227318/189858

In short, the answer details how to switch your running system to use an in-memory only root filesystem, without restarting. This allows installing a new OS, resizing the OS disks, etc.

It's a risky operation, but the linked answer covers many pitfalls that you might run into - I recently used it to shrink the root partition on a remote server, very much appreciated the detail.

[aaronmdjones]

I've been using this procedure to remotely replace operating systems for years. The most common scenario is a VPS provider that doesn't give you the choice of OS you want.

Infact, I will be using it later today, to replace a Debian system with Gentoo, no less. The README in OPs link is spot-on here (the last few paragraphs).

[drvdevd]

Thanks for this! Ive been doing wacky stuff like kexec which was completely unecessary when I could've just done a pivot_root...

[rdslw]

Another nice trick of this family (with reboots, or without with using systemd-nspawn) lies with clever btrfs usage. Long story short:

* use btrfs, and create your main root filesystem as a btrfs partition subvolume and another btrfs subvolume for snapshots (also a sub of master btrfs partition)

* to start any experiment (e.g. installing whole gnome and 500 different packages you MIGHT WANT TO REVERT in the future) create before the risky operation snapshot (btrfs subvolume snapshot / /.snapshots/yournameofsnap) of current filesystem

* experiment in any way :)

* switch between old root (snapshot you created) or the new one with (btrfs subvoulme set-default)

* delete any of them (btrfs subvolume delete)

btrfs copy-on-write allows all of these commands to happen instantly without (almost) any actual copying. Also booting from both volumes is possible without any additional steps as long as master btrfs partition is the one to be booted from UEFI.

https://wiki.archlinux.org/index.php/Btrfs

[floatboth]

Congratulations, you've reinvented ZFS Boot Environments :)

[notaplumber]

This sounds similar to the more cleverly named FreeBSD Depenguinator project which could be written over top of a remote Linux server replacing it with FreeBSD, without console.

If you have remote console access, a similar thing can be done for OpenBSD by dd(1)'ing a miniroot ramdisk install image.

[camtarn]

Seeing if I understand what this is doing: this keeps running the same Linux kernel and kernel modules, but swaps out absolutely everything else up to and including the init system - is that right?

[efficax]

More precisely, what it does is start a new init system on a pseudofilesystem, starts an sshd chrooted to that new system, and lets you then login there, where you can then, if you want, umount the original root fs, wipe it, install a new os, then reboot the system.

[simon1573]

I guess this could be really useful for installing distributions that are not available at some VPS providers.

[tyingq]

Another trick to do that is to use the VPS's rescue mode, download qemu, and start it with the remote desktop / VNC console. Then, do the install from there. Poor man's IPMI.

[tomsmeding]

There is a similar script on github somewhere that does some more work and can install arch over debian on a DigitalOcean host. It's beautiful.

[ce4]

Reminds me of Debian Takeover from more than 10 years ago :-)

https://wiki.debian.org/DebianTakeover

[zimbatm]

https://github.com/elitak/nixos-infect is similar but doesn't require to pivot root.

[Aissen]

FYI, there's vps2arch that does the same thing with a different approach: https://github.com/drizzt/vps2arch

Edit: it doesn't really do the same thing. vps2arch could be implemented on top of takeover.sh for better reliability.

[nashashmi]

Somebody correct if I am wrong, but this script somehow allows the session to live in the RAM. Once the OS is running directly from the RAM, the hard drive can be wiped and a new OS can be installed. The system is then booted to run off of the hard drive.

[VoidWhisperer]

I feel like this is, on one hand, an amazing thing to be able to do because it removes the necessity to have IPMI to remotely re-install a system.

On the other hand, this seems like it would be an incredibly easy thing to screw up and potentially leave yourself with a corrupted or unbootable system.

[dredmorbius]

Which is fundamentally how an installer works already.

Look up the chroot-install method for Debian/Ubuntu, as an example of this in a user-accessible manner. The process uses debootstrap, but as I understand was originally developed via a chroot shell on a running system and either partition or ramdisk-based bootstrapping of a new system.

[lisivka]

This scripts creates another root, then substitutes /sbin/init using `mount --bind`, then reloads /sbin/init, which pivots into new root with fakeinit.

IMHO, systemd in container mode can be used instead of fakeinit.

[dredmorbius]

This is conceptually similar to the chroot installation method, which has been a documented, if not entirely standard, method on Debian for quite some time.

https://www.debian.org/releases/stable/amd64/apds03.html.en

https://wiki.debian.org/chroot

[yellowapple]

It's also a common installation method for the more "advanced" distros. The "SAG Trifecta" (Slackware/Arch/Gentoo) uses a lot of chrooting in its installation procedures.

What's interesting about takeover.sh, though, is that it goes a step further and causes the chrooted system to actually replace an existing OS without a reboot and (theoretically) without involving additional boot media.

[geoffmcc]

I wonder if this would help me switch from Ubuntu Desktop to Ubuntu Server on my laptop that has a broken screen.

[pipo098]

you can always connect an external screen and go from there

[NGTmeaty]

Holy shit, that's really cool.

[technologyvault]

Wish I had known about this hack before today, even if it is just experimental at this point.

[dpweb]

removed. No delete on HN comments? interesting

[bjacobel]

I think you meant to comment on the GitLab postmortem thread.

[rocky1138]

You can delete comments but only up until a certain amount of time has passed, I think. I've been able to do it.

[andruby]

Where does this takeover.sh readme mention anything about missing pg_dump files?