No-reboot patching comes to Linux 4.0

[matteotom]

More info here: http://lwn.net/SubscriberLink/634649/d31644d65227ead6/

TD;DR: There are still issues to work out, and working live patching might not be in for 4.0.

> "So, while it is still possible that the kernel will have an essentially complete live-patching feature by the end of the year, it may happen rather closer to the end of the year than the developers involved might have hoped for."

[justincormack]

Live patching has its serious critics, and it is also not nearly finished "I think they are fundamentally misguided in both implementation and in design, which turns them into an (unwilling) extended arm of the security theater." [1]

[1] http://lwn.net/SubscriberLink/634649/f6a2aa1f39c538b7/

[azakai]

Thanks for the link! Reading that, it looks pretty pessimistic actually. There seems to be nowhere near a clear path forward for how to actually do this.

One concern is that stack traces are "best-effort" and not guaranteed to work. Isn't there a way to force them to work, by disabling some stack-related optimizations (like no-omit-frame-pointer)? I imagine there would be a little slowdown, but maybe worth it for the ability to do quick stacktrace-based live patching?

[yungchin]

Bizarre to think that at the time nobody stopped Oracle from picking up Ksplice (they offered rebootless security patches for all the larger distros back in what, 2009?). I really loved their service because it meant I could keep my half million or so terminals in gnu screen open for months.

(Also, their blog was great - see eg this really scary proof-of-concept of what you can do when you know how to live-patch a kernel https://blogs.oracle.com/ksplice/entry/hosting_backdoors_in_... )

[mmagin]

Dare I ask how having half a million terminals open in screen is useful in any way?

[ianlevesque]

What approach did ksplice take?

[PythonicAlpha]

But the Kernel is not the only reason, a system has to be restarted. What about changed dynamic libraries? It can be a mess to fiddle out, which processes has to be restarted, so a reboot is normally the best solution for critical patches (eg. in the glibc).

[hobarrera]

I agree. Especially stuff like init, or non obvious non-trivial system services (and, of course, in the case of these distros, systemd).

[ams6110]

One reason to love Linux on your servers or in your data-center is that you so seldom needed to reboot it. True, critical patches require a reboot, but you could go months without rebooting.

Doesn't seem so lately. I haven't actually plotted it but seems like kernel security updates drop every few weeks these days.

[riffraff]

I wonder if it might be an issue with recent kernels (in the sense of not being out for a while).

I recall running some form of "stable" red hat without rebooting for very long time years ago, while my ubuntu LTS from last year has a "System restart required" MOTD every other week.

[greglindahl]

Is there any solution for processes using obsolete libraries? It's a pain to figure out how to restart all of them without rebooting.

lsof | grep lib | egrep '(DEL|deleted\))'

[eudemo]

If you are on Debian, you could try checkrestart

[ciupicri]

Run needs-restarting on Fedora.

See also https://rwmj.wordpress.com/2014/07/10/which-services-need-re...

[bradfordboyle]

You could check out `needrestart`---I've used this on Ubuntu and Debian, so I can't vouch for other distros.

https://github.com/liske/needrestart

[drzaiusapelord]

What does this mean on the VPS front? Will EC2, Rackspace, and Linode be able to patch Xen or KVM without rebooting all their clients now?

[dezgeg]

This support is for the Linux kernel only. So if the Xen hypervisor itself needs to be updated, it's reboot time.

[rodgerd]

My understanding of all the kernel splicing technologies is that it will depend heavily on the nature of the patch.

[Thaxll]

Not really since the libraries remain affected.

[twic]

This seems like a bit of a smell. The desire for no-reboot patching stems from the fact that reboots are painful. But since hardware and software failure is an inevitable fact of life, it seems like it would be better to make reboots less painful, rather than try to make them less frequent.

This is the idea underlying crash-only software:

http://lwn.net/Articles/191059/

[bigbugbag]

Ironic while the kernel gains the ability to patch without rebooting as the same time, systemd makes it mandatory to reboot after each update.

[the_mitsuhiko]

Why does systemd make a reboot necessary?

[seccess]

This seems like it will be really useful for servers---I wonder how much it will matter for desktops though. Logging out and logging back in seems like a requirement for xorg, groups, etc. While it isn't a reboot, it still kills my windows and thus my workflow.

[atonse]

I've always wondered what something like CoreOS uses to "switch" kernels in a live system - is that also something like ksplice or a different technology?

[Titanous]

I believe that CoreOS can use kexec to update the kernel. kexec basically looks the same as a reboot except instead of stopping fully, and going through the BIOS/bootloader again (which can take a long time on server-grade hardware), the old kernel 'execs' the new kernel and boots userspace back up.

https://en.wikipedia.org/wiki/Kexec

[rckclmbr]

How do you mean? I thought CoreOS required reboot https://coreos.com/docs/cluster-management/setup/update-stra...

[jonalmeida]

I remember talking to a rep at Morgan Stanley that was explaining to me that you can perform patches on kdb as well so that you have zero downtime. Although, I believe that was implemented a bit differently where you have a compiled program and the patch that is read by an interpreter. I didn't get down to learning how it would eventually take the interpreted patches to be part of the compiled binary.

[xroche]

For those interested, here's a very nice presentation on kGraft by Jiri Slaby (Suse): https://kernel-recipes.org/fr/2014/patcher-le-noyau-en-temps...

Note: the page is in French, but both the presentation and the slides are in English.

[dantillberg]

This article's title really ought to include a hyphen: "No reboot patching" reads very differently from "No-reboot patching". The former suggests that this feature ("reboot patching") did not make it into Linux 4.0, while the latter accurately suggests that the feature called "no-reboot patching" is (edit: or may be) included in Linux 4.0.

[sctb]

Thanks, we updated the title.