[amiga386]

Lovely video, going into almost everything...

...and yet, zero mention of systemd's recommendation for programs to link in the libsystemd kitchen sink just to call sd_notify() (which should really be its own library)

...and no mention of why systemd felt the need to preemptively load compression libraries, which it only needs to read/write compressed log files, even if you don't read/write log files at all? Again, it's a whole independent subsystem that could be its own library.

The video showed that xz was a dependency of OpenSSH. It showed on screen, but never said aloud, that this was only because of systemd. Debian/Redhat's sshd [0] was started with systemd and they added in a call to the sd_notify() helper function (which simply sends a message to the $NOTIFY_SOCKET socket), just to inform systemd of the exact moment sshd is ready. This loads the whole of libsystemd. That loads the whole of liblzma. Since the xz backdoor, OpenSSH no longer uses the sd_notify() function directly, it writes its own code to connect to $NOTIFY_SOCKET. And the sd_notify manpage begrudgingly gives a listing of code you can use to avoid calling it, so if you're an independent program with no connection to systemd, you just want to notify it you've started... you don't need to pull in the libsystemd kitchen sink. As it should've been in the first place.

Is the real master hacker Lennart Poettering, for making sure his architectural choices didn't appear in this video?

[0]: as an aside, the systemd notification code is only in Debian, Redhat et al because OpenSSH is OpenBSD's fork of Tatu Ylönen's SSH, which went on to become proprietary software. systemd is Linux-only and will never support OpenBSD, so likewise OpenBSD don't include any lines of code in OpenSSH to support systemd. Come to think of it, "BSD" is another thing they don't mention in the script, despite mentioning the AT&T lawsuit (https://en.wikipedia.org/wiki/USL_v._BSDi)

[rwmj]

When I was being interviewed, we did talk about exactly this, including that libsystemd is a kitchen sink, and that eventually OpenSSH went with open-coding the equivalent to sd_notify instead of depending on libsystemd. (Also that ahem Red Hat added the dependency on libsystemd in a downstream patch oops).

However the editors (correctly IMHO) took the decision to simplify the whole story of dependencies. In an early draft they simplified it too much, sort of implying that sshd depended directly on liblzma, but they corrected that (adding the illustration of dependencies) after I pointed out it was inaccurate.

I agree with everything you say, but you have to pick your battles when explaining very complicated topics like shared libraries to a lay audience.

In general I was impressed by their careful fact checking and attention to detail.

Sadly they missed the misspelling (UNRESOVLED) even though I pointed it out last week :-( But that's literally the only thing they didn't fix after my feedback.

[dralley]

It did get mentioned - in the context of the upstream change to dynamically load those libraries being a threat to the hack's viability which may have caused "Jia Tan" to rush and accidentally make mistakes in the process.

[amiga386]

They say "an open-source developer requests to remove the dependency that links xz to OpenSSH" while showing https://github.com/systemd/systemd/pull/31550 on screen, zoomed and focused so the word "systemd" does not appear.

They never once utter the word "systemd", anywhere in the script... isn't that strange for such a key dependency?

[mayama]

From my vague memory of xz backdoor, I don't even recall systemd being involved. Now, I get what people are talking about when they said systemd is taking over everything and why there was so much pushback to systemd when it was being added to distros. For me as a end user/dev, it mattered little whether services were started by systemd, openrc etc.

[amiga386]

systemd was the key to the whole backdoor.

OpenSSH is maintained by the OpenBSD developers. OpenSSH does not use liblzma (xz) at all.

Linux distros which chose to switch to systemd also chose to patch OpenSSH to call systemd's sd_notify() function, to inform systemd when sshd is fully started.

This sd_notify() function is in the huge, sprawling kitchen sink of a library called libsystemd. sd_notify() is only a few lines of code, but it's convenient (to Linux distro packagers) to make systemd a dependency of OpenSSH, link in the whole library and call that one function. It makes their patches of the upstream software smaller and easier to review for correctness.

In the sprawling libsystemd is an entire subsystem for reading/writing systemd's famous binary log files, and the user can choose compression (xz, zstd or lz4). It depended on and loaded all three of these compression libraries, whether you read/write compressed logs or not. In the video you hear about the imminent request to load these libraries dynamically on demand -- https://github.com/systemd/systemd/pull/31550 -- but this arrives many years adding these functions to the libsystem kitchen sink, and generally speaking most programs shouldn't use the libsystemd functions for reading/writing log files, they only need to send log messages to journald via syslog() or sd_journal_print()

So you can see this unwarranted dependency chain was introduced by Linux distros adding systemd to everything, and nation-state level hackers saw and tried to exploit it, seeking out the xz maintainer for social engineering.

[rwmj]

libsystemd was the indirect dependency that caused liblzma to be pulled into sshd.

[bigbadfeline]

> Is the real master hacker Lennart Poettering, for making sure his architectural choices didn't appear in this video?

systemd is doing what it was designed to do... Cute videos are doing what they were designed to do too - hiding that!

> OpenBSD don't include any lines of code in OpenSSH to support systemd. Come to think of it, "BSD" is another thing they don't mention in the script

And this!