Back in 2016, systemd started killing user processes on logout (rather than send them the SIGHUP signal, as POSIX says should happen). This caused problems for programs like nohup, screen and tmux, which deliberately keep running. Systemd's response was to say that they should incorporate systemd's library, and use systemd's new daemonization API. As far as I know, none of them did.
Two years later, you can find hundreds of support requests across the internet, from frustrated users who are having their sessions killed by systemd.
Bugs are annoying, but that's life. On the other hand, when you're an impacted user who's lost work, and researching the bug leads you to a years-old discussion in which someone is actively denying that the bug exists and refusing to fix it, that's infuriating. I don't think systemd's developers deserve the trust that maintaining a core piece of infrastructure requires; they don't seem to care enough about whether they've broken things.
You know, because we knew this would be controversial we made sure it was both a compile-time option and a runtime option. Yes the upstream default of both defaults to on, but that's just upstream. We made it very easy and supported for downstream distros to switch between opt-out and opt-in of this option for their users. We have encouraged distributions to leave it on, but we were fully aware that for compatibility reasons this is something downstreams likely wanted to turn off, and most compat-minded distros did, as we expected.
Now I am used to taking blame for apparently everything that every went wrong on Linux, but you might as well blame your downstream distros for this as much you want to blame us upstream about this, as it's up to them to pick the right compile-time options matching their userbase and requirements in compatibility, and if they didn't do that to your liking, then maybe you should complain to them first.
(And yes, I still consider it a weakness of UNIX that "logout" doesn't really mean "logout", but just "maybe, please, if you'd be so kind, i'd like to exit, but not quite". I mean, that's not how you build a secure system. We fixed that really, fully knowing it would depart from UNIX tradition, but that's why we made it both compile-time and runtime configurable)
(Also, nobody has to "incorporate" systemd's library to avoid the automatic clean-up. In fact, there's no library we provide that could do that. What was requested though is to either run things as child of systemd --user or just register a separate PAM session, neither of which requires any systemd-specific library.)
Absolutely. I can understand implementing this feature for some special cases, like containers that should clear all hint of a user away on log off. It should never have been the default, and breaks an entire category of software. In my standard .bashrc file, I have the following snippet to warn me if I am on a system with that stupid setting enabled.
if which loginctl > /dev/null && loginctl >& /dev/null; then
if loginctl show-user | grep KillUserProcesses | grep -q yes; then
echo "systemd is set to kill user processes on logoff"
echo "This will break screen, tmux, emacs --daemon, nohup, etc"
echo "Tell the sysadmin to set KillUserProcesses=no in /etc/systemd/login.conf"
fi
fi
Systemd developers, if you're reading this: this isn't the sort of bug where people grumble for awhile and then get over it, because things are still broken, and the workaround being circulated (KillUserProcesses=no) doesn't fully work. (https://github.com/systemd/systemd/issues/8486) As long as people continue to encounter this issue anew--and they still are--people will be angry at the systemd maintainers.
That comes down entirely to how systemd is configured. If you don't like what your chosen distro has picked as the default then complain to them. systemd didn't force anyone's hand on the subject, they just added the feature. It's a pretty natural design choice IMHO. When I want to log out, I don't want to let some hung up daemon keep running just because it wasn't able to process the SIGHUP sent to it.
How else do you propose to make sure that when I log off my ssh-agent is really terminated and not just locked up with my keys still in memory? The POSIX approach is insufficient, there's no way to know if a process received a signal and chose to ignore it and keep running or if it received a signal but it was deadlocked and kept running.
I agree that on Linux-based systems, SIGHUP is a reasonable mechanism for killing processes when a user closes an ssh session, and that ignoring SIGHUP is a reasonable way to avoid getting terminated.
I disagree that POSIX says that processes should expect a SIGHUP when a user logs out (SIGHUP means the controlling terminal was closed). I am not at all a POSIX expert, so please correct me if I misunderstand, but afaict POSIX explicitly does not specify what happens to the controlling terminal when a user logs out (http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_...):
> POSIX.1 does not specify how controlling terminal access is affected by a user logging out (that is, by a controlling process terminating). 4.2 BSD uses the vhangup() function to prevent any access to the controlling terminal through file descriptors opened prior to logout. System V does not prevent controlling terminal access through file descriptors opened prior to logout (except for the case of the special file, /dev/tty). Some implementations choose to make processes immune from job control after logout (that is, such processes are always treated as if in the foreground); other implementations continue to enforce foreground/background checks after logout. Therefore, a Conforming POSIX.1 Application should not attempt to access the controlling terminal after logout since such access is unreliable. If an implementation chooses to deny access to a controlling terminal after its controlling process exits, POSIX.1 requires a certain type of behavior (see Controlling Terminal ).
There is no NOHUP signal, you're referring to SIGHUP.
See the enable-linger option for loginctl and KillUserProcesses for logind.conf. KillUserProcesses was set to default enabled on 4/9/2016, prior to that it didn't happen, but was configurable if desired. So you were always able to change the config to restore the previous behavior from the moment the default turned it on.
Eleven years earlier when SMF was added to what would eventually be Solaris 10, we had this same problem. Some of us had to drop everything to fix "bugs" in cron, sshd, ... introduced by SMF.
Systemd is basically SMF, done poorly, because NIH.
Systemd's response was to say that they should incorporate systemd's library, and use systemd's new daemonization API.
By "use systemd's new demonization API" you mean, instead of
$ screen
systemd asks you to write
$ systemd-run --scope --user screen
instead. Annoying to have to learn a new thing, but hardly the unbearable burden.
On the other hand, when you're an impacted user who's lost work, and researching the bug leads you to a years-old discussion in which someone is actively denying that the bug exists and refusing to fix it, that's infuriating.
Because it's a bug for some, and intended behavior for others. Look, you make it as if they introduced a bug on purpose to screw with some people. It's clearly not the case, there was a specific tradeoff involved.
> This argument, he said, seems to be predicated on the notion that systemd is a single, monolithic binary.
Can we please stop misrepresenting the complaints against systemd? The only time I ever hear this "monolithic binary" argument is from systemd advocates. The actual complaint is about tightly coupling important features together. Not only does this make it difficult (often impossible) to replace individual components, when tight coupling happens at the (internal) protocol level, any replacement component ne4cessarily hast to implement a bunch of (sometimes unwanted) systemd baggage.
Busybox implements all of its features in single monolithic binary, but it isn't a monolithic design that tightly couples those specific components together. Replacing one of busybox's components is often as simple as removing busybox's symlink and installing the replacement. This isn't even a "Unix philosophy" issue. Even inexperienced designers shouldn't have as hard time Understanding why systemd is a monolithic design but busybox isn't.
https://suckless.org/sucks/systemd/ has items like "pid 1 does DNS". It's an incorrect complaint that exists in the wild, though it certainly isn't the basis of all accusations that it violates the Unix philosophy.
> "It's software" so of course it's buggy, he said. The notion that systemd has to be perfect, unlike any other system, raises the bar too high.
systemd is a PID 1 program, it means it have to raise bar higher. When troubles begin, you would need tools to fix them, and if PID1 is crashed, you are out of luck. If system cannot boot into shell, you'd need to fix it from initrd shell. Or to boot other system, to fix this one. It sucks.
Linux kernel chases very high standards of reliability, because when kernel panics it is even worse than PID1 crash. Init system should follow the same standards as linux.
Have you ever had pid 1 (systemd or any other init) crash? For the last ~three years I've been paid to maintain high reliability algorithmic trading systems that ran systemd and a whole lot of other stuff, and systemd has never crashed on me. Lots of other stuff, including the kernel itself, has crashed.
The bar is higher for pid 1 - if I were designing systemd I would have made a tiny pid 1 that just did message-passing to a more complex secondary process that could be restarted, or something, just to be safe - but I think systemd has empirically cleared the bar.
This article is a bit of a joke "It's software so of course it's buggy" isn't a great argument when you're replacing something that didn't suffer the same issues.
I just count my blessings that runit is widely packaged in every major distro because it can just happily sit on top of sysvinit, systemd, upstart, pretty much any init system and does things in a very simple shell script style, I really wasn't a fan of the weird ini-like format for systemd or several different tools I'm expected to learn just to read my (now binary) log files competently.
If you're sick of switching init systems constantly or don't want to have to write separate scripts for your linux box and your freebsd box even, I highly recommend checking runit out.
I'm sure I'll give it a serious shot eventually... in about 3 years once they work all the Poettering kinks out, just like PulseAudio. They're doing some cool things with cgroups and stuff, so I hope it gets there eventually.
I second using runit. We use runit to be able to use the same service definitions inside docker, on a VM or bare metal.
If you've ever tried to use systemd inside docker to bring up a couple of services, you would know the hoops you have to jump through to get it working.
(I understand that docker wasn't invented to run multiple services in the one container, but sometimes it can't be avoided and simplifies app deployment vastly I.e, using CI to test your service actually starts up as per its definition: just run up a quick docker image with runit and a service definition file)
> When Lennart Poettering started thinking about the problem, he first looked at Upstart, which was an event-based system that was still based around scripts, but he concluded that he could do a better job.
I think I'd like systemd more if I had more confidence that he had done a better job.
I never really understood why Upstart didn't get more traction; was it just the Canonical backing made other distros avoid it, or did it have drawbacks I never ran across?
I used upstart deeply around 2012-2014 in a commercial product based on Ubuntu. I remember two problems with it which had me waiting for Ubuntu to switch to systemd (unfortunately my employer ran out of money right when that happened). The first was that Upstart's model is upside-down: where systemd has units that depend on units, Upstart has jobs that are activated as other jobs complete. Whereas you would tell systemd that (say) Apache depends on syslog and ask it to try to start Apache, and therefore systemd would first try to start syslog, you tell Upstart that Apache can be started when syslog starts, and that you should start syslog, and then Apache's dependencies are met and Apache can start too. Instead of having a single target (like sysvinit runlevels or systemd's multi-user.target) that it works towards, Upstart just starts ... stuff ... untill it's done. Upstart did have a runlevel concept, but as I recall it was an event that triggered jobs to start, not a target. See http://upstart.ubuntu.com/cookbook/#critique-of-dependency-b... for Upstart's defense of this approach.
(systemd is hardly unique in being dependency-based instead of event-based - but it is a strike against Upstart in particular. Also, while one of the stronger claims of why Upstart should work this way is it matches the evented mindset of udev, systemd can handle events from udev just fine, by just having it add additional targets - and in the end, systemd consumed udev.)
The other was that it was hard to deal with its bugs. I had a job that wasn't activating. I spent days chasing it, including attaching a debugger to Upstart, and eventually gave up. I've used systemd extensively and have not had "Why isn't this service starting" bugs. (I've certainly had that confusion, but I've always been able to figure it out quickly.) systemd has in my experience been reliable and trustworthy. I'm not going to say that it's bug-free, or that I like its implementation, or that I have no complaints about it. I am going to say I haven't been frustrated by it - even when running very old distro versions with lots of known bug fixes upstream. That's very important for whether I'm happy with it or whether I'm going to switch back to sysvinit and find something else for service monitoring.
The problem is full process management, and in that regard, upstart was pretty much still just scripts. I will say this, systemd tries to solve the problem of tracking all children/grandchildren which is difficult to do with sysvinit or scripts by themselves.
Systemd also solves the problem of unified ways to start services across many distributions. Packaging things is easier. I don't like the systemd ini target format and wish something else had won out instead. With something like upstart, in theory you could rewrite the upstart daemon and use the same scripts, since the format/standard was much simpler.
I feel like there must have been a lot of bullying to get systemd across so many systems. It just seems weird to see so many major distros all decide to accept something so complex if it wasn't for Redhat cramming it into everything they maintained and helped fund.
I think it largely was Canonical that was the issue. Every time they've pushed something that they designed/developed it always seemed to me that they were more attempting to solve their business model problem rather than a Linux architectural problem. (i.e. get people dependent on them rather than just providing a solution)
Even though Red Hat is a much larger organization, their contributions have never felt like power plays in the same way that Canonical's have. Perhaps it's just the way that Canonical tries to shove their stuff through while Red Hat seems to get more organic buy-in. For example, as controversial as Systemd was (mainly from sys admins and power users), it did have a level buy-in from distro maintainers and developers which Upstart never really did. Canonical just tried to say 'we're doing this.'
Scott James Remnant
+
4
1
2
1
Reply
+Michael Hasselmann at the point that Kay, Lennart and
I sat down and discussed all this stuff, I don't think
Upstart was perceived as "shitty" at all. We'd had
on/off discussions for ages, but the big one I remember
was the LF Collab Summit in SF in April 2010.
Hindsight certainly lends a different perspective, and
I'd be the first person to say that Upstart doesn't
work as intended. +Lennart Poettering makes a great
point about mountall in a recent post, it was written
because Upstart couldn't do the complex filesystem
cases it was designed to be able to do; and I was very
aware even at the time that was a failure that would
need to be addressed.
Had the CLA not been in place, the result of the LF
Collab discussions would have almost certainly been
contributions of patches from +Kay Sievers and Lennart
(after all, we'd all worked together on things like
udev, and got along) that would have fixed all those
design issues, etc.
But the CLA prevented them from doing that (I won't
sign the CLA myself, which is one reason I don't
contribute since leaving Canonical - so I hold no
grudges here), so history happened differently. After
our April 2010 meeting, Lennart went away and wrote
systemd, which was released in July 2010 if memory
serves.
So I don't think I can claim that the perceived
shittiness of Upstart spawned systemd, because at the
time it wasn't seen that way. I don't think I can even
claim that it provoked Lennart in any way, init was an
area all distributions were fiddling with, so it was
inevitable anyway.
I entirely agree with Kay and +Greg Kroah-Hartman that
it was the CLA that caused systemd to be written
instead of Upstart.
But I don't need that self-affirmation anyway :) I
wrote Upstart, I got paid for it, I moved on to do
other things, something else came along and replaced
it. If Upstart hadn't been under the CLA, and systemd
hadn't've happened, all my code would have long since
been rewritten by now anyway.
That's the nature of the software world, there's no
point getting precious over things. Do your bit, have
fun doing it, move on and let others do their bit,
etc.
One of my "favorite" corner cases caused by systemd was that if an NIS user tried to login to the system, it would cause my X session to restart:
1. They changed the defaults for systemd-logind to disallow network access
2. When systemd-logind attempts to send a message to the NIS server, it instead hangs (or has a timeout greater than the systemd watchdog anyways)
3. The watchdog timer causes all systemd-logind services to be restart, which then tears down all of the child processes. My X session is a child process.
However, I believe the implementation is not great.
A couple specific observations:
* lots of binaries. /lib/systemd, the logs. This more than anything else is a
tragedy. Binaries get in the way of viewing, diagnosing, understanding and
changing things. It seems like premature optimization to me - I don't think
speed requires it. I think the majority are completely unnecessary.
* copying from launchd? Commercial software vendors ship binaries. They
are aligned with them. They resist disclosure, which protects from reverse
engineering, lawsuits and user modifications. Does that align with linux?
* the config files are a disorganized mess. It is like /etc/init.d but poorly
understood and then /etc/rc.d was lumped into the same directory and organization
became chaos. I looked at one system here and in /etc/systemd there are 16 directories,
23 symbolic links and just 11 files. Try unraveling the interdependencies.
additionally the config files (which mimic windows config files) lack depth.
Simple config files can have benefits, say if a GUI read and wrote them to change
settings, but I don't think that applies here. What I noticed is that they
cannot do anything, so additional logic requires an intermediate script or
compiled binary elsewhere.
* it is responsible for too many functions. I bet it didn't solve so many in
its first iteration, but sucked in more over time. It reminds me of the
accounting program that grew to rule everything in Tron.
I wonder if these problems could be resolved. It's hard to remove complexity
and add elegance later.
I hate how the Unix philosophy argument is boiled down to "one vs many" binaries. It's about independence to the degree of reusability. There's nothing in systemd suite that's useful unless you're two feet in. You can say things about streams and text and binary formats, but those are distractions. Systemd tooling is tightly coupled to each other. It's a huge codebase, a huge system that shoddily replaces that which was historically done with less. And it's the biggest reason I don't run a Linux workstation anymore.
systemd feels like it is well thought out and modern, like it's an integrated rethink and rebuild of lots of various software utilities that have evolved over many years.
I'm investing all my learning in systemd where appropriate instead of things I used previously like cron and supervisor.
Some years ago, I was developing a Linux distro for embedded systems. Some of these embedded systems come with old kernels, and you will never get a recent kernel for them.
Systemd is written in C and relies on kernel headers. Which means that they need to have a layer that deals with different versions of the kernel headers. I remember it was headers about network.
At the time I was working on this, I had to contribute to systemd to add support for my kernel version. Got a few patches accepted in their master. And things were working fine for a while.
Until they suddenly decided to drop <= 3.10 kernel support. Yep.
Afterwhat I had to maintain patch sets for systemd, which is a great amount of job, given that my upstream was updating systemd quite often.
Now my good people, explain me how such a mess could have happen with a previous init based on shell?
It is my opinion that Lennart Poettering is the software equivalent of Thomas Midgley Jr., the inventor of both leaded gasoline and chlorofluorocarbons.[1]
This is the person behind PulseAudio, Avahi, and systemd. In any sane world, these software projects would have been stillborn. Sensible alternatives would have outcompeted them. Instead, Red Hat did their best to foist them upon the community. They promised that after the initial adoption, all the wrinkles would be ironed out. All three times this was not true.
I hate to appear so cynical, and I'm certain that Red Hat didn't intentionally pursue this strategy, but it sure is convenient for them that the harder Linux is to use, the more they can sell support contracts. Not so convenient for people who actually want to use Linux.
> It is my opinion that Lennart Poettering is the software equivalent of Thomas Midgley Jr., the inventor of both leaded gasoline and chlorofluorocarbons.
Considering that Midgley Jr. is (indirectly) responsible for the suffering and potentially death of many humans and other beings, I submit that this (in spirit) triggers Godwin's Law. The debate is therefore over.
>Sensible alternatives would have outcompeted them.
Of course, and they didn't, because there were no sensible alternatives on linux. All these projects were clones of decently engineered macos projects, before they were created, linux people used some truly silly staff like sysvinit.
I would use something better than pulse and systemd, but there is nothing better available.
Which alternative to PulseAudio, Avahi, and Systemd exists for Linux that provide the same features?
I'm not aware of any audio daemon that is as feature rich and flexible and PulseAudio. None of the alternatives fit the same general purpose use case that is expected in a modern desktop that needs to compete with Mac OS and Windows 10.
You might be giving Lennart too much credit. As the author of the article points out, Lennart didn't invent PulseAudio and systemd so much as he ripped them off (poorly) from coreaudio and launchd. To compare him to any inventor seems too flattering.
I hate systemd with a passion, however I understand the reason for it. I think that systemd is a bit Franklinstien by the nature of having to pull udev and d-bus into it. So I guess it is a victim of having to deal with the system it is on. Oh well.
I think the only Unix system to get this right was likely Solaris via SMF.
systemd reminds me of the efforts in some cities at clearing 'slums' to replace them with 'modern' apartment blocks.[1]
sytemd seems like a similar technocratic, modernist effort aimed at UNIX.
Operating systems are kind of like cities. One the surface they are often inefficient and messy, and people often wish they could come in and make them logical and rational, but people live in those cities and neighbourhoods, and sometimes the postbox is where it is for a reason.
If Jane Jacobs[2] were around, she could write a book called: "The Death and Life of Great American Operating Systems"
If the current state is the tragedy, I'm terrified of the farce.
My favorite systemdisms:
* (already mentioned several times above; sorry) even detached and nohup'd processes will be killed if the systemd context that transitively led to the creation of the process is killed: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394
Damn. So I'm just a simple Debian user. Systemd showed up, so I learned as much as I needed. And in some ways, I found it easier to use than init scripts.
Now I've been using Docker. And there is no systemd in containers. But of course, why would you want that? But the problem is that many packages now depend on systemd. Or at least, useful features do. So I'm back to fighting with scripts. It's funny, no?
For all those lamenting the death of standards, change for the sake of change, system breakage, bloat, and lack of user control: give FreeBSD a try (or any BSD, really).
systemd broke UNIX. Everything that worked before, now is a mine field. Last week I had a SLES virtual machine refusing to boot, no network, just emergency shell. The reason was a forgotten line in fstab (without nofail option of course), so systemd couldn't mount some file system. Coming from Gentoo, this is just unbelievable.
I particularly like how it ripped out the old logging system but completely failed to replace it properly. Local might be ok, but shipping to a central log server? Just save yourself the headache and use syslog or some other shipper like filebeat, etc. How can something that can't even do logging properly suddenly become the defacto standard?
[+] [-] jimrandomh|7 years ago|reply
Two years later, you can find hundreds of support requests across the internet, from frustrated users who are having their sessions killed by systemd.
Bugs are annoying, but that's life. On the other hand, when you're an impacted user who's lost work, and researching the bug leads you to a years-old discussion in which someone is actively denying that the bug exists and refusing to fix it, that's infuriating. I don't think systemd's developers deserve the trust that maintaining a core piece of infrastructure requires; they don't seem to care enough about whether they've broken things.
[+] [-] poettering|7 years ago|reply
Now I am used to taking blame for apparently everything that every went wrong on Linux, but you might as well blame your downstream distros for this as much you want to blame us upstream about this, as it's up to them to pick the right compile-time options matching their userbase and requirements in compatibility, and if they didn't do that to your liking, then maybe you should complain to them first.
(And yes, I still consider it a weakness of UNIX that "logout" doesn't really mean "logout", but just "maybe, please, if you'd be so kind, i'd like to exit, but not quite". I mean, that's not how you build a secure system. We fixed that really, fully knowing it would depart from UNIX tradition, but that's why we made it both compile-time and runtime configurable)
(Also, nobody has to "incorporate" systemd's library to avoid the automatic clean-up. In fact, there's no library we provide that could do that. What was requested though is to either run things as child of systemd --user or just register a separate PAM session, neither of which requires any systemd-specific library.)
Lennart
[+] [-] MereInterest|7 years ago|reply
[+] [-] jimrandomh|7 years ago|reply
[+] [-] MertsA|7 years ago|reply
How else do you propose to make sure that when I log off my ssh-agent is really terminated and not just locked up with my keys still in memory? The POSIX approach is insufficient, there's no way to know if a process received a signal and chose to ignore it and keep running or if it received a signal but it was deadlocked and kept running.
[+] [-] cout|7 years ago|reply
I disagree that POSIX says that processes should expect a SIGHUP when a user logs out (SIGHUP means the controlling terminal was closed). I am not at all a POSIX expert, so please correct me if I misunderstand, but afaict POSIX explicitly does not specify what happens to the controlling terminal when a user logs out (http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_...):
> POSIX.1 does not specify how controlling terminal access is affected by a user logging out (that is, by a controlling process terminating). 4.2 BSD uses the vhangup() function to prevent any access to the controlling terminal through file descriptors opened prior to logout. System V does not prevent controlling terminal access through file descriptors opened prior to logout (except for the case of the special file, /dev/tty). Some implementations choose to make processes immune from job control after logout (that is, such processes are always treated as if in the foreground); other implementations continue to enforce foreground/background checks after logout. Therefore, a Conforming POSIX.1 Application should not attempt to access the controlling terminal after logout since such access is unreliable. If an implementation chooses to deny access to a controlling terminal after its controlling process exits, POSIX.1 requires a certain type of behavior (see Controlling Terminal ).
[+] [-] newnewpdro|7 years ago|reply
See the enable-linger option for loginctl and KillUserProcesses for logind.conf. KillUserProcesses was set to default enabled on 4/9/2016, prior to that it didn't happen, but was configurable if desired. So you were always able to change the config to restore the previous behavior from the moment the default turned it on.
Edit:
Here is the commit where it happened
https://github.com/systemd/systemd/commit/97e5530cf2076a2b4f...
[+] [-] cryptonector|7 years ago|reply
Systemd is basically SMF, done poorly, because NIH.
[+] [-] jesuslop|7 years ago|reply
[+] [-] masonic|7 years ago|reply
[+] [-] JdeBP|7 years ago|reply
* https://news.ycombinator.com/item?id=12335128
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394#221
[+] [-] majewsky|7 years ago|reply
TIL what nohup(1) is for.
[+] [-] xyzzyz|7 years ago|reply
By "use systemd's new demonization API" you mean, instead of
$ screen
systemd asks you to write
$ systemd-run --scope --user screen
instead. Annoying to have to learn a new thing, but hardly the unbearable burden.
On the other hand, when you're an impacted user who's lost work, and researching the bug leads you to a years-old discussion in which someone is actively denying that the bug exists and refusing to fix it, that's infuriating.
Because it's a bug for some, and intended behavior for others. Look, you make it as if they introduced a bug on purpose to screw with some people. It's clearly not the case, there was a specific tradeoff involved.
[+] [-] pdkl95|7 years ago|reply
Can we please stop misrepresenting the complaints against systemd? The only time I ever hear this "monolithic binary" argument is from systemd advocates. The actual complaint is about tightly coupling important features together. Not only does this make it difficult (often impossible) to replace individual components, when tight coupling happens at the (internal) protocol level, any replacement component ne4cessarily hast to implement a bunch of (sometimes unwanted) systemd baggage.
Busybox implements all of its features in single monolithic binary, but it isn't a monolithic design that tightly couples those specific components together. Replacing one of busybox's components is often as simple as removing busybox's symlink and installing the replacement. This isn't even a "Unix philosophy" issue. Even inexperienced designers shouldn't have as hard time Understanding why systemd is a monolithic design but busybox isn't.
[+] [-] duckerude|7 years ago|reply
[+] [-] JdeBP|7 years ago|reply
* https://news.ycombinator.com/item?id=13390245
[+] [-] chris_wot|7 years ago|reply
[+] [-] ordu|7 years ago|reply
systemd is a PID 1 program, it means it have to raise bar higher. When troubles begin, you would need tools to fix them, and if PID1 is crashed, you are out of luck. If system cannot boot into shell, you'd need to fix it from initrd shell. Or to boot other system, to fix this one. It sucks.
Linux kernel chases very high standards of reliability, because when kernel panics it is even worse than PID1 crash. Init system should follow the same standards as linux.
[+] [-] geofft|7 years ago|reply
The bar is higher for pid 1 - if I were designing systemd I would have made a tiny pid 1 that just did message-passing to a more complex secondary process that could be restarted, or something, just to be safe - but I think systemd has empirically cleared the bar.
[+] [-] shittyadmin|7 years ago|reply
I just count my blessings that runit is widely packaged in every major distro because it can just happily sit on top of sysvinit, systemd, upstart, pretty much any init system and does things in a very simple shell script style, I really wasn't a fan of the weird ini-like format for systemd or several different tools I'm expected to learn just to read my (now binary) log files competently.
If you're sick of switching init systems constantly or don't want to have to write separate scripts for your linux box and your freebsd box even, I highly recommend checking runit out.
I'm sure I'll give it a serious shot eventually... in about 3 years once they work all the Poettering kinks out, just like PulseAudio. They're doing some cool things with cgroups and stuff, so I hope it gets there eventually.
[+] [-] cetra3|7 years ago|reply
If you've ever tried to use systemd inside docker to bring up a couple of services, you would know the hoops you have to jump through to get it working.
(I understand that docker wasn't invented to run multiple services in the one container, but sometimes it can't be avoided and simplifies app deployment vastly I.e, using CI to test your service actually starts up as per its definition: just run up a quick docker image with runit and a service definition file)
[+] [-] antt|7 years ago|reply
Good luck. If you need anything more than "I play a three minute song" on Linux audio you need both some type of real time kernel and jack.
[+] [-] Lazare|7 years ago|reply
I think I'd like systemd more if I had more confidence that he had done a better job.
I never really understood why Upstart didn't get more traction; was it just the Canonical backing made other distros avoid it, or did it have drawbacks I never ran across?
[+] [-] geofft|7 years ago|reply
(systemd is hardly unique in being dependency-based instead of event-based - but it is a strike against Upstart in particular. Also, while one of the stronger claims of why Upstart should work this way is it matches the evented mindset of udev, systemd can handle events from udev just fine, by just having it add additional targets - and in the end, systemd consumed udev.)
The other was that it was hard to deal with its bugs. I had a job that wasn't activating. I spent days chasing it, including attaching a debugger to Upstart, and eventually gave up. I've used systemd extensively and have not had "Why isn't this service starting" bugs. (I've certainly had that confusion, but I've always been able to figure it out quickly.) systemd has in my experience been reliable and trustworthy. I'm not going to say that it's bug-free, or that I like its implementation, or that I have no complaints about it. I am going to say I haven't been frustrated by it - even when running very old distro versions with lots of known bug fixes upstream. That's very important for whether I'm happy with it or whether I'm going to switch back to sysvinit and find something else for service monitoring.
[+] [-] djsumdog|7 years ago|reply
Systemd also solves the problem of unified ways to start services across many distributions. Packaging things is easier. I don't like the systemd ini target format and wish something else had won out instead. With something like upstart, in theory you could rewrite the upstart daemon and use the same scripts, since the format/standard was much simpler.
I feel like there must have been a lot of bullying to get systemd across so many systems. It just seems weird to see so many major distros all decide to accept something so complex if it wasn't for Redhat cramming it into everything they maintained and helped fund.
[+] [-] blihp|7 years ago|reply
Even though Red Hat is a much larger organization, their contributions have never felt like power plays in the same way that Canonical's have. Perhaps it's just the way that Canonical tries to shove their stuff through while Red Hat seems to get more organic buy-in. For example, as controversial as Systemd was (mainly from sys admins and power users), it did have a level buy-in from distro maintainers and developers which Upstart never really did. Canonical just tried to say 'we're doing this.'
[+] [-] the_why_of_y|7 years ago|reply
https://web.archive.org/web/20140928104327/https://plus.goog...
[+] [-] na85|7 years ago|reply
[+] [-] aidenn0|7 years ago|reply
1. They changed the defaults for systemd-logind to disallow network access
2. When systemd-logind attempts to send a message to the NIS server, it instead hangs (or has a timeout greater than the systemd watchdog anyways)
3. The watchdog timer causes all systemd-logind services to be restart, which then tears down all of the child processes. My X session is a child process.
[+] [-] m463|7 years ago|reply
However, I believe the implementation is not great.
A couple specific observations:
* lots of binaries. /lib/systemd, the logs. This more than anything else is a tragedy. Binaries get in the way of viewing, diagnosing, understanding and changing things. It seems like premature optimization to me - I don't think speed requires it. I think the majority are completely unnecessary.
* copying from launchd? Commercial software vendors ship binaries. They are aligned with them. They resist disclosure, which protects from reverse engineering, lawsuits and user modifications. Does that align with linux?
* the config files are a disorganized mess. It is like /etc/init.d but poorly understood and then /etc/rc.d was lumped into the same directory and organization became chaos. I looked at one system here and in /etc/systemd there are 16 directories, 23 symbolic links and just 11 files. Try unraveling the interdependencies.
additionally the config files (which mimic windows config files) lack depth. Simple config files can have benefits, say if a GUI read and wrote them to change settings, but I don't think that applies here. What I noticed is that they cannot do anything, so additional logic requires an intermediate script or compiled binary elsewhere.
* it is responsible for too many functions. I bet it didn't solve so many in its first iteration, but sucked in more over time. It reminds me of the accounting program that grew to rule everything in Tron.
I wonder if these problems could be resolved. It's hard to remove complexity and add elegance later.
[+] [-] atombender|7 years ago|reply
[+] [-] mixmastamyk|7 years ago|reply
[+] [-] squirrelicus|7 years ago|reply
[+] [-] andrewstuart|7 years ago|reply
It's got tons of really great functionality.
systemd feels like it is well thought out and modern, like it's an integrated rethink and rebuild of lots of various software utilities that have evolved over many years.
I'm investing all my learning in systemd where appropriate instead of things I used previously like cron and supervisor.
[+] [-] Kivutar|7 years ago|reply
Systemd is written in C and relies on kernel headers. Which means that they need to have a layer that deals with different versions of the kernel headers. I remember it was headers about network.
At the time I was working on this, I had to contribute to systemd to add support for my kernel version. Got a few patches accepted in their master. And things were working fine for a while.
Until they suddenly decided to drop <= 3.10 kernel support. Yep. Afterwhat I had to maintain patch sets for systemd, which is a great amount of job, given that my upstream was updating systemd quite often.
Now my good people, explain me how such a mess could have happen with a previous init based on shell?
[+] [-] chroma|7 years ago|reply
This is the person behind PulseAudio, Avahi, and systemd. In any sane world, these software projects would have been stillborn. Sensible alternatives would have outcompeted them. Instead, Red Hat did their best to foist them upon the community. They promised that after the initial adoption, all the wrinkles would be ironed out. All three times this was not true.
I hate to appear so cynical, and I'm certain that Red Hat didn't intentionally pursue this strategy, but it sure is convenient for them that the harder Linux is to use, the more they can sell support contracts. Not so convenient for people who actually want to use Linux.
1. https://en.wikipedia.org/wiki/Thomas_Midgley_Jr.
[+] [-] majewsky|7 years ago|reply
Considering that Midgley Jr. is (indirectly) responsible for the suffering and potentially death of many humans and other beings, I submit that this (in spirit) triggers Godwin's Law. The debate is therefore over.
[+] [-] ernst_klim|7 years ago|reply
Of course, and they didn't, because there were no sensible alternatives on linux. All these projects were clones of decently engineered macos projects, before they were created, linux people used some truly silly staff like sysvinit.
I would use something better than pulse and systemd, but there is nothing better available.
[+] [-] Sammi|7 years ago|reply
I'm not aware of any audio daemon that is as feature rich and flexible and PulseAudio. None of the alternatives fit the same general purpose use case that is expected in a modern desktop that needs to compete with Mac OS and Windows 10.
[+] [-] asark|7 years ago|reply
Huh, I didn't know he was behind Avahi. That... explains some things. Thanks.
[+] [-] darkpuma|7 years ago|reply
[+] [-] myrandomcomment|7 years ago|reply
I think the only Unix system to get this right was likely Solaris via SMF.
https://en.m.wikipedia.org/wiki/Service_Management_Facility
WindowsNT also did it correctly.
[+] [-] voltagex_|7 years ago|reply
[+] [-] gerbilly|7 years ago|reply
systemd reminds me of the efforts in some cities at clearing 'slums' to replace them with 'modern' apartment blocks.[1]
sytemd seems like a similar technocratic, modernist effort aimed at UNIX.
Operating systems are kind of like cities. One the surface they are often inefficient and messy, and people often wish they could come in and make them logical and rational, but people live in those cities and neighbourhoods, and sometimes the postbox is where it is for a reason.
If Jane Jacobs[2] were around, she could write a book called: "The Death and Life of Great American Operating Systems"
[1] See: https://en.wikipedia.org/wiki/Pruitt%E2%80%93Igoe, https://en.wikipedia.org/wiki/Quarry_Hill,_Leeds, https://www.youtube.com/watch?time_continue=862&v=Dxr2tEGlWU...
[2] https://www.citymetric.com/fabric/against-modernist-nightmar...
[+] [-] _pmf_|7 years ago|reply
My favorite systemdisms:
* (already mentioned several times above; sorry) even detached and nohup'd processes will be killed if the systemd context that transitively led to the creation of the process is killed: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394
* mounts being in a separate "mount" namespace that is not accessible normally: http://www.volkerschatz.com/unix/advmount.html
[+] [-] mirimir|7 years ago|reply
Now I've been using Docker. And there is no systemd in containers. But of course, why would you want that? But the problem is that many packages now depend on systemd. Or at least, useful features do. So I'm back to fighting with scripts. It's funny, no?
[+] [-] ComputerGuru|7 years ago|reply
[+] [-] pgtan|7 years ago|reply
[+] [-] j16sdiz|7 years ago|reply
Yes, it is faster. But when it breaks because bad dependency, it is hard to reason the breakage on a initrd shell on a small remote console.
I value debuggability, reliablity and manual fallback options on server.
[+] [-] arminiusreturns|7 years ago|reply