At the risk of sounding heretical, I kind of find myself in the middle ground regarding systemd - I was initially highly skeptical of it, and I still think it's problematic that it is so Linux-centric and will cause problems maintaining software to run both on Linux/systemd and on *BSD. The way it was pushed on distros was problematic, in my opinion.
But having used a couple of Linux systems running systemd - Raspian Jessie and openSUSE - I have to admit it's not that bad. In practice - on laptops and desktop systems (assuming one counts the Pi as a desktop system; for my use case, I do) - I had no problems with it. Enabling and disabling services is a lot easier. I do not think is as great as its proponents claim, but it's not as bad as some people think, either. Personally, I have come to appreciate journald, even though I still agree that binary logs are a bad idea. At least there is still the option of installing a syslog daemon.
I am also in the middle ground, but I have moved from the opposite direction. I used to like systemd a lot, especially for its simple unit files (compared to ugly System V shell scripts) and the fact that it could properly track processes and restart them on failure.
I have become a bit more skeptical, because most of the problems that I recently had seemed to be related to systemd. Including some networking problems, long boot delays because systemd decides to wait 90 seconds be default on some conditions that it considers to be errors, and problems such as having to restart systemd-logind manually because of some d-bus update [1]. Before the update, logging in via SSH blocked for a large amount of time.
The most annoying part is that some of the problems take quite a bit of work to debug due to the opaque nature of modern systemd/d-bus/...-based systems.
I'm a bit more on the positive side. I actually really like systemd. When it was first introduced I spent 10-15 minutes learning how it operated and after that it has been so much better than the old init systems.
And the best part is that it's on all the major distros so I don't have to keep relearning the init system if I try something new.
My guess has been that the majority of users enjoy it and a few very vocal opponents have such hate for anything new that they will latch on to every small bug as if it's the worlds end.
(Anecdotally, this is the first time I have written a comment about systemd as other discussions have been non-productive flame fests without substance).
> The way it was pushed on distros was problematic, in my opinion.
The technical term for this is "precommitment", and it comes from Tom Schelling. It's actually quite advantageous: once you're precommitted to a particular direction, you're emotionally invested in getting the most out of it. By precommitting you to systemd, the distros hope to get the community to leverage systemd's benefits.
I'd probably respect SystemD a lot more if it measured itself against a modern init system like Gentoo's OpenRC instead of pretending it's invented dependency handling.
I'm on Gentoo and remember the switch to OpenRC. I was nervous. But all the old stuff worked! And my custom init scripts? Changed only one or two lines to make them work and a few more edits to make them fully OpenRC like. Backwards compatible FTW!
And, now with OpenRC it still "just works". Fast, predictable boot with logs in normal places that I can easily debug, manage and modify.
Systemd isn't an init system, it's a service (a.k.a daemon) management daemon. Its primary purpose is to restart and diagnose failing daemons cleanly.
Systemd won for one simple reason: it's the only tool that accomplishes this task without bugs. We've been running daemontools for almost a decade in production, and it's a nightmare of bugs. Very glad to be finally switching to systemd.
OpenRC is still more difficult to write init scripts for than systemd; and offers less flexibility in scheduling inits. For example, I can add a 'before' clause to a custom systemd service unit, and I don't need to modify the subsequent units to depend on my custom unit. This is especially useful when you want a custom forking or oneshot service to reliably run before a standard service shipped with a package in your distro.
In a similar vein, systemd targets are considerably more useful than sysv/OpenRC runlevels. A typical sysadmin can read a couple paragraphs of manpage and set up a new target (analogous to runlevel) in systemd; and it can be used for debugging or recovery in the same fashion that runlevels are in other init systems.
On top of this, there are powerful system services developed in tandem with systemd, in the same repository, which offer well-integrated standard alternatives to things which were superficially different on ever distro only four years ago.
OpenRC's the only init system I've actually liked. It's one of the things I really miss from when I used Gentoo, and I don't know why more distros didn't adopt it years ago.
> The justification for storing logs in a binary format was speed and performance, they are more easily indexed and faster to search.
Are there any benchmarks for this? I don't know why, and I may be doing things wrong, but journalctl -lu <unit> --since yesterday in our prod env takes a couple of seconds before I see any output, while a (z)grep on a date rotated, potentially compressed log file on a non-journald system is generally instantaneous.
On a side note, I really like machine parsable, human readable logs, and I've never had any speed or performance issues with it when dealing with volumes of hundreds of millions of log entries, though that may be because I don't know any better.
My question would be "is searching worth optimizing for anyway?". Log files are typically only read when things go wrong or by automated systems where a few seconds here or there doesn't make much of a difference.
Faster to search often means slower to write too, I'd prefer faster to write and slower to search if I had to choose.
Of course nobody benchmarked. Rarely are there programmers in OSS these days that benchmark anything. (not nearly as common as those who claim "speed")
There was a bug though where the journaling was so slow it caused the whole server to crash. (sorry, can't find it right now; something about them using mmaped files and the access pattern was.. or something, interesting stuff:)
Binary logs should be faster, in theory, and should/could be as robust as text logs (if not more), in theory. edit: note that GNU grep is insanely optimized.
At first I didn't care much for the idea of having to learn yet another init system but as I had to write ansible automation stuff for services on Centos 7 it was kind of required that I have some basic understanding of systemd. I have to say now that I'm more familiar with it it has begun to grow on me.
There is something subjectively nice to me about running systemctl status and seeing a nice, clear picture of what the current state of services are on a system, being able to control the life cycle of a service including having systemd monitor and restart it, not having to deal with improperly written init scripts, etc.
The idea of systemd isn't bad. What's annoying is the friction involved in using it.
It's just a shade too ornate, a little too magical, in both cases only by a small degree, but it's an important one.
Creating a workable systemd init script is actually pleasant. Getting it running is easy. Checking for errors with status is nice, but searching the logs is annoying.
pm2 (https://github.com/Unitech/pm2) has a neat feature where you can watch logs easily, someting that systemd should totally steal and pack into journalctl, like "systemctl logs sshd" shows it in real-time, an alias to the obnoxiously verbose "journalctl -u sshd -f"
Russ Allbery's investigation [1] into alternative init systems, in the Debian's project to select the best one, also viewed this systemd property in a similar way -- a surprising and unexpected benefit.
Systemd's native understanding of cgroups, process hierarchy, and logging makes this view possible.
* it replaces significant part of the OS and creates a new APIs. It's not really problem of systemd itself, but once other applications start depend on the API you would need systemd to use it. This is especially bad for non-linux systems, like BSD for example, which need to write tools that will emulate such API. It doesn't help that Lennart openly is against any non-linux system so he won't make things easier
* it's buggy, this kind of reminds me of Pulse Audio (also created by Lennart) although eventually things will improve, but it sucks on early versions of RedHat/CentOS 7.
Best thing of systemd is that it's putting people behind alternate systemd-free Linuxen. Devuan, Alpine, Gentoo, Void (and particularly the BSDs) come along nicely.
I can tolerate systemd on desktops as long as I don't have to deal with it. The moment it craps out with Java-esque error traces in binary logs I'll install Slackware (or is there a modern desktop Linux without systemd I'm not aware of?).
Other than for desktops with complex run-time dependencies (WLAN, USB devices, login sessions), what are the benefits of systemd for servers that warrant a massive, polarizing, tasteless monolith which puts you up for endless patches and reboots, especially in a container world? (Re-)starting and stopping a daemon isn't rocket science; it's like 10 lines of shell code.
SystemD was so polarizing for me, I was a Fedora user and RHEL user for work- but it started consuming everything and giving really bizarre issues... I tried reaching out and explaining to people that it wasn't working in the way I expected or, asking them to point me towards the docs so I can at least learn how to use journalling properly so it doesn't hide issues from me.. and was met with some hostility.
When I asked to disable binary logging entirely because it kept getting corrupted and was opaque I was met with "Unlearn your old ways old man"-style responses and "You can just log to rsyslog too".. No, I want it disabled.. I don't have a desire to log twice..
It was then I realised I was at the mercy of systemd, they can push whatever and reject whatever and I'm completely out of control- I cannot introspect their service manager effectively, it's non-deterministic. It just reeks of hubris from the maintainers. It does mostly the right thing for most people, and they compare it to sysvinit which admittedly needed love. (and, was not a process manager, was only a process starter).
So, I adopted *BSD on the server. And christ is it wonderful.
I still use Arch/SystemD on my desktop at home, because it's actually pretty useful on laptops/desktops. But I swore a vow never to manage a server with SystemD on it.
I'm still runing RHEL6 at work, for the next OS, I'm pushing my very large corp to adopt FreeBSD as an alternative. That's a fairly large amount of money that Red Hat will lose and I don't particularly feel bad about it.
It's 10 lines of shell code to do it badly. Poor Man's Dæmon Supervisors written as shell scripts are inevitably flawed in one way or another. It is, however, easy to do with one of the many toolsets that have been around since the 1990s for doing this.
One of the exhibits in the systemd House of Horror results from people taking multiple Poor Man's Dæmon Supervisors written badly in shell script, nesting them inside one another, nesting the result inside another service manager, and then recommending against service management in toto because this madness goes badly wrong.
Because on servers, even the huge repository isn't that much of a gain. But running "apt-get upgrade" beats rebuilding a machine by orders of magnitude. Even more for rented VPS where I must either add an instance or get downtime.
I'll put my opinion in the middle ground with systemd as well. I like a lot about it, and I dislike a lot about it. I really think the best thing they could have done would have been to make it modular. If people could just turn off the "features" that they don't want, there wouldn't be so much bitching about it.
Instead they keep trying to shove everything into one giant pile, and don't understand why people get upset.
>the best thing they could have done would have been to make it modular.
They did, systemd is very modular in nature. The only things that aren't modular that spring to mind is the journal and systemd the init requires dbus (sort of).
systemd the project has a little bit of everything that you need in an operating system but systemd the init is stripped down to being able to parse unit files, supervise services, and generate and walk the dependency graph from where you currently are to where you're aiming to be.
All of the random features that you always hear about are all individual self contained services. So for example, systemd the init knows nothing about the format of /etc/fstab or SysVInit scripts. It only understands how to work with unit files, the way this works is that there's separate self contained generators that parse /etc/fstab and all of the old init scripts and creates unit files that only exist in a tmpfs directory.
If you don't want something like systemd-networkd then you don't have to have it. You can just compile systemd without it, turn it off at runtime, or just configure systemd to not start said service. You can take systemd and make it suitable for everything from embedded use, to servers, to a multiseat desktop.
You'll find that there's pretty much nothing you can't outright disable, except for journald. journald needs to be running, but you can turn off the binary logging and redirect everything to syslog.
systemd itself is a collection of system daemons, as well as small programs to interact with those daemons, and almost all of them are disabled by default. That's my experience on Arch, and they adhere strictly to upstream defaults.
If that weren't enough you could simply leave out the daemons you don't like at compile time.
The repository and build system are set up to allow you to disable almost everything (except core services like evdev, the "systemd" init process, journald, and D-Bus).
Distributions have just found it so pointless to strip it down that they've left it largely as one package. You're entirely welcome to use your distribution's packaging scripts to produce a reduced package, if one doesn't exist yet.
As far as I'm concerned, I'm happy that it's a single repository tracking consistent, interoperable versions of each of the systems in systemd. This means that there are few dependencies to track.
People get upset because they're too lazy to run the build system themselves. People who are unwilling to run the build system and read the mailinglist are unfit to ship anything short of the whole thing anyway. In my opinion, if these people are complaining, it is meaningless. They are not the sort of people who can figure out how to fit their system into 8MB of flash on a tiny SoC, so them complaining that the systemd package is more than 10 megs is like a child complaining that their bedsheets are too barbie and not buzz lightyear enough.
Personally, I love Systemd for Web development as it makes the job of managing Node.js projects a breeze. I simply pack a Systemd unit file into a project's repository as part of its multi-environment configuration and construct deployment tasks to copy, enable, then start the unit file.
1. Copy <unit.service> to the sever, which looks like:
Systemd ideas I'm all for but it only hints at linux lack of clean abstraction power. sysvinit was full of redundancy; apparently BSD found a way to make a thin abstraction layer to make init files clean.
Bash isn't "good" at hinting proper abstractions, I rarely see talks about this, maybe gurus can see through the rain.
I keep seeing a place for a tiny lambda friendly intermediate layer .. Just so you can compose properties of your services like clojure ring middleware.
ps: the idea is that (retry 5) is a normal language function, and not a systemd opaque built-in, you can write your own thing, and the most used can go upstream. Hoping to avoid the sysvinit fragility and redundancy.
Shell and bash are actually excellent at this, but people don't like writing shell scripts. This is just process chaining. Take a look at DJB's or the more modern runit for init toolkits that compose.
> sysvinit was full of redundancy; apparently BSD found a way to make a thin abstraction layer to make init files clean.
They are completely different systems of init scripts. Just for starters, BSD does not have run levels, so the rcn.d directories and the maze of symlinks do not exist.
BSD init scripts themselves are much simpler mostly because there is a good system of helper functions which they all use, rather than every single script inventing its own wheel in sysvinit. Of course, this is just a discipline or convention, and sysvinit could be vastly improved if anybody was similarly industrious.
BSD init scripts also have completely different dependency management (PROVIDE and REQUIRE instead of numerical priorities). And the mechanism for disabling and enabling a script is via dead simple definitions in the single file /etc/rc.
I feel like at some point, news aggregation sites will need moratoriums on this topic. Systemd is an architectural decision with very philosophical effects, so by its very nature be very divisive. I'm, personally, not a fan at all of systemd, but am tired of seeing these stupid arguments everywhere all the time.
There are plenty of places to find competent summaries of both the technical and political arguments for and against systemd and we don't need yet another.
I don't feel they should. These are arguments I haven't heard before either and I'm glad to see more of these weird cases laid out.
I run Gentoo (OpenRC) and Void (Runit) on my own systems, and although I do like both of them, I find the total lack of alternatives to the systemd ecosystem troubling.
As a package maintainer, I do like being able to create rpms/deb files and only needing one standardized init script, so there is are a few advantages to systemd, but not many. It's complexity makes it difficult to create drop in replacements (work has stopped on uselessd and others).
There needs to be more community support for distros like Void and real alternatives. Articles like this encourage that kind of thinking.
If a forum can't handle a discussion of UNIX software, then it should probably stick to kitten gifs. The art of moderation and community management is to allow discussion while containing hostility.
Personally I feel that the sheer controversialness of systemd and its chief maintainer is a bug in and of itself, which the maintainers should address.
Why would you run the Exec command through systemd-escape? The help text says that's for the NAME of the unit... Am I missing something?
Also, I'm failing to think of a scripting purpose that requires you to place non-trivial bash directly in a systemd unit that couldn't be solved by writing out the script somewhere and just invoking the script in the unit.
> Now agreed, if your workflow demands that you embed a Bash while loop in a unit, you’re already in a bad place, but there are times where this is required for templating purposes.
It's right there in the post. I have indeed had to do something like this in the wild to run a Docker container with special needs.
I agree, Upstart was great for most things and was simple to understand. Though I ran into some difficult fork, exec issues for certain processes. I sometimes had to run a wrapper to use upstart, but those seemed to be edge cases.
I'd prefer if we stuck with Upstart and improved on it, though it already seems like a distant dream.
The problem with systemd is not that it is "bad". There's a lot worse software out there. Systemd is relatively competently programmed. And it's even useful!
The problem with systemd has always been that it forces you to do everything the systemd way, and usually to use its tools. It goes against everything that has helped GNU/Linux become a great system: that it wasn't really one system. It was bits and pieces, and you could add them together however you wanted to get something you liked.
Systemd is the opposite. It is inflexible and clunky, monolithic and proprietary, binary and difficult. Everything has to be made to work with it, not vice versa. It doesn't follow any of the old conventions that made it simple to combine one tool with another. I can compare it to Windows, but I feel like that would be an insult to Windows' usability.
If this one single fact was different, everyone would love systemd, because it has plenty of useful features. But because it is designed specifically to please a single quirky user, people hate it.
> It is inflexible and clunky, monolithic and proprietary, binary and difficult. Everything has to be made to work with it, not vice versa. It doesn't follow any of the old conventions that made it simple to combine one tool with another.
Could you share a few specific examples? That hasn't been the case at all for me. I have worked with systemd and have had the opposite experience. I simply write textual config files describing how to launch the service, which can be done in a variety of ways. Could you share more about how you tried to use systemd?
I'd encourage you to read Russ Allbery's analysis of systemd [2], which includes a description of his experience converting one of his packages to use systemd. He remarks on the well-done integration and its compatibility:
* Integrated daemon status. This one caught me by surprise, since the
systemd journal was functionality that I expected to dislike. But I was
surprised at how well-implemented it is, and systemctl status blew me
away. I think any systems administrator who has tried to debug a
running service will be immediately struck by the differences between
upstart:
lbcd start/running, process 32294
and systemd:
lbcd.service - responder for load balancing
Loaded: loaded (/lib/systemd/system/lbcd.service; enabled)
Active: active (running) since Sun 2013-12-29 13:01:24 PST; 1h 11min ago
Docs: man:lbcd(8)
http://www.eyrie.org/~eagle/software/lbcd/
Main PID: 25290 (lbcd)
CGroup: name=systemd:/system/lbcd.service
└─25290 /usr/sbin/lbcd -f -l
Dec 29 13:01:24 wanderer systemd[1]: Starting responder for load balancing...
Dec 29 13:01:24 wanderer systemd[1]: Started responder for load balancing.
Dec 29 13:01:24 wanderer lbcd[25290]: ready to accept requests
Dec 29 13:01:43 wanderer lbcd[25290]: request from ::1 (version 3)
Both are clearly superior to sysvinit, which bails on the problem
entirely and forces reimplementation in every init script, but the
systemd approach takes this to another level. And this is not an easy
change for upstart. While some more data could be added, like the
command line taken from ps, the most useful addition in systemd is the
log summary. And that relies on the journal, which is a fundamental
design decision of systemd.
And yes, all of those log messages are also in the syslog files where
one would expect to find them. And systemd can also capture standard
output and standard error from daemons and drop that in the journal and
from there into syslog, which makes it much easier to uncover daemon
startup problems that resulted in complaints to standard error instead
of syslog. This cannot even be easily replaced with something that
might parse the syslog files, even given output forwarding to syslog
(something upstart currently doesn't have), since the journal will
continue to work properly even if all syslog messages are forwarded off
the host, stored in some other format, or stored in some other file.
systemd is agnostic to the underlying syslog implementation.
I think Russ's passage exhibits the opposite sentiment from the one you wrote. Russ says you can get the benefits of systemctl and journalctl if you want, but you can also use syslog in the old-fashioned way if you want. This has been my experience with systemd as well. I still have a habit of writing "service postfix restart", which still works but is not the systemd-idiomatic way. I don't even have the "correct" way memorized because the old way that I'm used to using works.
So what I'm asking is, have you tried to achieve something with systemd and found that you were forced to do it differently? I'd be curious to learn more about what those cases were. I'm not affiliated with systemd project, but I have a general interest in wanting to see it improve.
(Please note that, barring conventions of some kind that span multiple tools, every tool naturally requires you to do things that tool's own way. If you are holding systemd to the standard that its job should be able to be performed by two different tools, each with the same interface, while supporting advanced features, then you should have other examples of this as well.)
As are most Bash init files with InitV... At some point we should consider that the init system as a whole (including the configuration scripts for each daemon) _is_ complicated and often confusing. In this regard, writing something like SystemD to elegantly handle most of the usecases is actually a good idea.
Systemd has a lot of attention and people working on it and it will eventually become good enough for the vast majority.
But there were questionable tie-ins with various pieces like udev, consolekit and even gnome that allowed Systemd to become defacto init. The call for a kernel bus promotes a similar lock in with Systemd and this makes the use or development of alternatives and choice difficult.
There are things like predictable network names which are useful for 1% of users and are anything but predictable. Binary logging makes sense for the security industry Redhat serves but again has no use for the 99% others who anyway have to put up with it. There is a pattern of forcing things onto everyone that make sense for a tiny minority.
The big problem is open source funding. No one is interested in just supporting projects they benefit from. Acquisitions or hiring developers put these projects and developers under the control of companies like Redhat. Redhat has become a cathedral and a cathedral by sheer size and nature is always interested in securing and furthering its own influence and interests. When you allow such forces to become too powerful they will subsume the public interest to their interest.
The sad thing is, you do not really have a second option, yes I know some distros say they have alternatives, but SystemD becomes the "preferred" init system nearly everywhere now.
I just hope Debian to get rid of SystemD and return to whatever else. I know I'm biased, just failed to find a reason to love SystemD, tried a few times.
I've found that devuan is a joy to use on my laptop.
I'm also rotating BSDs and open source solaris variants on to the home network machines. (Switching to solaris to get rid of systemd would be overkill; I switched to joyent triton for better containers and zfs...)
More and more systemd is becoming symptomatic of deeper divide within the Linux "community".
The split being between those that embraced Linux for being a free, in both senses, _nix unburdened by AT&T and running on commodity hardware, and those that got to know it after the dot-com crash as the L in LAMP.
The former cares for Linux as a _nix, the latter could not care less about _nix and may see it as a vestigial appendage that should have been amputated long ago.
Let me just say that I disagree: I am squarely in the former camp – I started with SunOS 4 in 1992, learned to use the Internet before the WWW, and am a long-time GNU advocate, and I am also a professional system administrator. I like systemd just fine. In fact, I started using it at home and at work just as soon as the package became available in Debian 7 (wheezy).
I think, however, that you have a point in that some people ascribe some mystical properties to the design of _nix and _nix-like systems. I have never done this, perhaps because I remember the early days of OS/platform wars (Amiga or Atari? PC or console? Unix or VMS?), first on BBSes, then on Usenet. I also, early on, read the Unix-Haters Handbook, and I found it enlightening, even though it is not always spot-on.
I'm not so sure if it really has that much to do with heritage. A lot of us like the design principles you can see in Unix and that are pretty much absent in systems like Windows NT. They very much reflect the conclusions I came to after over fifteen years of building and debugging systems.
But, yes, I have the impression that a lot of Linux users would just as gladly use a modern BeOS or a ReactOS, regardless of the underlying design.
And a lot of systemd criticism comes from people who are just averse to any kind of change and maybe long for the bygone days of HP-UX and Irix.
I've never particularly liked init systems that restart jobs when they die. Normally, I don't want daemons that crash to restart. They should die, be caught by monitoring and the server bypassed. I would accept a single restart but after that, there's clearly a problem, and the systems should fail, rather than restarting the process again and again and again.
As a single user with a number of personal/hobbyist machines, I actually find systemd annoying to work with for precisely this reason. If a daemon is misconfigured and fails to launch several times in a row, it inevitably triggers the systemd "too many retries" error, after which systemd will refuse to start the daemon until some timeout has expired or the counter is cleared. This makes troubleshooting more difficult and frustrating.
One of the things I like about systemd is you can get a graph of the boot time along with critical path via the following: systemd-analyze plot >boot.svg
Most people I hear having problems with Systems is either on Redhat or Debian - not realising the both are mangling systemd badly. RHEL took in systemd way too early and are missing a lot of needed functionality. Also the have an ungodly amount of patches on top - so much you could argue it should be named redhatd instead. Debian just chooses to take the worst possible middle position due to politics. All of the disadvantages from systemd, but not really any of the great stuff as systemd units often just start shellscripts due to compatibility.... If you want to try systemd in all its glory try Arch or something downstream from it... I for one think that systemd has made my admin-life so much better ..
I am running systemd on Gentoo. (like Sabayon) While some of the features are nice, it is a royal pain to upgrade due to intertwined deps and forced restarts.
And it doesn't do anything OpenRC couldn't do, in fact its journald is a pain which had to be worked around. OpenRC way of doing socket activations and dbus activations also works slightly better (in case something crashes) It does parallelism just as well, service dependencies too.
Gentoo does not mangle anything related to systemd unlike the mentioned distributions.
(from tfa): "Now, let us turn our attention to the benefits that systemd brings us. I believe that these are the reasons that all Linux distributions have adopted systemd."
- Handwaves real criticisms and instead tries to look like an objective writing by presenting a trivial issue
- Have no idea about advances in sysvinit in the last decade that allowed support for parallel boot and dependency relations (and which was supported natively on mainstream distribution)
- Totally ignores the other "modern" and widely used init system and instead compares systemd with sysvinit of 80's (which wasn't used by any mainstream distro in that form)
- Overrates process supervision features which were already as equally as easy to use with supervision suites like runit (IMHO even easier)
Meh, what did I expect? Bandwagon is going full speed.
It does log the standard error stream. The default value of StandardError= is inherit, which will cause stderr to go to the same place that stdout goes, which by defaults to journal. This is documented in systemd.exec(5).
If not overriden in an individual service's unit file, perhaps you have set DefaultStandardError= in /etc/systemd/system.conf?
I get the feeling anyone who complains about systemd is just a hobbyist jumping on a hate bandwagon. For anyone who manages production systems, systemd is a godsend compared to the homegrown-per-distro shell based init systems.
krylon|9 years ago
But having used a couple of Linux systems running systemd - Raspian Jessie and openSUSE - I have to admit it's not that bad. In practice - on laptops and desktop systems (assuming one counts the Pi as a desktop system; for my use case, I do) - I had no problems with it. Enabling and disabling services is a lot easier. I do not think is as great as its proponents claim, but it's not as bad as some people think, either. Personally, I have come to appreciate journald, even though I still agree that binary logs are a bad idea. At least there is still the option of installing a syslog daemon.
microtonal|9 years ago
I have become a bit more skeptical, because most of the problems that I recently had seemed to be related to systemd. Including some networking problems, long boot delays because systemd decides to wait 90 seconds be default on some conditions that it considers to be errors, and problems such as having to restart systemd-logind manually because of some d-bus update [1]. Before the update, logging in via SSH blocked for a large amount of time.
The most annoying part is that some of the problems take quite a bit of work to debug due to the opaque nature of modern systemd/d-bus/...-based systems.
[1] https://major.io/2015/07/27/very-slow-ssh-logins-on-fedora-2...
digi_owl|9 years ago
Damn it, the LFS team basically adopted eudev, of Gentoo fame, because extracting udev from systemd required manual intervention over and over.
tyfon|9 years ago
And the best part is that it's on all the major distros so I don't have to keep relearning the init system if I try something new.
My guess has been that the majority of users enjoy it and a few very vocal opponents have such hate for anything new that they will latch on to every small bug as if it's the worlds end.
(Anecdotally, this is the first time I have written a comment about systemd as other discussions have been non-productive flame fests without substance).
pmlnr|9 years ago
You're lucky. For me, systemd randomly fails to execute a reboot, then, because of this, fails to do halt, and I have to force power off the machine.
And this is just one.
bitwize|9 years ago
The technical term for this is "precommitment", and it comes from Tom Schelling. It's actually quite advantageous: once you're precommitted to a particular direction, you're emotionally invested in getting the most out of it. By precommitting you to systemd, the distros hope to get the community to leverage systemd's benefits.
FeepingCreature|9 years ago
https://wiki.gentoo.org/wiki/Comparison_of_init_systems
> OpenRC provides a number of features touted as innovative by recent init systems like systemd ...
edoceo|9 years ago
And, now with OpenRC it still "just works". Fast, predictable boot with logs in normal places that I can easily debug, manage and modify.
otabdeveloper|9 years ago
Systemd won for one simple reason: it's the only tool that accomplishes this task without bugs. We've been running daemontools for almost a decade in production, and it's a nightmare of bugs. Very glad to be finally switching to systemd.
microcolonel|9 years ago
In a similar vein, systemd targets are considerably more useful than sysv/OpenRC runlevels. A typical sysadmin can read a couple paragraphs of manpage and set up a new target (analogous to runlevel) in systemd; and it can be used for debugging or recovery in the same fashion that runlevels are in other init systems.
On top of this, there are powerful system services developed in tandem with systemd, in the same repository, which offer well-integrated standard alternatives to things which were superficially different on ever distro only four years ago.
ashark|9 years ago
digi_owl|9 years ago
dom0|9 years ago
sebcat|9 years ago
Are there any benchmarks for this? I don't know why, and I may be doing things wrong, but journalctl -lu <unit> --since yesterday in our prod env takes a couple of seconds before I see any output, while a (z)grep on a date rotated, potentially compressed log file on a non-journald system is generally instantaneous.
On a side note, I really like machine parsable, human readable logs, and I've never had any speed or performance issues with it when dealing with volumes of hundreds of millions of log entries, though that may be because I don't know any better.
flukus|9 years ago
Faster to search often means slower to write too, I'd prefer faster to write and slower to search if I had to choose.
gens|9 years ago
There was a bug though where the journaling was so slow it caused the whole server to crash. (sorry, can't find it right now; something about them using mmaped files and the access pattern was.. or something, interesting stuff:)
Binary logs should be faster, in theory, and should/could be as robust as text logs (if not more), in theory. edit: note that GNU grep is insanely optimized.
amyjess|9 years ago
alyandon|9 years ago
There is something subjectively nice to me about running systemctl status and seeing a nice, clear picture of what the current state of services are on a system, being able to control the life cycle of a service including having systemd monitor and restart it, not having to deal with improperly written init scripts, etc.
Stockholm Syndrome perhaps?
astrodust|9 years ago
It's just a shade too ornate, a little too magical, in both cases only by a small degree, but it's an important one.
Creating a workable systemd init script is actually pleasant. Getting it running is easy. Checking for errors with status is nice, but searching the logs is annoying.
pm2 (https://github.com/Unitech/pm2) has a neat feature where you can watch logs easily, someting that systemd should totally steal and pack into journalctl, like "systemctl logs sshd" shows it in real-time, an alias to the obnoxiously verbose "journalctl -u sshd -f"
jcrites|9 years ago
Systemd's native understanding of cgroups, process hierarchy, and logging makes this view possible.
[1] https://news.ycombinator.com/item?id=13387989
takeda|9 years ago
* it replaces significant part of the OS and creates a new APIs. It's not really problem of systemd itself, but once other applications start depend on the API you would need systemd to use it. This is especially bad for non-linux systems, like BSD for example, which need to write tools that will emulate such API. It doesn't help that Lennart openly is against any non-linux system so he won't make things easier
* it's buggy, this kind of reminds me of Pulse Audio (also created by Lennart) although eventually things will improve, but it sucks on early versions of RedHat/CentOS 7.
tannhaeuser|9 years ago
I can tolerate systemd on desktops as long as I don't have to deal with it. The moment it craps out with Java-esque error traces in binary logs I'll install Slackware (or is there a modern desktop Linux without systemd I'm not aware of?).
Other than for desktops with complex run-time dependencies (WLAN, USB devices, login sessions), what are the benefits of systemd for servers that warrant a massive, polarizing, tasteless monolith which puts you up for endless patches and reboots, especially in a container world? (Re-)starting and stopping a daemon isn't rocket science; it's like 10 lines of shell code.
dijit|9 years ago
When I asked to disable binary logging entirely because it kept getting corrupted and was opaque I was met with "Unlearn your old ways old man"-style responses and "You can just log to rsyslog too".. No, I want it disabled.. I don't have a desire to log twice..
It was then I realised I was at the mercy of systemd, they can push whatever and reject whatever and I'm completely out of control- I cannot introspect their service manager effectively, it's non-deterministic. It just reeks of hubris from the maintainers. It does mostly the right thing for most people, and they compare it to sysvinit which admittedly needed love. (and, was not a process manager, was only a process starter).
So, I adopted *BSD on the server. And christ is it wonderful.
I still use Arch/SystemD on my desktop at home, because it's actually pretty useful on laptops/desktops. But I swore a vow never to manage a server with SystemD on it.
I'm still runing RHEL6 at work, for the next OS, I'm pushing my very large corp to adopt FreeBSD as an alternative. That's a fairly large amount of money that Red Hat will lose and I don't particularly feel bad about it.
They forced my hand.
JdeBP|9 years ago
* http://jdebp.eu./FGA/daemontools-family.html
One of the exhibits in the systemd House of Horror results from people taking multiple Poor Man's Dæmon Supervisors written badly in shell script, nesting them inside one another, nesting the result inside another service manager, and then recommending against service management in toto because this madness goes badly wrong.
* http://jdebp.eu./FGA/systemd-house-of-horror/wso2.html
aaronmdjones|9 years ago
I've been using Linux Mint, Debian Edition, as my main workstation OS for more than 3 years now.
It has systemd, but does not use it as init; it has a Makefile-style boot process as far as I know.
vbernat|9 years ago
That's why half of the SysV-style scripts to restart call sleep. No such unreliable hacks with systemd...
marcosdumay|9 years ago
Because on servers, even the huge repository isn't that much of a gain. But running "apt-get upgrade" beats rebuilding a machine by orders of magnitude. Even more for rented VPS where I must either add an instance or get downtime.
CaptSpify|9 years ago
Instead they keep trying to shove everything into one giant pile, and don't understand why people get upset.
MertsA|9 years ago
They did, systemd is very modular in nature. The only things that aren't modular that spring to mind is the journal and systemd the init requires dbus (sort of).
systemd the project has a little bit of everything that you need in an operating system but systemd the init is stripped down to being able to parse unit files, supervise services, and generate and walk the dependency graph from where you currently are to where you're aiming to be.
All of the random features that you always hear about are all individual self contained services. So for example, systemd the init knows nothing about the format of /etc/fstab or SysVInit scripts. It only understands how to work with unit files, the way this works is that there's separate self contained generators that parse /etc/fstab and all of the old init scripts and creates unit files that only exist in a tmpfs directory.
If you don't want something like systemd-networkd then you don't have to have it. You can just compile systemd without it, turn it off at runtime, or just configure systemd to not start said service. You can take systemd and make it suitable for everything from embedded use, to servers, to a multiseat desktop.
srslack|9 years ago
microcolonel|9 years ago
Distributions have just found it so pointless to strip it down that they've left it largely as one package. You're entirely welcome to use your distribution's packaging scripts to produce a reduced package, if one doesn't exist yet.
As far as I'm concerned, I'm happy that it's a single repository tracking consistent, interoperable versions of each of the systems in systemd. This means that there are few dependencies to track.
People get upset because they're too lazy to run the build system themselves. People who are unwilling to run the build system and read the mailinglist are unfit to ship anything short of the whole thing anyway. In my opinion, if these people are complaining, it is meaningless. They are not the sort of people who can figure out how to fit their system into 8MB of flash on a tiny SoC, so them complaining that the systemd package is more than 10 megs is like a child complaining that their bedsheets are too barbie and not buzz lightyear enough.
msimpson|9 years ago
1. Copy <unit.service> to the sever, which looks like:
2. Enable the unit file in place: 3. Start the unit file: And, of course, destroying that deployment is just as easy.agumonkey|9 years ago
Bash isn't "good" at hinting proper abstractions, I rarely see talks about this, maybe gurus can see through the rain.
I keep seeing a place for a tiny lambda friendly intermediate layer .. Just so you can compose properties of your services like clojure ring middleware.
Is this ugly to your eyes ?ps: the idea is that (retry 5) is a normal language function, and not a systemd opaque built-in, you can write your own thing, and the most used can go upstream. Hoping to avoid the sysvinit fragility and redundancy.
chubot|9 years ago
https://cr.yp.to/daemontools.html
http://smarden.org/runit/
Here is a bash function that retries N times:
Then you can compose with a timeout function, which already exists: You can pass the names of other functions as pre and post hooks as well.https://www.gnu.org/software/coreutils/manual/html_node/time...
Shell has a very forth-like quality to it, and Forth is sort of like a backwards Lisp as well (postfix rather than prefix).
fnj|9 years ago
They are completely different systems of init scripts. Just for starters, BSD does not have run levels, so the rcn.d directories and the maze of symlinks do not exist.
BSD init scripts themselves are much simpler mostly because there is a good system of helper functions which they all use, rather than every single script inventing its own wheel in sysvinit. Of course, this is just a discipline or convention, and sysvinit could be vastly improved if anybody was similarly industrious.
BSD init scripts also have completely different dependency management (PROVIDE and REQUIRE instead of numerical priorities). And the mechanism for disabling and enabling a script is via dead simple definitions in the single file /etc/rc.
djsumdog|9 years ago
ioddly|9 years ago
Karrot_Kream|9 years ago
There are plenty of places to find competent summaries of both the technical and political arguments for and against systemd and we don't need yet another.
djsumdog|9 years ago
I run Gentoo (OpenRC) and Void (Runit) on my own systems, and although I do like both of them, I find the total lack of alternatives to the systemd ecosystem troubling.
As a package maintainer, I do like being able to create rpms/deb files and only needing one standardized init script, so there is are a few advantages to systemd, but not many. It's complexity makes it difficult to create drop in replacements (work has stopped on uselessd and others).
There needs to be more community support for distros like Void and real alternatives. Articles like this encourage that kind of thinking.
CaptSpify|9 years ago
pjc50|9 years ago
Personally I feel that the sheer controversialness of systemd and its chief maintainer is a bug in and of itself, which the maintainers should address.
colemickens|9 years ago
Also, I'm failing to think of a scripting purpose that requires you to place non-trivial bash directly in a systemd unit that couldn't be solved by writing out the script somewhere and just invoking the script in the unit.
naftulikay|9 years ago
It's right there in the post. I have indeed had to do something like this in the wild to run a Docker container with special needs.
chungy|9 years ago
You're not. Contrived example is contrived.
awinder|9 years ago
nictrix|9 years ago
I'd prefer if we stuck with Upstart and improved on it, though it already seems like a distant dream.
0xbadcafebee|9 years ago
The problem with systemd has always been that it forces you to do everything the systemd way, and usually to use its tools. It goes against everything that has helped GNU/Linux become a great system: that it wasn't really one system. It was bits and pieces, and you could add them together however you wanted to get something you liked.
Systemd is the opposite. It is inflexible and clunky, monolithic and proprietary, binary and difficult. Everything has to be made to work with it, not vice versa. It doesn't follow any of the old conventions that made it simple to combine one tool with another. I can compare it to Windows, but I feel like that would be an insult to Windows' usability.
If this one single fact was different, everyone would love systemd, because it has plenty of useful features. But because it is designed specifically to please a single quirky user, people hate it.
reflexive|9 years ago
The other points can be debated, but please don't call it proprietary. It is free software as in "freedom" (LGPL).
jcrites|9 years ago
Could you share a few specific examples? That hasn't been the case at all for me. I have worked with systemd and have had the opposite experience. I simply write textual config files describing how to launch the service, which can be done in a variety of ways. Could you share more about how you tried to use systemd?
I'd encourage you to read Russ Allbery's analysis of systemd [2], which includes a description of his experience converting one of his packages to use systemd. He remarks on the well-done integration and its compatibility:
I think Russ's passage exhibits the opposite sentiment from the one you wrote. Russ says you can get the benefits of systemctl and journalctl if you want, but you can also use syslog in the old-fashioned way if you want. This has been my experience with systemd as well. I still have a habit of writing "service postfix restart", which still works but is not the systemd-idiomatic way. I don't even have the "correct" way memorized because the old way that I'm used to using works.So what I'm asking is, have you tried to achieve something with systemd and found that you were forced to do it differently? I'd be curious to learn more about what those cases were. I'm not affiliated with systemd project, but I have a general interest in wanting to see it improve.
(Please note that, barring conventions of some kind that span multiple tools, every tool naturally requires you to do things that tool's own way. If you are holding systemd to the standard that its job should be able to be performed by two different tools, each with the same interface, while supporting advanced features, then you should have other examples of this as well.)
[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...
[2] https://lists.debian.org/debian-ctte/2013/12/msg00234.html
Esau|9 years ago
kalenx|9 years ago
aaronky|9 years ago
seanp2k2|9 years ago
Disclosure: I liked this so much that I gave the developer money.
emmelaich|9 years ago
So, not sure what you have to do to not be 'opaque'
throw2016|9 years ago
But there were questionable tie-ins with various pieces like udev, consolekit and even gnome that allowed Systemd to become defacto init. The call for a kernel bus promotes a similar lock in with Systemd and this makes the use or development of alternatives and choice difficult.
There are things like predictable network names which are useful for 1% of users and are anything but predictable. Binary logging makes sense for the security industry Redhat serves but again has no use for the 99% others who anyway have to put up with it. There is a pattern of forcing things onto everyone that make sense for a tiny minority.
The big problem is open source funding. No one is interested in just supporting projects they benefit from. Acquisitions or hiring developers put these projects and developers under the control of companies like Redhat. Redhat has become a cathedral and a cathedral by sheer size and nature is always interested in securing and furthering its own influence and interests. When you allow such forces to become too powerful they will subsume the public interest to their interest.
ausjke|9 years ago
I just hope Debian to get rid of SystemD and return to whatever else. I know I'm biased, just failed to find a reason to love SystemD, tried a few times.
hedora|9 years ago
I'm also rotating BSDs and open source solaris variants on to the home network machines. (Switching to solaris to get rid of systemd would be overkill; I switched to joyent triton for better containers and zfs...)
digi_owl|9 years ago
The split being between those that embraced Linux for being a free, in both senses, _nix unburdened by AT&T and running on commodity hardware, and those that got to know it after the dot-com crash as the L in LAMP.
The former cares for Linux as a _nix, the latter could not care less about _nix and may see it as a vestigial appendage that should have been amputated long ago.
teddyh|9 years ago
I think, however, that you have a point in that some people ascribe some mystical properties to the design of _nix and _nix-like systems. I have never done this, perhaps because I remember the early days of OS/platform wars (Amiga or Atari? PC or console? Unix or VMS?), first on BBSes, then on Usenet. I also, early on, read the Unix-Haters Handbook, and I found it enlightening, even though it is not always spot-on.
I leave all die-hard advocates with this quote:
“All Software Sucks, All Hardware Sucks”
(http://www.absurdnotions.org/an20010622.gif)
(Source: (http://www.absurdnotions.org/page75.html). Start of storyline: (http://www.absurdnotions.org/page74.html))
vidarh|9 years ago
AnonymousPlanet|9 years ago
But, yes, I have the impression that a lot of Linux users would just as gladly use a modern BeOS or a ReactOS, regardless of the underlying design.
And a lot of systemd criticism comes from people who are just averse to any kind of change and maybe long for the bygone days of HP-UX and Irix.
secabeen|9 years ago
plorg|9 years ago
pkaye|9 years ago
weberc2|9 years ago
I know almost nothing about systemd, but you can define environment variables inline instead of in a file: `Environment=ENV_VAR=value`.
svennek|9 years ago
AstralStorm|9 years ago
And it doesn't do anything OpenRC couldn't do, in fact its journald is a pain which had to be worked around. OpenRC way of doing socket activations and dbus activations also works slightly better (in case something crashes) It does parallelism just as well, service dependencies too.
Gentoo does not mangle anything related to systemd unlike the mentioned distributions.
_ZeD_|9 years ago
ahem slackware has not adopted systemd
yjftsjthsd-h|9 years ago
Sir_Cmpwn|9 years ago
kasabali|9 years ago
- Handwaves real criticisms and instead tries to look like an objective writing by presenting a trivial issue
- Have no idea about advances in sysvinit in the last decade that allowed support for parallel boot and dependency relations (and which was supported natively on mainstream distribution)
- Totally ignores the other "modern" and widely used init system and instead compares systemd with sysvinit of 80's (which wasn't used by any mainstream distro in that form)
- Overrates process supervision features which were already as equally as easy to use with supervision suites like runit (IMHO even easier)
Meh, what did I expect? Bandwagon is going full speed.
z3t4|9 years ago
yrro|9 years ago
If not overriden in an individual service's unit file, perhaps you have set DefaultStandardError= in /etc/systemd/system.conf?
nailer|9 years ago
Everything I find just says to install syslog, which surprises me.
nailer|9 years ago
JdeBP|9 years ago
baybal2|9 years ago
[deleted]
Sembiance|9 years ago
[deleted]
unknown|9 years ago
[deleted]
unknown|9 years ago
[deleted]
nvarsj|9 years ago
msimpson|9 years ago