Locality of behavior is important. Note how this systemd-thingy is split into two separate files? It's like that a lot in systemd, whoever is doing the architecture doesn't seem to understand why locality of behavior would be desirable, instead taking the IDE approach of writing a bunch of tools to make managing the increased complexity more manageable. Don't glance at a crontab to see when things will next execute, run some other tool that will inspect everything for you. Make things more complex and difficult to read, and then write tools to make it easier to read again.
They could have just added "periodic" service type that accepts timer options, and at least then it wouldn't have to be two files.
Most of the advantages really don't seem like advantages to me. Templated unit files seems especially silly since it's equivalent to just copy and pasting a line in your crontab and changing an argument.
Seems to work for a lot of people, it's just not to my tastes I guess. Still, I find the whole thing pretty mystifying.
Even your own link talks about the Subjectivity of that opinion, and how LoB is often in conflict with DRY and SoC, both of which I support far more than LoB.
In this context systemd favors Separation of Concerns, the the service is what is being run, and the timer is when is it being run...
it is not that the devs do not "seem to understand why locality of behavior would be desirable" it is they disagree that is more desirable than having a logical separation of concerns...
To me I prefer greater separation of concern than locality
Binary log files is another example of something that could have advantages in theory but never seems to work well in practice. What used to be "tail /var/log/nginx.log" is now
man journalctl
journalctl -u nginx
<G to skip to latest, ctrl+C because it's taking too long>
journalctl | grep nginx
<wait...>
go back to man page to look for more options to search
It gets the "one file to define the job" right but that's the only thing that it gets right.
There is no holistic view on the jobs so you can't easily say "show me last jobs that failed" and shove that into alert.
There is email on failure but only on failure so you can't also do simple "if this mail returns fail, mark check as failed, if it returns okay, mark it as okay". You can get that info from logs but you can't get the content of error message of the app from those logs...
Environmental options history is.... interesting, better to just make script that sets everything up beforehand instead of even trying to do it via cron.
From ops perspective of managing multiple servers its utter garbage. From single user perspective it kinda works but is annoying
Systemd's timers should just be [Timer] block inside the .service
Some people just seem to love reinventing square wheels. I suspect this "spiral of complexity" is largely the result of people trying to justify their salary, because it lends itself to generating a lot of otherwise unnecessary work.
Starting a service is way different than the service itself, what if I want it to be started when a request hits a given port? Also, the service files themselves are usually packaged up and are immutable, and every user will want different behavior on when it is started.
To me it sounds like proper architecture and your solution would be needless close coupling.
This split between service file (what is going to happen exactly) and a timer file (when it's going to happen) is ok for complex use cases, but it really feels a bit silly when a single cron line would do the trick instead.
NixOS fortunately solves this problem. Here's a daily rsync with some names changed to protect any innocent hosts on my network. Note that the contents of "sync.sh" could easily go in there as well, but I was previously running it via cron, so didn't bother to move that over.
Counter argument: Try implementing support for Alpine, Debian and other distros whilst using a cronjob.
It's impossible. All that opinionated /etc/crontab and /etc/cron.daily shit can go to hell for all I care. Every opinionated distro uses whatever kind of file format that the next best sh guru came up with, which id also prone for exploitation btw.
With systemd there's a failsafe way for maintainers to offer a cross-distro way to integrate their software.
And that's what systemd is about: automation of integration through convention.
> Don't glance at a crontab to see when things will next execute, run some other tool that will inspect everything for you. Make things more complex and difficult to read, and then write tools to make it easier to read again.
systemctl list-timers is vastly more informative in a practical sense than just looking at the crontab. It just straight up tells you exactly when it ran last, how long ago that was, when it will run next, and how much time is left until that happens.
'Make everything simple as it can be but no simpler'. To me your ask violates this.
Timers augment the variety of existing service types, so your idea just doesnt work straight up, unless we severely limit the flexibility of what timers can be via the 'periodic' type. Or we make 'periodic-oneshot' 'periodic-serice' et cetera, which feels absurd.
Perhaps we could leave Service= alone & just shoehorn each systemd.timer option into system.service. But then we need some way to view the timer statuses & logs, which we can easily review and handle by having a separate unit. 'list-timers' is ok & might still work but we cant filter and log as well if everything is jammed together. We also cant just disable a timer temporarily, & go manage the service manually for a bit, if somethings going sideways & we need to step in; the two are now one in your world.
And what if you want multiple timers, with different configurations between them (maybe one long running one with WakeSystem= and a shorter one without?). We eliniate a lot of creative ability by glomming stuff together.
I dont like this idea. I think it's a misservice to jam everything together. Systemd has similar patterns of representing different pars in different places that I think has served it well. It's an even more trivial case that is more supportable by this idea you've floated, but there is a systemd.mount and systemd.automoumt. It's just good, as an operator, having clear boundaries between these units, and has always made it clearer to me what aspect Im operating on, and enabled healthier patterns of system growth.
I guess preferences will mostly depend on how the system is being used. If this is your desktop machine sure, do crontab -e and add a line. If it is a system sitting in a colo or cloud provider, it's likely that it's been configured through automation anyways so "locality" doesn't matter. Your scripts know where stuff is located.
While I agree Systemd timers aren't as ergonomic to set up - there are some benefits that I would basically never give up at this point.
* Running the service on command ( this is the best way to troubleshoot issues with your slightly different environments / cron's version of shell. I've spent many minutes over my career - setting the crontab one minute and the future and waiting for it to run / fail )
* Easily inspecting when the next run of the service is. ( list-timers )
* Straightforward access to the logs ( Always journalctl )
Speaking on OnFailure, one of the unfortunate aspects of systemd timers (which are otherwise quite nice) is you need to do extra work to get proper emails when something fails. Here's how I do this:
Define email-unit-status@.service
[Unit]
Description=Email status for %i to root
After=network.target
[Service]
Type=oneshot
ExecStart=email-unit-status %i
Define email-unit-status (a script)
#!/usr/bin/env bash
set -eu
dest="admin@whatever.com"
userflag=
if [[ $HOME == /home/* ]]; then
userflag=--user
fi
html=$(SYSTEMD_COLORS=1 systemctl $userflag status --full "$1" | ansi2html -w -c)
/usr/sbin/sendmail -t <<ERRMAIL
To: $dest
From: systemd <${USER}@${HOSTNAME}>
Subject: SystemD unit failure alert: $1
Content-Transfer-Encoding: 8bit
Content-Type: text/html; charset=UTF-8
Mime-Version: 1.0
$html
ERRMAIL
Then you can add to a service:
OnFailure=email-unit-status@%n
You will need to install the ansi2html tool. This stuff should come with systemd really.
I seemed to never care until they took my sysvinit away (or maybe any decent init system :D). Then they took resolv.conf control away. Then when I shutdown my computer, some random process takes 1 min 30 seconds to shutdown when it probably doesn't need to. Then I read on hacker news recently that Fedora uses a systemd daemon for handling OOM and the writer said it was terribly misconfigured particularly when it shutdown his X session and all processes related to it when an OOM condition happened. I am not a Linux admin (or at least a sophisticated one so I can look smart with systemd), but now they are taking cron away too? :D
I kid about this, in a way, and I know I should accept the inevitable, but I feel like just moving to Devuan on my laptop and use a nice init system, like OpenRC :D
Cron requires manual locking, which results in an old service sometimes aborting without cleaning that up, and then refusing to start ever since.
I'm precisely in the middle of an update that's going to throw that out, and just change to a systemd service/timer. Then that problem will go away for good.
Funnily at my previous job once upon a time the cron daemon crashed for some obscure reason, so the default monitoring template for years afterwards included a check if the cron process is up and running.
This is exactly what happens, just like the old XKCD: Now there are N+1 competing standards.
It's why I have to check /etc/profile, ~/.profile, /etc/profile.d/*, ~/.bashrc (or whatever shell), /etc/environment, ~/.env, and a few others I don't even remember to figure out why an $ENVVAR is set.
I have to look at two cron-like things to figure out what's scheduled when. systemd-timers has been around and in use for a long while.
I've had the opportunity to delve into systemd in a previous role.
This sounds a bit weird, but its not easy to get started with, unless you're used to reading man pages. But once you're able to read man pages, all of systemd behavior is documented in the man pages. I do wish there were more human friendly docs though, especially for younger engineers used to better sorts of docs.
Anyways, once you do get the hang of it, its an immensely powerful system, but like any such system it has its quirks and edge cases. You can get 99% of what you need with it though, and for low level tasks on machines, its pretty amazing.
Some time ago I wanted the best bits from both worlds:
- from cron: specifying all jobs in one file instead of scattering it across dosens of unit files. In 90%
of cases I just want a regular schedule and the command, that's it
- from systemd: mainly monitoring and logging. But also flexible timers, timeouts, resource management, dependencies -- for the remaining 10% of jobs which are a little more complicated
So I implemented a DSL which basically translates a python spec into systemd units -- that way I don't have to remember systemd syntax and manually manage the unit files. At the same time I benefit from the simplicity of having everything in one place.
An extra bonus is that the 'spec' is just normal python code
- you can define variables/functions/loops to avoid copy pasting
- you can use mypy to lint it before applying the changes
- I have multiple computers that share some jobs, so I simply have a 'common.py' file which I import from `computer1.py` and `computer2.py` -- the whole thing is very flexible.
I've been using this tool for several year now, with hundreds of different jobs across 3 computers, and it's been working perfectly for me. One of the best quality of life improvements I've done for my personal infrastructure.
The one time I've tried to use this in anger, it said my script was running when it was supposed to and exiting cleanly, everything green, but nothing was happening.
Put it in cron, worked first try, and forgot about it.
When I set up a new box, the first thing I do is disable all sorts of systemd-* nonsense, keeping systemd as an initd service only (which it should be). Then I replace them with serious, battle-tested tools like ntp/chrony, unbound, etc...
It looks like systemd.timer will continue this practice ;) I'm expecting someone at Canonical/IBM/RH will be happy to put this as the default scheduler in the next distro release, marking it as a "significant improvement".
Systemd and NixOS feels like a great fit. It is easier to configure than the real thing and also short scripts are easier to tie together with services and timers. All in one file instead of 2 or 3.
Nix/Guix users would think that way. They've alrady given up on the concept of system libraries or a unified OS. systemd's writing of 2 config files per timer would be right at home for people that have to set up an entire OS environment every time they want to try to compile something somone else hasn't already written a nix script for.
> Job output will automatically be written to systemd-journald
This is a bad thing, not something I’d boast about.
I actually really enjoy using systemd. Being able to start a process with complete isolation without heavyweight docker downloading mystery meat off the internet is a huge boon.
However, one thing systemd is logging. Jounalctl sucks. Grep, cat, etc, work infinitesimally better.
'Infinitesimally' means 'by a negligible amount'. Which is (accidentally) accurate. There's nothing preventing you from using grep, cat, etc. with the output of journalctl. It's just a tool in the pipeline.
I never quite understood that criticism, since systemd is extremely unapologetically Linux centric.
It uses all the cool features of the kernel. It's configured with text files, and has an override system that works great with the Linux filesystem layout and package manager -- much better than SysV init, by the way. And it handles a lot of little annoyances that have long been a thorn in an admin's/user's side on Linux.
God, I wish Linux would copy more Windows features. Service management before systemd was a hell of intermingled shell scripts (that often specified sh but only ran on bash) and side effects everywhere. Cron is a very stable tool but it should've been replaced when the first GUI ran on Linux.
I don't understand this idolisation of 70s mainframes that "hardcore" Linux users all seem to fight for. Systemd features can all be turned off if you want them to, but every day more and more alternatives go into maintenance mode because nobody uses them anymore. Inetdwas great when it was invented but we've moved past that point more than ten years ago.
When I'm tinkering, I love the great hacks I can apply by modifying system shell scripts, but I mostly just want my computers to work for me. The old ways of randomly placed dot files, custom service scripts (and formats) and modifying scripts that I shouldn't need to ever touch anyway because they conflict with another cobbled together script have caused me more headaches than systemd ever has. I wish system's docs were easier to find, but even the obscure man page names are better than comments in bash scripts somewhere in /etc/init.d.
traverseda|3 years ago
Locality of behavior is important. Note how this systemd-thingy is split into two separate files? It's like that a lot in systemd, whoever is doing the architecture doesn't seem to understand why locality of behavior would be desirable, instead taking the IDE approach of writing a bunch of tools to make managing the increased complexity more manageable. Don't glance at a crontab to see when things will next execute, run some other tool that will inspect everything for you. Make things more complex and difficult to read, and then write tools to make it easier to read again.
They could have just added "periodic" service type that accepts timer options, and at least then it wouldn't have to be two files.
Most of the advantages really don't seem like advantages to me. Templated unit files seems especially silly since it's equivalent to just copy and pasting a line in your crontab and changing an argument.
Seems to work for a lot of people, it's just not to my tastes I guess. Still, I find the whole thing pretty mystifying.
phpisthebest|3 years ago
Even your own link talks about the Subjectivity of that opinion, and how LoB is often in conflict with DRY and SoC, both of which I support far more than LoB.
In this context systemd favors Separation of Concerns, the the service is what is being run, and the timer is when is it being run...
it is not that the devs do not "seem to understand why locality of behavior would be desirable" it is they disagree that is more desirable than having a logical separation of concerns...
To me I prefer greater separation of concern than locality
ghostpepper|3 years ago
man journalctl journalctl -u nginx <G to skip to latest, ctrl+C because it's taking too long> journalctl | grep nginx <wait...> go back to man page to look for more options to search
etc
ilyt|3 years ago
It gets the "one file to define the job" right but that's the only thing that it gets right.
There is no holistic view on the jobs so you can't easily say "show me last jobs that failed" and shove that into alert.
There is email on failure but only on failure so you can't also do simple "if this mail returns fail, mark check as failed, if it returns okay, mark it as okay". You can get that info from logs but you can't get the content of error message of the app from those logs...
Environmental options history is.... interesting, better to just make script that sets everything up beforehand instead of even trying to do it via cron.
From ops perspective of managing multiple servers its utter garbage. From single user perspective it kinda works but is annoying
Systemd's timers should just be [Timer] block inside the .service
bravetraveler|3 years ago
Speaking of behavior, let's look at cron.
Consider every time you've written one. There's a good chance that it failed for some time because you forgot PATH is make believe there
Or logging. How many times has rotation been rewritten (and broken)?
Without digging through logs that may or may not exist, can you tell me when it last ran successfully?
I can with timers, 'for free'
userbinator|3 years ago
kaba0|3 years ago
To me it sounds like proper architecture and your solution would be needless close coupling.
marbu|3 years ago
lazide|3 years ago
aidenn0|3 years ago
cookiengineer|3 years ago
It's impossible. All that opinionated /etc/crontab and /etc/cron.daily shit can go to hell for all I care. Every opinionated distro uses whatever kind of file format that the next best sh guru came up with, which id also prone for exploitation btw.
With systemd there's a failsafe way for maintainers to offer a cross-distro way to integrate their software.
And that's what systemd is about: automation of integration through convention.
dralley|3 years ago
systemctl list-timers is vastly more informative in a practical sense than just looking at the crontab. It just straight up tells you exactly when it ran last, how long ago that was, when it will run next, and how much time is left until that happens.
rektide|3 years ago
Timers augment the variety of existing service types, so your idea just doesnt work straight up, unless we severely limit the flexibility of what timers can be via the 'periodic' type. Or we make 'periodic-oneshot' 'periodic-serice' et cetera, which feels absurd.
Perhaps we could leave Service= alone & just shoehorn each systemd.timer option into system.service. But then we need some way to view the timer statuses & logs, which we can easily review and handle by having a separate unit. 'list-timers' is ok & might still work but we cant filter and log as well if everything is jammed together. We also cant just disable a timer temporarily, & go manage the service manually for a bit, if somethings going sideways & we need to step in; the two are now one in your world.
And what if you want multiple timers, with different configurations between them (maybe one long running one with WakeSystem= and a shorter one without?). We eliniate a lot of creative ability by glomming stuff together.
I dont like this idea. I think it's a misservice to jam everything together. Systemd has similar patterns of representing different pars in different places that I think has served it well. It's an even more trivial case that is more supportable by this idea you've floated, but there is a systemd.mount and systemd.automoumt. It's just good, as an operator, having clear boundaries between these units, and has always made it clearer to me what aspect Im operating on, and enabled healthier patterns of system growth.
outworlder|3 years ago
luckycharms810|3 years ago
* Running the service on command ( this is the best way to troubleshoot issues with your slightly different environments / cron's version of shell. I've spent many minutes over my career - setting the crontab one minute and the future and waiting for it to run / fail )
* Easily inspecting when the next run of the service is. ( list-timers )
* Straightforward access to the logs ( Always journalctl )
* Specifying environment variables easily
* OnFailure
jamespwilliams|3 years ago
* Built in functionality for telling when the cron last ran, and whether it was successful or not (so you can do alerting)
* Ability to specify human-readable schedules
* Built-in feature for randomising cron start time to avoid stampeding
mike_hearn|3 years ago
Define email-unit-status@.service
Define email-unit-status (a script) Then you can add to a service: You will need to install the ansi2html tool. This stuff should come with systemd really.sparker72678|3 years ago
Seriously, though, cron has been so utterly reliable for me for so many decades now, it will be really hard for me to ever give it up.
bigpeopleareold|3 years ago
I kid about this, in a way, and I know I should accept the inevitable, but I feel like just moving to Devuan on my laptop and use a nice init system, like OpenRC :D
dale_glass|3 years ago
Cron requires manual locking, which results in an old service sometimes aborting without cleaning that up, and then refusing to start ever since.
I'm precisely in the middle of an update that's going to throw that out, and just change to a systemd service/timer. Then that problem will go away for good.
phpisthebest|3 years ago
Tradition: Just because you've always done it that way doesn't mean it's not incredibly stupid. -- Despair, inc
Every dev and administrator, or office user should have that poster above their desks
sofixa|3 years ago
eulers_secret|3 years ago
It's why I have to check /etc/profile, ~/.profile, /etc/profile.d/*, ~/.bashrc (or whatever shell), /etc/environment, ~/.env, and a few others I don't even remember to figure out why an $ENVVAR is set.
I have to look at two cron-like things to figure out what's scheduled when. systemd-timers has been around and in use for a long while.
pm90|3 years ago
This sounds a bit weird, but its not easy to get started with, unless you're used to reading man pages. But once you're able to read man pages, all of systemd behavior is documented in the man pages. I do wish there were more human friendly docs though, especially for younger engineers used to better sorts of docs.
Anyways, once you do get the hang of it, its an immensely powerful system, but like any such system it has its quirks and edge cases. You can get 99% of what you need with it though, and for low level tasks on machines, its pretty amazing.
dinosaurdynasty|3 years ago
karlicoss|3 years ago
- from cron: specifying all jobs in one file instead of scattering it across dosens of unit files. In 90% of cases I just want a regular schedule and the command, that's it
- from systemd: mainly monitoring and logging. But also flexible timers, timeouts, resource management, dependencies -- for the remaining 10% of jobs which are a little more complicated
So I implemented a DSL which basically translates a python spec into systemd units -- that way I don't have to remember systemd syntax and manually manage the unit files. At the same time I benefit from the simplicity of having everything in one place.
An extra bonus is that the 'spec' is just normal python code
- you can define variables/functions/loops to avoid copy pasting
- you can use mypy to lint it before applying the changes
- I have multiple computers that share some jobs, so I simply have a 'common.py' file which I import from `computer1.py` and `computer2.py` -- the whole thing is very flexible.
You can read more about it here:
- https://beepb00p.xyz/scheduler.html
- https://github.com/karlicoss/dron#what-does-it-do
I've been using this tool for several year now, with hundreds of different jobs across 3 computers, and it's been working perfectly for me. One of the best quality of life improvements I've done for my personal infrastructure.
yamtaddle|3 years ago
Put it in cron, worked first try, and forgot about it.
dig1|3 years ago
It looks like systemd.timer will continue this practice ;) I'm expecting someone at Canonical/IBM/RH will be happy to put this as the default scheduler in the next distro release, marking it as a "significant improvement".
ciupicri|3 years ago
Same with the last - templated unit files. Add a parameter or more to the script and you're done.
whateveracct|3 years ago
mongol|3 years ago
superkuh|3 years ago
tb_technical|3 years ago
yawaramin|3 years ago
hulitu|3 years ago
This is a trend in SW: facelift every couple of years.
unknown|3 years ago
[deleted]
exabrial|3 years ago
This is a bad thing, not something I’d boast about.
I actually really enjoy using systemd. Being able to start a process with complete isolation without heavyweight docker downloading mystery meat off the internet is a huge boon.
However, one thing systemd is logging. Jounalctl sucks. Grep, cat, etc, work infinitesimally better.
yawaramin|3 years ago
kaba0|3 years ago
mnd999|3 years ago
Gualdrapo|3 years ago
dale_glass|3 years ago
And it's not half-bad, I think, because GRUB2 is just way, way more complicated than most setups need these days.
unknown|3 years ago
[deleted]
smm11|3 years ago
[deleted]
dale_glass|3 years ago
It uses all the cool features of the kernel. It's configured with text files, and has an override system that works great with the Linux filesystem layout and package manager -- much better than SysV init, by the way. And it handles a lot of little annoyances that have long been a thorn in an admin's/user's side on Linux.
jeroenhd|3 years ago
I don't understand this idolisation of 70s mainframes that "hardcore" Linux users all seem to fight for. Systemd features can all be turned off if you want them to, but every day more and more alternatives go into maintenance mode because nobody uses them anymore. Inetdwas great when it was invented but we've moved past that point more than ten years ago.
When I'm tinkering, I love the great hacks I can apply by modifying system shell scripts, but I mostly just want my computers to work for me. The old ways of randomly placed dot files, custom service scripts (and formats) and modifying scripts that I shouldn't need to ever touch anyway because they conflict with another cobbled together script have caused me more headaches than systemd ever has. I wish system's docs were easier to find, but even the obscure man page names are better than comments in bash scripts somewhere in /etc/init.d.
MikusR|3 years ago