Advent of Sysadmin 2025

[0xbadcafebee]

Here's 12 Sysadmin/DevOps (they're synonyms now!) challenges, straight from the day job:

  1.  Get a user to stop logging in as root.
  2.  Get all users to stop sharing the same login and password for all servers.
  3.  Get a user to upgrade their app's dependencies to versions newer than 2010.
  4.  Get a user to use configuration management rather than scp'ing config files from their laptop to the server.
  5.  Get a user to bake immutable images w/configuration rather than using configuration management.
  6.  Get a user to switch from Jenkins to GitHub Actions.
  7.  Get a user to stop keeping one file with all production secrets in S3, and use a secrets vault instead.
  8.  Convince a user (and management) you need to buy new servers, because although "we haven't had one go down in years", every one has faulty power supply, hard drive, network card, RAM, etc, and the hardware's so old you can't find spare parts.
  9.  Get management to give you the authority to force users to rotate their AWS access keys which are 8 years old.
  10. Get a user to stop using the aws root account's access keys for their application.
  11. Get a user to build their application in a container.
  12. Get a user to deploy their application without you.

After you complete each one, you get a glass of scotch. Happy Holidays!

[cobertos]

Re: 6. ... Github Actions

Github Actions left a bad taste in my mouth after having it randomly removed authenticated workers from the pool, after their offline for ~5 days.

This was after setting up a relatively complex PR workflow (always on cheap server starts up very expensive build server with specific hardware) only to have it break randomly after a PR didn't come in for a few days. And no indication that this happens, and no workaround from GitHub.

There are better solutions for CI, GitHub 's is half baked.

[jagged-chisel]

> … from Jenkins to GitHub Actions.

Oh, good lord why?

[n4bz0r]

> Sysadmin/DevOps (they're synonyms now!)

I've notified the authorities and social services.

[betaby]

5. and 6. are a matter of taste (trade-offs), the rest is spot on!

[daemonologist]

You get me the permissions to do half of this stuff, and I'll do whatever you want.

[Waterluvian]

Here’s the first step to all of these that I often see sysadmins stumbling on: communicate in written, non-abstract terms why each of these matter.

Most are obvious to most people. None are obvious to everybody.

[Nextgrid]

> Get a user to stop logging in as root.

It really depends if the machine is hosting anything that you don't want some users to access. If the machine is single-purpose and any user is already able to access everything valuable from it (DB with customer data, etc) or trivially elevate to root (via sudo, docker access, etc) then it's just pointless extra typing and security theatre.

[f1shy]

>> Sysadmin/DevOps (they're synonyms now!)

Is this really like that? Isn't there any Unix/DBA anymore? I associate DevOps to what at my time we called "operations" and "development". We had 5 teams or so:

1) Developers, who would architect and write code, 2) Operations who would deploy, monitor and address customer complaints, 3) Unix (aka SYS) administrators, who would take care of housekeeping of well, the OS (and web servers/middleware), 4) DBA who would be monitoring and optimizing Oracle/Postgres, and 5) Network admins, who would take care of Load Balancers, Routers, Switches, Firewalls (well, there were 2 security experts for that also)

So I think DevOps would be a mix of 1&2, to avoid the daily wars that would constantly happen "THEY did it wrong!"

Can somebody clear my mind, please!? It seems I was out of it for too long?!

[technion]

I know its a common view that sysadmin/devops are the same these days, but witha current sysadmin role nothing youve mentioned sounds relevant. Let's give you my list:

1. Patch Microsoft exchange with only a three hour outage window 2. Train a user to use onedrive instead of emailing 50mb files and back and forth 3. Setup eight printers for six users. Deal with 9gb printer drivers. 4. Ask an exec if he would please let you add mfa to their mailbox. 5. Sit there calmly while that exec yells like a wwe wrestler about the ways he plans to ruin you in response 6. Debate the cost of a custom mouse pad for one person across three meetings 7. Deploy any standard windows app that expects everyone be an administrator without making everyone an administrator 8. Deploy an app that expects uac disabled without disabling uac 9. Debug some finance persons 9000 line excel function

[alberth]

I’d be super interested to see solutions to each, just to learn from.

[athrowaway3z]

  9.  Get management to give you the authority to force users to rotate their AWS access keys which are 8 years old.

Saying "keys which are 8 years old" implies you're worried about the keys themselves, which is just wrong. (Their security state depends on monitoring)

You can definitely make a strong argument that the organization needs practice rotating, so I would advise reframing it as an org-survivability-planning challenge and not a key-security issue.

[DoctorOW]

> Get a user to use configuration management rather than scp'ing config files from their laptop to the server.

Damn, this one I'm guilty of. Though, I'm not real Sysadmin/DevOps, I'm just throwing something together and deploying it on a LAN-only VM for security reasons (I don't trust the type of code I would write)

[infogulch]

Q: 3. Get a user to upgrade their app's dependencies to versions newer than 2010.

A: Calculate the average age in years of all dependencies calculated by: (max(most recent version release date, date of most recent CVE on library) - used version release date). Sleep for that many seconds before the app starts.

[JuniperMesos]

A lot of these problems seem pretty solveable, if you're the admin of the machine (or cloud system) and the user isn't.

If you don't want a user to log in as root, disable the root password (or change it to something only you know) and disable root ssh. If you want people to stop sharing the same login and password across all servers, there's several ways to do it but the most straightforward one seems like it would be to enforce the use of a hardware key (yubikey or similar) for login. If people aren't using configuration management software and are leaving machines in an inconsistent state, again there are several options but I'd look into this NixOS project: https://github.com/nix-community/impermanence + some policy of rebooting the machines regularly.

If you don't like how users are making use of AWS resources and secrets, then set up AWS permissions to force them to do so the correct way. In general if someone is using a system in a bad or insecure way, then after alerting them with some lead time, deliberately break their workflow and force them to come to you in order to make progress. If the thing you suggest is actually the correct course of action for your organization, then it will be worthwhile.

[AstroJetson]

I think the BOFH answer would be “They ride Elevator #2 to sub-basement 3.” Plot twist, there is only sub-basement 2.

Two pints of ale please!

[UltraSane]

Best practice is to use IP-restricted keys.

[melvinodsa]

When I get sad and nothing to do in the world, may be hacking into a sad server's problem seems very interesting

[alexpotato]

We use Sad Servers for evaluating candidates for DevOps/SRE roles and it's phenomenal.

Feedback from candidates is that they find it a bit stressful during the actual interview but love the approach once it's completed.

The interview option also makes it trivial to just send to a candidate via Zoom chat, ask them to share their screen and "just works".

Happy to answer questions folks may have about how we use it.

[zenoprax]

This is heartening - I'm about to start with the daily challenges today and document my experience and that sort of thing.

Any other suggestions? I have sysadmin experience as a homelabber and at work with a small company as a "tech lead" but have not yet had the chance to do it full time in a larger company. Currently focused on back-filling knowledge gaps and adding certs to support my existing experience.

[kralos]

    imagine typing in a terminal...
    you want to delete the previous word so press ctrl+w...
    actually you're in a browser; the window closes...

:sadness:

[melvinodsa]

We used to run terminal in browser using https://github.com/yudai/gotty and the entire dev team remapped their Ctrl+w to Ctrl+`. We did frontend and backend development with this setup almost for 1.5 years. Muscles memory and till this date, always have the fear if my actual terminal will get closed if I use Ctlr+w :P

[tambourine_man]

Which is why macOS command key is such an undervalued nicety. One key for GUI stuff, one for command-line stuff.

[protomikron]

You can use ctrl+shift+t to open the recently closed tab again.

[fduran]

hello, creator here, sorry about that. In this case you can click again on the "Open the Server Terminal in a New Window" button

[CoolCold]

I feel your pain - bites me from time to time, especially in KVM ;)

[scubbo]

Maybe I'm just extremely dumb, but I can't find how to edit files? Neither `vi` nor `nano` are installed, I don't have internet access to `apt-get update`, and I'm not about to learn `emacs` for this...

EDIT: Ah, ok, `vi` is installed on the server _itself_, just not in the Docker containers. So I guess I'm going to have to `docker cp` them in. Can do o7

[Erwyn]

Cool, might try it out! Are there any solutions repositories for them. I’d love to get an explanation for the ones I’m about to fail.

[gautamsomani]

Personal advice: don't use solutions repo. Googling the problem and then digging deep into the solutions will teach you hell lot more. Read the man pages of commands that turn up on Google, try them with different options, try to find different commands which can do almost the same thing may be a bit differently .... all these will help you learn things lot more.

[irusensei]

It seems it's called SRE nowadays right? I hate how things keep being renamed for no reason other than making more buzzwords for suits.

[phrotoma]

The definition I liked best, which I _think_ came from one of the Google SRE books though I'm not certain, was: "SRE is what happens when you consider operations to be a software problem".

[oarmstrong]

I share your disdain for buzzwords but SRE is definitely a different role.

[kortilla]

Nope, SREs keep applications running on a platform. Lots of metrics, tools to deploy apps in whatever rollout process the company has, etc.

In small companies, sysadmin might be a duty of the SRE team, but they definitely diverge if you have a large on-prem deployment or work with bespoke VMs in the cloud.

[teddyh]

[flagged]

[thatxliner]

well advent of code also needs an account

[fduran]

Checking out how the platform works was two clicks away: home -> give me a server.

I don't know of any other SaaS which gives you a VM with one click without any registration but we do it.

In any case thanks for the feedback, I've put a button on this /advent page for clarity, cheers

[fragmede]

how do you want it to work? do you even sysadmin?

[ofrzeta]

It doesn't seem to record my progress.

[fduran]

Hello, creator here. Have you checked your dashboard? otherwise please contact us (email or form in the website) and we'll be happy to help

[dontdoxxme]

Without sharing too many spoilers... I solved the challenge but the check script was unhappy. The curl commands in the script worked fine, the earlier parts of the script failed, i.e. it didn't like how I'd decided to make that work.

This kind of thing annoys me. This is why CTFs are great, where the goal is to get the flag string. Obviously harder to do for sysadmin, but expecting a particular configuration when I managed to make it work without doing things exactly as they wanted is no better than a poorly written exam.

[fduran]

hello, thanks for the feedback. Just deployed a new image that only checks for the objective, not at what docker network somebody uses.

It is hard to have a checker that eliminates both false positives and false negatives in general, but we always try to minimize false negatives and we failed initially here.

[truekonrads]

I absolutely love the sadservers. Can’t wait for windows version.

[fduran]

(Creator here) thanks! and we may at some point get into Windows :-)

[NooneAtAll3]

what's the deal with 12-days advent calendars lately?

[nstart]

Time pressures during christmas/holidays mean that the original calendars were becoming too stressful to handle. Seen several calendars switching to 12 consecutive days or 1 every 2 days challenges.

[aljaz823]

Advent of Code went from 25 days to 12 days starting this year, as it took too much time each year to come up with 25 unique challenges [1].

[1] https://adventofcode.com/2025/about#faq_num_days

[swyx]

aren't they canonically 12? 12 days of christmas etc

[fyltr]

Well, 12+12=24, so now we can complete two advents

[rconti]

Shrinkflation?

[tonyhart7]

now we need advent of arts,math etc

[dubya]

For math, the AMC 10 and AMC 12 tests have 25 questions each, some of them quite challenging. Both are high school level math, no calculus. Search "2025 amc 10" for this year's problems and solutions.

[udev4096]

I wonder if we could get something like that for k8s, docker and other container ecosystem

[fduran]

Hello, SadServers guy here.

We have scenarios running on k8s, both on single VMs (the ones you can see in the scenario list) and we also have a beta/PoC k8s cluster where we currently run a couple of scenarios as single pod (a docker container) or as a full system (the "kubernetes playgrounds", which is kind of hidden while we test it).

Is this what you were wondering? we do have pending to introduce podman scenarios as well

[gryfft]

Don't drag me into this.

[mekoka]

Could you elaborate?