top | item 43477057

You might want to stop running atop

323 points| subract | 11 months ago |rachelbythebay.com

140 comments

order

dgacmu|11 months ago

There's a lot of speculation about why, with the answer almost certainly security / exploitable (or backdoor), and I'll just throw an extra little tidbit in:

atop seems to run persistently as root, which may be the reason for preventing it from running/uninstalling.

the netatop part of atop installs a persistent kernel module, netatop.ko, as part of its installation. The module hooks netfilter to be able to monitor all traffic.

If there's an exploitable flaw in the kernel module, this would be a max-severity CVE.

netatop _also_ runs a persistent daemon, netatopd, which I believe from inspecting the source runs as root.

The article's language about uninstalling it kinda sorta makes you think one of these three parts is in some way exploitable or backdoored -- any which way it's a privileged process, and one that's monitoring network traffic.

(I'm not sure if netatop is installed by default on systems when you install atop, per czk's comment below)

cesarb|11 months ago

> atop seems to run persistently as root, which may be the reason for preventing it from running/uninstalling.

Some distributions (like Ubuntu) enable that service by default, but some others (like Fedora) don't.

tptacek|11 months ago

How severe it would be would depend on how exploitable it was in likely configurations.

netham45|11 months ago

When we tried deploying it we had netatop crashing kernels with a use after free on a linked list, based on the stack traces and kernel dumps. Every box we trialed it on started going down multiple times a week.

czk|11 months ago

I'm not familiar with atop but the website mentions netatop is optional and what I've found suggests you have to manually install it. Do you know if any distributions/packages install this by default alongside the atop install?

AlexClickHouse|11 months ago

I vaguely remember an old bug in atop, leading to a very unusual consequence.

Atop will do an invalid memory write and crash with a segfault. But this writing is performed on a memory page mapped to a hardware timer. Despite not being able to write into that page, just touching it somehow changes how this hardware timer works. Then, the OS detects that this timer is inaccurate and switches to a different clock source (which you can see in /sys/devices/system/clocksource/clocksource0/current_clocksource). As a result, every call to clock_gettime becomes slower, and the system becomes slower as a whole until it restarts.

In short, a segfault in atop leads to the whole system's performance degradation. But this was found around maybe 7 years ago.

anitil|11 months ago

That is such an interesting bug!

jofzar|11 months ago

This screams NDA/disclosure but things are so mega super fucked that they feel obligated to pre warn as early as possible.

I wonder how long/old the problem is in atop?

plorkyeran|11 months ago

Yeah, from a rando this would be just bad vagueposting but Rachel is absolutely someone who could know about a very good reason why we should uninstall atop but be unable to legally say why. I would heed her warning.

mjevans|11 months ago

That last line for sure reads as '(author) can't tell you now, but can (plans to) tell you later'; NDA and/or CVE as most likely reasons.

rybosome|11 months ago

Agreed. Severe CVE seems like the ticket here given the context.

imoreno|11 months ago

Why would there be an NDA on atop? It's under GPL.

spudlyo|11 months ago

At a previous gig, atop was running fleet-wide (> 1k servers) as sort of a resource monitoring tool of last resort, in a similar way as is described in this article[0]. I left a few years ago, but if memory serves, this thing was baked into base-image Puppet configs, and proved itself handy in past investigations of hard-to-find problems. If this turns to be real threat, I wouldn't be surprised if the blast radius for this is substantial.

[0]: https://www.bodhost.com/kb/how-to-monitor-system-resource-us...

rdtsc|11 months ago

It's Rachel. If she says to remove it, I'll remove it. I see people are suspicious, but I think I'll trust someone like her at least once to do this.

agnishom|11 months ago

I am out of the loop. Who is Rachel, and what are they famous for?

TheDong|11 months ago

No one else seems to have run 'grep system(', so I will:

https://github.com/Atoptool/atop/blob/037a6d3e4ace6c7be6c5dc...

> system ("gunzip -c %s > %s", tmpname1, tmpname2")

tmpname2 is hardcoded as "/tmp/atopwrkXXXXXX", so that's fine. tmpname1 is '$irawname.gz'. '$irawname' is set by the '-r' flag.

So, presumably if you can get the rest of the code to play nice and get you there, you can escalate from having shell access to run atop, to having shell access. Oh, I guess that's nothing.

Anyway, still a really bad use of system + user-controlled input, don't do that.

lnxdork|11 months ago

Agree as a basic example. tmpname1 = "/tmp/file.txt; rm -rf /"; becomes gunzip -c /tmp/file.txt; rm -rf / > /tmp/atopwrkXXXXXX

Also tmpname2 could be symlinked to /etc/passwd before it is unlinked..

isotopp|11 months ago

Eh? Calling system() for a binary without a path? And why system() using execl() in the first place, when you could do something using execve() without a sh inbetween instead?

Even w/o an exploit this can be prettier and more secure.

emmelaich|11 months ago

There's a bunch of interesting recent commits from someone without a public signing key.

    Removed excess checks before free()
    Fixed possible wrong result bit shifting on 64bit after left op type overflow
    Fixed possible wrong result bit shifting on 64bit after left operand type overflow
    Fixed possible access out-of-bounds items array better check index before using
Could be legit or flawed. Or even fixes for the possible flaw.

TheDong|11 months ago

1. Unsigned commits is the norm. It's weird to sign git commits. It's weird to upload your gpg key to github. gpg is a nightmare mess.

2. They aren't introducing the bug, those are all unreleased commits, so advice to "uninstall now" for something no distros are shipping would be silly.

3. The diff is trivial, you can read it and figure out if it looks like they're fixing a real exploitable thing. The answer is obviously no.

wejick|11 months ago

Seems they also are not coming PR. Sus

teekert|11 months ago

I recently had a course from the author of atop. Seemed like a straight up FOSS friendly guy, I’ll forward him this page.

Hello71|11 months ago

I stopped using atop when I found it installs several hooks which automatically run code as root and deposit files around the filesystem, including a "power management" hook.

lolinder|11 months ago

Do you have any references that describe this behavior? That sounds like exactly the kind of thing that could conceal a backdoor of the sort this seems to be warning about.

bitbasher|11 months ago

Pure speculation; but it sounds to me like she was doing some sysadmin triage and possibly stumbled onto a backdoor/exfiltration through atop.

She likely can't disclose anything right now.

nodesocket|11 months ago

Except, she kinda did disclose already. Seems a bit strange to circumvent standard embargo practices, only to publicly hint of an exploit but not give any details.

keyle|11 months ago

Is atop included in any distributions?

Is there even a tool to search what is pre-installed in each major distribution(s)?

JamesLeonis|11 months ago

I can confirm my FreeBSD, Debian, and NixOS boxes don't have it installed by default. It's also not installed on my TrueNAS box.

Intralexical|11 months ago

Check in your available container images:

  docker images -q | xargs -I{} -t docker run --rm {} sh -c 'type atop && echo "DANGER!!!"'
May produce false negatives, because container images tend to be stripped down compared to desktop and server releases. Probably won't produce false positives, so use as a minimum.

I'd be surprised if any large distros shipped it in a stock configuration.

senectus1|11 months ago

not a default on my Debian bookwork, Ubuntu 24.04.2 LTS, Fedora 41, Proxmox 8.3.4 or OPNsense 25.1.3

vanc_cefepime|11 months ago

I typed 'atop' in my Linux Mint 22.1 laptop/desktop, says it's not found but can be installed. So I think Linux Mint is in the clear, I tried my Ubuntu 24.04 server and same thing there as well as my proxmox home lab instance. I checked that Repology link and I did see Ubuntu, but I guess that is for Ubuntu desktop but not sever edition?

ps. If I said anything wrong, please correct me. I'm a linux newb who jumped from Microsoft's world after getting fed up with their Win11 BS. I'm still learning quite a bit about linux daily.

LinuxBender|11 months ago

"Ubuntu, Debian, Red Hat Enterprise Linux, Fedora, Linux Mint, SUSE Linux Enterprise, CentOS, Manjaro, elementary OS, Gentoo, Oracle Linux, and Pop!_OS" ~--Google's AI.

I am not aware of any that install it by default.

imoreno|11 months ago

Luckily I use a much better *top, btop.

d3Xt3r|11 months ago

This. Not only that, I don't know of a single person (IRL or online) who used atop, like, ever. In fact, this is the first time I'm even hearing of atop.

IIRC, most folks went from top -> htop -> glances -> various btop variants (bashtop, bpytop, btop++ etc)

geerlingguy|11 months ago

atop was a happy medium between btop/htop and a full tracing tool like perf.

I used it when debugging deeper PCIe, CPU, or networking bottlenecks.

crmrc114|11 months ago

Well that ansible job was quickly ran, buhbye atop. Very concerning coming from Rachel and not some rando. I know a number of fortune 5's that use atop for troubleshooting as well. So as others have commented if you had this baked into images or loaded with puppet etc than now may be the time to cleanup.

geenat|11 months ago

Probably a backdoor.

Repositories controlled by accounts based in mainland China and Russia are always a risk- it's too easy for a dictatorship to force something to happen even if the authors themselves are trying to act in good faith.

XZ, Swoole... examples off the top of my head.

protocolture|11 months ago

> it's too easy for a dictatorship to force something

We really need to get rid of this mentality. Australia has laws that allow undisclosed, compelled, software updates. Verbally by ministers, but written (confidential) changes can be requested by federal agencies. Many western countries have followed to various degrees. There's no stable trusted government that doesn't want its fingers in your code.

tetromino_|11 months ago

Where did you see signs of control by Russia or China? The project's github repo states that the project currently has one maintainer, and that maintainer has a very Dutch name and a .nl website.

lionkor|11 months ago

What about the fact that software is hosted on US/German/Australian/whatever else platforms and infrastructure, what's different with that, technically speaking? The fact that a majority of software we rely on is hosted on GitHub, isn't that scary the same way that a repo owned by someone in a other country is scary?

Does a government need to openly act in a specific way for there to be a risk, or is this perceived risk due to a media bias?

I'm genuinely curious if there's a good answer

LorenDB|11 months ago

For openSUSE users: `sudo zypper al atop` will prevent atop from being installed for any reason, as long as it's uninstalled before you add the lock.

jaffa2|11 months ago

enigmatic? Anyone know why

subract|11 months ago

No clue personally, but the author is prolific enough here that I thought it merited posting.

iJohnDoe|11 months ago

Very simple. From a state level, if they are trying to compromise a system, get persistent access, already have access, but need to escalate, then atop is a solution if it's already on the system.

Just like Notepad++ back in the day.

nextts|11 months ago

Is there a mechanism where this sort of advice can flow through security teams to everyone (assuming it is about security) without dropping the details. How are zero days dealt with?

alsetmusic|11 months ago

I’m actually surprised I didn’t have it installed, what with all the packages I check out just through sheer curiosity. Thanks Rachel! I’ll avoid it in the future.

stavros|11 months ago

Alarmingly, I had it installed on my home server, for some odd reason. I don't remember ever using it.

vanc_cefepime|11 months ago

Linux newbie here. Jumped into the Linux world after getting tired of Microsoft's BS with Win 11. Running Linux mint on my laptop and desktop. Looks like 'atop' is not installed by default, but regular 'top'. Anyone know which distros I should be worried about that have it? Also I have been dabbling with proxmox, I checked and looks like 'top' is the default there too.

grayfaced|11 months ago

You're probably not running either unless you know what they are. Top is an equivalent of windows taskmanager, most often to used identify "top" processes using memory/cpu (and other resources) and only ran briefly. Atop is a different long-running version used to create logs of the same data to understand trends.

xyst|11 months ago

[deleted]

mvdtnz|11 months ago

Not everything is about American politics.

slicktux|11 months ago

Ominous, I’ll heed the warning!