Out of Memory Shenanigans

[jeffbee]

Note to anyone who thinks they have "disabled swap": most programs, unless they go out of their way to remap themselves, are backed by a file on disk, and Linux will cheerfully discard the executable text pages of the program to reclaim, and then immediately have to read them back in from disk. This is at least as bad as swap, and worse in some ways.

[zokier]

One of the key observations is that you should basically always run your applications with some memory limits so that the system and management services will remain responsive etc even if the application is misbehaving. Luckily stuff like docker and systemd makes setting limits pretty easy

[omerban]

Writer of the blog post here. Great point, that's exactly what we did. Setting a "C Group" in docker compose is easily done with the "mem_limit" field.

[iTokio]

TLDR:

- OOM will swap or trigger the OOM killer but Linux might try to reclaim memory from caches before and slow down (using a cgroup limit solved this).

- glibc malloc can easily cause memory fragmentation with many threads (switching to Jemalloc solved this)

[omerban]

A couple of points I would add here:

- You should regulate your process's memory usage and throttle when needed, even at the cost of performance.

- Memory regulation is harder in a multi-process architecture because you can't set a memory limit on the total memory usage of all the processes.