[EdiX]

This is completely wrong. First, disabling overcommit is wasteful because of fork and because of the way thread stacks are allocated. Sorry, you don't get exact memory accounting with C, not even Windows will do exact accounting of thread stacks.

Secondly, memory is a global resource so you don't get local failures when it's exhausted, whoever allocates first after memory has been exhausted will get an error they might be the application responsible for the exhaustion or they might not be. They might crash on the error or they might "handle it", keep going and render the system completely unusable.

No, exact accounting is not a solution. Ulimits and configuring the OOM killer are solutions.

[inkyoto]

Without going into a discussion about whether this is right or wrong, how is fork(2) wasteful?

fork(2) has been copy-on-write for decades and does not copy the entire process address space. Thread stacks are a non-sequitur either as stack uses the data pages, the thread stacks are rather small in size in most scanarios, hence the thread stacks are also subject to copy-on-write.

The only overhead that the use of fork(2) incurs is an extra copy of process' memory descriptior pages, which is a drop in the ocean for modern systems for large amounts of RAM.

[barchar]

Unless you reserve on fork you're still over committing because after the fork writes to basically any page in either process will trigger memory commitment.

Thread stacks come up because reserving them completely ahead of time would incur large amounts of memory usage. Typically they start small and grow when you touch the guards. This is a form of overcommit. Even windows dynamically grows stacks like this

[otabdeveloper4]

> because of fork and because of the way thread stacks are allocated

For modern (post-x86_64) memory allocators a common strategy is to allocate hundreds of gigabytes of virtual memory and let the kernel handle deal with actually swapping in physical memory pages upon use.

This way you can partition the virtual memory space into arenas as you like. This works really well.

[zrm]

Which is a major way turning off overcommit can cause problems. The expectation for disabling it is that if you request memory you're going to use it, which is frequently not true. So if you turn it off, your memory requirements go from, say, 64GB to 512GB.

Obviously you don't want to have to octuple your physical memory for pages that will never be used, especially these days, so the typical way around that is to allocate a lot of swap. Then the allocations that aren't actually used can be backed by swap instead of RAM.

Except then you've essentially reimplemented overcommit. Allocations report success because you have plenty of swap but if you try to really use that much the system grinds to a halt.

[kccqzy]

Yeah but these only request address space. The memory is neither readable nor writable until a subsequent mprotect call. Ideally reserving address space only shouldn’t be counted as overcommit.