top | item 39322248

(no title)

gorset | 2 years ago

I struggle to use lmdb when using resource isolation (like docker, nomad exec driver). The hidden complexities really become apparent when you build largish systems using shared infrastructure. Anything is easy if you have a dedicated machine or have dataset smaller than your memory allocation.

Mmap is so simple for the application/developer, but the price you pay is giving up clear ownership of memory and clean separation between kernel page cache and application space. The interaction between the mmu, mmap, tlb and the linux kernel IO layer is complicated. You don't want to touch that complexity at all if you care about ultra low latency.

The article rephrases the question into

    "do you really want to reimplement everything the OS already does for you?"
but I would rephrase it again into questioning why fairly straight forward concepts like a buffer cache can't be implemented directly in the application?

discuss

order

natmaka|2 years ago

> You don't want to touch that complexity at all if you care about ultra low latency.

This is less about a given criterion (such as ultra low latency) than about an adequate global performance (multiple requirements: max latency, average latency, perceived interactive latency, throughput...) on a high-yield machine (more often than not the implication being that it simultaneously has many different and changing missions).