[codaphiliac]

Wonder how many sqlite databases would be too many. At one point I assume not all databases can be kept opened at all time. what sort of overhead would there be serving a tenant not opened up yet? there has to be caches etc. not warmed up causing lots of disk IO

[toast0]

At some level, it doesn't make a big difference if you've got a file open or not once the file's data falls out of the disk cache, you'll have the same kinds of latency to get it back. Sure, you'll avoid a round or two of latency pulling in the filesystem data to get to the file, but probably not a big deal on SSD.

Chances are, the popular databases stay in cache and are quick to open anyway; and the unpopular ones are rarely accessed so delay is ok. But you'd also be able to monitor for disk activity/latency on a system level and add more disks if you need more throughput; possibly disks attached to other machines, if you also need more cpu/ram to go with it. Should be relatively simple to partition the low use databases, because they're low use.

[koakuma-chan]

In this scenario you would use short-lived connections, and the overhead would probably be approximately the same as reading a file.

[julik]

That would be the FD limit, divided by 3 (the DB itself, the shm file and the WAL).

[ncruces]

But each SQLite connection (even to the same DB) will also consume 2 FDs for the DB and the WAL.

You'll more easily pool connections to the same DB, of course, but the difference might not be so stark.