top | item 20621589

(no title)

throwaway082729 | 6 years ago

Isn't the speed bottlenecked by the storage speed. Is the data fully loaded into memory first?

discuss

order

tmostak|6 years ago

OmniSci transparently caches data across the memory of the CPUs and GPUs on a server, so after the initial read, it is likely that the data for subsequent queries will be in memory.

We've also optimized our storage formats and multithreaded our disk reads, such that we can easily hit many gigabytes per second on flash storage. Plus, new persistent memory technologies like Intel Optane will enable even more instant reads from "cold" storage.