top | item 37430797

(no title)

__all__ | 2 years ago

> Whatever eventually supplants Postgres is quite likely going to be based on Arrow - polyglot zero-copy vector processing is the future.

Can you elaborate this? I understand it's a very opinionated statement but still I don't see how "polyglot" and "vector processing" could be considered the future of OLTP and general purpose DBMS.

discuss

order

refset|2 years ago

Polyglot means not having to fight with marshaling overheads when integrating bespoke compute functions into SQL, or when producing input to / consuming the output from queries. This could radically change the way in which non-expert people construct complex queries and efficiently push more logic into the database layer, and open the door to bypassing SQL as the main interface to the DBMS altogether.

Vector processing means improved mechanical sympathy. Even for OLTP the row-at-a-time execution model of Postgres is leaving a decent chunk of performance on the table because it doesn't align with how CPU & memory architectures have evolved.

__all__|2 years ago

Thanks!

Honestly, I can't envision a near future where SQL is not the main interface. Happy to see the future proving me wrong here though!

Despite I can buy the arguments about how having a better data structure to communicate between processes (in the same server) could help, it's a bit difficult to wrap my mind around how Arrow will help in distributed systems (compared to any other performant data structure). Do you have any resources to understand the value proposal in that area?

Same for vector processing, would be great to read a bit more about some optimizations that would help improving Postgres leaving out pure analytical use cases.