I have implemented a query optimization algorithm for PostgreSQL. The novelty here was supposed to be that the query optimization phase could be split up into sub-optimization steps which could be executed on multiple workers parallely. I think there are not many real world use cases for this. I could only think of a contrived setting where one has to do some data-analysis on many (100s) of data sources, where searching for a join plan might be a bottleneck.
Anyway, as you will find, if you visit the repository, that I found it difficult to use threads/fork processes in Postgres in a way that was reliable and safe. I would like people's opinion on whether the use case I described earlier is a real one and if so, can we parallelize join optimization in Postgres?
[+] [-] matroid|4 years ago|reply
I have implemented a query optimization algorithm for PostgreSQL. The novelty here was supposed to be that the query optimization phase could be split up into sub-optimization steps which could be executed on multiple workers parallely. I think there are not many real world use cases for this. I could only think of a contrived setting where one has to do some data-analysis on many (100s) of data sources, where searching for a join plan might be a bottleneck.
Anyway, as you will find, if you visit the repository, that I found it difficult to use threads/fork processes in Postgres in a way that was reliable and safe. I would like people's opinion on whether the use case I described earlier is a real one and if so, can we parallelize join optimization in Postgres?