[kipukun]

To the second footnote: you could utilize Polar's lazyframe API to do that cosine similarity in a streaming fashion for large files.

[minimaxir]

That would get around memory limitations but I still think that would be slow.

[kipukun]

You'd be surprised. As long as your query is using Polars natives and not a UDF (which drops it down to Python), you may get good results.