The Missing ”WHERE” Clause for Vector Search

[tobrien6]

Amazon's Opensearch (fork of Elasticsearch) natively supports vector-based approximate KNN (using https://github.com/nmslib/nmslib/) which is integrated with Opensearch's native filtering functionality. Elasticsearch also has similar functionality, but I don't know if their KNN code scales quite as well.

[gk1]

Opensearch only supports "pre-filtering" or "post-filtering," which leads to either high latency or incomplete results, as explained in the article.

This is why single-stage filtering was the most-requested feature for us.

From the Opensearch docs:

> You should not use approximate k-NN if you want to apply a filter on the index before the k-NN search, which greatly reduces the number of vectors to be searched.

> Because the graphs are constructed during indexing, it is not possible to apply a filter on an index and then use this search method. All filters are applied on the results produced by the approximate nearest neighbor search.

> If you use the knn query alongside filters or other clauses (e.g. bool, must, match), you might receive fewer than k results.

(https://opensearch.org/docs/search-plugins/knn/approximate-k...)

I know Elasticsearch is working on introducing vector search but it is not yet available. I don't know how they will support filtering.

[sysctl21]

I've already subscribed to pinecone newsletter. but accidentally i clicked on this post when i opened HN , what a coincidence .

[jamesbriggs]

Hi all, I'm the author - I'll be here to answer any questions you have!

[Noe2097]

Hi! Yes! Nice article (articles actually).

I realize you are hinting at it as the subject of a later article, but could you share with us a little bit more about single-stage-filtering :D ? how does it work? How can metadata and vector data "coexist" in a same index?

[throwaway_pdp09]

Your damn animated gif has sucked up >4.7GB in my browser and crashed it. Not impressed.

Did animation actually add anything significant anyway.

[sysctl21]

may i know what tool was used to create the gifs in this blog post ? . Thanks in advance .

[jamesbriggs]

Excalidraw.com for the style/graphs, then Adobe animate for the animation!

[gk1]

I believe it’s Excalidraw and some animation software like Adobe Animate.

[TOMDM]

Not quite what you're after, but there's a matplotlib style made to look like xkcd

https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot....

[londons_explore]

This works by streaming results from the similarity search to the filter, in descending order of similarity. As soon as enough matches are found, the similarity search is terminated.

Postgres does the same thing for many queries if you look at the query plan.

Not really worthy of a blog post - especially one that says "wait till the next blog post to find out how it works!!".

[willvarfar]

How is what you describe _not_ just an efficiently implemented post filter with early out?

If it turns out that’s all pinecone have, then yeah, I’m gonna be disappointed. My mind is working overtime imagining prefixing vectors with their filter terms to root everything and other naive things…

[sgt101]

You have high standards about what should be in a blog post.

[WithinReason]

And how do construct a search that can stream results in descending order of similarity, that's faster than O(N) for N values in the database?

[gk1]

No, this isn’t at all how it works.

This is not a trivial problem when dealing with vector similarity search, which Postgres does not have.