[regiswilson]

The short answer is that, yes, embeddings are probably a hack in the same way that using bits or short variable names were hacks to reduce memory usage. At some point you are correct: someone would prompt "given <large amount of data>, answer <user request>".

[lukev]

But embedding-based semantic search can handle arbitrary sized databases. I fully believe context windows are going to grow: I am skeptical they will grow to cover "all your company's documents" or even "the full encyclopedia" sizes.

[kenjackson]

> I fully believe context windows are going to grow: I am skeptical they will grow to cover "all your company's documents" or even "the full encyclopedia" sizes.

This is the type of statement that I feel like is often/usually wrong -- at least for the common case. The last time I had this argument was about CDs and how eventually we'll start burning them because they'll be in the cloud, and my friend arguing that storage and network bandwidth would make that impractical if everyone did it.

I expect context window compression or smart ways to embed them so they still provide useful context in "most" cases, even if not-lossless, will be an active area of research.

EDIT: That said, looking a the original question -- I do think vector embeddings are still useful in their own right and somewhat orthogonal to context window sizes. IMO.

[dmix]

It's more than just optimizing for space (which is still going to be important), it's also about using vector databases to seed the data from a wider dataset and translating that into something the AI can use. I mean technically in the far future you could dump a whole database into the 'context' and work off of it, but Vector DBs will fill that role in the meantime and add a memory layer on top of it for future queries.

[redskyluan]

Can't Agree with that more.

LLMs should not be trained to simply memorize information. Instead, they should be designed to understand and identify patterns in the data, and use the knowledge stored in vector databases to organize and summarize information.

Vector databases can be used to store and organize knowledge in a way that is more accessible to LLMs. By using vector representations, LLMs can easily access and manipulate knowledge, allowing them to more effectively process and analyze large amounts of information.

[joe_the_user]

I'd say:

Yes - embeddings are a hack:

No - there won't anything like a "real API" unless there's a new discovery or a shift in the way LLMs are constructed. It's not theoretically impossible but there's no clear way to get guaranteed results from present day LLMs, all they do output guesses from their input text (combining prompt text and then user text).