Creating AI assistant with GPT and Ruby and Redis using embeddings

[Mizza]

From a discussion with a friend today..

Are embeddings a hack? Is building out tooling and databases and APIs and companies around embeddings all going to be for naught as soon as there's a solid LLM/API with a big enough context window?

[jrpt]

I don't think the context window will ever be big enough for some use cases. There was a recent paper talking about a million tokens, but that's just the Harry Potter books. Which is amazing, but you know there's going to be use cases that will have more than 7 books worth they want to use. Furthermore, the performance will be better when you don't have to give it all 1 million tokens, but just the most relevant parts of a context.

[regiswilson]

The short answer is that, yes, embeddings are probably a hack in the same way that using bits or short variable names were hacks to reduce memory usage. At some point you are correct: someone would prompt "given <large amount of data>, answer <user request>".

[jeremy_k]

I can't say I'm very well versed in all of this but I was asking my coworkers today about whether embeddings were the way forward or if doing your own training would be more beneficial. Or even yet, could you take an open source model and train it specifically on just your content; would that wield better results?

Expanding context seems like an approach, but if you're trying to get an answer about your company's documentation, why would you need the entirety of GPT-X?

[fzliu]

Even with an incredibly long context window (say, 1M tokens), attention still suffers from a problem with long-term dependencies. This is probably why OpenAI hasn't publicly released their 32k token length model just yet.

[toxicFork]

Embeddings are useful for sentiment analysis and search in general, but given a "powerful enough AI with enough of a context window" they may be obsolete indeed, if it can do all of those things.

[welfare]

That's gotta be a Hacker News bingo if I've ever seen one.

[taf2]

Think we can optimize this with rust

[strudey]

Can use https://github.com/alexrudall/ruby-openai to do this sort of thing also :)