Hi all - Jerry (co-founder/CEO) here, here to help answer any questions you might have!
We're building a data framework to unlock the full capabilities of LLMs on top of your private data. We can’t wait for the future - this space is moving so rapidly and there’s so many things we want to do on both the open-source and enterprise side.
Feel free to shoot me a personal note on Twitter/Discord as well.
alright HN - langchain and llamaindex - they are not-competitors in that you can use either standalone or you can use both together - who is using which and why?
I'm using LlamaIndex way more these days, mostly because the big use cases from enterprise orgs is around using their own data in novel ways. it makes much more sense for backend tasks.
that said, while there's some clear crossover between the two, i find myself using langchain for things like huggingface embeddings for local models, and other helpers that work well with llamaindex.
somewhat akin to a data warehouse and all the techniques and abstractions that go into modeling it for non-technical end users, llamaindex makes a lot of that much easier to work with as a developer. structured and unstructured data can be indexed side by side, and the auto retriever functions they've recently built out work really well once you've got data indexed in a sensible way. our next step is to put a simple UI on top of it all with filters (like a dashboard) that pass metadata filters to the llamaindex autoretriever.
these patterns may not be exactly right today, but I don't see any others focusing on this area. just throwing all of your docs haphazardly into an index and calling it a day is no different than tossing all your data into a single database schema without any rhyme or reason, and hoping your dashboards can do 'magic' on top of it.
AFAIK llamaindex is built on langchain to save you a lot of time and lines of code, and also constantly adding new stuff (as does langchain) as dataloaders, different indeces, all kinds of cool stuff like nodes pre and post processing, reranking, query optimization, query decomposition, custom query engines and whatnot. you can still use langchain, for example when using agents or more realistically when going the route of textsplitting by token size. I'd say its complementary, do whatever you can do with llama_index and whatever you cant use langchain.
freezed8|2 years ago
We're building a data framework to unlock the full capabilities of LLMs on top of your private data. We can’t wait for the future - this space is moving so rapidly and there’s so many things we want to do on both the open-source and enterprise side.
Feel free to shoot me a personal note on Twitter/Discord as well.
chenjerry|2 years ago
archimedeslever|2 years ago
swyx|2 years ago
towelpluswater|2 years ago
that said, while there's some clear crossover between the two, i find myself using langchain for things like huggingface embeddings for local models, and other helpers that work well with llamaindex.
somewhat akin to a data warehouse and all the techniques and abstractions that go into modeling it for non-technical end users, llamaindex makes a lot of that much easier to work with as a developer. structured and unstructured data can be indexed side by side, and the auto retriever functions they've recently built out work really well once you've got data indexed in a sensible way. our next step is to put a simple UI on top of it all with filters (like a dashboard) that pass metadata filters to the llamaindex autoretriever.
these patterns may not be exactly right today, but I don't see any others focusing on this area. just throwing all of your docs haphazardly into an index and calling it a day is no different than tossing all your data into a single database schema without any rhyme or reason, and hoping your dashboards can do 'magic' on top of it.
batou069|2 years ago
towelpluswater|2 years ago
thekelvinwong|2 years ago
bobvanluijt|2 years ago
pandajelly68|2 years ago
dazzeloid|2 years ago
taimurrashid|2 years ago
crocsnsocks|2 years ago
gevansgreylock|2 years ago
unknown|2 years ago
[deleted]
andre-z|2 years ago
jgoulie|2 years ago