[jwuphysics]

How much do you expect auto-context and clustering+re-ranking to help for cases in which documents already have high-quality summaries? For context, I parse astrophysics research papers from arXiv and simply embed by paper abstracts (which must be of a certain size), and then append (parts of) the rest of the paper for RAG.

[zmccormick7]

So the point of AutoContext is so you don't have to do that two-step process of first finding the right document, and then finding the right section of that document. I think it's cleaner to do it this way, but it's not necessarily going to perform any better or worse. But then spRAG also has the RSE part which is what identifies the right section(s) of the document. Whether or not that helps in your case is going to depend on how good of a solution you already have for that.

[jwuphysics]

That makes sense and I'll run a few evals. Many thanks for open sourcing your work!