[ofermend]

RAG Evaluation is difficult, primarily because it's hard to come up with "golden answers" (or golden chunks).

We made Open-RAG-Eval to solve this - RAG Eval that only requires the question, yet provides great metrics for retrieval, generation, hallucination and citations for any RAG setup.

This was in collaboration with Jimmy Lin and his students at UWaterloo.

It has connectors to LangChain, LlamaIndex and Vectara, and hoping others can contribute more connectors to other RAG systems.

repo: https://github.com/vectara/open-rag-eval

UI for reviewing eval results: https://openevaluation.ai/

Papers: https://arxiv.org/pdf/2406.06519 and https://arxiv.org/abs/2504.15068