[liliuleo93]

Thanks for your suggestion and this is a super good question. I was asked some times and please allow me quote one of my response in the repo

" With respect to recall vs Performance, your idea is indeed correct. However, several reasons have guided us to our current approach:

1. We are not solely benchmarking open-source systems; we are also focusing on cloud services. Some of these services, such as Zilliz and Pinecone, don't allow users to customize their parameters to tune the recall, aiming to simplify their usage. Consequently, creating a recall vs Performance graph is not feasible. Also this benchmark allow users to customize their parameters for systems allowing tuning to get their own result to do comparison.

2. There already exists a number of benchmarks doing what you've suggested, which target individuals with ANN search backgrounds. Our goal is to make this benchmark as straightforward as possible and to assist people who lack understanding about the inner workings of each system.

3. Concerning reproducibility, generating a recall vs QPS graph that you mentioned, would require conducting a multitude of tests to obtain enough data points, which considerably reduces reproducibility. "

the link is: https://github.com/zilliztech/VectorDBBench/issues/200#issue...