Can you tell us about your scoring function? Selecting 40M results from a dataset of ~1B and returning the top 10 based on some trivial scoring function is easy, any reasonable search system will handle that. The problem is when you have to run your scoring function on all 40M matching docs to decide which 10 are the most relevant. It's even more of a problem when your scoring function captures some of the complexities of the real word, rather than something trivial like tf/idf or bm25.
lacksconfidence|4 years ago