We use grid search to figure out what's the best chunking strategy to use. Create a bunch of different strategies such as recursive chunking, semantic, etc, and parameterize them and see which one works best. The "best" chunking strategy depends on the nature of the documents and the questions being asked.
No comments yet.