[jaggirs]

Not sure, havent done so myself but I think you can use MTEB maybe. Or otherwise a llm benchmark on large inputs (and compare your chunking with naive chunking)