[thisdougb]

https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-th...

This is interesting on the subject. Maybe gives you some understanding. Posted on HN a few minutes ago.