[throwawayscrapd]

And maybe you could Bing and learn what "cache eviction" is and why that happens when a crawler systematically hits every page on your site.

[OutOfHere]

Maybe because it's an overly simplistic LRU cache, in which case a different eviction algorithm would be better.

It's funny really since Google and other search engines have been crawling sites for decades, but now that search engines have competition, sites are complaining.