[CraigRood]

I've personally noticed loads of crawls too, but what has been interesting is they have been crawling effectively the same pages, over and over, for months. Also, they don't seem to be caching DNS at all.

I guess I'm in the minority that I don't care too much about crawlers, even the use of my work for AI, but I think the current hit rates just can't be justified.