[codemusings]

For what it's worth: they do honor the robots.txt file. I had the same problem with a client's CMS and denying all AI crawler user agents did the trick.

It's clear they've all gone mad. The traffic spiked 400% overnight and made the CMS unresponsive a few times a day.