top | item 42768957

(no title)

codemusings | 1 year ago

For what it's worth: they do honor the robots.txt file. I had the same problem with a client's CMS and denying all AI crawler user agents did the trick.

It's clear they've all gone mad. The traffic spiked 400% overnight and made the CMS unresponsive a few times a day.

discuss

order

No comments yet.