[awsanswers]

Unacceptable, sorry this is happening. Do you know about fail2ban? You can have it automatically filter IPs that violate certain rules. One rule could be matching on the bot trying certain URLs. You might be able to get some kind of honeypot going with that idea. Good luck

[thayne]

They said that it is coming from different ip addresses every time, so fail2ban wouldn't help.

[jsheard]

Amazon does publish every IP address range used by AWS, so there is the nuclear option of blocking them all pre-emptively.

https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-rang...

[SteveNuts]

It’ll most likely eventually help, as long as they don’t have an infinite address pool.

Do these bots use some client software (browser plugin, desktop app) that’s consuming unsuspecting users bandwidth for distributed crawling?

[keisborg]

Monitor access logs for links that only crawlers can find.

Edit: oh, I got your point now.