top | item 43430347

(no title)

bno1 | 11 months ago

What stops a scraper from detecting Anubis and just removing "Mozilla" from the user-agent string?

discuss

order

TonyTrapp|11 months ago

That would allow you to specifically lock out that bot based on its user-agent string. That's the main problem with AI scrapers, many of them normally use user-agents that cannot be easily blocked, so other means have to be found to keep them off your grounds.

xena|11 months ago

It breaks websites when you do that.