top | item 37221884

(no title)

catboybotnet | 2 years ago

Seconding. robots.txt is just a way of "asking nicely." If somebody wants to scrape, they're spoofing their UA and ignoring it. Can't do anything other than monitor the logs and ban IPs one by one.

discuss

order

No comments yet.