top | item 37221884 (no title) catboybotnet | 2 years ago Seconding. robots.txt is just a way of "asking nicely." If somebody wants to scrape, they're spoofing their UA and ignoring it. Can't do anything other than monitor the logs and ban IPs one by one. discuss order hn newest No comments yet.
No comments yet.