top | item 22181946

(no title)

vectorEQ | 6 years ago

i'm wondering if that robots.txt might then get you sued due to blocking scrapers / bots?

discuss

order

herval|6 years ago

A robots.txt file doesn’t block scrapers, it’s the equivalent of a no trespassing sign. I don’t think putting up signs will suddenly become illegal?

setr|6 years ago

Robots.txt doesn't interfere with anything. It's a suggestion.