top | item 45711875

(no title)

Max-Ganz-II | 4 months ago

The instant I dropped this on HN, gptbot was scanning the site.

(Updated robots.txt, this site https://robotstxt.com/ai to the rescue. I had to update robots.txt on the Amazon Redshift site fairly recently - I had gone from something like 10mb of log file per day to something like 50mb of log file per day, from LLM crawlers.)

discuss

order

No comments yet.