(no title)
Max-Ganz-II | 4 months ago
(Updated robots.txt, this site https://robotstxt.com/ai to the rescue. I had to update robots.txt on the Amazon Redshift site fairly recently - I had gone from something like 10mb of log file per day to something like 50mb of log file per day, from LLM crawlers.)
No comments yet.