[rimeice]

Bots could be crawlers gathering data to periodically be used as raw training data or the requests could just be from a web search agent of some form like ChatGPT finding latest news stories on topic X for example. I don’t know if robots.txt can distinguish between the two types of bot request or whether LLM providers even adhere to either.

[jay_kyburz]

Wow, Just reading the headline I had assumed they were giving the new article as a document, then asking it to summarize the the document given.