[CaptainFever]

I'm surprised older OpenAI agents respected robots.txt.

Expecting AI agents to respect robots.txt is like expecting browser extensions like uBlock Origins to respect "please-dont-adblock.txt".

Of course it's going to be ignored, because it's an unreasonable request, it's hard to detect, and the user agent works for the user, not the webmaster.

Assuming the agent is not requesting pages at an overly fast speed, of course. In that case, feel free to 429.

Q: but what about botnets-

I'm replying in the context of "Users will be installing browser extensions or full browsers that run the actions on their local computer with the user's own cookie jar, IP address, etc."