[thiagowfx]

There are two to distinguish: "Googlebot" and "Google-Extended".

[lxgr]

That seems to be more like a courtesy that Google could stop extending at any point than a requirement grounded in law or legal precedent.

[Palmik]

Same goes for OpenAI ignoring these "blocks".