(no title)
AkshatM | 8 months ago
The idea behind the headers is to allow bots to bypass automatic bot filtering, not blockade all regular traffic. In other words:
- we block bots (the website owner can configure how aggressively we block) - unless they say they're from an AI crawler we've vetted, as attested by the signature headers - in which case we let them pay - and then they get to access the content
(Disclosure: I wrote the web bot auth implementation Cloudflare uses for pay per crawl)
xg15|8 months ago
grg0|8 months ago