[Toritori12]

Overall I agree with the idea, but prob will be cheaper to bypass CF considering the amount of data that big techs are consuming (also Google with get it for free because Google Search?). If successful, I wonder how agents will transfer this cost to the user.

[jimbohn]

>Google with get it for free because Google Search

What if the second step is that Google pays the page it visits? By enabling a crawler fee per page, news websites could make some articles uncrawlable unless a huge fee is paid. Just thinking aloud, but I could easily see a protocol stating pricing by different kinds of "licensing" e.g. "internal usage", "redistribution" (what google news did/does?), "LLM training", etc. Cloudflare, acting as a central point for millions of websites, makes this possible.

[vbezhenar]

The question is: who has the leverage?

If some small news website denies Google Bot crawling, it'll disappear from Google and essentially it'll disappear from the Internet. People do a great lengths to appease the Google Crawler.

If some huge news website demands fees from Google, it might work, I guess. But I'm not sure that it would work even for BBC or CNN.

[ethbr1]

It'd be a fitting solution if news closed the loop, crawled Google et al. to see if any of their content showed up there, then repriced future cotent higher for any search engines that reproduced content via genai.

[figassis]

More publishers will start blocking google bots as well, bc google is already killing their revenue with AI results.