[octoberfranklin]

I don't think it has anything to do with LLMs.

I think the big cloud companies (AWS) figured out that they could scrape compute-intensive pages in order to drive up their customers' spend. Getting hammered? Upgrade to more-expensive instances. Not using cloud yet? We'll force you to.

The other possibility is cloudflare punishing anybody who isn't using it.

Probably a combination of these two things. Whoever's behind this has ungodly supplies of cheap bandwidth -- more than any AI company does. It's a cloud company.

[yellowapple]

> Whoever's behind this has ungodly supplies of cheap bandwidth -- more than any AI company does. It's a cloud company.

Most of the major cloud companies are themselves also AI companies, so I don't think the “cloud companies are artificially driving up compute spend” hypothesis is mutually exclusive with the “AI companies are doing a very bad job at scraping” hypothesis.