[spencerchubb]

You seem to be misunderstanding why a website would make llms.txt

Obviously, they would not make it just for an AI company to scrape

Here's an example. Let's say I run a dev tools company, and I want users to be able to find info about me as easily as possible. Maybe a user's preferred way of searching the web is through a chatbot. If that chatbot also uses llms.txt, it's easy for me to deliver the info, and easy for them to consume. Win-win

Of course adoption is not very widespread, but such is the case for every new standard.

[ppqqrr]

The point of LLMs is they are able to make sense of the web the same way humans can (roughly speaking); so why do they get the special treatment of having direct, ad-free, plain text version of the actual info they’re looking for, while humans aren’t allowed to scroll through a salad recipe without being bombarded with 20 ads?

[spencerchubb]

A human could read the llms.txt if they want to. And a developer could put ads in llms.txt if they wanted to!