top | item 44488269

(no title)

ouked | 7 months ago

The link https://github.com/mandarwagh9/llmstxt is broken, and gives 404.

I assume this is to help tool-enabled agents navigate websites? I'm curious how this balances against the otherwise anti-LLM position a lot of website owners / artists are taking.

discuss

mandarwagh|7 months ago

The link originally pointed to the GitHub repository, but I've made it private for obvious reasons.

The purpose of this file is to help AI scrapers index your website more effectively. OpenAI and other platforms are planning to prioritize these files, which can reduce scraping costs and improve the quality of indexed content.

This also gives your site an advantage in AI-powered search engines like Perplexity and ChatGPT Search.

For anyone in the emerging tech space, adding this file is a no-brainer — it can significantly improve your site's visibility compared to others.