top | item 44790364 (no title) dgreensp | 6 months ago Googlebot respects robots.txt. And Google doesn't use the fetched data from users of Chrome to supplement their search index (as a2128 is speculating that Perplexity might do when they fetch pages on the user's behalf). discuss order hn newest foota|6 months ago Yes, but there's no way to say "allow indexing for search, but not for AI use", right? warkdarrior|6 months ago But there is: https://developers.google.com/search/docs/crawling-indexing/...There is an user agent for search that you can control in robots.txt. user-agent: Googlebot There is another user agent for AI training. user-agent: Google-Extended load replies (1)
foota|6 months ago Yes, but there's no way to say "allow indexing for search, but not for AI use", right? warkdarrior|6 months ago But there is: https://developers.google.com/search/docs/crawling-indexing/...There is an user agent for search that you can control in robots.txt. user-agent: Googlebot There is another user agent for AI training. user-agent: Google-Extended load replies (1)
warkdarrior|6 months ago But there is: https://developers.google.com/search/docs/crawling-indexing/...There is an user agent for search that you can control in robots.txt. user-agent: Googlebot There is another user agent for AI training. user-agent: Google-Extended load replies (1)
foota|6 months ago
warkdarrior|6 months ago
There is an user agent for search that you can control in robots.txt.
There is another user agent for AI training.