(no title)
imilk | 7 months ago
https://developers.google.com/search/docs/crawling-indexing/...
Google-Extended is what is associated with AI crawling, but GoogleBot also crawls to produce AI overviews in addition to indexing your website in Google search.
While the number of crawlers and their overlapping responsibilities makes it difficult to know which ones you can safely block, I should also say that pure AI company bots behave 1000x worse than Google crawlers when it comes to not flooding your site with scraping requests.
No comments yet.