top | item 44501578

(no title)

imilk | 7 months ago

There are about a dozen Google crawlers that can hit your website for different reasons:

https://developers.google.com/search/docs/crawling-indexing/...

Google-Extended is what is associated with AI crawling, but GoogleBot also crawls to produce AI overviews in addition to indexing your website in Google search.

While the number of crawlers and their overlapping responsibilities makes it difficult to know which ones you can safely block, I should also say that pure AI company bots behave 1000x worse than Google crawlers when it comes to not flooding your site with scraping requests.

discuss

order

No comments yet.