top | item 46280564

(no title)

ccgreg | 2 months ago

> Some search engines provide a list of their scraper IP ranges

Common Crawl's CCBot has published IP ranges. We aren't a search engine (although there are search engines using our data) and we like to describe our crawler as a crawler, not a "scraper".

discuss

order

No comments yet.