top | item 37749082 (no title) jwilliams | 2 years ago It's not necessarily a lot of sites that block non-popular bots - but often it's big sites (i.e. content-centric sites such as Social Media). Think Yelp, Twitter, LinkedIn, Instagram, etc.That can add up to a serious percentage of the web. discuss order hn newest No comments yet.
No comments yet.