top | item 37032008 (no title) dmitrysergeyev | 2 years ago They could in theory combat it by comparing results with a second crawler that uses a different User Agent. discuss order hn newest mdaniel|2 years ago If they were going to the amount of energy to do a 2nd crawl using a different user agent, then why bother advertising the user agent at all and just feed it the Chrome one like every other home-grown spider does unknown|2 years ago [deleted]
mdaniel|2 years ago If they were going to the amount of energy to do a 2nd crawl using a different user agent, then why bother advertising the user agent at all and just feed it the Chrome one like every other home-grown spider does unknown|2 years ago [deleted]
mdaniel|2 years ago
unknown|2 years ago
[deleted]