top | item 45713014 (no title) lolpython | 4 months ago How would the links be prioritized? If the bots goal is to crawl all content would they have prioritization built-in? discuss order hn newest HumanOstrich|4 months ago How would they prioritize things they haven't crawled yet? lolpython|4 months ago It's not clear that they are doing that. Web logs I've seen from other writing on this topic show them re-crawling the same pages at high rates, in addition to crawling new pages load replies (1)
HumanOstrich|4 months ago How would they prioritize things they haven't crawled yet? lolpython|4 months ago It's not clear that they are doing that. Web logs I've seen from other writing on this topic show them re-crawling the same pages at high rates, in addition to crawling new pages load replies (1)
lolpython|4 months ago It's not clear that they are doing that. Web logs I've seen from other writing on this topic show them re-crawling the same pages at high rates, in addition to crawling new pages load replies (1)
HumanOstrich|4 months ago
lolpython|4 months ago