WingNews logo WingNews
top | new | best | ask | show | jobs | submit
GitHub [2]
top | item 19481655

Using Sitemaps in Web Scraping

3 points| webrobots | 7 years ago |webrobots.io | reply

2 comments

order
[+] [-] akriukas|7 years ago|reply
Very useful for some big eshops if they list their sitemaps. However not all stores have sitemaps and even robots.txt site...
[+] [-] HackerDalius|7 years ago|reply
Good article, useful general tips that work for many scraping tools and platforms.
powered by hn/api // news.ycombinator.com