What does one get for actually being nice and being an "ethical web scraper"?
I don't see many people bothering to be honest, developers who are out there scraping are mostly just doing a job, they want to be done as quick as possible and most don't care about things like this unless there are repercussions
It is this type of attitude that is why websites are becoming so aggressive in blocking web scrapers.
Being an "ethical web scraper" is about your own ethics, not abusing other peoples servers/data, and preserving the open internet for everyone.
Yes, if you slam someones website you will probably get the data you want but it will damage it for the collective. If everyone just slammed websites then:
1) the website owners will just get pissed off and either shut data behind logins or make it inaccessible. This is basically what happened with LinkedIn.
2) make it much harder and costlier to scrape by using more advanced anti-bots, requiring everyone to use more expensive residential proxies and headless browsers to get past them.
Web scraping can be a burden for websites, so everyone should approach it in as responsible and ethical way as possible.
joe_91|3 years ago
I don't see many people bothering to be honest, developers who are out there scraping are mostly just doing a job, they want to be done as quick as possible and most don't care about things like this unless there are repercussions
Ian_Kerins|3 years ago
Being an "ethical web scraper" is about your own ethics, not abusing other peoples servers/data, and preserving the open internet for everyone.
Yes, if you slam someones website you will probably get the data you want but it will damage it for the collective. If everyone just slammed websites then:
1) the website owners will just get pissed off and either shut data behind logins or make it inaccessible. This is basically what happened with LinkedIn.
2) make it much harder and costlier to scrape by using more advanced anti-bots, requiring everyone to use more expensive residential proxies and headless browsers to get past them.
Web scraping can be a burden for websites, so everyone should approach it in as responsible and ethical way as possible.