top | item 27970276

(no title)

ksahin | 4 years ago

I'm the founder of https://www.scrapingbee.com/, an API for web scraping.

At the beginning we had a banner on the dashboard with a CTA to book a call with me. To incentivize users to book the call, we offered free API credits: https://twitter.com/PierreDeWulf/status/1329765250707623939/...

I had 100+ calls with users in the first months, it helped us a lot to understand our market, what they needed etc.

It worked so well that we had to remove the free credit incentive because taking those calls took all my time.

discuss

ChemSpider|4 years ago

But I would guess that a significant number of your users are doing illegal web scraping (Otherwise, why use proxy rotation). So you advise them how to get around website protections?

AznHisoka|4 years ago

I'm not a ScrapingBee customer (but I've looked into and was interested in it). A lot of websites have started blocking any datacenter IP, even if you are just scraping it once. In order to get the contents of those sites, you need to use residential proxies.

jazzyjackson|4 years ago

When is it illegal to scrape a website?

capableweb|4 years ago

How is it relevant to the topic of "establishing and maintaining relationships with your users" what the users are doing by using the service in their jurisdiction?

CodesInChaos|4 years ago

I'm not sure if all web scraping that's undesired by the website you're scraping is illegal.