[tootahe45]

Would be cool if there was something like this for Python. Last time i tried to scrape something interesting i found that one of Cloudflare's enterprise options was easily blocking all of the main http libraries due to the identifiable TLS handshake.

[ricardo81]

Are you sure they blocked you because of the handshake?

Always thought it was the myriad of cookies and expiry time of said cookies that tend to make non-browser clients more obvious to CF.

[tootahe45]

The site wasn't using it to block me, just to prompt a captcha, without doing so to 'real' browsers.

The HTTP requests were exact copies of browser requests (in terms of how the server would've seen them), so it was something below HTTP. I ended up finding a lot of info about Cloudflare and the TLS stuff on StackOverflow, with others having similar issues. Someone even made an API to do the TLS stuff as a service, but was too expensive for me. https://pixeljets.com/blog/scrape-ninja-bypassing-cloudflare...

[tyingq]

I think most of the scraping libraries have stagnated since it's hard to scrape without a headless browser these days...too many sites with client-side rendered content.