[hartator]

If API doesn’t have the data you want, this point is moot.

[dotancohen]

Not GP, but I disagree. I've written successful, robust web scrapers without LLMs for decades.

What do you think the E in perl stands for?

[llbbdd]

This is probably just a parallel discussion. I written plenty of successful web scrapers without LLM's, but in the last couple years, I've written a lot more where I didn't need to look at the web markup for more than a few seconds first, if at all. Often you can just copy-paste an example page into the LLM and have it generate accurate, consistent selectors. It's not much different when integrating with a formal API, except that the API usually has more explicit usage rules, and APIs will also often restrict data that can very obviously be used competitively.

[llbbdd]

Double-posting so I'm sorry but the more I read this the less it makes sense. The parent reply was talking about data that was straight-up not available via the API, how does perl help with that?