[PathfinderBot]

What about something like Nitter? Archiving? Adversarial bridging between different platforms? Automation?

How will well-behaved scrapers undermine the sustainability of a business? I guess adblocking is one, but we can already do that with uBlock and that's legal. Or adversarial bridging, but that only serves to boost competition.

In other words, the question is flipped; why would well-behaved (i.e. non-DDoSing) scrapers be illegal?

[jastingo]

I think you're conflating automation and intentional avoidance of bot detection as part of automation. The issue I have is not that this service allows users to automate browsing activities. The issue is that this service deliberately tries to circumvent being detected as automating browser activities because websites are trying to prevent bots. There are LOTS of services that allow users to create automations without disguising themselves. If you are using well-behaved scrapers that respect TOS then you shouldn't have to use a service like this.

Nitter is an example of a service that explicitly disrupts Twitter/X's way to make money. If they can't make money then they can't provide the service, there would be no Twitter/X, and hence no Nitter. Of course they would try to prevent that kind of behavior and it should be obvious why. Resorting to using a service like this in order to continue using Nitter should raise some alarm bells. Sure you can still do it and rationalize it however you want, but you have to acknowledge you're trying to get the value of the service without paying for it.

Perhaps there are cases where there is a dissonance between a website's TOS and how they are blocking bot traffic? That sounds like a valid gripe. Otherwise, I don't buy the argument.

[PathfinderBot]

That's fair enough. I think that falls under similar arguments to adblocking; it's against ToS, and affects the revenues of ad-supported businesses, but it seems like the popular view is to use it regardless.

[judge2020]

Legality isn't the question here. If you want to speak to the legality, anyone circumventing a robots txt that explicitly has your bot's user-agent and 'disallow: *' is unauthorized access (I imagine it's more nuanced for 'user-agent: *'). No website is required to allow anyone to visit and can discriminate against any client or software any way they want.

[yjftsjthsd-h]

> Legality isn't the question here.

The question was literally,

> What are the legitimate (i.e. legal) use cases for a product such as this?