[re-lre-l]

Don’t get me wrong, but what’s the problem with scrapers? People invest in SEO to become more visible, yet at the same time they fight against “scraper bots.” I’ve always thought the whole point of publicly available information is to be visible. If you want to make money, just put it behind a paywall. Isn’t that the idea?

[georgefrowny]

There's a difference between putting information easily online for your customers or even people in general (eg as a hobby), and working in concert with scraping for greater visibility via search, and giving that work away, or at a cost, to companies who at best don't care and possibly may be competition, see themselves as replacing you or otherwise adversarial.

The line is "I technically and able to do this" and "I am engaging with a system in good faith".

Public parks are just there and I can technically drive up and dump rubbish there and if they didn't want me to they should have installed a gate and sold tickets.

Many scrapers these days are sort of equivalent in that analogy to people starting entire fleets of waste disposal vehicles that all drive to parks to unload, putting strain on park operations and making the parks a less tenable service in general.

[akoboldfrying]

> The line is "I technically and able to do this" and "I am engaging with a system in good faith".

This is where the line should be, always. But in practice this criterion is applied very selectively here on HN and elsewhere.

After all: What is ad blocking, other than direct subversion of the site owner's clear intention to make money from the viewer's attention?

Applying your criterion here gives a very simple conclusion: If you don't want to watch the ads, don't visit the site.

Right?

[nrhrjrjrjtntbt]

The old scrapers indexed your site so you may get traffic. This benefits you.

AI scrapers will plagiarise your work and bring you zero traffic.

[ProofHouse]

Ya make sure you hold dear that grain of sand on a beach of pre-training data that is used to slightly adjust some embedding weights

[saltysalt]

You are correct, and the hard reality is that content producers don't get to pick and choose who gets to index their public content because the bad bots don't play by the rules of robots.txt or user-agent strings. In my experience, bad bots do everything they can to identify as regular users: fake IPs, fake agent strings...so it's hard to sort them from regular traffic.

[Dilettante_]

Did you read TFA?

These scrapers drown peoples' servers in requests, taking up literally all the resources and driving up cost.