[jruohonen]

It affects science too (and there you'd want solid archiving as much as possible). Increasingly, meta-data is full of errors and general purpose search engines for science are breaking down, including even things like Google Scholar. I suppose some big science publishers are blocking AI bots too.

[shevy-java]

Google ruined its own search engine on top of that as well though.

We are increasingly becoming blind. To me it looks as if this is done on purpose actually.

[terminalshort]

Did Google ruin it, or did advesarial activity between Google's algorithm and SEO ruin it? The latter seems more likely because the incentives make sense, and also inevitable.

[salawat]

It was. Advertising is incompatible with accurate data retrieval/routing. We've also implemented "obligation to deindex". So providing an unbiased index of the web as she is is essentially (in the U.S.) verboten.

[ninjagoo]

> I suppose some big science publishers are blocking AI bots too.

That's a travesty, considering that a huge chunk of science is public-funded; the public is being denied the benefits of what they're paying for, essentially.

[galleywest200]

The public can still access the sites themselves.

[upboundspiral]

If anyone wants the surreal experience of seeing blogs and websites made by real humans they should check out https://marginalia-search.com

It's far from perfect but it does achieve its stated goal: of resurfacing real people on the internet.

It recently got some NLNet funding and I hope to see it flourish - to my knowledge there aren't any other projects trying to claw back control of the internet towards the commons.

https://about.marginalia-search.com

[asdff]

Thank god for pubmed and deterministic search operators.