[634636346]

Like one of the commenters in that thread said, this sounds like they were using the noidex feature to use the IA as a personal private backup, and thus abusing it, and ruined it for everyone else. The IA is great as a personal public backup. (For example, I've deliberately submitted copies of certain OSS projects I've worked on to the Wayback Machine.)

[fieryskiff11]

The IA can always defer to putting a census of noindexed items as a json item though if they want to stay true to their motto of "universal access to all knowledge". For every invalid reasons there are always a valid reason for retaining such features. Some people might not want to see their works easily scraped by AI companies and so on which then profit off their backs, among all other sensitive use cases. If they do away with the self noindex option, the search results will one day become very clogged if you want to look for a refined result.

Permanent.org which is more geared toward the personal private backup use cases were only formally released in 2020. Before it there was only IA which could really provide digital preservation for all swathes of people. Even then I occasionally saw that Permanent.org cross-pollinates their datasets into the IA.

Contrary to the claims that I'm not looping in with them about noindex, in fact they had already been keenly aware of my (and probably others) use cases for a while. Before then I'd sometimes even contacted their staff members for help if I for some reason cannot no-index the items alone. Their tone changed slightly to say that "IA is a library" around the time of the bookseller's lawsuit but coincidentally because all the items I wanted to be archived, had been so, I was able to promise them to not upload any private items, at least I could discuss with Mr. Kahle about the issue. I did sent an email to the latter afterwards, but perhaps because he's heavily distracted by things like the lawsuits, it was eventually let slide and forgotten, until now.

I actually conversed with the founder somewhere before this year where I floated the possibility of the IA becoming an all-out archive and digital vault, more than just a library, in which he seemed to be welcoming with.

Edit: At one point IA even endorsed/recommended the use of noindex as an option to hide bulk-uploaded files from search results to avoid clogging it.

https://archive.org/post/1086430/should-i-archive-this-ever-...