[super256]

I prefer archive.today because the Internet Archive’s Wayback Machine allows retrospective removals of archived pages. If a URL has already been crawled and archived, the site owner can later add that URL to robots.txt and request a re-crawl. Once the crawler detects the updated robots.txt, previously stored snapshots of that page can become inaccessible, even if they were captured before the rule was added.

Unfortunately this happens more often than one would expect.

I found this out when I preserved my very first homepage I made as a child on a free hosting service. I archived it on archive.org, and thought it would stay there forever. Then, in 2017 the free host changed the robots.txt, closed all services, and my treasured memory was forever gone from the internet. ;(

[pgalvin]

This information is now many years out of date - they no longer have this policy.

[extraduder_ire]

Any idea when that changed? I've been unable to access historical sites in the past because someone parked the domain and had a very restrictive robots.txt on it.

[snigsnog]

Even so you can still just request your site to be removed: https://help.archive.org/help/how-do-i-request-to-remove-som...