Ask HN: How to Resurrect a Site from Archive.org?

[rrr_oh_man]

I recently bought the expired domain of a niche interest site because the previous owner was determined to let it die and did not want to put any effort in it anymore.

Is there a way I can "revive" it from archive.org in a more or less automated fashion? Have you ever encountered anything like it? I am familiar with web scraping, but archive.org has its peculiarities.

I really, really love the content on it.

It's a very niche site, but I would love for it to live on.

[duskwuff]

> I recently bought the expired domain of a niche interest site because the previous owner was determined to let it die and did not want to put any effort in it anymore. Is there a way I can "revive" it from archive.org in a more or less automated fashion?

Buying a domain name does not award you ownership of the content it previously hosted. If you have not come to some agreement with the previous owner, you should not proceed.

[aspenmayer]

Well we can't really assume either way, as OP was vague about how the site was left abandoned. They may have some arragement that would make this not copyright infringing. In the absence of any affirmative assent in writing reviewed by legal counsel, I'd be inclined to agree with you, and yet I sought to provide the best answer to the question provided, as the legal issues were outside the scope of the question as asked, and the legal issues you raised seem obvious to you and I, and ought also to be so to OP, but we can't make assumptions about the license of the content in question and/or the relevant jurisdiction(s), which may make these points all moot.

[moralestapia]

How's that different from the site being hosted at archive.org?

[lhamil64]

What if OP just had the domain redirect to the archive.org page? Then they wouldn't be hosting the content themselves

[ksec]

Which is the part that really annoys me. The owner would much rather shut it off than selling it or letting others to run even in archive mode.

I recently learned CGTalk was completely shut down and ALL the information shared over the pass 20 years are gone. It never received the attention like DPreview. There are plenty of other examples where forum owner no longer wants the burden of owning it.

It really is a sad state of things.

Is there a site or exchange somewhere where owner could sell their site or at least put up a whole archive as asset?

[ulrischa]

I’ve seen a lot of people do this when resurrecting old niche sites. The high-level approach usually involves grabbing all the snapshots from archive.org, stripping out their timestamped URLs, and consolidating everything into a local mirror. In practice, you want to:

1. Collect a list of archived URLs (via archive.org’s CDX endpoints). 2. Download each page and all related assets. 3. Rewrite all links that currently point to `web.archive.org` so they point to your domain or your local file paths.

The tricky part is the Wayback Machine’s directory structure—every file is wrapped in these time-stamped URLs. You’ll need to remove those prefixes, leaving just the original directory layout. There’s no perfect, purely automated solution, because sometimes assets are missing or broken. Be prepared for some manual cleanup.

Beyond that, the process is basically: gather everything, clean up links, restore the original hierarchy, and then host it on your server. Tools exist that partially automate this (for example, some people have written scripts to do the CDX fetching and rewriting), but if you’re comfortable with web scraping logic, you can handle it with a few careful passes. In the end, you’ll have a mostly faithful static snapshot of the old site running under your revived domain.

[Gualdrapo]

I was commissioned to recover ideawave.ca from archive.org as its owner lost its database so pretty much all what was left was only on archive.org. I think it was under WordPress but he asked me to port it to Jekyll.

I scraped its contents (blog posts, pages, etcetera) with Python's beautifulsoup and redid its styling "by hand", which was not something otherworldy (the site was from line 2010 or so) and had the chance to put some improvements.

The thing with the scraping was that the connection was lost after a while and it was reaaaaaaaaaally sloooooooooow so I had to keep a register on memory of what was the last successful scraped post/page/whatever and, if something happened, restart from it as a starting point.

Got pennies for it, mostly because I lowballed myself, but got to learn a thing or two.

[janesvilleseo]

This something that used to be done quite a bit in the SEO world. Not sure if still holds and SEO value. Probably some, but maybe not the same level.

Anyways there are tools out there. I haven’t used them

But a tool like https://www6.waybackmachinedownloader.com/website-downloader...

Or

https://websitedownloader.com/

Should do the trick. Depending on the size of the site a small cost is involved.

They can even package them into unusable files.

[cdr420]

I'm hoping you meant "usable" and not "unusable". Or maybe you did. Funny either way!

[moxvallix]

You can use wayback_machine_downloader to automate downloading the archived pages https://github.com/hartator/wayback-machine-downloader/

[d3VwsX]

That used to work great for me, but recently it started to fail. It downloads a few pages but then it gets errors, as if it is detected and prevented by the server from scraping.

[latexr]

Have you tried searching for your question online? I found plenty of results.

https://superuser.com/questions/828907/how-to-download-a-web...

[aspenmayer]

Specifically:

https://wiki.archiveteam.org/index.php?title=Restoring

which mentions

https://github.com/hartator/wayback-machine-downloader

and also this tip:

> This is undocumented, but if you retrieve a page with id_ after the datecode, you will get the unmodified original document without all the Wayback scripts, header stuff, and link rewriting. This is useful when restoring one page at a time or when writing a tool to retrieve a site:

> http://web.archive.org/web/20051001001126id_/http://www.arch...

From the downloader's issues, you may or may not need to use this forked version if you encounter some errors:

https://github.com/hartator/wayback-machine-downloader/issue...

https://github.com/ShiftaDeband/wayback-machine-downloader

[01jonny01]

Gosh. No one answers the question directly.

1) Download HTTrack if its a large websit with alot of pages 2) Download Search and Replace program, theres many of them. 3) The search and replace program allows you to remove the appended web archive url from the pages in bulk. 4. Upload to your host. 5. Run the site through a bulk link checker, that test for broken links. There is plenty of them online.

[bagpuss]

Archivarix is the most fully formed easiest way to do this, free https://archivarix.com/

[aspenmayer]

This site is misrepresenting itself as open source and free, while simultaneously having an affiliate program and pricing page, which, as I've said, isn't free. It's unverifiable whether or not it's open source, as you don't even download/run the software yourself: it's a web app, which is beside the point, as web apps could also be open source, but as there's no way to self-host this, let alone download it and/or run it, for free or otherwise. I think it's safe to avoid this scammy site.

None of my ire is directed at you, as I don't assume you knew any of this. I just wanted to let you know, in case you were mislead as to what the site does by its ad copy.

https://archivarix.com/en/affiliate/

https://archivarix.com/en/#show-prices-wbm

[bagpuss]

> to clarify, i have nothing to do with this site, i used it once, years back and there was a free tier or at least a free/crippled version at that time

posters, enhance your calm

- bagpuss, fat furry cat puss

[toast0]

I did this for a niche site, but it was only 20 pages.

I pulled each page off internet archive, saved it as an archive; then did some minor tidying up, setting viewports for mobile, updating the linkback html snippet to go to my url instead of the old dead one, changing the snippet to not suggest hotloading the link image, crop the dead url out of the link image, pngcrush the images, put it on cheap hosting for static pages.

I did a bit of poking around trying to find a way to contact the owner, but had no luck. If they come back and want it down, I'll take it down. Copyright notices are intact. I'm clearly violating the author's copyrights, and I accept that.

[gopher_space]

> I'm clearly violating the author's copyrights, and I accept that.

I'm looking at combining several old message boards into something useful, and I'd like to be proactive regarding copyright. My approach so far:

- I'm assuming that everyone owns their own post/comment.

- I'm assuming that submitting content meant they intended to grant rights to community members.

- I'm assuming that work done in support of the original community would be welcomed by members.

- And I'm assuming this all changes if I want money.

So I'm preserving attributions when I can, but treat content like it's CC or similar as long as I'm operating within the original authors area of concern. Anything that actually gets released will be as open as possible... and probably start with telling you how to download files. Entirely walling off my code makes sense but then it is no longer a fun little project, it is a framework.

[Sysreq2]

You could also consider using the Common Crawl dataset provided by Amazon. Archive.org is more or less a wrapper around it anyways.

https://registry.opendata.aws/commoncrawl/

[paxys]

Have you spoken to the previous owner about any of this? Otherwise it's pretty crazy to just take ownership of the site and all its content without a written agreement in place. You are opening yourself up to a massive amount of liability for no reason.

[aspenmayer]

I agree with your points, but as the original host of the site no longer is continuing to host it, I doubt they would be any more interested in what others do regarding it, but a lawsuit with a potential payday might motivate them. I broadly agree with you though.

[aoipoa]

This was posted 6 days ago but it's reappeared now 4 hours ago. What happened?

https://hn.algolia.com/?q=ask+hn+resurrect+site+archive

Very odd.

Even the times of the comments have changed, this is what the post looked like yesterday:

https://web.archive.org/web/20241205054108/https://news.ycom...

[denotational]

HN has a “resubmit” mechanism whereby the mods can resubmit interesting posts if they think they might stimulate more interest by being posted at a different time (or just by having better luck).

To avoid a dupe, this mechanism post-dates the original post.

[alsetmusic]

I’ve been thinking about buying a sibling domain (.net instead of .com) to re-host a fantastic essay that disappeared from the web some years back. I would make it clear that I didn’t write it and offer to remove it if the original author contacted me requesting that I remove it (it did not include attribution in its original form). But the issue has been enough of a grey area that I haven’t pulled the trigger.

For anyone who may be curious, wayback machine has an archive: fuckthesouth.com

[Alifatisk]

HTTrack? You should not do it without the owners consent though.

[aspenmayer]

Seems legit.

> HTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility.

Available on Windows, Mac, Linux, and Android.

[pabs3]

Unless you are going to continue to run the site and have it change etc, there is no point doing this since archive.org already hosts static snapshots of sites.

Depending on the site you would use different tools, for eg for MediWiki/DokuWiki sites you would import the latest database dump on archive.org.

I have used wayback-machine-downloader before for completely static sites before:

https://github.com/hartator/wayback-machine-downloader/

[donalhunt]

Did this 10+ years ago for a circa-2000 band website (was a few html pages). Was fairly straightforward to achieve. Some content (embedded from 3rd party websites) was not recoverable.

[joshdavham]

Can I ask what site it was? Reading this made me think of a very specific site that I'd also like to see revived and I'm wondering if we're thinking of the same site.

[davidjhall]

I have a similar one -- site just sent down a few days ago. A special library....

[canU4]

Isn't a simple wget -r enough?

[ddgflorida]

web scraping but be careful about using copyrighted images.

[comboy]

  wget --mirror --convert-links --page-requisites --no-parent URL

But yeah it's also not clear to me regarding copyrights and such.