Totally anecdotal but just a month ago I opened a box from my last moving (ended 4 years ago...) and I found a CD-RW labeled "Backup 2001", which survived 5-6 relocations, even an international one.
Well, I put it into an USB DVD player I have aaaand, it starts spinning and spinning and spinning, and nothing shows up in Nautilus. Looking at dmesg I see a few I/O errors on the device.
Then, I try ddrescue [1], start extracting an image, leave it doing its work for a couple of hours and... magic! I have a perfectly mountable disk image!
I got to live again some >20 years old memories, pictures of younger me with my friends. stupid things you would save as a late nerd teen with access to the Internet and a ton of emails in Outlook Express format (guilty!) that I could not open yet. I guess the format is probably not super-hard to parse (most of the content is plain text actually) but since I'm here, if you know of something already existing to convert it to mbox/maildir it would be cool!
Even with ddrescue you might lose a few bits. You will only find out when accessing those specific files and bits.
For my photo history I prefer to keep PAR files next to it. On a running mechanical disc I haven't seen issues when running the repair. I do expect to see some errors in the future and as long as it stays under a certain percentage, things will be fixable.
I started doing this long time in the past, when harddisks were awful. I would often lose mp3 files due to damage. With mp3 you can easily hear it. With jpeg images you can often see damage as well.
I recently got rid of all my CD-Rs I'd use for archiving back in the day. Moved everything to a NAS and dumped CDs.
The earliest CDs I had were from 1994 or so, though most were from early 2000s. They were mostly cheapest CD-Rs you could buy back then, so I was expecting most of them be completely unusable.
But most of them still read perfectly. I had maybe half a dozen (out of hundreds) that were completely unreadable, mostly due to peeling, about a dozen that were functionally unreadable, because there were so many unreadable files I couldn't save anything complete out of them, and maybe 10% that had less than 10 unreadable files on them.
I was surprised. I was really expecting almost all the early ones to be dead.
I recently sampled a CD-R from late 90s, mostly videos. I couldn't open a few files. Storage: inside home, temperature range from maybe 18 at night to 35 max. I'll give a try to ddrescue.
I've oddly noticed zero issues reading CD-R's going back to 1999 as recently as last year. I could certainly have some unnoticed bitrot but compressed archives have restored fine.
I've stopped using burned CDs and DVDs like 15 years ago, long before they were considered obsolete, after losing a huge amount of data, including multiple backups of projects. All of them were kept in their own case on a wall CD/DVD holder, no weights, no humidity, no excessive heat or any sun exposition. After only about 5 years like 30% of them were unreadable at all and others developed errors, so as soon as I realized what was going on I backed up everything I could to hard drives in multiple copies, dumped them all and stopped using that media completely.
Printed media last a lot more, but those CD/DVD R/RW to me are too unreliable to be used for anything beyond the occasional OS image install where USB boot isn't available.
Me seemed to be a little bit luckier as only 80% seemed problematic after around 10 years. But the result were the same, I had to dump all of them directly except those I did not wish to lose. But even after multiple attempts, still suffered very significant loss.
After reviving a PC i had in the late 90s a couple of weeks ago i investigated this big box of CD-Rs standing besides it (in an unisolated attic in germany... so temperatures from about -5 °C to >35 °C), most of the CDs were burnt in a period from around '96 - 2005, many ... ehm... "off site backups" of games and other software but also personal files.
What can i say... every CD i put in the drive worked flawlessly. I think it sometimes comes down to pure dumb luck.
Quality can vary a lot, back in the late 90's you had certain brands and disc you would buy because of how much extra data you could write to them and also testing to see which disc performed better at higher writing speeds. It was a big deal to some folks back then and you likely bought the high quality ones.
I used to be concerned about optical media longevity. I bought Taiyo-Yuden discs. I checked their integrity. I stored them carefully. But the format became obsolete (for me) long before the media degraded. Recordable discs are just too small for the backup use case I had. Optical drives are still available of course, but they are now rarely standard with a new PC or laptop, and soon will become specialist components, and then it won’t be long before the drives become a greater limit to longevity than the media.
After quite a bit of research into this about a year back I decided to archive everything on archive-grade CD-Rs. 20-50 years is more than enough and realistically in the correct climate (a safe or something that humidity would have trouble getting into along with dessicants, etc) could last much longer. I evaluated magnetic tape, HDDs, and a few other options are CD-Rs are as good as magnetic tape (in longevity) and will likely have readers well into the next century.
But you have to be very careful. There are a few brands, in particular Taiyo Yuden that are the absolute best. CMC Pro is a related brand with the same technology/longevity (and easier to get). You will pay quadruple the cost per CD but its worth every penny.
Avoid Memorex, JVC, etc. Basically any brand you can pick up at your local big box retailer.
I've always only bought Taiyo Yuden media since ~2000. Every CD-R and DVD-R I still have that I've tried is still readable.
Taiyo Yuden at least used to be white labeled sometimes, so you'd just look up online what white-label media at Best Buy or CompUSA at the time was Taiyo Yuden.
I haven't burned a disc in probably close to a decade at this point though. SpiderOak is my backup.
There's allegedly nothing different special about blu-ray M-DISCs
>Digging further around the M-disc site, it is noted that their section on longevity testing actually features no data about the BD-R discs at all. Indeed, if there were tests, the reports were not actually featured on the site at all. Further poking around in M-Disc Technology pages shows only references to their DVD products. In fact, it seems their patented “etched in stone” material may only apply to their DVD products.
I have some M-DISC DVD's from ~15 years ago that still read fine.
That said I still have a lot of cheap CDs from 20 years ago that still read fine as well.
I've had a 25GB M-DISC BD-R kicking around on my desk without a sleeve or case for a while now to test durability. Sometimes I take it outside and leave it on my patio table for a few days to add some extra wear. I slide it around the desk, pick it up with normal greasy/dirty hands and handle it non-gently, flex it a bit, wash it off in the sink, and wipe it off with a cotton cloth. After four years of this abuse so far it still hasn't lost a single bit of data. I have noticed some of it takes a bit longer to get a good read so I'll probably lose data soon but this has been a hell of a lot of abuse so far.
yep, I have about 1 TB of data I backed up 5 years ago that I was looking thru. I used the BDXL 100 GB disks.
I’m a huge fan of mdisk but I’m not sure it makes much sense in the era of cheap cloud storage. There are many companies now that will give TB of storage for 9$/month. Also there is cheap archival storage on AWS.
I just started using M-Discs last month. Bought a compatible burner, a Pioneer Blu-ray Disc (BDR-XD07B), CAD $170. I'm pleased to report that my burning software (ImgBurn) worked fine with the new Pioneer DVD drive.
At $10 a pop, one is very prudent when choosing what to backup.
When I was using dual-layer DVD+R blanks for off-site backups, the folk wisdom on blanks I heard from someone who burnt discs a lot was: to get a particular type and brand, and make sure the unit had been manufactured in Singapore. The country/factory of manufacture was indicated on the packaging, but the SKU and UPC didn't reflect that.
More recently, I looked into re-adding optical disks (this time, M-Disc at BD capacities) as one backup medium for current home servers, but the tech manufacturers just seemed even more user-hostile and Linux-hostile than they used to be. Maybe it's due to the intersection with piracy, maybe it's due to market forces of consumers who aren't very savvy about their own interests, but whatever the reason, it didn't look like an attractive option.
(Low-level burning software for optical drives was always crazy, and required some tricky alchemy by saintly open source programmers, as well as a user tolerance for burning the occasional "coaster" (failed burn). I layered my own, simpler, software atop their critical work. But I'm not interested in reinvesting in a category of media that in some ways seems to be getting worse, or even customer-hostile, at the same time that it's fallen out of popularity. If someone has better information, about how some corner of this is rock-solid, non-hostile, and otherwise viable, then I'd be happy to reconsider.)
All of the CDs in my old collection are fine, but I’ve had major problems with my DVD collection. I haven’t spent any time trying to figure it out, but it looks like some kind of unusual mold that I’ve never seen or some kind of deterioration of the substrate occurred inside some of the discs and they became completely unplayable. Attempts at cleaning completely failed to solve the problem. Is there really a major difference between DVD and CD manufacturing on a commercial level?
Print media is often considered 'more permanent' that other digital storage media. But consider: every courthouse I ever visited had records 'back to the fire', the time the records room burned. Every one.
Considering the flammability of paper multiplied by the probability of a fire annually gives you the means to come up with a mean-time-between-failure for paper records. And I'd guess it isn't very long - maybe 50 years to a century.
Unless you have ideal storage conditions, print media won't last more than a few hundred years - and that is with acid free paper, acid treated paper won't last that long.
There is a reason ancient documents are found in Egypt "all the time" and almost no place else in the world: Egypt is an ideal climate to store paper. The rest of the world paper will decay in a few hundred years. (of course modern technology can create better conditions - if we choose to keep it running)
Sprinklers, as well as waterless fire suppression systems, are fairly recent inventions so their mettle has not been really tested on the time scale of document archiving.
There was a panic circa 1980 about the longevity of color cinema film. Martin Scorsese pointed out the cultural loss but it was also a big dollar and cents loss because home video was on its way as a way to turn old movies into cash.
People looked at quite a few different answers, including copying the movies and separating them into three films, one for each color, but the problem was simply solved when people discovered you can greatly extend the life of dyes by storing them in the freezer.
You have to keep them out of light also. Conversely, high temperatures accelerate aging. I have a room in my house (an old farmhouse with uncontrolled humidity) which is very bright and somewhat intemperate (and also has a lot of indoor wildlife) and I’ve found prints made with fugitive dyes fade badly in six months — I didn’t get mad instead I use it for accelerated aging tests, I make two prints and store one in the dark and put the other in the wall.
“Handing someone a demo CD is like saying to them, Hey, you throw this away.”
(Can’t recall the comedian, could be Dimitri Martin)
It’s funny but burned CDs are a great handout for me. I can see through DistroKid that some people loaded them into iTunes and I got a small match but if money for it. Considering how cheap they are and “put in walk away repeat” the process is, I’m still happy to crank them out now and again.
Plus! I have CD labels so they have personality and flair. Also they fit a 40-60 minute DJ set perfectly. Burn, pop in the car, party on the go baby.
> BD-R (dye or non-dye, single layer or dual layer) 5 to 10 years
> BD-R (non-dye, gold metal layer) 10 to 20 years
> BD-RE (erasable Blu-ray) 20 to 50 years
I find hard to believe that info. Seems to be sustained by a single study. That would mean that bluray (specially the BR-R format) is a pretty flawed technology.
My educated guess would be something along these two threads:
* Higher data density => tighter tolerance => same physical degradation is more catastrophic (DVD ~7x density of CD, BD-R ~20/10/5x DVD density = up to 140x CD density)
* Produced later and therefore like all consumer goods, less regard for durability and more regard for absolute cost optimization
I am surprised that we're still stuck at Blu-Ray levels of data storage on disc. I would have thought we would be up to a terabyte or so by now, but everything has stagnated.
Tape remains expensive for the home data hoarder, and there is a point at which cloud storage becomes more expensive than tape.
I should check my archive, but as of years ago, I have yet to find one bad disk. Anyone who cared could easily read up about it: writing at maximum speed produces marginal media. Writing at lower speed produces media with lower initial error rate and higher contrast, so much better longevity.
That was the kind of thing that kind of just seemed logical to me back then. I don't remember reading it anywhere, I think I just had the perception from VHS tape recording lengths.
Once made a quick $5k, because a former client had started using CD-RW for their "enterprise" backups, 10-years later they had unreadable data and contacted me to see if I still had some source-code.
Checked my stack of hard-drives, and... Yes... yes, I did have a working copy.
That cemented it in my personal world - NAS with also HD's is the way to go.
What would one use as a media for long-long-term storage, taking only into account the resisiliance of the ability to read the data from it?
For example if you wanted to store data in a time capsule for like 100 or 200 and you where ganrateed the people on the other side had a reader capable of reading whatever media, and whatever format you chose.
Interesting that it looks like Blu-Rays in all forms have much shorter longevity than the preceding DVD and CD formats. Presumably due to tighter tolerances required to fit more data on the same platter size?
I had switched to large format 50-100GB M-DISC BD-Rs a few years ago, but to be honest I'm often too lazy to bother with the process of burning discs for offline storage.
I should probably upgrade my NAS/external HDD&SDD stack again.
I would advise anyone going the NAS route to think carefully and read reviews on RAID devices. It's probably safer & easier to keep 2 copies in 2 formats on 2 different media in the end. I have had numerous consumer grade NAS RAID devices fail to the point I almost find it an additional failure mode rather than level of safety. Fragility of consumer grade RAID reminds me there's a reason our employers hire storage engineers!
[+] [-] darkwater|3 years ago|reply
I got to live again some >20 years old memories, pictures of younger me with my friends. stupid things you would save as a late nerd teen with access to the Internet and a ton of emails in Outlook Express format (guilty!) that I could not open yet. I guess the format is probably not super-hard to parse (most of the content is plain text actually) but since I'm here, if you know of something already existing to convert it to mbox/maildir it would be cool!
[1] https://www.gnu.org/software/ddrescue/
[+] [-] mpol|3 years ago|reply
For my photo history I prefer to keep PAR files next to it. On a running mechanical disc I haven't seen issues when running the repair. I do expect to see some errors in the future and as long as it stays under a certain percentage, things will be fixable.
I started doing this long time in the past, when harddisks were awful. I would often lose mp3 files due to damage. With mp3 you can easily hear it. With jpeg images you can often see damage as well.
[+] [-] gorbachev|3 years ago|reply
The earliest CDs I had were from 1994 or so, though most were from early 2000s. They were mostly cheapest CD-Rs you could buy back then, so I was expecting most of them be completely unusable.
But most of them still read perfectly. I had maybe half a dozen (out of hundreds) that were completely unreadable, mostly due to peeling, about a dozen that were functionally unreadable, because there were so many unreadable files I couldn't save anything complete out of them, and maybe 10% that had less than 10 unreadable files on them.
I was surprised. I was really expecting almost all the early ones to be dead.
[+] [-] pmontra|3 years ago|reply
[+] [-] grouseway|3 years ago|reply
https://learn.microsoft.com/en-us/windows/win32/stg/structur...
e.g. https://www.mitec.cz/ssv.html
[+] [-] iicc|3 years ago|reply
http://fileformats.archiveteam.org/wiki/Outlook_Express_Data...
[+] [-] ja27|3 years ago|reply
[+] [-] dehrmann|3 years ago|reply
[+] [-] polygloty|3 years ago|reply
[+] [-] xerox13ster|3 years ago|reply
[+] [-] shostack|3 years ago|reply
[+] [-] squarefoot|3 years ago|reply
[+] [-] yosito|3 years ago|reply
[+] [-] bitL|3 years ago|reply
[+] [-] mrjin|3 years ago|reply
[+] [-] RalfWausE|3 years ago|reply
What can i say... every CD i put in the drive worked flawlessly. I think it sometimes comes down to pure dumb luck.
[+] [-] nekoashide|3 years ago|reply
[+] [-] jl6|3 years ago|reply
[+] [-] Test0129|3 years ago|reply
But you have to be very careful. There are a few brands, in particular Taiyo Yuden that are the absolute best. CMC Pro is a related brand with the same technology/longevity (and easier to get). You will pay quadruple the cost per CD but its worth every penny.
Avoid Memorex, JVC, etc. Basically any brand you can pick up at your local big box retailer.
[+] [-] TedDoesntTalk|3 years ago|reply
I have not been able to buy these in the last few years. Also, when I was able to buy them, one had to be careful of counterfeits.
Do you have a source for them?
[+] [-] cdr|3 years ago|reply
Taiyo Yuden at least used to be white labeled sometimes, so you'd just look up online what white-label media at Best Buy or CompUSA at the time was Taiyo Yuden.
I haven't burned a disc in probably close to a decade at this point though. SpiderOak is my backup.
[+] [-] ridgered4|3 years ago|reply
[+] [-] mixmastamyk|3 years ago|reply
[+] [-] smarx007|3 years ago|reply
https://en.wikipedia.org/wiki/M-DISC
https://web.archive.org/web/20130422110903if_/http://www.mdi...
[+] [-] gruez|3 years ago|reply
>Digging further around the M-disc site, it is noted that their section on longevity testing actually features no data about the BD-R discs at all. Indeed, if there were tests, the reports were not actually featured on the site at all. Further poking around in M-Disc Technology pages shows only references to their DVD products. In fact, it seems their patented “etched in stone” material may only apply to their DVD products.
https://goughlui.com/2015/10/16/review-tested-verbatim-lifet...
[+] [-] vel0city|3 years ago|reply
That said I still have a lot of cheap CDs from 20 years ago that still read fine as well.
I've had a 25GB M-DISC BD-R kicking around on my desk without a sleeve or case for a while now to test durability. Sometimes I take it outside and leave it on my patio table for a few days to add some extra wear. I slide it around the desk, pick it up with normal greasy/dirty hands and handle it non-gently, flex it a bit, wash it off in the sink, and wipe it off with a cotton cloth. After four years of this abuse so far it still hasn't lost a single bit of data. I have noticed some of it takes a bit longer to get a good read so I'll probably lose data soon but this has been a hell of a lot of abuse so far.
[+] [-] jti107|3 years ago|reply
I’m a huge fan of mdisk but I’m not sure it makes much sense in the era of cheap cloud storage. There are many companies now that will give TB of storage for 9$/month. Also there is cheap archival storage on AWS.
[+] [-] GnarfGnarf|3 years ago|reply
At $10 a pop, one is very prudent when choosing what to backup.
[+] [-] neilv|3 years ago|reply
More recently, I looked into re-adding optical disks (this time, M-Disc at BD capacities) as one backup medium for current home servers, but the tech manufacturers just seemed even more user-hostile and Linux-hostile than they used to be. Maybe it's due to the intersection with piracy, maybe it's due to market forces of consumers who aren't very savvy about their own interests, but whatever the reason, it didn't look like an attractive option.
(Low-level burning software for optical drives was always crazy, and required some tricky alchemy by saintly open source programmers, as well as a user tolerance for burning the occasional "coaster" (failed burn). I layered my own, simpler, software atop their critical work. But I'm not interested in reinvesting in a category of media that in some ways seems to be getting worse, or even customer-hostile, at the same time that it's fallen out of popularity. If someone has better information, about how some corner of this is rock-solid, non-hostile, and otherwise viable, then I'd be happy to reconsider.)
[+] [-] bibaheu|3 years ago|reply
[+] [-] sammalloy|3 years ago|reply
[+] [-] JoeAltmaier|3 years ago|reply
Considering the flammability of paper multiplied by the probability of a fire annually gives you the means to come up with a mean-time-between-failure for paper records. And I'd guess it isn't very long - maybe 50 years to a century.
[+] [-] bluGill|3 years ago|reply
There is a reason ancient documents are found in Egypt "all the time" and almost no place else in the world: Egypt is an ideal climate to store paper. The rest of the world paper will decay in a few hundred years. (of course modern technology can create better conditions - if we choose to keep it running)
[+] [-] xattt|3 years ago|reply
[+] [-] PaulHoule|3 years ago|reply
People looked at quite a few different answers, including copying the movies and separating them into three films, one for each color, but the problem was simply solved when people discovered you can greatly extend the life of dyes by storing them in the freezer.
You have to keep them out of light also. Conversely, high temperatures accelerate aging. I have a room in my house (an old farmhouse with uncontrolled humidity) which is very bright and somewhat intemperate (and also has a lot of indoor wildlife) and I’ve found prints made with fugitive dyes fade badly in six months — I didn’t get mad instead I use it for accelerated aging tests, I make two prints and store one in the dark and put the other in the wall.
There is a book about this stuff at
http://wilhelm-research.com/
[+] [-] 6stringmerc|3 years ago|reply
(Can’t recall the comedian, could be Dimitri Martin)
It’s funny but burned CDs are a great handout for me. I can see through DistroKid that some people loaded them into iTunes and I got a small match but if money for it. Considering how cheap they are and “put in walk away repeat” the process is, I’m still happy to crank them out now and again.
Plus! I have CD labels so they have personality and flair. Also they fit a 40-60 minute DJ set perfectly. Burn, pop in the car, party on the go baby.
[+] [-] FroshKiller|3 years ago|reply
[+] [-] mrfinn|3 years ago|reply
> BD-R (non-dye, gold metal layer) 10 to 20 years
> BD-RE (erasable Blu-ray) 20 to 50 years
I find hard to believe that info. Seems to be sustained by a single study. That would mean that bluray (specially the BR-R format) is a pretty flawed technology.
[+] [-] ricardobeat|3 years ago|reply
[+] [-] steveBK123|3 years ago|reply
* Higher data density => tighter tolerance => same physical degradation is more catastrophic (DVD ~7x density of CD, BD-R ~20/10/5x DVD density = up to 140x CD density)
* Produced later and therefore like all consumer goods, less regard for durability and more regard for absolute cost optimization
[+] [-] _int3_|3 years ago|reply
[+] [-] at_a_remove|3 years ago|reply
Tape remains expensive for the home data hoarder, and there is a point at which cloud storage becomes more expensive than tape.
[+] [-] ahartmetz|3 years ago|reply
[+] [-] xerox13ster|3 years ago|reply
[+] [-] rvieira|3 years ago|reply
These are all made from my personal music collection (which is not backed up in optical media).
Just saying, because CDs to me became just a transient media to, say, transport music to my car's stereo[1].
Because of this I never really bothered about longevity, they are just digital cassettes.
[1] - Yes, I could replace my car CD player with something else, but now it's a nostalgia thing :)
[+] [-] hyperluz|3 years ago|reply
[+] [-] stared|3 years ago|reply
[+] [-] jjkaczor|3 years ago|reply
Checked my stack of hard-drives, and... Yes... yes, I did have a working copy.
That cemented it in my personal world - NAS with also HD's is the way to go.
[+] [-] OptionX|3 years ago|reply
For example if you wanted to store data in a time capsule for like 100 or 200 and you where ganrateed the people on the other side had a reader capable of reading whatever media, and whatever format you chose.
[+] [-] steveBK123|3 years ago|reply
I had switched to large format 50-100GB M-DISC BD-Rs a few years ago, but to be honest I'm often too lazy to bother with the process of burning discs for offline storage.
I should probably upgrade my NAS/external HDD&SDD stack again. I would advise anyone going the NAS route to think carefully and read reviews on RAID devices. It's probably safer & easier to keep 2 copies in 2 formats on 2 different media in the end. I have had numerous consumer grade NAS RAID devices fail to the point I almost find it an additional failure mode rather than level of safety. Fragility of consumer grade RAID reminds me there's a reason our employers hire storage engineers!