This came in handy once when I accidentally did the equivalent of rm -rf ~/git. Lost 54 minutes of work.
Arq is Mac-only, but you could accomplish the same thing with https://www.tarsnap.com/ and a cron job. Both are changes-only backups so there shouldn't be a huge performance hit.
I also use Time Machine when I'm at home but only sort of trust it.
On Linux it is extremely easy do use rdiff-backup for local and remote incremental backups. When I work, I have a minutely cronjob for my work directory. For the whole system I have a hourly schedule. When I resume work the next day, the previous mess of diff files from the work rdiff target directory is deleted and a new clean "emergency" cycle is started. Losing one hour of great concentrated coding can be extremely annoying and hard to reproduce so this is a cheap way to prevent it. But I guess even an hourly backup would be a good first step for many people.
Remember, a harddisk probably costs less than one day of your work. Or just get an USB stick for the most important bits.
Has anyone got any experience with using Dropbox as a backup service? They provide 30 days of file recovery for free, and for a few dollars a month you can get packrat, which provides eternal point-in-time recovery. Seems like a great use case of a git folder backup?
I have a Intel 520 SSD. Fantastic thing I must say and I don't think I can ever use a HDD again, but somehow -- and I realize this may just be baseless -- I don't trust it as much as I trusted my old Seagate HDD, which lasted 7 years.
SSDs are getting better now, but are they as reliable as HDD? Can anyone shares his/her expertise on this?
From what I've read online, SSD now should last the same, if not more than HDD.
The nice thing about them is that you know how many writes they should handle, and get a new one before some disaster happen. Sure, you can get a faulty unit, but you could get one with HDDs as well.
Instead, for HDDs, you couldn't be really sure. Yes, there's the "broken sector" count, but this isn't always reliable. Disaster could always happen randomly, without any warning before.
If you're interested in more stat, some sites are starting to run endurance tests [1][2][3][4], to see how much on average an SSD is supposed to last, and which brand is better.
Anyway, the only real solution is to run a raid, and keep frequent backups.
You can certainly buy SSDs now that are rated to more read/write cycles than your 7-year-old Seagate will have gone through. The ones we buy are rated for 3+ years of constant I/O at full bus speed. I don't know how much of that tech is in the sort of SSD you're likely to buy for a desktop, but they're certainly available.
I've been using Intel SSDs in a datacenter for over 3 years now. X25-E, X25-M, 320, 520, DC S3500, DC S3700. Over 1,000 of them. Not one has failed. Just my experience though.
A few are getting low on their media wear indicator (25% remaining) but they are getting replaced due to capacity issues.
My SSD died. Much like a USB drive, when it died, I could still read from it. Just made a direct ghost of it onto another SSD. Was up and running in less than 15 minutes.
SSDs are fast - but you better back them up. When an SSD fails its usually complete and without warning.
Spinning drives will often give you some warning signs, clicking, slowing down, bad sector errors etc.
Spinning disks can be rebuilt with new heads or platter swapped into a new body.
With SSDs you cant just remove the NAND chips and recover the data either. SSDs use dynamic wear leveling to shuffle pages of blocks around - in other words - block 0 is not at block 0 -it could be anywhere... and if the internal map is lost - good luck.
The data in the NAND chips is also encrypted... they do this to ensure that erased sectors are actually erased. Because of the wear-leveling routine spies could possibly recover data from unused sectors - the encryption makes this impossible.
And for extra fun - intel likes to epoxy the chips to the board - just to be sure they don't fall off....
I compare them to a hover-car, with wheels you can get a flat and still make it off the freeway... but with the hover-car your just a sitting duck...
You don't have a reliable option. It's that simple.
If you value your data, keep backups. And if you want to avoid hiccups in productivity, use RAID. Data that you don't have backed up is data that you don't have.
And keep in mind that RAID, while useful, is not a backup. It doesn't protect against theft, whole-computer destruction, human error, or many other things.
In our data center we have several thousand hard drives and nearly two thousand SSDs churning away day in and day out, indexing and caching Etc. So far the failure rates has been about 1/10th as much for the SSDs per-capita than the Hard drives.
Not definitive, just a data point. (oh and the SSDs are the Intel X25-M (160G version))
> SSDs are getting better now, but are they as reliable as HDD? Can anyone shares his/her expertise on this?
I'm especially interested to hear from people who use many discs, and if they can share some information about the kind of use those discs get.
From what I've heard SSDs are more reliable that traditional spinning platter drives, unless you have a bad batch. This is especially true if there is more than one platter in the drive.
But I also recognise that it doesn't really matter. All drives fail, and no drive is reliable, so you assume the life is going to be 3 years and you have robust mirroring and backups and replacement policies.
It really depends. If you keep your PC running all the time and are always writing to it at pretty high bandwidth then you'll wear out a SSD really quickly and a HDD won't have the spinup/spindown cycles that impact it's lifetime. If you're not in that group than, on a mechanical level, a SSD will tend to last much longer.
On the other hand, the abstraction layer that lets a SSD be so reliable is complicated and it's possible for companies to screw up complicated things. There have been several instances of SSD firmware bugs that have caused data loss for users before they were fixed, so maybe be wary of new SSD models.
I got a new Thinkpad with an SSD at my job last year and it died after 10 months. Luckily I had pushed everything but it was still mighty inconvenient. I just put one in my desktop and will be backing up religiously.
This is just one data point, I know, but I have never (knock on wood) had a hard drive of any kind fail before.
On the Mac with Time Machine, I can lose at most an hour of work. Not saying that to praise the Mac; I'm saying that because that's how all computers ought to be.
This would be a swell time for Dropbox to offer Linus a free large business account so he can store everything.
I ha(ve/d) a small external hard drive enclosure which holds/held two 2 TB drives in RAID-1 mode (recently tested that it’s indeed RAID 1: The enclosure left out smoke and one of the drives now works happily in a single-drive enclosure), attache(s/d) via eSATA and is used every hour by an rsync backup run using hard links. Every evening, I run another script that cleans up most of the older backups of the last week, keeping one per day.
So, yes, while at home, I also lose at most one hour of stuff, provided that nothing catastrophic happens that damages both the drive and the notebook – unfortunately until very recently, the internet connection to backup offsite was simply not there, and even with 50/10 MBit/s it will be annoying.
First thing I thought, Linus should evaluate a MacBook. If I remember correctly he once stated that the MacBook Air was the perfect machine for him, so it's not that far-fetched.
I find if vaguely disturbing that John Stoffel suggests to Linus that he mirror two discs in the future (presumably meaning a RAID-1 arrangement) and Linus responds with "I long ago gave up on doing backups."
Linus conflating RAID with backup?!? Surely not. My world feels turned upside down... >smile<
If you are doing something as important as this, why aren't you at least running some sort of RAID setup that would have been fine with losing a disk? Or at the very minimum had a decent backup setup so that you would not lose data.
For someone who created Linux in the first place, this must/should be very embarrassing.
OTOH, it must be nice to know that even if your house blows up (hopefully when you're not in it), your life's work is mirrored and backed up on millions of computers. All he lost were some pull requests which have to be re-sent. The submitters know who they are and if the changes don't show up in a day they just send another email - no harm, no foul.
So... does this mean that Apple's flash storage is low-quality?
I use the same MacBook Air that Linus does (except it's a 13" model), never turn it off, have reinstalled the OS multiple times, archive my data, do around ~3GB worth of transfers every day, and run 3 different VMs more often than not.
Maybe I should just upgrade to a newer SSD. Anyone had luck with OWC?
[+] [-] masnick|12 years ago|reply
This came in handy once when I accidentally did the equivalent of rm -rf ~/git. Lost 54 minutes of work.
Arq is Mac-only, but you could accomplish the same thing with https://www.tarsnap.com/ and a cron job. Both are changes-only backups so there shouldn't be a huge performance hit.
I also use Time Machine when I'm at home but only sort of trust it.
[+] [-] hnha|12 years ago|reply
Remember, a harddisk probably costs less than one day of your work. Or just get an USB stick for the most important bits.
[+] [-] stingraycharles|12 years ago|reply
[+] [-] thex86|12 years ago|reply
SSDs are getting better now, but are they as reliable as HDD? Can anyone shares his/her expertise on this?
[+] [-] Spittie|12 years ago|reply
The nice thing about them is that you know how many writes they should handle, and get a new one before some disaster happen. Sure, you can get a faulty unit, but you could get one with HDDs as well.
Instead, for HDDs, you couldn't be really sure. Yes, there's the "broken sector" count, but this isn't always reliable. Disaster could always happen randomly, without any warning before.
If you're interested in more stat, some sites are starting to run endurance tests [1][2][3][4], to see how much on average an SSD is supposed to last, and which brand is better.
Anyway, the only real solution is to run a raid, and keep frequent backups.
[1] http://techreport.com/review/24841/introducing-the-ssd-endur... / http://techreport.com/review/25320/the-ssd-endurance-experim...
[2] http://www.xtremesystems.org/forums/showthread.php?271063-SS...
[3] http://ssdendurancetest.com/
[4] "SSD Endurance test" on Google
[+] [-] regularfry|12 years ago|reply
[+] [-] chrisbolt|12 years ago|reply
A few are getting low on their media wear indicator (25% remaining) but they are getting replaced due to capacity issues.
[+] [-] ambiate|12 years ago|reply
[+] [-] wayne_h|12 years ago|reply
SSDs are fast - but you better back them up. When an SSD fails its usually complete and without warning.
Spinning drives will often give you some warning signs, clicking, slowing down, bad sector errors etc.
Spinning disks can be rebuilt with new heads or platter swapped into a new body.
With SSDs you cant just remove the NAND chips and recover the data either. SSDs use dynamic wear leveling to shuffle pages of blocks around - in other words - block 0 is not at block 0 -it could be anywhere... and if the internal map is lost - good luck.
The data in the NAND chips is also encrypted... they do this to ensure that erased sectors are actually erased. Because of the wear-leveling routine spies could possibly recover data from unused sectors - the encryption makes this impossible.
And for extra fun - intel likes to epoxy the chips to the board - just to be sure they don't fall off....
I compare them to a hover-car, with wheels you can get a flat and still make it off the freeway... but with the hover-car your just a sitting duck...
[+] [-] ori_b|12 years ago|reply
If you value your data, keep backups. And if you want to avoid hiccups in productivity, use RAID. Data that you don't have backed up is data that you don't have.
And keep in mind that RAID, while useful, is not a backup. It doesn't protect against theft, whole-computer destruction, human error, or many other things.
[+] [-] ChuckMcM|12 years ago|reply
Not definitive, just a data point. (oh and the SSDs are the Intel X25-M (160G version))
[+] [-] DanBC|12 years ago|reply
I'm especially interested to hear from people who use many discs, and if they can share some information about the kind of use those discs get.
From what I've heard SSDs are more reliable that traditional spinning platter drives, unless you have a bad batch. This is especially true if there is more than one platter in the drive.
But I also recognise that it doesn't really matter. All drives fail, and no drive is reliable, so you assume the life is going to be 3 years and you have robust mirroring and backups and replacement policies.
[+] [-] Symmetry|12 years ago|reply
On the other hand, the abstraction layer that lets a SSD be so reliable is complicated and it's possible for companies to screw up complicated things. There have been several instances of SSD firmware bugs that have caused data loss for users before they were fixed, so maybe be wary of new SSD models.
[+] [-] wizzard|12 years ago|reply
This is just one data point, I know, but I have never (knock on wood) had a hard drive of any kind fail before.
[+] [-] cjensen|12 years ago|reply
This would be a swell time for Dropbox to offer Linus a free large business account so he can store everything.
[+] [-] claudius|12 years ago|reply
So, yes, while at home, I also lose at most one hour of stuff, provided that nothing catastrophic happens that damages both the drive and the notebook – unfortunately until very recently, the internet connection to backup offsite was simply not there, and even with 50/10 MBit/s it will be annoying.
[+] [-] quarterto|12 years ago|reply
[+] [-] cliveowen|12 years ago|reply
[+] [-] primelens|12 years ago|reply
[+] [-] EvanAnderson|12 years ago|reply
Linus conflating RAID with backup?!? Surely not. My world feels turned upside down... >smile<
[+] [-] Dylan16807|12 years ago|reply
[+] [-] unknown|12 years ago|reply
[deleted]
[+] [-] flexd|12 years ago|reply
For someone who created Linux in the first place, this must/should be very embarrassing.
[+] [-] primelens|12 years ago|reply
[+] [-] lutusp|12 years ago|reply
http://www.wired.com/wiredenterprise/2012/10/linus-torvalds-...
And when Linus speaks, a lot of people listen.
[+] [-] bsg75|12 years ago|reply
[+] [-] consider_this|12 years ago|reply
It is only the linux kernel afterall.
[+] [-] jamesaguilar|12 years ago|reply
[+] [-] hakcermani|12 years ago|reply
[+] [-] astrodust|12 years ago|reply
[+] [-] kunai|12 years ago|reply
I use the same MacBook Air that Linus does (except it's a 13" model), never turn it off, have reinstalled the OS multiple times, archive my data, do around ~3GB worth of transfers every day, and run 3 different VMs more often than not.
Maybe I should just upgrade to a newer SSD. Anyone had luck with OWC?
[+] [-] hnha|12 years ago|reply
[+] [-] hannibal5|12 years ago|reply
"Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;)"
source: https://groups.google.com/forum/#!msg/linux.dev.kernel/2OEgU...
[+] [-] LukeShu|12 years ago|reply