I was at Pixar when this happened, but I didn't hear all of the gory details, as I was in the Tools group, not Production. My memory of a conversation I had with the main System Administrator as to why the backup was not complete was that they were using a 32-bit version of tar and some of the filesystems being backed up were larger than 2GB. The script doing the backup did not catch the error. That may seems sloppy, but this sort of thing happens in the Real World all the time. At the risk of spilling secrets, I'll tell a story about the animation system, which I worked on (in the 1996-97 time frame).
The Pixar animation system at the time was written in K&R C and one of my tasks was to migrate it to ANSI C. As I did that I learned that there were aspects of this code that felt like a school assignment that had escaped from the lab. While searching for a bug, I noticed that the write() call that saved the animation data for a shot wasn't checked for errors. This seemed like a bad idea, since at the time the animation workstations were SGI systems with relatively small SCSI disks that could fill up easily. When this happened, the animation system usually would crash and work would be lost. So, I added an error check, and also code that would save the animation data to an NFS volume if the write to the local disk failed. Finally, it printed a message assuring the animator that her files were safe and it emailed a support address so they could come help. The animators loved it! I had left Pixar by the time the big crunch came in 1999 to remake TS2 in just 9 months, so I didn't see that madness first hand. But I'd like to think that TS2 is just a little, tiny bit prettier thanks to my emergency backup code that kept the animators and lighting TDs from having to redo shots they barely had time to do right the first time.
The point is that one would like to think that a place like Pixar is a model of Software Engineering Excellence, but the truth is more complex. Under the pressures of Production deadlines, sometime you just have to get it to work and hope you can clean it up later. I see the same things at NASA, where, for the most part, only Flight Software gets the full on Software Engineering treatment.
Right on the money with the "Real World" anecdote.
We do penetration tests for a wide range of clients across many industries. I would say that the bigger the company, the more childish flaws we find. For sure the complexity, scale, and multiple systems do not help towards having a good security posture , but never assume that because you are auditing a SWIFT backend you will not find anything that can lead to direct compromise.
Maybe not surprisingly, most startups that we work with have a better security posture than F500 companies. They tend to use the latest frameworks that do a good job of protecting against the standard issues, and their relatively small attack landscape doesn't leave you with much to play.
One of the really interesting artifacts from the NASA flight software programs is that it helps put an upper bound of god honest ground truth level of effort to produce "perfect" software. Everything else we do is approximation to some level of fidelity. The only thing even reasonably close is maybe SQLite, and most people think the testing code for it is about 10x overkill.
It makes one start to contemplate how little we really understand about software and how nascent the field really is. We're basically stacking rocks in a modern age where other engineering disciplines are building half-km tall buildings and mile-spanning bridges.
Fast forward 2500 years and the software building techniques of the future must be as unrecognizable to us as rocket ships are to people who build mud huts.
> The script doing the backup did not catch the error. That may seems sloppy, but this sort of thing happens in the Real World all the time.
I disagree. I mean, I agree those things happen, but the system administrator's job is to anticipate those Real World risks and manage them with tools like quality assurance, redundancy, plain old focus and effort, and many others.
The fundamental of backups is to test restoring them, which would have caught the problem described. It's so basic that it's a well-known rookie error and a source of jokes like, 'My backup was perfect; it was the restore that failed.' What is a backup that can't be restored?
Also, in managing those Real World risks, the system administrator has to prioritize the value of data. The company's internal newsletter gets one level of care, HR and payroll another. The company's most valuable asset and work product, worth hundreds of millions of dollars? A personal mission, no mistakes are permitted; check and recheck, hire someone from the outside, create redundant systems, etc. It's also a failure of the CIO, who should have been absolutely sure of the data's safety even if he/she had to personally test the restore, and the CEO too.
Cool story :) You are bang on point when it comes to software engineering at what are thought to be "top tier" development houses. In the ideal world sure they will build the very best software but the real world has [unrealistic] deadlines and when you have deadlines it means corners get cut. Not always but very often. This leads to the whole "does it do exactly what is required?" and if it does then you are moved onto the next thing often with the "promise" that you will be able to go back and "fix things" at a later date. Of course we all know that promise is never kept.
"Backups aren't backups until they've been tested."
They really are Schrödinger's backups until a test restore takes place. This is one area where people cut corners a lot because no one cares about backups until they need them. But it's worth the effort to do them right, including occasional, scheduled manual testing. If you can't restore the data/system you're going to be the one working insane hours to get things working when a failure occurs.
And then there's the aftermath. Unless you are lucky enough to work for a blame free organization major data loss in a critical app due to a failure of the backup system (or lack thereof) could be a resume generating event. If you're ordered to prioritize other things over backups make sure you get that in writing. Backups are something everyone agrees is "critical" but no one wants to invest time in.
I've heard about NASA's Flight Software teams being very strict on 9-5 work hours, with lots of code review and tests. I was under the impression this wasn't as strict with the competition from SpaceX and Blue Origin now that we aren't sending people to space on our (USA) own rockets. Is my impression incorrect?
My recollection of the details is lacking but this jives with what I remember about a talk I attended by a Pixar sysadmin. I think there was only a couple slides about it since it was just one part of a "journey from there to here" presentation about how they managed and versioned large binary assets with Perforce.
There are other anecdotes online about this catastrophic data loss and backup failure but I think it was, funny enough, the propensity for some end users of Perforce to mirror the entire repository that saved their bacon. I say funny because this is something a Perforce administrator will generally bark at you about since your sync of this enormous monolithic repo would be accompanied by an associated server side process that would run as long as your sync took to finish and thanks to some weird quirks of the Perforce SCM long running processes were bad and would/(could?) fuck up everyone else's day. In fact I think a recommendation from Pixar was to automatically kill long running processes server side and encourage smaller workspaces. Anyway, I digress. They were able to piece it together using local copies from certain workstations that contained all or most of the repo. Bad practices ended up saving the day.
Was that menv? I've heard stories that Pixar builds these crazy custom apps that rival things like 3D Studio and Maya but that never leave their campus!
i think it might have turned out better if they had lost the movie - Toy Story I and III have really good plots, but the screenplay of Toy Story II isn't that stellar; is it possible that they would have changed the story of II if it had been lost? (Mr. Potato head might have said that they lost the movie on purpose)
The biggest difference, I think, was leaving the hunting for a head for a second moment, or even not doing it at all.
Commitment would be very different if people were being asked to help while some heads were rolling. Because you're a real team when everybody is going in the same direction. Any call on "people, work hard do recover while we're after the moron who deleted everything" wouldn't have done it.
You just commit to something when you know that you won't be under the fire if you do something wrong without knowing it.
I never understood the attitude of some companies to fire an employee immediately if they make a mistake such as accidentally deleting some files. If you keep this employee, then you can e pretty sure he'll never made that mistake again. If you fire him and hire someone else, that person might not have had the learning experience of completely screwing up a system.
I think that employees actually makes less mistakes and are more productive if they don't have be worried about being fired for making a mistake.
Very good point. Aviation is awesome in that sense - accident investigations are focused on understanding what happened, and preventing re-occurence. Allocating blame or punishment are not part of it, at least in enlightened jurisdictions.
Furthermore, a lot of individual error's are seen in an institutionalised "systems" framework - given that people invariably will make mistakes, how can we set up the environment/institutions/systems so that errors are not catastrophic.
Not sure how that applies to movie animation, to be honest, but not primarily looking for whom to blame was certainly a very good move.
I remember this from the Field Guide to Understanding Human Error. Making recovering from human error a well-understood process is important, and as you point out, that process will work best if people aren't distracted by butt-covering.
The fault here does not lie with just one person. One person ran the rm -rf command. Other people failed to check the backups. Others made the decision to give everyone full root access. Really it was a large part of the company that was to blame.
Ed Catmull discusses this incident thoroughly in Creativity Inc.. He believed seeking retribution for this incident would've been counterproductive and discouraged Pixar's overall ethos as a safe place to experiment and make mistakes. It is this ethos and culture of vociferous, thorough experimentation and casting everyone's performance in the light of "What can we learn from that?" rather than "What ROI did we get from the last 3 months?" that Catmull credits for Pixar's success (paraphrasing here, but I believe this is an accurate summary).
Since Catmull has an engineering background (his PhD involved the invention of the Z-buffer, and he was doing computer graphics before anyone knew anything about it), he understands that mistakes and failed projects, when combined with an forthright and collaborative feedback loop, are not problematic detours, but rather necessary mile markers on the path to real innovation. We'd be so much further ahead if we put more men like Catmull in charge of things.
The biggest problem with reading Creativity Inc. is that it will rekindle the hope that there may be a sane workplace out there somewhere, when practically speaking, we know that few of us will ever find employment in one. It gave me a number of disquieting feelings as I read that the attributes of a workplace that all good engineers crave actually can and sometimes do exist out there. I had convinced myself that these things were myths, so now I'm sad that my boss isn't Ed Catmull.
That said, I do believe some evaluation and/or discipline would've been appropriate in this case, not for the person who accidentally executed a command in the wrong directory, but for the people who were supposed to be maintaining backups and data integrity.
Assuming that your primary job duties involve data integrity and system uptime, having non-functional backups of truly critical data stretches beyond the scope of "mistakes" and into the scope of incompetence.
It is, I'm sure, very possible that no one was really assigned this task at Pixar and that it would therefore by improper to punish anyone in particular for the failure to execute it, but I do believe there is a limit between mistakes en route to innovation and negligence. My experience has been that most companies strongly take one tack or the other: they either let heads roll for minor infractions (and thus never allow good people to get established and comfortable), or they never fire anyone and let the dead weight and political fiefdoms gum up the works until the gears stop altogether. There needs to be a balance, and that's a very hard thing to find out there.
There was an incident where I work where an employee (a new hire) set up a cron job to delete his local code tree, re-sync a new copy, then re-build it using a cron job every night. A completely reasonable thing for a coder to automate.
In his crontab he put:
rm -rf /$MY_TREE_ROOT
and as everyone undoubtedly first discovered by accident, the crontab environment is stripped bare of all your ordinary shell environment. So $MY_TREE_ROOT expanded to "".
The crontab ran on Friday, IIRC, and got most of the way through deleting the entire project over the weekend before a lead came in and noticed things were disappearing. Work was more or less halted for several days while the supes worked to restore everything from backup.
Could the blunder have been prevented? Yes, probably with a higher degree of caution, but that level of subtlety in a coding mistake is made regularly by most people (especially someone right out of university); he was just unlucky that the consequences were catastrophic, and that he tripped over the weird way crontab works in the worst possible usage case. He probably even tested it in his shell. We all know to quadruple-check our rm-rfs, but we know that because we've all made (smaller) mistakes like his. It could have been anyone.
Dragging him to the guillotine would have solved nothing. In fact, the real question is "how is it possible for such an easy mistake to hose the entire project?" Some relatively small permissions changes at the directory root of the master project probably would have stopped stray `rm -rf`s from escaping local machines without severely sacrificing the frictionless environment that comes from an open-permissions shop. So if anything, the failure was systems's fault for setting up a filesystem that can be hosed so easily and so completely by an honest mistake.
The correct thing to do was (and is) to skip the witch hunt, and focus on problem-solving. I am not sure, but I think the employee was eventually hired on at the end of his internship.
For me the principle is: Standards and habits are teachable. Competence and attitude, less so. Educate and train for the former, and a failure of the former should cause you to look first at your procedures, not the people. Hire and fire for the latter.
> You just commit to something when you know that you won't be under the fire if you do something wrong without knowing it.
The other side is if you play a key role (and head could roll after the hard work is done) to simply leverage that fact (perhaps with others) as an advantage such that you have get a new contract and can't be fired for X amount of time.
In the comments, Oren addressed exactly that particular question:
"We didn't scrap the models, but yes, we scrapped almost all the animation and almost all the layout and all the lighting. And it was worth it.
Changing the script saved the film, which in turn allowed Buzz and Woody to carry on for future generation (see ToyStory3 for how awesome that universe continues to be - well done to everyone who worked on the lastest installment!) and, in some ways, set a major cornerstone in the culture of Pixar. You may have heard John or Steve or Ed mention "Quality is a good business model" over the years. Well, that moment in Pixar's history was when we tested that, and it was hard, but thankfully I think we passed the test. Toy Story 2 went on to became one of the most successful films we ever made by almost any measure.
So, suffice it to say that yes, the 2nd version (which you saw in theatres and is now on the BluRay) is about a bagillion times better than the film we were working on. The story talent at the studio came together in a pretty incredible way and reworked everything. When they came back from the winter holidays in January '99, their pitch, and Steve Jobs's rallying cry that we could in fact get it done by the deadline later that year, are a few of the most vivid and moving memories I have of my 20 years at the studio."
A large percentage of files in a CG production are not shot specific, so no, the recovery work was definitely not wasted. There are sets, lighting setups, props, layouts, models, textures, shaders, character rigs, procedural and effects systems, etc., etc. A few of those things might have to be redone, but when those things are set up and the script changes, the main bulk of the work is cameras and character animation, and then re-rendering.
I believe that the original version that was scrapped was intended to be a straight-to-video release. It was completely reworked when the company decided to give the project bigscreen treatment.
I worked at a few VFX studios, and everyone has deleted large swathes of shit by accident.
My favourite was when an rsync script went rogue and started deleted the entire /job directory in reverse alphabet order. Mama-mia[1] was totally wiped out, as was Wanted (that was missing some backups, so some poor fucker had to go round fishing assets out of /tmp, from around 2000 machines.)
From what I recall (this was ~2008) There was some confusion as to what was doing the deleting. Because we had at the time a large(100 or 300tb[2]) lustre file system, it didn't really give you that many clues. They had to wait till it went on a plain old NFS box before they could figure out what was causing it.
Another time honoured classic is matte painters on OSX boxes accidentally dragging whole film directories into random places.
[1]some films have code names, hence why this was first
[2]That lustre was big, physically and IO, it could sustain something like 2-5 gigabytes a second, It had at least 12 racks of disks. Now a 4u disk shelf and one server can do ~2gigabyes sustained
We lost a good chunk of Tintin (I think) when someone tried to use the OSX migration assistant to upgrade a Macbook that had some production volumes NFS mounted. It was trying in vain to copy several PB of data (I am convinced that nobody at Apple has ever seen or heard of NFS), and because it was so slow the user hit cancelled and it somehow tried to undo the copy and started deleting all the files on the NFS servers.
There was another incident where there was a script that ran out of cron to cleanup old files in /tmp, and someone NFS mounted a production volume into /tmp...
Eventually we put tarpit directories at the top of each filesystem (a directory with 1000 subdirectories each with 1000 subdirectories, several layers deep) to try and catch deletes like the one you saw, then we would alert if any directories in there were deleted so we could frantically try and figure out which workstation was doing the damage.
In the most recent VFX company I worked at, with some of the same guys, the backup sys was fucking ace. Firstly rm was aliased to a script that just moved stuff, not deleted it.
Second, there were very large nearlines that took hourly snapshots. Finally, lots and lots of tape for archive.
> The command that had been run was most likely ‘rm -r -f *’, which—roughly speaking—commands the system to begin removing every file below the current directory. This is commonly used to clear out a subset of unwanted files. Unfortunately, someone on the system had run the command at the root level of the Toy Story 2 project and the system was recursively tracking down through the file structure and deleting its way out like a worm eating its way out from the core of an apple.
As a linux neophyte, I once struggled to remember whether the trailing slash on a directory was important. So I typed "rm -rf", and pasted the directory name "schoolwork/project1 " (with a trailing space), but then I waffled and decided to add a trailing slash. So I changed it to "rm -rf schoolwork/project1 /".
rm just unlinks the files at the inode level, seems like a disk forensics utility like the imagerec suite could have restored alot of the 'lost' data. In fact i've done it on source code after learning that the default behavior of untar was to overwrite all of your current directory structure. since it was text i didnt need anything fancy like imagerec, instead i just piped the raw disk to grep, and looked for parts i knew were in files i needed, then output them and the surrounding data to an external hard drive.
John Cleese's talk on Creativity recently made it to the front page of HN again https://www.youtube.com/watch?v=9EMj_CFPHYc and if you haven't watched it I highly recommend it.
I believe it was in this talk that he says the best work he ever did was when he scrapped and started over. Which from practice I think we can all admit that while its the hardest to do, it is always for the best.
Not necessarily. People often underestimate (in engineering fields) how much work it will take to rebuild something. In software there is a high degree of creativity which can have large downstream effects. You need to architect your system in such a way to make it possible to replace components when needed, this is where strong separation of concerns is important.
One thing that I've seen happen time and again is an organization bifurcating itself, so that there is one team working on the new cool replacement, and the other working on the old dead thing that everyone hates. Needless to say this creates anymosity and serverely limits an organizations ability to respond to customer demands.
> And then, some months later, Pixar rewrote the film from almost the ground up, and we made ToyStory2 again. That rewritten film was the one you saw in theatres and that you can watch now on BluRay.
At first I was feeling how it would feel to lose all that work, so frustrating! But then even if you hadn't, it turns out management was gonna throw it all away anyway!
Funny that this comes up a few weeks after I finished Ed Catmull's "Creativity Inc." If you want a little more detail about this (and other Pixar related things and Steve Jobs) read the book. It is a really good one.
A good way to fuckup on windows/C#: I Was iterating through network folders to delete (which all start with "\\servername"), except that I had a bug and instead was iterating through the characters of the first network folder path. And that's how I discovered that in windows, "\" means root of the current active drive. And that's also why I value my automatic backup to a NAS twice a day.
This brings up a great practical question. What's the state of the art of this sort of thing for more modest but still modern data storage requirements?
Context: For the last five years, my backup system has been to have Time Machine do hourly backups on my MBP (main development machine, just shy of 1TB data), with key spots on my Linux server (3TB data at the moment) backed up daily to my in-laws' house using cron and rsync, and spot directories on the MBP backed up there as well.
But the hard drive on the Time Capsule I've used seems to have gotten unreliable, and the external USB drive I bought to replace it has not worked reliably for more than a day or two at a time. And even when it was all working properly, I was never really verifying my backups.
Do people have suggestions for secure, reliable, verifiable, easy backup systems capable of handling 4+ TB of data? I don't mind if it takes work or money to set it up; the important thing is once it's working I can mostly forget about it.
For an offsite backup, Backblaze is fantastic. Unlimited storage for $5/month and the client works perfectly. It's not highly-redundant or anything, so use it in addition to a local backup.
CrashPlan is the next-best option if you need Linux support, but the client isn't as good.
I replaced my Time Capsule with a Synology NAS. Backups are still handled by Time Machine on the Mac (so they're still mostly unobtrusive and idiot-proof), but now they're stored on a mirrored RAID of cheap 3.5" disks.
The Synology box is basically just an ARM Linux machine, SSH/root is not locked out if you want it, so if you want to get fancy with off-site backups, you can set up rsync or whatever you want on it. They even ship with some GUIs for mirroring to Dropbox, S3, rsync, another remote Synology NAS, etc.
[+] [-] rootbear|9 years ago|reply
The Pixar animation system at the time was written in K&R C and one of my tasks was to migrate it to ANSI C. As I did that I learned that there were aspects of this code that felt like a school assignment that had escaped from the lab. While searching for a bug, I noticed that the write() call that saved the animation data for a shot wasn't checked for errors. This seemed like a bad idea, since at the time the animation workstations were SGI systems with relatively small SCSI disks that could fill up easily. When this happened, the animation system usually would crash and work would be lost. So, I added an error check, and also code that would save the animation data to an NFS volume if the write to the local disk failed. Finally, it printed a message assuring the animator that her files were safe and it emailed a support address so they could come help. The animators loved it! I had left Pixar by the time the big crunch came in 1999 to remake TS2 in just 9 months, so I didn't see that madness first hand. But I'd like to think that TS2 is just a little, tiny bit prettier thanks to my emergency backup code that kept the animators and lighting TDs from having to redo shots they barely had time to do right the first time.
The point is that one would like to think that a place like Pixar is a model of Software Engineering Excellence, but the truth is more complex. Under the pressures of Production deadlines, sometime you just have to get it to work and hope you can clean it up later. I see the same things at NASA, where, for the most part, only Flight Software gets the full on Software Engineering treatment.
[+] [-] iraklism|9 years ago|reply
We do penetration tests for a wide range of clients across many industries. I would say that the bigger the company, the more childish flaws we find. For sure the complexity, scale, and multiple systems do not help towards having a good security posture , but never assume that because you are auditing a SWIFT backend you will not find anything that can lead to direct compromise.
Maybe not surprisingly, most startups that we work with have a better security posture than F500 companies. They tend to use the latest frameworks that do a good job of protecting against the standard issues, and their relatively small attack landscape doesn't leave you with much to play.
Of course there are exceptions.
[+] [-] bane|9 years ago|reply
It makes one start to contemplate how little we really understand about software and how nascent the field really is. We're basically stacking rocks in a modern age where other engineering disciplines are building half-km tall buildings and mile-spanning bridges.
Fast forward 2500 years and the software building techniques of the future must be as unrecognizable to us as rocket ships are to people who build mud huts.
[+] [-] hackuser|9 years ago|reply
I disagree. I mean, I agree those things happen, but the system administrator's job is to anticipate those Real World risks and manage them with tools like quality assurance, redundancy, plain old focus and effort, and many others.
The fundamental of backups is to test restoring them, which would have caught the problem described. It's so basic that it's a well-known rookie error and a source of jokes like, 'My backup was perfect; it was the restore that failed.' What is a backup that can't be restored?
Also, in managing those Real World risks, the system administrator has to prioritize the value of data. The company's internal newsletter gets one level of care, HR and payroll another. The company's most valuable asset and work product, worth hundreds of millions of dollars? A personal mission, no mistakes are permitted; check and recheck, hire someone from the outside, create redundant systems, etc. It's also a failure of the CIO, who should have been absolutely sure of the data's safety even if he/she had to personally test the restore, and the CEO too.
[+] [-] satysin|9 years ago|reply
[+] [-] brazzledazzle|9 years ago|reply
"Backups aren't backups until they've been tested."
They really are Schrödinger's backups until a test restore takes place. This is one area where people cut corners a lot because no one cares about backups until they need them. But it's worth the effort to do them right, including occasional, scheduled manual testing. If you can't restore the data/system you're going to be the one working insane hours to get things working when a failure occurs.
And then there's the aftermath. Unless you are lucky enough to work for a blame free organization major data loss in a critical app due to a failure of the backup system (or lack thereof) could be a resume generating event. If you're ordered to prioritize other things over backups make sure you get that in writing. Backups are something everyone agrees is "critical" but no one wants to invest time in.
[+] [-] tekklloneer|9 years ago|reply
from a brief stint in the gfx industry, you are correct.
Pixar isn't a model of software excellence, it's a model of process and (ugh) culture excellence.
[+] [-] sachinag|9 years ago|reply
[+] [-] brazzledazzle|9 years ago|reply
There are other anecdotes online about this catastrophic data loss and backup failure but I think it was, funny enough, the propensity for some end users of Perforce to mirror the entire repository that saved their bacon. I say funny because this is something a Perforce administrator will generally bark at you about since your sync of this enormous monolithic repo would be accompanied by an associated server side process that would run as long as your sync took to finish and thanks to some weird quirks of the Perforce SCM long running processes were bad and would/(could?) fuck up everyone else's day. In fact I think a recommendation from Pixar was to automatically kill long running processes server side and encourage smaller workspaces. Anyway, I digress. They were able to piece it together using local copies from certain workstations that contained all or most of the repo. Bad practices ended up saving the day.
[+] [-] santaclaus|9 years ago|reply
Was that menv? I've heard stories that Pixar builds these crazy custom apps that rival things like 3D Studio and Maya but that never leave their campus!
[+] [-] MichaelMoser123|9 years ago|reply
[+] [-] woliveirajr|9 years ago|reply
Commitment would be very different if people were being asked to help while some heads were rolling. Because you're a real team when everybody is going in the same direction. Any call on "people, work hard do recover while we're after the moron who deleted everything" wouldn't have done it.
You just commit to something when you know that you won't be under the fire if you do something wrong without knowing it.
[+] [-] lokedhs|9 years ago|reply
I think that employees actually makes less mistakes and are more productive if they don't have be worried about being fired for making a mistake.
[+] [-] FabHK|9 years ago|reply
Furthermore, a lot of individual error's are seen in an institutionalised "systems" framework - given that people invariably will make mistakes, how can we set up the environment/institutions/systems so that errors are not catastrophic.
Not sure how that applies to movie animation, to be honest, but not primarily looking for whom to blame was certainly a very good move.
[+] [-] mr_luc|9 years ago|reply
[+] [-] weavie|9 years ago|reply
[+] [-] cookiecaper|9 years ago|reply
Since Catmull has an engineering background (his PhD involved the invention of the Z-buffer, and he was doing computer graphics before anyone knew anything about it), he understands that mistakes and failed projects, when combined with an forthright and collaborative feedback loop, are not problematic detours, but rather necessary mile markers on the path to real innovation. We'd be so much further ahead if we put more men like Catmull in charge of things.
The biggest problem with reading Creativity Inc. is that it will rekindle the hope that there may be a sane workplace out there somewhere, when practically speaking, we know that few of us will ever find employment in one. It gave me a number of disquieting feelings as I read that the attributes of a workplace that all good engineers crave actually can and sometimes do exist out there. I had convinced myself that these things were myths, so now I'm sad that my boss isn't Ed Catmull.
That said, I do believe some evaluation and/or discipline would've been appropriate in this case, not for the person who accidentally executed a command in the wrong directory, but for the people who were supposed to be maintaining backups and data integrity.
Assuming that your primary job duties involve data integrity and system uptime, having non-functional backups of truly critical data stretches beyond the scope of "mistakes" and into the scope of incompetence.
It is, I'm sure, very possible that no one was really assigned this task at Pixar and that it would therefore by improper to punish anyone in particular for the failure to execute it, but I do believe there is a limit between mistakes en route to innovation and negligence. My experience has been that most companies strongly take one tack or the other: they either let heads roll for minor infractions (and thus never allow good people to get established and comfortable), or they never fire anyone and let the dead weight and political fiefdoms gum up the works until the gears stop altogether. There needs to be a balance, and that's a very hard thing to find out there.
[+] [-] tbabb|9 years ago|reply
There was an incident where I work where an employee (a new hire) set up a cron job to delete his local code tree, re-sync a new copy, then re-build it using a cron job every night. A completely reasonable thing for a coder to automate.
In his crontab he put:
and as everyone undoubtedly first discovered by accident, the crontab environment is stripped bare of all your ordinary shell environment. So $MY_TREE_ROOT expanded to "".The crontab ran on Friday, IIRC, and got most of the way through deleting the entire project over the weekend before a lead came in and noticed things were disappearing. Work was more or less halted for several days while the supes worked to restore everything from backup.
Could the blunder have been prevented? Yes, probably with a higher degree of caution, but that level of subtlety in a coding mistake is made regularly by most people (especially someone right out of university); he was just unlucky that the consequences were catastrophic, and that he tripped over the weird way crontab works in the worst possible usage case. He probably even tested it in his shell. We all know to quadruple-check our rm-rfs, but we know that because we've all made (smaller) mistakes like his. It could have been anyone.
Dragging him to the guillotine would have solved nothing. In fact, the real question is "how is it possible for such an easy mistake to hose the entire project?" Some relatively small permissions changes at the directory root of the master project probably would have stopped stray `rm -rf`s from escaping local machines without severely sacrificing the frictionless environment that comes from an open-permissions shop. So if anything, the failure was systems's fault for setting up a filesystem that can be hosed so easily and so completely by an honest mistake.
The correct thing to do was (and is) to skip the witch hunt, and focus on problem-solving. I am not sure, but I think the employee was eventually hired on at the end of his internship.
For me the principle is: Standards and habits are teachable. Competence and attitude, less so. Educate and train for the former, and a failure of the former should cause you to look first at your procedures, not the people. Hire and fire for the latter.
[+] [-] gist|9 years ago|reply
The other side is if you play a key role (and head could roll after the hard work is done) to simply leverage that fact (perhaps with others) as an advantage such that you have get a new contract and can't be fired for X amount of time.
[+] [-] smcl|9 years ago|reply
So that effort to recreate it (not to mention produce it in the first place) was pretty much all for naught? That must have been soul destroying
[+] [-] stpe|9 years ago|reply
"We didn't scrap the models, but yes, we scrapped almost all the animation and almost all the layout and all the lighting. And it was worth it.
Changing the script saved the film, which in turn allowed Buzz and Woody to carry on for future generation (see ToyStory3 for how awesome that universe continues to be - well done to everyone who worked on the lastest installment!) and, in some ways, set a major cornerstone in the culture of Pixar. You may have heard John or Steve or Ed mention "Quality is a good business model" over the years. Well, that moment in Pixar's history was when we tested that, and it was hard, but thankfully I think we passed the test. Toy Story 2 went on to became one of the most successful films we ever made by almost any measure.
So, suffice it to say that yes, the 2nd version (which you saw in theatres and is now on the BluRay) is about a bagillion times better than the film we were working on. The story talent at the studio came together in a pretty incredible way and reworked everything. When they came back from the winter holidays in January '99, their pitch, and Steve Jobs's rallying cry that we could in fact get it done by the deadline later that year, are a few of the most vivid and moving memories I have of my 20 years at the studio."
https://www.quora.com/Did-Pixar-accidentally-delete-Toy-Stor...
[+] [-] dijit|9 years ago|reply
"We have to keep this scene even though it's not quite perfect because otherwise it's a waste of money".
Maybe this is a bad example actually, movie industry is something you launch and market and leave.
But the best architectures I've seen have been demolished, destroyed and rebuilt from the ground up for their purpose.
Same with code.
[+] [-] xja|9 years ago|reply
So often something happens which seems like a total disaster, the end of the world, and you struggle desperately to fix it.
In hindsight it turns out it didn't matter as much as you thought it did anyway. Has happened in so many startups I've worked at.
[+] [-] dahart|9 years ago|reply
[+] [-] mabbo|9 years ago|reply
I bet you they were suddenly industry experts in source control and data backups.
[+] [-] Donzo|9 years ago|reply
[+] [-] huherto|9 years ago|reply
Reminds me of Fred Brooks quote. "Plan to throw one away. You will anyhow".
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] KaiserPro|9 years ago|reply
I worked at a few VFX studios, and everyone has deleted large swathes of shit by accident.
My favourite was when an rsync script went rogue and started deleted the entire /job directory in reverse alphabet order. Mama-mia[1] was totally wiped out, as was Wanted (that was missing some backups, so some poor fucker had to go round fishing assets out of /tmp, from around 2000 machines.)
From what I recall (this was ~2008) There was some confusion as to what was doing the deleting. Because we had at the time a large(100 or 300tb[2]) lustre file system, it didn't really give you that many clues. They had to wait till it went on a plain old NFS box before they could figure out what was causing it.
Another time honoured classic is matte painters on OSX boxes accidentally dragging whole film directories into random places.
[1]some films have code names, hence why this was first
[2]That lustre was big, physically and IO, it could sustain something like 2-5 gigabytes a second, It had at least 12 racks of disks. Now a 4u disk shelf and one server can do ~2gigabyes sustained
[+] [-] mprovost|9 years ago|reply
There was another incident where there was a script that ran out of cron to cleanup old files in /tmp, and someone NFS mounted a production volume into /tmp...
Eventually we put tarpit directories at the top of each filesystem (a directory with 1000 subdirectories each with 1000 subdirectories, several layers deep) to try and catch deletes like the one you saw, then we would alert if any directories in there were deleted so we could frantically try and figure out which workstation was doing the damage.
[+] [-] KaiserPro|9 years ago|reply
Second, there were very large nearlines that took hourly snapshots. Finally, lots and lots of tape for archive.
[+] [-] dvdhnt|9 years ago|reply
> The command that had been run was most likely ‘rm -r -f *’, which—roughly speaking—commands the system to begin removing every file below the current directory. This is commonly used to clear out a subset of unwanted files. Unfortunately, someone on the system had run the command at the root level of the Toy Story 2 project and the system was recursively tracking down through the file structure and deleting its way out like a worm eating its way out from the core of an apple.
[+] [-] knodi123|9 years ago|reply
That's my theory as to what they did.
[+] [-] leecarraher|9 years ago|reply
[+] [-] hybridtupel|9 years ago|reply
[+] [-] mentos|9 years ago|reply
I believe it was in this talk that he says the best work he ever did was when he scrapped and started over. Which from practice I think we can all admit that while its the hardest to do, it is always for the best.
[+] [-] bluejekyll|9 years ago|reply
Not necessarily. People often underestimate (in engineering fields) how much work it will take to rebuild something. In software there is a high degree of creativity which can have large downstream effects. You need to architect your system in such a way to make it possible to replace components when needed, this is where strong separation of concerns is important.
One thing that I've seen happen time and again is an organization bifurcating itself, so that there is one team working on the new cool replacement, and the other working on the old dead thing that everyone hates. Needless to say this creates anymosity and serverely limits an organizations ability to respond to customer demands.
Starting over should be taken very seriously.
[+] [-] abe_duarte|9 years ago|reply
[+] [-] creaghpatr|9 years ago|reply
[+] [-] jrochkind1|9 years ago|reply
> And then, some months later, Pixar rewrote the film from almost the ground up, and we made ToyStory2 again. That rewritten film was the one you saw in theatres and that you can watch now on BluRay.
At first I was feeling how it would feel to lose all that work, so frustrating! But then even if you hadn't, it turns out management was gonna throw it all away anyway!
[+] [-] yawz|9 years ago|reply
[+] [-] cm2187|9 years ago|reply
[+] [-] colomon|9 years ago|reply
Context: For the last five years, my backup system has been to have Time Machine do hourly backups on my MBP (main development machine, just shy of 1TB data), with key spots on my Linux server (3TB data at the moment) backed up daily to my in-laws' house using cron and rsync, and spot directories on the MBP backed up there as well.
But the hard drive on the Time Capsule I've used seems to have gotten unreliable, and the external USB drive I bought to replace it has not worked reliably for more than a day or two at a time. And even when it was all working properly, I was never really verifying my backups.
Do people have suggestions for secure, reliable, verifiable, easy backup systems capable of handling 4+ TB of data? I don't mind if it takes work or money to set it up; the important thing is once it's working I can mostly forget about it.
[+] [-] rossng|9 years ago|reply
CrashPlan is the next-best option if you need Linux support, but the client isn't as good.
[+] [-] kalleboo|9 years ago|reply
The Synology box is basically just an ARM Linux machine, SSH/root is not locked out if you want it, so if you want to get fancy with off-site backups, you can set up rsync or whatever you want on it. They even ship with some GUIs for mirroring to Dropbox, S3, rsync, another remote Synology NAS, etc.
[+] [-] eva1984|9 years ago|reply
[+] [-] franze|9 years ago|reply
[+] [-] ihaveahadron|9 years ago|reply
[deleted]
[+] [-] 1denis|9 years ago|reply
[+] [-] unknown|9 years ago|reply
[deleted]