The way I read it, you didn't destroy anything. You had a moment of panic, and could have created real problems.
Regardless, same lessons learned right? :) Whether the server is gone or not, it could have been, and the way we work changes drastically after a real world scare like this.
Did you ever tell the boss, or did you just let it ride? There's nothing like going to the boss in a cold sweat and the old "I f*ed up" look on your face.
I just let it ride at the time. When I joined the company there wasn't any other tech guy around except me (our tech guy was on a vacation). So I was left alone for a while, and had to sort out of lot of craziness :)
He put himself in a position where only chance protected him from a disaster. (Imagine if the update did trash the running server.) That's reason enough for a good sysadmin to be worried, IMO.
Never log into your production servers via shell? This seems a tad over the top.. especially if you are planning on writing a script to perform operations and need to discover dependencies / etc.
Well, the idea is that instead of logging into production, you log into your development environment, do the testing there--then you write a script that replicates what you just did. Lastly, you run the script on staging, and if that seems to work, then run it on production.
There's a pretty handy tool I use now (fabric: docs.fabfile.org) that makes it pretty simple.
"If you join a project with no version control, don't convince yourself that everything will be ok. Refuse to do any work until you version control it." Embargo on.
cd /tmp. Unzip some file with some system image - usr, etc, lib, the usual - that should be unpacked in /. Do what needs to be done. rm -fr /usr.
WAIT, NO! ^C.
"Backup, could you be so kind as to restore the ABC server /usr tree? I think I made a huge mistake. Thank you."
Good times. Learned a lot about why I need a VM with an image of a given system.
[+] [-] dpcan|15 years ago|reply
Regardless, same lessons learned right? :) Whether the server is gone or not, it could have been, and the way we work changes drastically after a real world scare like this.
Did you ever tell the boss, or did you just let it ride? There's nothing like going to the boss in a cold sweat and the old "I f*ed up" look on your face.
[+] [-] b14ck|15 years ago|reply
[+] [-] mkelly|15 years ago|reply
[+] [-] bschlinker|15 years ago|reply
[+] [-] b14ck|15 years ago|reply
There's a pretty handy tool I use now (fabric: docs.fabfile.org) that makes it pretty simple.
[+] [-] armored|15 years ago|reply
[+] [-] trbecker|15 years ago|reply
Good times. Learned a lot about why I need a VM with an image of a given system.