What I find curious in this article is that they gloss over that this is also a people management problem. Sure you can put restrictive policies in place but apparently things went down at Citibank that caused so much ire for the person in question to take action like that.
I like to think that even though people see sysadmins are grumpy rage-machines in a basement most are super-decent people caring deeply for the systems under their control. Perhaps I've got a too rosy view of my fellow engineers.
I'd be very curious to hear what went on there on the people side of things. What led to this person feeling so angry that they would take an action like this. I'm also curious how it could even get so far. I have the benefit of being able to regularly talk to my manager so even if we did performance reviews in that way they wouldn't come as a surprise.
> I'd be very curious to hear what went on there on the people side of things.
It's important to study that in these cases. I was convicted in a similar case and am actually a chapter in a CERT book, but they never reached out to me for my input, so they're completely oblivious to my motives and missing key facts about the case. I'd write more about it or speak at conferences but I've been able to bury that past and move on. Maybe some day.
I think he had a similar case to mine where he felt he was stuck in a situation with his manager(s) and felt like he had no recourse. HR should be involved in employee reviews and should provide a way for the employee to give feedback on his own review.
Another aspect of it is education about the law. I see case after case where the defendant had no idea he could face federal charges at all, much less one that can result in such stiff penalties. (The penalties he was facing were much, much worse if he had been tried and convicted rather than taking this plea deal.) That could easily be part of any degree program and/or employee orientation.
People have different reactions to bad news, especially life altering news like "you're about to be fired." These reactions stem from personality differences that form early in development and throughout life.
Someone could be a great team member, love their peers, and still react badly to the news they might be fired soon.
I don't think any amount of "people management" can fix someone who is already a bad egg, "configured" to react poorly to bad news. Some people will be out for revenge no matter what.
The more important thing to analyze here might be the technical measures in place to respond to configuration files being wiped out. Sure, an "insider" could maliciously wipe out the files... but couldn't a technical glitch just as easily do the same? The important lesson from this should be including adequate detection measures to immediately know when configuration files have been changed outside of the normal build process, and the ability to rollback when changes are detected.
One person should not be able to do any more damage than a technical glitch.
If you follow the link to the justice department website, you'll see that the guy saw himself as a martyr.
"They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team. Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
Used to be there was "the guy" that knew the "secret formula." Like the guy that knew the Coke secret formula, or the guy that used to maintain Lay's potato chip fryers. Most everyone else was just a cog in a big machine with very little power individually.
What this guy - and Snowden - show is that today's IT professionals have tremendous power that generally they never choose to exercise.
If an IT shop in a major corporation decided to unionize and turn on its management, they would have the company totally by the balls. So many people in an IT shop are positioned to control so much, and management often treats its IT staff like disposable widgets.
It would take two+ hands to count the number of major multinationals that I could have seriously damaged singlehandedly in a matter of hours if I had ever gone rogue back in the day. Few non-IT rank-and-file employees have such power.
Of course the fact that I never did such a thing is one of the reasons I remain a well-paid consultant (trustworthiness and honesty is a consultant's #1 asset), but the fact is that many corporations are quite blind to the level of risk they undertake when they mistreat their IT staff.
Eh, pretty much any employee can cause an almost arbitrarily large amount of damage if they choose to be actively malicious. Drop a box of rat poison into the cookie dough mixer on the day you leave and watch a hundred million dollar recall roll into place.
The reason why this doesn't happen is because it's an exceedingly dumb crime and most people are not sociopaths. Plenty of people like to fantasize about scenarios like this but it basically takes a terrorist mindset to actually go through with it.
I worked in Citi some years before, during, and after the 2008 crisis. Held a two positions, IT and a business aligned roles. I left IT because the treatment of staff was atrocious, and I could see the cost cutting hammer dropping on IT.
In regards to the integrity of their employee evaluation process, I have an anecdote to share. We used a 360 evaluation process. My review set was my direct manager, plus 5 other people with whom I directly worked. That year, I received a 5 out of 5, the highest rating possible.
During the official manager / employee review sign off meeting, my manager noticed that my 5 was no longer a 5, but a 3. He canceled the meeting to investigate what happened. Apparently, his manager with collusion with a senior HR manager, went into the employee review system and manually overwrote the ratings provided by my manager and colleagues. The reasoning provided was that there were too many high ratings within my employee pool and I was selected by "someone" to be downgraded.
I can see going into a Citi employee review meeting, coming out, and wanting to nuke the entire site from orbit.
That's surprisingly common. Where I am, every department, no matter how big or small, has to be within x points of the average rating on a bell curve. If not, your whole dept gets scaled to it.
Where my friend works, it's more like yours and a select few get the stick. But they have this feature where you get reviewed formally if you have 3 consecutive "bad ratings", so mysteriously, no one ever gets the stick if they got it last cycle.
Just, for both of us, it's company-wide knowledge.
I don't want to know about what he did unless there's some information on why he did it also. Was he pushed over the edge after years of mistreatment? Or was he mentally unstable and it was petty revenge?
Most of us have the keys to the kingdom in one way or another and while I (and others) could never do this it doesn't mean I don't have sympathy for someone who does - for some justifiable reason. I can't imagine any that are justifiable... but I can ask.
I found the other linked articles about, "The Malicious Insider", a bit fucked in the head. Unapproved hardware! Email misuse! Unapproved workaround! Unapproved software! Are you fucking kidding me? That's a pretty far cry from this guy.
From the SMS quoted in TFA, seems like he at least thought he and his colleagues were being continually mistreated by upper management:
“They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team.”
“Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
This happened when I was at Google, but it was an accident.
On someone's last day they ran rm -rf / to wipe their desktop machine. I can't remember whether they typed it into the wrong terminal window or had mounted an external filesystem somewhere deep under their home directory and forgot about it, it was something like that. They wound up wiping some unfathomably large amount of data (I think most of a data center) before noticing what they'd done.
There were backups, but it was still a huge hassle.
I managed to do that one time but on a much smaller scale (home server). Forgot about my network mounts, rm -Rf, noticed that it was taking longer than expected to delete just a few dirs... "oh crap." :) Luckily I run zfs with zfs-auto-snap on my home server so everything was back the way it was after a quick 'zfs restore'.
I once left a "production" connection string in my local config while fixing a bug in my last few days on a job, this got checked into source control (yeah, configuration in source control is bad)... a couple weeks after I left, someone had used that connection string to connect to the database server, and truncated a table.... needless to say, there were some things that changed after that.
One of my clients, a well known fixture in American finance, has been hacked by employees or contractors several times over the past 10 years. These cases have been public and supposedly cost millions (it's so hard to know how accurate the assessed damages in these cases are).
Because they now fear employees and contractors more than external threats it takes 30 tickets to different groups to set up a server. You can't chown a file in a directory you own without a ticket. You can use one of two old text editors. No new IDE's or modern text editors. You can't upgrade language runtimes without corp. approval. It's nuts. Ironically it leads to more attack vectors and it's ground their software dev to a halt.
I worked for a large company that went through drastic downsizing at one point. Some ppl who were getting let go were "well positioned" to do some damage. The way the company handled this is there was no advance notice. Your manager and a couple of security dudes shows up at your desk with boxes and you're outta there in 5 min.
There were things like managers getting punched, ppl screaming etc but no malicious hacking that I'm aware of.
Somehow one dude (I'm assuming :)) managed to take a dump in the elevator on the way out :)
At financial institutions I've worked at, if you're getting fired you are taken to a room (managers office, conf room etc.) and immediately escorted out of the building after. No returning to your desk at all. Someone else will gather the belongings that you need and bring them like wallets/handbags. The rest of your stuff will be mailed/couriered to you later after someone has had the chance to go through them to make sure that there is no IP in them.
Sorry for the naive question, but in that kind of situation, how do you assure the continuity of the organization ? What if the people being let got were supporting customers, about to close a deal, and so on ? How do you make sure people are not leaving with a critical "know-how" ?
> “They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team.”
> “Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
Ah yes, the old double 'they are harming me by holding me accountable' and 'I am helping them by holding them accountable' defense.
What I'd like to know is how mindsets like this are formed in life, and how to prevent them. One's employer doesn't owe one a job; one owes one's employer fair labour in return for one's wages. The world does not consist of shadowy forces plotting against one. Don't minimise one's own agency: be an active force for good, not a passive subject of whatever happens.
Labor laws disagree to a certain degree -- at least in that you can't fire someone for any reason you choose (see the EEOC poster in your breakroom/mailroom/kitchen for examples). Also employment contracts can stipulate various things that prevent a company from firing you, particularly in union jobs.
I've found going by the motto "don't be a dick" tends to be a decent deterrent against doing something stupid like maliciously wiping company data and getting sued and/or going to jail for it. If it's your employer doing it to you, there is plenty enough opportunity in IT to find work at a place that doesn't have a hostile work environment.
I started on a help desk, from my experience the management knew which ones were smart enough to do real damage and triggered enough to actually do it.
If I had any advice it would simply be to avoid letting a corporate environment allow you to forget you have real people as employees with feelings. "It's just business" doesn't translate to everyone.
Yeah or don't be shitty to your employees, or don't hire managers that are so bad at giving negative performance reviews that the employee goes out for revenge. This is definitely a people management problem.
The only way to rid yourself of the threat posed by other humans is to run a single-person company, and even then you might screw yourself over.
You will never be able to completely neuter a threat that does any useful service for your company, and it's foolish to think you can. Any useful amount of delegated power or responsibility without the ability to misuse it is very rare, if not impossible.
You cannot control the human factor so easily but you can definitely keep you employees happy! I don't get why the article does not mention the quality of working environment. IT systems can be extremely stressful if there is not help from management, and most of us know what bad management means...
I wonder if the fact that this was a bank had something to do with the short sentence. Back when I was a pr activist, it was known that the level of public anger - the public's trigger - on an issue would sway the decision of high-profile cases - that is, the public's perception would factor into the judge's decisionmaking. If crimes were committed inside certain "pathways", those crimes were easier to get away with under certain constraints. This man has sabotaged a bank using a computer and made pro-union statements. If he had made broadly anarchist statements I wonder how this sentence would have changed.
I argue that these dynamics are relevant from a criminal psychology and a government-hacking standpoint. Being mostly derived from firsthand resources, Toffler, Bernays, et. al, along with some authors whose mention is straight-up dangerous, I am constantly seeking additional resources.
This is reminiscent of the San Francisco network administrator who took it upon himself to "protect" the city from his supervisors and would only divulge his password to mayor Newsom.[1]
IIRC it was also precipitated by perf reviews or discipline.
Hope AWS has better systems in place to make something like this nearly impossible to carry out.
This is a total nightmare scenario for a lot of organizations, and is quiet doable for a lot of systems. Specifically on the router side (thanks Cisco) I haven't seen a lot of controls that would stop this. Network Engineers need full access to get things done in a timely fashion, and the limitations of control in TACACS. Recover from insider attacks of this nature are all based on how much to drill to recover from this style of attack.
What precautions can one take against their sysadmin? I can't think of many places that have stricter security policies than 3-letter agencies, still Snowden got away with a USB stick (man, can't wait to see the movie - I wonder if he really passed the USB stick out using the rubik's cube as in the movie trailer).
The thing is that that you can't do much against someone who has root access for a living.
There are other ways to make a point, especially when you're prepared to get fired. Damaging everyone else is selfish and stupid.
Sounds like a management issue more than security. However, post-Snowden, I did some exploration of a strategy for countering sysadmin risk in organizations whose IT activities were pretty regular.
It's a little unsettling to think about how the biggest threats to an organization can be from within. The inside hacker doesn't even have to be disgruntled: they could just download the wrong link or put in a flash stick from the parking lot and BAM! More hacking in a minute than any Hollywood effort could get you.
The worst part about it is that to be completely on the ball with internal threats, you need to be constantly paranoid, assuming by default that any employee with enough access is a threat, or a threat waiting to happen. Such things do not add to team morale or cohesion, but in this case, a pound of prevention is still better than a metric ton of cure.
I would think the first and easiest thing to do to prevent situations like this is to make it so a single person cannot run certain commands. Sort of like the digital version of two keys on the opposite side of the room to launch a missile.
[+] [-] daenney|9 years ago|reply
I like to think that even though people see sysadmins are grumpy rage-machines in a basement most are super-decent people caring deeply for the systems under their control. Perhaps I've got a too rosy view of my fellow engineers.
I'd be very curious to hear what went on there on the people side of things. What led to this person feeling so angry that they would take an action like this. I'm also curious how it could even get so far. I have the benefit of being able to regularly talk to my manager so even if we did performance reviews in that way they wouldn't come as a surprise.
[+] [-] alt2319|9 years ago|reply
It's important to study that in these cases. I was convicted in a similar case and am actually a chapter in a CERT book, but they never reached out to me for my input, so they're completely oblivious to my motives and missing key facts about the case. I'd write more about it or speak at conferences but I've been able to bury that past and move on. Maybe some day.
I think he had a similar case to mine where he felt he was stuck in a situation with his manager(s) and felt like he had no recourse. HR should be involved in employee reviews and should provide a way for the employee to give feedback on his own review.
Another aspect of it is education about the law. I see case after case where the defendant had no idea he could face federal charges at all, much less one that can result in such stiff penalties. (The penalties he was facing were much, much worse if he had been tried and convicted rather than taking this plea deal.) That could easily be part of any degree program and/or employee orientation.
[+] [-] chatmasta|9 years ago|reply
Someone could be a great team member, love their peers, and still react badly to the news they might be fired soon.
I don't think any amount of "people management" can fix someone who is already a bad egg, "configured" to react poorly to bad news. Some people will be out for revenge no matter what.
The more important thing to analyze here might be the technical measures in place to respond to configuration files being wiped out. Sure, an "insider" could maliciously wipe out the files... but couldn't a technical glitch just as easily do the same? The important lesson from this should be including adequate detection measures to immediately know when configuration files have been changed outside of the normal build process, and the ability to rollback when changes are detected.
One person should not be able to do any more damage than a technical glitch.
[+] [-] pdpi|9 years ago|reply
"They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team. Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] reitoei|9 years ago|reply
I'd be interested to hear it too, but in the grand scheme of things, it doesn't matter one iota.
[+] [-] riprowan|9 years ago|reply
What this guy - and Snowden - show is that today's IT professionals have tremendous power that generally they never choose to exercise.
If an IT shop in a major corporation decided to unionize and turn on its management, they would have the company totally by the balls. So many people in an IT shop are positioned to control so much, and management often treats its IT staff like disposable widgets.
It would take two+ hands to count the number of major multinationals that I could have seriously damaged singlehandedly in a matter of hours if I had ever gone rogue back in the day. Few non-IT rank-and-file employees have such power.
Of course the fact that I never did such a thing is one of the reasons I remain a well-paid consultant (trustworthiness and honesty is a consultant's #1 asset), but the fact is that many corporations are quite blind to the level of risk they undertake when they mistreat their IT staff.
[+] [-] Retric|9 years ago|reply
[+] [-] shalmanese|9 years ago|reply
The reason why this doesn't happen is because it's an exceedingly dumb crime and most people are not sociopaths. Plenty of people like to fantasize about scenarios like this but it basically takes a terrorist mindset to actually go through with it.
[+] [-] kovrik|9 years ago|reply
1) Nowadays almost everything depends on IT.
2) Majority of people still perceive IT as a "black magic". Arcane knowledge, something they don't understand, but depend on.
[+] [-] rabboRubble|9 years ago|reply
In regards to the integrity of their employee evaluation process, I have an anecdote to share. We used a 360 evaluation process. My review set was my direct manager, plus 5 other people with whom I directly worked. That year, I received a 5 out of 5, the highest rating possible.
During the official manager / employee review sign off meeting, my manager noticed that my 5 was no longer a 5, but a 3. He canceled the meeting to investigate what happened. Apparently, his manager with collusion with a senior HR manager, went into the employee review system and manually overwrote the ratings provided by my manager and colleagues. The reasoning provided was that there were too many high ratings within my employee pool and I was selected by "someone" to be downgraded.
I can see going into a Citi employee review meeting, coming out, and wanting to nuke the entire site from orbit.
[+] [-] uremog|9 years ago|reply
Where my friend works, it's more like yours and a select few get the stick. But they have this feature where you get reviewed formally if you have 3 consecutive "bad ratings", so mysteriously, no one ever gets the stick if they got it last cycle.
Just, for both of us, it's company-wide knowledge.
[+] [-] pc86|9 years ago|reply
[+] [-] overgryphon|9 years ago|reply
[+] [-] sqldba|9 years ago|reply
Most of us have the keys to the kingdom in one way or another and while I (and others) could never do this it doesn't mean I don't have sympathy for someone who does - for some justifiable reason. I can't imagine any that are justifiable... but I can ask.
I found the other linked articles about, "The Malicious Insider", a bit fucked in the head. Unapproved hardware! Email misuse! Unapproved workaround! Unapproved software! Are you fucking kidding me? That's a pretty far cry from this guy.
[+] [-] semi-extrinsic|9 years ago|reply
“They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team.”
“Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
[+] [-] throwaway11teen|9 years ago|reply
On someone's last day they ran rm -rf / to wipe their desktop machine. I can't remember whether they typed it into the wrong terminal window or had mounted an external filesystem somewhere deep under their home directory and forgot about it, it was something like that. They wound up wiping some unfathomably large amount of data (I think most of a data center) before noticing what they'd done.
There were backups, but it was still a huge hassle.
[+] [-] vmp|9 years ago|reply
[+] [-] dba7dba|9 years ago|reply
Always a fear of mine. I always try to pause for a second before I run a command like that.
[+] [-] tracker1|9 years ago|reply
[+] [-] ryanmarsh|9 years ago|reply
Because they now fear employees and contractors more than external threats it takes 30 tickets to different groups to set up a server. You can't chown a file in a directory you own without a ticket. You can use one of two old text editors. No new IDE's or modern text editors. You can't upgrade language runtimes without corp. approval. It's nuts. Ironically it leads to more attack vectors and it's ground their software dev to a halt.
[+] [-] ElijahLynn|9 years ago|reply
[+] [-] st3v3r|9 years ago|reply
[+] [-] arcanus|9 years ago|reply
[+] [-] ChemicalWarfare|9 years ago|reply
There were things like managers getting punched, ppl screaming etc but no malicious hacking that I'm aware of. Somehow one dude (I'm assuming :)) managed to take a dump in the elevator on the way out :)
They had EMS parked outside too. Wasn't pretty.
[+] [-] celticninja|9 years ago|reply
[+] [-] h1srf|9 years ago|reply
[+] [-] seren|9 years ago|reply
[+] [-] pc86|9 years ago|reply
[+] [-] zeveb|9 years ago|reply
> “Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
Ah yes, the old double 'they are harming me by holding me accountable' and 'I am helping them by holding them accountable' defense.
What I'd like to know is how mindsets like this are formed in life, and how to prevent them. One's employer doesn't owe one a job; one owes one's employer fair labour in return for one's wages. The world does not consist of shadowy forces plotting against one. Don't minimise one's own agency: be an active force for good, not a passive subject of whatever happens.
[+] [-] oxryly1|9 years ago|reply
Labor laws disagree to a certain degree -- at least in that you can't fire someone for any reason you choose (see the EEOC poster in your breakroom/mailroom/kitchen for examples). Also employment contracts can stipulate various things that prevent a company from firing you, particularly in union jobs.
[+] [-] st3v3r|9 years ago|reply
That kind of thinking is exactly what breeds this kind of stuff, though.
[+] [-] tracker1|9 years ago|reply
[+] [-] autotune|9 years ago|reply
I've found going by the motto "don't be a dick" tends to be a decent deterrent against doing something stupid like maliciously wiping company data and getting sued and/or going to jail for it. If it's your employer doing it to you, there is plenty enough opportunity in IT to find work at a place that doesn't have a hostile work environment.
[+] [-] kjs3|9 years ago|reply
[deleted]
[+] [-] carlmcqueen|9 years ago|reply
If I had any advice it would simply be to avoid letting a corporate environment allow you to forget you have real people as employees with feelings. "It's just business" doesn't translate to everyone.
[+] [-] hardwaresofton|9 years ago|reply
The only way to rid yourself of the threat posed by other humans is to run a single-person company, and even then you might screw yourself over.
You will never be able to completely neuter a threat that does any useful service for your company, and it's foolish to think you can. Any useful amount of delegated power or responsibility without the ability to misuse it is very rare, if not impossible.
[+] [-] agounaris|9 years ago|reply
[+] [-] 123456Seven|9 years ago|reply
I argue that these dynamics are relevant from a criminal psychology and a government-hacking standpoint. Being mostly derived from firsthand resources, Toffler, Bernays, et. al, along with some authors whose mention is straight-up dangerous, I am constantly seeking additional resources.
Uh. Hack the planet.
Edit: or don't. U du u.
[+] [-] mc32|9 years ago|reply
IIRC it was also precipitated by perf reviews or discipline.
Hope AWS has better systems in place to make something like this nearly impossible to carry out.
[1]http://m.sfgate.com/bayarea/article/S-F-officials-locked-out...
[+] [-] 893helios|9 years ago|reply
[+] [-] nstj|9 years ago|reply
Take note - just in case, don't wear a 'hoody' as people may think you are a 'hacker'.
[+] [-] GrantSolar|9 years ago|reply
[+] [-] atmosx|9 years ago|reply
The thing is that that you can't do much against someone who has root access for a living.
There are other ways to make a point, especially when you're prepared to get fired. Damaging everyone else is selfish and stupid.
[+] [-] nickpsecurity|9 years ago|reply
https://www.schneier.com/blog/archives/2013/08/nsa_increasin...
[+] [-] rm_-rf_slash|9 years ago|reply
The worst part about it is that to be completely on the ball with internal threats, you need to be constantly paranoid, assuming by default that any employee with enough access is a threat, or a threat waiting to happen. Such things do not add to team morale or cohesion, but in this case, a pound of prevention is still better than a metric ton of cure.
[+] [-] typetypetype|9 years ago|reply