The top answer has it exactly right. This should be Root Cause, since ultimately even bugs caused by human errors are really caused by systemic flaws. Some examples:
* SQLite has 1177 times as many tests as code (not a typo)
* Live television is broadcast with a 5-10 second delay
* IMVU automatically reverts commits pushed to the site if regressions are detected
* Netflix implemented a system called the Chaos Monkey, which randomly shuts
down processes. Ensures the system can survive any failure
* VLC, Unity3D, Windows, 3D Studio Max and many more applications phone
home crashes, which allows developers to quickly patch frequent issues
* Code reviews and pair programming ensure no one person's mistake
can break critical code sections
* Similarly, multiple people should sign off on copy written. For newsletters
and press releases the whole team should, since they can't be withdrawn
* Well designed systems automatically backup, and those backups are
automatically tested, so nothing is deleted forever
Change "who" to "why" and a horrible idea turns into a brilliant one. Well designed systems can reduce the risk of almost any mistake, at the cost of speed and flexibility.
Ultimately it's up to the company to decide where the balance lies, and to live with the consequences. Startups will accept a drastically different risk profile to banks and Fortune 500 companies.
I'm usually arguing against stereotyping management around here, but I find it a stretch to believe a boss who thinks "Person to Blame" is going to be willing to pay the cost for true root cause analysis.
I have a feeling it's more like: <git annotate>; "Oh, it's Bob's fault."
While the why is important the who is important as well. In any organization of sufficient size there are going to be B grade developers, or even just young developers. It is important to understand if certain people are writing most of the major bugs. Now there might be a good reason for it such as the difficulty of certain pieces of the application, so bug density is not the only thing to look at. However knowing who is writing bugs that cause outages is very important for proper accountability.
* Well designed systems automatically backup, and those backups
are automatically tested, so nothing is deleted forever
How much software automatically backs itself up? I'm solving this problem with my startup, http://www.restbackup.com/ . I provide online backups that software makers can bundle and sell to users.
Joel Spolsky refused to add a "blame" field to FogBugz for years, he used this as an example of how you should not give the customers every feature they demand, and he said it was one of the top most requested features. He explained that once you add that field it will be the end of fixing bugs and making honest bug reports. Developers will work with QA so that bugs don't get entered into the system. Long arguments will ensue about whose fault a bug really is, and massive time will be wasted on this. One of the jobs of a software designer is to construct a tool that encourages good ways of working and not dysfunctional ones. New features need to be evaluated with this in mind.
Eventually though after he turned over project management of the software to the employees, they responded to the constant bug reports about the lack of this feature and added the ability to have this field. I recall he made a note about it on his blog, but that the project belonged to them and it wasn't his project to micromanage, so now they have the feature whether he likes it or not.
Caution: I probably have half the details wrong, I'll try to find some links.
One tradition I really appreciate at Google is the post-mortem. Any serious outage or bug will be followed by a detailed post-mortem detailing a timeline of what went wrong and why.
Sometimes this will go as far as establishing a "war room", possibly for weeks, dragging in people from a number of different areas to address a particular issue.
It's not about pointing the finger or otherwise apportioning blame. It's about learning from mistakes and preventing them from happening again.
That being said, I've worked at other places where shame is attempted to be used in such situations (which is what this is). In my experience it just fosters a hostile environment.
At a prior job I was a key member of a team that did a 2 week long, intense root cause analysis of a prod issue that threatened a multi million dollar contract. I consider it one of the most interesting and rewarding things I've done in my career. All obstacles were removed, we could follow the clues wherever they took us, use any resources etc. In the end it took careful analysis, many simulations and hard work to find and fix the root cause, and no fingers were pointed, instead we learned some good lessons and were better for it.
That sort of sounds like how Morbidity and Mortality conferences at hospitals are supposed to work. It isn't about assigning blame(unless someone really screwed up) but examining the mistake in front of the whole department so that everybody learns from it.
My company, LiveOps, does the same thing. Every customer impacting issue gets a post-mortem, and some non-customer impacting problems get one.
We have good stability (we hit four 9s recently), but we're always pushing for me. The post-mortem process is very positive and affirming. That's because it's transparent, as it should be.
Interesting. That reminded me of this book "Start-up Nation: The Story of Israel's Economic Miracle" which says their military runs the same procedure in a similar way after missions.
I don't know... maybe it's where I work, but the post-mortems I've been to all too often mandate some useless extra layer of review or process on top of an already byzantine change control scheme. Sometimes, "shit happens", and I wish our availability teams appreciated that.
And even if there is justification for additional process, they should let the ops/dev teams come up with ideas/solutions themselves: we know our tools and teams best.
That is a toxic work environment, and I'm not sure from verb-tenses if you're still there or not. If you are, find another place to work. Your quality of life will improve.
Oh, how I know this. The worst part is, this tends to stay with you in other jobs. In takes time to consider the possibility that maybe bringing a new idea to the table isn't a risky affair.
Now, others already advised you to quit the job — but that's obviously not so easy. But be careful, and if you can, find another outlet for creativity, because this can seriously damage both what you can offer to other companies, and how much you'll be able to enjoy any job.
Sorry to hear that. There are a lot of insecure nerds who work in technology and they enjoy making others feel as insecure as they themselves do. The best advice I can give you is to ignore their ridicule (unless you can learn from it), and move on. And don't be like them when you see a young person make a mistake. Make the world a better place.
Sounds identical to my first job outta college. I knew that some of the senior neckbeards were just toxic people, but at the same time I knew I was really inexperienced so it was hard not to second guess everything I did. In the end I quit (5 years later) but felt like at least I learned a lot about code and even more about people and how to not run a company. Thicker skin too. (p.s. thanks for posting, your comment was a nice change of pace from the usual pissing contest fare)
That is exactly the same situation I was in over a period of years working in a company with someone similar.
In the end I decided that I would do the best job that I could possibly do with the knowledge I had and to hell with the over-inflated opinions of my line manager.
The best seniors are those that answer questions you have, steer you in the right direction and are open to new ideas and suggestions.
I was a senior manager in an organization where one of my peers essentially inserted a similar blame regime into our incident management process. It is a poisonous practice, but I actually loved my job and wanted to improve the place. So my team fought back.
The way that you defeat a system like this is to use it. Be humble, honest and calm, and go out of your way take the hits. But refuse to be blamed for things that aren't your responsibility. Force the problem people to do the same.
That undermines the system, as your putting the problem person in the uncomfortable position of taking responsibility for his actions. The whole point of "blame assignment" is to deflect blame from the golden children.
In my case, the blame regime lasted a few months. It collapses when the person pushing the regime is on the defensive too often.
That doesn't work when you have somebody like me who would never accept blame if it was to be written in a permanent file, because I know it would end up being used to fuel some KPI.
And in development there is always somebody else to blame if you look hard enough (because every time a choice is made, it has costs).
This is so much better advice than "fight the system", "convince your boss", "use these talking points", etc.
If the poster really wants to keep that particular job, but wants to also improve the environment, they need to start following all of the rules they don't agree with totheletter. The worst thing that can happen is that it actually improves the team somehow.
It's not always the right way to make change, but in these situations it really can be the best route. Bad rules and procedures can only be (officially) recognized as such when they're actually being used. There's no need to make a mockery of this stuff, either, because time will tell whether the new rule is really helping or hurting.
The more blame you try to place the fewer commits you're going to get (and the ones you do get will be larger and full of "defensive code"). Less commits, less often will make the problem worse, not better because the merges will be larger and more painful, and more subtle systemic errors will become the norm.
This is one of those solutions that causes more of the problem its trying to solve. Management will love it.
I had a new PM come in and give a similar speech. "If we fail spectacularly, it is going to be my job. But if we just fail, it is also going to be my job. More importantly, in neither case is it going to be your job. So now is the time when you can do spectacular things in utter safety; I'm asking you to take this opportunity and run with it!"
The way I've heard that principle phrased is "praise in public, criticize in private."
It's good advice, because criticizing people in public (i.e. in front of their peers) always backfires -- it just makes the criticized person defensive, so they stop being open to learning from their mistake.
We have such field in JIRA too. At first it wasn't required, so nobody filled it. So it was made required by upper management.
People filled it mostly with our boss (we agreed with him, that when it's hard to say whose bug it's - we assign it to him), or old workers, that work here no longer, that first made commits to the feature where bug is. Or people solving the bug assigned it to themselves. Only sometimes somebody assigned a bug to another programmer who still works here.
I wish bug trackers were better equipped to help find the real root cause of bugs, and more teams actually tried to track that data. Were the specs bad? Is there a problem with our process? Did we not write enough tests? Did someone just screw up?
As it is, bug trackers are good at collecting information and letting you create a process that is as convoluted as you'd like to track the work towards coming up with and releasing a fix, but leave much desired when it comes to trying to reduce your overall defect rate.
That said, trying to pin the blame on someone is the absolute worst way to accomplish any of this, and virtually guarantees that any "root cause" information you try to gather is going to be completely useless.
> I wish bug trackers were better equipped to help find the real root cause of bugs, and more teams actually tried to track that data. Were the specs bad? Is there a problem with our process? Did we not write enough tests? Did someone just screw up?
Some of that comes back to the concept of requirements traceability. Unfortunately it didn't quite make the agile cut. So it's languished in the pre-agile SEng world, locked up in clunky, heinously expensive tools.
The European Directive 94/56/EC [1] "on the investigation and prevention of accidents and incidents in civil aviation and repealing" states that:
> ‘causes’ means actions, omissions, events, conditions, or a combination thereof, which led to the accident or incident; the identification of causes does not imply the assignment of fault or the determination of administrative, civil or criminal liability;_
(Art. 2 part 4).
This is said again in the preamble of the same directive, paragraph 4:
> The sole objective of safety investigations should be the prevention of future accidents and incidents without apportioning blame or liability.
I'm sure there are many other legal texts (including in the US) that state the same thing, as it's plain common sense: assigning blame and getting to the root cause of a problem are two very different and separate businesses; both are necessary but shouldn't be conducted by the same personnel and not at the same time.
Although I tend to agree that morale would decline I do not know the full story of the working environment.
If you have an idiot coder that has been there forever and is hard to get rid of then this field would come up in their evaluation. Needless to say it may get overlooked on other peoples evaluation. It may be used as a way to manage some muppet out the company for wasting everyone's time.
But I don't know the full story, the boss may be a control freak and an idiot himself but he may be actually trying to improve working conditions, he just chose the wrong terminology for the field.
and if he'd rather check it out in his car, have him listen to John Allspaw and Mike Rembetsy talking about blameless postmortem on the food fight podcast (the relevant part starts around the middle of the show):
http://www.foodfightshow.org/2012/05/episode-14-live-post-mo...
A company can likely be 10x (or more) productive if they're shooting for 99% of code being "good enough". It's understood that sometimes mistakes will happen, but you get a lot more done when you know 99% bug-free is OK. What happens with the blame game is that many employees now start to focus on hitting that 100% level, which significantly slows things down, and probably still doesn't result in hitting that 100% bug-free mark. It's just a really bad idea, and is probably instituted by non-technical management.
A lot of ideas sound great to management, and they work too! I mean, other than killing employee morale and resulting in long term loss ;-)
At one of my past jobs, we worked like consultants (we quoted hours that a task would take to get done, tested etc). We filled daily timesheets to ensure that our hours did not exceed our estimates. In fact, the time on those timesheets had to total 40 hours every week. So we worked the full 40 hours. And we billed those 40 hours to the client. In those few months, I was as productive as I've ever been.
Unfortunately, we weren't paid like consultants.. we got a regular salary. Management was doing nothing wrong. They failed to realize that the 'consultant lifestyle' of billing and working in hours only works if there is an equally motivating paycheck to go with it. I've known multiple people to leave because of how hard it was.
You shouldn't need a person to blame field. Good responsible employees will know what they did wrong and work to fix it. And if they don't, having the field in there will only increase finger pointing and people trying to protect their own jobs. And your bottom line will still not improve. So why do it?
"Fix the problem, not the blame." - I first heard this in the movie Red October, I think. The other one that come to mind is John Wooden, the famous basketball coach, claiming that he liked players who made mistakes because those were the ones trying to make things happen.
[+] [-] reitzensteinm|13 years ago|reply
Ultimately it's up to the company to decide where the balance lies, and to live with the consequences. Startups will accept a drastically different risk profile to banks and Fortune 500 companies.
[+] [-] SoftwareMaven|13 years ago|reply
I have a feeling it's more like: <git annotate>; "Oh, it's Bob's fault."
[+] [-] ww520|13 years ago|reply
[+] [-] sseveran|13 years ago|reply
[+] [-] mleonhard|13 years ago|reply
[+] [-] droithomme|13 years ago|reply
Eventually though after he turned over project management of the software to the employees, they responded to the constant bug reports about the lack of this feature and added the ability to have this field. I recall he made a note about it on his blog, but that the project belonged to them and it wasn't his project to micromanage, so now they have the feature whether he likes it or not.
Caution: I probably have half the details wrong, I'll try to find some links.
Update: http://www.joelonsoftware.com/news/20020912.html
[+] [-] xtracto|13 years ago|reply
Instead, filing a bug for Firefox, Gnome or the majority of popular open source projects requires you to provide even the color of your socks.
[+] [-] excuse-me|13 years ago|reply
[+] [-] cletus|13 years ago|reply
Sometimes this will go as far as establishing a "war room", possibly for weeks, dragging in people from a number of different areas to address a particular issue.
It's not about pointing the finger or otherwise apportioning blame. It's about learning from mistakes and preventing them from happening again.
That being said, I've worked at other places where shame is attempted to be used in such situations (which is what this is). In my experience it just fosters a hostile environment.
[+] [-] quux|13 years ago|reply
When done right, RCA is a great thing.
[+] [-] wccrawford|13 years ago|reply
A bad programmer shouldn't work there in the first place, and if they need to be blamed, they're probably doing other things to destroy morale.
[+] [-] quantumhobbit|13 years ago|reply
[+] [-] Diederich|13 years ago|reply
We have good stability (we hit four 9s recently), but we're always pushing for me. The post-mortem process is very positive and affirming. That's because it's transparent, as it should be.
[+] [-] swah|13 years ago|reply
[+] [-] lhnn|13 years ago|reply
And even if there is justification for additional process, they should let the ops/dev teams come up with ideas/solutions themselves: we know our tools and teams best.
[+] [-] unknown|13 years ago|reply
[deleted]
[+] [-] scott_s|13 years ago|reply
[+] [-] LaGrange|13 years ago|reply
Now, others already advised you to quit the job — but that's obviously not so easy. But be careful, and if you can, find another outlet for creativity, because this can seriously damage both what you can offer to other companies, and how much you'll be able to enjoy any job.
Good luck.
[+] [-] 16s|13 years ago|reply
[+] [-] sgoranson|13 years ago|reply
[+] [-] matthewl|13 years ago|reply
In the end I decided that I would do the best job that I could possibly do with the knowledge I had and to hell with the over-inflated opinions of my line manager.
The best seniors are those that answer questions you have, steer you in the right direction and are open to new ideas and suggestions.
[+] [-] Spooky23|13 years ago|reply
The way that you defeat a system like this is to use it. Be humble, honest and calm, and go out of your way take the hits. But refuse to be blamed for things that aren't your responsibility. Force the problem people to do the same.
That undermines the system, as your putting the problem person in the uncomfortable position of taking responsibility for his actions. The whole point of "blame assignment" is to deflect blame from the golden children.
In my case, the blame regime lasted a few months. It collapses when the person pushing the regime is on the defensive too often.
[+] [-] tomjen3|13 years ago|reply
And in development there is always somebody else to blame if you look hard enough (because every time a choice is made, it has costs).
[+] [-] m104|13 years ago|reply
If the poster really wants to keep that particular job, but wants to also improve the environment, they need to start following all of the rules they don't agree with to the letter. The worst thing that can happen is that it actually improves the team somehow.
It's not always the right way to make change, but in these situations it really can be the best route. Bad rules and procedures can only be (officially) recognized as such when they're actually being used. There's no need to make a mockery of this stuff, either, because time will tell whether the new rule is really helping or hurting.
Like you say, humble, honest, and calm.
[+] [-] noonespecial|13 years ago|reply
This is one of those solutions that causes more of the problem its trying to solve. Management will love it.
[+] [-] dhimes|13 years ago|reply
[+] [-] DanielBMarkham|13 years ago|reply
I was sitting down with a team getting started on Agile/Lean, and I was trying to lighten things up.
"Well, the first thing to do is pick somebody to blame when it all goes south," I paused for effect, "and that's usually me, the consultant."
It was dead silent. Everyone in the room was looking at me seriously.
Tough crowd.
[+] [-] roguecoder|13 years ago|reply
[+] [-] starpilot|13 years ago|reply
[+] [-] smacktoward|13 years ago|reply
It's good advice, because criticizing people in public (i.e. in front of their peers) always backfires -- it just makes the criticized person defensive, so they stop being open to learning from their mistake.
[+] [-] ajuc|13 years ago|reply
People filled it mostly with our boss (we agreed with him, that when it's hard to say whose bug it's - we assign it to him), or old workers, that work here no longer, that first made commits to the feature where bug is. Or people solving the bug assigned it to themselves. Only sometimes somebody assigned a bug to another programmer who still works here.
Now the field is optional again.
[+] [-] suresk|13 years ago|reply
As it is, bug trackers are good at collecting information and letting you create a process that is as convoluted as you'd like to track the work towards coming up with and releasing a fix, but leave much desired when it comes to trying to reduce your overall defect rate.
That said, trying to pin the blame on someone is the absolute worst way to accomplish any of this, and virtually guarantees that any "root cause" information you try to gather is going to be completely useless.
[+] [-] jacques_chester|13 years ago|reply
Some of that comes back to the concept of requirements traceability. Unfortunately it didn't quite make the agile cut. So it's languished in the pre-agile SEng world, locked up in clunky, heinously expensive tools.
[+] [-] bambax|13 years ago|reply
The European Directive 94/56/EC [1] "on the investigation and prevention of accidents and incidents in civil aviation and repealing" states that:
> ‘causes’ means actions, omissions, events, conditions, or a combination thereof, which led to the accident or incident; the identification of causes does not imply the assignment of fault or the determination of administrative, civil or criminal liability;_
(Art. 2 part 4).
This is said again in the preamble of the same directive, paragraph 4:
> The sole objective of safety investigations should be the prevention of future accidents and incidents without apportioning blame or liability.
I'm sure there are many other legal texts (including in the US) that state the same thing, as it's plain common sense: assigning blame and getting to the root cause of a problem are two very different and separate businesses; both are necessary but shouldn't be conducted by the same personnel and not at the same time.
[1]: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2...
(Why HN doesn't support Markdown, we'll never know.)
[+] [-] zdw|13 years ago|reply
[+] [-] chrislomax|13 years ago|reply
If you have an idiot coder that has been there forever and is hard to get rid of then this field would come up in their evaluation. Needless to say it may get overlooked on other peoples evaluation. It may be used as a way to manage some muppet out the company for wasting everyone's time.
But I don't know the full story, the boss may be a control freak and an idiot himself but he may be actually trying to improve working conditions, he just chose the wrong terminology for the field.
[+] [-] notJim|13 years ago|reply
[+] [-] jacques_chester|13 years ago|reply
That the records are personal -- they belong to the programmer -- is repeated again and again.
Because when you keep management-accessible records like this, three stages occur:
1. "We won't use this to judge your performance and it will not be connected to reward or punishment".
2. It is eventually connected, directly or indirectly, to reward or punishment.
3. The indicator is gamed.
[+] [-] ragincajun|13 years ago|reply
PS. Your boss is stupid.
[+] [-] alan_cx|13 years ago|reply
[+] [-] kepioo|13 years ago|reply
and if he'd rather check it out in his car, have him listen to John Allspaw and Mike Rembetsy talking about blameless postmortem on the food fight podcast (the relevant part starts around the middle of the show): http://www.foodfightshow.org/2012/05/episode-14-live-post-mo...
[+] [-] viscanti|13 years ago|reply
[+] [-] hackoder|13 years ago|reply
At one of my past jobs, we worked like consultants (we quoted hours that a task would take to get done, tested etc). We filled daily timesheets to ensure that our hours did not exceed our estimates. In fact, the time on those timesheets had to total 40 hours every week. So we worked the full 40 hours. And we billed those 40 hours to the client. In those few months, I was as productive as I've ever been.
Unfortunately, we weren't paid like consultants.. we got a regular salary. Management was doing nothing wrong. They failed to realize that the 'consultant lifestyle' of billing and working in hours only works if there is an equally motivating paycheck to go with it. I've known multiple people to leave because of how hard it was.
You shouldn't need a person to blame field. Good responsible employees will know what they did wrong and work to fix it. And if they don't, having the field in there will only increase finger pointing and people trying to protect their own jobs. And your bottom line will still not improve. So why do it?
[+] [-] tallpapab|13 years ago|reply