[donarb]

Years ago, I was a second-shift operator in a computer center for an insurance company. We ran production jobs on an IBM mainframe. When jobs would crash we would write up the error on an ABEND form (IBM called crashes ABENDs for ABnormal END), collect the printout and call the programmer responsible. One night a production job crashed late in my shift about 10:30-11 pm and I woke up the programmer responsible. He seemed really groggy and it took a few minutes for me to describe the crash to him. I would always try to be helpful and suggest options for recovery (you got to know the programmers and what their recommendations would be based on the type of production job). Usually they would hold the job, restart it or say they were coming in to fix, this was back in the day where if they could log in remotely, it was with a clunky CRT terminal.

The programmer told me to just restart the job, I noted that on the form. I came in the next day and my boss called me into his office, his boss was there too. They wanted to know why I restarted the job, which caused all kinds of corruption to the database. They had spent the better part of the day recovering the database, then running the batch job, which meant that that system was unavailable for use by the agents.

The programmer swore up and down he did not tell me to restart the job, said I never called him! He was that deep into sleep. But on the form I noted the time I called him and his response to restart the job, so they believed me.

This highlights a problem with people responsible for multi-million dollar systems being woken up in the middle of the night and having to make quick critical decisions.