top | item 5560135

All American Airlines Flights Grounded, Experiencing Nationwide Computer Outage

155 points| jstreebin | 13 years ago |techcrunch.com

135 comments

order
[+] akcreek|13 years ago|reply
I'm at DFW right now heading to LAX on AA. Computers just came up and they are going to begin boarding shortly. Lots of unhappy travelers here as you can imagine I'm sure.

Edit--

To follow up. It's been about an hour since the computers came back up in the terminal and I'm finally in my seat on the plane. About 3 hours after it was supposed to depart.

There were several canceled flights in my terminal and no other flights going out so people had to stay overnight. Most were very mad as American was not supplying food or hotel vouchers - all those people are coming out of pocket I guess.

I heard quite a few saying things like, "No wonder they are going out of business" and "I will never fly American again no matter what".

[+] w1ntermute|13 years ago|reply
> I heard quite a few saying things like, "No wonder they are going out of business" and "I will never fly American again no matter what".

Yeah, they'll never fly American again no matter what...until it has the lowest price ticket for the time/destination they want.

[+] akcreek|13 years ago|reply
Another update. I'm still here. We are parked at the gate waiting for additional fuel before we can leave. Not sure why they boarded us just to have us sit here for another hour (or more - no clue how long this will end up taking). The good news is the plane has AC power in my row so I was able to build the navigation for a site I'm working on.
[+] impendia|13 years ago|reply
> American was not supplying food or hotel vouchers - all those people are coming out of pocket I guess.

Why are they not subject to a class action lawsuit? The problems rather obviously seem to be American's fault.

Seems like some enterprising lawyer should buy a ticket, drive to the airport, go to the AA gates, and start handing out business cards to everyone in sight.

[+] bargl|13 years ago|reply
Because it's a mechanical failure and not an "act of God" they are responsible for any lodging/food for their passengers due to this delay... I don't know if that's a legal thing, but I do know that's how every airline I've ever flown on works.
[+] reagan83|13 years ago|reply
I fly the DFW to LAX route once a week and have done it on many different carriers (United, AA, and Virgin America). Even though VA offers fewer flights on this route, that has become my sole carrier (even outside of today's events). They handily beat AA on experience, and sell at the exact same ticket price (main cabin, not FC).
[+] akcreek|13 years ago|reply
Okay, last update. I made it to LAX, picked up my rental car then received an email from American notifying me that my flight was on time - the same flight I just got off of that landed over 4 hours after it should have. I had to laugh at that one.
[+] TallboyOne|13 years ago|reply
I like how it says "apologize for ANY inconvenience" as if theres the possibility that all planes being grounded making hundreds of people late for meetings offers any room for there not to be an inconvenience.
[+] jmathai|13 years ago|reply
Or worse, parents with young children stuck at airports losing their mind.
[+] akcreek|13 years ago|reply
I don't have anything scheduled until the morning luckily so no big deal for me. I was more inconvenienced by the fact that they don't have enough power outlets to charge devices here (DFW).
[+] ryanmarsh|13 years ago|reply
To me the tone of that tweet sounded very insincere.
[+] ari_elle|13 years ago|reply
"American said the issue was caused by an inability to get access to its reservations system, called Sabre. The electronic system, often described as the brains of an airline, is responsible for bookings and reservations but also manages a wide variety of functions related to flights, including printing boarding passes, online check-ins, ticketing, and tracking checked bags. [...]

Sabre, meanwhile, said American’s system outage did not come from its own computer systems. Other airlines, including Southwest Airlines and JetBlue, use the reservation system and have not experienced any outages, said Nancy St. Pierre, a spokeswoman for Sabre."

Source:

http://www.nytimes.com/2013/04/17/business/american-airlines...

[+] ryanmarsh|13 years ago|reply
SABRE was fine. Some of the systems at AA couldn't talk to it. Having seen their systems none of this surprises me.
[+] cyanoacry|13 years ago|reply
A similarly described SABRE outage in 1989 seems to have caused minimum trouble, because they had the real capability to work off-line with paper:

http://www.nytimes.com/1989/05/13/us/computer-chaos-for-air-...

> Apparently, no information about reservations and other travel plans was lost, and American said there were no serious disruptions of its 2,300 daily flights.

[+] pjbrunet|13 years ago|reply
From the NYT article, sounds like it was a connection problem. Which makes you wonder, was it a DoS attack (either end of the connection?) preventing AA from connecting to Sabre?
[+] mikeash|13 years ago|reply
Not really related to this outage, but I think it's interesting to note that Sabre dates back sixty years! The ball got rolling in 1953, with a chance meeting between AA's president and an IBM salesman. IBM was working on a massive air defense computer system at the time, and they decided that a similar system would be good for airline reservations. The system went live in 1960, and has been running continuously ever since.
[+] tpowell|13 years ago|reply
I can't wrap my head around how extraordinarily complex airline reservations/logistics systems are. I'm kind of surprised you don't see this more often.
[+] mimiflynn|13 years ago|reply
Looks like they use Sabre:

UPDATE, 1:43 P.M.: The Federal Aviation Administration has confirmed the ground stop. “GROUNDSTOP IMPLEMENTED BY ATCSCC ON AMERICAN AIRLINES NATIONWIDE AND AMR AMERICAN EAGLE INTO DFW, LGA, AND ORD AT THE REQUEST OF AMERICAN OPS. STOP REQUESTED AS RESULT OF AAL SABRE SYSTEM FAILURE.”

from http://www.dallasnews.com/business/airline-industry/20130416...

[+] jcdavis|13 years ago|reply
United's failed in November, although for a much shorter duration (hour or so). As luck would have it, I was flying United that day, and I'm flying American today =/
[+] ryanmarsh|13 years ago|reply
It's not that hard of a problem to solve really. Weve come a long way as an industry since SABRE came out 53 years ago. AA's biggest problem is that their IT dept is 80% low grade contractors and they've been unsuccessful in delivering their own solution for what SABRE does. They've been pushing back the sunset of SABRE for over a decade. Many days I'm surprised anything written at AA runs at all.
[+] sterlingross|13 years ago|reply
I am surprised no one has mentioned Rule 240. I have, more than once, gone up to the agent when a flight was delayed and asked about this rule. They always hesitate at first, but if you insist, they quickly make provisions to put you on another flight.

This only works when it is the airlines fault and not an issue with weather.

Rule 240: http://en.wikipedia.org/wiki/Rule_240

[+] nlh|13 years ago|reply
Putting my tinfoil hat on for a moment...

I wonder if there was a threat against the airline? Perhaps something enough to be credible but not so specific as to be pinpointable, and rather than cause widespread panic by saying "ground every AA flight until we sort it out" (which would freak people out beyond belief), someone figured out that pulling the plug on their mainframe for a few hours would have the same result without the panic. Just some pissed off passengers.

Or maybe I've just been watching too much Homeland. Sigh.

[+] ryanmarsh|13 years ago|reply
That's just your paranoia. I wish I could speak more freely than I already have (see my other comments) but the fact is, crappy software, written by crappy contractors all tied to a 53 year old system of record.
[+] simcop2387|13 years ago|reply
I can't find the article, but this isn't the first time that an airline has had a problem that grounded all of their flights. There was one in recent memory (last 20 years?) that had a bug where if there were more than 32k changes to the flights in a day it would cause a horrific crash in their system. They never hit it for so long, and then one day, boom all flights just disappeared.
[+] clueless123|13 years ago|reply
Never ascribe to malice that which is adequately explained by incompetence.

Hanlon's razor.

[+] tquai|13 years ago|reply
Just a general comment on airlines:

With US-based carriers, expect low prices, inflating awards points/miles, and bottom-level service.

With non-US based carriers, expect higher prices, no awards programs, and excellent service.

This is my experience flying in N. America, Europe, and Asia.

[+] mjn|13 years ago|reply
You must not be flying the low-cost airlines in Europe. I have never seen any US-based airline that rivals the combination of low-price and bottom-level service provided by Ryanair. Unfortunately, the low-end airlines are so popular within Europe that it's increasingly hard to avoid them, even if you want to pay more, because they have many of the direct flights (especially to vacation destinations).
[+] k-mcgrady|13 years ago|reply
I don't think this is true at all. Europe's low cost airline business is major. Easyjet, Ryanair, Flybe and Jet2 just to name a few. Obviously the service on these carriers is awful but flying between European cities for $50 a flight isn't uncommon. Personally my most consistent good service experiences have been with Air Canada. BA was good too as was Aer Lingus. I've only had two flights with them but I wasn't fond of AA.
[+] JimboOmega|13 years ago|reply
Even the US-based "legacy" carriers find themselves directly competing with Southwest, which has forced them to gravitate towards the low-cost model. The only thing that differentiates them in such an environment is their award systems. Except Southwest which (ironically?) has the best service and perks now.
[+] dagw|13 years ago|reply
It's admittedly been a few years since I flew a US low cost airline. But at least then they where pure luxury compared to Ryanair or Easyjet in Europe.
[+] monksy|13 years ago|reply
Thats not always the case.

I've been badly screwed over by Leftouthansa many times, their response: "Nope, we don't care. We look forward to seeing you onboard." [After 2 support tickets, one to complain and then the other to report the unanswered first ticket after 2 months]

I have been treated well by KLM.

[+] deelowe|13 years ago|reply
Huh... Explain how Southwest has done so well then? They are cheap, profitable, and have the best service.
[+] seanmcdirmid|13 years ago|reply
> With non-US based carriers, expect higher prices, no awards programs, and excellent service.

Cough...China Eastern Airlines...cough. And oh my god, Air China. You guys have nothing on inconveniences when compared to the good ole PRC.

[+] dia80|13 years ago|reply
I feel for the poor sods thrashing away trying to make this right. Do developers in mission critical areas earn a premium? This could knock a couple of years off your life in stress.
[+] nwh|13 years ago|reply
I'm picturing a disk failure, or a RAID5 with two dead drives. Someone at AA franticly running disk recovery software.
[+] edandersen|13 years ago|reply
Nice to see their legacy systems were kept up to date and maintained.
[+] drcoopster|13 years ago|reply
As a frequent US Airways flier, this makes me feel really warm and fuzzy about the upcoming merger with American Airlines.
[+] capkutay|13 years ago|reply
I'd like to see a paper on American Airlines software architecture. Although I doubt it'll pop up on "High Scalability" any time soon. Do we know anything about Sabre's platform? I imagine its highly redundant..how could they have a nationwide outage?
[+] eddmc|13 years ago|reply
The hosting company that American Airlines uses is Sabre

This is the same company used by Virgin America, US Airways, JetBlue and Alaska Airlines

[+] maxdemarzi|13 years ago|reply
I got stuck for 2 hours waiting for "paperwork" flying from Chicago to Philly. At least I managed to finish my book.
[+] w1ntermute|13 years ago|reply
Why haven't carriers like AA been displaced by more competent (and passenger friendly) alternatives yet?
[+] SODaniel|13 years ago|reply
For a minute I had a '24' flashback and started packing canned goods and ammunition into a 'bug out bag'..