I don't usually prefer a video over text. But if anyone is interested in a video about this, Mentour Pilot has an excellent one[1], including a bunch of details about how they ended up in this crazy situation, and how they figured out what's going on.
It's truly remarkable that they managed to land that plane, against all odds.
The obvious solution — using connectors that cannot be fastened to the wrong cable — reminds me of Bob Hoover, an absolutely stunning test and display pilot, who had a forced landing after his fuel tank was filled with jet fuel rather than standard avgas. He devised a bell-shaped nozzle that is now universally used to dispense jet fuel, and cannot be inserted into a standard fuel filling neck.
In this case, though, better design, better tooling, and better training are probably the best fix; there are too many variations in stick-to-aileron connection for a single standard to be accepted.
Even though the author is critical of the GPIAAF's report, it's amazing to see how thoroughly incidents like this one are investigated - With a real focus on process improvement.
Frustrating that the obvious mechanical fix for this (have different shaped connectors for each aerilon cable) wasn't adopted by Embraer, but still a great reminder of how we can do incident handling better in software dev
I find Embraer's rationalization for this (and, presumably, its acceptance by the relevant regulatory agencies) to be particularly galling. Even if it was true at the time that no procedure called for both cables to be disconnected at the same time, did they have any equivalent proof that no such procedure might be introduced later? Did they have any equivalent proof that no-one would misunderstand the instructions, or that it could go undetected if that happened? Could they be sure that this constraint, or its relevance, would not be overlooked at some point in the future? Well, we know they did not. Nobody addressed the question, "what could possibly go wrong?" with sufficient rigor.
When I first started flying GA planes I did the control check and assumed that seeing the ailerons move was sufficient to check that item off. It never occurred to me that you _could_ reassemble the plane in such a way that the ailerons would be inverted. Some older pilot mentioned his friend wrecking a Mooney on takeoff that way (luckily walking out of the wreck) and it scared the hell out of me.
Hats off to the Air Astana guys. That's some solid debugging work and very impressive ability to keep calm and focused under pressure.
In my private pilot training, I was taught to move the wheel both ways and visually check that the ailerons moved in the expected directions. Even if the connections hadn't been changed recently, they could still jam or a cable could have broken.
They were fortunate enough that cables on both sides were inverted; therefore the ailerons "merely" worked in reverse. If they had reversed the cables on one side and not the other, that would have been even tougher to sort out and maybe impossible.
Yep, that would have been a second elevator. There would be completely no way to roll the plane in that case, though I wonder if the plane has a natural tendency to fly level anyway.
What reminds me of computer systems is how the spoilers partly compensated for the opposite role, the computer modes (direct vs. assisted) lead to different behaviors, the test display (green == good?) was mostly self-referential and did not indicate what they thought it should, etc.
In short, the mental model required to understand the plane and its indicia of state was way too complex. These guys literally had to read technical manuals and follow (run-book) procedures step-by-step while the plane was cavorting.
Dynamic stability is the tendency of a system to restore stability, given external forces. There should be a term for dynamic-breaking stability, the ability to restore stability when broken, while also maintaining dynamic and static stability.
This is probably the only real benefit of devops: designing systems on the assumption people will need to intervene to fix them during moments of highest demand.
I watched a lot of accident videos and that kinda is the common trend. Many of them were caused not only by actual failure but by pilots not understanding what the plane was telling them, or plane sending mixed signals (whether actually or by pilot's misunderstandings)
I remember one accident where part of the cause was that both pilots were trying to yank the plane in different direction, and the system just averaged out both inputs and as it was fly by wire and there was no feedback and "too much to do", pilots didn't notice. That's the reason why many planes have pilot's controls connected with eachother - so one can feel what the other one is trying to do.
> This is probably the only real benefit of devops: designing systems on the assumption people will need to intervene to fix them during moments of highest demand.
I have noticed the interesting trend when it comes to automating and "cattlefication" of everything - the less something breaks the less everyone involved understands it. Because it's automated and because nobody needs to even read anything to deploy it (just add a line in Puppet manifest and you're done) when it does break not even the author has a clue why. Documentation helps a little bit but there is a gulf between "describing what it is and how to use it" and "knowing enough to find where the problem is.
I guess that goes to show that no matter the warnings and training, if something can be put in reverse, it eventually will be. Reminds me a bit of Proton crash where accelerometer (which even had arrow marking "up") was mounted in reverse and the rocket wanted to accelerate in direction of the ground.
I bet designers of software were like "there is no need to check whether moving it in direction actually moves it in that direction, we have thousand procedures to make sure hardware is done right". Defense in depth really is the best approach, not just strictly for security
The 2013 Proton crash was more similar to the Embraer incident than you might think. Human pilots can sometimes step outside the box and re-learn how to fly but automated guidance systems aren't that flexible.
The Proton crash involved angular velocity sensors -- rotational accelerometers -- installed upside down. It's not that it wanted to fly into the ground, it was that it couldn't fly straight because (from the guidance computer's perspective) it was responding backwards to the behavior commanded by the guidance system. And rockets just don't fly sideways very well.
Watch the direction of thrust from the engines (the Proton, like most rockets, steers by thrust vectoring) in this slowed-down video of the incident:
Thanks to the mhandley for posting this. I have now read a bunch of these posts (on 'admiralcloudberg') and they are absolutely fascinating.
The ones I find most interesting are those where I remember the media coverage at the time. So often, the post has fascinating details which emerged later. For example:
I remember this crash of a Jamaica-bound plane after 9/11, and I remember the press reporting initially that the tail had been repaired at the factory, and then a few months later that the NTSB had issued a directive that pilots shouldn't respond to wake turbulence too aggressively.
However, the cloudberg post has fascinating details about how the particular characteristics of a training scenario for the A300 had led many pilots to conclude that the proper response to wake turbulence was massive inputs to the controls. It would have been easy to conclude the pilot who made them was just incompetent. But in fact, evidence was he was very competent, and that he had learned what the training taught him quite well. The trainers did not understand what the implementation of their simulation was teaching until after the accident was investigated.
Also, the industry did not fully understand what tests for structural integrity of the tail really said about its capacity to withstand inputs at lower speeds until after this crash. The details are just fascinating.
About a plane where everyone is unconscious and it circles until it runs out of gas. I remember the coverage at the time, which presented it as a mystery. It turns out it wasn't such a mystery, the compression system on the plane was left in an incorrect state by maintenance, and the pilots repeatedly failed to recognize it. The details behind those short statements are fascinating.
> Fortunately, however, that’s something only pilots will have to worry about, because the first flight after heavy maintenance is never conducted with passengers on board
It's an extraordinary achievement by the pilots to keep going under the immense pressure of a 2 hour impromptu rollercoaster ride in the clouds with 6 lives on the line.
It seems color coded cables would be a nearly zero cost solution. I wonder why identical steel cables were preferred, and how this wasn't an issue in manufacturing?
I suspect having a color coding material/technique that can apply to random bits of random plane parts, that won't get rubbed off / broken off / obscured by dirt/grease/grime/etc, consistently throughout the plane, is probably harder than it appears at first glance.
What I am curious about whether it would have been possible to deactivate the spoilers, at least in roll-assistance?
From the MH-370 discussions here I had the impression that the pilot can shut-off every system.
I don't think the article was completely clear about this, but I was left with the impression that the solution was to always turn the yoke far enough that the spoilers overrode the ailerons. The alternative - to use reversed controls - is so against one's experience that it may not be feasible even if you know what is going on.
Are there any home-computer flight simulators (or their controllers) that can be hacked to simulate reversed ailerons?
Not an embraer pilot, but I suspect because (like many larger aircraft) you can't physically see the ailerons from the cockpit – that is why the big jets measure the deflection and (should) show it to you.
Part of me wonders if also as long as you see the ailerons _move_ you assume they've moved in the right direction. There have been similarly tragic rigging accidents in GA aircraft as well – and 38 incidences of aileron mis-connection in gliders between 1974 and 2014 reported in the UK. [1]
[+] [-] elromulous|3 years ago|reply
It's truly remarkable that they managed to land that plane, against all odds.
[1]https://youtu.be/5ywaMkMTwWk
[+] [-] coldcode|3 years ago|reply
[+] [-] JasonFruit|3 years ago|reply
In this case, though, better design, better tooling, and better training are probably the best fix; there are too many variations in stick-to-aileron connection for a single standard to be accepted.
[+] [-] carbonatedmilk|3 years ago|reply
[+] [-] mannykannot|3 years ago|reply
[+] [-] dingaling|3 years ago|reply
[+] [-] unsafecast|3 years ago|reply
[+] [-] BWStearns|3 years ago|reply
Hats off to the Air Astana guys. That's some solid debugging work and very impressive ability to keep calm and focused under pressure.
[+] [-] HeyLaughingBoy|3 years ago|reply
[+] [-] wazoox|3 years ago|reply
[+] [-] stavros|3 years ago|reply
[+] [-] w10-1|3 years ago|reply
In short, the mental model required to understand the plane and its indicia of state was way too complex. These guys literally had to read technical manuals and follow (run-book) procedures step-by-step while the plane was cavorting.
Dynamic stability is the tendency of a system to restore stability, given external forces. There should be a term for dynamic-breaking stability, the ability to restore stability when broken, while also maintaining dynamic and static stability.
This is probably the only real benefit of devops: designing systems on the assumption people will need to intervene to fix them during moments of highest demand.
[+] [-] ilyt|3 years ago|reply
I remember one accident where part of the cause was that both pilots were trying to yank the plane in different direction, and the system just averaged out both inputs and as it was fly by wire and there was no feedback and "too much to do", pilots didn't notice. That's the reason why many planes have pilot's controls connected with eachother - so one can feel what the other one is trying to do.
> This is probably the only real benefit of devops: designing systems on the assumption people will need to intervene to fix them during moments of highest demand.
I have noticed the interesting trend when it comes to automating and "cattlefication" of everything - the less something breaks the less everyone involved understands it. Because it's automated and because nobody needs to even read anything to deploy it (just add a line in Puppet manifest and you're done) when it does break not even the author has a clue why. Documentation helps a little bit but there is a gulf between "describing what it is and how to use it" and "knowing enough to find where the problem is.
[+] [-] ilyt|3 years ago|reply
I bet designers of software were like "there is no need to check whether moving it in direction actually moves it in that direction, we have thousand procedures to make sure hardware is done right". Defense in depth really is the best approach, not just strictly for security
[+] [-] bewaretheirs|3 years ago|reply
The Proton crash involved angular velocity sensors -- rotational accelerometers -- installed upside down. It's not that it wanted to fly into the ground, it was that it couldn't fly straight because (from the guidance computer's perspective) it was responding backwards to the behavior commanded by the guidance system. And rockets just don't fly sideways very well.
Watch the direction of thrust from the engines (the Proton, like most rockets, steers by thrust vectoring) in this slowed-down video of the incident:
https://www.youtube.com/watch?v=vqW0LEcTAYg
[+] [-] blueflow|3 years ago|reply
[+] [-] ttcbj|3 years ago|reply
The ones I find most interesting are those where I remember the media coverage at the time. So often, the post has fascinating details which emerged later. For example:
https://admiralcloudberg.medium.com/days-of-our-discontent-t...
I remember this crash of a Jamaica-bound plane after 9/11, and I remember the press reporting initially that the tail had been repaired at the factory, and then a few months later that the NTSB had issued a directive that pilots shouldn't respond to wake turbulence too aggressively.
However, the cloudberg post has fascinating details about how the particular characteristics of a training scenario for the A300 had led many pilots to conclude that the proper response to wake turbulence was massive inputs to the controls. It would have been easy to conclude the pilot who made them was just incompetent. But in fact, evidence was he was very competent, and that he had learned what the training taught him quite well. The trainers did not understand what the implementation of their simulation was teaching until after the accident was investigated.
Also, the industry did not fully understand what tests for structural integrity of the tail really said about its capacity to withstand inputs at lower speeds until after this crash. The details are just fascinating.
Or this one:
https://admiralcloudberg.medium.com/lost-souls-of-grammatiko...
About a plane where everyone is unconscious and it circles until it runs out of gas. I remember the coverage at the time, which presented it as a mystery. It turns out it wasn't such a mystery, the compression system on the plane was left in an incorrect state by maintenance, and the pilots repeatedly failed to recognize it. The details behind those short statements are fascinating.
Anyway, thanks for posting this.
[+] [-] twelve40|3 years ago|reply
whew : ) but yeah, a pretty epic save
[+] [-] MichaelZuo|3 years ago|reply
[+] [-] deeblering4|3 years ago|reply
[+] [-] elfchief|3 years ago|reply
[+] [-] ilyt|3 years ago|reply
> and how this wasn't an issue in manufacturing?
Manufacturing does same exact thing every single time in order and then someone checks it.
But this was something staff don't do often.
[+] [-] chopin|3 years ago|reply
[+] [-] mannykannot|3 years ago|reply
Are there any home-computer flight simulators (or their controllers) that can be hacked to simulate reversed ailerons?
[+] [-] tetris11|3 years ago|reply
[+] [-] mgsouth|3 years ago|reply
[+] [-] mrlonglong|3 years ago|reply
[+] [-] azalemeth|3 years ago|reply
Part of me wonders if also as long as you see the ailerons _move_ you assume they've moved in the right direction. There have been similarly tragic rigging accidents in GA aircraft as well – and 38 incidences of aileron mis-connection in gliders between 1974 and 2014 reported in the UK. [1]
[1] https://members.gliding.co.uk/wp-content/uploads/sites/3/201...
[+] [-] hyperific|3 years ago|reply