In my experience as a mechanical designer physical testing, especially system level integrated testing, is the only way to find errors in the “unknown unknown” category. In a complex system (mechanical, electrical, software, biological, basically all of them), there usually exist interactions that are very difficult to predict. Software-based simulation can help, but is only as good as the models which are used to describe the underlying physics. In my experience, these models aren’t nearly good enough, and without testing, you can’t even say what is wrong with them.
Full-up testing isn’t always possible, but it shouldn’t be abandoned only because it is costly or difficult.
One of the things I continue to find disturbing when interacting with the markets we work in (e.g. self-driving & other highly-automated systems) is the growing hope that simulation will somehow bail people out of having to do real world validation, let them take significant shortcuts, or make assurance claims that can't really be made.
We deal in formal verification for some aspects of what we do, and that lead to a conversation that went roughly like this:
AV Exec: "Can we use formal methods to prove that our simulator is as good as reality?"
Me: "No. You can use formal methods to prove your simulator implementation more closely adheres to your simulation model."
AV Exec: "Isn't that the same thing?"
Me: "No. Your simulation model is definitely wrong."
AV Exec: (looking at me both disappointed and despondent)
Me: "Look. It's important to understand that simulation isn't creating an approximation with a known correspondence to reality. Simulation is fabricating an entirely new reality with an opaque correlation to our reality."
AV Exec: "Then how am I supposed to use simulation to prove our system is safe?"
I agree with what your saying. As they optimise more and more, they "fly closer to the sun". That would be ok as long as the models/sim tools were accurate, but as you say they are not so much, which means there is not as much room for error.
Here's the thing. I don't doubt that since the controversies with the MAX earlier this year, a great deal many people at Boeing are taking a hard look at everything around them and figuring out how to do better.
I'm also sure that this article could be embellishing the facts. It's entirely possible that the very smart people who think about aviation engineering all day are supremely convinced that some of the digital tests can absolutely replace physical tests that were in place for decades. I'm not even remotely qualified enough to offer an opinion on that, so I won't.
So here is my question. Boeing is a very large company beholden to shareholders, and kept in check by a declining number of other checks and balances. Should we, the public, trust that Boeing will pay close enough attention to maintaining safety as an utmost priority? Or should we be demanding that more oversight and regulation are put in place?
> It's entirely possible that the very smart people who think about aviation engineering all day are supremely convinced that some of the digital tests can absolutely replace physical tests that were in place for decades. I'm not even remotely qualified enough to offer an opinion on that, so I won't.
This is really the foundation of the problem. Members of Congress are not aeronautical engineers, so they're in the same boat as you. Aeronautical engineers work for aircraft makers, so they have a conflict of interest.
Then, because they don't really know what they're doing, they tend to require things that reduce safety, either by requiring measures that aren't cost effective and thereby blow your entire budget including the money that could have been spent on other safety measures with better cost/benefit, or are overly rigid in requiring the specific solution to a problem which was the state of the art three decades ago when the rule was enacted even though safer alternatives are known today.
So why doesn't the government higher some regulators with some relevant industry experience? Well, they tried that, and then we got all of this revolving door nonsense where, to use an example closer to home here, former Verizon lawyer Ajit Pai is now running the FCC as a wholly owned subsidiary of Verizon Communications.
Then we get proposals to prevent regulators from going back to industry when they finish, to try to prevent that. But the government already has trouble attracting talent when they're paying substantially less than private industry does, and the jobs tend to only last for one administration until the next one comes in and replaces them with their own people, so how are they ever going to get anyone good to do a job that will a) pay less than they're making already and b) by law end their private career even though their tenure in government is likely to be less than a decade?
It's possible that we're better off not specifying how to make airplanes safe but instead imposing significant liability on companies and individual engineers who make ones that aren't. (This works in industries where the smallest company has a multi-billion dollar market cap. It's obviously less of a deterrent when the manufacturer has no exposure to the jurisdiction of your courts or is small enough to file bankruptcy every time there is a problem, but then we're no longer talking about Boeing and Airbus.)
It sounds like Reuters is trying to draw a connection between this and the 737 Max crashes, but I don’t think there’s any reason to believe that better testing in silico would not have caught that issue just as well as physical testing
And let’s not forget that too strict requirements for new airframes is part of what led Boeing to keep modifying the original 737 design instead of building something that would require new certification for both planes and pilots.
You're correct that this is different from the 737 MAX MCAS crashes since those involved more human factors though I think it's fair to take a longer look at the proposed shortcuts.
Boeing's corner cutting and lack of regulatory oversight seem to have contributed to the crashes and they rightly should have to prove that this testing will be equivalent.
It's time for the Feds to insist that Boeing return to developing aircraft in an adversarial manner, where dedicated teams try to break everything the primary engineers build. Adversarial engineering is the only proven reliable way to build safety-critical systems. It costs more initially, but it's less expensive than the penalties and lawsuits later.
> It's time for the Feds to insist that Boeing return to developing aircraft in an adversarial manner,
Generally speaking, the government cannot maintain a staff of aerospace engineers since they would rapidly fall behind industry engineers, you know, actually building airplanes with current materials.
However, the FAA could pay for engineers to provide oversight and also test pilots. Either group would have caught the MCAS issues.
Most FAA oversight has always been paperwork-related. You don't need to be a pilot to be an FAA employee.
I'm not really against testing something in a simulation, but in this case would the simulation be using the same software that designed the thing in the first place? Because in that case you're not so much testing it with software as just removing any testing altogether.
> expanding the use of digital analysis over costlier physical testing
Oh no.
> For example, when vibrating a fuselage on an enormous platform to expose weaknesses - known as fatigue testing - the vast majority of the time the tool itself breaks instead of the airframe, according to a person with knowledge of past tests. Such work is costly and has reliably confirmed engineers’ expectations, he added.
Is this just hubris, or is this real? That's a lot of confidence to place in a software model.
Yeah, and it seems that if the tools your using the test for fatique to destruction are themselves too weak for the task, then you upgrade the tools, not abandon physical testing. No computer model will tell you if there are internal microfractures to a component, and any number of other possibilities.
It's not software: it's mathematics. You don't build a bridge by successively creating bridges with more and more structural reinforcement until the bridges stop falling down. Instead, you calculate the appropriate size of the columns using a model and only build a single bridge. This is no different.
Finite element methods are such a known quantity that we're able to design nuclear weapons without ever testing them in meatspace. If it's good enough for nuclear weapons, I am strongly convinced it's good enough for something as simple as stress analysis.
This is another instance of market failure with the mantra of 'freedom', self regulation and 'good intentions' spectacularly coming undone.
Boeing's CEO is incredibly still in office inspite of damning evidence of incompetence which is a straight indictment the whole concept of 'shareholder interest' and accountability.
Can anyone provide one instance where shareholder interest has ensured some kind of accountability of management? Why shouldn't Boeings top management be fired for seriously damaging the company and the brand?
Boeing cost cutting as the Max runtime defects pile up - aircraft groundings, returns, and cancelled sales. My confidence in this rationale is non-existent.
[+] [-] torpfactory|6 years ago|reply
Full-up testing isn’t always possible, but it shouldn’t be abandoned only because it is costly or difficult.
[+] [-] im_down_w_otp|6 years ago|reply
We deal in formal verification for some aspects of what we do, and that lead to a conversation that went roughly like this:
AV Exec: "Can we use formal methods to prove that our simulator is as good as reality?"
Me: "No. You can use formal methods to prove your simulator implementation more closely adheres to your simulation model."
AV Exec: "Isn't that the same thing?"
Me: "No. Your simulation model is definitely wrong."
AV Exec: (looking at me both disappointed and despondent)
Me: "Look. It's important to understand that simulation isn't creating an approximation with a known correspondence to reality. Simulation is fabricating an entirely new reality with an opaque correlation to our reality."
AV Exec: "Then how am I supposed to use simulation to prove our system is safe?"
Me: "You can't."
Despair on both sides of the table ensues.
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] laythea|6 years ago|reply
[+] [-] samcday|6 years ago|reply
I'm also sure that this article could be embellishing the facts. It's entirely possible that the very smart people who think about aviation engineering all day are supremely convinced that some of the digital tests can absolutely replace physical tests that were in place for decades. I'm not even remotely qualified enough to offer an opinion on that, so I won't.
So here is my question. Boeing is a very large company beholden to shareholders, and kept in check by a declining number of other checks and balances. Should we, the public, trust that Boeing will pay close enough attention to maintaining safety as an utmost priority? Or should we be demanding that more oversight and regulation are put in place?
[+] [-] AnthonyMouse|6 years ago|reply
This is really the foundation of the problem. Members of Congress are not aeronautical engineers, so they're in the same boat as you. Aeronautical engineers work for aircraft makers, so they have a conflict of interest.
Then, because they don't really know what they're doing, they tend to require things that reduce safety, either by requiring measures that aren't cost effective and thereby blow your entire budget including the money that could have been spent on other safety measures with better cost/benefit, or are overly rigid in requiring the specific solution to a problem which was the state of the art three decades ago when the rule was enacted even though safer alternatives are known today.
So why doesn't the government higher some regulators with some relevant industry experience? Well, they tried that, and then we got all of this revolving door nonsense where, to use an example closer to home here, former Verizon lawyer Ajit Pai is now running the FCC as a wholly owned subsidiary of Verizon Communications.
Then we get proposals to prevent regulators from going back to industry when they finish, to try to prevent that. But the government already has trouble attracting talent when they're paying substantially less than private industry does, and the jobs tend to only last for one administration until the next one comes in and replaces them with their own people, so how are they ever going to get anyone good to do a job that will a) pay less than they're making already and b) by law end their private career even though their tenure in government is likely to be less than a decade?
It's possible that we're better off not specifying how to make airplanes safe but instead imposing significant liability on companies and individual engineers who make ones that aren't. (This works in industries where the smallest company has a multi-billion dollar market cap. It's obviously less of a deterrent when the manufacturer has no exposure to the jurisdiction of your courts or is small enough to file bankruptcy every time there is a problem, but then we're no longer talking about Boeing and Airbus.)
[+] [-] tempguy9999|6 years ago|reply
[+] [-] elil17|6 years ago|reply
[+] [-] alkonaut|6 years ago|reply
[+] [-] xvf22|6 years ago|reply
Boeing's corner cutting and lack of regulatory oversight seem to have contributed to the crashes and they rightly should have to prove that this testing will be equivalent.
[+] [-] dreamcompiler|6 years ago|reply
[+] [-] supermanfan|6 years ago|reply
Generally speaking, the government cannot maintain a staff of aerospace engineers since they would rapidly fall behind industry engineers, you know, actually building airplanes with current materials.
However, the FAA could pay for engineers to provide oversight and also test pilots. Either group would have caught the MCAS issues.
Most FAA oversight has always been paperwork-related. You don't need to be a pilot to be an FAA employee.
[+] [-] dsfyu404ed|6 years ago|reply
[+] [-] contravariant|6 years ago|reply
[+] [-] mhb|6 years ago|reply
[+] [-] conistonwater|6 years ago|reply
Oh no.
> For example, when vibrating a fuselage on an enormous platform to expose weaknesses - known as fatigue testing - the vast majority of the time the tool itself breaks instead of the airframe, according to a person with knowledge of past tests. Such work is costly and has reliably confirmed engineers’ expectations, he added.
Is this just hubris, or is this real? That's a lot of confidence to place in a software model.
[+] [-] ineedasername|6 years ago|reply
[+] [-] rjsw|6 years ago|reply
Now probably isn't the best time to put out press releases on this though.
[+] [-] chroem-|6 years ago|reply
Finite element methods are such a known quantity that we're able to design nuclear weapons without ever testing them in meatspace. If it's good enough for nuclear weapons, I am strongly convinced it's good enough for something as simple as stress analysis.
[+] [-] taneq|6 years ago|reply
[+] [-] noir_lord|6 years ago|reply
> Beware of bugs in the above code; I have only proved it correct, not tried it. - Knuth.
[+] [-] throw2016|6 years ago|reply
Boeing's CEO is incredibly still in office inspite of damning evidence of incompetence which is a straight indictment the whole concept of 'shareholder interest' and accountability.
Can anyone provide one instance where shareholder interest has ensured some kind of accountability of management? Why shouldn't Boeings top management be fired for seriously damaging the company and the brand?
[+] [-] astrange|6 years ago|reply
[+] [-] kjar|6 years ago|reply
[+] [-] anticensor|6 years ago|reply
[+] [-] solarkraft|6 years ago|reply