It is absolutely a Boeing failure and not due to subcontractors/outsourcing as the clickbait title seems to imply.
If you parcel out parts of your job in something as critical as Airplane Construction, it is your responsibility to validate all specifications/testing/verification/etc. If you treat "Airplane Engineering" as mere "Embedded Engineering" you need to be put out of business.
The article explicitly says MCAS and "cockpit warning light not working" problems (which were the cause of the two crashes) were NOT outsourced.
Key points;
“Boeing was doing all kinds of things, everything you can imagine, to reduce cost, including moving work from Puget Sound, because we’d become very expensive here,” said Rick Ludtke, a former Boeing flight controls engineer laid off in 2017. “All that’s very understandable if you think of it from a business perspective. Slowly over time it appears that’s eroded the ability for Puget Sound designers to design.
Rabin, the former software engineer, recalled one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature. “I was shocked that in a room full of a couple hundred mostly senior engineers we were being told that we weren’t needed,” said Rabin, who was laid off in 2015.
“Engineering started becoming a commodity,” said Vance Hilderman, who co-founded a company called TekSci that supplied aerospace contract engineers and began losing work to overseas competitors in the early 2000s.
The last point is the most damning; Airplane Engineering cannot be commodity engineering.
No disagreements on your main points, but I would dispute calling that headline clickbait. We published this story in 2019 when software issues were the central issues at Boeing, so establishing how the company was coding was very newsworthy. I typically associate clickbait with bait-and-switch tactics, tricking readers into reading content unrelated to the headline. This article provides material directly related to its title.
Robert Schoenberger, editor-in-chief, IndustryWeek
Most big "industrial-like" product companies are pinchers when it comes to SW. They offshore a lot hence why they pay SW devs so little. And it mostly works for them.
It works for them, because like Boeing said, they have old battles tested processes which usually means big mistakes can easily get caught ahead of time. And if big mistakes don't get caught they usually can't kill anyone and generate a huge public scandal.
The article seems to conflate engineering salaries in a low cost of living country with the quality of the software.
As I understand it, the software performed exactly as designed, forcing the nose of the plane downwards to counter the upwards torque produced by the engines being offset relative to the centre of gravity of the plane.
Mentour Pilot had a great episode on the two tragic accidents. It appears that Boeing did not expose the way the software was designed to behave, and the system silently turned itself back on even after the pilots disengaged it.
My experience is that contract work like this ends up being done by ordertakers; the contractors execute the design, no matter how stupid.
Whereas, the best performing companies hire engineers who are empowered to understand the goals, understand the problem, and push back when appropriate.
The results are much better, because the people working on each system, understand it deeply and care.
A low-cost contractor isn’t allowed to care. They aren’t even allowed to talk to somebody who matters.
The theme throughout has been cost cutting and poor oversight for the sole benefit of shareholder profits.
A proper in house team would probably say, hey what happens when the sole sensor feeding our software is obstructed by a birthday balloon? Maybe we need an exponentially backoff instead of every 30 seconds. Maybe nose down actions shouldn't be automatically executed at low altitude.
What we see playing out is the result of delegation and "not my problem", outsourcing safety critical systems to the lowest bidder...
The biggest problem with offshore engineering teams is their dedication to delivering exactly what you ask for. One of the qualities of a good software developer is to take the requirements and interpret what needs to happen then deliver appropriately. Delivering exactly what's written without thinking about the implications is a huge problem and one of the biggest issues I've run into working with offshore teams.
> As I understand it, the software performed exactly as designed
This is very likely correct.
People on HN don't seem to understand how software is created in regulated industries outside the tech bubble.
In a complex system such an aircraft, the behavior of the system is modeled and detailed requirements are generated.
These models are created by the system engineers.
The models and requirements are then handed off to the software engineers to implement the modeled behaviour in code.
The software is then tested to see if the behaviour matches the models.
So it doesn't matter how much the SWEs were paid, if the software met the requirements, and implemented the models as designed, then the software engineers did their jobs.
Yep, the problem with the software wasn't the implementation, it was the spec: to operate contrary to the expectations and experience of pilots.
The problem was that they didn't outsource the design spec. I doubt an externally developed design would be so contrary, with a flippant we'll document it and include it in the short brush up training.
I am not surprised. When in doubt and when you have screwed things so badly, blame the people who don't look or speak or earn like you.
Boeing did have $9 software engineers working for them. They were working on the accounting and ERP softwares. But as soon as 700 people lost their lives, apparently they were working on MCAS, a core system that only NASA engineers should have worked.
I dont believe anything that Boeing says or does, the company is pure evil.
If it's Boeing, I'm not going.
I think the big question is how did no FAA or whatever inspection show all those safety implications until now? What are they missing at other Aircraft manufacturers?
> The Max became Boeing’s top seller soon after it was offered in 2011.
Seriously? I mean, could you still order anything but a Max from Boeing in that size? And that being the most useful size, of course it became the top seller. Because it was the only offering.
The country is not what counts, there are good and bad engineers in any location. The hourly rate is a much more reliable statistic, though without a good quality control feedback loop anything can happen.
Not that you're incorrect in all cases, but generally speaking, you do get what you pay for. SWEs who are charging $500/hr can charge $500/hr due to significant domain knowledge or overall experience, and as such their opinions and work are _worth_ that amount per hour to the market. SWEs who are charging $9/hr are either selling themselves extremely short, lack experience, or operate in an organization that doesn't allow them to negotiate their rate (in which case they should move to a new organization) and thus most likely generate lower quality results. To imply there's no relationship between the hourly rate and the quality of work is to pretend that there's no underlying reason why certain SWEs can charge premium hourly rates.
rramadass|1 year ago
If you parcel out parts of your job in something as critical as Airplane Construction, it is your responsibility to validate all specifications/testing/verification/etc. If you treat "Airplane Engineering" as mere "Embedded Engineering" you need to be put out of business.
The article explicitly says MCAS and "cockpit warning light not working" problems (which were the cause of the two crashes) were NOT outsourced.
Key points;
“Boeing was doing all kinds of things, everything you can imagine, to reduce cost, including moving work from Puget Sound, because we’d become very expensive here,” said Rick Ludtke, a former Boeing flight controls engineer laid off in 2017. “All that’s very understandable if you think of it from a business perspective. Slowly over time it appears that’s eroded the ability for Puget Sound designers to design.
Rabin, the former software engineer, recalled one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature. “I was shocked that in a room full of a couple hundred mostly senior engineers we were being told that we weren’t needed,” said Rabin, who was laid off in 2015.
“Engineering started becoming a commodity,” said Vance Hilderman, who co-founded a company called TekSci that supplied aerospace contract engineers and began losing work to overseas competitors in the early 2000s.
The last point is the most damning; Airplane Engineering cannot be commodity engineering.
RobertSchoenb|1 year ago
aslihana|1 year ago
Shame. Industrial companies, if you decide as pincher, you will be doomed in your little world... as it always has been :)
FirmwareBurner|1 year ago
It works for them, because like Boeing said, they have old battles tested processes which usually means big mistakes can easily get caught ahead of time. And if big mistakes don't get caught they usually can't kill anyone and generate a huge public scandal.
jkic47|1 year ago
As I understand it, the software performed exactly as designed, forcing the nose of the plane downwards to counter the upwards torque produced by the engines being offset relative to the centre of gravity of the plane.
Mentour Pilot had a great episode on the two tragic accidents. It appears that Boeing did not expose the way the software was designed to behave, and the system silently turned itself back on even after the pilots disengaged it.
hgfghj|1 year ago
Whereas, the best performing companies hire engineers who are empowered to understand the goals, understand the problem, and push back when appropriate.
The results are much better, because the people working on each system, understand it deeply and care.
A low-cost contractor isn’t allowed to care. They aren’t even allowed to talk to somebody who matters.
This is a management disgrace.
uuddlrlrbaba|1 year ago
The theme throughout has been cost cutting and poor oversight for the sole benefit of shareholder profits.
A proper in house team would probably say, hey what happens when the sole sensor feeding our software is obstructed by a birthday balloon? Maybe we need an exponentially backoff instead of every 30 seconds. Maybe nose down actions shouldn't be automatically executed at low altitude.
What we see playing out is the result of delegation and "not my problem", outsourcing safety critical systems to the lowest bidder...
rednerrus|1 year ago
wannacboatmovie|1 year ago
This is very likely correct.
People on HN don't seem to understand how software is created in regulated industries outside the tech bubble.
In a complex system such an aircraft, the behavior of the system is modeled and detailed requirements are generated.
These models are created by the system engineers.
The models and requirements are then handed off to the software engineers to implement the modeled behaviour in code.
The software is then tested to see if the behaviour matches the models.
So it doesn't matter how much the SWEs were paid, if the software met the requirements, and implemented the models as designed, then the software engineers did their jobs.
silent_cal|1 year ago
acchow|1 year ago
karmakaze|1 year ago
The problem was that they didn't outsource the design spec. I doubt an externally developed design would be so contrary, with a flippant we'll document it and include it in the short brush up training.
renewiltord|1 year ago
gregors|1 year ago
zamfi|1 year ago
tledakis|1 year ago
lencastre|1 year ago
fargle|1 year ago
what is this? do you have a link?
tremorscript|1 year ago
unknown|1 year ago
[deleted]
reboot_boom|1 year ago
lavezzi|1 year ago
ilrwbwrkhv|1 year ago
nottorp|1 year ago
Seriously? I mean, could you still order anything but a Max from Boeing in that size? And that being the most useful size, of course it became the top seller. Because it was the only offering.
darkhorn|1 year ago
Madmallard|1 year ago
ametrau|1 year ago
[deleted]
nuancebydefault|1 year ago
playingalong|1 year ago
fwungy|1 year ago
[deleted]
methodical|1 year ago
jbreckmckye|1 year ago
There are great engineers in non-Western countries. It's just that they realise talent is so rare, they can charge more than local rates.