”My emails produced only well-worded refutations. They explained quite factually why the setup is the way it is, and implicitly therefore why it could not change”
This landed so truly for me,
it felt like a punch in the stomach.
I wouldn’t dare count the number of times I’ve been told the technical details of why something is the way it is, without anyone ever saying the reason why we actually wanted it to be this way. My thesis was usually:
we don’t.
In my career I feel like I have seen hundreds of examples of me saying the systems equivalent of “lets put the dining table indoors?” to be told that the dining table is outside because the original budget meant the front door could only be yay wide so we had to leave the table in the yard and put a tent over it. And I’m just left standing there agape at how we eat in a cold wet tent every night instead of fixing it.
Except it’s usually more like: why do we have to spend $9k on a commercial dishwasher repair contract? Because we have a commercial dishwasher … to get the rust off the silverware … because we eat outdoors every night … because the front door was too small to get the dining table in the house.
Somehow, when the real examples of this stuff are clever engineering around build / docker / polyrepo / release / feature flags / third party bugs, the cleverness makes people think the existence of the workaround should be tolerated. It’s infuriating to join a new team held hostage by years and years of band aids because they never suffer the bigger picture consequences.
The whole article was fantastic. I hope the author has the engineering leadership role they deserve. We need more people like this.
If you spend all your time refactoring, cleaning up after legacy design constraints, fixing ossified errors, then you run out of time and fail to write actual meaningful income generating features. Conversely if you make none of those improvements, eventually the weight of bad architecture slows all progress to a halt and no new income generating features get delivered.
One of the hardest parts about advancing as a developer, in my opinion, is being able to tell when you should refactor versus just leaving the old working mess alone.
With your example, it's like if it turns out that the dining table is embedded into the ground with concrete because it kept blowing over, and moving it indoors would require getting a carpenter to create new legs. And also that because the dining table has been there for so long, someone decided to run electricity cables through it, so rerouting it requires an electrician and will shut down the factory at the bottom of the garden for half a day. We could buy a separate table for indoors, to try and slowly migrate to the new table, but then we'd have two tables to maintain and we all know how that usually goes.
At a certain point, you look at it and go well the commercial dishwasher is just $9k and we can focus our efforts on building that loft conversion for now.
> In my career I feel like I have seen hundreds of examples of me saying the systems equivalent of “lets put the dining table indoors?” to be told that the dining table is outside because the original budget meant the front door could only be yay wide so we had to leave the table in the yard and put a tent over it. And I’m just left standing there agape at how we eat in a cold wet tent every night instead of fixing it.
I have, too. And then I usually haven't managed to put the dining table indoors. And then new people came in and asked the same question you ask, and by then I was one of the people who tried to put the dining table indoors, and explained how it wouldn't fit through the front door, and how I tried to get it in through the window. And then the new people try to put the table indoors and fail and next thing you see they're either leaving the house or explaining to the newcomers why the table is outdoors.
Ultimately, I've realized that talk like this is cheap, unless you can actually improve things. That requires leadership skills and some political capital in your organization. I don't think the author of the article deserves an engineering leadership role simply for complaining about things. (They might still deserve an engineering leadership role for other reasons, what do I know...)
One of the biggest systems-level failures in recent memory is the Boeing 737 MAX story. I read this article and your comment and then went looking for an autopsy, found this:
"The Boeing 737 MAX: Lessons for Engineering Ethics (2020)"
It's an example of a workaround that should not have been tolerated:
> "The Maneuvering Characteristics Augmentation System (MCAS) software was intended to compensate for changes in the size and placement of the engines on the MAX as compared to prior versions of the 737."
Rather shockingly this wasn't even an engineering problem workaround; it does seem that it was solely designed to avoid an aeronautical reclassification of the aircraft that would have required pilots to undergo an expensive retraining program on flight simulators, which might have caused lost orders.
This does look like a systems-level failure, but one at an organizatonal level: the system went from a state where engineering took priority, to a state where financialization took priority. In systems thinking, this could be called a state transition: a fluctuation takes place, and afterwards the system settles down to a new (apparently) stable state quite different from the old state:
> "One factor in Boeing’s apparent reluctance to heed such warnings may be attributed to the seeming transformation of the company’s engineering and safety culture over time to a finance orientation beginning with Boeing’s merger with McDonnell–Douglas in 1997 (Tkacik 2019; Useem 2019). Critical changes after the merger included replacing many in Boeing’s top management, historically engineers, with business executives from McDonnell–Douglas and moving the corporate headquarters to Chicago, while leaving the engineering staff in Seattle (Useem 2019). According to Tkacik (2019), the new management even went so far as “maligning and marginalizing engineers as a class”."
> It’s infuriating to join a new team held hostage by years and years of band aids because they never suffer the bigger picture consequences.
What’s even more infuriating is seeing new engineers join that team, question why the hell something insane is insane, and then slowly grow used to the insane thing. Only for the cycle to repeat when the next new person joins.
I make a huge point of confirming to them, that yes we do acknowledge it’s insane. Even if management doesn’t care.
> In my career I feel like I have seen hundreds of examples of me saying the systems equivalent of “lets put the dining table indoors?” to be told that the dining table is outside because the original budget meant the front door could only be yay wide so we had to leave the table in the yard and put a tent over it. And I’m just left standing there agape at how we eat in a cold wet tent every night instead of fixing it.
Oh wow, that hits home. To be fair, the historical context for the decision can be valuable information, the problem is the next step. Even if you can't fix it right now, you might make steps towards that. Or you might say: Now that we have this heavy table outside, why not attach more things to it?
everyone's worked in companies like this. did you try formulating very specific, actionable migration plans at any of these jobs? It's one thing to say, "this is stupid! we should use XYZ" and expect everyone to say "wow! you're right, we'll do that right away", and another (quite another) to actually formulate the superior architecture concretely, break it into digestible migration steps, sell the organization on if not the whole architecture at once, then at least on the first several migration steps, and to guide the organization into that new architecture for real.
obviously this is more or less possible given specific organizations and personalities at said organizations. my point is more that the magnitude of the task of migrating the ossified organization to a better architecture, even with a fully pliant staff totally on board with changing, should not be underestimated.
... because we forgot that you can take the legs off of the table ... because the screwdriver we needed to take off the legs was in use elsewhere when the table arrived ...
There are two of me. The me of now, and the me of hindsight. Hindsight me is way smarter, and able to criticise every decision we made leading us into the mess that is now. Now me needs to make quick decisions based on imperfect information, budgets and tight timelines.
If it were up to hindsight me, we'd have carefully designed and orchestrated every past decision. We'd do full design qualification and change control on projects and purchases, researching carefully to never make a mistake. We'd sit as a team and brainstorm every possible implication. We'd write, execute and document tests for everything to prove ourselves. Any sniff of inefficiency and we would stop everything and fix it, no matter the cost. We'd take time to document, investigate and follow through. It would be a glorious cavalcade of plans and CAPAs, qualifications and tests. And reports! Binders and binders of wonderful validation reports everywhere!
If it were entirely up to hindsight me, we'd run our little widget company like we're building a space shuttle. Of course, we'd never make any money. But we'd be doing it right, by gum!
Most companies need to be somewhere in the middle. No hindsight and you end up a tangled mess of short-sighted kludges, all hindsight and you can't move forward. Either way you risk ending up a lead balloon.
If you find yourself in the kludge company territory, then here's some advice from a talk I recently attended[1]:
Start by training yourself and your team in Root Cause Analysis. Empower them to start thinking deeply and critically about what's really causing the problems and inefficiencies you encounter. Understanding root causes naturally translate into solutions that aren't just bandaids. Use these skills in your day to day, and you'll start building a culture of quality around your systems.
> I wouldn’t dare count the number of times I’ve been told the technical details of why something is the way it is, without anyone ever saying the reason why we actually wanted it to be this way. My thesis was usually: we don’t.
In my experience, at the time the decision was made, folks did want it that way. The organization has lost that context as to why and has only documented the technical design.
A curse shared by less effective engineers I've worked with is to rage at legacy decisions unable to convince the organization to revise them. They lack the ability to understand the various stakeholders involved and to come up with a plausible plan. A systems engineer (as referenced in the blog) would understand the various sub-systems that make up an organization and be able to drive the change they desired (the conclusion that it's irreparably broken or you lack the expertise to fix it would be fine too).
Maybe it's just over my head, but the article was quite a letdown after your great analogy. It may have valuable information in it, but I don't see how I could share the article with the people who might need to hear it and have them understand or care. It's preaching to the choir.
I'm reminded of back when I was studying pattern recognition for a system that would become an Expert System (this was before that term was used). I would read many articles saying what techniques would work. I had the urge to ask "But this doesn't show how you got there. Show me your discarded solutions that didn't work." I would like to see your wastebasket.
Similarly, I am inclined to someone who acts as an expert to tell me five ways that won't work.
> Somehow, when the real examples of this stuff are clever engineering around build / docker / polyrepo / release / feature flags / third party bugs, the cleverness makes people think the existence of the workaround should be tolerated. It’s infuriating to join a new team held hostage by years and years of band aids because they never suffer the bigger picture consequences.
The only logical reason to do this is because it has no impact on the business. Or at least, smaller impact than a total rewrite/refactor would have.
If an engineer presented a case where fixing an underlying issue resulted in better business outcomes vs. a short term band-aid, then I don’t know anyone that would tell them no. Businesses want to succeed. They want to make money. If you can help me make more money I’ll let you do whatever you want (within the confines of the law and civil society).
I think half the time there was no need for a dining table in the first place, just a shiny solution waiting for the question to be figured out afterwards.
I work in a small organization which is kinda chaotic, meaning we have a lean process. Very agile, short standup each morning, but nearly no sprint planning.
I think, it's liberating to work in such a company as an experienced developer. You get difficult tasks sometimes, where you need to be the System Engineer and understand the system fully. But you're free to approach the problem in any way you see fit. This requires a lot of trust from the company and your team, but I believe that's a good thing.
It's also a nice for customers, who sometimes have crazy requirements, but they still get results in a reasonable amount of time.
On the other hand, this approach completely destroys newcomers. It's nearly impossible for them to approach a System which they can't fully grasp and is in constant flux. We mentor them, give them easy and introductory task, review their code, but still... It takes a massive amount of time. And I think, it's one of the reasons a company like this has a hard time growing.
And there's the risk of the 'not so competent programmer', who knows how to fix stuff, without thinking far enough.
I don't know, if I'm a System Engineer, but I think it aligns with the description in the article. However, I fear the day, we all agree to overhaul our processes, after which we need to plan and document and review everything.
It seems to be a necessary evil for a company to grow, but at the same time we would also destroy a lot of the liberty we currently have and I don't know which side is worse.
> And there's the risk of the 'not so competent programmer', who knows how to fix stuff, without thinking far enough.
I would actually claim that majority of non-seasoned programmers fall into this category.
It's rare to see an engineer who would actively set out to destroy or cripple the system they are working on - but it is incredibly common to see one fix the problem they have at hand, using what happens to be available, and make the overall system just a tiny bit less pleasant to work with. Or understand. Let alone maintain.
Repeat the above a hundred times and your codebase would make Lovecraft take note. If each such modification reduces the quality just 0.5%, after 100 rounds you're looking at an aggregate damage of 40%.
> It seems to be a necessary evil for a company to grow
Yep, the needs of a company change when and if it grows. Early days, a small team "getting shit done" is probably what you want to find your place in the market, make early customers happy, etc. But as you scale on all axes: time, headcount, headcount turnover, customer count, feature count and so on, that early pile of shit that got done can start to bog you down. Even worse if no one recognises the need to change and continues adding to the pile.
I've seen a few really brilliant engineers struggle with this in various ways:
1. Start hating their job but being confused because they love the company -- not realising that the company has become a very different beast to the early days.
2. Struggling to get out of the "get shit done" mentality and just floundering. Even becoming a net negative contributor in some cases.
Worth remembering this I think in an industry where rapidly changing companies are so common. Your company might be a good fit _now_ but that does not mean it will stay that way!
I work within an organization resembling this, and yet it has remained profitable for multiple decades. I forever wonder when the technical naked short options[0] will get called, but they never have in a serious way. My suspicion is that the chaos is S.O.P. among peer companies, so the market has never experienced a competitor not so hobbled. It may also be that controlling chaos is so expensive it's a non-obvious competitive advantage not to do so.
[0] technical debt is when you have a plan to pay it down. A technical naked short option is when your plan is to hope the subsystem goes away before its deficiencies have catastrophic effects.
I think the underlying reason for why IT companies are so chaotic and inefficient, is because they simply don't need enough people, if they are well structured, do quality work and use simple and powerful tools/solutions.
You could run a super high valued company on just a handful of people, and society is just not ready for that. It's a case of technology advancing faster than human organisation and economic system, and the multiples are just too large, it's a big challenge for society to handle the impact.
So it's better for the organisations to be less efficient and chaotic, and to use dumbed down tools, overcomplicated solutions etc. This will fill the void created by the advancement in technology, so that the organisation can continue to function by at least resembling the traditional model.
You would need to significantly shorten the working hours in order to enable more efficient companies. Otherwise you'd get even more hyper concentration of wealth in a very short time.
I think badly run tech organisations still work fine. Most of the time it doesn’t matter whether your problem is solved with spaghetti and 300 hours, or 1 hour and a fantastic design, since the gains are still orders of magnitude higher than the cost.
It’s just that such an organisation is unpleasant to work for.
> I forever wonder when the technical naked short options[0] will get called, but they never have in a serious way.
What does «called» mean in this analogy? That the subsystem doesn’t go away?
So your experience is that: Stuff tends to stick around? That’s my experience. I have never encountered a throw-away MVP that was actually thrown away if the product continued in the same direction (and didn’t pivot).
Complexity never really goes away, it can merely be transformed from one type to another. This organization has apparently decided it rather deals with the complexity of not having processes rather than the complexity of maintaining them. Which is a valid choice: look at India. A society totally chaotic to a stranger but one that has stuck around for millenia.
> the curse of systems thinkers is to be correct, but never valued.
"Systems" are a mixed blessing, but system thinking is almost always
good and useful. We can add value but not bask in it. As a someone
invested in the _idea_ here are a few of my favourite quotes that
illustrate:
- People don't like systems. Especially new ones.
- Systems ossify and become the problem themselves.
- The ideal system exists only in the mind of its designer.
- The ideal systems designer is invisible and can never take credit.
"I mistrust all systematizers and avoid them. The will to a system is a
lack of integrity." -- Nietzsche
"The English have a system, which is *no system*, which is also a
system, only better." - (?? British political philosopher c 1900 -
does anyone know this one?)
"A complex system that works is invariably found to have evolved
from a simple system that worked. A complex system designed from
scratch never works and cannot be patched up to make it work. You
have to start over with a working simple system" -- Gall
Overall, I think the thing is that systems are brilliant, until you
try to actually build them and encounter _people_, who have other
ideas. Neither the force of the better argument, nor punishment,
reward, bribery, or flattery will move things. This is neither the
fault of systems thinkers nor people but the misunderstanding that
(outside the immediacy of war) systems can be imposed. Working
systems evolve and are, if the individuals are mentally healthy and
motivated by good attitude, generally such that people are doing the
thing they would naturally be doing anyway were a formal system not
there.
A good system is like cat that falls off a tall building and by luck
lands on its feet in a box of wool, and licks itself as if to say -
sure I meant to do that.
> "The English have a system, which is no system, which is also a
system, only better." - (?? British political philosopher c 1900 -
does anyone know this one?)
I don't, but that quote was used in a comment [1] about five or six weeks ago, and the commenter's relevant bit was:
Nietzche said it best:
I mistrust all systematizers and avoid them. The will to a system is a lack of integrity.
Or maybe (I think Sidgwick):
The English system is "No system", Which is also a system, only better.
Your last quote reminds me so much of the Gaia X project. The short version of that project is: a standardized protocol of privacy preserving data exchange. But suffering from design by committee and wanting to throw in everything in existence (including blockchains, cause why not) makes it into something that on release (if ever) will just crumble under its own weight.
I think cross-discipline training is really under-rated and important. For instance, as a 3.5th year civil engineering student, I’ve been taught systems engineering and project management multiple times and in multiple different contexts. These are integral to ‘physical’ engineering, but seem (to me) to be missing from software engineering. I’m dabbling in programming and software eng now and I’m constantly surprised by the lack of standardisation and the sort of ‘wild west’ approach to things. This is fine for getting things done, but in terms of liability and responsibilities (like what the post talks about), it seems that many jobs are ill-defined and poorly scoped.
Overall, I think software and ‘physical’ engineers should swap experience. Physical engineering could use a tech-injection, and software could use a ‘structure’-injection.
Before clicking, I thought the article is about someone who only thinks in "system" and not able to deliver concrete things. How wrong I was.
"Give yourself permission to let the organisation fail".
I agree. As someone is similar situation, the job is to let the "design of system" be heard, be debated and be implemented once green-light. You cannot convince the decision making body (be it the CEO, management, or a design committee) that your idea is the right one for 3 reasons IMO
- idea could be wrong
- People won't know what you are talking about unless they had the first hand experience. (A la you don't know what it is like to be a bat)
- Your meritocracy is limited to a small group of decision making body
If you see the "system" broad enough, you will see a market. It is essentially preaching your "system" to a wider audience. Your system maybe wrong, for which your start up will die or you devise another system. Or you are reaching whole bunch of people who understands your (the initial niche), and your living does not depend on a single decision body but a market.
Convincing one body is hard, but broadcasting whatever you believe, some will respond eventually. This is why start up is great.
>Steve recommended investigating Systems Engineering as a distinct subject. Specifically, reading the engineering histories of the Gemini and Apollo projects, and especially about the culture clash between the experimental aircraft guys who built Mercury, and the ICBM teams.
Not a book, but MIT OCW has a class on Aircraft Systems Engineering 16.885J (focusing on the Space Shuttle) that is an excellent example of this. The class starts by looking at the requirements for the system, then examines every single subsystem on the Shuttle, and explains, in detail, why it was built the way it was. Almost always the answer is "because of the constraints placed on the system by the rest of the vehicle" + it has to be as light as possible.
I initially watched the lectures because of an interest in aerospace and it is a fascinating historical series in its own right with some incredible speakers. The lessons for systems engineers are numerous too.
Wow, this whole thing sounds exactly like me. I'm definitely at the stage where I've given the organization permission to fail and am just drudging on now.
I mean, the system I started working on 2 years ago was using a business text field for processing decisions. That caused changes to the system when the business wants to make changes. If they do make changes, the reporting queries have to be modified to look for all the historical versions of that text for audit reasons. If the team asking that change forgets to tell one of the numerous other systems that also uses that field, then there are errors. I proposed we add a field with a code that represents this field so that the business can change the display text without affecting the systems that currently use it. It's been at least 18 months, and nothing has changed. You would think that this is a basic design best practice that should have been implemented from the beginning...
This rang so true. I recently left a large multi-national aerospace company where I and my team had developed local processes that put systems engineering first.
Unfortunately our main contract was developing a component of a larger system being developed by our parent organisation who didn’t have a concept of systems engineering. We tried for years to educate them and I watched aghast as their program costs and schedule continued to spiral out of control.
Basically a bomb burst of engineers all doing what they thought was the right thing but no one owning the system design and saying no to good ideas.
In the words of their chief engineer “its like 10 different black boxes, I don’t know what I’m getting and I don’t know when it’ll be finished!”
There’s much to dive into here — ty — from Steve’s “Systems … where it is all about interfaces and trade-offs” to points you make throughout supporting his conclusion
> we define a stable interface
> reproduce behaviour reliably
> Predictability is great
Within being believed and valued for protecting engineers and organizations, so much progress applying these principles relies on individuals’ and teams’ readiness to adopt and advocate for them. Hope to see your experience with that in part II.
[My colleague] would appear to see time spent in planning and writing documents as essentially wasted time, since he asserts that without process we are faster. We are not faster in reality. We are merely faster to say we're finished. That is not the same thing as actually being finished: we can all think of examples of that. And the uncertainty induced by the unknown magnitude of correction required is, IMHO, the biggest contribution to our inefficiency and ineffectiveness.
I've spent my entire career working for a medium sized organization and in the past few years it has tried to become "agile". Most of this push is predicated on the idea that we will be able to go faster this way. As a result we have deconstructed ourselves. What's weird is that now we aren't actually even "faster to say we are finished." No, now we are only faster to say that we are going to be faster to be finished. We still end up taking a long time but as long as the slide deck says we are going to be done in a short amount of time, then all is believed to be running smoothly. It's very strange/depressing/unsettling.
Systems thinking is great but he way we reason about organizations and systems in a pretty medieval way.
Most systems are described in a simple way: box and line charts which are just voodoo.
We need to develop a algebraic notation for how to represent the various configurations/states of an organization, along with its operations.
With a notation, you can describe what you “feel” and share the knowledge and improve on it. You can also define the organizations desired state and track your progress. You can plop a new manager in it and they will know what to do. You can also do basic engineering like pick a configuration that has the desired cost, throughout and latency that you need.
The author is Irish, and as an anglo-saxon who has lived in Sweden for the last 17 years, I can relate. We get more systems engineers in the Anglo-saxon world because in part because our education at university tends to be more generalist and also in part because of our more hierarchical organization does not involve everybody in decision making, so people at the bottom let their "mother hen" boss take more responsibility. In strong engineering cultures like Sweden and Germany, we tend to have more specialization and consensus building. There will always be systems engineers, but they are more prevalent in some cultures (anglo-saxon).
The most successful companies I have ever worked for hired smart developers and got out of the way. The least successful company I have worked for spent a huge amount of time in meetings, planning and documenting everything, while failing in the market place against competitors that didn’t. The more structure you impose, the less competitive you are. It is OK if you are a monopoly (NASA) but if not then be careful what you wish for.
Thought it was a post arguing AGAINST Systems Thinkers. Was slighlty provoked and curious to read it, since I’m a fan of systems thinking. Found out it was arguing FOR Systems Thinkers…
consider a slow-growth company free of a title system (e.g. where every engineer is just a "member of engineering"). the less-credentialed nature means you don't have to be as specialized to float ideas. the slow-growth nature allows the company to prioritize process over chasing chaotic payoffs to stay afloat. YMMV and i'm not sure all slow-growth, titleless engineering organizations turn out this way, but there's some correlation.
[+] [-] gorgoiler|3 years ago|reply
This landed so truly for me, it felt like a punch in the stomach.
I wouldn’t dare count the number of times I’ve been told the technical details of why something is the way it is, without anyone ever saying the reason why we actually wanted it to be this way. My thesis was usually: we don’t.
In my career I feel like I have seen hundreds of examples of me saying the systems equivalent of “lets put the dining table indoors?” to be told that the dining table is outside because the original budget meant the front door could only be yay wide so we had to leave the table in the yard and put a tent over it. And I’m just left standing there agape at how we eat in a cold wet tent every night instead of fixing it.
Except it’s usually more like: why do we have to spend $9k on a commercial dishwasher repair contract? Because we have a commercial dishwasher … to get the rust off the silverware … because we eat outdoors every night … because the front door was too small to get the dining table in the house.
Somehow, when the real examples of this stuff are clever engineering around build / docker / polyrepo / release / feature flags / third party bugs, the cleverness makes people think the existence of the workaround should be tolerated. It’s infuriating to join a new team held hostage by years and years of band aids because they never suffer the bigger picture consequences.
The whole article was fantastic. I hope the author has the engineering leadership role they deserve. We need more people like this.
[+] [-] cameronh90|3 years ago|reply
If you spend all your time refactoring, cleaning up after legacy design constraints, fixing ossified errors, then you run out of time and fail to write actual meaningful income generating features. Conversely if you make none of those improvements, eventually the weight of bad architecture slows all progress to a halt and no new income generating features get delivered.
One of the hardest parts about advancing as a developer, in my opinion, is being able to tell when you should refactor versus just leaving the old working mess alone.
With your example, it's like if it turns out that the dining table is embedded into the ground with concrete because it kept blowing over, and moving it indoors would require getting a carpenter to create new legs. And also that because the dining table has been there for so long, someone decided to run electricity cables through it, so rerouting it requires an electrician and will shut down the factory at the bottom of the garden for half a day. We could buy a separate table for indoors, to try and slowly migrate to the new table, but then we'd have two tables to maintain and we all know how that usually goes.
At a certain point, you look at it and go well the commercial dishwasher is just $9k and we can focus our efforts on building that loft conversion for now.
[+] [-] tasuki|3 years ago|reply
I have, too. And then I usually haven't managed to put the dining table indoors. And then new people came in and asked the same question you ask, and by then I was one of the people who tried to put the dining table indoors, and explained how it wouldn't fit through the front door, and how I tried to get it in through the window. And then the new people try to put the table indoors and fail and next thing you see they're either leaving the house or explaining to the newcomers why the table is outdoors.
Ultimately, I've realized that talk like this is cheap, unless you can actually improve things. That requires leadership skills and some political capital in your organization. I don't think the author of the article deserves an engineering leadership role simply for complaining about things. (They might still deserve an engineering leadership role for other reasons, what do I know...)
[+] [-] photochemsyn|3 years ago|reply
"The Boeing 737 MAX: Lessons for Engineering Ethics (2020)"
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7351545/
It's an example of a workaround that should not have been tolerated:
> "The Maneuvering Characteristics Augmentation System (MCAS) software was intended to compensate for changes in the size and placement of the engines on the MAX as compared to prior versions of the 737."
Rather shockingly this wasn't even an engineering problem workaround; it does seem that it was solely designed to avoid an aeronautical reclassification of the aircraft that would have required pilots to undergo an expensive retraining program on flight simulators, which might have caused lost orders.
This does look like a systems-level failure, but one at an organizatonal level: the system went from a state where engineering took priority, to a state where financialization took priority. In systems thinking, this could be called a state transition: a fluctuation takes place, and afterwards the system settles down to a new (apparently) stable state quite different from the old state:
> "One factor in Boeing’s apparent reluctance to heed such warnings may be attributed to the seeming transformation of the company’s engineering and safety culture over time to a finance orientation beginning with Boeing’s merger with McDonnell–Douglas in 1997 (Tkacik 2019; Useem 2019). Critical changes after the merger included replacing many in Boeing’s top management, historically engineers, with business executives from McDonnell–Douglas and moving the corporate headquarters to Chicago, while leaving the engineering staff in Seattle (Useem 2019). According to Tkacik (2019), the new management even went so far as “maligning and marginalizing engineers as a class”."
[+] [-] Aeolun|3 years ago|reply
What’s even more infuriating is seeing new engineers join that team, question why the hell something insane is insane, and then slowly grow used to the insane thing. Only for the cycle to repeat when the next new person joins.
I make a huge point of confirming to them, that yes we do acknowledge it’s insane. Even if management doesn’t care.
[+] [-] codeflo|3 years ago|reply
Oh wow, that hits home. To be fair, the historical context for the decision can be valuable information, the problem is the next step. Even if you can't fix it right now, you might make steps towards that. Or you might say: Now that we have this heavy table outside, why not attach more things to it?
[+] [-] zzzeek|3 years ago|reply
obviously this is more or less possible given specific organizations and personalities at said organizations. my point is more that the magnitude of the task of migrating the ossified organization to a better architecture, even with a fully pliant staff totally on board with changing, should not be underestimated.
[+] [-] infogulch|3 years ago|reply
[+] [-] earth_walker|3 years ago|reply
If it were up to hindsight me, we'd have carefully designed and orchestrated every past decision. We'd do full design qualification and change control on projects and purchases, researching carefully to never make a mistake. We'd sit as a team and brainstorm every possible implication. We'd write, execute and document tests for everything to prove ourselves. Any sniff of inefficiency and we would stop everything and fix it, no matter the cost. We'd take time to document, investigate and follow through. It would be a glorious cavalcade of plans and CAPAs, qualifications and tests. And reports! Binders and binders of wonderful validation reports everywhere!
If it were entirely up to hindsight me, we'd run our little widget company like we're building a space shuttle. Of course, we'd never make any money. But we'd be doing it right, by gum!
Most companies need to be somewhere in the middle. No hindsight and you end up a tangled mess of short-sighted kludges, all hindsight and you can't move forward. Either way you risk ending up a lead balloon.
If you find yourself in the kludge company territory, then here's some advice from a talk I recently attended[1]:
Start by training yourself and your team in Root Cause Analysis. Empower them to start thinking deeply and critically about what's really causing the problems and inefficiencies you encounter. Understanding root causes naturally translate into solutions that aren't just bandaids. Use these skills in your day to day, and you'll start building a culture of quality around your systems.
[1] Steve Gompertz
[+] [-] ctvo|3 years ago|reply
In my experience, at the time the decision was made, folks did want it that way. The organization has lost that context as to why and has only documented the technical design.
A curse shared by less effective engineers I've worked with is to rage at legacy decisions unable to convince the organization to revise them. They lack the ability to understand the various stakeholders involved and to come up with a plausible plan. A systems engineer (as referenced in the blog) would understand the various sub-systems that make up an organization and be able to drive the change they desired (the conclusion that it's irreparably broken or you lack the expertise to fix it would be fine too).
[+] [-] darkerside|3 years ago|reply
Love your post, though!
[+] [-] wglb|3 years ago|reply
I'm reminded of back when I was studying pattern recognition for a system that would become an Expert System (this was before that term was used). I would read many articles saying what techniques would work. I had the urge to ask "But this doesn't show how you got there. Show me your discarded solutions that didn't work." I would like to see your wastebasket.
Similarly, I am inclined to someone who acts as an expert to tell me five ways that won't work.
[+] [-] ryanSrich|3 years ago|reply
The only logical reason to do this is because it has no impact on the business. Or at least, smaller impact than a total rewrite/refactor would have.
If an engineer presented a case where fixing an underlying issue resulted in better business outcomes vs. a short term band-aid, then I don’t know anyone that would tell them no. Businesses want to succeed. They want to make money. If you can help me make more money I’ll let you do whatever you want (within the confines of the law and civil society).
[+] [-] headsoup|3 years ago|reply
[+] [-] zubspace|3 years ago|reply
I think, it's liberating to work in such a company as an experienced developer. You get difficult tasks sometimes, where you need to be the System Engineer and understand the system fully. But you're free to approach the problem in any way you see fit. This requires a lot of trust from the company and your team, but I believe that's a good thing.
It's also a nice for customers, who sometimes have crazy requirements, but they still get results in a reasonable amount of time.
On the other hand, this approach completely destroys newcomers. It's nearly impossible for them to approach a System which they can't fully grasp and is in constant flux. We mentor them, give them easy and introductory task, review their code, but still... It takes a massive amount of time. And I think, it's one of the reasons a company like this has a hard time growing.
And there's the risk of the 'not so competent programmer', who knows how to fix stuff, without thinking far enough.
I don't know, if I'm a System Engineer, but I think it aligns with the description in the article. However, I fear the day, we all agree to overhaul our processes, after which we need to plan and document and review everything.
It seems to be a necessary evil for a company to grow, but at the same time we would also destroy a lot of the liberty we currently have and I don't know which side is worse.
[+] [-] bostik|3 years ago|reply
I would actually claim that majority of non-seasoned programmers fall into this category.
It's rare to see an engineer who would actively set out to destroy or cripple the system they are working on - but it is incredibly common to see one fix the problem they have at hand, using what happens to be available, and make the overall system just a tiny bit less pleasant to work with. Or understand. Let alone maintain.
Repeat the above a hundred times and your codebase would make Lovecraft take note. If each such modification reduces the quality just 0.5%, after 100 rounds you're looking at an aggregate damage of 40%.
[+] [-] necrobrit|3 years ago|reply
Yep, the needs of a company change when and if it grows. Early days, a small team "getting shit done" is probably what you want to find your place in the market, make early customers happy, etc. But as you scale on all axes: time, headcount, headcount turnover, customer count, feature count and so on, that early pile of shit that got done can start to bog you down. Even worse if no one recognises the need to change and continues adding to the pile.
I've seen a few really brilliant engineers struggle with this in various ways: 1. Start hating their job but being confused because they love the company -- not realising that the company has become a very different beast to the early days. 2. Struggling to get out of the "get shit done" mentality and just floundering. Even becoming a net negative contributor in some cases.
Worth remembering this I think in an industry where rapidly changing companies are so common. Your company might be a good fit _now_ but that does not mean it will stay that way!
[+] [-] zmgsabst|3 years ago|reply
That hasn’t been my experience (in general; had some bad companies for sure).
[+] [-] nine_k|3 years ago|reply
[+] [-] thyrsus|3 years ago|reply
[0] technical debt is when you have a plan to pay it down. A technical naked short option is when your plan is to hope the subsystem goes away before its deficiencies have catastrophic effects.
[+] [-] jseban|3 years ago|reply
You could run a super high valued company on just a handful of people, and society is just not ready for that. It's a case of technology advancing faster than human organisation and economic system, and the multiples are just too large, it's a big challenge for society to handle the impact.
So it's better for the organisations to be less efficient and chaotic, and to use dumbed down tools, overcomplicated solutions etc. This will fill the void created by the advancement in technology, so that the organisation can continue to function by at least resembling the traditional model.
You would need to significantly shorten the working hours in order to enable more efficient companies. Otherwise you'd get even more hyper concentration of wealth in a very short time.
[+] [-] Aeolun|3 years ago|reply
It’s just that such an organisation is unpleasant to work for.
[+] [-] redbar0n|3 years ago|reply
What does «called» mean in this analogy? That the subsystem doesn’t go away?
So your experience is that: Stuff tends to stick around? That’s my experience. I have never encountered a throw-away MVP that was actually thrown away if the product continued in the same direction (and didn’t pivot).
[+] [-] andreskytt|3 years ago|reply
[+] [-] nonrandomstring|3 years ago|reply
"Systems" are a mixed blessing, but system thinking is almost always good and useful. We can add value but not bask in it. As a someone invested in the _idea_ here are a few of my favourite quotes that illustrate:
- People don't like systems. Especially new ones.
- Systems ossify and become the problem themselves.
- The ideal system exists only in the mind of its designer.
- The ideal systems designer is invisible and can never take credit.
Overall, I think the thing is that systems are brilliant, until you try to actually build them and encounter _people_, who have other ideas. Neither the force of the better argument, nor punishment, reward, bribery, or flattery will move things. This is neither the fault of systems thinkers nor people but the misunderstanding that (outside the immediacy of war) systems can be imposed. Working systems evolve and are, if the individuals are mentally healthy and motivated by good attitude, generally such that people are doing the thing they would naturally be doing anyway were a formal system not there.A good system is like cat that falls off a tall building and by luck lands on its feet in a box of wool, and licks itself as if to say - sure I meant to do that.
[+] [-] samizdis|3 years ago|reply
I don't, but that quote was used in a comment [1] about five or six weeks ago, and the commenter's relevant bit was:
Nietzche said it best:
I mistrust all systematizers and avoid them. The will to a system is a lack of integrity.
Or maybe (I think Sidgwick):
The English system is "No system", Which is also a system, only better.
https://news.ycombinator.com/item?id=30598863
[Edit to add: I've been dumb there. You were that commenter, so this won't have helped - sorry.]
[+] [-] hef19898|3 years ago|reply
[+] [-] mhitza|3 years ago|reply
[+] [-] catlifeonmars|3 years ago|reply
The problem here is not including people in your “system”. System design needs to be holistic in order to be effective.
[+] [-] giantg2|3 years ago|reply
It's only useful and adds value if your idea actually gets used. Otherwise, it's still pretty crushing.
[+] [-] tasuki|3 years ago|reply
I like how this can be read in two ways and both make sense.
People don't like new systems. But also: New people often don't like (the pre-existing) systems.
[+] [-] joeman1000|3 years ago|reply
Overall, I think software and ‘physical’ engineers should swap experience. Physical engineering could use a tech-injection, and software could use a ‘structure’-injection.
[+] [-] a_c|3 years ago|reply
"Give yourself permission to let the organisation fail".
I agree. As someone is similar situation, the job is to let the "design of system" be heard, be debated and be implemented once green-light. You cannot convince the decision making body (be it the CEO, management, or a design committee) that your idea is the right one for 3 reasons IMO
- idea could be wrong
- People won't know what you are talking about unless they had the first hand experience. (A la you don't know what it is like to be a bat)
- Your meritocracy is limited to a small group of decision making body
If you see the "system" broad enough, you will see a market. It is essentially preaching your "system" to a wider audience. Your system maybe wrong, for which your start up will die or you devise another system. Or you are reaching whole bunch of people who understands your (the initial niche), and your living does not depend on a single decision body but a market.
Convincing one body is hard, but broadcasting whatever you believe, some will respond eventually. This is why start up is great.
[+] [-] dusklight|3 years ago|reply
Anyone know any good books covering this history?
[+] [-] dabiged|3 years ago|reply
I initially watched the lectures because of an interest in aerospace and it is a fascinating historical series in its own right with some incredible speakers. The lessons for systems engineers are numerous too.
[+] [-] giantg2|3 years ago|reply
I mean, the system I started working on 2 years ago was using a business text field for processing decisions. That caused changes to the system when the business wants to make changes. If they do make changes, the reporting queries have to be modified to look for all the historical versions of that text for audit reasons. If the team asking that change forgets to tell one of the numerous other systems that also uses that field, then there are errors. I proposed we add a field with a code that represents this field so that the business can change the display text without affecting the systems that currently use it. It's been at least 18 months, and nothing has changed. You would think that this is a basic design best practice that should have been implemented from the beginning...
[+] [-] jarl-ragnar|3 years ago|reply
Unfortunately our main contract was developing a component of a larger system being developed by our parent organisation who didn’t have a concept of systems engineering. We tried for years to educate them and I watched aghast as their program costs and schedule continued to spiral out of control.
Basically a bomb burst of engineers all doing what they thought was the right thing but no one owning the system design and saying no to good ideas.
In the words of their chief engineer “its like 10 different black boxes, I don’t know what I’m getting and I don’t know when it’ll be finished!”
[+] [-] smcnally|3 years ago|reply
> we define a stable interface > reproduce behaviour reliably > Predictability is great
Within being believed and valued for protecting engineers and organizations, so much progress applying these principles relies on individuals’ and teams’ readiness to adopt and advocate for them. Hope to see your experience with that in part II.
[+] [-] Josteniok|3 years ago|reply
I've spent my entire career working for a medium sized organization and in the past few years it has tried to become "agile". Most of this push is predicated on the idea that we will be able to go faster this way. As a result we have deconstructed ourselves. What's weird is that now we aren't actually even "faster to say we are finished." No, now we are only faster to say that we are going to be faster to be finished. We still end up taking a long time but as long as the slide deck says we are going to be done in a short amount of time, then all is believed to be running smoothly. It's very strange/depressing/unsettling.
[+] [-] hasmanean|3 years ago|reply
Most systems are described in a simple way: box and line charts which are just voodoo.
We need to develop a algebraic notation for how to represent the various configurations/states of an organization, along with its operations.
With a notation, you can describe what you “feel” and share the knowledge and improve on it. You can also define the organizations desired state and track your progress. You can plop a new manager in it and they will know what to do. You can also do basic engineering like pick a configuration that has the desired cost, throughout and latency that you need.
[+] [-] Terretta|3 years ago|reply
[+] [-] jamesblonde|3 years ago|reply
[+] [-] mbrodersen|3 years ago|reply
[+] [-] redbar0n|3 years ago|reply
[+] [-] voiper1|3 years ago|reply
[+] [-] langsoul-com|3 years ago|reply
Like the trade-off between creativity and efficiency
[+] [-] wallacoloo|3 years ago|reply
[+] [-] contingencies|3 years ago|reply
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] ctvo|3 years ago|reply