I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written.
Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet after only writing one or two integrations on the new platform. Nobody over-engineered here. Everything was built to spec. But three years down the line the new dev is going to get assigned a bug in one of the integrations and lament how the code was over-engineered. Hindsight is 20/20.
Lots of stories from the trenches including many in this thread make this mistake. The same goes for 'tech debt'. If you weren't there, you don't know.
One thing that bugs me is the notion that "Software rewrites are something you should never do", which is a mantra so often repeated that it has acquired the status of self-evident truth, despite the only evidence being (usually) presented is an example of a web browser from 20 years ago! (Which incidentally spawned Mozilla, so not exactly a complete loss; especially from the POV of society rather than shareholders, but I digress).
Having rewritten a bunch of systems (sometimes several times) I can attest that it will not always lead to the death of the company. The trick is of course having modular enough systems that they can be rewritten from scratch in a reasonable amount of time.
It can also be a great way to increase the simplicity of the system as typically the old version was designed with a very imperfect understanding of the problem and no operational experience servicing it; further learning were usually crudely patched on top and you often end up in a conceptual hodge-podge where words mean subtly different things depending on the context and translation layers need to be inserted between the contexts etc.
Often a (good) rewrite starts by clarifying the conceptual model. I like the saying "clear writing reflects clear thinking", and in programming there is a lemma "clear thinking produces clear code".
I suspect that the main issue with rewrites is that the users or product managers see it as an oportunity to add new features or redo old ones extensively. In the end the scope of the rewrite is no longer a rewrite but a new product that is incompatible with the original it was supposed to replace. I have seen this happen a couple of times. A straight rewrite for technical reasons and well defined scope does not suffer these issues.
Old developers have left. New ones come in. The systems the old devs built suck, so the new team convinces management to do a rewrite. The new devs don’t really understand the problems encountered by the old ones so they repeat all the same mistakes. New system, same problems.
The lesson we’re supposed to take away “no one should ever do full rewrites”. It’s a stretch, but IMO the proper takeaway should be 1) really understand the old system and 2) have a very good reason before doing full rewrites.
The way I've heard it is is "Evolution, not revolution".
You evolve your code with refactoring and rewriting only pieces at a time. This is opposed to revolution, also known as "the big rewrite", which replaces the entire application all at once.
Your "modular enough systems" seems to indicate that you also favor evolution over revolution.
> despite the only evidence being (usually) presented is an example of a web browser from 20 years ago!
Actually normally the evidence is lots of other companies that failed to do rewrites. It's just that one was a full scale fuck up. I'm currently working at a company that literally it's echoing Netscape. The issue wasn't the rewrite it was rushing a half finished rewrite out the door. It was stopping product development for so long.
My current employer started a rewrite but called it a migration gave it a 3 month deadline. 3-months to write all the features it took 7 years to write. They realised this was impossible and remove a bunch of features and decided this rewrite would remove features they will add back later. But they still kept on setting months for something that has taken 18 months so far with even more features removed. It almost a daily thing that yet another product feature was removed to cut down time. They claimed they were feature complete in september because it had to be done under all circumstances, they found out they hadn't done 50% of the features they said they would. So with more rushing of the features they hadn't written they then started talked about releasing it before it had passed QA. They announced the release date before it had passed QA. We have partners using it and saying it's broken for them. They don't have all the data. And yet they're still releasing it on Monday. Why? Because it had to be done in 2021. They're rushing a half finished rewrite out the door to hit a target set by management. So they spent 18-months removing features and when they release it, it will be buggy.
So, yea I mention Netscape a lot because honestly, this sounds the same. Rush out a half done rewrite while allowing the competitors to improve their product while we made ours worse.
> Having rewritten a bunch of systems (sometimes several times) I can attest that it will not always lead to the death of the company. The trick is of course having modular enough systems that they can be rewritten from scratch in a reasonable amount of time.
I would say that the trick is not to do the rewrite. You refactor each part until the entire system is rewritten.
Having dealt with a similar setup that basically had to be re-written, I totally agree. I would also like to add that if one senses themselves to be in a situation like this where the system is a messy build-up over years, try to resist adding things that are absolutely not essential. Otherwise, the guy doing the clean-up/re-write down the line may be forced to take not-so-clean approaches to cater to those non-essential bits mainly for backwards compatibility.
Rewrites are also sometimes very good for your career. A friend working at Google for about ~1.5 years said a new Sr. Director of Engineering joined and their first decision was to rewrite the project, and it would take 2 years.
So now the Director has job security for about 2 years, gets the big launch near a promotion cycle they have a small chance of being considered for promo, and gets to blame the predecessor for all the problems.
Complexity kills your product - Overengineering is just one instance of complexity - Technical Debt like having state and data all over the place is another one - I quit my last job and I happily blame this article for convincing me to quit: https://itnext.io/the-origin-of-complexity-8ecb39130fc - coordination causes complexity and this killed me - we had everything not once but twice or more in different places - just one example - it's much more complex but me and my colleagues were grinded between an old codebase that nobody understood anymore and kubernetes, ci, etc.pp on the other side because management sold this without understanding that you need a process and time for digesting and applying the concepts in the team.
This sounds exactly like the last place where I worked. I quit in anger Not because I don't like Kubernetes but because management sold it as the fixall solution.
This is like saying that water is lethal - both are true in the sense that some x (water, complexity) is ok, but too much is bad.
Complexity is an unfortunate (but necessary) side-effect of (1) adding features and (2) optimizing for performance, both of which are critical for building a product.
If you try to remove all complexity from your product, you won't have a product. Instead, you have to try to minimize complexity while still delivering the same set of features and level of performance.
I interpret “early over engineering” not as adding complexity necessarily. It could also be about decreasing complexity but by taking too much time. Early on you’re supposed to rush to your MVP and add tech debt, not spend too much time to design something pretty and modular. Later on once you know your company can survive multiple quarters, then you can spend more time refactoring.
Six months ago I left a company that was working on an overengineered product. Even worse than it being overengineered was that it was also under documented. Working on anything was a pain, because the CTO wanted everything to follow his well thought out, and frankly very cleverly engineered design patterns, but he couldn't clearly communicate what those patterns were. And the entire company amounted to transforming and cleaning data sets using in-house tools, which could easily be done with existing tools too. Both myself and the other senior engineer on the team left at the same time. I felt bad leaving them, because they were trying to grow and had a ton of funding and deals with FANG companies but they were struggling to find engineers that the CTO thought were smart enough. I didn't want to burn bridges, so I didn't end up telling them that the problem wasn't a lack of qualified engineers, it was an over-engineering CTO who struggled to communicate.
> they were struggling to find engineers that the CTO thought were smart enough.
If you have to find the smartest people to keep the wheels on, you’ve already lost.
Disdain for the bell curve is the fastest way to get overengineering. Very few things have to be rocket science to create a good company. Everything else should be dead simple so that anyone can work on it. That also means you have to compartmentalize the hard bits so they don’t leak across the entire codebase.
But some people get bored with mundane code and will make things interesting by replacing a boring task with an exciting one. It’s part of the Curse of Knowledge. You can’t run a company like an action movie. Everyone gets burnt out.
Being ins a similar role myself, how do I ensure that engineers stay happy working on the project that we're working on? I'm finding myself actually doing the opposite of the CTO you mentioned and pushing them towards adopting more off-the-shelf components instead of maintaining homegrown stuff but I think I'm causing a degree of upheaval by doing this. Their justifications for push back however, often smell of sunken cost fallacy to me.
Same been in that role and in a similar role at the moment. It's hard but this is the industry I can't change that anymore I just have to work with these people long enough so I can retire.
The most important thing is to find the right balance. This article goes into one direction. But to be honest, most of the time I see it shift into the other direction: in product driven orgs the drive to implement features and bring them to market quickly is more important than engineering quality. But in the end you end up with something where implementing new features takes so much time because of the complexity that built up because you started to implement a product where the specs where unknown in the beginning and only materialized later. That's the point when you should re-engineer your system. But yeah... it's all about balancing and finding the sweet spot.
I think there's a very specific form of overengineering afflicting products currently, which I'd classify as "endless revisiting". This is where companies build something which works well enough, but then get trapped in a cycle of endlessly tweaking that one thing. Inevitably the amount of code churn in this one area combined with the need for some semblance of backward compatibility results in something that is fragile, complex and slow.
Plus the annoyance as a user that whenever you use this thing, all your tools are in a slightly different place and work slightly differently to how you left them. IMO there's a need for better balance between "it works well enough, leave it alone" and "we haven't fully optimised this workflow yet" in product development.
The issue here isn't so much over or under-engineering, but rather "Are we building the right thing?" or "Are we building the thing right?"
In a startup you don't know if you're building the right thing so trying to build it right is premature optimization. Since you have limited resources you really have to focus on ensuring that you're building the right thing...if you aren't it doesn't matter how well designed or built it is no one is going to use it.
Once you've validated you're building the right thing you can start focusing on building it right, but by they you probably know where the pain points are and where you need to spend the effort.
The trick with this though is that everyone needs to be aligned on that approach and be honest about the fact that corners may be being cut to get something working fast. Where I've seen this go badly is where a crappy initial version was built to get to market fast, but then no time was made available to address the defencies in the initial release.
The "super simple code" at both ends of the graph aren't equivalent. The latter is more "simplest code possible".
Overly simple, hacky, narrowly spec'd code produces the same tech debt as overengineering. Anybody who's worked in a move-fast-break-things type startup will know how much engineering resources are wasted on rewrites/bugfixes due to this.
Ultimately, as with many things in life, you need to find the right balance.
I think the graph needs to continue a few more years, but the author hasn't lived that yet. The "simplify" mindset is also something you can take too far without experience.
On the other hand, simplifying an already over-engineered product is almost next to impossible simply because many jobs depend on it. Maybe I'm cynical but I'm beginning to think that software complexity grows until it justifies all the head counts in the department.
Yeah it is like this. Even simple sites require a ton of engs thrown at it. On the business side I guess they figure spending more money on tech will make them safe, but it's like of you wanted a bowl of soup and hired ten chefs to do it. But I can't complain since I am one of the soup chefs, and I help make sure we have tests that prevent the soup from becoming too salty
This is the same phenomenon as the Peter Principle, where people Rose to the level of their incompetence. It sounds like a joke at first, but, of course they do. People get promoted when they excel. When you are in over your head, you stop getting promoted. Of course, over time, people grow into their roles, and regain their competence.
Devs will build software until they can no longer do so because the codebase is larger than their collective abilities to manage.
I'm wondering about this now I guess it is old fashioned now to use environment variables and bare EC2 servers, managing your own APIs and websockets/DB on same server as opposed to breaking everything out. You need to use cloud formation and "oh did you know there is an AWS service for that?" Then you are using 5 services instead of 2. This twelve factor app concept. Don't know when is the right time to do this/at what scale.
My last startup was spending stupid money on AWS services and horizontal scaling for an app which had, when I left, about 100 users and maybe 2 concurrent at a given time. And they had been doing this since before I joined when the numbers were much lower. The complexity was idiotic and no one but the devops guy who set it up could grok it. We still managed to have frequent downtime
my two cents : until you hit scaling wall (and when you will congratulations, you are either successfull or a video streaming platform), a big server to upgrade is the best way to go and focus on building product.
Then you hit performance problems, any sysadmin could help you handle 2x/4x/10x scaling with simple separations of service and maybe some hours of downtime. In the meantime you probably have weeks/months to think about going really crazy with your infrastructure.
As others have said, it is unecessary complexity. Over-engineering is an ambiguous term.
For example, premature optimisation is frequently mentioned as over-engineering but is it premature optimisation to use a map/dictionary instead of an array for key-based access? Nope. That is just correct. Is it over-engineering to know that if your product succeeds, you could end up with X-hundred objects which will use up all the RAM you are storing them in and therefore you might want to make them smaller/store them off-board? Of course, you can come back and refactor later but it is so much easier for the person who writes that code to understand what can be done on day 1 rather than assuming it is cheaper to refactor later only when needed.
I think a better take is for people to accept that no app lasts forever. If we build that assumption into our worldview, it will help us make better decisions. I still think some engineers think there is some perfection that means the app will live forever.
> use a map/dictionary instead of an array for key-based access? Nope. That is just correct.
Maybe, maybe not. While the map/dictionary is easier to use, if the number of items is small (which it often is) the array will be substantially faster because the CPU will pre-fetch the next element into the cache while you are still checking if the current one is the right key. Maybe - check with a cache aware profiler, and you need to design your array to be cache friendly.
Of course the above assumes your map/dictionary is on the hot path (seems unlikely), your code is too slow (probably subjectively, but in some real time applications this can be objectively measured), and you are writing in an efficient language. IF All of the above are true, only then should you go to the complexity of using an array to do the job of a map/dictionary.
I've written a lot of complex stuff. In fact, I'm doing it right now.
There needs to be a "sweet spot," where we have enough complexity to achieve goals (which can be business and cultural, as well as technical), and as much simplicity as possible.
A lot of folks think that using dependencies makes the project "simpler."
It does, for the final implementation team, but that complexity still lives there; along with a whole bunch of cruft, tech debt, potential legal issues, security issues, etc.
Unfortunately, you can't do complex stuff, simply. Some level of complexity is always required. Managing that complexity is where we get to play adults.
Right, using too dependencies don't make things simpler. They make them easier as long as the dependencies can do exactly what you need. However sometimes you come to a point where the dependency can't do something that's needed (often by the time I realized a third party lib can't do sth, I could have implemented it by myself without the dependency) or where you've accumulated so many dependencies that they conflict with each other or where connecting two different libraries is the really hard part. No silver bullets.
Figuring out the sweet spot in architectural design seems to me still an art/craft, that comes from intuition and by experience.
[As far as dependencies... using dependencies may sometimes be simpler than the alternative(s), and sometimes not, sure. It depends on the nature of the dependencies, the problem, and the alternatives. :) ]
Complexity is also one of the best moats in the business, if you have a "gross" problem with many edge cases there's few people who will want to eat your lunch, and most next generation up and comers often tout their simplicity compared to your old complex product because they certainly cant argue feature parity ;)
> It does, for the final implementation team, but that complexity still lives there; along with a whole bunch of cruft, tech debt, potential legal issues, security issues, etc.
I call this squeezing the balloon, the complexity doesn't go anywhere, it's inherent to the requirements of the system. You just put it somewhere else.
This is just code factoring at a macro(-ish) level.
For well maintained deps there is the extra boon that it takes work off your hands though. For instance building a React app with Next.js saves you from ever having to deal with webpack and you get big upgrades for free.
1. Without good development practices due to lack of Experience or plain old Rational Thought the only possible outcome is Operational Deficiency.
Every Best Practice is context-dependent, thus having Deficient Resources makes them inapplicable in certain cases. Some teams can really suck with the same tech stack when others flourishing using it.
2. Basic organizational anti-patterns, like Mushroom Management, and broken retrospective lead to rediculous outcomes.
Even plain old Micro-services and Micro-frontends can be a basis of Stovepiping and applying Mushroom Management. Usually, again, due to Lack of Competence and Sheer Hubris.
3. “Premature Optimization” only used in context of over-engineering by those who didn’t read the book, but use Halo-effect cognitive bias to project compensated qualities onto the term itself. There are a lot of Psychological Compensational Needs under the hood.
It’s like “Why Agile has nothing to do with Discipline ?” or “Why senior developers turning the project into a sandbox due to the lack of self-fulfillment ?” or “Why most of the MVP’s lack Concise and Validated Definition of Viability ?”
Complex doesn’t mean Hard or Expensive. Simple doesn’t mean Easy or Cheap.
Too often “over-engineering” is just an organizational and psychological issue and not an Engineering one.
Stop operating on Feelings.
Six Sense of the Fifth Body Anchor Point is not a reliable Key Performance Indicator.
"Stop overengineering" was the excuse I heard back on a rough project when I insisted we add logging beyond just the returned HTTP status code to our service before shipping to 200M+ people. But that would take an extra week or two in the current system and that's just too much investment for some silly over engineering.
Had we gotten decent logging in place early, we could have saved a terrible change that got passed our canary rings, which got us called to see the CEO and made news.
So, I learned early that "overengineering" can also be a management excuse for cutting corners that shouldn't be cut.
Overengineering is typically a term thrown around when a manager wants to ship a system in half the time it actually would take to make a half-decent system!
The biggest consequence for me was not mentioned: that your carefully planned design very soon becomes an obstacle to something a user actually wants done, at which point you say "we can't do that". Which is one of the worst things you can do. In this sense almost all of the systems I interact with, modify etc. are over engineered.
Interestingly, dang (HN mod) once mentioned in a feature request comment that he has a mental model of a complexity budget. This very closely fits my own perception: You can only add so much complexity to a system before it starts to break down under the load. Estimating that budget, and how much new features will consume, makes a good engineer in my opinion.
Another problem is that these overly complex systems are often very fragile when something changes. In a very theoretical situation there could be a case where someone with an overengineered client consumes your JSON API and their client breaks when you add a field to a certain service response. Something that should have been no problem suddenly causes total breakage of your software and then you'll have to alter that very complex piece of software. Something that could've been prevented if you kept things simple and robust and simple chose to ignore extra fields or headers you weren't using anyway.
That's indeed the irony. The over-engineering happened in an attempt to be resilient to future change, but the outcome can often be the opposite. We've all been there, I think?
I've had this exact ticket in the past: A customer built a complex Java client for our public API which assumed no surplus fields in the responses, and as we added a new field, their entire application broke down, causing huge losses for the customer. I wasted so much time on explaining how our stability promise does not extend to added fields!
Oh boy, does this ring a bell with me. I've already written 5575LOC according to cloc (and threw away 7449LOC in the process) and I'm far from finished writing the code to email the data from a contact form in PHP. But it ticks all the boxes!
It's all OOP
SOLID principles
99,6713% typed (according to psalm)
Purposely written to be unit-testable in PHPUnit strict mode (but no actual harness yet)
I suspect under-producting has killed far more products than over-engineering. The ratio of decision making power to decision making abilities is way out of balance for most Product Managers. Even at the big tech companies I feel most PMs are unimaginative MBA types that can optimize but not innovate and have no grasp of the concept of opportunity cost.
In terms of power structures, Product Managers decisions largely go unchecked in a lot of places. Engineering decisions face significantly more scrutiny, especially in places that work in short sprint cycles.
From a UI/usability perspective, one of the dangers with overengineering is the "wall of options" issue, where all users - even the majority of them that just need something simple - still need to read and understand all these advanced options. As a product manager and UI designer you have a couple ways to deal with this. You can choose an opinionated subset of features that make sense for a given niche and target only that demographic. You can go the corporate way, keep the wall of options and just require training for users. Or you can try the balancing act - choose a sane subset of features as the default, and hide the more advanced options, so they don't bother normal users, but still have them as possible options. There are many important choices regarding how much to hide, and where, and how to make it discoverable, and how to make it possible to gradually dig deeper, and for users to self-identify as someone that needs to dig deeper - and those details are often as much art as science. But get it right, and you've got one of those rare killer apps that both newbies and experienced users enjoy.
[+] [-] somehnacct3757|4 years ago|reply
Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet after only writing one or two integrations on the new platform. Nobody over-engineered here. Everything was built to spec. But three years down the line the new dev is going to get assigned a bug in one of the integrations and lament how the code was over-engineered. Hindsight is 20/20.
Lots of stories from the trenches including many in this thread make this mistake. The same goes for 'tech debt'. If you weren't there, you don't know.
[+] [-] gampleman|4 years ago|reply
Having rewritten a bunch of systems (sometimes several times) I can attest that it will not always lead to the death of the company. The trick is of course having modular enough systems that they can be rewritten from scratch in a reasonable amount of time.
It can also be a great way to increase the simplicity of the system as typically the old version was designed with a very imperfect understanding of the problem and no operational experience servicing it; further learning were usually crudely patched on top and you often end up in a conceptual hodge-podge where words mean subtly different things depending on the context and translation layers need to be inserted between the contexts etc.
Often a (good) rewrite starts by clarifying the conceptual model. I like the saying "clear writing reflects clear thinking", and in programming there is a lemma "clear thinking produces clear code".
[+] [-] cfn|4 years ago|reply
[+] [-] aranchelk|4 years ago|reply
Old developers have left. New ones come in. The systems the old devs built suck, so the new team convinces management to do a rewrite. The new devs don’t really understand the problems encountered by the old ones so they repeat all the same mistakes. New system, same problems.
The lesson we’re supposed to take away “no one should ever do full rewrites”. It’s a stretch, but IMO the proper takeaway should be 1) really understand the old system and 2) have a very good reason before doing full rewrites.
[+] [-] justin_oaks|4 years ago|reply
You evolve your code with refactoring and rewriting only pieces at a time. This is opposed to revolution, also known as "the big rewrite", which replaces the entire application all at once.
Your "modular enough systems" seems to indicate that you also favor evolution over revolution.
[+] [-] that_guy_iain|4 years ago|reply
Actually normally the evidence is lots of other companies that failed to do rewrites. It's just that one was a full scale fuck up. I'm currently working at a company that literally it's echoing Netscape. The issue wasn't the rewrite it was rushing a half finished rewrite out the door. It was stopping product development for so long.
My current employer started a rewrite but called it a migration gave it a 3 month deadline. 3-months to write all the features it took 7 years to write. They realised this was impossible and remove a bunch of features and decided this rewrite would remove features they will add back later. But they still kept on setting months for something that has taken 18 months so far with even more features removed. It almost a daily thing that yet another product feature was removed to cut down time. They claimed they were feature complete in september because it had to be done under all circumstances, they found out they hadn't done 50% of the features they said they would. So with more rushing of the features they hadn't written they then started talked about releasing it before it had passed QA. They announced the release date before it had passed QA. We have partners using it and saying it's broken for them. They don't have all the data. And yet they're still releasing it on Monday. Why? Because it had to be done in 2021. They're rushing a half finished rewrite out the door to hit a target set by management. So they spent 18-months removing features and when they release it, it will be buggy.
So, yea I mention Netscape a lot because honestly, this sounds the same. Rush out a half done rewrite while allowing the competitors to improve their product while we made ours worse.
> Having rewritten a bunch of systems (sometimes several times) I can attest that it will not always lead to the death of the company. The trick is of course having modular enough systems that they can be rewritten from scratch in a reasonable amount of time.
I would say that the trick is not to do the rewrite. You refactor each part until the entire system is rewritten.
[+] [-] noisy_boy|4 years ago|reply
[+] [-] inerte|4 years ago|reply
So now the Director has job security for about 2 years, gets the big launch near a promotion cycle they have a small chance of being considered for promo, and gets to blame the predecessor for all the problems.
[+] [-] nisa|4 years ago|reply
[+] [-] oweiler|4 years ago|reply
[+] [-] throw10920|4 years ago|reply
This is like saying that water is lethal - both are true in the sense that some x (water, complexity) is ok, but too much is bad.
Complexity is an unfortunate (but necessary) side-effect of (1) adding features and (2) optimizing for performance, both of which are critical for building a product.
If you try to remove all complexity from your product, you won't have a product. Instead, you have to try to minimize complexity while still delivering the same set of features and level of performance.
[+] [-] baby|4 years ago|reply
[+] [-] jasfi|4 years ago|reply
[+] [-] d0m3|4 years ago|reply
[+] [-] yosito|4 years ago|reply
[+] [-] hinkley|4 years ago|reply
If you have to find the smartest people to keep the wheels on, you’ve already lost.
Disdain for the bell curve is the fastest way to get overengineering. Very few things have to be rocket science to create a good company. Everything else should be dead simple so that anyone can work on it. That also means you have to compartmentalize the hard bits so they don’t leak across the entire codebase.
But some people get bored with mundane code and will make things interesting by replacing a boring task with an exciting one. It’s part of the Curse of Knowledge. You can’t run a company like an action movie. Everyone gets burnt out.
[+] [-] abraxas|4 years ago|reply
[+] [-] Dave3of5|4 years ago|reply
[+] [-] zeitgeist_y2k|4 years ago|reply
[+] [-] MattKimber|4 years ago|reply
Plus the annoyance as a user that whenever you use this thing, all your tools are in a slightly different place and work slightly differently to how you left them. IMO there's a need for better balance between "it works well enough, leave it alone" and "we haven't fully optimised this workflow yet" in product development.
[+] [-] am391|4 years ago|reply
In a startup you don't know if you're building the right thing so trying to build it right is premature optimization. Since you have limited resources you really have to focus on ensuring that you're building the right thing...if you aren't it doesn't matter how well designed or built it is no one is going to use it.
Once you've validated you're building the right thing you can start focusing on building it right, but by they you probably know where the pain points are and where you need to spend the effort.
The trick with this though is that everyone needs to be aligned on that approach and be honest about the fact that corners may be being cut to get something working fast. Where I've seen this go badly is where a crappy initial version was built to get to market fast, but then no time was made available to address the defencies in the initial release.
[+] [-] izolate|4 years ago|reply
Overly simple, hacky, narrowly spec'd code produces the same tech debt as overengineering. Anybody who's worked in a move-fast-break-things type startup will know how much engineering resources are wasted on rewrites/bugfixes due to this.
Ultimately, as with many things in life, you need to find the right balance.
[+] [-] jeremyjh|4 years ago|reply
[+] [-] a_square_peg|4 years ago|reply
[+] [-] darepublic|4 years ago|reply
[+] [-] darkerside|4 years ago|reply
Devs will build software until they can no longer do so because the codebase is larger than their collective abilities to manage.
[+] [-] jcun4128|4 years ago|reply
[+] [-] blindmute|4 years ago|reply
[+] [-] GaelFG|4 years ago|reply
Then you hit performance problems, any sysadmin could help you handle 2x/4x/10x scaling with simple separations of service and maybe some hours of downtime. In the meantime you probably have weeks/months to think about going really crazy with your infrastructure.
[+] [-] jcun4128|4 years ago|reply
[+] [-] lbriner|4 years ago|reply
For example, premature optimisation is frequently mentioned as over-engineering but is it premature optimisation to use a map/dictionary instead of an array for key-based access? Nope. That is just correct. Is it over-engineering to know that if your product succeeds, you could end up with X-hundred objects which will use up all the RAM you are storing them in and therefore you might want to make them smaller/store them off-board? Of course, you can come back and refactor later but it is so much easier for the person who writes that code to understand what can be done on day 1 rather than assuming it is cheaper to refactor later only when needed.
I think a better take is for people to accept that no app lasts forever. If we build that assumption into our worldview, it will help us make better decisions. I still think some engineers think there is some perfection that means the app will live forever.
[+] [-] bluGill|4 years ago|reply
Maybe, maybe not. While the map/dictionary is easier to use, if the number of items is small (which it often is) the array will be substantially faster because the CPU will pre-fetch the next element into the cache while you are still checking if the current one is the right key. Maybe - check with a cache aware profiler, and you need to design your array to be cache friendly.
Of course the above assumes your map/dictionary is on the hot path (seems unlikely), your code is too slow (probably subjectively, but in some real time applications this can be objectively measured), and you are writing in an efficient language. IF All of the above are true, only then should you go to the complexity of using an array to do the job of a map/dictionary.
[+] [-] ChrisMarshallNY|4 years ago|reply
There needs to be a "sweet spot," where we have enough complexity to achieve goals (which can be business and cultural, as well as technical), and as much simplicity as possible.
A lot of folks think that using dependencies makes the project "simpler."
It does, for the final implementation team, but that complexity still lives there; along with a whole bunch of cruft, tech debt, potential legal issues, security issues, etc.
Unfortunately, you can't do complex stuff, simply. Some level of complexity is always required. Managing that complexity is where we get to play adults.
T.A.N.S.T.A.A.F.L.
[+] [-] pkolaczk|4 years ago|reply
[+] [-] jrochkind1|4 years ago|reply
Figuring out the sweet spot in architectural design seems to me still an art/craft, that comes from intuition and by experience.
[As far as dependencies... using dependencies may sometimes be simpler than the alternative(s), and sometimes not, sure. It depends on the nature of the dependencies, the problem, and the alternatives. :) ]
[+] [-] hobs|4 years ago|reply
[+] [-] Griffinsauce|4 years ago|reply
I call this squeezing the balloon, the complexity doesn't go anywhere, it's inherent to the requirements of the system. You just put it somewhere else.
This is just code factoring at a macro(-ish) level.
For well maintained deps there is the extra boon that it takes work off your hands though. For instance building a React app with Next.js saves you from ever having to deal with webpack and you get big upgrades for free.
[+] [-] yuriy-yarosh|4 years ago|reply
Probably missing few core points:
1. Without good development practices due to lack of Experience or plain old Rational Thought the only possible outcome is Operational Deficiency.
Every Best Practice is context-dependent, thus having Deficient Resources makes them inapplicable in certain cases. Some teams can really suck with the same tech stack when others flourishing using it.
2. Basic organizational anti-patterns, like Mushroom Management, and broken retrospective lead to rediculous outcomes.
Even plain old Micro-services and Micro-frontends can be a basis of Stovepiping and applying Mushroom Management. Usually, again, due to Lack of Competence and Sheer Hubris.
3. “Premature Optimization” only used in context of over-engineering by those who didn’t read the book, but use Halo-effect cognitive bias to project compensated qualities onto the term itself. There are a lot of Psychological Compensational Needs under the hood.
It’s like “Why Agile has nothing to do with Discipline ?” or “Why senior developers turning the project into a sandbox due to the lack of self-fulfillment ?” or “Why most of the MVP’s lack Concise and Validated Definition of Viability ?”
Complex doesn’t mean Hard or Expensive. Simple doesn’t mean Easy or Cheap.
Too often “over-engineering” is just an organizational and psychological issue and not an Engineering one.
Stop operating on Feelings. Six Sense of the Fifth Body Anchor Point is not a reliable Key Performance Indicator.
[+] [-] HEmanZ|4 years ago|reply
Had we gotten decent logging in place early, we could have saved a terrible change that got passed our canary rings, which got us called to see the CEO and made news.
So, I learned early that "overengineering" can also be a management excuse for cutting corners that shouldn't be cut.
[+] [-] _drimzy|4 years ago|reply
[+] [-] locallost|4 years ago|reply
[+] [-] 9dev|4 years ago|reply
[+] [-] FriedrichN|4 years ago|reply
[+] [-] jrochkind1|4 years ago|reply
[+] [-] 9dev|4 years ago|reply
I've had this exact ticket in the past: A customer built a complex Java client for our public API which assumed no surplus fields in the responses, and as we added a new field, their entire application broke down, causing huge losses for the customer. I wasted so much time on explaining how our stability promise does not extend to added fields!
[+] [-] pdenton|4 years ago|reply
It's all OOP
SOLID principles
99,6713% typed (according to psalm)
Purposely written to be unit-testable in PHPUnit strict mode (but no actual harness yet)
...I'll show myself out
[+] [-] jamil7|4 years ago|reply
[+] [-] rubiquity|4 years ago|reply
In terms of power structures, Product Managers decisions largely go unchecked in a lot of places. Engineering decisions face significantly more scrutiny, especially in places that work in short sprint cycles.
[+] [-] m12k|4 years ago|reply