Software takes a long time to build because it's always new.
That is, software is trivially copy-able, so there is no reason to spend effort duplicating any software that already exists. (Legal reasons and "not-invented-here" syndrome notwithstanding.) This is a huge different from how the "real" world works, where almost all of the work involved in building, say, a car, is actually just the work of assembling identical copies of that product so that it can be physically sold to more than one person.
The insane amounts of repeated effort involved in building any physical product at scale have a silver lining, in that the repetition enables a very good and stable estimate of how long another repetition will take. Software is the opposite situation --- we only need to build anything once, but not having built that thing before, we don't really know how long it will take.
I think focusing on this comparison to the real world assemblies and constructions doesn't yield much insight. The equivalent of a car being assembled is a compiler building an executable from source. You can consider factories to be just very, very long compiles, and forget about it.
The question of why software takes a long time to build pertains to the coding phase, to which real life's analogue is drafting/design phase. It is my impression that the drafting/design phase in "hard" industries can be just as messy and hard to schedule as software development is - we just don't pay attention, because for physical products, drafting/design is nearly free (capex & opex of the assembly phase dominates), whereas in software, the construction is nearly free (compiler time is too cheap to meter).
It's true that, on the surface, software is "always new". But after doing it for a while, I have this sinking feeling that a lot of it is old and repetitive work that we hadn't yet automated - there are deep similarities between pieces of code we're writing, but we don't have a language to fully capture them and abstract them away.
Best comparison (car/machine manufacturing vs software development) I've ever heard on this topic is as following;
While producing a machine, there are two steps;
1) designing the blueprint (takes a lot of time, unpredictable)
2) mass production (takes a lot of time, predictable)
Those steps in software development;
1) building the software (takes a lot of time, unpredictable)
2) deploying the code (takes virtually no time, predictable)
So the real difference is this: software is almost all about doing something new, each and every time.
It's all flesh and has no bone: e.g. devoid of a long, very stable and predictable mass production stage. It's almost all about R&D, all of it's length. Hence 98% of the process is novel work therefore takes time and hard to predict.
Yes, building cars in a factory yields repetitive results, but what about all the different engineering projects. The truth is that this is not a new problem, we just lack discipline of fields that have been refined over thousands of years.
Because of the quick idea to prototype rate, product research and refinement takes a back seat. All engineering projects have massive research before the first stone is placed. Nobody mid way through building a sky scraper decides "oh, do we need to redo the foundation?"
I would also like to note, nobody is gonna give a self-taught architect who's 19 the ability to build that skyscraper foundation. Each building technique is developed, tested, and then once determined good, is actually used by others, and supervised by an industry veteran with underlings who learn and eventually themselves supervise such projects. Meanwhile in software it is the opposite.
Not to say that there isn't a reason software grows so fast. Things change daily. If software worked like architecture we'd be licensing even the right to use ruby on rails-style modeling. But software works differently, thus faster iteration, thus more loosey-goosey, thus harder to estimate.
And add to that the number of custom software are many times more than most things in real life. Nobody will asks for their BMW to transport a cow, or for their BMW to connect / communicate with other's Mercedes.
But we often asked for accounting app to able to handle employee's payroll and paid leaves.
Then with that many demand for custom software, we also got many times software manufacturer than car manufacturer. Inexperienced manufacturer (programmer) will make development even slower. Not to mention that the client even don't know what they want.
One aspect that is not mentioned is that we build software on top of an ever-increasing number of first-to-market, low-quality, building blocks. And by low-quality I mean "worse is better"/MVP/"everyone makes mistakes"/"leaky abstractions"/etc -- pick your favorite. As a result, we spend more and more time dealing with someone else's mistakes rather than making forward progress.
I dunno, there's a lot of really high quality stable software out there. It's not all crap.
But as someone who has had the luxury of taking time to do things the Right Way let me tell you from firsthand experience: doing things the Right Way is incredibly hard in no small measure because figuring out what you actually want to do is incredibly hard. There have been many times when I thought I was building something for the ages only to discover that I had made a bad assumption, or technology changed, or the market changed, or my own desires changed. The process of meeting human needs is messy because both human needs and the tools we have at our disposal are a constantly moving target.
Move fast and break things? Let’s not. Let’s build carefully and methodically. Teach others how to build quality software. Stop regurgitating what you watch on YouTube 4 hour course. I’ve seen horrific, I mean absolutely bottom of the barrel code being taught to others. Especially in JS community - yes, I’m picking at you guys again.
When teaching goes to shit, you’re breeding and propagating, institutionalizing horrible ways to do something - amplified 100x because YouTubers are chasing viewership. That code camp 8 hour course is better replaced by reading good books and docs. Actually build something by thoroughly reading the docs.
Now you got 100x more developers building foundational blocks that other developers blindly build atop.
Study what Unix did when they were building small composable highly quality building blocks. Still used today after 45 years!
This has not been my experience in backend development (and some dabbling in building small React frontends). Our building blocks are FOSS components who are quite robust and widely tested, and the bugs, mistakes and shortcuts we have to deal with are almost exclusively of our own making.
At first I was inclined to comment "it doesn't" because I can easily build small but useful tools in a matter of days.
But your comment made me realize that maybe the reason is just that I keep using the same old C++ libraries to avoid surprises.
In my last Ruby project, critical APIs changed multiple times during development. But Boost / openssl / curl / TBB / MKL are surprisingly API stable, given how much is changed under the hood.
Maybe conservative languages attract conservative programmers who conserve time by conserving APIs.
It's not mentioned specifically, but I think this very much falls into the "Accidental complexity" bucket. In the same way he describes someone choosing to use Mathematica for solving a problem - a developer choosing an obscure technology or writing poor code is just more accidental complexity.
I think you are using the wrong building blocks. Build your own blocks or use ones that are solid with good test coverage and many of those issues go away.
I really like Dorian Taylor's analogy of software to movie production: https://doriantaylor.com/softwares-ailing-mythology
Their point is that software is a refinement of a process to the point where it can be executed by a computer (in a generally bug-free manner). Just like a movie spends a lot of time in pre-production and writing the script, we must spend a lot of time on refining our own understanding of our processes, before trying to explain them to computer via code. The faster one jumps to code, the more likely one is to have to go back and fix things, delaying the delivery of software.
So software is more similar to writing novels or movies, than construction, where a lot of time is spent just thinking through things, and going through the ups and downs of "writer's block". Perhaps one way out is to loosen the delineation between making software and using software (a la spreadsheets, Smalltalk) and moving towards more programmable environments, so one doesn't deliver software, but rather allow users to build their own as part of refining their processes.
It's because most software developers add negative value. Most not them aren't investing in learning at all and are just playing around and understanding things on the surface level. I'm fairly certain it takes much more learning and effort to keep up to date as an accountant than a software engineer.
I've been getting into elixir and was very sad to realise that a lot of the best practices have been enshrined into the erlang otp library for 20 years and we've forgotten about it.
Think how many engineers you know that can reason well about db indexes or could implement something like redis given enough time.
On the one hand it's fairly depressing, on the other hand it makes sense from the incentive structure - business people treat them like "nerds" they're not on the same career path as someone in management, PMs come from weird professions that have nothing to do with software and are focussing on weird scrum processes and jira tickets...
At least that has been my experience. I've seen massive amounts of wasted resources that has made me a bit jaded
"it takes much more learning and effort to keep up to date as an accountant than a software engineer"
you are wrong in many levels.... accounting is a field that moves slowly and it is highly regulated
Software is still in the wild wild west phase, and it catches and loses trends all the time. We are currently using languages and platforms that didn't exist at all 10-12 years ago, yet, in 10 years whatever you are doing now, might be completely obsolete and you have to retrain-retool.
You may be jaded, but it doesn't mean you're wrong.
Framework churn is such an issue, and so it language and all the other fashions that we are victim too. People don't spend enough time getting value from what they decide to use.
I think a more relaxing way to think about software, is to think about bridges and buildings. Sometimes bridges and buildings fall down. It happens less now. However, Civil engineers have had about ~2500 years to try all the combos and now they can both make things both robust and pretty.
If software is bridges, we as an industry spend a lot of effort on the gold plating very rickety construction.
Gold plating in the form of over security engineering inconsequential little things, over complicating UI's, rapidly building new languages because of a perceptual deficiency in language Y.
It'll stabilize, and success will go up. The business / PM side is also new at this and doesn't control risk well yet. Agile and Scrum attempt it but .... the same fundamentals that get a building built well and on budget also apply to good software.
This is but one part of it, but what I see is that software has a uniqueness to it because so much of the complexity is hidden. Really, nobody can see it being created except the actual developer building that piece and maybe one level up if they are good enough to review the code. But to everybody else its a mysterious black box. Now this not only hides the complexity but actually amplifies it because at the ground level people are highly incentivised to inject complexity into their work. It feels good to make something complex. You feel productive and it definitely helps your resume if you can claim to have worked on something highly complex. Everybody else is impressed. The team gets more resources to support this amazing thing being created.
So hiding the complexity not only prevents exposure but means many people in the production chain incentivised to create complexity can get away with gratuitously injecting it into things. Whole frameworks will be deployed where 10 lines of code would have worked, new libraries will mysteriously appear and get used for 1 function. Layers of indirection in the design will appear to abstract things that have only 1 concrete implementation. etc etc. All of this is invisible above 1 management layer and therefore unmeasurable and unmanageable. But the outcome is, the end result takes much longer than you would surmise from the bare bones functionality or requirements.
I fundamentally disagree with the conclusions of this article.
It doesn't take long to "build software" - git was built in a weekend, Facebook a similar timeline. When the environment is right there's no upper bound on the pace of delivery, it's just that modern day dev is, in many places, less about quality of the output (in terms of product/market fit and user value, not code quality) and more about the satisfaction of ceremony. You couldn't product manage Google into existence, or any truly valuable piece of software, hardware, or any other innovation, but it's customary for organisations to create structures which feel grown-up in order to yield software.
The talents now required to enter the field are also different, which has naturally played a part in the perceived slow-down. Development is a new field, and research into what makes teams effective is limited - we're still using the ideas of industrial steel production to try and bolster the present manufacturing line. This means there are plenty of undesirable behaviours.
But ultimately, it's about what you're optimising for. If you hire outstanding people and give them an excellent brief, you can deliver incredible software at pace. Most modern companies don't optimise for this, it's more about satisfying sprawling teams, polishing egos, and chasing incremental gains. In that environment you can bet delivery is slow.
That's an overstatement. You can build impressive MVP and tech demos in a short amount of time, yes. But that requires that the author has already been thinking about the types of problems that come up in this domain for a long long time. Rest assured that Git certainly hasn't been created in a weekend.
Also, you can't create a product in this short amount of time. (There are some websites that do almost nothing at all, and that are very profitable, but those aren't engineering problems). Think how long git has been maintained, polished, and extended since its inception. And still people complain.
Software is a learning process, otherwise people wouldn't be so obsessed about it. And learning takes time. No matter which way you're approaching it.
While using git as an example is a bit over simplifying I agree with the comment. Over time, if you plan ahead, your build up a set of building blocks and at some point you can put together an application very quickly. I suppose it is all relative though. While a month might seem fast to me someone else might expect a solid, fully featured application in a day or a weekend.
A lot of accidental complexity arises from adding more people to the mix. There is the communication overhead to deal with. Even when things go well this consumes time. When it doesn't: more time. Then there's of course the constant arguing and bickering over what is best; more people means more opinions on this. And of course the more people you have the more complexity gets introduced. Conway's law is a thing.
The benchmark for me is what I think I can pull off in a few days vs. what I would build with a whole team in a few months. You can build a lot in a weekend. But not not with a whole team around you questioning every move. So, when I need quick results, I give the right people a lot of freedom and not too many distractions and see what happens.
I'm building a new webapp for a startup with a few people since three weeks. We're launching with customers next month. That involves a lot of pragmatic decision making and taking on a few risks. Most of the hard choices were made in the first week and at this point the app is starting to look alright.
Here's what we did not do: we had no designers involved. We also had no lengthy meetings about what features we were going to have. We're patching up our existing backend and adding some new features to it.
Otherwise, it's an emergency rebuild of an app that just wasn't good enough (wrong mobile platform, developer is out of the picture, and it's a lot of technical debt and stability issues). I took the decision in December and gave our inexperienced frontend person some guidance to investigate the technologies I picked for him. By December we had a clear idea that this was doable. January we started doing this. That's 3 weeks ago. By now it's largely done. We're adding a few missing features and then we have an MVP that will run on IOS, Android and the web.
If I had a few million, I could not do this any faster. But I'd be able to ship beautifully native apps six months later. If fast is a goal, keep your team lean.
When you throw as many coders on to a project as anybody has ever done, in order to rush to the goal with maximum urgency, those man-months really add up fast and it does end up taking longer in the end because of it but look how many people got to the goal and it wouldn't have been possible without all those executive-months struggling to move the goalposts toward the team as much as possible before everything craters.
If you were committed to efficient progress to reach your original goal you're going to need a different approach.
And start all over again.
Carefully read every word of their comments from the programmer having two decades of single-handed operation, and the 42-year corporate retiree now making single handed progress himself, these are long comments but here's an excerpt:
>This all being said, one of the major causes of complexity in software development, which has always been an impediment to quality productivity is the ridiculous demand for unsuited deadlines in development efforts. This is a result of weak and incompetent management that runs up and out of the technical departments right up to division heads and the like. Deadlines, cause their own deference to the development of complexities, which often include defects. Deadlines cause stress and other factors, which often disallow most developers from thinking out their designs and problems clearly with time to tinker with the best solutions.
On a somewhat separate topic and line of thinking than the post (Great article btw, I agree with it completely) -- I've often thought that, as a software engineer, whenever giving estimates, any number of management folks must be thinking "why does it take so long?". And I think part of the reasons is this. If you simply _ignore_ most of the accidental complexity, you often can build 80% of the functionality of a requested product/feature in 20% of the time or less. What's more, for any upper management who's worked in the industry long enough, over the past two decades, they've most likely worked with some whiz kid who had done it (I was one myself in my early 20s a decade+ ago) -- so they know it can be done and have witnessed it.
Let's say the requested product is some CRUD web app. At one point in their career, this executive worked in a growing tech startup and on the team was a talented fresh grad. That engineer built a product very similar to that entirely, front and back ends, in two days. 15 years later you're working in a different company as an executive and you're talking about building some CRUD web app just like that. Why is the estimates being given by the dev team 4 weeks with 4 developers?
What wasn't seen and remembered was that the one built in two days by the smart kid had no unit tests (or any kind of tests), didn't cover many edge cases (at least in its first version), a UI with barebones CSS, no responsive layout, no component-based frontend (e.g. React), no scalability concerns, etc. But all the requested functionality is there, it's demo-able, and usable even. A similar product is being requested now, but it needs to be built to withstand today's quality standards, robustness, maintainability, etc. So unit tests are part of the requirements, a component-based frontend is needed for maintainability, you have to handle mobile, etc.
It's pretty much all the same points the author is pointing out about accidental complexity; just kind of showing why someone would ask that question.
It sounds like the kid in your example built an MVP product for the executive to demonstrate/test business value. Expectations matter. If the executive thought that was a finished product and is holding everyone to that standard, they were mistaken. I don't see any reason to write extensive unit tests, a component-based frontend, responsive layout, and scalable architecture for the first draft of a product. In my opinion, the first draft of a product in a business environment should test business value of the idea as rapidly as possible, as many of the ideas are half-baked
Some interesting thoughts in here. I guess the main thing is that there is basically infinite demand for more things in software. You can take almost any feature from any product and come up with new things you would like it to do that would take entire teams of developers to work on.
Software will never get quicker to develop because we will always request the absolute maximum amount of features that the market can stand to afford.
Basically every area has exploded with more features and is still way way behind what people would ideally want. Just look at the difference between Discord and IRC.
But do we actually have more features? I don't think Google Docs has more features than Ami Pro.
I think the problem is the modern focus on adding features, without consideration whether and how these compose. Not all features are equal - some can be used as a foundation on which more can be easily built, while some are a detriment to that. Just adding features without architectural consideration leads to - surprisingly - bad architecture.
Razors today looks very different from the ones when it was invented. Same goes for vacuum cleaners, cars, tools for construction/manual task, the list goes on.
The only notable difference is that each iteration of hardware have a vividly clear version start/end date compared to the rolling versioning of software. First vacuum cleaner was invented 1908...
"the last 20 years has been the drastic reduction in the ratio of essential to accidental complexity"
I know this is nitpicking, but the author is using mathematics as an example and then uses ratio the wrong way... The point he/she is trying to make is that tools got better. That would mean the accidental complexity goes down. That means the ratio of essential to accidental complexity goes up.
I wondered about that as well, because much of the content in the middle of the article, where the author lists changes in the industry such as automated infrastructure and frequent deployments, would suggest to me that accidental complexity really has increased and the author's statement is actually correct as written even if that wasn't the intent.
My take is that programming languages have an inadequate ratio between abstract and concrete data flows.
You are supposed to build code by stitching together commands for all the data handling functions required to handle each case in the data. Yet development environments make it really hard to actually _see_ what effect the functions are having on the data. You need to enter a special debug mode and place strategic breakpoints, logs and watched expressions, instead of making easy to compare the full program state before and after executing each function.
In that respect, spreadsheets and reactive functional languages make it easier to prototype new software fast, since they have close together both functions and the data on which the functions operate; and in spreadsheets, you also get the intermediate steps for chains of functions.
His visual interactions for programming are a great advance in the state of the art for fast development, and they've greatly influenced recent advances in development environments - making everything more reactive, and with code and data more intertwined.
In an average replicate (FB, HN, Twitter) tutorial, students can see how with an amazing developer skills, someone creates the finished product from 15 minutes to 2 hours. But, when they try to do the same, it takes days, weeks or months.
And this also applies to an experienced developer; the reason is depth. (How deep to the rabbit hole do we want to go).
There are always hidden complexities, configuration, installations, validations, issues, known obstacles and unknown obstacles; there is time for meetings, negotiations, time for thinking, time for merely trying to come up with the best approach long before we start coding.
A good example from my early career was "just one field in the database", a client says "hey this will take you just 1 minute, why you are arguing with me about this", but that "one field" is so much intertwined that can affect half of the codebase. It means, going through all procedures, views, models, tests, validations, then there is an impact on historical data etc...
Also there is a huge difference between: spike, MVP, and finished product.
I think there is a cost disease in software where baseline requirements only increase depending on the maturity of a company. Accessibility, localization, scalability, permissioning, observability and alerting, reliable deploy and testing pipeline... all these things are table stakes for a new project at the company I work at, represent A LOT of complexity, yet an application delivered without them might have a nearly identical user experience...
That’s not even to mention the added burdens on the planning side of things, e.g. getting infosec and legal and compliance to sign off before breaking ground, gathering requirements from 10 different teams across the world, appeasing more stakeholders, etc.
The difference just in table stakes between a young startup and a mature company can easily account for an order of magnitude difference in effort required to deliver a feature, and it shouldn’t be surprising.
Edit: and this is why I weep when I see mature companies under-investing in developer tooling.
The pace of change is too quick and the permutations of how to do X are necessarily too numerous.
As far as I know there is no “standards body” that governs these things across organisations.
Ie: if you’re developing a web app that needs to poll other systems for data every X minutes there should be a standard that governs the best way of doing this in the major 5 languages.
Taking into account SRE principles like logging, scaling, security etc. And some clear code examples using the simplest and least OO-functional-prototypical-new age code possible
What I find the most painful is that you can't rely on much to decrease accidental complexity.
A couple of years ago I settled on the JavaScript Dojo Framework. It was the most solid framework available, but then AngularJS came out, and it had some very compelling arguments on it's side.
So I moved over to AngularJS, being happy that there finally a good, modern solutions to many problems.
Then Angular 2 came. Then AngularJS 1.x stopped being updated. Angular 2 was nowhere near what I needed, so I had to find an alternative.
I switched to Vue, being really happy about it. Now I'm finding myself in the transition to Vue3, where frameworks like Quasar still need to get ported and it is unclear what will happen to all the code I already wrote.
Or Python2 vs Python3.
All this progress makes me feel burned out, barely having time to tackle the actual problem to be solved.
Typical internal CRUD applications took about 1/3 the time and 1/3 the coding to produce in my experience with the desktop IDE's of the late 90's. The web bloated things up so that you spend time hooking layers to more layers and debugging layers instead of focusing on business logic and UI in terms of users' needs. Now you micromanage technology instead of actual work.
People often say, "yes, but we have so much wonderful choice now!" Perhaps, but organizations are paying dearly for that choice.
Oracle Forms may have been esthetically ugly, but it did the CRUD job fairly well without fuss and muss, and developers were quite productive with it. The client was kind of a GUI browser so that you didn't need to install each application on each user's PC. It was also multi-platform. (Oracle mucked up the client by migrating it to Java applets.)
Some point out they don't work well on phones, but few of our apps are used on phones at our org. The UI was crippled to be mobile-friendly, but nobody is mobile-ing for real-work stuff. Make a special phone portal for the 5% or so of apps that need it instead of tangle up the 95% "incase" someday someone will want mobile. YAGNI.
If the industry would just learn to live with ugly K.I.S.S., they could save a lot of money and time. Warren Buffett says one of his greatest strengths is the courage to say "no". Few have that, and bloat the IT world as a result. Maybe fear of missing the Buzzword of the Week turns our stacks into buzzword pack-rats.
Maybe because successful companies in tech got rich so quick that their leaders never learned how to code properly.
None of the developers who work for big successful companies know how to code efficiently. Initially, they were hacking together all the code as quickly as possible (of course that didn't scale and had to be rewritten completely) and then they later got enough money to just brute-force development by throwing hundreds of developers at it.
Then they started pushing their naive ideas to everyone else (e.g. static typing, functional programming, 100% unit test coverage, Elixir...) and they put 0 focus on stuff that really matters such as keeping method parameters and return values as simple as possible, consuming events using for-await loops instead of event listener callbacks, naming variables well, separation of concerns, etc...
So now most of the industry knows two kinds of code: Dirty hacks or bureaucratic bloatware. Nobody knows how to code properly. There is nobody around to teach people the correct approach and there are a lot of people around to teach the incorrect approach.
When someone asks "Why does it take so long to build software?" and you want to verify whether said person has any intelligent thoughts on the matter, ask them: "How long do you think it should take to build software?"
> How long do you think it should take to build software?
About as long as it takes to figure out what the software should do. Add one day for each 1000 lines of code for typing carefully and adding automated tests.
Aren't most developers in the world paid a salary or day rate and thus are paid for their time? What incentive is there for them to "hurry up" rather than sandbag?
A marketing page
A dashboard set of elements
A payment system
A billing history, refund, change card, invoice system
A token system
A auth system
A email system or integration with a email provider.
A complaint system
A admin dashboard
A often custom database schema
A analytics system and useage
A log tracker
A cloud infrastructure for hosting
Rate limiting
A api
A event logging system
A marketing blog
I have strayed too far.
My point is each one if these things is usually build from scratch. It takes a lot of time, then you have to wire everything together. In typical custom ways. If someone makes a off the shelf version typically the learning curve is greater than building a custom version.
I think software takes so log because each dev tries to rebuild the wheel. And often times the wheel does need to be rebuilt. But due to time issues things never get finished to the degree you planned and your custom code has been rewritten 10 times and each new feature take a logx longer to not break the old stuff.
[+] [-] caseyross|5 years ago|reply
That is, software is trivially copy-able, so there is no reason to spend effort duplicating any software that already exists. (Legal reasons and "not-invented-here" syndrome notwithstanding.) This is a huge different from how the "real" world works, where almost all of the work involved in building, say, a car, is actually just the work of assembling identical copies of that product so that it can be physically sold to more than one person.
The insane amounts of repeated effort involved in building any physical product at scale have a silver lining, in that the repetition enables a very good and stable estimate of how long another repetition will take. Software is the opposite situation --- we only need to build anything once, but not having built that thing before, we don't really know how long it will take.
[+] [-] TeMPOraL|5 years ago|reply
The question of why software takes a long time to build pertains to the coding phase, to which real life's analogue is drafting/design phase. It is my impression that the drafting/design phase in "hard" industries can be just as messy and hard to schedule as software development is - we just don't pay attention, because for physical products, drafting/design is nearly free (capex & opex of the assembly phase dominates), whereas in software, the construction is nearly free (compiler time is too cheap to meter).
It's true that, on the surface, software is "always new". But after doing it for a while, I have this sinking feeling that a lot of it is old and repetitive work that we hadn't yet automated - there are deep similarities between pieces of code we're writing, but we don't have a language to fully capture them and abstract them away.
[+] [-] lazysheepherd|5 years ago|reply
While producing a machine, there are two steps;
1) designing the blueprint (takes a lot of time, unpredictable)
2) mass production (takes a lot of time, predictable)
Those steps in software development;
1) building the software (takes a lot of time, unpredictable)
2) deploying the code (takes virtually no time, predictable)
So the real difference is this: software is almost all about doing something new, each and every time. It's all flesh and has no bone: e.g. devoid of a long, very stable and predictable mass production stage. It's almost all about R&D, all of it's length. Hence 98% of the process is novel work therefore takes time and hard to predict.
[+] [-] Justsignedup|5 years ago|reply
Yes, building cars in a factory yields repetitive results, but what about all the different engineering projects. The truth is that this is not a new problem, we just lack discipline of fields that have been refined over thousands of years.
Because of the quick idea to prototype rate, product research and refinement takes a back seat. All engineering projects have massive research before the first stone is placed. Nobody mid way through building a sky scraper decides "oh, do we need to redo the foundation?"
I would also like to note, nobody is gonna give a self-taught architect who's 19 the ability to build that skyscraper foundation. Each building technique is developed, tested, and then once determined good, is actually used by others, and supervised by an industry veteran with underlings who learn and eventually themselves supervise such projects. Meanwhile in software it is the opposite.
Not to say that there isn't a reason software grows so fast. Things change daily. If software worked like architecture we'd be licensing even the right to use ruby on rails-style modeling. But software works differently, thus faster iteration, thus more loosey-goosey, thus harder to estimate.
[+] [-] sriku|5 years ago|reply
And yet it feels like authentication and authorization code need to be rewritten for every application.
[+] [-] TuringTest|5 years ago|reply
The conclusion is that the tools to build software from a known specification to working code are sub-optimal.
[+] [-] fendy3002|5 years ago|reply
But we often asked for accounting app to able to handle employee's payroll and paid leaves.
Then with that many demand for custom software, we also got many times software manufacturer than car manufacturer. Inexperienced manufacturer (programmer) will make development even slower. Not to mention that the client even don't know what they want.
[+] [-] boris|5 years ago|reply
[+] [-] lisper|5 years ago|reply
But as someone who has had the luxury of taking time to do things the Right Way let me tell you from firsthand experience: doing things the Right Way is incredibly hard in no small measure because figuring out what you actually want to do is incredibly hard. There have been many times when I thought I was building something for the ages only to discover that I had made a bad assumption, or technology changed, or the market changed, or my own desires changed. The process of meeting human needs is messy because both human needs and the tools we have at our disposal are a constantly moving target.
[+] [-] systemvoltage|5 years ago|reply
Move fast and break things? Let’s not. Let’s build carefully and methodically. Teach others how to build quality software. Stop regurgitating what you watch on YouTube 4 hour course. I’ve seen horrific, I mean absolutely bottom of the barrel code being taught to others. Especially in JS community - yes, I’m picking at you guys again.
When teaching goes to shit, you’re breeding and propagating, institutionalizing horrible ways to do something - amplified 100x because YouTubers are chasing viewership. That code camp 8 hour course is better replaced by reading good books and docs. Actually build something by thoroughly reading the docs.
Now you got 100x more developers building foundational blocks that other developers blindly build atop.
Study what Unix did when they were building small composable highly quality building blocks. Still used today after 45 years!
[+] [-] dxdm|5 years ago|reply
[+] [-] fxtentacle|5 years ago|reply
At first I was inclined to comment "it doesn't" because I can easily build small but useful tools in a matter of days.
But your comment made me realize that maybe the reason is just that I keep using the same old C++ libraries to avoid surprises.
In my last Ruby project, critical APIs changed multiple times during development. But Boost / openssl / curl / TBB / MKL are surprisingly API stable, given how much is changed under the hood.
Maybe conservative languages attract conservative programmers who conserve time by conserving APIs.
[+] [-] joekrill|5 years ago|reply
[+] [-] peterohler|5 years ago|reply
[+] [-] nsm|5 years ago|reply
So software is more similar to writing novels or movies, than construction, where a lot of time is spent just thinking through things, and going through the ups and downs of "writer's block". Perhaps one way out is to loosen the delineation between making software and using software (a la spreadsheets, Smalltalk) and moving towards more programmable environments, so one doesn't deliver software, but rather allow users to build their own as part of refining their processes.
[+] [-] mstipetic|5 years ago|reply
[deleted]
[+] [-] mstipetic|5 years ago|reply
I've been getting into elixir and was very sad to realise that a lot of the best practices have been enshrined into the erlang otp library for 20 years and we've forgotten about it.
Think how many engineers you know that can reason well about db indexes or could implement something like redis given enough time.
On the one hand it's fairly depressing, on the other hand it makes sense from the incentive structure - business people treat them like "nerds" they're not on the same career path as someone in management, PMs come from weird professions that have nothing to do with software and are focussing on weird scrum processes and jira tickets...
At least that has been my experience. I've seen massive amounts of wasted resources that has made me a bit jaded
[+] [-] ardit33|5 years ago|reply
you are wrong in many levels.... accounting is a field that moves slowly and it is highly regulated
Software is still in the wild wild west phase, and it catches and loses trends all the time. We are currently using languages and platforms that didn't exist at all 10-12 years ago, yet, in 10 years whatever you are doing now, might be completely obsolete and you have to retrain-retool.
[+] [-] gonzo41|5 years ago|reply
I think a more relaxing way to think about software, is to think about bridges and buildings. Sometimes bridges and buildings fall down. It happens less now. However, Civil engineers have had about ~2500 years to try all the combos and now they can both make things both robust and pretty.
If software is bridges, we as an industry spend a lot of effort on the gold plating very rickety construction. Gold plating in the form of over security engineering inconsequential little things, over complicating UI's, rapidly building new languages because of a perceptual deficiency in language Y.
It'll stabilize, and success will go up. The business / PM side is also new at this and doesn't control risk well yet. Agile and Scrum attempt it but .... the same fundamentals that get a building built well and on budget also apply to good software.
[+] [-] zmmmmm|5 years ago|reply
This is but one part of it, but what I see is that software has a uniqueness to it because so much of the complexity is hidden. Really, nobody can see it being created except the actual developer building that piece and maybe one level up if they are good enough to review the code. But to everybody else its a mysterious black box. Now this not only hides the complexity but actually amplifies it because at the ground level people are highly incentivised to inject complexity into their work. It feels good to make something complex. You feel productive and it definitely helps your resume if you can claim to have worked on something highly complex. Everybody else is impressed. The team gets more resources to support this amazing thing being created.
So hiding the complexity not only prevents exposure but means many people in the production chain incentivised to create complexity can get away with gratuitously injecting it into things. Whole frameworks will be deployed where 10 lines of code would have worked, new libraries will mysteriously appear and get used for 1 function. Layers of indirection in the design will appear to abstract things that have only 1 concrete implementation. etc etc. All of this is invisible above 1 management layer and therefore unmeasurable and unmanageable. But the outcome is, the end result takes much longer than you would surmise from the bare bones functionality or requirements.
[+] [-] tom_walters|5 years ago|reply
It doesn't take long to "build software" - git was built in a weekend, Facebook a similar timeline. When the environment is right there's no upper bound on the pace of delivery, it's just that modern day dev is, in many places, less about quality of the output (in terms of product/market fit and user value, not code quality) and more about the satisfaction of ceremony. You couldn't product manage Google into existence, or any truly valuable piece of software, hardware, or any other innovation, but it's customary for organisations to create structures which feel grown-up in order to yield software.
The talents now required to enter the field are also different, which has naturally played a part in the perceived slow-down. Development is a new field, and research into what makes teams effective is limited - we're still using the ideas of industrial steel production to try and bolster the present manufacturing line. This means there are plenty of undesirable behaviours.
But ultimately, it's about what you're optimising for. If you hire outstanding people and give them an excellent brief, you can deliver incredible software at pace. Most modern companies don't optimise for this, it's more about satisfying sprawling teams, polishing egos, and chasing incremental gains. In that environment you can bet delivery is slow.
[+] [-] jstimpfle|5 years ago|reply
Also, you can't create a product in this short amount of time. (There are some websites that do almost nothing at all, and that are very profitable, but those aren't engineering problems). Think how long git has been maintained, polished, and extended since its inception. And still people complain.
Software is a learning process, otherwise people wouldn't be so obsessed about it. And learning takes time. No matter which way you're approaching it.
[+] [-] gilbetron|5 years ago|reply
And git wasn't even built in 10 days, as Linus has said elsewhere. The core structure was, kind of. But there's been a large effort on top of that.
And the idea that Facebook was built in a weekend is just completely wrong. Sure, some basic Facebook-ish app was created over a weekend, sorta.
[+] [-] peterohler|5 years ago|reply
[+] [-] mbrodersen|5 years ago|reply
[+] [-] jillesvangurp|5 years ago|reply
The benchmark for me is what I think I can pull off in a few days vs. what I would build with a whole team in a few months. You can build a lot in a weekend. But not not with a whole team around you questioning every move. So, when I need quick results, I give the right people a lot of freedom and not too many distractions and see what happens.
I'm building a new webapp for a startup with a few people since three weeks. We're launching with customers next month. That involves a lot of pragmatic decision making and taking on a few risks. Most of the hard choices were made in the first week and at this point the app is starting to look alright.
Here's what we did not do: we had no designers involved. We also had no lengthy meetings about what features we were going to have. We're patching up our existing backend and adding some new features to it.
Otherwise, it's an emergency rebuild of an app that just wasn't good enough (wrong mobile platform, developer is out of the picture, and it's a lot of technical debt and stability issues). I took the decision in December and gave our inexperienced frontend person some guidance to investigate the technologies I picked for him. By December we had a clear idea that this was doable. January we started doing this. That's 3 weeks ago. By now it's largely done. We're adding a few missing features and then we have an MVP that will run on IOS, Android and the web.
If I had a few million, I could not do this any faster. But I'd be able to ship beautifully native apps six months later. If fast is a goal, keep your team lean.
[+] [-] fuzzfactor|5 years ago|reply
If you were committed to efficient progress to reach your original goal you're going to need a different approach.
And start all over again.
Carefully read every word of their comments from the programmer having two decades of single-handed operation, and the 42-year corporate retiree now making single handed progress himself, these are long comments but here's an excerpt:
>This all being said, one of the major causes of complexity in software development, which has always been an impediment to quality productivity is the ridiculous demand for unsuited deadlines in development efforts. This is a result of weak and incompetent management that runs up and out of the technical departments right up to division heads and the like. Deadlines, cause their own deference to the development of complexities, which often include defects. Deadlines cause stress and other factors, which often disallow most developers from thinking out their designs and problems clearly with time to tinker with the best solutions.
[+] [-] godot|5 years ago|reply
Let's say the requested product is some CRUD web app. At one point in their career, this executive worked in a growing tech startup and on the team was a talented fresh grad. That engineer built a product very similar to that entirely, front and back ends, in two days. 15 years later you're working in a different company as an executive and you're talking about building some CRUD web app just like that. Why is the estimates being given by the dev team 4 weeks with 4 developers?
What wasn't seen and remembered was that the one built in two days by the smart kid had no unit tests (or any kind of tests), didn't cover many edge cases (at least in its first version), a UI with barebones CSS, no responsive layout, no component-based frontend (e.g. React), no scalability concerns, etc. But all the requested functionality is there, it's demo-able, and usable even. A similar product is being requested now, but it needs to be built to withstand today's quality standards, robustness, maintainability, etc. So unit tests are part of the requirements, a component-based frontend is needed for maintainability, you have to handle mobile, etc.
It's pretty much all the same points the author is pointing out about accidental complexity; just kind of showing why someone would ask that question.
[+] [-] errantmind|5 years ago|reply
[+] [-] SulfurHexaFluri|5 years ago|reply
Software will never get quicker to develop because we will always request the absolute maximum amount of features that the market can stand to afford.
Basically every area has exploded with more features and is still way way behind what people would ideally want. Just look at the difference between Discord and IRC.
[+] [-] js8|5 years ago|reply
I think the problem is the modern focus on adding features, without consideration whether and how these compose. Not all features are equal - some can be used as a foundation on which more can be easily built, while some are a detriment to that. Just adding features without architectural consideration leads to - surprisingly - bad architecture.
[+] [-] NalNezumi|5 years ago|reply
Razors today looks very different from the ones when it was invented. Same goes for vacuum cleaners, cars, tools for construction/manual task, the list goes on.
The only notable difference is that each iteration of hardware have a vividly clear version start/end date compared to the rolling versioning of software. First vacuum cleaner was invented 1908...
[+] [-] loteck|5 years ago|reply
Law is very similar to software in this way. They both also take a long time to write & are never actually "finished."
[+] [-] fxtentacle|5 years ago|reply
I know this is nitpicking, but the author is using mathematics as an example and then uses ratio the wrong way... The point he/she is trying to make is that tools got better. That would mean the accidental complexity goes down. That means the ratio of essential to accidental complexity goes up.
[+] [-] Silhouette|5 years ago|reply
[+] [-] TuringTest|5 years ago|reply
You are supposed to build code by stitching together commands for all the data handling functions required to handle each case in the data. Yet development environments make it really hard to actually _see_ what effect the functions are having on the data. You need to enter a special debug mode and place strategic breakpoints, logs and watched expressions, instead of making easy to compare the full program state before and after executing each function.
In that respect, spreadsheets and reactive functional languages make it easier to prototype new software fast, since they have close together both functions and the data on which the functions operate; and in spreadsheets, you also get the intermediate steps for chains of functions.
[+] [-] TuringTest|5 years ago|reply
http://worrydream.com/LearnableProgramming/
His visual interactions for programming are a great advance in the state of the art for fast development, and they've greatly influenced recent advances in development environments - making everything more reactive, and with code and data more intertwined.
[+] [-] NiceWayToDoIT|5 years ago|reply
A good example from my early career was "just one field in the database", a client says "hey this will take you just 1 minute, why you are arguing with me about this", but that "one field" is so much intertwined that can affect half of the codebase. It means, going through all procedures, views, models, tests, validations, then there is an impact on historical data etc...
Also there is a huge difference between: spike, MVP, and finished product.
[+] [-] hamandcheese|5 years ago|reply
That’s not even to mention the added burdens on the planning side of things, e.g. getting infosec and legal and compliance to sign off before breaking ground, gathering requirements from 10 different teams across the world, appeasing more stakeholders, etc.
The difference just in table stakes between a young startup and a mature company can easily account for an order of magnitude difference in effort required to deliver a feature, and it shouldn’t be surprising.
Edit: and this is why I weep when I see mature companies under-investing in developer tooling.
[+] [-] holografix|5 years ago|reply
As far as I know there is no “standards body” that governs these things across organisations.
Ie: if you’re developing a web app that needs to poll other systems for data every X minutes there should be a standard that governs the best way of doing this in the major 5 languages.
Taking into account SRE principles like logging, scaling, security etc. And some clear code examples using the simplest and least OO-functional-prototypical-new age code possible
[+] [-] qwertox|5 years ago|reply
A couple of years ago I settled on the JavaScript Dojo Framework. It was the most solid framework available, but then AngularJS came out, and it had some very compelling arguments on it's side.
So I moved over to AngularJS, being happy that there finally a good, modern solutions to many problems.
Then Angular 2 came. Then AngularJS 1.x stopped being updated. Angular 2 was nowhere near what I needed, so I had to find an alternative.
I switched to Vue, being really happy about it. Now I'm finding myself in the transition to Vue3, where frameworks like Quasar still need to get ported and it is unclear what will happen to all the code I already wrote.
Or Python2 vs Python3.
All this progress makes me feel burned out, barely having time to tackle the actual problem to be solved.
[+] [-] tabtab|5 years ago|reply
People often say, "yes, but we have so much wonderful choice now!" Perhaps, but organizations are paying dearly for that choice.
Oracle Forms may have been esthetically ugly, but it did the CRUD job fairly well without fuss and muss, and developers were quite productive with it. The client was kind of a GUI browser so that you didn't need to install each application on each user's PC. It was also multi-platform. (Oracle mucked up the client by migrating it to Java applets.)
Some point out they don't work well on phones, but few of our apps are used on phones at our org. The UI was crippled to be mobile-friendly, but nobody is mobile-ing for real-work stuff. Make a special phone portal for the 5% or so of apps that need it instead of tangle up the 95% "incase" someday someone will want mobile. YAGNI.
If the industry would just learn to live with ugly K.I.S.S., they could save a lot of money and time. Warren Buffett says one of his greatest strengths is the courage to say "no". Few have that, and bloat the IT world as a result. Maybe fear of missing the Buzzword of the Week turns our stacks into buzzword pack-rats.
[+] [-] cryptica|5 years ago|reply
Then they started pushing their naive ideas to everyone else (e.g. static typing, functional programming, 100% unit test coverage, Elixir...) and they put 0 focus on stuff that really matters such as keeping method parameters and return values as simple as possible, consuming events using for-await loops instead of event listener callbacks, naming variables well, separation of concerns, etc...
So now most of the industry knows two kinds of code: Dirty hacks or bureaucratic bloatware. Nobody knows how to code properly. There is nobody around to teach people the correct approach and there are a lot of people around to teach the incorrect approach.
[+] [-] junippor|5 years ago|reply
[+] [-] username90|5 years ago|reply
About as long as it takes to figure out what the software should do. Add one day for each 1000 lines of code for typing carefully and adding automated tests.
[+] [-] marklit|5 years ago|reply
[+] [-] fxtentacle|5 years ago|reply
There's a minimum quality level required to not get fired, and I wouldn't be surprised if that's exactly what most employees deliver.
[+] [-] errantmind|5 years ago|reply
[+] [-] ohadron|5 years ago|reply
[+] [-] godmode2019|5 years ago|reply
What do you need
A marketing page A dashboard set of elements A payment system A billing history, refund, change card, invoice system A token system A auth system A email system or integration with a email provider. A complaint system A admin dashboard A often custom database schema A analytics system and useage A log tracker A cloud infrastructure for hosting Rate limiting A api A event logging system A marketing blog
I have strayed too far.
My point is each one if these things is usually build from scratch. It takes a lot of time, then you have to wire everything together. In typical custom ways. If someone makes a off the shelf version typically the learning curve is greater than building a custom version.
I think software takes so log because each dev tries to rebuild the wheel. And often times the wheel does need to be rebuilt. But due to time issues things never get finished to the degree you planned and your custom code has been rewritten 10 times and each new feature take a logx longer to not break the old stuff.