All this smokes and mirrors, when the reality is: if you gave accurate estimates, you'd get reprimanded, not get the project approved, etc. Same reason that infrastructure project go over budget. The way anything actually gets done is by initially being overly optimistic, ignoring potential future problems, getting the project approved, then lock it in via sunk costs, now as problems turn up, you can claim it could never had been foreseen, it's an impossibly hard problem bla bla. But best to bail from the vicinity of the project once reality starts to manifest itself. By that time you should be on the next project perhaps at the next company again promising the stars.
Another factor here is in a meeting-heavy agile environment, many engineers can't even give a good forward estimate of the number of "in the zone" hours they will have in the coming week.
Will I be on 0 outage calls or 5? Doesn't matter if I am on pager duty rotation or not..
Will I get pulled into 3 calls this afternoon or 0?
Is my boss gonna come bother me with some urgent non-ticket task tomorrow morning? Etc.
I'm currently in a no-jira, no-meeting, get-it-done environment and I find myself getting tremendous amounts of work done for the first time in nearly a decade. I would not have estimated my ability to get 1/4 of what I have gotten done, done. My last 2 jobs were basically the opposite. Everything took 4x as long as it should..
Another factor here is in a meeting-heavy agile environment, many engineers can't even give a good forward estimate of the number of "in the zone" hours they will have in the coming week.
As a developer my highest level of productivity was on a 2 person team with a bi-weekly meeting with our manager. We would have impromptu meetings when needed, but for the most part I was free to work 6-15 hours in a row whenever I got in the zone, and if I slacked off other times noone noticed.
Once we got more people and switched to scrum my productivity plummeted. My anxiety went through the roof. Most days the standup left me feeling like I wanted to go back to bed. Most of the time when I was in the zone and feeling engaged I had to waste it on meetings that had no tangible benefits.
Busy work is draining, management knows this, they don't care.
Everytime I see that, I cringe. But sadly, you're not wrong.
Agile is anti-meeting, anti-heavy and definitely anti-meeting-heavy.
I really don't know what to call this weird thing that was created.
Just as an example: why are there standups? Because in XP, meetings are frowned upon. So the idea is not to have meetings whenever humanly possible (better to pair up, talk to individuals etc.).
And if there's a meeting you really really can't avoid or eliminate, let's at the very least incentivise everyone to keep it as short and meaningful as possible. By having everyone stand and thus be slightly uncomfortable.
How that fun, focused, no-nonsense, get-things-done approach got turned into ... whatever that "meeting-heavy" thing is ... is in some senses fascinating.
And terrifying.
> I'm currently in a no-jira, no-meeting, get-it-done environment
I think we went from one extreme to the other. While being an "island in the zone" is super productive, some communication and team work needs to be there to avoid the high bus factor. It's nice when a team member leaves and no one is freaking out because there is a level of redundancy in the knowledge.
That said, the modern communication culture is out of control. We spend way more time talking about what needs to be done instead of actually doing something.
"Let's break the whole thing into 40 microservices" does not help. Unlike the neat theory of "everyone maintains their own set of microservices", every change and new feature needs to be over-analyzed for side effects. You are stuck in analysis paralysis. Not only do you not know what you should talk to other teams about, you may not even know who you should be talking to.
This doesn't really sound like an Agile problem specifically, more like a broader problem of "your on-call technical support roles shouldn't be shouldered by your core development team."
I've worked in an Agile team on a pre-launch product and after the 10min morning scrum (which served more to cement personal accountability and discover blockers than as the showboating and bikeshedding venue that some people here seem to have encountered) the rest of the day was more or less uninterrupted dev time with a few impromptu soccer games thrown in (for research purposes). It worked pretty well.
I'm currently in a 100% Agile-free environment, experiencing the delightful after-effects of good product-market fit, which means even if I have a clear schedule I never know if I'm going to spend half the day trying to help troubleshoot issues or helping end users integrate our product. I don't mind doing this, I want our customers to have the best experience possible and so far it seems to be working pretty well. It does, however, wreak havoc on my ability to 'get in the zone' because subconsciously I know I'll probably just be interrupted anyway.
I don't do "Agile" and never have. But every time I read or hear about Agile it's described like above. Everyone seems to hate it and it all sounds terribly inefficient and frustrating.
I scratch my head...
Why? Why are people still doing it? Who actually _wants_ that?
Management has a solution for that, though - they'll bring in an efficiency coach who will talk to you for an hour insisting that every time you say "I can't", replace that with "I will".
Management: We've got a great new project that has all the features that our users have been asking for. We need it done as fast possible, it's the number one priority. How fast can we get it done?
Engineers: For all of it? At least a year, probably more. Even a subset of those features will take six months, minimum.
Management: That's too long. You can have all the resources you want, we need to have it done in 90 days.
Engineers: That cannot be done. No amount of effort or engineers will get it done in 90 days.
Tell them to rank the features by importance. Go down the list with individual ETA estimates and they can deduce the final ETA. For bonus points provide error bars for each item so they can see how it adds up...
I'm genuinely looking for a "calm" company. Is there such a thing? I have a few anecdotal stories of companies being absolutely chaotic (my current one included). I don't know where to point fingers to. I could start at pointing myself. Customers demanding custom features. Execs and sales people asking for unreasonable estimates. Engineers not feeling safe enough to say "no" but have to make something work, introducing tech debts. Engineers picking technology tools without much research, because there is no time, which increases complexity. People leaving, and then new people can't understand everything holistically. All of these factors combined into one gigantic bowl of mess.
Go find a well established company where you can work on an existing product. I work on a legacy point of sale product that started life in the 80’s and is still powering a good chunk of commerce around the world. I work on a small yet efficient team who customize the product for retailers. It is all about the team you’re on though. My company is also releasing all kinds of new products and services. I doubt anyone would describe working on those teams as calm.
I'm contracted to the IT branch of a certain pharma company and it's been calm the whole time.
Projects are either about regulatory compliance (boring, but not challenging) or supporting research - the big brains do the thinking, while you just klaka klaka klaka away at the implementation.
A visible chunk of the latter goes nowhere and eventually gets cancelled, but everyone is fine with that.
Both companies I've worked for rely on government contracts, and in general feel pretty calm. It's hard to feel super rushed when the software isn't delivered for a few years. I don't think I make anywhere near industry salary though.
I had calmness when I was individual contributor working alone on difficult long-term projects, involving both research and development.
Bi-monthly or weekly meeting with the CTO, and that was it.
Delivery was on time simply because I adjusted the work and constraints to match the expected deadline. If something wasn't possible or required compromises, I would just let the CTO know. It was often fine.
The problem for me starts when you have chaotic product people in-between business owners and you.
I work as a developer for a company that sell a somewhat niche B2B software that integrates with a lot of customer systems. However we got several large, well-known companies that rely on our products for their daily operation (and a ton of smaller ones).
We got a lot of stuff to do, between new customers pouring in and gov't changing their systems with little warning[1] and such.
However we also don't have most of the other stuff you talk about. Sales ask for estimates, but if our provided estimates don't work for the client, say because the contract with our competitor is due in a month, then they'll work with us to try to find some way of making it work rather than force it through.
There's a lot of freedom with responsibility, so sure for low-impact stuff a developer might try some new tech to learn. However for larger shifts it'll have to be discussed in the dev group, especially if it impacts support. We do have a mature codebase so some tech dept is inevitable, but we have it as a goal to try to improve those things if we need to work on a particularly bad area of the code.
As for people leaving, in the years I've been here there's been a very stable group. So stable our customers ask us how their systems work that we integrate with, as they have much higher churn.
I don't think we're particularly unique though. But we're a relatively small company with a name that you can't flash on a CV, and at first glance our niche might sound boring.
[1]: "yea we redesigned our API, we will be doing a hard cut-over in a couple of months"
That might be unrealistic in some cases but one way to deal with this is to take control over the situation. If you have a recipe to fix a particular problem whether it’s about unrealistic expectations or people not feeling safe - propose it, take full responsibility for the implementation and push it through. It takes a long while though.
Yes, that's absolutely possible. My company is making a b2b product. We have challenges and sometimes the scope of the new features can be huge. But I never felt big pressure.
This model is working to identify the Inspection Paradox [0] - most time is spent on doing tasks that take a long time to do. Combine that with high variance, and one task will blow out and a project will be spent working on something that is much harder than people thought, and that the estimate process didn't see coming.
My interpretation here is software engineers are better at estimating tasks than I thought. The next step is analysing tasks being performed in serial to show how many sub-tasks a project can have before the estimates are effectively meaningless / likely to blow out an unpredictable amount. I bet it turns out to be about a sprint's worth of work. Real people have a knack for feeling out those sort of tipping points empirically.
I have two simple parameters I use to apply to all estimates. How many people are involved? And how many times has each person done exactly this thing before?
The more people involved, the more invisible work there is. Not just meetings, but handoffs and general coordination.
And things are only predictable if they are done to the point of being known things. Rehearsed activities are known activities. Even if you shake up a part of it, people that have rehearsed are more capable of dealing with the shake ups and keeping things moving.
What I find ironic is that everyone in the industry knows that estimating is very hard, thus estimations are nearly useless, yet everyone insists on the importance of planning and on basing such planning on void estimates.
Nobody has yet had the balls to state the obvious: we have to learn to work without estimations.
> What I find ironic is that everyone in the industry knows that estimating is very hard, thus estimations are nearly useless, yet everyone insists on the importance of planning and on basing such planning on void estimates.
Obviously, planning is critical. Planning means resource allocation. How is this not a critical aspect of any project?
The mistake you're making is presuming that if estimates are not crisp then they have no value.
> Nobody has yet had the balls to state the obvious: we have to learn to work without estimations.
This belief is detached from reality. Failing to provide estimates means a failure to scope how much work is required, which translates to not even knowing how many people should be working on a simple task.
The problem you're failing to address is limited information, and specifically limited context from developers and emerging requirements. Projects have been adjusting to this for decades. Agile developed concepts such as sprints and spikes for this very reason. It's not only about changes in requirements. It's mainly about gathering info and updating projects based on that.
You estimate as a forcing function to know what you are trying to accomplish. The nuances of scope, how to do it and tradeoffs get discussed.
I think estimates are a problem where you have say 10 estimates, you add then up to 47 days, divide by 3 people, call it 16 days… ah about 3 weeks that’ll be done. That is dumb!
Planning and estimating are orthogonal. I don't think inaccurate estimates should prevent you from anticipating the future.
> have to learn to work without estimations.
A closely related, but different time management tool is a budget. Consider the same website but made in: 3 hours, 3 weeks, or 3 years. The methodology and tradeoffs the engineer approaches each of those projects is very different and needs to be known up front.
Part of the goal of estimation should be to create a budget for various key areas of the project, not guess how long it will take to fix 5 critical system bugs.
That's all well and good in a world that's pure software (and that does happen in those environments). It's not realistic in a world with lead times on physical items that are at least 1 year. For example, if you want to set up a high volume manufacturing line for a new product that takes a lot of planning in advance. You need to spend 10s to 100s of millions of dollars on a contract and schedule with the manufacturer and you need to be able to have SOME indication of where in that process firmware might be released (or more realistically - given the timescale, what scope of software is possible).
I've been advocating for 3-point estimation for quite a while. [1]
Unfortunately it doesn't happen very often (mainly because tooling support for this has been missing in popular trackers such as Jira).
In situations where I was the decisionmaker on the methodology I've had excellent success with it. (using about 90%-95% confidence interval was typically right on the mark)
I write code for 15 years and I still have no idea how to do any estimations. All my estimations are off by 2-200 times. Basically they’re useless. I write code until I’m happy with it. Sometimes I need to rewrite code 5 times before I’m satisfied. Sometimes I have extremely bad mood and can’t do anything creative at all, so even expecting me to work a hour tomorrow is not reliable. Or may be I’ll be in perfect mood and conditions and will spend weekends writing perfect code for 30 hours.
I envy people who can work on schedule. I’m so terrible on that.
> All my estimations are off by 2-200 times. Basically they’re useless.
I know, hyperboles and all, but if your initial estimate for a task is 1 week but you end up requiring 4 years to deliver it then something went terribly wrong, and asking your for estimates ain't it.
Even story points scope tasks with granularities that range different orders of magnitude for this reason. One day, one week, epic/spike. It's ok if one day means 4 days, or one week means two. If one day turns to 3 years then the developer was completely clueless and unfit to continue working on the project.
In one of the Agile training courses I took we were taught the concept of "horizon of predictability". The base one is two weeks.
Within two weeks you more or less can estimate accurately. Beyond the two weeks, the estimates become pretty much useless. "Three months" is not an estimate. It just can't be one in good faith.
This is a simple project, it shouldn't take nearly that long. Let's set the deadline a little earlier and leave the rest as buffer.
This project is ahead of schedule, so you're not going to need the buffer, we'll reallocate resources for that time.
The project is on schedule, let's move up the deadline a little just so we have a bit of buffer at the end.
This project is slightly behind schedule, let's schedule a daily hour long meeting to go over everything to get things back on schedule.
The project is further behind schedule, how about you redo a major part of the project to try a different approach that will hopefully be sufficiently faster that you'll not only recover the cost of the reimplementation but also get back onto the original schedule.
The project has fallen even further behind schedule, and other things are starting to pile up, we need to pull some resources away to deal with them.
The project is ridiculously far behind schedule, people are losing faith in your ability to deliver, you need to get back on schedule before they'll approve more resources for this project.
The project has fallen too far behind schedule, we're pulling the plug. I've scheduled a lengthy post-mortem meeting so we have plenty of time to go over how you failed to bring this project to completion so you will learn from your mistakes.
I know a plan is good when my bottom-up estimates (padded with a buffer for unforeseen contingencies) align with my top-down gut instinct of how long a project of a certain complexity "should take".
As an experienced professional, one often compares with past projects and
looks at the duration of a similar one - not necessarily regarding content but complexity.
Start from an empty file and write to an imaginary API or pseudocode what you actually want to do and coding is easy and fast!
I think the high level imaginary API/pseudocode of any system is straightforward and fast when you start from scratch. (Because you're not focused on detail and just intent)
But when you start from an existing codebase, existing infrastructure, existing APIs, it slows to molasses. You've got to understand and have a mental model of ALL OF THAT before you can complete your intent.
I think this is why software takes longer than expected. Your mental model has to be accurate, and there is always more to learn.
How do you solve the problem?
I'm an advocate and trying to design what I call "commutative programming", which is another way of saying that the behaviour is the product of every statement about the behaviour that is desired, not an explicit instruction of what to do next which is what modern programming is and it is slow and tedious and doesn't compose. In this dream, if you want to change behaviour, you just add statements. Programming is additive! Not edit sourcecode or try to find out where to insert your lines of code or tweak existing lines. Let the computer tell you about conflicting requirements or missing criteria.
We need query engines and cost based optimisers for behaviour.
Imagine taking lots of files that are simple "writing to an imaginary API" - that's your intent specification. You just need to merge them together and your system is finished.
I started work on a commutative GUI https://github.com/samsquire/additive-guis but I'm also thinking from time to time about commutative code where we define refinements to desired behaviour and the computer behaviour query engine or rule engine generates the code or configuration that exhibits this behaviour. (A bit like Kubernetes for behaviour.)
In the future: we don't get LLMs to code for us, but we have highly expressive semantic vector spaces that map to architectural relationships and meaning of behaviour, and changing the architecture of a system to support new behaviour is a matter of changing the serialisation of semantic vectors via the prompt.
> way of saying that the behaviour is the product of every statement about the behaviour that is desired, not an explicit instruction of what to do next
I believe you'll enjoy reading "Notes on the Synthesis of Form" by Christopher Alexander [1]. There are many ideas in it, but at the later chapters the author proposes a system for solving design problems as a large graph of interacting requirements, where smaller sub-graphs are (hopefully reusable) components solving a smaller set of requirements.
Sounds similar to this idea of yours, just with a key difference in that the author sees it as a two-pass solution: first you "carve" out your solution by establishing a context boundary, and then do you "fill" it with an implementation - such that the solution is the minimal implementation that fits the context, and not anymore.
After reading the book, I have also adopted a similar approach (but with less formalism) to gather requirements for projects in the past.
I have thought about developing the ideas of the book further, if you find this subject interesting we can discuss (I can ping your profile's email). I'm certainly interested in hearing more of your own ideas, as it seems to intersect.
[1] Same author of "A Pattern Language", that inspired the design patterns concept in software
We (or at least I) may base our estimates on the happy-path.
That is, I can see how to solve a problem in the common case. But as I code, I encounter the myriad of error cases, exceptions to the common problems, and especially the interactions of all these. Then add in performance considerations and the complexity can multiply quickly. The long-tail of those errors and exceptions can be very time-consuming to solve, so the mean is likely to be way off.
It's obviously much easier to estimate when you're experienced with a problem space. This should mean that you know how to solve a particular problem as well as the likely exceptions (non happy-path) you'll encounter.
If you're experienced with the domain, you can fit the relevant partial solutions in your head and more accurately judge what could go wrong with your code. But more generally I think this is similar to solving problems in other domains as well, such as architecture, building a car, or anything with many moving parts. These domains do not usually have loosely coupled sub systems.
Cars and buildings are probably more loosely coupled than most HN readers imagine. Up through the engine in the 1995 Mustang (I think that was the cutoff), the Ford small block engine line would all bolt into Mustangs all the way back to 1965. I've got a 1989 block, a modern 5-speed transmission with overdrive, hydraulic clutch, disk brakes, dual circuit braking, limited slip differential, and a semi-modern radio in my 1965. All of those were either direct bolt-on or straightforward upgrades with minimal field engineering needed. Many of the VW and Audi engines also have common engine/trans mounts and interchange across years and brands. You can often bolt heavier duty suspension or brake packages onto base model cars/trucks. "Parts bin engineering" is a phrase to search on to learn more. It's done aftermarket, but it's also done by the manufacturers.
There's a similar story in the mechanical engineering for buildings. Sure, some architectural choices will make it harder or easier, but most buildings can be retrofitted system-by-system over the decades a building is in service.
Part of that is to allow configuration pre-sale, but a lot of it is just "it's too damn complex if everything affects everything else", which is not that different from software.
This seems pretty reasonable to me. Maybe some of the details are wrong, I don't know.
But my big question from this article is... why isn't this a well known aspect of project planning, scheduling, and software engineering? It's not even specific to software really - I mean it's possible that software tasks have higher variance than others but I'd be surprised if task variance is irrelevant for scheduling non-software projects.
Is it really the case that no one has applied mathematical modelling to the problem of project planning, scheduling and estimation before? Do the ideas in this blog post represent the core of a new avenue of research? Seems unlikely. So how come this, or something like it, wasn't taught as a basic and critical part of project planning, in the same class that taught me the term "Gantt chart"?
Has anyone tried the Shape Up method by Basecamp? Their concept of “fixed time, variable scope” seems like it might break the Gordion Knot of software estimation challenges.
Software projects take longer than expected because every single time it turns out what actually gets build is nothing like what was originally planned.
[+] [-] bonoboTP|2 years ago|reply
[+] [-] steveBK123|2 years ago|reply
Will I be on 0 outage calls or 5? Doesn't matter if I am on pager duty rotation or not..
Will I get pulled into 3 calls this afternoon or 0?
Is my boss gonna come bother me with some urgent non-ticket task tomorrow morning? Etc.
I'm currently in a no-jira, no-meeting, get-it-done environment and I find myself getting tremendous amounts of work done for the first time in nearly a decade. I would not have estimated my ability to get 1/4 of what I have gotten done, done. My last 2 jobs were basically the opposite. Everything took 4x as long as it should..
[+] [-] dec0dedab0de|2 years ago|reply
As a developer my highest level of productivity was on a 2 person team with a bi-weekly meeting with our manager. We would have impromptu meetings when needed, but for the most part I was free to work 6-15 hours in a row whenever I got in the zone, and if I slacked off other times noone noticed.
Once we got more people and switched to scrum my productivity plummeted. My anxiety went through the roof. Most days the standup left me feeling like I wanted to go back to bed. Most of the time when I was in the zone and feeling engaged I had to waste it on meetings that had no tangible benefits.
Busy work is draining, management knows this, they don't care.
[+] [-] mpweiher|2 years ago|reply
Everytime I see that, I cringe. But sadly, you're not wrong.
Agile is anti-meeting, anti-heavy and definitely anti-meeting-heavy.
I really don't know what to call this weird thing that was created.
Just as an example: why are there standups? Because in XP, meetings are frowned upon. So the idea is not to have meetings whenever humanly possible (better to pair up, talk to individuals etc.).
And if there's a meeting you really really can't avoid or eliminate, let's at the very least incentivise everyone to keep it as short and meaningful as possible. By having everyone stand and thus be slightly uncomfortable.
How that fun, focused, no-nonsense, get-things-done approach got turned into ... whatever that "meeting-heavy" thing is ... is in some senses fascinating.
And terrifying.
> I'm currently in a no-jira, no-meeting, get-it-done environment
AKA, an agile environment. ¯\_(ツ)_/¯
[+] [-] renegade-otter|2 years ago|reply
That said, the modern communication culture is out of control. We spend way more time talking about what needs to be done instead of actually doing something.
"Let's break the whole thing into 40 microservices" does not help. Unlike the neat theory of "everyone maintains their own set of microservices", every change and new feature needs to be over-analyzed for side effects. You are stuck in analysis paralysis. Not only do you not know what you should talk to other teams about, you may not even know who you should be talking to.
[+] [-] taneq|2 years ago|reply
I've worked in an Agile team on a pre-launch product and after the 10min morning scrum (which served more to cement personal accountability and discover blockers than as the showboating and bikeshedding venue that some people here seem to have encountered) the rest of the day was more or less uninterrupted dev time with a few impromptu soccer games thrown in (for research purposes). It worked pretty well.
I'm currently in a 100% Agile-free environment, experiencing the delightful after-effects of good product-market fit, which means even if I have a clear schedule I never know if I'm going to spend half the day trying to help troubleshoot issues or helping end users integrate our product. I don't mind doing this, I want our customers to have the best experience possible and so far it seems to be working pretty well. It does, however, wreak havoc on my ability to 'get in the zone' because subconsciously I know I'll probably just be interrupted anyway.
[+] [-] intelVISA|2 years ago|reply
Best env, but sadly a bit rare?
I used to select for shops like that, no wonder lay-offs are scary if you've been trapped in meetings instead of shipping.
[+] [-] dgb23|2 years ago|reply
I scratch my head...
Why? Why are people still doing it? Who actually _wants_ that?
[+] [-] commandlinefan|2 years ago|reply
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] 0xblinq|2 years ago|reply
[+] [-] AssertErNullNPE|2 years ago|reply
Engineers: For all of it? At least a year, probably more. Even a subset of those features will take six months, minimum.
Management: That's too long. You can have all the resources you want, we need to have it done in 90 days.
Engineers: That cannot be done. No amount of effort or engineers will get it done in 90 days.
Management: It will be done in 90 days.
180 days later
Management: Why are so many of our projects late?
[+] [-] esafak|2 years ago|reply
[+] [-] denvaar|2 years ago|reply
[+] [-] sideproject|2 years ago|reply
[+] [-] heywire|2 years ago|reply
[+] [-] Tade0|2 years ago|reply
I'm contracted to the IT branch of a certain pharma company and it's been calm the whole time.
Projects are either about regulatory compliance (boring, but not challenging) or supporting research - the big brains do the thinking, while you just klaka klaka klaka away at the implementation.
A visible chunk of the latter goes nowhere and eventually gets cancelled, but everyone is fine with that.
[+] [-] Fourier864|2 years ago|reply
[+] [-] whstl|2 years ago|reply
Bi-monthly or weekly meeting with the CTO, and that was it.
Delivery was on time simply because I adjusted the work and constraints to match the expected deadline. If something wasn't possible or required compromises, I would just let the CTO know. It was often fine.
The problem for me starts when you have chaotic product people in-between business owners and you.
[+] [-] magicalhippo|2 years ago|reply
I work as a developer for a company that sell a somewhat niche B2B software that integrates with a lot of customer systems. However we got several large, well-known companies that rely on our products for their daily operation (and a ton of smaller ones).
We got a lot of stuff to do, between new customers pouring in and gov't changing their systems with little warning[1] and such.
However we also don't have most of the other stuff you talk about. Sales ask for estimates, but if our provided estimates don't work for the client, say because the contract with our competitor is due in a month, then they'll work with us to try to find some way of making it work rather than force it through.
There's a lot of freedom with responsibility, so sure for low-impact stuff a developer might try some new tech to learn. However for larger shifts it'll have to be discussed in the dev group, especially if it impacts support. We do have a mature codebase so some tech dept is inevitable, but we have it as a goal to try to improve those things if we need to work on a particularly bad area of the code.
As for people leaving, in the years I've been here there's been a very stable group. So stable our customers ask us how their systems work that we integrate with, as they have much higher churn.
I don't think we're particularly unique though. But we're a relatively small company with a name that you can't flash on a CV, and at first glance our niche might sound boring.
[1]: "yea we redesigned our API, we will be doing a hard cut-over in a couple of months"
[+] [-] kulikalov|2 years ago|reply
[+] [-] Makhini|2 years ago|reply
[+] [-] jrumbut|2 years ago|reply
[+] [-] roenxi|2 years ago|reply
My interpretation here is software engineers are better at estimating tasks than I thought. The next step is analysing tasks being performed in serial to show how many sub-tasks a project can have before the estimates are effectively meaningless / likely to blow out an unpredictable amount. I bet it turns out to be about a sprint's worth of work. Real people have a knack for feeling out those sort of tipping points empirically.
[0] https://en.wikipedia.org/wiki/Inspection_paradox#Inspection_...
[+] [-] taeric|2 years ago|reply
The more people involved, the more invisible work there is. Not just meetings, but handoffs and general coordination.
And things are only predictable if they are done to the point of being known things. Rehearsed activities are known activities. Even if you shake up a part of it, people that have rehearsed are more capable of dealing with the shake ups and keeping things moving.
[+] [-] pif|2 years ago|reply
Nobody has yet had the balls to state the obvious: we have to learn to work without estimations.
[+] [-] rewmie|2 years ago|reply
Obviously, planning is critical. Planning means resource allocation. How is this not a critical aspect of any project?
The mistake you're making is presuming that if estimates are not crisp then they have no value.
> Nobody has yet had the balls to state the obvious: we have to learn to work without estimations.
This belief is detached from reality. Failing to provide estimates means a failure to scope how much work is required, which translates to not even knowing how many people should be working on a simple task.
The problem you're failing to address is limited information, and specifically limited context from developers and emerging requirements. Projects have been adjusting to this for decades. Agile developed concepts such as sprints and spikes for this very reason. It's not only about changes in requirements. It's mainly about gathering info and updating projects based on that.
[+] [-] karles|2 years ago|reply
If you don't know if its 500 or 5000 hours - you're simply not buying the product.
[+] [-] quickthrower2|2 years ago|reply
I think estimates are a problem where you have say 10 estimates, you add then up to 47 days, divide by 3 people, call it 16 days… ah about 3 weeks that’ll be done. That is dumb!
But as an order of magnitude it is useful.
[+] [-] starcraft2wol|2 years ago|reply
> have to learn to work without estimations.
A closely related, but different time management tool is a budget. Consider the same website but made in: 3 hours, 3 weeks, or 3 years. The methodology and tradeoffs the engineer approaches each of those projects is very different and needs to be known up front.
Part of the goal of estimation should be to create a budget for various key areas of the project, not guess how long it will take to fix 5 critical system bugs.
[+] [-] rkangel|2 years ago|reply
That's all well and good in a world that's pure software (and that does happen in those environments). It's not realistic in a world with lead times on physical items that are at least 1 year. For example, if you want to set up a high volume manufacturing line for a new product that takes a lot of planning in advance. You need to spend 10s to 100s of millions of dollars on a contract and schedule with the manufacturer and you need to be able to have SOME indication of where in that process firmware might be released (or more realistically - given the timescale, what scope of software is possible).
[+] [-] dennis_jeeves1|2 years ago|reply
Age old conundrum: "Who will bell the cat?"
[+] [-] spion|2 years ago|reply
Unfortunately it doesn't happen very often (mainly because tooling support for this has been missing in popular trackers such as Jira).
In situations where I was the decisionmaker on the methodology I've had excellent success with it. (using about 90%-95% confidence interval was typically right on the mark)
[1]: https://en.wikipedia.org/wiki/Three-point_estimation
[+] [-] vbezhenar|2 years ago|reply
I envy people who can work on schedule. I’m so terrible on that.
[+] [-] rewmie|2 years ago|reply
I know, hyperboles and all, but if your initial estimate for a task is 1 week but you end up requiring 4 years to deliver it then something went terribly wrong, and asking your for estimates ain't it.
Even story points scope tasks with granularities that range different orders of magnitude for this reason. One day, one week, epic/spike. It's ok if one day means 4 days, or one week means two. If one day turns to 3 years then the developer was completely clueless and unfit to continue working on the project.
[+] [-] renegade-otter|2 years ago|reply
Within two weeks you more or less can estimate accurately. Beyond the two weeks, the estimates become pretty much useless. "Three months" is not an estimate. It just can't be one in good faith.
[+] [-] jjk166|2 years ago|reply
This project is ahead of schedule, so you're not going to need the buffer, we'll reallocate resources for that time.
The project is on schedule, let's move up the deadline a little just so we have a bit of buffer at the end.
This project is slightly behind schedule, let's schedule a daily hour long meeting to go over everything to get things back on schedule.
The project is further behind schedule, how about you redo a major part of the project to try a different approach that will hopefully be sufficiently faster that you'll not only recover the cost of the reimplementation but also get back onto the original schedule.
The project has fallen even further behind schedule, and other things are starting to pile up, we need to pull some resources away to deal with them.
The project is ridiculously far behind schedule, people are losing faith in your ability to deliver, you need to get back on schedule before they'll approve more resources for this project.
The project has fallen too far behind schedule, we're pulling the plug. I've scheduled a lengthy post-mortem meeting so we have plenty of time to go over how you failed to bring this project to completion so you will learn from your mistakes.
[+] [-] jll29|2 years ago|reply
As an experienced professional, one often compares with past projects and looks at the duration of a similar one - not necessarily regarding content but complexity.
(I also conduct research on methodology, e.g. https://arxiv.org/abs/2201.07725 / https://dl.acm.org/doi/abs/10.1007/978-3-031-08473-7_48 .)
[+] [-] fzeindl|2 years ago|reply
Also check out this study which links software quality to deviations in estimation: https://dl.acm.org/doi/10.1145/3524843.3528091
[+] [-] samsquire|2 years ago|reply
I think the high level imaginary API/pseudocode of any system is straightforward and fast when you start from scratch. (Because you're not focused on detail and just intent)
But when you start from an existing codebase, existing infrastructure, existing APIs, it slows to molasses. You've got to understand and have a mental model of ALL OF THAT before you can complete your intent.
I think this is why software takes longer than expected. Your mental model has to be accurate, and there is always more to learn.
How do you solve the problem?
I'm an advocate and trying to design what I call "commutative programming", which is another way of saying that the behaviour is the product of every statement about the behaviour that is desired, not an explicit instruction of what to do next which is what modern programming is and it is slow and tedious and doesn't compose. In this dream, if you want to change behaviour, you just add statements. Programming is additive! Not edit sourcecode or try to find out where to insert your lines of code or tweak existing lines. Let the computer tell you about conflicting requirements or missing criteria.
We need query engines and cost based optimisers for behaviour.
Imagine taking lots of files that are simple "writing to an imaginary API" - that's your intent specification. You just need to merge them together and your system is finished.
I started work on a commutative GUI https://github.com/samsquire/additive-guis but I'm also thinking from time to time about commutative code where we define refinements to desired behaviour and the computer behaviour query engine or rule engine generates the code or configuration that exhibits this behaviour. (A bit like Kubernetes for behaviour.)
In the future: we don't get LLMs to code for us, but we have highly expressive semantic vector spaces that map to architectural relationships and meaning of behaviour, and changing the architecture of a system to support new behaviour is a matter of changing the serialisation of semantic vectors via the prompt.
[+] [-] hcarvalhoalves|2 years ago|reply
I believe you'll enjoy reading "Notes on the Synthesis of Form" by Christopher Alexander [1]. There are many ideas in it, but at the later chapters the author proposes a system for solving design problems as a large graph of interacting requirements, where smaller sub-graphs are (hopefully reusable) components solving a smaller set of requirements.
Sounds similar to this idea of yours, just with a key difference in that the author sees it as a two-pass solution: first you "carve" out your solution by establishing a context boundary, and then do you "fill" it with an implementation - such that the solution is the minimal implementation that fits the context, and not anymore.
After reading the book, I have also adopted a similar approach (but with less formalism) to gather requirements for projects in the past.
I have thought about developing the ideas of the book further, if you find this subject interesting we can discuss (I can ping your profile's email). I'm certainly interested in hearing more of your own ideas, as it seems to intersect.
[1] Same author of "A Pattern Language", that inspired the design patterns concept in software
[+] [-] tcard|2 years ago|reply
[1] https://witheve.com/
[+] [-] vjaswal|2 years ago|reply
That is, I can see how to solve a problem in the common case. But as I code, I encounter the myriad of error cases, exceptions to the common problems, and especially the interactions of all these. Then add in performance considerations and the complexity can multiply quickly. The long-tail of those errors and exceptions can be very time-consuming to solve, so the mean is likely to be way off.
It's obviously much easier to estimate when you're experienced with a problem space. This should mean that you know how to solve a particular problem as well as the likely exceptions (non happy-path) you'll encounter.
If you're experienced with the domain, you can fit the relevant partial solutions in your head and more accurately judge what could go wrong with your code. But more generally I think this is similar to solving problems in other domains as well, such as architecture, building a car, or anything with many moving parts. These domains do not usually have loosely coupled sub systems.
[+] [-] sokoloff|2 years ago|reply
There's a similar story in the mechanical engineering for buildings. Sure, some architectural choices will make it harder or easier, but most buildings can be retrofitted system-by-system over the decades a building is in service.
Part of that is to allow configuration pre-sale, but a lot of it is just "it's too damn complex if everything affects everything else", which is not that different from software.
[+] [-] azornathogron|2 years ago|reply
But my big question from this article is... why isn't this a well known aspect of project planning, scheduling, and software engineering? It's not even specific to software really - I mean it's possible that software tasks have higher variance than others but I'd be surprised if task variance is irrelevant for scheduling non-software projects.
Is it really the case that no one has applied mathematical modelling to the problem of project planning, scheduling and estimation before? Do the ideas in this blog post represent the core of a new avenue of research? Seems unlikely. So how come this, or something like it, wasn't taught as a basic and critical part of project planning, in the same class that taught me the term "Gantt chart"?
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] markhalonen|2 years ago|reply
[+] [-] magicink81|2 years ago|reply
Their book is available for free online, here's a link to the chapter "Estimates don’t show uncertainty " https://basecamp.com/shapeup/3.4-chapter-13#estimates-dont-s...
[+] [-] rnburn|2 years ago|reply
Research has shown a power-law distribution to be a better fit.
See "The Empirical Reality of IT Project Cost Overruns: Discovering a Power-Law Distribution" https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4204819
[+] [-] d3ckard|2 years ago|reply
It’s as simple as that.