top | item 40584901

Study finds 268% higher failure rates for Agile software projects

221 points| johnsutor | 1 year ago |theregister.com

190 comments

order
[+] CSMastermind|1 year ago|reply
Obviously, it's a biased study that shouldn't be taken seriously, but while we're on the topic of Agile:

Consulting Agile, grew out of web development consulting firms as a defensive methodology for managing shitty stakeholders. They incorporated some genuine industry best practices, but they also have a lot there that's simply meant to blunt the worst impacts of having to deliver software in an environment where the person paying you doesn't understand the basics of software development (or project management really).

It actually works pretty well in those environments - you don't get hyper-productive engineering teams, but you do get somewhat consistent delivery and a stable enough working environment that the worst outcomes (not delivering anything at all) are avoided.

No large tech company that I'm aware of implements this type of Agile but plenty of big non-tech companies do. If you have a CIO in your company and the business refers to software engineers as 'IT' there's a good chance you do this. This is inefficient but it's probably good enough for those companies, especially if it's established, the switching costs alone would be a nightmare.

If you're using it as a startup then something has gone terribly wrong and you should seriously reevaluate what you think the future of the company will be.

[+] tinco|1 year ago|reply
It's not just a biased study, it's an article about a study of which the data is not published, so it's impossible to tell if there's any merit to it.

The excerpts of the study published in the article do not inspire confidence either:

1. Agile ("Agile Requirements Engineering") is defined as: Development starts before clear requirements, no complete specification, significant changes late in development.

I think even those who are opposed to Agile would agree that's a severely lacking if not completely incorrect characterisation of Agile development practices. It sounds more like a list of things that would cause an Agile project to fail.

2. Impact Engineering is defined as: Use of all engineering practices studied which increase success rates.

Why doesn't the author dare to give any properties of Impact Engineering up front? This sounds like he's just baiting for people to buy the book so they could learn what magical engineering practices are studied therein. It smells scammy.

[+] giantg2|1 year ago|reply
Yep, and the big companies basically use Agile as an excuse for not thoroughly planning and vetting ideas. Just build it and figure it out as you go. Then rebuild everything in 2 years since you're left with outdated and unmanageable spaghetti code and you burnt out all your SMEs so they took the knowledge with them.
[+] marginalia_nu|1 year ago|reply
The by far strongest correlate of a complete clusterfuck of a project is how often, per meeting, agile is invoked.

In fact, I've discovered you can use agile as a sort of meeting hand grenade, if you don't like the direction a meeting is headed, like they're about to decide on something stupid, you can just throw in "wait, is that agile though?" and the rest of the meeting will discuss methodology, never arriving at any sort of conclusion.

[+] brightball|1 year ago|reply
Yea. I do Fractional CTO consulting and usually have good results with developers because I focus heavily on removing bottlenecks, unhelpful processes and meetings, etc.

The challenge is always when you run into sacred cows from other levels of management.

- Hard deadlines while acknowledging we don't have enough information to commit to those deadlines

- Story points as a measure of time

- Absolutely overloaded teams being asked to deliver new features without being given time to improve underlying issues in the system...and then having to constantly stop what they are working on to deal with production issues caused by those underlying problems.

It's like clockwork sometimes and the conversations are always hard.

[+] cjk2|1 year ago|reply
This is my methodology. Set two PMs on each other and go and do some work that generates ROI.
[+] dmix|1 year ago|reply
So kind of like the stereotypical communist meetings where they spend all their time in meetings debating each other and obsessively discuss the finer points of socialism instead of ever accomplishing anything.
[+] kevin_nisbet|1 year ago|reply
I'm not really sure what to make of this article, it says itself it's a study commissioned by promoters of another methodology. If they're consultants trying to pitch their consulting services, they need these types of things to sell.

But my fundamental reaction is, what does failure mean. Because in my world view failure should be expected and accepted and learned from. It's entirely possible to spend a bunch of time avoiding every possible failure mode, and really not delivery much value at all... but we've successfully avoided the work being considered a failure.

[+] signal|1 year ago|reply
- Based on a random survey of software engineers (who somehow had visibility to project inception & outcome)

- Doesn't define project

- Success was defined by not violating the iron triangle (so nobody knew what they were saying OR they're working on trivial "projects")

- If you read reviews for the book it's totally AI generated nonsense

[+] commandlinefan|1 year ago|reply
> what does failure mean

I've been reading that 50+% of software projects "fail" ever since I started programming in 1992 (long before the term "agile" crept into our lexicon). They consistently fail because they consistently have deadlines but no actual definitions. That was true before Agile, was true during Agile, and will be true after Agile.

[+] fisf|1 year ago|reply
> But my fundamental reaction is, what does failure mean. Because in my world view failure should be expected and accepted and learned from.

That's fine in some domains. It's not acceptable in others (i.e. you cannot just try things and see if they stick).

But regardless of that, you still want to minimize failure. So if one methodology (Agile) pushes to spend less time on fixed and clear requirements upfront, and this leads to higher failure rates, this is still an issue.

[+] pavel_lishin|1 year ago|reply
The article also doesn't go into what kind of software was being produced, etc.

If you're building control systems for the ISS, or an interplanetary probe, of course you need requirements hammered out pretty well before you start development; you can't just continually push updates out incrementally.

But if you're building Yet Another CRUD App at your startup's third pivot, then getting every requirement written down beforehand is virtually impossible.

[+] rawgabbit|1 year ago|reply
This is the issue of agile. It is a blanket prescription for every scenario which of course … fails.

If you are talking about customer expectations, yes, you can manage customer expectations. As long as you are honest and upfront, customers tend to be forgiving.

On the other hand, if you are talking about a core API or building block that downstream APIs will depend on, then no. You can't deliver a API that writes to the database/storage layer that only works 90% of the time.

[+] boxed|1 year ago|reply
> "Our research has shown that what matters when it comes to delivering high-quality software on time and within budget is a robust requirements engineering process and having the psychological safety to discuss and solve problems when they emerge, whilst taking steps to prevent developer burnout."

I almost laughed out loud. Isn't that agile? :P

[+] oriel|1 year ago|reply
Probably the originally intended Agile [1]

These days its been devolved into micromanagement musical chairs of pain. The kind that promotes what it professes to prevent and provide.

[1] https://agilemanifesto.org/

[+] Aurornis|1 year ago|reply
Every agile coach always tells me that agile is whatever works for the team.

Conveniently, if something isn’t working they dismiss it as not true agile.

Of course, all of the processes and meetings they push on to the team don’t actually work out, but they’re long gone by then.

[+] giantg2|1 year ago|reply
"robust requirements engineering process"

I worked on one team that had this and it was amazing. It implicitly helped with burnout too. It was an "Agile" team, but we followed more of a scrumifall model that worked well for that project.

I've been on a bunch of Agile teams (some more than others) since then, but the requirements processes are all garbage. Then the leaders wonder why things take so long, there's so many bugs, burnout is high, etc. It's really not surprising.

[+] fook-dang|1 year ago|reply
"prevent developer burnout."

...Frankly I think all the standups & the pressure from constantly asking me if I am contributing (every single day?!) instead of trusting that I am... burns me out.

I don't mind 2-3 standups per week, 15 mins max.

But 5x 30 minute standups... Is exhausting. It feels like a lack of trust from product managers.

Oh yeah... and unqualified, zero engineering experience, diversity-hire product managers. That's super annoying-- being managed by people who did not obtain their role via the route expected of an engineer (having demonstrable engineering skills)

[+] karaterobot|1 year ago|reply
> However, while the Agile Manifesto might have its problems, those stem more from its implementation rather than the principles themselves.

Is the author an Agile coach? This is the classic response when people observe that Agile has failed to transform their team: you're doing it wrong, the problem is you. When systems seem good in theory but tend to fail in practice, you should just try to learn what you can from them and move on to the next thing.

[+] thaanpaa|1 year ago|reply
No, it really is true. Agile is widely misunderstood. The way scrum is implemented in 90+% of the cases is a complete mockery of what it was supposed to be. A lot of coaches completely miss the mark as well.

And the fundamental problem is that agile is completely incompatible with the way most enterprises are built. A truly agile team talks to the users of the product, and they have complete control and responsibility over the development process. That just doesn't happen because managers want to manage, and as much as developers would like freedom, they'd often rather not have the responsibility.

[+] Sammi|1 year ago|reply
If a process fails most of the time, then the process is wrong. You can't keep blaming the implementors.
[+] hk1337|1 year ago|reply
Honestly, 99% of the time people are doing it wrong. People are either too strict with it or too "loosey goosey" with it.

*EDIT* It doesn't have to be correct, it just has to work for the people on the team. If it works, then use it, if it doesn't then change it. That's agile.

[+] pydry|1 year ago|reply
That certainly applies to Scrum but agile is more of a religion than a system. It's a bunch of extremely vague "principles" whose meaning is often anyone's guess.

What constitutes agile is basically a matter of opinion.

[+] signal|1 year ago|reply
- Based on a random survey of software engineers (who somehow had visibility to project inception & outcome)

- Doesn't define project

- Success was defined by not violating the iron triangle (so nobody knew what they were saying OR they're working on trivial "projects")

[+] gortok|1 year ago|reply
The study isn't shared, so we can't see the methodology to determine its accuracy. We see the results of the survey; but that's not a study, and the survey itself isn't shared with us.

I don't dispute that agile has its issues -- and I'd be surprised if 'agile' failure rates weren't an issue. We can empirically see the result of 25 years of 'agile', and it's not somewhere we should aspire to be.

The issues comes down to this:

1. Is adopting an 'agile' methodology sufficient to ensure (insert goal here)? 2. Has the software industry adopted a method of operating that creates a statistical quality control; and adopted practices that investigate and remediate special causes and common causes of variation in quality?

To spoil it: for #1 the answer is 'no', and for #2 the answer is a resounding 'no'.

There are practices that are ingredients to the world described in #2, but we as an industry have not yet adopted a systematic way of thinking and resolving problems in the software development process.

I think we'd be better off adopting Deming's method of management and his System of Profound Knowledge and be far better off than adopting the agile fad of the week; even as I admit I still have open questions on how to apply his work to Software. (See "The New Economics" 3rd Edition, and "Out of the Crisis", both written by Deming).

[+] JohnMakin|1 year ago|reply
In any discussion on this site or others about Agile, you will find a flurry of PM's rigorously defending it with the typical line of reasoning -

If you tried agile and it failed, it wasn't actually agile. When you show how it was actually agile, you didn't do it right. When you show you did it right - go back to argument 1. It's a really circular no true scotsman style of argument.

Obviously this "study" is not good or even really a study, but I think the general arguments it makes aren't tremendously out of line. I have been an IC on agile teams, an IC on more traditional teams that I guess would be called "waterfall," and I've been tasked with implementing a scrum process both as a team leader and as a PM in my career - I've personally never seen it work on Ops/Infrastructure teams. In fact, I'm convinced it can't. It's far too difficult to determine the scope of work, far too interrupt driven, far too difficult to measure actual velocity, way too easy for engineers and managers to bullshit the system.

All the arguments I see in favor of it over the years strike me as pretty dogmatic. The way I like to manage projects these days is a kind of kanban system with a task queue, and 1-2x a week brief cadence meetings to discuss priority/alignment and any blockers.

edit: should probably clarify I'm using agile == scrum here, which I know isn't technically correct, but that's how I've seen it implemented in 99% of cases so far in my career.

[+] whynotminot|1 year ago|reply
> The way I like to manage projects these days is a kind of kanban system with a task queue, and 1-2x a week brief cadence meetings to discuss priority/alignment and any blockers.

To me, distilled down to its core, agile is all about faster feedback loops and course corrections.

If you’re regularly getting good, actionable feedback on your work, and maintaining alignment, I think you’re living up the spirit of agile.

There are plenty of “agile” teams going through the motions, doing the ceremonies, but not actually reaping what the real benefit is supposed to be: frequent, actionable feedback that helps guide and align the next phase of work.

[+] lloydatkinson|1 year ago|reply
I agree with you completely and what you said brings to mind something I wrote on my blog a few days ago:

> I posit that if I showed the Agile Manifesto bullet points (without the title) to a group of people with cringe-worthy titles like Scrum Master, Agile Delivery Manager, Agile Coach, or Head of Agile Delivery, and asked them “Is this agile?” almost all of them would say it isn’t. I can only imagine the effect that revealing the title and explaining that, actually, this really is the agile, would have on their Jira-addled brains.

[+] Attrecomet|1 year ago|reply
>If you tried agile and it failed, it wasn't actually agile. When you show how it was actually agile, you didn't do it right. When you show you did it right - go back to argument 1. It's a really circular no true scotsman style of argument.

It'd be interested in seeing that argument in the wild. Mostly I see complaints like yours when people point out that an hour-long daily that the manager uses to harangue the developers isn't actually agile, and that is perfectly true. Or that a blame-assigning retro or measuring dev productivity using story points is not actually agile, which is at least true insofar that it clearly is forbidden in Scrum, and stupid in any case.

Though I find it interesting that you end with kanban -- an agile methodology. It seems you do like to use agile after all, just not scrum.

[+] brazzy|1 year ago|reply
> The way I like to manage projects these days is a kind of kanban system with a task queue, and 1-2x a week brief cadence meetings to discuss priority/alignment and any blockers.

Sounds pretty agile to me.

While the "defenders" of agile might be employing No True Scotsman fallacies, it's detractors (which I personally seem quite a bit more numerous on HN) are often doing the same in reverse: refusing to define what "agile" actually means, and just throwing together a bunch of anecdotes and feelings about excessive meetings and estimates being misused to measure productivity.

[+] affinepplan|1 year ago|reply
this smells very much like selection bias, in that it seems pretty plausible that

* if you have a project that is Very Important To Succeed, a team might be more likely to adopt waterfall

* if you have a project you just want to trial and don't care if it fails, maybe agile practices are more likely

[+] btutt|1 year ago|reply
I don't agree that on time and on budget are the correct measures for determining overall project success or failure. By these standards a multi-year project delivered a single day after the original estimate, or a massively profitable project that went a dollar over the original budget are counted as a failed project. Ideally some actual business outcomes need to be included before determining if a project succeeded or failed.

I think this problem persists for a lot of research and discourse about software project success and failure where we conflate whether we estimated accurately with whether or not the project succeeded or met the needs of the business.

[+] paulsutter|1 year ago|reply
Clear requirements are needed regardless of methodology. The simpler (briefer) the requirements, the more likely to be understood and followed. Iterative development processes can leverage more concise (and thus clearer) requirements

> One standout statistic was that projects with clear requirements documented before development started were 97 percent more likely to succeed.

Also, if a project doesnt have requirements, how can it either fail or succeed?

EDIT: agile is one type of iterative, and iterative exists in contrast to waterfall. go ahead flame away :)

[+] msurekci|1 year ago|reply
"One standout statistic was that projects with clear requirements documented before development started were 97 percent more likely to succeed"

The issue with a majority of projects are the requirements aren't always clear from the start or even if they are it tends to deviate while the project is in progress. Agile tries to mitigate that by allowing teams to be able to react more easily to changing requirements.

That being said not many companies understand or implement agile correctly, unfortunately.

[+] giancarlostoro|1 year ago|reply
Most times I've been in a doomed to fail agile project, it was almost always a manager or higher who overpromised with deadlines, not accounting for real world blockers. One of said projects has insanely awful app store ratings for their mobile apps.
[+] wild_egg|1 year ago|reply
Article source seems to be mainly an ad for some agile alternative which makes the whole thing suspect.

Most of the "results" can boil down to: scope creep kills projects. That's entirely independent of whatever methodology your team follows and they don't make it clear how their "Impact Engineering" concept would address it

[+] moi2388|1 year ago|reply
I’m sure that agile done right is great.

I’m also fairly sure most organisations don’t do Agile right.

At least within my organisation, we do not design anything up front. We’re agile.

We don’t think about proper api modelling. We’re agile.

We also do barely any testing. We’re agile.

We do rewrite the UI a dozen times based on user feedback. After all, we’re agile.

[+] thaanpaa|1 year ago|reply
> At least within my organisation, we do not design anything up front. We’re agile

Agile doesn't mean forgo design completely.

> We don’t think about proper api modelling. We’re agile.

See previous point.

> We also do barely any testing. We’re agile.

It's difficult, if not impossible, to be agile without testing. If you want to move fast, you want to be confident that your latest change didn't break anything.

> We do rewrite the UI a dozen times based on user feedback. After all, we’re agile.

Sounds like you built a complete UI before any users could give you feedback. That's the opposite of agile.

[+] trickstra|1 year ago|reply
Can you point to any of the 12 principles which says that we should do "barely any testing"? I can point to some that say "deliver working software" and "it's the only valid measure of progress". If testing helps you deliver working software, then do testing, that's agile.
[+] zzzeek|1 year ago|reply
who really does "Agile" or "waterfall" or anything ? I'm in this job for 30 years. I've never seen any formal methods my whole career. it's like here's the thing we need or what the customer needs, gather some vague requirements, write some prototypes, do internal demos, now more requirements are there, more clarity, iterate on the prototypes, etc., repeat. use issue trackers, use "stories", use kanban boards sure, but are we adhering to some rigid notion of "agile"? no way. I can't imagine the "waterfall" method actually working either.
[+] oytis|1 year ago|reply
V-Model is a variation of waterfall which is still used in regulated industries (automotive, medical, aerospace etc.) as far as I know.
[+] renegade-otter|1 year ago|reply
Agile in itself is not a development methodology - it's a communication one. A small team working on a stealth R&D project, where everyone is same rank and in the same boat, would be suicidal to use Agile. This is what Kanban is for.

Agile is an overhead. It doesn't make things faster, or more efficient, quite the opposite. It's there to avoid arbitrary deadlines coming from the management and developers not working on an island, with no visibility. That's a recipe for some nasty surprises. "But we thought you were working on THAT and it would be ready NOW!"

[+] macNchz|1 year ago|reply
It seems like this is in service of promoting some new competing methodology. I also can’t find any details on the research methods, but it seems like it may be relying on software engineers’ judgment of the success/failure of projects they’ve been involved with, which…I think is pretty questionable. The reality is that there are a lot of software engineers who might describe a project that was hugely profitable as a failure because it has a shitty database schema and lots of dead code because of a last minute change.
[+] ChrisMarshallNY|1 year ago|reply
I've always liked the Agile Manifesto, but it's fairly vague, and leaves a lot of details to be provided at implementation time.

I suspect that a lot of Agile is actually "agile™." A branding facade, over an unstructured, ad hoc development system. Not even Waterfall, which is actually a very robust system (just robustly inflexible, which often gives bad results).

I like the idea of evolutionary design, and adapting to change, but I have found that it needs to be done carefully, and that having experienced developers; concentrating on results, is a must.

As always, I think the proper answer is "It depends." The search for The One True Methodology is one that will never be satisfied. Even different projects, under a single organization, probably need to take different approaches.

I just believe that we always need to keep our eye on the end result, and all development needs to be done, with that in mind. I think that (for me), Agile is accepting that we don't actually know what "done" looks like, when we start, but we have to have a heuristic to help us to understand when we're done.