top | item 14085716

Cargo Cult Agile (2008)

111 points| ekzy | 9 years ago |jamesshore.com | reply

109 comments

order
[+] struppi|9 years ago|reply
I have been thinking about this a lot, ever since I first read this article, some years ago.

Most of my freelance projects and technical coaching gigs in the last few years where at companies that were in or after their "Agile transition". And most got "Agile" horribly wrong.

I think cargo cult is an issue, but the problem goes deeper. I wrote a conference talk titled "Your Company Will Never Be Agile" around that topic. The gist is:

Many C-level executives want "true business agility" for their company: They want to be able to react quickly when circumstances change. Many "workers in the trenches" (developers, testers, ...) want to be able to do good work and provide value to the customer.

But in established companies, the organizational structure, processes and policies, annual budgets, KPI and bonus systems and middle management prevent the company from becoming truly agile.

So, young, small companies are more agile, because (among other things) they did not become a bureaucracy yet, they are mostly run by engineers with not much middle management, and because of survivor bias (they do not have a war chest yet, so those that were not able to react quickly do not exist anymore).

Unfortunately, there is no English recording (but the slides are here: https://speakerdeck.com/dtanzer/your-company-will-never-be-a... ).

[+] taneq|9 years ago|reply
> Many C-level executives want "true business agility" for their company: They want to be able to react quickly when circumstances change.

True, and this isn't a bad goal in and of itself, but what they don't realise is that these 'quick reactions' (aka fundamental low level changes in the product) may require orders of magnitude more work than said executives expect (and in fact, probably the work required is at least linear in the age of the company).

> (they do not have a war chest yet, so those that were not able to react quickly do not exist anymore)

I'd argue that having a 'war chest' is actually detrimental to agility (if you're thinking of agility as 'the ability to perform at our current level in a different field'). Small new companies can pivot extremely fast because even if it means throwing away their whole codebase, they've only lost a few months' work.

[+] isoos|9 years ago|reply
This matches my experience, although I was lucky that I stopped doing "Agile transitions" gigs early enough. My question is: how do you discuss this with a customer?

At one point I had a talk with a director of a smaller bank, and he was enthusiastic to bring me on board, even wanted to pay me just to spend my workday in the office doing nothing specifically. I declined, because it was a lost cause: their team was too fragmented, their processes were too convoluted, their estimate for a very minor change added up to several months...

Yet, I felt bad about it, because I wanted to help them, but in that environment, I was not able to do anything. How do you approach such situations?

[+] ryanmarsh|9 years ago|reply
Agile coach here. You nailed it.

One way of looking at these companies is that they're organisms adapted to their environment. If there are weak selection pressures around agility and software delivery in their market "environment" then you can expect that they're well adapted to other pressures (i.e. credit risk, or buying labor at one price and selling it at another).

Often times these businesses might see reasons for agility, opportunities they could take advantage of, still they really aren't feeling selection pressures so there's no real need to change. How they deliver software with really bad agile is "good enough". Well it will be until it isn't and then it might be too late.

So you have all the structures and processes that make one organism good at being a bottom feeder and it's like pushing water up a hill to make them change when they REALLY DONT HAVE TOO.

Every once in a while I have a client where this stuff actually matters to them and we have a lot of fun and make amazing stuff.

[+] exelius|9 years ago|reply
Actually (and I do this for a living as a management consultant) we are pushing all of our clients to an Agile operating model. There are many reasons for this, but the primary one being that consumers now expect to interact with all companies like CPGs.

Agile lends itself very well to a top-down brands-and-products hierarchy. You build the core of the products in a shared engineering group (again, just like most CPGs) then apply localized branding and product focus as needed.

And here's the thing: adopting Agile in the corporate world is less about "reacting quickly" (though that is one of the benefits, at least in comparison to the legacy methods) but really more about cost control. Agile is simply cheaper because you're not building shit that nobody wants. If you can build an Agile culture of "if it's not in the backlog that was approved by product and reviewed every couple weeks, you don't build it" you cut down on a lot of pet projects, deprecated features and duplication of effort.

Is there still duplication? Sure; but often times companies will have 5 groups build the same thing and they just choose the one that gets the most internal traction. That's actually a viable strategy when you don't know the product you need to build, but you need it to work. It also tends to help find the right "home" within the organization politically (i.e. the people who really need said functionality will build it, use it, make it popular, and then get the money to maintain it while fringe users lose funding and migrate to the shared platform).

The key thing is that the engineers aren't in control. And with good reason; engineers are paid to build products, product managers are paid to design products and brand managers are paid to build portfolios of interrelated products. You cannot run a large company without this kind of P&L discipline.

[+] tomelders|9 years ago|reply
I could wax lyrical about Agile all day long. If you google 'Agile Bullshit', a comment I wrote on HN a few years ago is usually in the top three results, and because of that I get an email once a month from someone driven to despair by said 'Agile Bullshit' who feels compelled to get in touch with me and vent a little. I enjoy the emails, and I think they've given me a much broader perspective on agile than my own experience alone could offer.

I've been writing an article about Agile for years now, that I can never finish. But here's what I consider to the be the two biggest problems with Agile.

1. Agile can never work in an agency, where the client has control over budgets, deadlines, and features. There is simply no incentive for the client to compromise when your agency has made some pretty ridiculous promises and put themselves on the hook to deliver.

2. There are some teams that deliver well under agile (as per the metrics that the business has defined), and some that don't. In my experience, the ones that deliver well produce the worst work because they won't push back on a design or a feature that's incomplete, nonsensical, or plain impossible. They do what they're told and add no value to the overall output. This is especially true of offshore development teams.

The teams that produce the better software do push back, but the Agile process as implemented everywhere on earth does not support them. Instead, animosity grows between the developers who push back, and the product-owners|BA's|designers who can't or wont understand the problems with their own work.

But! Some process is better than no process. And I've worked in some companies that take Agile very seriously and are willing to put in the hard work to make it work. And I suppose that' the secret sauce that's missing from most organisations that are "Agile", they think it's a trick, a hack, a cheat, a short cut.

It's not. Like everything else worth doing, it's hard work.

[+] jaggederest|9 years ago|reply
> Agile can never work in an agency, where the client has control over budgets, deadlines, and features. There is simply no incentive for the client to compromise when your agency has made some pretty ridiculous promises and put themselves on the hook to deliver.

There's an interesting triad that I've been talking to people about a lot recently.

1. Authority (the ability to make decisions of importance)

2. Accountability (a way of telling if you are doing better or worse)

3. Responsibility (pride and reward proportional to the results you deliver)

Essentially when you lack any of those three, you cannot operate effectively.

Without accountability, you have no way of knowing what success is. Without authority, you have no power to recruit help or make the necessary changes to be successful, and without responsibility, no amount of accountability or authority matters.

This is especially true in asymmetrical relationships like customer-vendor and employee-employer.

[+] seanwilson|9 years ago|reply
> 1. Agile can never work in an agency, where the client has control over budgets, deadlines, and features. There is simply no incentive for the client to compromise when your agency has made some pretty ridiculous promises and put themselves on the hook to deliver.

Yes, I've worked at places where they'd accept fixed budget and short timescale projects while also preaching about the benefits of agile to the client. This meant there was minimal upfront design and the customer was given a chance to change their mind often. This lead to crunch time and having to make changes that could have easily been predicted if planning had been done from the start. If you suggested the latter advice though, you'd be accused of being a "waterfall" practitioner and ignored so it felt very cargo cult.

My view is that projects with short time scales where it's predictable early on what all the functionality should be, you should plan out as much as you can from the start and get the client to OK as much as you can early on (e.g. with mockups, interactive prototypes). Agile is more suitable for larger projects with flexible budgets where there are so many unknowns that planning ahead isn't productive.

[+] flukus|9 years ago|reply
I'd go a step further and say that agencies can never work, at least past the small project or website level. The customer is oblivious to the maintenance required for a software project (even if it's just maintaining developer familiarity with the code base) and the agency can only ever get them to pay for features.
[+] parasubvert|9 years ago|reply
For #1, I'd say a dedication to Agile requires significant business commitment on the contractual side to ensuring there basically ARE no commitments beyond the backlog/stories. Needless to say, some clients can't handle this and you need to to turn business away.

There are agencies that act this way like Tribalscale or Pivotal Labs, both of which are pretty dedicated XP shops but also with significant design and product capabilities.

[+] chrisan|9 years ago|reply
I have definitely lived through #1. I started out at a "web agency for marketing agencies". Most of the time the actual client never knew we existed. Budget and features were set up front and the deadline was immovable as other forms of media would be directing to our apps/webpages starting on a specific date.

Despite all of that we called ourselves agile. This amounted to pretending to be agile for 90% of the project and then as the deadline approached just staying up late/working weekends to make sure it was complete and out the door. Naturally it was worse when we had competing deadlines between different projects. And when do different clients ever have deadlines that fit nicely in between other clients?

Being at a "product company", it is much easier to be agile about stuff where features and/or deadlines can be shifted

[+] misja111|9 years ago|reply
Well said. The funny thing about point 1. is that the Agile Manifesto implies already that it could never work: "Customer collaboration over contract negotiation"

Nevertheless it doesn't stop many agencies from claiming that their fixed price projects are agile.

[+] danpalmer|9 years ago|reply
Agile is the only word I know that inverts its meaning when you capitalise it.

The team I work in is fairly agile, in that we communicate well, react to changing priorities very quickly, and ship things regularly. We're not perfect by any means, but whenever we consider adding more of the typical "Agile" process we always realise it would slow us down, and we see other companies with more set "Agile" processes being much slower.

[+] mpweiher|9 years ago|reply
Yes. The funny thing is that at a company I worked for in the 90ies, we were very agile, before that became a thing, and it was GREAT. Empowering, fun, engineering-driven, effective, efficient. The things we did with our tiny team were amazing.

When I read about XP, what I read very much resonated with what we were doing, without ever having a name for it other than "what we do and like that works well".

However, whenever I've seen Agile applied at other companies, it was horrible. And yet they refer to the same words. Someone recently told me "Agile/XP/etc. sound like slaves trying to figure out better ways to please their masters". And when I take that frame of reference, yep, that is exactly what it sounds like.

So something weird is going on with those words.

[+] theprotocol|9 years ago|reply
I enjoyed your comment. The rush to formally define and implement "agile" is pretty hysterical on a fundamental level.
[+] _betty_|9 years ago|reply
I typically find the same about Software Architect vs software architect - eg the Job Title vs the activity that decent developers do.
[+] flukus|9 years ago|reply
I'd argue stand up meetings are the opposite of agile, the insistence on them is literally breaking the first rule of the agile manifesto:

> Individuals and interactions over processes and tools

Probably breaking the fourth too:

> Responding to change over following a plan

How many companies have done a retrospective on whether the daily recitation of Kanban board movements​ is providing any value?

[+] jwdunne|9 years ago|reply
To me, a lot of Agile breaks the first rule. A Certified SCRUM Master comes in and overhauls your processes. That is, in effect, processes and tools straight up, from the minute that scrum master goes on a course.

It wouldn't be much of a course if it didn't teach some process for implementing SCRUM.

It wouldn't be much of a consultation if the consultant tells you to go speak to your customers and find out what is really bothering them and how you can deliver working software quickly to alleviate that.

So you have daily standup a, weekly retrospectives, a designated master, product owner, a process for estimation, a n-week sprint. Those are must haves. That's all process.

[+] Swizec|9 years ago|reply
> I'd argue stand up meetings are the opposite of agile

Those 15min when the team gets together and talks about what they're doing and where they need help ... that's the most useful meeting I've ever been a part of. 80% of all other interactions could be removed with a kanban board and some patience.

Like, shit man, sometimes it takes me more than 3 minutes to reply on Slack because I'm doing shit. Go do some stretches or something ...

[+] PJDK|9 years ago|reply
We've talked about the stand up a few times in retros. We've mixed it up a few times (we now do a task based stand up rather than a person based one) but always felt it was more valuable than not having it.
[+] Nursie|9 years ago|reply
To echo another comment on here -

Daily standups have been the most useful development I've seen in my 17 year career so far, I'm not sure why there's the hate for them.

It's a small time allotment where everyone gets just a couple of minutes to sum up where they were yesterday, where they're going today and what's holding them up.

I've seen them go wrong, when managers decide that they need to attend and use part of the meeting for "management announcements". I've seen it go wrong when one team member cannot shut up and regularly took over the whole meeting (yes, the scrum master was ineffectual). But in general I've found the daily rapid-fire catchup a thing of beauty.

The rest I can take or leave. One of the worst projects I've ever observed had a highly paid SCRUM Master/Consultant/Trainer and that team seemed to spend several hours each day on process. They failed to deliver anything, ever, the product was scrapped and everyone was laid off, including the development manager.

[+] dasmoth|9 years ago|reply
Daily standups have been the most useful development I've seen in my 17 year career so far, I'm not sure why there's the hate for them.

Two things that bother me:

1) You either have to hold them at the start of the working day (in which case they become a synchronization point and everyone has to turn up at the same time -- which will inevitable be a bad time for some people) or a bit later (in which case you end up with an odd slot at the start of the day which usually isn't long enough to start on anything substantive).

2) They mandate daily visibility, and (by design, I think) prevent people going off for a week or two and solving something substantial on their own.

There are possible solution to (1) -- e.g. asynchronous stand ups in a chat client (and it's probably not coincidental that the only times I've found these things at all helpful, they've followed this model rather than the actual stand-up-in-circle thing). But (2) is kind-of the whole premise of "agile teams", so not sure what can be done about that.

[+] humanrebar|9 years ago|reply
> ...everyone gets just a couple of minutes to sum up where they were yesterday, where they're going today and what's holding them up.

Peer-to-peer, listing blockers is more or less the only thing that matters, followed distantly by making sure we're not stepping on each other (deploying the same thing, refactoring the same class).

It's also important to sync up that designs are good and work together, but that's outside the scope of Agile (and standups).

The bulk of the info in the yesterday-today-blockers standup is to let the manager get a quick rundown of what the employees are up to. It's a management tool, basically. And that info is all duplicated (if you're doing it correctly) in the agile board, the burndown chart, and in the version control system. So management could just create better queries and dashboards on top of existing data and get the same information.

[+] flukus|9 years ago|reply
> It's a small time allotment where everyone gets just a couple of minutes to sum up where they were yesterday, where they're going today and what's holding them up.

Don't you have a kanban board or equivalent where you can see what everyone's been up to and what their upcoming tasks are? Do you need to know what's holding them up if it's not you?

For those of us complaining, it's not that we don't see the value, we just see other, arguably better ways to get the same value.

[+] overgard|9 years ago|reply
Maybe its useful to you, but to be honest, I cant remember a single thing about the standup meeting I had today and I made a point of paying attention. People talk about like "blockers", but if something is blocking me I just go talk to whoever can fix that, and if I need to know about something I just ask people to loop me in. I dont need agile for basic communication skills
[+] valuearb|9 years ago|reply
I had a standup at a 24 person company where every single person (except the CEO of course) had to attend standup and report. That's a long standup, esp. when someones report spirals into a discussion.

Wasn't very agile.

[+] cvs268|9 years ago|reply
Based on my personal experience, most projects are doomed from the start due to these 2 MAJOR problems:

1. Not prioritizing the task of gathering, decomposing and documenting use-cases at the project kick-off.

2. Chronically under-estimating effort required.

[1] http://thecodeartist.blogspot.com/2015/04/use-case-is-everyt...

[2] http://thecodeartist.blogspot.com/2017/04/effort-estimation....

[+] BaronSamedi|9 years ago|reply
I often see these as well. I would even go so far as to say these problems are the rule rather than the exception. Why is software development still so high risk?

I don't know the answer but it seems to me that software development is much too bespoke. Take a relatively simple case of web store front development. What if the development company said, OK, for your store you can have option A or option B. Period. The developer doesn't gather requirements at all, and all the tools and libraries are known and have been used before. The customer gets to pick a few options like when you buy a car (e.g., say some look&feel options). This is a low risk, predictable, "there's only one way to do it" approach.

I'd like to see software development become vastly more predictable and stable. This will bore some developers who always want to try the latest framework and technology, but novelty adds risk.

[+] fiftyacorn|9 years ago|reply
I dont think its "under-estimating" as much as to get buy-in in a corporate environment means promising to over-deliver

Architects/Developers can also be their own worst enemy - chosing a new stack over the existing stack

[+] _pmf_|9 years ago|reply
Also nice: the joke by Rich Hickey that Agile solving software development via sprints is like someone revolutionizing the marathon by firing a starter pistol every 100 meters.
[+] zer0tonin|9 years ago|reply
I should explain that to my boss, we're literally doing waterfall but with standup meetings...
[+] adrianmsmith|9 years ago|reply
Great stuff.

I asked one Agile coach about how and when one should do software architecture decisions for a larger project, he said you can take an extra sprint/spike at the start to do software architecture if necessary.

At another place, the testing members of the team always tested the previous sprint's work (otherwise they'd be doing nothing for the first 80% of the sprint, if they tested that sprint's software).

So if you had a requirements sprint, a software architecture sprint, a development sprint, then that gets tested on the next sprint, what process does that remind you of? :)

[+] flukus|9 years ago|reply
"Agile waterfall" may be a meme but I have seen job listings use that exact term to describe the company. Out of everywhere I've ever interviewed though, no company that has described themselves as Agile have been remotely Agile, ones that don't spend so much time thinking about process generally are Agile to a degree.
[+] liamt|9 years ago|reply
I've been there. Now we do scrum and do standup meetings but it's a lot less strict. Most of the team talks constantly anyway as we're all in the same room.

Our old boss is long gone.

[+] overgard|9 years ago|reply
Is waterfall even a thing? I feel like the software industry is constantly coming up with solutions to cultural problems that were more relevant in 1977 than 2017

Ive worked at places that werent "agile" and it wasnt the strawman experience of waterfall.. it was just people doing work. They just didnt make a big deal about squeezing everything in a two week window and avoided uncomfortable/useless planning and retrospective meetings.

[+] overgard|9 years ago|reply
This is anecdotal, but I've found that the more "agile" a place is, the more underworked I am. In most non-agile places, if im out of work its up to me to find a way to make myself useful (agency!), but in agile environments I usually am told not to take something on because "it wont fit in the sprint". I generally think agiles main benefit is bringing the low performers up to average, but at the cost of handcuffing your top performers.
[+] pdelbarba|9 years ago|reply
When I worked with a bigger agile group one of the things I liked was having a task list put together and then divided up among everyone but also having an 'on-deck' group for stuff we didn't think we'd have time for. Anyone who got things done faster would have their pick of the on deck work and was some extra freedom and recognition for doing so.
[+] shanemhansen|9 years ago|reply
> bringing the low performers up to average, but at the cost of handcuffing your top performers.

Sounds about right. If you view humans as interchangeable resources then predictable performance is good. Most businesses don't need or benefit from a Linus Torvalds to build their CRUD app.

[+] scruple|9 years ago|reply
Ugh, I deal with this, too. "It won't fit in to the current sprint." Who cares? Nothing bad is going to happen! I just take them anyway and take people to task to explain why this is a worse situation than my salary burning up while I do nothing instead.
[+] seanwilson|9 years ago|reply
I don't see value in stand ups most of the time and usually they just get in the way of your work e.g. "I'll start this feature in 30 mins because we've got a standup in 10 mins". I already know what most people are doing from either group chat or the planning board, and a lot of details won't have much relevance to everyone so you end up zoning out just like you would in typical bad meetings.

The "what did you do yesterday?" part can also encourage people to scramble to justify they worked hard enough yesterday when it's tricky to condense why a task isn't as trivial as it sounds.

What did you do yesterday? What are you doing today? What is getting in your way? That's exactly what planning boards are for! Create a Slack bot which automatically summarises and posts this for each team member every day if you really must.

Planning boards are great though. They give you an overview of what everyone is up to, what progress is being made and they're little effort to keep up-to-date compared to having to reply to lots of emails about "what's going on?".

[+] tempodox|9 years ago|reply
I have yet to see an implementation of "Agile" that's not a cargo cult pretext for micromanagement.
[+] partycoder|9 years ago|reply
I think the article is correct. Agile has valid motivations and solves concrete problems.

But in practice, you will eventually see confirmation bias and cargo cult.

The motto of delivering customer value has been used as an excuse to emphasize the surfaces exposed to the customer (e.g: features) and neglect non-features (non-functional requirements, e.g: maintainability, security, performance, scalability, configuration, testing).

[+] codingmyway|9 years ago|reply
Microservices seems to be the latest cargo cult technique being pushing without really considering whether the benefits outweigh the costs. Some consultants always need another thing to push that a particular company's problems for them so therefore it must be applied everywhere.
[+] 9gunpi|9 years ago|reply
Another kind of cult is religious adherence to methodology while ruining the very value the methodology is supposed to enhance.

When you think something is stupid (like adopting only standups), frequently it's because you don't understand the motivation behind it. Standups are one of the first obvious changes any team, no matter how oldschool and rigid, can adopt and see the results behind it quickly, to encourage further transformation.

When you're changing something big (like a development process that is stable, consistent and filled with people, resources, operational and deployment plans, etc.), you want to change it either radically, or chunk by chunk. Radical changes are easy to decide yet hard to pull off while preserving business continuity, it's a bit of a gamble not everybody is willing to take. So, many executives want the "transformation" to actually be "step by step evolution". And one of the first steps are standups and enhancing the planning.

All of the above doesn't debate the fact that there are companies which get Agile terribly wrong, and companies which can't get Agile right due to extremely rigid structure. However, not all companies who fail to visibly transform into Agile in a few easy steps are like that: sometimes they're stuck on early iterations, slowly working their ways further while preserving continuity and market momentum, and there's nothing wrong with it.

[+] RiffyRiffy|9 years ago|reply
All managers not only engage in waterfall, but covet all of the privileges of waterfall. The ability to tell lessers what to program down to the line of code is the ultimate expression of power for people who will never, ever be C-level. And to top it off, they use their powers to force you to compel you to agree that their practices are agile.

Agile has never, and will never exist when micromanagement is a good enough drug for those who will never reach actual influence.

[+] 9gunpi|9 years ago|reply
This is very true, but Agile isn't the only un-achievable property in this case. Even basic healthy atmosphere and efficiency barely have a chance in these situations, so expecting someone in a wheelchair to run feels a bit naive.
[+] rhapsodic|9 years ago|reply
I think the most critical factor in the success or failure of a software development project is the skill and dedication of the developers. Yet I never seem to hear or "Agile Coaches" mention that. I guess clients would not be receptive if someone were to tell them that their best course of action was to replace half of their team.