(Author of The Art of Agile Development here.) Lots of negativity in this thread. I can't say I'm surprised. For what it's worth, I mostly agree with the negativity—"Agile," as commonly practiced, tends to be terrible. That's part of why I wrote the new edition.
I say as much in the first paragraphs of the book:
> Agile is everywhere. And paradoxically, nowhere.
> In the 20 years after the Agile freight train roared into software developers’ consciousness, the number of companies calling themselves “Agile” increased by orders of magnitude. The number of teams actually taking an agile approach to their work? Not so much. “Agile,” the easily repeated name, is enormously successful. The ideas behind Agile—well, most of them are ignored.
> Let’s fix that.
If anybody would like to talk about the book, I'm happy to engage. If you're curious about the book, the first three chapters are currently up for free on my website, and there's a lot of excerpts from the remaining chapters posted too. Follow the links in the table of contents at the bottom of https://www.jamesshore.com/v2/books/aoad2.
I think many aspects of Agile are sensible enough in most situations.
My negative experience with Agile has mainly been through people who are ideological about it and insist, when agile fails, that it was not the fault of the process or some other factor, but rather the fault of the people who "didn't implement agile right."
A bit of humility from the Agile community would be very welcome, as well as clarification of what types of problems it works well for and which ones it doesn't, where the dangers are, etc.
I have been a proponent of Agile approach since the 90s (XP, Tom Gilb, RAD, Lean). But lately, I am really really hating it. People no longer understand what Agile is about. They just follow some Scrum practices and that´s it. I feel disempowered. How do I find a job that gets it ?
My own experience with (lack of) Agile development:
- Water Epic Fall. An obvious symptom of Cargo Cult Agile. This happens when the phases of a waterfall project are disguised as epics that represent a huge fraction of the development. In my own experience this is caused when the MBAs try to force Scrum from top to bottom so they can pretend that they're doing agile on paper and thus "fulfill the values(tm) of the company". This is even worse than usual waterfall because it places an artificial limitation on making multiple components of the project live and evolve together and so "world collides" when each time an epic "is closed" and the new one has to find a place to fit, somehow. Proposition: epics are a way to breakdown work and work is done by a the team and so leave it to the team doing the work. Rule of thumb: stories are defined by time while epics are defined by stories with a common concrete idea.
- Daily meetings are mostly reports to the boss. Proof of that was when the boss wasn't there meetings that normally endured for 30 minutes and more would only take 10 minutes. In none of these meetings the boss helped to resolve the blockers; other devs did. Proposition: no boss in daily standup; DSMs are a tool for and by the dev team.
- Devs are blamed for not finishing stories in time. This is usually due to a bad backlog grooming where definition of ready is left as an exercise to the reader. It is also a symptom of the Water Epic Fall described above. Devs has to deal with the fact that the code isn't simply ready to integrate the new feature but the exercise to check if it was was never done in the first place. This is made worse when it is the dev who estimated the story and is now "responsible" for it. Also made even worse when "architects" with veto power decides what is ready or not and what the stories are. Proposition: again, let the devs have the final words on the backlog and its stories.
- Asking for permission to fix problems and remove friction. This is usually a problem caused by survivor biased MBAs who think of the heroics they did 20 years ago as war medals and a good development methodology because after all "this is why we could afford our new building". This problem often forms a symbiotic relationship with a culture of fear lead by barons aka "team leads". Proposition: revolt. Seriously. "We can't work this way anymore" during the quarterly report mega-meeting in front of the C*Os has its impact. Otherwise change jobs and make sure to let HR know by naming names. It's sad to see this kind of situation but if you don't it will poison every aspect of your life.
- Nobody RTFM. Seriously. Who can say they have read at least one book on Lean/Agile/Scrum/DevOps from cover to cover? I would say most MBAs didn't because there's an entire industry based on giving very expensive courses and on agile consulting just to repeat verbatim what are on these books. And even then most of the time you will never heard of concepts such as "lead time", "feed back", "release often", "no wip", etc. Some devs do tho, but they usually quit when they realize that their conversion efforts go to waste and provoke the wrath of the MBAs and the Teams Leads(tm) because either "we're not a startup" or "we can't be like Google"
- Large teams. Scrum doesn't really say much about how to work with multiple teams. Even less so where you're dealing with hardware and firmware development.
In my experience, management cherry pick aspects of the agile method that disempower knowledge workers and destroy any sense of ownership over the product, while ignoring all practices and intentions of empowering knowledge workers. Until the Agile custodians address how their philosophy has been corrupted in the real world, I want nothing else to do with it.
Agile in practice is like conveyor belt mass production but with each produced piece having unique requirements from each worker. Managers love it, the conveyor belt is constantly moving so a worker can't look at any piece for too long, so throughput is steady and the team always looks very productive!
Edit: And then some will say "If you are stressed, why don't you just reduce the speed of the conveyor belt? Agile is meant to be customized!", or "If an item wasn't done properly you can just move it back to the start of the belt so you can look at it again!". My answer is "I just don't want that damn conveyor belt in the first place!".
1. I suspect you're ascribing a level of power to "the Agile custodians" (who, exactly?) that doesn't exist. Nobody has the power to prevent people from misinterpreting Agile, no matter how much as they might want to.
2. The first chapter of my book (the one under discussion) does discuss how the philosophy has been corrupted in the real world. It's a major theme of the book.
Yep. I have done it well but I have also experienced exactly as your comment describes. In those scenarios, some poor tech lead is appointed whose responsibilities include estimation, knowing the whole backlog, managing tech debt, managing risks and hiring for the team (instead of an empowered team doing all this)
I don’t follow. If your managers have disempowered you, it’s not the “agile custodians” fault, nor is it Agile, it’s your management’s fault and you should want nothing else to do with them instead
That or invent something better that management cannot infect
The problem with any development philosophy or dogma at this point is, that it will eventually be led by people that want to make money from it trough books, consultancies, certificates, etc. People that are willing brush over any downsides or compromise its original ideas in whatever way necessary to sell.
We could replace agile with something else and it will end up the same way.
Can anyone point me in the direction of companies that are pointedly not using Agile/Scrum?
I understand that embedded and large systems are most likely to eschew agile as it's frequently not a good fit for the work, but I work in Python/Django mostly at the moment.
Scrum especially seems to fail in spectacular ways vs. other production methodologies in that when Scrum fails, it can be a net negative to the company. I feel as if I were to restrict my job search to companies that have anti-"Agile" methodologies, I would be more likely to work in a true agile environment.
I understand the pressures that create Scrum, as nature hates a vacuum, but I can't understand why there isn't any slack at all to experiment and fail in most smaller organizations. Perhaps we're disinclined to "hire smart people and get out of their way" nowadays, because we can only make data-driven decisions.
I'm sure that the best products are made by a small dedicated team of competent individuals communicating openly with one another, and I'm equally sure that success is rarer among those orgs. But, as someone with a degree in psychology, it's plain to see how operationalizing healthy communication patterns can have a counterintuitive result.
Kanban or Lean are usually much better for smaller teams.
For a larger company, SAFe is a much better option but there’s a danger that by default people will suggest that individual dev teams follow Scrum (because that’s generally the default). Individual devs teams can use anything they want within each PI and usually Kanban or Lean will go much more smoothly for everybody.
Pretty much almost every Silicon Valley VC funded company or startup I know are not using Agile/Scrum. My impression is Agile/Scrum is popular on consulting where the software buyer is external and therefore it's somewhat a mystery what they want so you hedge the risk and in companies where engineering is considered more of a "cost" and not well understood than something that drives the whole business.
Anecdotally, Coinbase and Airbnb didn't use Agile/Scrum and I never have interviewed in a company that mentioned that. Usually how it works that there the company knows what they want and there is some kind of roadmap of projects or experiments which are based on improving some kind of goal or metrics. Teams usually work in somewhat iterative phases, were you start with design, build the first prototype/mvp, test internally until you get to a stage to run an experiment with the users. During this time, your team meets every now and then, gets feedback from others and you do reviews with leadership. Then eventually if the experiment seems promising and the feature is good enough, it gets shipped. You can have sprints or not but no-one talks about or uses story points, retrospectives, planning pokers, product owners, user stories or other agile ceremonies. You have goal and project to accomplish, and then the management/leaderships tries to estimate with the team how long things will take.
The FAANG companies mostly don't, likely many companies similar to those also doesn't. If you can get into this sort of companies it isn't terribly hard to avoid, I'll never work at an Agile shop again.
By 'agile', as one of the first people around the manifesto, Fowler means product development applied to software. If you're curious which fields benefit from it and which don't, it is all fields in which you create a product that users use or will use. Agile (product development) doesn't help any process which is not supposed to be used by users. Examples: software products with 100%, and no less, defined specs. Second use case is never released vaporware, such as everything for which waterfall methodology is used and specs are assumed to be 100% correct upfront and it turns out they are not. The only type of project with 100% defined requirements I'm aware of are one-to-one software rewrites.
Agile for majority of devs and consultants (and not for the people around manifesto, such as Fowler) is conflated with a 2 week or so status reporting and planning process. It has nothing to do with product development as these status reports typically don't ask 'what is the status of the product' but the status report is typically about 'what everybody is doing and if they are busy'.
This has nothing to do with the original agile manifesto and if you read through it you'll notice that this modern 'Agile/Scrum' time-boxed busywork and status report is in contradiction with its principles. Keeping devs busy is obviously the goal of this consulting scrum as that is what consultants charge for.
Fowler himself gave up years ago explaining the difference. When words become popular, their meaning gets hijacked and repurposed. He explained this in a blog post of his
https://martinfowler.com/bliki/SemanticDiffusion.html
The highest quality software I ever worked on was waterfall all the way, and by God, was it high quality.
The requirements were hammered out in brutal detail by an experienced specialist. Each requirement agreed with the customer, and then each requirement carefully worked into a design. Each requirement clearly linked with every part of the design that fulfilled it. The design reviewed and approved. The code written in literal style (using a variant of noweb), with discussion and commentary, and that noweb document compiling into the source code to be onwards compiled and also a beautiful document listing the full source code and the various commentaries and discussions of it (such that one understood the code not by looking at the code, but by reading the document, which was like a structured guided tour of the code, and to look at the actual code was to see a one-to-one relationship with the documentation).
Each part of that document linked to a part of the design, and thus to the requirements. Tests had been written independently of the code, based on the design and the original requirements. Test documents were printed out, and an individual conducted every test on that document, writing down version numbers and so on, and writing down each results, circling PASS or FAIL, signing their name to it. Successful test documents were sealed in an envelope, with a date and signature, and put into storage against the customer requesting to see one (which happened every so often).
The original requirements could, years later, be followed through the design, through the code, through the tests; the whole chain verified on demand.
The software I worked on was developed over most of a decade (I was only there about four years), with a delivery a couple of times a year. The customer never once reported a bug. Every requirement was met. Other contractors had their part taken away from them and given to us, and when it was finished they took it and asked us if we could start working on the next generation product of the same.
I have never before or since worked on such high quality software. It was only possible because we happened to have really high quality customers, who actually knew what they wanted and had the ability to say it and agree to it.
Doing Waterfall correctly is easy. Spend 1 year planning, 3-5 years building, 1 year testing, then release. Find out you built the wrong thing, enjoy the earnings from your $10 billion government contract, and start another cycle.
Waterfall with WIP limits and continuous delivery is far less painful than a lot of processes I've used.
Because waterfall generally has milestones as a coarser grained way of ensuring evidence of forward progress, all deliverables generated around the time of a milestone just become candidates for delivery. Do you want this version that is missing two features, this version that is missing one but the other might be buggy, or to wait another two weeks to get the whole thing? If you have anything that looks like QA or acceptance tests, those can be run on deliverables, keeping people a little more comfortable with the process than they would with strict Waterfall.
To quote Fowler and all of his ilk, this means you're doing it wrong, and you must contemplate the principles of Agile further until you achieve enlightenment and discover True Agile.
As someone who joined the movement, believed in i, and still does, I had a lot to say about point a. I think the following best wraps it all up though.
"Every great cause begins as a movement, becomes a business, and eventually degenerates into a racket" -Eric Hoffer
I think another important thing to point is that most* software projects fail. Trying a new methodology is a good way to deflect blame and extend life into a bad business, team, manager, product.
b) I don't think so. I think companies are still chasing agile. I personally hope it makes a resurgence. Unfortunately Waterfall is perceived as an urban legend vs the monstrosity it is/was on the industry by the new generations
* I thought it was 50%. [1] suggests it's 75%, and this[2] suggests 90%+
The very first link of the article “Manifesto for Agile Software Development” isn’t working sine it’s pointing at www.agilemanifesto.org but the correct link is agilemanifesto.org :-)
[+] [-] jdlshore|4 years ago|reply
I say as much in the first paragraphs of the book:
> Agile is everywhere. And paradoxically, nowhere.
> In the 20 years after the Agile freight train roared into software developers’ consciousness, the number of companies calling themselves “Agile” increased by orders of magnitude. The number of teams actually taking an agile approach to their work? Not so much. “Agile,” the easily repeated name, is enormously successful. The ideas behind Agile—well, most of them are ignored.
> Let’s fix that.
If anybody would like to talk about the book, I'm happy to engage. If you're curious about the book, the first three chapters are currently up for free on my website, and there's a lot of excerpts from the remaining chapters posted too. Follow the links in the table of contents at the bottom of https://www.jamesshore.com/v2/books/aoad2.
[+] [-] recursivedoubts|4 years ago|reply
My negative experience with Agile has mainly been through people who are ideological about it and insist, when agile fails, that it was not the fault of the process or some other factor, but rather the fault of the people who "didn't implement agile right."
A bit of humility from the Agile community would be very welcome, as well as clarification of what types of problems it works well for and which ones it doesn't, where the dangers are, etc.
[+] [-] corpMaverick|4 years ago|reply
[+] [-] TeeMassive|4 years ago|reply
- Water Epic Fall. An obvious symptom of Cargo Cult Agile. This happens when the phases of a waterfall project are disguised as epics that represent a huge fraction of the development. In my own experience this is caused when the MBAs try to force Scrum from top to bottom so they can pretend that they're doing agile on paper and thus "fulfill the values(tm) of the company". This is even worse than usual waterfall because it places an artificial limitation on making multiple components of the project live and evolve together and so "world collides" when each time an epic "is closed" and the new one has to find a place to fit, somehow. Proposition: epics are a way to breakdown work and work is done by a the team and so leave it to the team doing the work. Rule of thumb: stories are defined by time while epics are defined by stories with a common concrete idea.
- Daily meetings are mostly reports to the boss. Proof of that was when the boss wasn't there meetings that normally endured for 30 minutes and more would only take 10 minutes. In none of these meetings the boss helped to resolve the blockers; other devs did. Proposition: no boss in daily standup; DSMs are a tool for and by the dev team.
- Devs are blamed for not finishing stories in time. This is usually due to a bad backlog grooming where definition of ready is left as an exercise to the reader. It is also a symptom of the Water Epic Fall described above. Devs has to deal with the fact that the code isn't simply ready to integrate the new feature but the exercise to check if it was was never done in the first place. This is made worse when it is the dev who estimated the story and is now "responsible" for it. Also made even worse when "architects" with veto power decides what is ready or not and what the stories are. Proposition: again, let the devs have the final words on the backlog and its stories.
- Asking for permission to fix problems and remove friction. This is usually a problem caused by survivor biased MBAs who think of the heroics they did 20 years ago as war medals and a good development methodology because after all "this is why we could afford our new building". This problem often forms a symbiotic relationship with a culture of fear lead by barons aka "team leads". Proposition: revolt. Seriously. "We can't work this way anymore" during the quarterly report mega-meeting in front of the C*Os has its impact. Otherwise change jobs and make sure to let HR know by naming names. It's sad to see this kind of situation but if you don't it will poison every aspect of your life.
- Nobody RTFM. Seriously. Who can say they have read at least one book on Lean/Agile/Scrum/DevOps from cover to cover? I would say most MBAs didn't because there's an entire industry based on giving very expensive courses and on agile consulting just to repeat verbatim what are on these books. And even then most of the time you will never heard of concepts such as "lead time", "feed back", "release often", "no wip", etc. Some devs do tho, but they usually quit when they realize that their conversion efforts go to waste and provoke the wrath of the MBAs and the Teams Leads(tm) because either "we're not a startup" or "we can't be like Google"
- Large teams. Scrum doesn't really say much about how to work with multiple teams. Even less so where you're dealing with hardware and firmware development.
[+] [-] trhway|4 years ago|reply
> The number of teams actually taking an agile approach to their work? Not so much.
Classic no true Scotsman. That inept excuse gets always invoked by the proponents of Agile, Lean, Scrum, Six Sigma and Communism.
[+] [-] sul_tasto|4 years ago|reply
[+] [-] Jensson|4 years ago|reply
Edit: And then some will say "If you are stressed, why don't you just reduce the speed of the conveyor belt? Agile is meant to be customized!", or "If an item wasn't done properly you can just move it back to the start of the belt so you can look at it again!". My answer is "I just don't want that damn conveyor belt in the first place!".
[+] [-] jdlshore|4 years ago|reply
1. I suspect you're ascribing a level of power to "the Agile custodians" (who, exactly?) that doesn't exist. Nobody has the power to prevent people from misinterpreting Agile, no matter how much as they might want to.
2. The first chapter of my book (the one under discussion) does discuss how the philosophy has been corrupted in the real world. It's a major theme of the book.
The first chapter is online for free at present: https://www.jamesshore.com/v2/books/aoad2/what_is_agile
[+] [-] Graffur|4 years ago|reply
[+] [-] throwoutway|4 years ago|reply
That or invent something better that management cannot infect
[+] [-] smoe|4 years ago|reply
We could replace agile with something else and it will end up the same way.
[+] [-] about3fitty|4 years ago|reply
I understand that embedded and large systems are most likely to eschew agile as it's frequently not a good fit for the work, but I work in Python/Django mostly at the moment.
Scrum especially seems to fail in spectacular ways vs. other production methodologies in that when Scrum fails, it can be a net negative to the company. I feel as if I were to restrict my job search to companies that have anti-"Agile" methodologies, I would be more likely to work in a true agile environment.
I understand the pressures that create Scrum, as nature hates a vacuum, but I can't understand why there isn't any slack at all to experiment and fail in most smaller organizations. Perhaps we're disinclined to "hire smart people and get out of their way" nowadays, because we can only make data-driven decisions.
I'm sure that the best products are made by a small dedicated team of competent individuals communicating openly with one another, and I'm equally sure that success is rarer among those orgs. But, as someone with a degree in psychology, it's plain to see how operationalizing healthy communication patterns can have a counterintuitive result.
[+] [-] brightball|4 years ago|reply
Kanban or Lean are usually much better for smaller teams.
For a larger company, SAFe is a much better option but there’s a danger that by default people will suggest that individual dev teams follow Scrum (because that’s generally the default). Individual devs teams can use anything they want within each PI and usually Kanban or Lean will go much more smoothly for everybody.
[+] [-] enra|4 years ago|reply
Anecdotally, Coinbase and Airbnb didn't use Agile/Scrum and I never have interviewed in a company that mentioned that. Usually how it works that there the company knows what they want and there is some kind of roadmap of projects or experiments which are based on improving some kind of goal or metrics. Teams usually work in somewhat iterative phases, were you start with design, build the first prototype/mvp, test internally until you get to a stage to run an experiment with the users. During this time, your team meets every now and then, gets feedback from others and you do reviews with leadership. Then eventually if the experiment seems promising and the feature is good enough, it gets shipped. You can have sprints or not but no-one talks about or uses story points, retrospectives, planning pokers, product owners, user stories or other agile ceremonies. You have goal and project to accomplish, and then the management/leaderships tries to estimate with the team how long things will take.
Pragmatic Engineer had article about this too: https://blog.pragmaticengineer.com/project-management-at-big...
[+] [-] clumsysmurf|4 years ago|reply
[+] [-] Jensson|4 years ago|reply
[+] [-] snidane|4 years ago|reply
Agile for majority of devs and consultants (and not for the people around manifesto, such as Fowler) is conflated with a 2 week or so status reporting and planning process. It has nothing to do with product development as these status reports typically don't ask 'what is the status of the product' but the status report is typically about 'what everybody is doing and if they are busy'.
This has nothing to do with the original agile manifesto and if you read through it you'll notice that this modern 'Agile/Scrum' time-boxed busywork and status report is in contradiction with its principles. Keeping devs busy is obviously the goal of this consulting scrum as that is what consultants charge for.
Fowler himself gave up years ago explaining the difference. When words become popular, their meaning gets hijacked and repurposed. He explained this in a blog post of his https://martinfowler.com/bliki/SemanticDiffusion.html
[+] [-] Ecstatify|4 years ago|reply
I love the idea of Agile, hasn’t worked at our company since we got spotified.
[+] [-] EliRivers|4 years ago|reply
The requirements were hammered out in brutal detail by an experienced specialist. Each requirement agreed with the customer, and then each requirement carefully worked into a design. Each requirement clearly linked with every part of the design that fulfilled it. The design reviewed and approved. The code written in literal style (using a variant of noweb), with discussion and commentary, and that noweb document compiling into the source code to be onwards compiled and also a beautiful document listing the full source code and the various commentaries and discussions of it (such that one understood the code not by looking at the code, but by reading the document, which was like a structured guided tour of the code, and to look at the actual code was to see a one-to-one relationship with the documentation).
Each part of that document linked to a part of the design, and thus to the requirements. Tests had been written independently of the code, based on the design and the original requirements. Test documents were printed out, and an individual conducted every test on that document, writing down version numbers and so on, and writing down each results, circling PASS or FAIL, signing their name to it. Successful test documents were sealed in an envelope, with a date and signature, and put into storage against the customer requesting to see one (which happened every so often).
The original requirements could, years later, be followed through the design, through the code, through the tests; the whole chain verified on demand.
The software I worked on was developed over most of a decade (I was only there about four years), with a delivery a couple of times a year. The customer never once reported a bug. Every requirement was met. Other contractors had their part taken away from them and given to us, and when it was finished they took it and asked us if we could start working on the next generation product of the same.
I have never before or since worked on such high quality software. It was only possible because we happened to have really high quality customers, who actually knew what they wanted and had the ability to say it and agree to it.
[+] [-] Jtsummers|4 years ago|reply
[+] [-] hinkley|4 years ago|reply
Because waterfall generally has milestones as a coarser grained way of ensuring evidence of forward progress, all deliverables generated around the time of a milestone just become candidates for delivery. Do you want this version that is missing two features, this version that is missing one but the other might be buggy, or to wait another two weeks to get the whole thing? If you have anything that looks like QA or acceptance tests, those can be run on deliverables, keeping people a little more comfortable with the process than they would with strict Waterfall.
[+] [-] thrower123|4 years ago|reply
[+] [-] Graffur|4 years ago|reply
A company doing Agile poorly most likely would do Waterfall poorly too.
[+] [-] Zababa|4 years ago|reply
[+] [-] kleiba|4 years ago|reply
As someone who has never worked in the industry (only in academia), I'm curious: what is it that turned the tide?
Could someone elaborate (a) why agile/scrum fell out of favor; and (b) what is its modern replacement? Thanks!
[+] [-] johncessna|4 years ago|reply
"Every great cause begins as a movement, becomes a business, and eventually degenerates into a racket" -Eric Hoffer
I think another important thing to point is that most* software projects fail. Trying a new methodology is a good way to deflect blame and extend life into a bad business, team, manager, product.
b) I don't think so. I think companies are still chasing agile. I personally hope it makes a resurgence. Unfortunately Waterfall is perceived as an urban legend vs the monstrosity it is/was on the industry by the new generations
* I thought it was 50%. [1] suggests it's 75%, and this[2] suggests 90%+
[1]https://www.geneca.com/why-up-to-75-of-software-projects-wil...
[2]https://www.quora.com/What-percentage-of-software-projects-f...
[+] [-] abacadaba|4 years ago|reply
0: http://programming-motherfucker.com/
[+] [-] kelseyfrog|4 years ago|reply
[+] [-] recursivedoubts|4 years ago|reply
from the darkness outside and within
by dreaming of systems so perfect that no one will need to be good
but the man that is will shadow
the man that pretends to be
[+] [-] NKosmatos|4 years ago|reply
[+] [-] willismichael|4 years ago|reply
[+] [-] aynyc|4 years ago|reply
[+] [-] FirstLvR|4 years ago|reply
The name sounds cool tho