The issue here is that it cleaves toward designing for the audience you have, as opposed to the audience you seek to appease.
Often, the design goals might exist on an alternate axis.
Random people offering random opinions amounts to random noise. However, the random noise will not appear neutral; it will appear as information.
If we consider many different things designed for particular audience members such as jet cockpits, medical tools, racing automobiles, we will see traits that exist that may seem nonsensical or otherwise when we divorce them from their designed contexts.
Bill Buxton covers this in Sketching User Interfaces when he describes Inuit coastal maps:
"The Inuit have used. [...] Tactile maps of the coastline, carved out of wood. They can be carried inside your mittens, so your hands stay warm. They have infinite battery life, and can be read, even in the six months of the year that it is dark. And, if they are accidentally dropped into the water, they float. What you and I might see as a stick, for the Inuit can be an elegant design solution that is appropriate for their particular environment."[1]
Focusing on complaints of the above design in all likelihood would, given the mad rabble of audiences online, result in discarding a solid bit of design.
[1] Bill Buxton, Sketching User Experiences, pg 37
Listening do user complaints doesn't always mean listening to their suggestions in fixing the problem. You shouldn't necessarily remove something that every user is asking you to remove if you feel it's a critical feature, but it definitely means you need to rethink how the feature works.
Isn't it more likely that the tactile maps are used because they are superior to any other kind of map the Inuits could have created (non tactile drawing, or a rock that doesn't float, for example)?
>That's how you suss out the rare 10% of community feedback that is amazing and transformative.
So don't obsess over every bit of user feedback, but figure out what matters, and what multiple people are complaining about, not just one loud person.
This is great for a startup but can be depressing for long term consulting.
I've had the unfortunate experience of building a product for someone else where the process was driven by a combination of Complain-Driven Development and Upper-Management Wish-lists. This alone might have been fine but at the same time anything positive like the real analytics about how successful the product was or any non-complaint communication coming from the users was hidden from me for fear, I suppose, that I might try to use that information get my company more money.
This became incredibly depressing. Everyday you show up to work putting in more and more hours into something that comes back with more and more complaints. It was hell and I did everything I could to end Complaint-Driven Development to no avail because thats how the customer liked to work. Eventually I finally just gave up and left to find something more rewarding and less soul crushing.
I am still trying to work out how to get useful feedback at all. Due to a fairly unique situation, people divide up between those folks who tell I am awesome (with zero useful feedback as to why) and folks who want my head on a pike) and zero useful feedback as to why). So far, backing away from the crowds of folks who want my head on a pike is the only constructive answer I have come up with, which leaves me more and more and isolated. I remain frustrated as to how to resolve this issue. I need skeptical feedback, not hatred and not idolatry. Neither of those is productive for anyone.
So, yes, there can be big problems with this, depending in part on the problem space you are addressing, I think.
"This became incredibly depressing. Everyday you show up to work putting in more and more hours into something that comes back with more and more complaints. It was hell and I did everything I could to end Complaint-Driven Development to no avail because thats how the customer liked to work. Eventually I finally just gave up and left to find something more rewarding and less soul crushing"
My current gig is very customer-driven and we hire a number of PMs to handle and incorporate the feedback. Our customers love us because the alternative is a particularly stodgy, poorly performing and relatively unchanging product. With end-users, yes. The flood of complaints and the complaints about the fixes for the complaints and on and on can be soulcrushing.
Finding a niche who loves you and your product is hopefully possible.
This article is about early stage products that are very new. If you're looking at a mature product that has been on the market for 10+ years, it's likely (I hope!) that most of the major obvious pain / complaint points have already been addressed. I don't think CDD would be as useful in that scenario -- you want a different model.
This process takes a lot of humility. It's not easy to say, "We are smart, but we don't know, so let's just get something out there." It's antithetical to the planning mania in so many big companies, and also why innovation like this is best from small places. I wish them the best!
The only thing I've ever seen work is getting down deep and dirty in the trenches with your users, communicating with them and cultivating relationships. That's how you suss out the rare 10% of community feedback that is amazing and transformative.
This is simple but practical advice I think far too many people ignore. You've inspired me with this article. Glad to see you've been successful from it!
I totally agree. The problem for us however is actually deciding what's the most complained-about request, and how to categorize those requests. I find that we each have our pet hates and loves, and even if we don't mean to, we develop selective hearing for complaints or praises.
We tried to add tags to support emails (via helpscout), but it's also hard to remember to tag things, and it's easy to use different tags for similar complaints.
I wonder about the best strategy to quantify complaints / suggestions and 'bucket' them correctly, so you can really choose the top ones.
That's when you need to look back, and not just focus on the most popular complained-about things, but also look at the qualitative feedback you're getting, and they to synthesize commonalities between different lines of feedback. Like another commenter said, sometimes feedback will help you reach a local maximum, but you need to infer /why/ complaints come in the first place, now just what people are complaining about, to know where to go next. If you have so many complaints and there's no clear direction, either you're at the nitpicking phase, or there are larger problems that you haven't identified yet.
If your current tagging system isn't helping you, flip the axis on which you're tagging to see if new patterns emerge. (area of complain on your product, versus types of tasks users are trying to achieve, for example.)
We've been using User Voice for the computer game Torment: Tides of Numenera. First, we used it to get community feedback on our rewards, tiers, and approach for our Kickstarter campaign. It was very helpful in testing our ideas in advance.
Throughout development, we've been using it to get feedback from the community: on their own ideas, their thoughts on design aspects we've shared, and questions they'd like us to answer.
User Voice lets people suggest and vote on ideas. We can label them with customizable tags like "Seen," "Considering," "Not Planned," as a very time-efficient means of communicating to many at once, without the expectation of personal replies.
The system isn't perfect, and we're not using it perfectly, but it's been helpful for getting and organizing feedback.
That's why I really liked Dell's Idea Storm. Instead of worrying about what a single user thinks, you can only worry about the current most popular issues or requests.
I don't think it has really helped Dell as to many of the things they simply said they aren't doing it (e.g. standardise power supplies in laptops) but the concept seems decent if they weren't very open to the feedback received.
Tried installing Discourse on Gentoo. Turns out to be a non-trivial project and I'm no Ruby on Rails newb. I'm no code-jock but I'd love if the install story was a tad easier. Maybe I should quit whining and figure out how to turn my pain into an ebuild :)
Agree that one issue in the Ruby ecosystem is that installs are too hard. Can you try our Docker install, we hope to move to this soon as our standard recommended install for that very reason, would love feedback:
Something is not clear about the character count requirement: did they set it to one or just finally found the right way to present it? If this is the former as the messages from the dialogs suggest, I think he should have highlighted the fact that the problem was not one of design (i.e. making users understand the limit), but functional (i.e. not forcing users to pad a too-short message).
No we left it as is, but it is configurable in the admin section. This is very important for non-English languages where one or two characters is often enough for a descriptive title.
One time I was talking to a designer about doing it this way. The designer said "Yeah, that's a great way to find a (look of total disdain) _local_maximum_." As usual, some truth to both points of view.
Terrible design. “20 to go for the reply” is comically bad UI writing. And then hiding it in a corner in grey text when the user is fixating on the field they are typing in? Not surprised at all it didn't work. And the "nuclear option" doesn't even meet basic usability guidelines of informing the user of input field requirements upfront, not just relying on error messaging.
While I grasp the reasoning in the article, and sometimes practice it, I also think it is often either counter productive or impossible.
For example, I have friends that released a rather ho-hum mobile app. They quickly garnered something like a 1.5 star rating, scathing reviews, and almost no conversions. The business cycle on this was a year, and they are still trying to claw back reputation and win users. It's a debacle. (the problems weren't their fault, but that is irrelevant to this point).
Then you have companies with secrecy, like Apple. I think this advice would be terrible for them (I have never worked there, and am open to correction). They can't dog food it widely due to the internal silos, and they certainly cannot test it with the public.
Then there are electronic systems - iterations on SW is easy, iterations on HW expensive and hard, even with simulations, mock ups, and what have you. I worked on an augmented reality hardware thingy several years ago; we went from foam cutouts to a couple of very expensive prototypes, and that was it.
It is awesome when we can completely sidestep a problem, and this process lets you sometimes sidestep the serious difficulty of UI design. I worry when it gets bandied about as a truism, or The One True Way (not saying Jeff is doing that, I'm remarking on the wider industry). Yes, Agile lets you sidestep the problem of scheduling and estimation - sometimes. Try that when you are making a new airliner, building a cloverleaf interchange, making a car entertainment system, and so on.
edit: the converse problem is equally as large. Someone below mentioned the 'planning mania' of companies. I don't mean to downplay that problem, just to point out the need to evaluate each situation on it's particular needs as opposed to a 'best practices' (oh, how I hate that term) unthinking approach.
"Then you have companies with secrecy, like Apple. I think this advice would be terrible for them (I have never worked there, and am open to correction). They can't dog food it widely due to the internal silos, and they certainly cannot test it with the public"
I fear that this myth is capitalized on by Apple under the hypercapitalist Auteur theory.[1]
The value of magic tricks is that they create awe and surprise, when in reality they can often be a by-product of misdirection and street-level sleights.[2]
This is not to suggest that Apple is going out onto street corners and asking random individuals for their opinions, but rather that part of the corporate myth making is likely obscuring process.
[2] I believe there is a court testimony document discussing Apple's market research team via Phil Schiller. Contrast this with Mr. Schiller's other comments about avoiding market research.
You can't release crap, even in your first version. Not saying your friends did that, but you have to have the kernel of something decent to show people before you release. Otherwise you're blowing the launch by taking all that initial interest and squandering it.
(I'd also argue mobile apps have a severe updating problem, that web apps do not.)
This definitely resonates. Although while feedback from discourse users naturally takes place on discourse, gathering feedback from users can often be much more difficult. We've enabled usersnap recently, which has been somewhat helpful, but hasn't created the kind of vociferous user-to-user and user-to-developer debates that you see on meta.discourse
When I'm giving feedback, I usually don't want the developer to tell me why it is wrong. I don't mean that in the sense that my perspective is correct and my feedback is never wrong, I mean that if I am irritated enough about something to report it as a flaw, the last thing I am going to want to do is engage in a debate about it.
BTW, discourse infinite scroll could behave better when the user input is the down arrow on the scroll bar.
Did he consider that the way he wanted Discourse to be used may not have tallied with the way his users wanted to use it?
That's the underlying disconnection.
I am now working on my 3rd message-transformation/integration middleware product for a large corp. In each case, the issue tracker has been stock full of problems from people who took the product, read all the manuals about what they were supposed to do, then did it their way anyway. Because, well if they can, they will.
I think the basic problem is not challenging your personal assumptions about how you want your beloved system to be used.
One problem with this is that users don't know what they want or what is easy / possible / hard.
I constantly get asked to add another "excel parser" to my in house Django app. These days I refuse (it is way too error prone) and build it directly into the web interface. Then they realize, that even with all the nice auto-complete and cut and paste features that excel comes with, clicking a few checkboxes and a submit button in a web interface is more efficient (and less error prone) than cutting and pasting the data into Exel in the first place.
I feel like responding to user complaints can help bring you closer to the nearest local maxima. But it often won't bring you beyond that unless your users are themselves UX designers or developers.
So it's useful to make your MVC as good as possible so that your users aren't forced to complain about the results of fundamental design shortcomings. That can result in more and more complex fixes, none of which should be necessary.
Because even the best experts are not going to be able to identify requirements the users won't discover until they begin using the product.
You can approximate some of this feedback by observing user interaction with prototypes, but if you pay attention, you'll learn more when they start using it in production.
In my experience as a designer that changes very little - In reality it can often mean you did a whole lot of upfront work just to have it change anyway. You can put 100 UX experts in the product design phase and users will still have complaints the second you release the product.
The point of this process is to start that complaint processes as early as possible so you're not delaying the inevitable or spinning your wheels on something users don't want anyway.
Right, this is why you listen to your community, but you don't blindly do everything they tell you to do. Filtering out the best 10% of feedback -- and resisting the transformation of the car you designed into a truck because "the community demanded it" -- is the way to go.
[+] [-] foolrush|12 years ago|reply
Often, the design goals might exist on an alternate axis.
Random people offering random opinions amounts to random noise. However, the random noise will not appear neutral; it will appear as information.
If we consider many different things designed for particular audience members such as jet cockpits, medical tools, racing automobiles, we will see traits that exist that may seem nonsensical or otherwise when we divorce them from their designed contexts.
Bill Buxton covers this in Sketching User Interfaces when he describes Inuit coastal maps: "The Inuit have used. [...] Tactile maps of the coastline, carved out of wood. They can be carried inside your mittens, so your hands stay warm. They have infinite battery life, and can be read, even in the six months of the year that it is dark. And, if they are accidentally dropped into the water, they float. What you and I might see as a stick, for the Inuit can be an elegant design solution that is appropriate for their particular environment."[1]
Focusing on complaints of the above design in all likelihood would, given the mad rabble of audiences online, result in discarding a solid bit of design.
[1] Bill Buxton, Sketching User Experiences, pg 37
[+] [-] jobu|12 years ago|reply
[+] [-] heartbreak|12 years ago|reply
[+] [-] jorgeleo|12 years ago|reply
It really does not matter what methodology, or tools you will use, if at the end it does not pass the user acceptance test
So why consider it a failure if you have users telling you what they want and how off you are? Better to embrace it and make a better product
[+] [-] csbrooks|12 years ago|reply
>That's how you suss out the rare 10% of community feedback that is amazing and transformative.
So don't obsess over every bit of user feedback, but figure out what matters, and what multiple people are complaining about, not just one loud person.
[+] [-] wavesounds|12 years ago|reply
I've had the unfortunate experience of building a product for someone else where the process was driven by a combination of Complain-Driven Development and Upper-Management Wish-lists. This alone might have been fine but at the same time anything positive like the real analytics about how successful the product was or any non-complaint communication coming from the users was hidden from me for fear, I suppose, that I might try to use that information get my company more money.
This became incredibly depressing. Everyday you show up to work putting in more and more hours into something that comes back with more and more complaints. It was hell and I did everything I could to end Complaint-Driven Development to no avail because thats how the customer liked to work. Eventually I finally just gave up and left to find something more rewarding and less soul crushing.
[+] [-] Mz|12 years ago|reply
So, yes, there can be big problems with this, depending in part on the problem space you are addressing, I think.
[+] [-] illuminate|12 years ago|reply
My current gig is very customer-driven and we hire a number of PMs to handle and incorporate the feedback. Our customers love us because the alternative is a particularly stodgy, poorly performing and relatively unchanging product. With end-users, yes. The flood of complaints and the complaints about the fixes for the complaints and on and on can be soulcrushing.
Finding a niche who loves you and your product is hopefully possible.
[+] [-] codinghorror|12 years ago|reply
This article is about early stage products that are very new. If you're looking at a mature product that has been on the market for 10+ years, it's likely (I hope!) that most of the major obvious pain / complaint points have already been addressed. I don't think CDD would be as useful in that scenario -- you want a different model.
[+] [-] pointernil|12 years ago|reply
[+] [-] mathattack|12 years ago|reply
[+] [-] atmosx|12 years ago|reply
"The fool doth think he is wise, but the wise man knows himself to be a fool."- William Shakespeare, As You Like It
[+] [-] jackson1988|12 years ago|reply
This is simple but practical advice I think far too many people ignore. You've inspired me with this article. Glad to see you've been successful from it!
[+] [-] gingerlime|12 years ago|reply
We tried to add tags to support emails (via helpscout), but it's also hard to remember to tag things, and it's easy to use different tags for similar complaints.
I wonder about the best strategy to quantify complaints / suggestions and 'bucket' them correctly, so you can really choose the top ones.
[+] [-] frandroid|12 years ago|reply
If your current tagging system isn't helping you, flip the axis on which you're tagging to see if new patterns emerge. (area of complain on your product, versus types of tasks users are trying to achieve, for example.)
[+] [-] ksaun|12 years ago|reply
Throughout development, we've been using it to get feedback from the community: on their own ideas, their thoughts on design aspects we've shared, and questions they'd like us to answer.
User Voice lets people suggest and vote on ideas. We can label them with customizable tags like "Seen," "Considering," "Not Planned," as a very time-efficient means of communicating to many at once, without the expectation of personal replies.
The system isn't perfect, and we're not using it perfectly, but it's been helpful for getting and organizing feedback.
[+] [-] UnoriginalGuy|12 years ago|reply
I don't think it has really helped Dell as to many of the things they simply said they aren't doing it (e.g. standardise power supplies in laptops) but the concept seems decent if they weren't very open to the feedback received.
[+] [-] igravious|12 years ago|reply
[+] [-] codinghorror|12 years ago|reply
https://github.com/discourse/discourse_docker
[+] [-] yoha|12 years ago|reply
[+] [-] EvilTrout|12 years ago|reply
[+] [-] dodger|12 years ago|reply
[+] [-] mrxd|12 years ago|reply
[+] [-] RogerL|12 years ago|reply
For example, I have friends that released a rather ho-hum mobile app. They quickly garnered something like a 1.5 star rating, scathing reviews, and almost no conversions. The business cycle on this was a year, and they are still trying to claw back reputation and win users. It's a debacle. (the problems weren't their fault, but that is irrelevant to this point).
Then you have companies with secrecy, like Apple. I think this advice would be terrible for them (I have never worked there, and am open to correction). They can't dog food it widely due to the internal silos, and they certainly cannot test it with the public.
Then there are electronic systems - iterations on SW is easy, iterations on HW expensive and hard, even with simulations, mock ups, and what have you. I worked on an augmented reality hardware thingy several years ago; we went from foam cutouts to a couple of very expensive prototypes, and that was it.
It is awesome when we can completely sidestep a problem, and this process lets you sometimes sidestep the serious difficulty of UI design. I worry when it gets bandied about as a truism, or The One True Way (not saying Jeff is doing that, I'm remarking on the wider industry). Yes, Agile lets you sidestep the problem of scheduling and estimation - sometimes. Try that when you are making a new airliner, building a cloverleaf interchange, making a car entertainment system, and so on.
edit: the converse problem is equally as large. Someone below mentioned the 'planning mania' of companies. I don't mean to downplay that problem, just to point out the need to evaluate each situation on it's particular needs as opposed to a 'best practices' (oh, how I hate that term) unthinking approach.
[+] [-] foolrush|12 years ago|reply
I fear that this myth is capitalized on by Apple under the hypercapitalist Auteur theory.[1]
The value of magic tricks is that they create awe and surprise, when in reality they can often be a by-product of misdirection and street-level sleights.[2]
This is not to suggest that Apple is going out onto street corners and asking random individuals for their opinions, but rather that part of the corporate myth making is likely obscuring process.
[1] http://www.youtube.com/watch?v=p9dmcRbuTMY
[2] I believe there is a court testimony document discussing Apple's market research team via Phil Schiller. Contrast this with Mr. Schiller's other comments about avoiding market research.
[+] [-] codinghorror|12 years ago|reply
(I'd also argue mobile apps have a severe updating problem, that web apps do not.)
[+] [-] jchung|12 years ago|reply
[+] [-] maxerickson|12 years ago|reply
BTW, discourse infinite scroll could behave better when the user input is the down arrow on the scroll bar.
[+] [-] kitd|12 years ago|reply
That's the underlying disconnection.
I am now working on my 3rd message-transformation/integration middleware product for a large corp. In each case, the issue tracker has been stock full of problems from people who took the product, read all the manuals about what they were supposed to do, then did it their way anyway. Because, well if they can, they will.
I think the basic problem is not challenging your personal assumptions about how you want your beloved system to be used.
[+] [-] collyw|12 years ago|reply
I constantly get asked to add another "excel parser" to my in house Django app. These days I refuse (it is way too error prone) and build it directly into the web interface. Then they realize, that even with all the nice auto-complete and cut and paste features that excel comes with, clicking a few checkboxes and a submit button in a web interface is more efficient (and less error prone) than cutting and pasting the data into Exel in the first place.
[+] [-] sopooneo|12 years ago|reply
So it's useful to make your MVC as good as possible so that your users aren't forced to complain about the results of fundamental design shortcomings. That can result in more and more complex fixes, none of which should be necessary.
[+] [-] leaxdc|12 years ago|reply
[+] [-] xfalcox|12 years ago|reply
It's great, users fells empowered, and you get the priority list for free with reddit upvote system.
As for discourse were planning to deploy it this year at our intranet, and we've got 130.000 employees.
[+] [-] the1|12 years ago|reply
Contact me if you need help. I'm a good UXpert.
[+] [-] kbutler|12 years ago|reply
You can approximate some of this feedback by observing user interaction with prototypes, but if you pay attention, you'll learn more when they start using it in production.
[+] [-] awesomerobot|12 years ago|reply
The point of this process is to start that complaint processes as early as possible so you're not delaying the inevitable or spinning your wheels on something users don't want anyway.
[+] [-] npsimons|12 years ago|reply
[+] [-] boohoofoodoo|12 years ago|reply
[+] [-] codinghorror|12 years ago|reply
[+] [-] chris_wot|12 years ago|reply