I stopped taking this seriously when they compared the accuracy of the "ask the audience" and "phone a friend" bits on Who Wants to be a Millionaire without considering that everyone used the ask the audience lifeline on an early question and saved the phone a friend for later questions.
Without controlling for when they are used the comparison of accuracy is literally meaningless.
The same stats show that, rightly or wrongly, the contestants didn't have as much faith in the audience for later stage, much harder questions.
I guess I should give the benefit of the doubt and assume the writer didn't notice this and the error is accidental but that's only slightly better than just ignoring it because you wanted a sexy anecdote to support your point.
You right that the two lifelines are used at differing stages in the game. I wonder though, what the accuracy rates look like after controlling for what stage of the game you are in. Maybe people SHOULD save their ask the audience for later in the game.
His further reading list includes Surowiecki's The Wisdom of Crowds, but his post doesn't look like he read it himself. Surowiecki has a long list of requirements for the "wisdom of crowds" to actually be helpful, for one example the individual judgments need to be independent, that the post ignores. Without the restrictions you get different failure modes, like bubbles, runs, and traffic jams.
As programmers we should be well aware that our customers' "needs" are bounded by what they a) know and b) believe is possible. Guiding them to technological enlightenment can and does provide rewards (financial and otherwise).
He isn't being very fair: where the goal at first was to be as right as possible, it later became "to piss off as few of the 300 guests as possible". I'm not sure what exact procedure he had in mind for businesses, but in a situation where "votes eliminate the interesting edges, leaving only the boring residue that no one hated enough to vote off the island" applies, you're just not talking about the wisdom of crowds anymore, this wasn't possible in his earlier examples either. An important factor to the wisdom of crowds, and a key difference between his earlier examples and his later examples, is that people vote independently, without even talking about what they're going to vote.
For demonstrational purposes, I will now take his holiday meal situation and apply actual wisdom of crowds to them. This isn't a very usual situation, but this is about properly applying the wisdom of crowds.
Instead of people stating what they dislike, people may nominate a meal, and afterwards, everyone votes for the meal they want to go with. Like in the Wants to be a Millionaire examples, people cannot talk in this process - they cannot state what they dislike. Instead, the fact that some people can't take spicy food will show in the results, but beside that, every individual will pick a meal that isn't boring, and the end result will be just that. "Allergic to garlic" and "doesn’t eat anything green" are rather specific complaints and were not taken into account, so let's count those 3 people as unsatisfied, but the success here is that most people will in fact be satisfied with the meal.
You can't use your technique for meals because everyone will suggest a different meal, and then you have no basis for a decision. That technique works when you're voting on one variable which has one CORRECT value which you're trying to seek. Creative work is not that.
The other problem with focus groups - they aren't independent. There will always be one person who sounds convincing, and the rest will be biased to whatever that person thinks.
The advantage of focus groups is, you can identify themes which resonate. For example, the group might say "integrated" sounds good. That doesn't mean you want an integrated product, it just means that you need a sticker which says "integrated", and some justification (We integrate with Facebook, via our "Like" buttons!)
>The other problem with focus groups - they aren't independent. There will always be one person who sounds convincing, and the rest will be biased to whatever that person thinks.
Experienced group moderators are aware of this take steps to mitigate and reflect in their notes.
There are other forms of market research beyond focus groups. Aggregated quantitative and qualitative one-on-ones also fit the OP's bill of "crowd wisdom"
This essey is weak, I mean author is comparing apples to oranges. First problem with jellies is Fermi problem, it is approximation. If you have more people giving they answers mean of approximation will be closer to actual value. But second part is about concrete thing it is not estimating or guessing. First part is totally irrelevant. Second part could be used alone. But you just know that you can't please everyone so it is common sense I think. Like going for trip with people who are working in different jobs on different schedules, somebody will have to be left out.
[+] [-] freshhawk|13 years ago|reply
Without controlling for when they are used the comparison of accuracy is literally meaningless.
The same stats show that, rightly or wrongly, the contestants didn't have as much faith in the audience for later stage, much harder questions.
I guess I should give the benefit of the doubt and assume the writer didn't notice this and the error is accidental but that's only slightly better than just ignoring it because you wanted a sexy anecdote to support your point.
[+] [-] hammock|13 years ago|reply
[+] [-] chmike|13 years ago|reply
[+] [-] billswift|13 years ago|reply
[+] [-] smartbear|13 years ago|reply
[+] [-] chmike|13 years ago|reply
[+] [-] smoyer|13 years ago|reply
[+] [-] fleitz|13 years ago|reply
[+] [-] antithesis|13 years ago|reply
For demonstrational purposes, I will now take his holiday meal situation and apply actual wisdom of crowds to them. This isn't a very usual situation, but this is about properly applying the wisdom of crowds.
Instead of people stating what they dislike, people may nominate a meal, and afterwards, everyone votes for the meal they want to go with. Like in the Wants to be a Millionaire examples, people cannot talk in this process - they cannot state what they dislike. Instead, the fact that some people can't take spicy food will show in the results, but beside that, every individual will pick a meal that isn't boring, and the end result will be just that. "Allergic to garlic" and "doesn’t eat anything green" are rather specific complaints and were not taken into account, so let's count those 3 people as unsatisfied, but the success here is that most people will in fact be satisfied with the meal.
[+] [-] smartbear|13 years ago|reply
[+] [-] wisty|13 years ago|reply
The advantage of focus groups is, you can identify themes which resonate. For example, the group might say "integrated" sounds good. That doesn't mean you want an integrated product, it just means that you need a sticker which says "integrated", and some justification (We integrate with Facebook, via our "Like" buttons!)
[+] [-] hammock|13 years ago|reply
Experienced group moderators are aware of this take steps to mitigate and reflect in their notes.
There are other forms of market research beyond focus groups. Aggregated quantitative and qualitative one-on-ones also fit the OP's bill of "crowd wisdom"
[+] [-] ozim|13 years ago|reply
[+] [-] neuroelectronic|13 years ago|reply
[deleted]