I don't understand what point he is trying to make about the Dunning-Kruger effect. The only pop-sci version of the study that I've heard is that the bottom percentile tends to overestimate their abilities while the top percentile tends to underestimate their abilities. The data shown certainly seems to support this interpretation?
edit: I read the article linked in another comment below [1] and now I think I understand what claim the original article was referring to. It seems a common misunderstanding of the D-K effect is that the bottom percentile would think that they are as competent or more competent as the top percentile. However, this is not what the study says, the bottom percentile is overestimating their ability but the estimations are still relatively lower than the self-estimations of those with higher ability.
When I read pop-sci interpretations of it (which is basically all I've ever read) my impression was that the top percentile estimate themselves as worse than the lower-percentile does.
The graphs cited in the article show that the top still estimate themselves better than the lower percentile, which is new to me.
"Rather, it’s that incompetent people think they’re much better than they actually are. But they typically still don’t think they’re quite as good as people who, you know, actually are good."
Unfortunately, while the incompetent people may believe that they are not quite as good as people who actually are good, the problem is that they are not qualified to identify people who are actually good. From the original paper:
"That is, the same incompetence that leads them to make wrong choices also deprives them of the savvy necessary to recognize competence, be it their own or anyone else’s."
"As predicted, participants who scored in the bottom quartile were less able to gauge the competence of others than were their top-quartile counterparts." [2]
So while the incompetent people may not believe themselves to be experts in general, they are less able to identify that the person sitting next to them is an expert.
And further: "Bottom-quartile participants failed to gain insight into their own performance after seeing the more competent choices of their peers. ... [they] tended to raise their already inflated self-estimates."
Not only do they fail to identify the expert next to them, but when presented with the expert's answers they will raise the own inflated estimate!
Therefore, I believe that in practice, the DK paper does support the belief that less competent people believe themselves to be more competent than experts.
[2] Kruger, Justin; Dunning, David (1999). "Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments". Journal of Personality and Social Psychology
The author states, as regards the interpretation of the
Dunning-Kruger diagrams, that
[i]n two of the four cases, there’s an obvious positive correlation between
perceived skill and actual skill, which is the opposite of the pop-sci
conception of Dunning-Kruger.
In my corner of the universe, you don't get to cherry-pick which pieces
of data (ie "what instances of two sets of random variables") you bestow
the golden twig of correlation upon. If I'm not entirely mistaken,
correlation is very much a global feature, not a measure of proximity of
two points on a chart.
So, yes, Dunning-Kruger (as evinced from the diagrams sported here) indeed seems to make a weaker claim: that there's no
correlation between “perceived ability” and “actual ability”. As such,
this claim is as far from the "pop-sci conception" of Dunning-Kruger as
it is from the author's.
The referenced graphs measure performance and perceived ability on 4 different tasks. You're right that ideally you'd pool this data to get to an overall correlation, but for the point the author makes, eyeballing it and taking a mental average does the trick, no?
Also, what corner of the universe are you from? Loess regression, hierarchical modeling, conditional analyses... methods for finding "non-global" correlations aplenty.
"It’s a little easier to see why people would pass along the wrong story here, since it’s easy to misinterpret the data when it’s plotted against a linear scale, but it’s still pretty easy to see what’s going on by taking a peek at the actual studies."
"People’s life evaluations rise steadily with income, but the reported quality of emotional daily experience levels off at a certain income level, according to a new study by two Princeton University professors [...]"
So seemingly that has nothing to do with the scale used, but with the definition of happiness. Guess someone didn't look "at the actual study"?
The author only thinks his understanding of Dunning-Kruger is better than average. No, seriously. As others have pointed out, the meme is not that actual ability and self-assessment are negatively correlated. It's just that ignorant people don't know how ignorant they are - i.e. that those at the low end of the scale overestimate their position relative to the maximum. That version is very well supported by the graphs the OP cites. He has scratched the iceberg of skepticism and now thinks he's enough of an expert to tell the rest of us that we're Doing It Wrong.
Hmm, as he kind of points out, there is a difference between knowing your ability in something is poor and being willing to admit it to a stranger. The usual meme of Dunning-Kruger tends to refer to people who are oblivious to their own incompetence. But there are valid reasons for the perceived score compression. I'd wager the people in the bottom quartile of grammar performance are well aware that their grammar isn't very good, they just weren't willing to write 20 on the paper (out of shame) and went for 60 instead.
> the meme is not that actual ability and self-assessment are negatively correlated
I don't think there's a single DK meme, but I do think that this is a common DK meme.
Somewhat-serious suggestion: maybe people who actually understand DK, have a tendency to underestimate how well they understand it relative to everyone else, and so assume that everyone else understands it about as well as they do.
I see the "actual ability and self-assessment are negatively correlated" version of the meme a majority of the time. Maybe I just hang out on the wrong parts of the internet?
> He has scratched the iceberg of skepticism and now thinks he's enough of an expert to tell the rest of us that we're Doing It Wrong.
This sort of gratuitous negativity isn't appropriate in a Hacker News comment. It is unnecessary to make your point and pollutes the atmosphere for all of us. Please don't do that.
This. I work with a support guy like this. He has, at best, a hobbyist's understanding of the technology we use. He tries to paint himself in meetings as this expert who often tries to correct us and says completely mind-numbing things. When discussing a fairly complex Drupal roll-out, "Oh, why even bother with this open source stuff? We can make an access database on the network shares and give people access to it! I hear it can do web now!" I've sat him down and patiently explained to him why we do these things, how they are common and best practices, etc and he just makes frowny face like I'm the one talking crazy and he doesn't seem to learn as he brings up the same suggestions over and over. Last week he was adament that a dedicated linux server for our Asterisk PBX was overkill and that WindowsXP running some windows freeware pbx on any old desktop would be better because PBXs need so little power and XP is a super lean OS.
At my previous job where I worked and managed a few young-ish support and jr devs, I came accross a lot of the same attitudes. The funny thing is that two or three years in, they tend to shed those attitudes (at least the smart ones do). They realize that this stuff is a lot bigger than the limited experience they've gotten in school or in their hobbyist projects. Suddenly the "lets toss out everything and do it my way" motor-mouting gets turned down a notch or two, especially after I let them do things "their way" once in a while only to have it explode in their faces. Hell, 90% of managing young techies is controlling their D-K until they mature into imposter syndrome. Then you have to manage that, which is a million times easier to deal because you're not being know-it-all'd to death in every meeting.
D-K is real. The graphs he points out literally supports it. Personally, I'm getting sick of this kind of uber-skeptic mentality that's popular here and on reddit. I think there's something ego gratifying for the INTJ male about being this loud-mouth contrarian who tells everyone they're wrong. Especially if its against some popular wisdom and if the argument is unusually pedantic and trivial.
In "Income & Happiness" you're missing that there are two complementary definitions of happiness - "satisfaction" and "affect". Satisfaction, or how you feel about your life, is what is shown in your graphs. "Affect" is your direct emotional experience of your life. What you call "wrongness" is just a failure to distinguish between the two - more money makes increasingly smug, but (beyond a certain point) doesn't make your like any more fun
You can still say type systems are obviously a good thing (I do too!). But you, sadly, can't say this is supported by scientific studies. It's an opinion based on your own personal experience.
The good news is that this opinion doesn't contradict evidence either! The research just isn't there yet. Some of research I've seen so far is pretty bad.
The fact that you actually recognized this response in yourself means that you're more than capable of adjusting that response to be less biased. It's a very good first step.
One option is to shrink your world to the things you have direct experience with. So yes, type systems are obviously a good thing - for you. Then the obvious question is why would somebody believe otherwise - and you can learn something by asking it. Worry about figuring out what the world is actually like once you've got a sample size of more than two.
When enough people are aware of the meme, and the meme is ingrained in culture enough to have a statistically significant effect on perceived ability. Or when people decide it is meaningless, out of desire to control their expectations of their performance.
"I suspect we find this sort of explanation compelling because it appeals to our implicit just-world theories: we’d like to believe that people who obnoxiously proclaim their excellence at X, Y, and Z must really not be so very good at X, Y, and Z at all, and must be (over)compensating for some actual deficiency; it’s much less pleasant to imagine that people who go around shoving their (alleged) superiority in our faces might really be better than us at what they do.
Unfortunately, Kruger and Dunning never actually provided any support for this type of just-world view; their studies categorically didn’t show that incompetent people are more confident or arrogant than competent people."
It's also worth pointing out that you almost never see people bringing up Dunning-Kruger to say that they might be over estimating their own understanding of a topic. It's almost always used as evidence that those that disagree with them are ignorant; that it's a study that only says something about their opponents, not themselves.
people on the internet misunderstanding a paper? Might that be because 99% of them don't have access to read the thing, since it's suck behind a paywall?
and to end a post about a paper people can't read with "Maybe I’m being naive here, but I think a major reason behind false memes is that checking sources sounds much harder and more intimidating than it actually is." -- that's just rubbing it in!
Scientific reporting sucks, even in quality media that take it seriously. If I had money I'd pay a select group of bloggers to write about how a popular bit of science has been misreported and then colate links to those on some central site.
A bit like Behind the Headlines, but about any science and by a wide range of science bloggers.
One of Norway's biggest bloggers is a guy named Tjomlid[1]. His blog is "skeptical", as in he looks into various things circulating in the media and either backs it up or debunks it.
When media says "X causes cancer", he comes with a level headed blog-post explaining the data. When anti-vaxxers point to a study, he shows that they have misinterpreted it, etc.
Unfortunately, it's in Norwegian. I'm interested in reading more in this vein, so please let me know if you have good, English resources.
However, I disagree with his interpretation of Dunning and Kruger's figures. There is limited data in those figures of course (3 sets with 12 points of data and 1 set with 8 points), and I don't know how big the actual sample sizes were. But in all 4 cases, it clearly and significantly shows that people with low actual ability overrate themselves and people with high actual ability underrate themselves.
> The pop-sci version of Dunning-Kruger is that, the less someone knows about a subject, the more they think they know.
I can only guess that he is interpreting that too literally. I always took the latter part as "the more they think they know [compared to what they actually do]". You could plot "perceived - actual" or "perceived / actual" instead of the figures in the paper to make it more obvious.
> But in all 4 cases, it clearly and significantly shows that people with low actual ability overrate themselves and people with high actual ability underrate themselves.
I read the data quite differently. I read that people don't use the full scale when rating themselves. The self rating scale goes from 60 to 90. If you correct for that, people kind of seem to do a nice job in self evaluation.
It can be explained, too. People have social pressure not to rate themselves near 100%, nor in the bottom half of the scale.
I always sucked at stats, so can someone clarify the income / hapiness case. From what I understand, we have :
* a straight line on a log scale - which is misleading, since the man on the street is used to straight lines on linear scales - telling that money buys happiness
* a "log-like" graph on a linear scale, which tells you that "money buys happiness a lot at the start, and then it doesn't change things a lot" - and the graph uses a linear scale on both axis, so there is less deformation.
People are making the claim that money buys happiness up to a certain number and then stops.
But there is no inflection point. 10% more money always buys you the same amount of happiness.
Nothing makes $75k the point where you have 'enough' money as opposed to $5k. 75 million is just as far ahead of 5 million (at least if it continues to be logarithmic).
I'm curious about this too. I'm familiar with the economic concept of diminishing marginal utility, and I get that the $10k you get from $120-$130k/yr is much less valuable to you than the $10 from $40-50k. But there still seemed (to my Stats 101-level interpretation) that there was a decent-enough jump between $75k and $130k and beyond that happiness/utility increased significantly as income did beyond the commonly cited figure.
For those of you who earn the requisite amount, is it true that it's easier to get a significant raise at the higher income levels? I live in a fairly rural area but I've found as my income rises the degree to which I need to fight for marginal pay increases has diminished, which seems counter-intuitive since each point gets more and more expensive, obviously.
What a gazillion pop-sci articles and blog posts suggest is that there is actually a negative correlation between self-assessment and performance -- i.e., that the higher people rate themselves, the worse they will perform. Looking at the graph we can see that while poor performers overestimate their performance relative to reality, there is still a positive correlation between performance and self-assessment.
Looks like only people in the 3rd quartile are any good at estimating performance, and so perhaps they should immediately all become project managers?! :)
I think there's an error in the blog post, because the 4th graph is the same as the 3rd.
It should probably be http://danluu.com/images/dunning-kruger/dunning_4.png which shows a positive correlation for "perceived logical reasoning ability and test performance". That then helps explain the statement "In two of the four cases, there’s an obvious positive correlation between perceived skill and actual skill".
This looks like a very uncharitable interpretation of the pop version of Dunning-Krugger. For one, people who use an explanation like that pretty much always talk about unskilled individuals in particular. I might be projecting but it seems like a more charitable interpretation of what they are saying would be 'The less someone knows about a subject, the more they [overestimate what] they know' which is roughly correct.
Without commenting on the various claims about memes in this article, I can offer a hypothesis on the closing paragraphs: the most obvious reason why misinterpreted science is much more commonly seen is that sharing an unsupported meme takes far less time and effort (~few clicks) than finding the research paper and reading it, so we would anticipate seeing a lot more of the former.
The pithy version is "a lie can run round the world before the truth has got its boots on", with thanks to Pratchett.
> the most obvious reason why misinterpreted science is much more commonly seen is that sharing an unsupported meme takes far less time and effort (~few clicks) than finding the research paper and reading it
I think your comment contains a highly recursive component:
[+] [-] krig|11 years ago|reply
edit: I read the article linked in another comment below [1] and now I think I understand what claim the original article was referring to. It seems a common misunderstanding of the D-K effect is that the bottom percentile would think that they are as competent or more competent as the top percentile. However, this is not what the study says, the bottom percentile is overestimating their ability but the estimations are still relatively lower than the self-estimations of those with higher ability.
[1]: http://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-k...
[+] [-] hasenj|11 years ago|reply
The graphs cited in the article show that the top still estimate themselves better than the lower percentile, which is new to me.
[+] [-] lowbloodsugar|11 years ago|reply
"Rather, it’s that incompetent people think they’re much better than they actually are. But they typically still don’t think they’re quite as good as people who, you know, actually are good."
Unfortunately, while the incompetent people may believe that they are not quite as good as people who actually are good, the problem is that they are not qualified to identify people who are actually good. From the original paper:
"That is, the same incompetence that leads them to make wrong choices also deprives them of the savvy necessary to recognize competence, be it their own or anyone else’s."
"As predicted, participants who scored in the bottom quartile were less able to gauge the competence of others than were their top-quartile counterparts." [2]
So while the incompetent people may not believe themselves to be experts in general, they are less able to identify that the person sitting next to them is an expert.
And further: "Bottom-quartile participants failed to gain insight into their own performance after seeing the more competent choices of their peers. ... [they] tended to raise their already inflated self-estimates."
Not only do they fail to identify the expert next to them, but when presented with the expert's answers they will raise the own inflated estimate!
Therefore, I believe that in practice, the DK paper does support the belief that less competent people believe themselves to be more competent than experts.
[2] Kruger, Justin; Dunning, David (1999). "Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments". Journal of Personality and Social Psychology
[+] [-] hf|11 years ago|reply
So, yes, Dunning-Kruger (as evinced from the diagrams sported here) indeed seems to make a weaker claim: that there's no correlation between “perceived ability” and “actual ability”. As such, this claim is as far from the "pop-sci conception" of Dunning-Kruger as it is from the author's.
[+] [-] stdbrouw|11 years ago|reply
Also, what corner of the universe are you from? Loess regression, hierarchical modeling, conditional analyses... methods for finding "non-global" correlations aplenty.
[+] [-] maxerickson|11 years ago|reply
[+] [-] throwaway9324|11 years ago|reply
"It’s a little easier to see why people would pass along the wrong story here, since it’s easy to misinterpret the data when it’s plotted against a linear scale, but it’s still pretty easy to see what’s going on by taking a peek at the actual studies."
Googling
http://wws.princeton.edu/news-and-events/news/item/two-wws-p...
"People’s life evaluations rise steadily with income, but the reported quality of emotional daily experience levels off at a certain income level, according to a new study by two Princeton University professors [...]"
So seemingly that has nothing to do with the scale used, but with the definition of happiness. Guess someone didn't look "at the actual study"?
[+] [-] notacoward|11 years ago|reply
[+] [-] gd1|11 years ago|reply
[+] [-] philh|11 years ago|reply
I don't think there's a single DK meme, but I do think that this is a common DK meme.
Somewhat-serious suggestion: maybe people who actually understand DK, have a tendency to underestimate how well they understand it relative to everyone else, and so assume that everyone else understands it about as well as they do.
[+] [-] anonymoushn|11 years ago|reply
[+] [-] dang|11 years ago|reply
This sort of gratuitous negativity isn't appropriate in a Hacker News comment. It is unnecessary to make your point and pollutes the atmosphere for all of us. Please don't do that.
[+] [-] logicallee|11 years ago|reply
[+] [-] drzaiusapelord|11 years ago|reply
At my previous job where I worked and managed a few young-ish support and jr devs, I came accross a lot of the same attitudes. The funny thing is that two or three years in, they tend to shed those attitudes (at least the smart ones do). They realize that this stuff is a lot bigger than the limited experience they've gotten in school or in their hobbyist projects. Suddenly the "lets toss out everything and do it my way" motor-mouting gets turned down a notch or two, especially after I let them do things "their way" once in a while only to have it explode in their faces. Hell, 90% of managing young techies is controlling their D-K until they mature into imposter syndrome. Then you have to manage that, which is a million times easier to deal because you're not being know-it-all'd to death in every meeting.
D-K is real. The graphs he points out literally supports it. Personally, I'm getting sick of this kind of uber-skeptic mentality that's popular here and on reddit. I think there's something ego gratifying for the INTJ male about being this loud-mouth contrarian who tells everyone they're wrong. Especially if its against some popular wisdom and if the argument is unusually pedantic and trivial.
[+] [-] deciplex|11 years ago|reply
This whole article is just a bag of fail.
[+] [-] circlefavshape|11 years ago|reply
[+] [-] rfergie|11 years ago|reply
The first three I was all like "yes! So glad someone articulate is finally saying this".
Then for type systems I was unable to overcome my current biases, thinking "what a load of tosh! Type systems are obviously a good thing".
Makes me wonder what it would actually take to break the Dunning-Kruger meme
[+] [-] kpmah|11 years ago|reply
The good news is that this opinion doesn't contradict evidence either! The research just isn't there yet. Some of research I've seen so far is pretty bad.
[+] [-] Kronopath|11 years ago|reply
[+] [-] nostrademons|11 years ago|reply
[+] [-] cheatsheet|11 years ago|reply
[+] [-] nickbauman|11 years ago|reply
[+] [-] praptak|11 years ago|reply
Key quote:
"I suspect we find this sort of explanation compelling because it appeals to our implicit just-world theories: we’d like to believe that people who obnoxiously proclaim their excellence at X, Y, and Z must really not be so very good at X, Y, and Z at all, and must be (over)compensating for some actual deficiency; it’s much less pleasant to imagine that people who go around shoving their (alleged) superiority in our faces might really be better than us at what they do.
Unfortunately, Kruger and Dunning never actually provided any support for this type of just-world view; their studies categorically didn’t show that incompetent people are more confident or arrogant than competent people."
[+] [-] Chathamization|11 years ago|reply
[+] [-] danohuiginn|11 years ago|reply
and to end a post about a paper people can't read with "Maybe I’m being naive here, but I think a major reason behind false memes is that checking sources sounds much harder and more intimidating than it actually is." -- that's just rubbing it in!
[+] [-] mhaymo|11 years ago|reply
[+] [-] m_y-n_a_m_e|11 years ago|reply
[+] [-] dang|11 years ago|reply
Angry venting against someone you think wrong is gratuitous negativity. So are generic dismissals like "This whole article is just a bag of fail."
If you think someone is wrong, please say why substantively, or say nothing.
[+] [-] lowbloodsugar|11 years ago|reply
[+] [-] DanBC|11 years ago|reply
A bit like Behind the Headlines, but about any science and by a wide range of science bloggers.
http://www.nhs.uk/news/Pages/NewsIndex.aspx
[+] [-] maaaats|11 years ago|reply
When media says "X causes cancer", he comes with a level headed blog-post explaining the data. When anti-vaxxers point to a study, he shows that they have misinterpreted it, etc.
Unfortunately, it's in Norwegian. I'm interested in reading more in this vein, so please let me know if you have good, English resources.
[1]: http://tjomlid.com/
[+] [-] cataflam|11 years ago|reply
However, I disagree with his interpretation of Dunning and Kruger's figures. There is limited data in those figures of course (3 sets with 12 points of data and 1 set with 8 points), and I don't know how big the actual sample sizes were. But in all 4 cases, it clearly and significantly shows that people with low actual ability overrate themselves and people with high actual ability underrate themselves.
> The pop-sci version of Dunning-Kruger is that, the less someone knows about a subject, the more they think they know.
I can only guess that he is interpreting that too literally. I always took the latter part as "the more they think they know [compared to what they actually do]". You could plot "perceived - actual" or "perceived / actual" instead of the figures in the paper to make it more obvious.
[+] [-] sergiosgc|11 years ago|reply
I read the data quite differently. I read that people don't use the full scale when rating themselves. The self rating scale goes from 60 to 90. If you correct for that, people kind of seem to do a nice job in self evaluation.
It can be explained, too. People have social pressure not to rate themselves near 100%, nor in the bottom half of the scale.
[+] [-] phtrivier|11 years ago|reply
* a straight line on a log scale - which is misleading, since the man on the street is used to straight lines on linear scales - telling that money buys happiness
* a "log-like" graph on a linear scale, which tells you that "money buys happiness a lot at the start, and then it doesn't change things a lot" - and the graph uses a linear scale on both axis, so there is less deformation.
What am I missing ?
[+] [-] Dylan16807|11 years ago|reply
But there is no inflection point. 10% more money always buys you the same amount of happiness.
Nothing makes $75k the point where you have 'enough' money as opposed to $5k. 75 million is just as far ahead of 5 million (at least if it continues to be logarithmic).
[+] [-] pc86|11 years ago|reply
For those of you who earn the requisite amount, is it true that it's easier to get a significant raise at the higher income levels? I live in a fairly rural area but I've found as my income rises the degree to which I need to fight for marginal pay increases has diminished, which seems counter-intuitive since each point gets more and more expensive, obviously.
[+] [-] Nimitz14|11 years ago|reply
Each figure shows how the bottom quartile is overestimating their ability. And that's exactly what people call the Dunning-Kruger effect.
[+] [-] emodendroket|11 years ago|reply
[+] [-] delibes|11 years ago|reply
[+] [-] serve_yay|11 years ago|reply
[+] [-] delibes|11 years ago|reply
It should probably be http://danluu.com/images/dunning-kruger/dunning_4.png which shows a positive correlation for "perceived logical reasoning ability and test performance". That then helps explain the statement "In two of the four cases, there’s an obvious positive correlation between perceived skill and actual skill".
[+] [-] Tenoke|11 years ago|reply
[+] [-] asuffield|11 years ago|reply
The pithy version is "a lie can run round the world before the truth has got its boots on", with thanks to Pratchett.
[+] [-] jacquesm|11 years ago|reply
I think your comment contains a highly recursive component:
http://quoteinvestigator.com/2014/07/13/truth/
[+] [-] unknown|11 years ago|reply
[deleted]