I actually interpret their data quite differently. There is a statistically robust and positive correlation between perceived and actual ratings:
- 44% of the time, the actual and perceived ranking was identical.
- 91% of the time, the difference between the rankings was at most one star.
- Only in 9% of cases was there was a strong disagreement about how well the interview went (2 or more stars out of four).
- There is also a bias towards engineers thinking they did slightly worse than they actually did.
Even in an alternative universe where the same interview was repeated, you would not expect rankings by the same individual to be perfect. A more meaningful benchmark (though probably unattainable) would be interviewer/interviewee consistency relative to interviewer self-consistency.
That's what I thought as well. I'm not sure it's a great idea to fit a straight line and then look at the R2. After all there's only three other values it can be, rather than a normal distribution.
The problem with that interpretation is that it ignores how much something deviates from the expected random. If everything was completely random:
- 25% of the time, the actual and perceived ranking would be identical.
- 62.5% of the time, the difference would be at most one.
- Only in 37.5% of cases would there be a strong difference in scores (2 or more stars out of four).
Would you say that 44% is a good score on a multiple choice test? Obviously candidates did better than random guessing, but not by enough to call it a strong correlation. Determining the strength of a correlation is what R-squared is for, which makes it the right choice for this analysis.
I think it's arguable that, when the question you're asking is 'how well do X and Y two things correlate with each other', the statistic you should be giving is r (which here is 0.49), not r^2. r^2 is the proportion of the variance of Y that's explained by X -- not that that isn't an interesting thing, but ITSM the correlation coefficient is more to what people expect a measure of relationship to be. (IANAStatistician)
Beat me to it. When I interviewed at Google there were multiple interviewers and then there were multiple committees and people who look at their feedback.
Comparing interviewers to other interviewers makes a "He did above average" from a person who's hard to impress as good as a "She did really great!" from someone else.
So many of you 20-somethings interviewing really make me cringe! Basically, many of you are a stark mirror in which I can see how terrible I was as an interviewer in my 20's and 30's.
Be honest now, do you do this?
1) See something or think of something
2) Decide that it's the most important point
3) Go on a "fishing" expedition to try and get the interviewee to see it
4) If the interviewee can't figure out what you're hinting at, decide
s/he's completely worthless! (Or if she gets it, decide she's the ultimate!)
A lot of you do this. Cut it out. It's idiotic. It presupposes a pretty arrogant view of the relative value of your own perceptions. What you really need to be doing is listening.
I had the experience of someone doing this to me with concurrency, then concluding I'm an impostor, while I'm telling him about 3 concurrent systems I've worked on. The disconnect? He was "fishing" for the current undergrad spiel about scheduling. Sorry, we covered it, but that wasn't emphasized in the same way when I was an undergrad, and the poorly thought out puzzle you came up with on the spot wasn't such a great hint.
Given the number of times I was mis-identified as an impostor programmer during interviews (a minority of instances, but still) there is clearly something going on. You guys are just like the young Ivy league grads in Mad Men. You just have different clothes and different language, but you are looking out for the signs of your own tribe just the same.
Eh - humans generally have a hard time rating things on a 5 star rating system. That's mainly because the discrete intervals between 1 and 5 stars aren't well defined so everyone will have a different opinion of what 3 stars "mean". I encountered this when I ran a rating system in Mechanical Turk. If you ask turkers to do something arbitrary like "rate the quality of this picture" it takes quite a number of ratings on the same image to get an accurate consensus.
It would be interesting to see if you can replicate experiment but replace the 5 stars with 5 yes/no questions regarding their performance and see if those match up
Whats worse is that any type of rating system has an utter bias towards the very very upper end of the scale by it's consumers.
If you have a true rating system then 2.5/3 out of 5 means average or more or less what you should be looking for in most cases 4 is out standing and 5 is bloody icecream pooping unicorn.
You see this every where a rating system is implemented heck in entertainment movies and games don't fight from 0 to 100 anymore but form the 90-100 if you see any score below 90 or 4 stars people usually avoid that product.
There are text descriptions under the stars, which are coincidentally rated out of 4, not 5. 3/4 is "Solid" for someone's technical skills. If someone does below average or amazing it's unlikely that the interviewer is going to select 3/4, regardless of their internal idea of what 3/4 means, when it says "Solid" underneath that.
Even worse because 1 star and 5 stars mean different things to different people, or even sometimes to the same person under different circumstances. If you asked me to rate my interview performance without first defining some objective criteria, for example, I'd have to spent a few minutes deciding whether (1 vs 5) stars is ("didn't get the job" vs "got hired with a nice title and negotiated a bunch of extra pay") or ("was arrested for assaulting the interviewer" vs "got the job").
I never cared about my own "interview performance".
But I always cared about my interviewer's assessment of me.
So much so that I've always reserved the last 10 minutes of every interview just to find that out.
They almost always ask if I have any questions, but even if they don't I ask them anyway:
Based on what you've just learned:
- How well would you say I did in this interview?
- How well do my skills fill your requirement?
- Where do you see me in the first month?
- Where do you see me in 3 years?
- What's the first thing you'd have me work on?
- What gaps do you think I have?
- What are your 3 biggest concerns about me?
- What can I do to address those concerns?
and of course:
- What is the next step?
I have never had an interviewer afraid to honestly (or appear to be honest) answer these questions. See how easy?
The last 10 minutes where you ask questions is a bad time to waste asking someone to judge your 3 year plan when they just met you. Most of these questions are better asked when you fail an interview and request feedback as to why.
I specifically state when I perform interviews that I will not answer the question "How well would you say I did in this interview?" The interview isn't over yet, and that's impossible to answer.
You should really spend this time to see how much you would want to work at the company. Ask about their processes, the things that would absolutely turn you off to a company or the things that would absolutely turn you on to a company and make you forget about all their other bullshit. Ask about the problems they are trying to solve. Ask about how smart the other people are who you'll be working with.
Don't waste time in the interview checking if you passed. You'll know very soon anyway and there is always the chance to get this feedback afterward.
Just to add another datapoint, some companies don't allow their interviewers to give feedback on how well you did right there. "How well would you say I did in this interview" would not be a question they could answer.
And "How well do my skills fill your requirement?" may not apply for some companies, where their interviewing practices don't look for a specific requirement but go for a more general set of characteristics.
Not commentating on whether this is effective or not, just that it is the case at several of the large companies you might imagine applying for these days.
I might point out that it's not in the company's best interest to answer any questions about your performance. If an interviewer gives you feedback, and they say something wrong, it could open the company up to a discrimination lawsuit.
You're better off asking questions about the business and trying to determine whether they're a good fit for you than you are deciding whether they're impressed by you or not. By assessing their business and their fit for you, you'll decide whether you want to accept any offer they may or may not decide to offer. If they don't offer, they weren't impressed, and they still shouldn't provide you much feedback because of aforementioned liability issues. If they do offer, then they were impressed, and you can ask those questions.
When I'm an interviewee I do the same thing. Opportunities to get interview feedback are already scarce, best to maximize them with questions like this.
When I'm interviewing, I always do a portion at the end to tell them at least one thing that impressed me and one thing that gave me pause.
I was once in an interview where the interviewer was a hard ass. Asking all kids of hard questions about languages I knew, then he got deeper and asked about specifics into those languages. Then started asking about really abstract stuff like different sorting algorithms, OOP concepts like polymorphism, etc. I felt I was holding my own but as the interview went on the interviewer seemed to grow more impatient and irritable. Needless to say I left the interview thinking I blew it. Two days later I got a call from the recruiter saying I was their top choice and got an offer soon after.
P.S. It turned out that guy was just grouchy because he had interviewed so many underwhelming candidates. He admitted he was just trying to break me.
Trying to break a candidate seems to be very related issue to the one the article is talking about. Most people will take that kind of interviewer attitude as a negative experience that reflects poorly on the company and would make them less interested in taking the job.
The goal of the interviewer should be to evaluate a candidate, not dig and dig until you find a weakness. If it takes that much effort it's not worth doing in the first place.
Terrific analysis, thank you! I love reading your posts.
I also think that the dynamics differ based on the mode of interviewing viz. Faceless (like http://interviewing.io does) vs face-ful (like normal onsite or video interviews, or the way http://interviewkickstart.com does).
I have an observation to share for the latter kind of interviews:
When I was hiring at Box for one of my teams (or even now), I'd generally go towards the end, to meet the candidate and talk to them about how the day went. I'd ask them about specific interviews, the interviewers and how they think they did on the given problems.
I found that it was surprisingly rare to get a candidate who was aware enough of how things went (positive or otherwise).
Digging into it, I realized that the correlation of what the candidate perceived, was simply to how expressive the interviewER was. If the interviewer was friendly and chatty, the candidate would feel like they have done well (or at least forgivably well). If the interviewer was more silent, the candidate would feel they haven't done very well.
Now that I do interview training for a living, I try and hammer this point home: that interviews are like a date. It depends a lot on the other person (in this case, the interviewer). Understanding this well often works like magic.
Nobody can judge their own interview performance - engineer or not. The hard part is that you as an interviewee don't really know what the interviewer is looking for. You may have nailed the back half of the interview, but you don't have the one skill they're looking for. Or sometimes you don't blow it at all and they love you, but there's one other person they interviewed for the position that blew them away.
Trying to guess how well you did in an interview is folly. Hell, one interview tactic I use with people is to push them until they break: I keep asking technical questions until I reach a point where you say "I don't know". This sometimes makes people think they "failed" the interview, when all I'm really trying to do is see how much you know while also seeing how much you'll bullshit me on topics you don't know as well. Nobody can be an expert on everything, and interviews are as much about behavioral profiling as they are subject matter expertise.
I've found this with myself as well. In one of my worst-ever interviews, I had the flu really badly the entire time, and read the signals of all of my interviewers to be both negative regarding my technical chops and negative toward my weakened state. I got the job, and was very well liked. In another trainwreck of an interview where I seriously considered standing up and leaving after getting extreme negative signals, I was offered the job as well-- but I should have paid attention to the negative signals, as they were indicative of larger problems with that group.
In other interviews, I've massively overestimated my own performance, seeing positive signals everywhere and being very impressed with myself-- of course, these didn't pull through, and my confidence was dashed. In another incidence, I did get the job offer.
The most confusing are the interviews in which I don't have a strong feeling of doing well or poorly either way-- not the median case, but certainly it happens frequently enough that I've gotten a job where I didn't think that I stood out. If anything, these cases are a consolation that others may see something positive that I don't see in myself.
This tells me that my ability to correctly predict my interview performance is no more efficacious than random chance. I think that there are a few confounding factors which make interviewee measuring interview performance as perceived by the interviewer difficult. Interviews are a time of endless posturing, flagrant lies, propaganda, and overt deception hopefully mingled in with actual mutual interest, excitement, and good will. There are many social signals to keep track of in addition to self-monitoring to ensure the correct outcome. For people who are not the strongest socially, I think that these social signals tend to get dropped intermittently as attention shifts inward (don't say the wrong thing) or outward (get this technical problem correct).
It's a fools errand to try and predict interview performance. I've been interviewing for a short while now and haven't gotten an offer yet despite feeling really great (and for some, really crappy) about my interviews. Feedback from the interviewers is equally inaccurate. On my most recent interview, I was told that my solution to their coding challenge was one of the best they'd ever seen, and that the feedback so far was extremely positive. Two days later I got the "another direction" email. I don't really take rejection to heart anymore but I hate not knowing what I need to do better.
Sometimes it has nothing to do with your performance. Budget cuts, downsizing, sudden hiring freezes, or possibly you were the last candidate but the VP is already sold on another guy. Sometimes job parameters shift, they are looking for different areas of expertise and they don't realize it til after they interview a bunch of people.
> Hopefully this gets more love than it did last time
We invited the submitter to repost it for that reason. I'll try to merge the comments from there into here.
Invited reposts are mostly deprecated now in favor of re-ups [1], but when it looks like the submitter might also be the author (as e.g. with Show HNs), we still send them. It's nice for an author to know that their post may still get discussed, and it's good for HN when an author jumps into the thread.
>And by extension, it means that in every interview cycle, some portion of interviewees are losing interest in joining your company just because they didn’t think they did well, despite the fact that they actually did.
I hate the assertion of causation here. As always, this relationship means that A causes B, B causes A, and/or A and B have a common cause (or even more possibilities, given that we're essentially conditioning on this event taking place). Aline argues that "some portion of interviewees are losing interest in joining your company just because they didn’t think they did well," but I think it's more plausible that there is something that happens in the interview that simultaneously makes the interviewer think they did poorly and makes them think poorly of the company. If you're rating me on my ability to name country capitals in a statistics interview, I'm going to think you don't want to hire me and I'll think you're an idiot I'd hate to work for. Or if you're being kind of a jerk or being impatient and checking your cell phone every 30 seconds, I'll think I'm doing poorly and that you're rude and I wouldn't like to work with you. Common cause seems much more likely to me.
Maybe resume's specifically do suck, but surely it's beneficial to have at least some way to see an overview of what a candidate has accomplished in the past. Especially when you're hiring for more senior roles where you need to place more value on experience and a proven track record than raw technical excellence. I can't really see these "anonymous" interviewing platforms working well for those kinds of roles.
I have seen hundreds of CVs that show no achievements of note - and tend to be more along the lines of describing the duties of the job and little else besides. I've also had hundreds of subsequent conversations with those people that revealed significant accomplishments.
CVs are better than nothing. But there's no set way to write them that's widely known, and people tend to be rather reticent about their own accomplishments - perhaps not even thinking of them as such. There's a lot of value being left on the table.
[+] [-] shoyer|10 years ago|reply
- 44% of the time, the actual and perceived ranking was identical.
- 91% of the time, the difference between the rankings was at most one star.
- Only in 9% of cases was there was a strong disagreement about how well the interview went (2 or more stars out of four).
- There is also a bias towards engineers thinking they did slightly worse than they actually did.
Even in an alternative universe where the same interview was repeated, you would not expect rankings by the same individual to be perfect. A more meaningful benchmark (though probably unattainable) would be interviewer/interviewee consistency relative to interviewer self-consistency.
[+] [-] lordnacho|10 years ago|reply
[+] [-] andimus|10 years ago|reply
- 25% of the time, the actual and perceived ranking would be identical.
- 62.5% of the time, the difference would be at most one.
- Only in 37.5% of cases would there be a strong difference in scores (2 or more stars out of four).
Would you say that 44% is a good score on a multiple choice test? Obviously candidates did better than random guessing, but not by enough to call it a strong correlation. Determining the strength of a correlation is what R-squared is for, which makes it the right choice for this analysis.
[+] [-] SEMW|10 years ago|reply
I think it's arguable that, when the question you're asking is 'how well do X and Y two things correlate with each other', the statistic you should be giving is r (which here is 0.49), not r^2. r^2 is the proportion of the variance of Y that's explained by X -- not that that isn't an interesting thing, but ITSM the correlation coefficient is more to what people expect a measure of relationship to be. (IANAStatistician)
[+] [-] mbizzle88|10 years ago|reply
[+] [-] jMyles|10 years ago|reply
[+] [-] JFlash|10 years ago|reply
Comparing interviewers to other interviewers makes a "He did above average" from a person who's hard to impress as good as a "She did really great!" from someone else.
[+] [-] krick|10 years ago|reply
[+] [-] unknown|10 years ago|reply
[deleted]
[+] [-] stcredzero|10 years ago|reply
Be honest now, do you do this?
A lot of you do this. Cut it out. It's idiotic. It presupposes a pretty arrogant view of the relative value of your own perceptions. What you really need to be doing is listening.I had the experience of someone doing this to me with concurrency, then concluding I'm an impostor, while I'm telling him about 3 concurrent systems I've worked on. The disconnect? He was "fishing" for the current undergrad spiel about scheduling. Sorry, we covered it, but that wasn't emphasized in the same way when I was an undergrad, and the poorly thought out puzzle you came up with on the spot wasn't such a great hint.
Given the number of times I was mis-identified as an impostor programmer during interviews (a minority of instances, but still) there is clearly something going on. You guys are just like the young Ivy league grads in Mad Men. You just have different clothes and different language, but you are looking out for the signs of your own tribe just the same.
[+] [-] ArkyBeagle|10 years ago|reply
It's interesting isn't it? This was all much easier in decades past. The questions don't matter. The answer doesn't matter.
What matters are the body language cues.
[+] [-] konradb|10 years ago|reply
[+] [-] invalidfunction|10 years ago|reply
It would be interesting to see if you can replicate experiment but replace the 5 stars with 5 yes/no questions regarding their performance and see if those match up
[+] [-] dogma1138|10 years ago|reply
If you have a true rating system then 2.5/3 out of 5 means average or more or less what you should be looking for in most cases 4 is out standing and 5 is bloody icecream pooping unicorn.
You see this every where a rating system is implemented heck in entertainment movies and games don't fight from 0 to 100 anymore but form the 90-100 if you see any score below 90 or 4 stars people usually avoid that product.
[+] [-] pc86|10 years ago|reply
[+] [-] saulrh|10 years ago|reply
[+] [-] zyxley|10 years ago|reply
[+] [-] InclinedPlane|10 years ago|reply
[+] [-] edw519|10 years ago|reply
But I always cared about my interviewer's assessment of me.
So much so that I've always reserved the last 10 minutes of every interview just to find that out.
They almost always ask if I have any questions, but even if they don't I ask them anyway:
Based on what you've just learned:
and of course: I have never had an interviewer afraid to honestly (or appear to be honest) answer these questions. See how easy?[+] [-] NickLarsen|10 years ago|reply
I specifically state when I perform interviews that I will not answer the question "How well would you say I did in this interview?" The interview isn't over yet, and that's impossible to answer.
You should really spend this time to see how much you would want to work at the company. Ask about their processes, the things that would absolutely turn you off to a company or the things that would absolutely turn you on to a company and make you forget about all their other bullshit. Ask about the problems they are trying to solve. Ask about how smart the other people are who you'll be working with.
Don't waste time in the interview checking if you passed. You'll know very soon anyway and there is always the chance to get this feedback afterward.
[+] [-] kajecounterhack|10 years ago|reply
And "How well do my skills fill your requirement?" may not apply for some companies, where their interviewing practices don't look for a specific requirement but go for a more general set of characteristics.
Not commentating on whether this is effective or not, just that it is the case at several of the large companies you might imagine applying for these days.
[+] [-] Jemaclus|10 years ago|reply
You're better off asking questions about the business and trying to determine whether they're a good fit for you than you are deciding whether they're impressed by you or not. By assessing their business and their fit for you, you'll decide whether you want to accept any offer they may or may not decide to offer. If they don't offer, they weren't impressed, and they still shouldn't provide you much feedback because of aforementioned liability issues. If they do offer, then they were impressed, and you can ask those questions.
[+] [-] someone7x|10 years ago|reply
When I'm interviewing, I always do a portion at the end to tell them at least one thing that impressed me and one thing that gave me pause.
[+] [-] kelukelugames|10 years ago|reply
Are they being mean because they think I wasted their time or because I killed it and they want to push me?
[+] [-] joeax|10 years ago|reply
P.S. It turned out that guy was just grouchy because he had interviewed so many underwhelming candidates. He admitted he was just trying to break me.
[+] [-] pgodzin|10 years ago|reply
The goal of the interviewer should be to evaluate a candidate, not dig and dig until you find a weakness. If it takes that much effort it's not worth doing in the first place.
[+] [-] kvcrawford|10 years ago|reply
Sounds like a great guy to work with.
[+] [-] soham|10 years ago|reply
I also think that the dynamics differ based on the mode of interviewing viz. Faceless (like http://interviewing.io does) vs face-ful (like normal onsite or video interviews, or the way http://interviewkickstart.com does).
I have an observation to share for the latter kind of interviews:
When I was hiring at Box for one of my teams (or even now), I'd generally go towards the end, to meet the candidate and talk to them about how the day went. I'd ask them about specific interviews, the interviewers and how they think they did on the given problems.
I found that it was surprisingly rare to get a candidate who was aware enough of how things went (positive or otherwise).
Digging into it, I realized that the correlation of what the candidate perceived, was simply to how expressive the interviewER was. If the interviewer was friendly and chatty, the candidate would feel like they have done well (or at least forgivably well). If the interviewer was more silent, the candidate would feel they haven't done very well.
Now that I do interview training for a living, I try and hammer this point home: that interviews are like a date. It depends a lot on the other person (in this case, the interviewer). Understanding this well often works like magic.
[+] [-] exelius|10 years ago|reply
Trying to guess how well you did in an interview is folly. Hell, one interview tactic I use with people is to push them until they break: I keep asking technical questions until I reach a point where you say "I don't know". This sometimes makes people think they "failed" the interview, when all I'm really trying to do is see how much you know while also seeing how much you'll bullshit me on topics you don't know as well. Nobody can be an expert on everything, and interviews are as much about behavioral profiling as they are subject matter expertise.
[+] [-] cryoshon|10 years ago|reply
In other interviews, I've massively overestimated my own performance, seeing positive signals everywhere and being very impressed with myself-- of course, these didn't pull through, and my confidence was dashed. In another incidence, I did get the job offer.
The most confusing are the interviews in which I don't have a strong feeling of doing well or poorly either way-- not the median case, but certainly it happens frequently enough that I've gotten a job where I didn't think that I stood out. If anything, these cases are a consolation that others may see something positive that I don't see in myself.
This tells me that my ability to correctly predict my interview performance is no more efficacious than random chance. I think that there are a few confounding factors which make interviewee measuring interview performance as perceived by the interviewer difficult. Interviews are a time of endless posturing, flagrant lies, propaganda, and overt deception hopefully mingled in with actual mutual interest, excitement, and good will. There are many social signals to keep track of in addition to self-monitoring to ensure the correct outcome. For people who are not the strongest socially, I think that these social signals tend to get dropped intermittently as attention shifts inward (don't say the wrong thing) or outward (get this technical problem correct).
[+] [-] vdnkh|10 years ago|reply
[+] [-] joeax|10 years ago|reply
[+] [-] vonmoltke|10 years ago|reply
I found the analysis very interesting and consistent with my personal bias (I almost always think I did worse than I actually did).
[+] [-] dang|10 years ago|reply
We invited the submitter to repost it for that reason. I'll try to merge the comments from there into here.
Invited reposts are mostly deprecated now in favor of re-ups [1], but when it looks like the submitter might also be the author (as e.g. with Show HNs), we still send them. It's nice for an author to know that their post may still get discussed, and it's good for HN when an author jumps into the thread.
[1] All the details at https://news.ycombinator.com/item?id=10705926 and previous posts linked there.
[+] [-] imh|10 years ago|reply
I hate the assertion of causation here. As always, this relationship means that A causes B, B causes A, and/or A and B have a common cause (or even more possibilities, given that we're essentially conditioning on this event taking place). Aline argues that "some portion of interviewees are losing interest in joining your company just because they didn’t think they did well," but I think it's more plausible that there is something that happens in the interview that simultaneously makes the interviewer think they did poorly and makes them think poorly of the company. If you're rating me on my ability to name country capitals in a statistics interview, I'm going to think you don't want to hire me and I'll think you're an idiot I'd hate to work for. Or if you're being kind of a jerk or being impatient and checking your cell phone every 30 seconds, I'll think I'm doing poorly and that you're rude and I wouldn't like to work with you. Common cause seems much more likely to me.
[+] [-] unknown|10 years ago|reply
[deleted]
[+] [-] lewisl9029|10 years ago|reply
Do they?
Maybe resume's specifically do suck, but surely it's beneficial to have at least some way to see an overview of what a candidate has accomplished in the past. Especially when you're hiring for more senior roles where you need to place more value on experience and a proven track record than raw technical excellence. I can't really see these "anonymous" interviewing platforms working well for those kinds of roles.
[+] [-] 6d0debc071|10 years ago|reply
I have seen hundreds of CVs that show no achievements of note - and tend to be more along the lines of describing the duties of the job and little else besides. I've also had hundreds of subsequent conversations with those people that revealed significant accomplishments.
CVs are better than nothing. But there's no set way to write them that's widely known, and people tend to be rather reticent about their own accomplishments - perhaps not even thinking of them as such. There's a lot of value being left on the table.
[+] [-] unknown|10 years ago|reply
[deleted]
[+] [-] lazyant|10 years ago|reply
[+] [-] skue|10 years ago|reply
[1] https://en.m.wikipedia.org/wiki/Dunning–Kruger_effect
[+] [-] cjrjdjcnd|10 years ago|reply
[+] [-] vonmoltke|10 years ago|reply
[+] [-] jnbiche|10 years ago|reply
Not to mention the not insignificant number of EE who have fully transitioned over to software, some of whom may have taken the exam at some point.
Likewise, not all "real" engineers have taken the PE or FE exam. Lots of electrical engineers don't bother unless their job requires it.