top | item 15599111

The two questions I ask every interviewer

307 points| deafcalculus | 8 years ago |blog.wesleyac.com

303 comments

order
[+] nmalaguti|8 years ago|reply
Each time I read a post like this I say to myself “they’re not wrong, but something is also missing.”

If there was a way to evaluate a candidate based on how well they would do at their actual job, in an hour or less, that couldn’t be games or cheated, we’d all be using it.

That isn’t to say we shouldn’t work on improving the hiring process, but how much time, both from the perspective of the candidate and the company, is too much time to invest in a potential hire?

Would we get better signal if you came to work for us for a week and paired with everyone on the team? Sure! But how many candidates can take a week off from their current job? The author talked to 11 companies. Could they invest 11+ weeks in their job search full time in order to find their next role? What about the loss in productivity? Can you have more than one candidate in the office in a given week? What if your ramp up time for full time employees is actually longer than a week? Are you biasing for people who ramp up fast instead of people who will be ultimately more productive and impactful?

I often feel that engineers write these posts because they know that they themselves are good at their job, but feel it is silly to have to develop an alternative set of skills in order to signal that they are a good hire. Or they get rejected and blame the process or those alternative skills. I’ve felt the same way and wanted the system to be better tailored for my skill set, but balanced against the time investment for some alternatives, I see why we have the process we do.

[+] grue2|8 years ago|reply
I use a work sample (short paid contract), but it's not perfect and boy is it labor intensive.

As a hirer, you really can't win. There's not one hiring process nor guiding principle that doesn't seem to bring out the pitchforks of those who were frustrated, disrespected, or rejected by said process:

- Show us your open source work. (You're excluding all but a lucky few who have the privilege of writing open source code!)

- Okay then, show us a personal project or some work you've done in your free time. (What, so I'm expected to live eat and breathe code 24/7 to get hired?!)

- Well, how about a short contract/work sample? (How am I supposed to find the time to do that? I have a day job and a life!)

- Shall we try whiteboarding/coding tests then? (This is so insulting! Solving CS puzzles isn't what the job is about!)

One cannot, sadly, rely on the résumé. I have interviewed multiple self-deemed "experts" in such-and-such language, only to find that they could not even write a basic for-loop on the whiteboard.

[+] bootsz|8 years ago|reply
> I often feel that engineers write these posts because they know that they themselves are good at their job, but feel it is silly to have to develop an alternative set of skills in order to signal that they are a good hire.

It does feel silly. But sadly it's often the way the world works. Standardized tests carry many of the same problems and benefits. Getting a perfect SAT score doesn't really speak to your level of intelligence as much as it speaks to your ability to dedicate lots of time and focused energy toward preparing for an arbitrary system of evaluation... which turns out to correlate somewhat well with being successful in school. There are few false positives but many false-negatives, just like with whiteboard-style coding interviews.

Is this fair? Maybe, maybe not.

What bothers me more than anything is that people either don't realize this distinction, or aren't honest about it. I was once rejected from a "big four" company after a whiteboarding on-site, and I was advised by the recruiter to "spend a year working somewhere with a very large codebase" and try again... As if that would contribute anything toward my ability to pass a whiteboarding interview. We all know the real way to pass whiteboarding interviews is to practice whiteboarding interview problems. A lot. I've been working as a software engineer on "very large codebases" for a while now and can confirm that I am no better prepared to pass that interview.

[+] crdoconnor|8 years ago|reply
There is. Take a set of realistic, representative tasks from your day to day work, decouple it from the context as much as possible to limit the need company specific services, software or knowledge while performing the task.

There's a few reasons this is rare:

* Setting up a test like this requires prep work, whereas a chat, recycling old tests and downloading questions from the internet does not.

* It's rare for hirers to evaluate or be evaluated on their own hiring process.

* Lots of developers want to cargo cult Google's hiring process so they ask you to whiteboard algorithms.

[+] lr4444lr|8 years ago|reply
I agree, and I'm not sure what these kinds of posts are trying to accomplish, because they're speaking from hindsight and ignore with many other counterfactual possibilities that don't happen to support their point. Every alternative I've seen proposed to FizzBuzz and Whiteboard style tactics is flawed or self-selecting in its own way, and often come at a higher cost to one or both of the interviewee or the company as the candidate pool grows. Cultural fit getting a lot of flack these days for reinforcing unconscious bias is hardly proven fact, but even if it were, it would not be too hard to identify and reach consensus beforehand on what the team lacks, e.g. someone who is a testing fanatic, a security wizard, a deep dive experimentalist, etc. and to instruct all interviewers on how to spot those needed elements.
[+] JamesBarney|8 years ago|reply
> but feel it is silly to have to develop an alternative set of skills in order to signal that they are a good hire

The problem isn't that they have to develop a set of skills that combined with their true skills gives the interviewer a glimpse into their abilities.

It's that it's an entirely different set of skills entirely. We might as well be asking people to learn woodworking in order to showcase their software skills. The ability to sound knowledgeable and make someone like you in 50 minutes is pretty orthogonal to writing good code as a member of a team.

[+] rlpb|8 years ago|reply
> I often feel that engineers write these posts because they know that they themselves are good at their job...

The Dunning-Kruger effect suggests that almost certainly, some non-zero proportion engineers who have written this type of posts are not.

https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

[+] logicallee|8 years ago|reply
>If there was a way to evaluate a candidate based on how well they would do at their actual job, in an hour or less, that couldn’t be gamed or cheated, we’d all be using it.

Right, because there's currently a ten million dollar bounty up at way-to-evaluate-a-candidate-based-on-how-well-they-would-do-at-their-actual-job-in-an-hour-or-less-that-cant-be-gamed-or-cheated-bounty .com - which everyone knows about and has known about for decades, and checks regularly. The site can't be gamed and it's run as a public service. Literally within a day, and I mean a day, that anyone ever uses the winning method a single time, God himself will write up the method and submit it on the interviewer's behalf (invisible hand hypothesis), the site will judge it correctly and approve it (just world hypothesis), and the next day we'd all be using it. Oh also there's a genie that makes people use the winning entry I forgot to mention that, otherwise people might not all use it even once it's well-known.

I mean just consider the wildly implausible set of circumstances it would take for not everyone in the world to be using the best interview process in the world!

It's clear that it just doesn't exist. How could it?

/s

-

Edit: someone didn't like this. Just imagine what method you suppose by which everyone would learn the perfect interview method from each other. If that method exists, why would everyone know it?

[+] dorfsmay|8 years ago|reply
The "good fit" sent shivers through my spine!

I'm a bit of an odd ball in terms of background, and in my early 20's I interviewed for really interesting jobs at really interesting companies (Intel, oracle, Digital, and some more). Every single time the tech staff I would have worked with finished the interviews by saying "You just need one last interview with HR, it will be nothing, looking forward to work with you!".

And every single time the answer from HR was that " I would not fit in".

After 4 or 5 companies turning me down I thought fuck this, I don't want to fit in, I want to produce good quality work and make customers happy, and started to look exclusively for contracts. Thirty years later, I'm still contracting, every single of my customers has been very happy with my work.

Funnily enough, nobody tried to hire me as an employee in those 30 years! And they were right, I'd probably wouldn't fit in. I don't accept status quo, I don't buy in cargo cult, I'm not interested in coming in to wait for the hours to go by.

Remember, "fitting in" is not necessarily a good thing. People hire me because I don't.

[+] vita17|8 years ago|reply
The purpose of HR departments is to avoid liabilities. The purpose of rejecting “bad fit” candidates is to avoid losing their job if someone who looks or acts odd ends up being a liability. “You should have know he’d burn the building down. He was weird. He had a nose piercing!” So they avoid hiring anybody with a characteristic that could be seen as a red flag in retrospect.

It’s a product of the corporate mentality.

[+] marnett|8 years ago|reply
Do you think the amount of contracts available has changed over your career?
[+] pcunite|8 years ago|reply
dorfsmay,

I'm trying to look at your résumé, following the link in your HN profile. It gives a 404.

[+] chrisbennet|8 years ago|reply
"In 1985, Freada Klein (then head of organizational development for Lotus) did an experiment... With Kapor’s permission, Klein pulled together the résumés of the first forty Lotus employees... Klein explained that most of these early employees had skills the growing company needed, but many had done “risky and wacko things” such as being community organizers, being clinical psychologists, living at an ashram, or like Kapor, teaching transcendental meditation.

Then Klein did something sneaky. She submitted all forty resumes to the Lotus human resources department.

Not one of the forty applicants, including Kapor, was invited for a job interview. The founders had built a world that rejected people like them."

http://vault.theleadershiphub.com/blogs/if-you-werent-boss-w...

[+] blfr|8 years ago|reply
It's unclear whether this was and is a mistake. You need different kind of people in the beginning and when the company is executing a tested business model.

We can usefully divide intellectual tasks into two sets: filling and framing. Fillers add more useful detail or content within some framework, while framers explore possible new frameworks. While both tasks are essential, framing has higher variance; most frame attempts fail, but a few produce great value.

http://www.overcomingbias.com/2006/12/fillers_neglect.html

This definitely causes issues later on when the organization or movement becomes incapable of replacing the original founders. The founders would need to both hire a lot of fillers and also mentor others like them to take over in the future. But this causes enormous resentment. Fillers will feel (absolutely correctly, this is exactly what happens) passed over for promotions that they by every other metric thoroughly deserve.

[+] jakobegger|8 years ago|reply
Why should a company hire people that are similar to the founders?

At an early stage, companies need very different people than later on. At first, they need generalists; once you reach a certain size, you need specialists.

The founders should be self-driven, they should have a vision, they should know about all parts of the business. Later employees should be excellent in their speciality, should have execution skills, should be able to follow someone else's vision.

[+] mlashcorp|8 years ago|reply
I always interview for "culture fit". In my experience, that's almost always the bottleneck, and the technical part is rarely a problem. Learning a new skill is not an earth-shattering problem if you have a good attitude towards work, and good interpersonal skills are a must in teams
[+] _chris_|8 years ago|reply
>If you want to make a change to your interview process, give it to some of your current employees first. If anyone fails it, ask yourself if that person should be fired. The answer is probably no.

That's a fascinating idea that's never crossed my mind. E.g., how many current employees can pass whiteboard coding exams?

[+] ssijak|8 years ago|reply
Interviewing is a random event for many companies. You as a candidate could fail miserably on one day and get great feedback the other day. It depends on so many factors in larger companies. For example in one of my previous employers, they would send a random engineer (juniors included) to interview a candidate and give a mark from 1-10. There were engineers who would not pass their own interview and questions, and on the other hand, there were others who would just see that you can code anything and you are cool to work with and would let you in. And anything in between. There were times where juniors wanted to be smart and asked some puzzles they got from the net, and when later I asked them to solve them they failed. So many interviews depend on so many random events and circumstances and.

The biggest takeaway I got from all this experience is that as a candidate I should not be intimidated by anything and should just be cool and myself, because there is so much randomness that trying to find reason in it will just make things worse. And ALWAYS negotiate as much as you can because your biggest raise is when you get hired.

[+] pavlov|8 years ago|reply
Many companies instruct candidates should study for interviews with specific books or courses (e.g. “Cracking the Code Interview”).

That indicates the idea is not that someone who is a competent professional should pass without studying. The passively hostile whiteboard coding exams are more like a hazing ritual than an actual job performance test.

[+] FeatureRush|8 years ago|reply
Apparently there are already companies that take it a step further. They test sample of their employees with some fancy IQ test that measures them across several dimensions. From those results profile of desirable new hire is build and all candidates get measured by the same test.

On reddit in threads about interviews I've already seen couple people discussing those strange tests where in many questions there are no obvious correct/wrong answers, like for example all presented shapes can in their own way fit in presented pattern.

[+] BeetleB|8 years ago|reply
>That's a fascinating idea that's never crossed my mind. E.g., how many current employees can pass whiteboard coding exams?

I once confronted a colleague who had refused a candidate because he couldn't answer a technical question. When I heard the question, I asked him "How many people in the current team can answer this?" (Hint: Only one). His response: "Yeah, but we told him to read the first 6 chapters of textbook X!"

So we denied him a job because he didn't do his homework.

[+] tomseldon|8 years ago|reply
Testing the change is one thing, but if the original people were hired and asked something very similar, you have an existing group selected based on that characteristic and so will likely do well.

e.g. if you have always done a whiteboard interview, then you've selected for people that do well in that type of scenario. Changing the question but keeping the format will likely just prove that those people are still suited to that type of task, but not that an otherwise good hire will be filtered out.

I'm not against whiteboard interviews or similar, but just pointing out that selection bias is an important factor here if you're judging effectiveness by comparing it to results for existing employees.

[+] lucideer|8 years ago|reply
> less than 10% of the companies that I've interviewed with have said that their interview process was designed to evaluate how effectively someone could do the job that they're being hired for.

I don't think an interview process can do this tbh.

A good interview process can evaluate if they can do the job that they're being hired for, in most cases, but I would say only an actual trial/probationary period can evaluate how effectively they can do it. And even then, how effective they are may be environmental - completely external to their own internal competence.

After concluding that a candidate can do the job, the secondary purpose of the interview process is filtering for people that will positively affect the work environment of their colleagues (and/or filter out those who may negatively affect others). As an interviewer, I would make some effort not to lean too heavily on my own biases and try and also consider how they'd get on with my colleagues that I know well, but there is unfortunately always going to be some implicit bias here.

[+] arkadiytehgraet|8 years ago|reply
As an interviewee, one of the questions (that I shamelessly stole from somewhere on HN) I like to ask recruiters is this: "Do you like working here?" The answers and the way the answers are given usually tell you quite a lot about the company and the specific place you are going to work.
[+] comstock|8 years ago|reply
It’s a fair question, but if you asked me that in front of my boss or coworkers I would likely immediately take a disliking to you because you put me in an awkward situation where there are few easy ways to answer honestly.
[+] dandare|8 years ago|reply
His point, that hiring process is not a scientific process, is probably correct. But the same is true about performance review process (in my opinion). At best, the performance review is unscientific and biased, at worst it is a popularity contest, while usually, it is a formality.

> I've never had an interview that tried to evaluate my ability to work in a team or prioritize tasks - both of which are probably more important to doing well in a software engineering job than being able to find anagrams in O(n) time.

And I have never seen a test that could reliably evaluate your "ability to work in a team or prioritize tasks".

The hiring process is usually unscientific because making it reliably scientific is very expensive.

[+] rafiki6|8 years ago|reply
Yes, but what about the commonly trotted out line of "hiring mistakes are expensive so we'd prefer to not hire than make a mistake". Perfect example of having cake and eating it...You have to eat the cost somewhere. Either it's in the hiring process, the firing of a bad employee, or spinning your wheels waiting for the right candidate.
[+] conductr|8 years ago|reply
I think this is a bit of a slippery slope. In that, if most employers really evaluated what skills they needed over cultural fit and their desire for _The Best_ then, I postulate many of these companies would also realize they have absolutely no reason to be paying super high US/EU salaries, especially the salaries for tech talent in the tech hubs.

They would realize that they probably just need a kick ass project management framework and access to a pool of low paid but capable programmers (eg. a remote team in India/China/etc). Most startup tech isn't solving any huge tech problems, it's just building to spec/vision and iterating based on feedback and maybe if you're lucky you get to do this under load/at scale.) I know things have been outsourced with countless examples of bad results but I think it's really just a project management problem - not a skills gap (again for most startup tech we see today, but probably a significant portion of enterprise jobs too).

I have more of a bootstrap mindset for most things. So I almost never understand the $1M seed funded startups that get an office and hire a few programmers in SV and instantly only have 6 months of runway just to build a Tinder clone or something with little tech complexity. In my mind, they could easily outsource development and spend 90% of their money on sales.

Edit: I say "outsource" a lot above, I actually mean remote teammates that you manage

[+] notacoward|8 years ago|reply
He almost gets it right. Hiring for some abstract "best" isn't so great. Hiring for "culture fit" is practically an invitation to discriminate. OTOH, people rarely do only one thing at a company, so it's important to remember that you're hiring for a job rather than a specific task. You need to evaluate fundamental skills, including collaboration or leadership skills (and styles), not just knowledge of specific subject matter.
[+] throw999888away|8 years ago|reply
I have mixed feelings on culture fit becoming a bad word.

I don't disagree with your point but I think there is a danger with ignoring its value.

For example --leaving aside your views on SV startup sweatshops-- if you're in a start up, you will at some point work late hours and you will at some point go through stressful times. Sometimes the things that get you through those times is that feeling of camaraderie and actually getting on with your colleagues.

Screening on culture-fit is a good way to guard for that, and at least for me it is unwise to not associate value with that.

Now, it could be that your culture is inclusivity. But then isn't hiring for people who fit that mould still hiring for culture fit? Isn't that still creating a monoculture? Isn't discriminating against a bigot brogrammer (who may well be great at coding) still discrimination?

[+] listentojohan|8 years ago|reply
One of the good interview experiences I had, included going through a project I had previously done, framing the problem, how I'd have worked through it (with class/function level whiteboarding), and what could be improved. It was a project from a previous job, but too high level to be replicated. But could see how going through work from a previous job could be problematic (no code was shown).
[+] pjc50|8 years ago|reply
Note that if your company is small, it can be very hard to do any kind of meaningful statistical process control on hiring.

The "track careers of people you rejected" idea is an interesting one, but depends very much on how public people make their careers. I suppose you can rely on LinkedIn for 90% of people.

(What is HN's opinion of the Stack Overflow Developer Story?)

[+] richmarr|8 years ago|reply
We run an early stage applicant tracking system and see ourselves broadly as a rejection engine. Something like 95% of the activity on our system relates to candidates who end up being rejected.

Since our assessment data is structured our current iteration of the product uses that to create value for rejected candidates, informing them where they were strong or weak in comparison to the rest of the group, and showing them some aggregate scoring.

It goes down very well with hiring managers, who typically don't have the time to respond in detail or at all to rejected candidates, and (for the most part) goes down very well with candidates.

[+] speby|8 years ago|reply
>The "track careers of people you rejected" idea is an interesting one, but depends very much on how public people make their careers.

I really like this idea. This reminds me a bit of the "anti-portfolio" that some VCs track (aka a bittersweet regret of the unicorns they passed on years ago). I think the same could be said of great talent that an organization passed on that then goes on to build great products, invent, and/or otherwise lead great teams despite that.

However, employment is a tad different than investment so, to me, the only caveat to this is that it simply isn't the case that every super talented person will bring about great results and value creation at every single organization.

Much of what a talented person's output depends on is the very room full of other people in which that person works with. And while "fit" is such a cliche thing these days, this is what matters so much in terms of whether or not an otherwise talented person will be successful or not within one organization versus another.

[+] ec109685|8 years ago|reply
Very good point. A nice blog post illustrating the problem with drawing conclusions on small sample sizes: https://jvns.ca/blog/2014/07/11/fun-with-stats-how-big-of-a-...

You need a lot more than a startup’s number of employees and their eventual job reviews to make strong conclusions on the quality of the interview process, and need to rely more on industry best practices instead.

[+] cubano|8 years ago|reply
There are so many cognitive biases in play when it comes to hiring its basically a total crapshoot no matter what processes and shit is in place.

People hire people they like, period.

[+] rafiki6|8 years ago|reply
Disagree, we can basically eliminate the personal part of it. Blind interviewing. I don't need to know name/gender/school etc. Technical skills test as a filter given to current employees to track if it actually tests for necessary skills. Second part of interviewing can be assessed by people who don't see the candidate and we can obscure their voices.
[+] comstock|8 years ago|reply
> If you want to make a change to your interview process, give it to some of your current employees

I don’t think I would respond well to this as an employee.

[+] dx034|8 years ago|reply
Why not? I think it's a valid test for a company. If the majority of your employees wouldn't pass the hiring test, why use it for new employees (unless you're looking for someone more senior)? Employees can probably best give feedback on how to adapt it so that it mirrors the actual work.
[+] ksk|8 years ago|reply
These posts always read to me as being a bit presumptuous. The fact that you're looking for a job at that particular company means they're doing _something_ right, with regards to hiring, or shipping quality products, or having a good work culture, etc, etc. If I was the interviewer, I would never hire a person with this sort of "know it all" attitude. If you think you can be a better interviewer, then you have to show you are, rather than coming up with a logical argument of "doing it this way makes sense". Basically, unless you yourself are in that position, or have intimate knowledge of it, you don't know why someone made those decisions.
[+] unoti|8 years ago|reply
In the original article the author said:

> I've never had an interview that tried to evaluate my ability to work in a team or prioritize tasks...

I found this interesting, because there are some good ways to look for this that I always ask when interviewing. I wonder if the author had been checked on these things and just didn’t notice.

One thing I always do in interviews is after I see the approach the candidate is taking, show them another approach and see if they can understand it and run with it. Many candidates struggle with wrapping their brain around any other approach than their own. This is about collaboration and teamwork.

It’s also very common to have behavioral questions that talk through what some challenges were and how you solved them, and to ask probing questions about how the candidate dealt with differing opinions in the best way to proceed. This also is looking at teamwork skills.

Regarding prioritizing tasks, it’s routine for me to ask a candidate to break a problem down into components. Then evaluate which are the most difficult, and which they’d tackle first and why. I’m looking to see if they try to reduce risks early in their process. This is something more senior people do better.

I wonder if the author has indeed been interviewed about teamwork skills and just didn’t realize it.

[+] j_s|8 years ago|reply
I would like to see YC take over the initial chunk of the hiring process for their companies, and publically document the heck out of how to do it best.

Posts like this one demonstrate that free access to the front page of HN for job listings isn't enough:

Don't work for CrateJoy | https://news.ycombinator.com/item?id=15486301

[+] kreetx|8 years ago|reply
A side note, but the latter two answers (2. being a good fit and 3. doing a good job) sound very much the same to me.
[+] 0xffff2|8 years ago|reply
I think "fit" is referring to ability to get along with the rest of the team, regardless of ability to do the job. They aren't quite opposites, but they are mostly unrelated.
[+] dvt|8 years ago|reply
> I've heard "Hmm, I don't actually know what it is that we're looking for" from a few recruiters

Why do you even care what recruiters say? They're there to make as many hires as possible and get their commission. Don't get me wrong, this is actually a decent question, but you might as well ask a car salesman what he likes doing in his free time. It doesn't matter -- he's trying to sell you a car.

> How do you evaluate how well you're meeting your goal? -- The majority of the companies that I ask this to essentially answer "we don't" to this question.

I'm going to go ahead and call BS on this one. Literally every company (outside of early-stage startups) has some kind of performance review. Most of these are completely artificial and completely suck, but they're definitely there.

[+] tehwalrus|8 years ago|reply
Performance reviews and their flaws are discussed in detail later in the article. In particular, they detect false positives but not false negatives.
[+] ndh2|8 years ago|reply
The point was the performance reviews aren't used to evaluate or iterate on the hiring process.