With programmers, the single easiest way to identify good candidates (in my experience) is sheer interest in what they do / desire to learn. This is a learn everyday field and if you're interested in what you're doing, you're going to do a lot better at it. It's hard to apply yourself mentally to something that you don't have a good level of interest in. Given that it's a learn everyday field, people with that level of interest will realistically be able to learn anything they need to do solve the problem you're hiring them to solve.
The only real differentiating factor is your tolerance for ramp up time. I expect a programmer to be able to pick up a new language or database within a couple of weeks (tops) in most cases. If I'm hiring full time, that's something I'll tolerate. If I'm hiring a contractor, I'm going to be uneasy about paying high hourly rates for him to learn the job.
The single most effective way that I've found to interview for "interest" is to just get them talking about something they've done before and ask them to go deep into the details. You get everything you need from watching somebody talk, with a smile on their face, about how they solved some problem in a creative way that makes them show some pride. Doesn't really matter what the problem was, if it was a business problem, code problem or hardware problem. The important thing is the level of attention to detail in addressing it.
I've been using this technique for about 8 years now and while I don't make it the exclusive criteria for hiring, every person I've ever hired who has passed that part has ended up in my "great hire" category.
> The single most effective way that I've found to interview for "interest" is to just get them talking about something they've done before and ask them to go deep into the details. You get everything you need from watching somebody talk, with a smile on their face, about how they solved some problem in a creative way that makes them show some pride.
Sorry, but you are being scammed. You are selecting for sales skills, not technical skills. Read what you've written. You're judging someone based on how they presents themselves - not anything about technical ability.
I have interviewed many people who are great at doing this impassioned sales speech, but are terrible at actual engineering. And in fact, many good engineers are not good at selling themselves in this way - they have spent most of their lives deep in code, not working on their social skills like normal people.
Why not actually test them similar to what they're going to be doing? If it's a sales or advocate role at your tech consultancy, well then sure, your approach works.
Interest is not enough, unfortunately. There are plenty of engineers who are attracted to the challenge and excitement of building new things, but have no appreciation for The Right Way to build things.
Great engineers think beyond "how" and ask the "should" questions as well. Mediocre engineers glue things together in a haphazard way with little thought about what's the best way to write things. Caring about maintainability, comprehensibility and extensibility is partially a function of experience, but I've met plenty of experienced engineers who still write bad code and design systems poorly. It has no correlation to how much of a tinkerer and curious person they are, in my experience.
> I expect a programmer to be able to pick up a new language or database within a couple of weeks (tops) in most cases.
They may be able to hack around, write a for loop, track down a bug....but you're not going to get the same caliber of work from someone who first saw python two weeks ago compared to someone whose been using the language for 5 years on real projects.
I would like to add that interviewing requires quite different skills from engineering. A good tech interviewer has to be a good engineer, but a good engineer might not be a good interviewer. I have seen good engineers with poor ability to read people, with strong bias of "curse of knowledge". They are more likely to fall for saleperson-typed candidates.
Therefore, a hiring manager should have the ability to tell which engineers are good interviewers, and give more weights to their opinions. Pay extra attention to those interviewers who is more of a big talker and less of a listener, if you have to ask him/her to interview. If one can't be a good listener to his/her teammates, I can't imagine he/she would listen to and observe a stranger with any substantial effort. Unless he/she is a genius in reading people, his/her feedback can be mostly ignored.
People who have natural abilities chronically underestimate their effects. Some, regardless of interests, will always struggle more with certain physical or mental tasks.
Interest is important, but ignoring ability is not the right way to interview.
Most interviewers don't ask enough technical questions to have any idea what a candidate knows or doesn't know. If their one or two questions happen to be something the candidate knows well, they'll call them a genius. If they happen to not know, they'll label them an idiot.
You can learn a lot more from 20+ rapid fire questions than forcing a candidate to eek out an answer to something they're not familiar with. And once you establish the areas they're familiar with, you can ask them truly useful questions.
The key is to look for people who have strengths and not worry at all about gaps in their knowledge. Anyone who has earned genuine expertise in one area will be able to do so in other areas.
Along these lines, I start out with very broad questions. Something like, "tell me how you'd troubleshoot a web service that's suddenly not accepting connections / suddenly performing badly." Different candidates will focus on different aspects of that problem depending on their background: low-level networking, cloud environments, application-level problems, databases etc. Based on their resume, I like to see if their expertise matches up with their experience. I like to see that they don't consider ONLY things within their expertise. I like to see if they can make reasonable guesses outside of their expertise without BS'ing me. I like to see if they have a good approach to exploring areas they're unfamiliar with and general problem-solving. I like to see if they recognize how valuable it is to have diagnostic / monitoring / change control tools in place and to have done pro-active testing. This can be tough to do with coding problems, but I still try give problems that should be familiar to experienced low-level C developers as well as high-level Python web devs and tailor my expectations to their experience. I'm more concerned with how well they've learned based on what they've done than if they've already learned what I'd like them to do.
I have been to lots of interviews, on both sides of the table. I find most interviewers unprepared to evaluate the person for the role, and instead exercise their own biases, stroke their egos, etc. It's largely a voodoo practice that we'll look back and laugh at as a civilization at some point..
I wonder how many employ Kahneman's recommendation based on his book, "Thinking, Fast and Slow":
> Suppose that you need to hire a sales representative for your firm. If you are serious about hiring the best possible person for the job, this is what you should do. First, select a few traits that are prerequisites for success in this position (technical proficiency, engaging personality, reliability, and so on. Don't overdo it — six dimensions is a good number. The traits you choose should be as independent as possible from each other, and you should feel that you can assess them reliably by asking a few factual questions. Next, make a list of those questions for each trait and think about how you will score it, say on a 1-5 scale. You should have an idea of what you will call "very weak" or "very strong."
> These preparations should take you half an hour or so, a small investment that can make a significant difference in the quality of the people you hire. To avoid halo effects, you must collect the information on one trait at a time, scoring each before you move on to the next one. Do not skip around. To evaluate each candidate add up the six scores ... Firmly resolve that you will hire the candidate whose final score is the highest, even if there is another one whom you like better — try to resist your wish to invent broken legs to change the ranking. A vast amount of research offers a promise: you are much more likely to find the best candidate if you use this procedure than if you do what people normally do in such situations, which is to go into the interview unprepared and to make choices by an overall intuitive judgment such as "I looked into his eyes and liked what I saw."
Same here, the problem is that it takes a lot of time and effort to set up and run a meaningful standardized hiring process (and is very hard to outsource) and also people don't even consider planning for it, thinking they can just ask some questions in a couple interviews.
For more junior candidates I'd go the "take home test" route, for senior candidates I don't see any other solution than sitting down and design the process properly.
Take home tests FTW. The thinking goes as follows:
1. If the candidate can't be bothered to complete a 2-4 hour (depending on claimed seniority) code test in the language of their choice, we can't be bothered to talk to them.
2. If the candidate does reasonably well by completing the code test somewhat on time (with a fat margin allowed for them, well, having a life) and within parameters of the task, they're invited for a mostly non-technical onsite meet-and-greet.
3. During the meet-and-greet we make sure that the candidate isn't an axe murderer, is able to hold a quasi-technical conversation, and that both sides aren't immediately scared of each other.
4. The meet-and-greet can also include some low-key architecture discussion. Any nerds worth their salt will be able to conduct this line of questioning without making it obvious that an interview is taking place. Hopefully this isn't a critical step, as a good take-home code test will require the candidate to spend a little time designing or architecting their solution.
After the above has taken place, it should be pretty clear whether the candidate in question is a fit or not. Note that this process is by design missing the useless traditional CS questioning component, contrived problem solving exercises, or a whiteboard code beatdown.
>1. If the candidate can't be bothered to complete a 2-4 hour (depending on claimed seniority) code test in the language of their choice, we can't be bothered to talk to them.
A good reason not to work at your company. Why would I want to invest 4(!) unpaid hours into something where I am not even considered seriously yet?
I recently had a coding challenge, which was not only vague, but also took up two hours of my time. The end result was... nothing. Not even a "thank you, but we chose someone else." Since every then, I chose to not bother with long coding challenges anymore.
Startup A asked me to do a take-home programming challenge to prove my skills, as well as a general algorithm test to give them ideas how to run their core architecture. This took hours, I felt I was being asked to do their job for free (on the second test), and the results were pathetic.
Startup B gave me an assignment to add a feature to their API and have my work merged with their active code base. I was compensated $300 for my time. The feature worked, although I don't know if they still use my code from 3 years ago.
In the end, I worked for neither. But if I were forced to choose between them, I know my answer in a heartbeat.
> 1. If the candidate can't be bothered to complete a 2-4 hour (depending on claimed seniority) code test in the language of their choice, we can't be bothered to talk to them.
The approach you describe strikes me as shortsighted as well as to no small degree selfish. I don't do take-home tests, as a rule, for one overriding reason: my time is too valuable, and I have so much less of it to expend in discretionary fashion than any potential client or employer, to spend it on a one-way interaction such as a coding test. I have a Github profile with more than enough stuff on it to demonstrate proficiency in some areas and mastery in others, and I learn nothing about your company by completing your test that will help me better understand if I want to work for you. If I really want the gig, maybe then I'll do a little additional work to demonstrate, in good faith and after you have likewise established that you are acting in good faith and aren't just cattle-calling me, that yes, I'm interested. But before? No way. Your company is just not special.
Conversing with a potential client or employer, on the other hand, is a two-way street. I learn things useful to me both in terms of deciding on a gig and things that are more generally useful, like the war stories that inevitably get swapped during those conversations. It helps me decide whether your company is worth it to me and, right now and in this market (for clued-in people), that's the biggest question: not "do we want this person to work for me?", but "do I want to work for this company?". Not operating with that in mind seems like a very, very poor idea to me.
I honestly feel like take home tests are "busy work" and just as volatile as any other sort of technical interview for showing actual skills versus a real world environment.
2-4 hours is a lot of time to ask of somebody, regardless of their circumstances. Would you pay a contractor's salary rate for that work? Are these take home tests results worth $100/hour? $40? Every take home test I've encountered I've wished someone would pay me for that time, based on the opportunity costs of what I could have been doing, but maybe I'm cynical and pessimistic.
I was bummed out recently when I did a take home test for a local start-up, which asked me to build a full web app. I spent 4-6 hours on it making sure it was perfect - just to turn it and and get told I did a great job but they decided they didn't really want to hire any non-senior engineers and where just testing the waters.
This is happening a lot in Seattle recently - the job market is here is terrible for anyone except senior devs (which it is great for!).
I have been interviewing quite a bit lately, and I have found that my favorite interviews are take home tests that are representative of the kind of work I would be doing at said company. I am more comfortable with that style and I perform better.
More than that though: it gives me an idea about what it will be like to work at the company. If my only experience is studying algorithms and coding on a white board, I have a hard time knowing whether or not I will enjoy working at this company for years of my life.
A promise that you can work on "interesting problems" is hardly enough. I'm curious why more companies don't take the opportunity to sell themselves.
Anyhow, I've hung a whiteboard in my room and try to practice for interviews as much as possible. It's incredibly draining and produces a feeling completely opposite to how much fun I have building things. This feels like getting started on the wrong foot.
Despite what others are saying here I think take-home tests are a great idea. What is a terrible idea however is requiring that a candidate complete your take-home test, that you wrote and will administer. It is simply not reasonable to expect a candidate to complete such an exam for every company he applies to, oftentimes before he even talks to a human being and usually before he has a face-to-face or a phone interview with an engineer. There simply aren't enough hours in the day, even if you're unemployed.
And there aren't, to my knowledge, any really good and widely-used sites that a prospective job-seeker could use to establish his skills, which any random company likely will look at and accept as canon. And so we're all in a bit of a bind.
I assume the take home test is after you have talked to them? I wouldn't spend multiple hours unless it was already clear there was some desire to move further
These can be great. Provided that they're not used as a first-pass filter and provided they give some insight into the work the person will actually be doing. Because otherwise you're going to lose a lot of talented people who can't be bothered, especially in this market.
I do agree that "homework" type tests are the way to go.
The only caveat is - someone needs to be REALLY interested in working for you to do this as a first step.
A 15 - 20 min conversation with a hiring manager or an HR person if you have HR :) to cover things like the specifics of the position, candidate's 5 min summary of their career, approximate compensation, etc as a first step. Then you decide whether you like what you've heard and the only thing standing between the applicant and the offer is the completed assignment.
4 hours is a lot of time out of someone's day, and a near-dealbreaker for anyone gainfully employed and having some semblance of a life. Do you not realize this?
Meanwhile, it really doesn't take long to find out if someone is "worth talking to." Ask them to provide a code sample, and talk to them about for 10 minutes or so. Or if you must use a coding exercise, really, just keep it short and sweet. A good coder can show you a lot about their sensibilities in their solution to even just a slightly non-standard sorting problem.
Technical interviews are a form of hazing. Engineers often suffer from imposter syndrome, especially during an interview. Those who have already been hazed and accepted to the club will turn around and put potential candidates through the same humiliating process. And what's worse is that demonstrating you have superior capabilities in one area or another can be seen as a threat to the interviewer and they may give you a thumbs down based purely on their own insecurities. What ends up happening is that, just like in college fraternities, is that everyone ends up being similar, both culturally and in terms of abilities.
If I had it my way I would do away with the interview process altogether and do something more akin to an internship. Potential employees could start their engagement with a company by working (for pay, mind you) on a very limited basis to solve actual problems that need solving (i.e. "write an algorithm that's 10% more efficient", "create a tooltip that's aware of the viewport in React", etc). Based on their output their engagement could be ramped up until they are brought on as a full time employee. That way it ends up being completely merit based. You can either solve these problems or you can't. And whether or not they ultimately end up becoming an employee doesn't end up mattering because both parties are compensated along the way.
This would obviously put the burden on the company to boil its problems down into smaller, isolated efforts but that's something all companies should be trying to do anyways. In the end, they just want some code written that will end up solving some problem for their customer.
Constantly confused by this, so much arguing back and forth, and yet the single easiest way to deal with this is a stripped down real world problem and then giving it to a whole bunch of different candidates. Some people adopt this some people don't some people argue that it's meaningless, and go back to the standard silly interview patterns of algorithm questions and meaningless complex fizzbuzz alternatives.
My personal conclusion is that most companies don't want this for two reasons:
1. culture fit is more important for people in a rigid hierarchical structure, partly because an out of the box thinker could be dangerous for that structure. too much questioning authority, too much pointing out flaws. It's much easier to have a good worker bee than wondering why you need 40 employees to build an automated gif platform.
2. in most companies everyone is very reluctant to make decisions. for example management struggles with clear direction because it opens them up to the question of liability. if they make a decision and it's wrong they might get fired. HR works the same way, if HR passes a resume along they want it to hit a list of keywords, so they can cover their asses if he turns out to be a bad hire.
Basically everyone is so scared to make a mistake that they make a lot more mistakes trying to avoid them.
The opening of the cracking the coding interview she talks about how they don't really care about false positives and negatives, they just want those to stay below a certain threshold. But consider the hiring scale of google compared to a small company and suddenly those things matter.
One bad hire can be toxic. And basing your hiring strategy on something a huge behemoth with infinite money does is kind of silly imho
Isn't this exactly what you'd expect? e.g. if get a random cohort of developers who all of their peers would say are amazing, if I subject them to a series of tests, they will do differently well at the different tests?
For example, so I set a test where you have to write some Java, if half the candidates haven't written any Java, they'd all surely do worse on the test than the other half?
Or is their a belief in the industry that there's some scale on which we can absolutely rank all developers - front end, back end, full stack, mobile, desktop, embedded? That sounds like a surprising belief which would require extra-ordinary evidence?
The conclusion is misleading due to 2 wrong assumptions:
1. The population is heterogeneous: interviews test different skills. All interviews don't test the same set of skills, which is mandatory to compare interview scores because scores are aggregates of these skill tests. Different job opportunities means different skills to test, so it seems reasonable to assume that people evaluation vary for different job opportunities, and thus their scores vary for different interviews.
2. The observations are not statistically independent: past interviews may influence future interviews. People may get better at passing interviews or conducting interviews over time. This would impact their score. It would be good to study the evolution of individual scores over time.
While (1) should strongly limit the conclusions of the study, the complete analysis may simply be irrelevant because of (2) if the statistical independence of observations is not demonstrated. Sorry guys but this is Statistics 101 introductory course.
(1) We listened to most interviews on the platform to establish homogeneity. Interviews were across the board, language agnostic, and primarily algorithmic in nature.
(2) We actually looked into this and noticed that time didn't really affect performance. Usually, people did their interviews over a pretty short time span and then found a job. Or, people were already experienced interviewers and had kind of hit a plateau. You can see the raw data and how it oscillates wrt time in the footnotes.
Not too surprising when you consider there isn't really a standardized guideline and every interviewer asks questions of varying difficulty. Sometimes interviewers don't even ask candidates the same question and tailor them based on the candidate's resume and experiences. I've had interviews as simple as writing a function that outputs if two strings are anagrams, and other interviews that test dynamic programming knowledge and other interviews that tested my knowledge of concurrency. At the end of the day, its luck of the draw which interviewer you get and what questions he decides to ask you.
Technical interviews are silly exercises IMO. You have a multi-month ramp up period and you have all kinds of environment specific things to deal with at just about every employer. Nobody in real life gets asked to program on stage or forced to answer esoteric questions as if your in some sort of math competition.
If you don't have confidence in someone's ability based on their experience and their interview but you did like them, give them a task to accomplish offline. See if their results are anything like your results would be, and bring them in to see how they respond to feedback both negative and positive.
I've seen too many interviews that go along the lines of "How would you rate your Java on a scale of 1-5?" "5" "So how would you fix the problem if your cache hit rate on SomeObscureCommercialProduct went from 94% to 82%?" "Forget that guy. Huge ego. Doesn't know anything."
I did run into one company that had an interesting process for technical validation. They actually hire people for two weeks as contractors and have them work with the team. Then they hold a vote and decide whether to extend an offer.
1) They usually just measure the amount of effort a person has put into studying interview questions. Whether or not the ability to do this translates to being a better engineer is debatable.
2) An interviewer almost always exercise some form of personal bias, whether it be educational, personal, etc. This doesn't always show up in written feedback, but the interviewers with stronger personalities usually dominate interview debriefs, and often influence others into hire/no-hire decisions. This is especially prevalent in smaller startups where the process is more informal, things move quickly, and decisions are based more on gut feelings.
technical interviews are a joke. the majority of the time they exist so the interviewer can try to feel smart and subject the interviewee to whatever whimsical problem they found on the internet.
how often do you do group coding in a whiteboard in your actual job? at one interview I was criticized for sitting and thinking about a problem for a minute without just blindly jumping into attempting to solve it. also tons of people are great at solving toy interview problems but can't debug their way out of a paper bag.
I once was called into an emergency meeting by the CEO of the company I was working for the time. When I entered the room, all of the top brass were seated around the table, some visibly agitated. The CEO proceeded to hand me a single black whiteboard marker as he stated "the fate of the company depends on you". The problem was outlined by a fellow engineer and it was explained that I had 10 minutes to solve it, or we might risk going out of business.
With that, I took a deep breath and went to work furiously scribbling away on the board while everyone watched with anticipation. As I closed the final bracket, and stepped back to examine my work, audible gasps could be heard from around the room. The grizzled old CTO whom everyone was a little scared of broke the silence at last: "God dammit, Mark!" I caught a few nervous glances from some of the less technical folks. You could hear a pin drop when he continued:
"you just saved the god damn company!"
A slow clap started, and soon everyone joined in with a round of applause and cheering. Over the outburst of joy and adoration the CEO shouted to one of the developers who had gathered outside of the conference room, "get this code into production!" Soon the whiteboard was being whisked away by a group of smiling engineers as they navigated with great purpose through a sea of high fives and back slaps.
And then, with a twinkle in his eye, the CEO shook my hand, and pulled a crisp $100 bill out of his wallet. As he handed it to me, he said "I knew from the moment I heard about your interview that you would be a great asset to this company, Mark. Go home and relax for the rest of the day and take your wife out for dinner tonight. You've earned it."
So you see, being able to solve complicated problems on a whiteboard in a crunch time is a valuable skill.
The headline is "interview performance is kind of arbitrary," but the data solution proposed in the article is "interviewers rate interviewees in a few different dimensions," which is not any less arbitrary.
I appreciate there is an appendix addressing this issue, but it does not absolve the issues the analysis, especially since the appendix uses a "Versus Rating" to justify the statistical accuracy of the system, which is also calculated somewhat arbitrarily (since the Versus Rating is derived from the calculated interview score, wouldn't it be expected that the two have a relationship?)
The fact that the results of the non-arbitrary score are centralized around 3 out of a 4 max (instead of the midpoint of 2) implies a potential flaw or bias in the scale criteria. (The post notes that people who get a 3 typically move forward; maybe selection bias is in play since companies would not interview unskilled people in the first place)
That's not to say that the statistical techniques in the analysis themselves are unimpressive though. I particularly like the use of FontAwesome icons with Plot.ly.
Thanks, as always, for the thoughtful notes. We certainly don't mean to imply that this is gospel, and there are limitations to the work, and fwiw, we are thinking about potentially moving to a 5 star scale.
That said, if click on the link in the footnotes to see the original data, you can get an idea of what we're working with.
And lastly, the fontawesome thingy isn't plotly. It was built using http://blockbuilder.org/ by @enjalot
> the data solution proposed in the article is "interviewers rate interviewees in a few different dimensions," which is not any less arbitrary.
This article's headline reveals the spin the HN community is trying to put on it. Many people in SF like to attack the very idea of meritocracy, because if you admit that meritocracy is a good idea, you're implicitly endorsing the idea that people have different levels of merit, and thus that between-group differences in representation might be due to something other than discrimination.
I had a really bizarre interview recently where, after the initial recruiter phone screen, I was rejected based on an in-person half-hour very simplistic paired coding exercise, only met with one person, and wasn't asked about my (imo very strong) resume once. I must have said something foolish at some point, which is on me, but the point is: interviews can be hit or miss. Fortunately you only need one hit.
Fundamentals: CS basics. I don't nitpick on details. It's more around if you've heard about it or not and if you could figure out how and when to use it.
Structure: I want to see a structured approach to problem solving. Doesn't matter if your code is perfect. Doesn't matter the programming language you want to use.
Curiosity: You need to be curios about things. Asking the "why".
It may not be arbitrary, but if so it's buried deep in the data.
I've turned down candidates who had impressive technical resumes, who had worked in startups that sold, who had been hired on as consultants at various places, etc, because they were unable to solve simple algorithms in a simple manner, and their code was atrocious. Does this mean they're "bad developers"? No. If we were a consulting firm or a startup they might well be worth it; where the important thing is getting code out the door quickly, and to have something that works, even if it's not easily maintainable. But I was hiring for a position that required someone who would keep solutions simple and maintainable ('craftsmanship' rather than productivity, if you will. Note that the former does not necessarily preclude the latter, but it's the trait that was necessary, and was lacking).
Google optimizes for people with strong algorithmic knowledge. It's debatable whether they need everyone to have that, but certainly, many shops don't. Again, I've hired people with no formal CS background, because most of my job's problems don't require you to have deep algorithmic knowledge (the ones that do we can have others address, or work together on).
We know that people can fail one technical interview, while being radiant in another, and the reality is that what we're looking for, and what others are looking for, are often different. That creates a lot of variance in the data when we compare them.
I'm curious about the interviewer community. Specifically things like how are they vetted and how often they come to conduct interviews. It would be cool if there was a community of interviewers for the betterment of the process, but I could see their retention for conducting interviews to only be 1 or 2 before they drop out. I see in the appendix that there are those that do more, but no indication about what percent leave quickly.
A better drinking game might be when a candidate offers a data structure they know nothing about. Would a red-black tree work here? No.. I guess not.
I dunno - if you look at the data, there are fairly clear clusters of people who are 'probably good' and 'probably not so good'.
'Programming' is necessary but not sufficient for product engineering and that is what most of these interviews are trying to tease out. Good companies will balance out 'programming' with other rounds like 'technical design' or 'pair programming' or even non-technical rounds with business analysts or product to gauge general ability.
I have learned that an (in)ability to program "in the small" correlates very well with an (in)ability to program in the large, and now ask mostly simply questions whose answers are things like one-line Boolean predicates to test for well-defined conditions. It is paradoxically easier for an inept candidate to fake his way through an algorithm design question than it is to fake the coding of a simple test for "determine whether two closed intervals [a,b] and [x,y] overlap each other".
I've moved to doing something extremely simple: just give me pseudo-code for indexOf given a string and a character. If you can't write the 4-5 liner for linearly searching a string for a character, then there's not much point in moving further. It is shocking how many people get tripped up by it.
Exactly this. I've been arguing for years that those simple string/array/link list/tree manipulation questions are simply a microcosm of the qualities that make for good developers in general. If those trip you up then you need to consider that there are serious holes in your abilities, not that the test is bad.
[+] [-] brightball|10 years ago|reply
The only real differentiating factor is your tolerance for ramp up time. I expect a programmer to be able to pick up a new language or database within a couple of weeks (tops) in most cases. If I'm hiring full time, that's something I'll tolerate. If I'm hiring a contractor, I'm going to be uneasy about paying high hourly rates for him to learn the job.
The single most effective way that I've found to interview for "interest" is to just get them talking about something they've done before and ask them to go deep into the details. You get everything you need from watching somebody talk, with a smile on their face, about how they solved some problem in a creative way that makes them show some pride. Doesn't really matter what the problem was, if it was a business problem, code problem or hardware problem. The important thing is the level of attention to detail in addressing it.
I've been using this technique for about 8 years now and while I don't make it the exclusive criteria for hiring, every person I've ever hired who has passed that part has ended up in my "great hire" category.
[+] [-] nvarsj|10 years ago|reply
Sorry, but you are being scammed. You are selecting for sales skills, not technical skills. Read what you've written. You're judging someone based on how they presents themselves - not anything about technical ability.
I have interviewed many people who are great at doing this impassioned sales speech, but are terrible at actual engineering. And in fact, many good engineers are not good at selling themselves in this way - they have spent most of their lives deep in code, not working on their social skills like normal people.
Why not actually test them similar to what they're going to be doing? If it's a sales or advocate role at your tech consultancy, well then sure, your approach works.
[+] [-] jessedhillon|10 years ago|reply
Great engineers think beyond "how" and ask the "should" questions as well. Mediocre engineers glue things together in a haphazard way with little thought about what's the best way to write things. Caring about maintainability, comprehensibility and extensibility is partially a function of experience, but I've met plenty of experienced engineers who still write bad code and design systems poorly. It has no correlation to how much of a tinkerer and curious person they are, in my experience.
[+] [-] jkyle|10 years ago|reply
They may be able to hack around, write a for loop, track down a bug....but you're not going to get the same caliber of work from someone who first saw python two weeks ago compared to someone whose been using the language for 5 years on real projects.
[+] [-] muddyrivers|10 years ago|reply
Therefore, a hiring manager should have the ability to tell which engineers are good interviewers, and give more weights to their opinions. Pay extra attention to those interviewers who is more of a big talker and less of a listener, if you have to ask him/her to interview. If one can't be a good listener to his/her teammates, I can't imagine he/she would listen to and observe a stranger with any substantial effort. Unless he/she is a genius in reading people, his/her feedback can be mostly ignored.
[+] [-] jchendy|10 years ago|reply
[+] [-] paulddraper|10 years ago|reply
People who have natural abilities chronically underestimate their effects. Some, regardless of interests, will always struggle more with certain physical or mental tasks.
Interest is important, but ignoring ability is not the right way to interview.
[+] [-] staunch|10 years ago|reply
You can learn a lot more from 20+ rapid fire questions than forcing a candidate to eek out an answer to something they're not familiar with. And once you establish the areas they're familiar with, you can ask them truly useful questions.
The key is to look for people who have strengths and not worry at all about gaps in their knowledge. Anyone who has earned genuine expertise in one area will be able to do so in other areas.
The other big mistake most interviews make is forgetting about the "Curse of knowledge" https://en.wikipedia.org/wiki/Curse_of_knowledge
I've seen people research the answer to a question before an interview and then expect candidates to be equally informed without that advantage.
[+] [-] TallGuyShort|10 years ago|reply
[+] [-] jchendy|10 years ago|reply
[+] [-] senekerim|10 years ago|reply
[+] [-] colmvp|10 years ago|reply
> Suppose that you need to hire a sales representative for your firm. If you are serious about hiring the best possible person for the job, this is what you should do. First, select a few traits that are prerequisites for success in this position (technical proficiency, engaging personality, reliability, and so on. Don't overdo it — six dimensions is a good number. The traits you choose should be as independent as possible from each other, and you should feel that you can assess them reliably by asking a few factual questions. Next, make a list of those questions for each trait and think about how you will score it, say on a 1-5 scale. You should have an idea of what you will call "very weak" or "very strong."
> These preparations should take you half an hour or so, a small investment that can make a significant difference in the quality of the people you hire. To avoid halo effects, you must collect the information on one trait at a time, scoring each before you move on to the next one. Do not skip around. To evaluate each candidate add up the six scores ... Firmly resolve that you will hire the candidate whose final score is the highest, even if there is another one whom you like better — try to resist your wish to invent broken legs to change the ranking. A vast amount of research offers a promise: you are much more likely to find the best candidate if you use this procedure than if you do what people normally do in such situations, which is to go into the interview unprepared and to make choices by an overall intuitive judgment such as "I looked into his eyes and liked what I saw."
[+] [-] lazyant|10 years ago|reply
For more junior candidates I'd go the "take home test" route, for senior candidates I don't see any other solution than sitting down and design the process properly.
[+] [-] maxaf|10 years ago|reply
1. If the candidate can't be bothered to complete a 2-4 hour (depending on claimed seniority) code test in the language of their choice, we can't be bothered to talk to them.
2. If the candidate does reasonably well by completing the code test somewhat on time (with a fat margin allowed for them, well, having a life) and within parameters of the task, they're invited for a mostly non-technical onsite meet-and-greet.
3. During the meet-and-greet we make sure that the candidate isn't an axe murderer, is able to hold a quasi-technical conversation, and that both sides aren't immediately scared of each other.
4. The meet-and-greet can also include some low-key architecture discussion. Any nerds worth their salt will be able to conduct this line of questioning without making it obvious that an interview is taking place. Hopefully this isn't a critical step, as a good take-home code test will require the candidate to spend a little time designing or architecting their solution.
After the above has taken place, it should be pretty clear whether the candidate in question is a fit or not. Note that this process is by design missing the useless traditional CS questioning component, contrived problem solving exercises, or a whiteboard code beatdown.
[+] [-] chrisper|10 years ago|reply
A good reason not to work at your company. Why would I want to invest 4(!) unpaid hours into something where I am not even considered seriously yet?
I recently had a coding challenge, which was not only vague, but also took up two hours of my time. The end result was... nothing. Not even a "thank you, but we chose someone else." Since every then, I chose to not bother with long coding challenges anymore.
[+] [-] rm_-rf_slash|10 years ago|reply
Startup B gave me an assignment to add a feature to their API and have my work merged with their active code base. I was compensated $300 for my time. The feature worked, although I don't know if they still use my code from 3 years ago.
In the end, I worked for neither. But if I were forced to choose between them, I know my answer in a heartbeat.
Will name companies if requested.
[+] [-] eropple|10 years ago|reply
The approach you describe strikes me as shortsighted as well as to no small degree selfish. I don't do take-home tests, as a rule, for one overriding reason: my time is too valuable, and I have so much less of it to expend in discretionary fashion than any potential client or employer, to spend it on a one-way interaction such as a coding test. I have a Github profile with more than enough stuff on it to demonstrate proficiency in some areas and mastery in others, and I learn nothing about your company by completing your test that will help me better understand if I want to work for you. If I really want the gig, maybe then I'll do a little additional work to demonstrate, in good faith and after you have likewise established that you are acting in good faith and aren't just cattle-calling me, that yes, I'm interested. But before? No way. Your company is just not special.
Conversing with a potential client or employer, on the other hand, is a two-way street. I learn things useful to me both in terms of deciding on a gig and things that are more generally useful, like the war stories that inevitably get swapped during those conversations. It helps me decide whether your company is worth it to me and, right now and in this market (for clued-in people), that's the biggest question: not "do we want this person to work for me?", but "do I want to work for this company?". Not operating with that in mind seems like a very, very poor idea to me.
[+] [-] WorldMaker|10 years ago|reply
2-4 hours is a lot of time to ask of somebody, regardless of their circumstances. Would you pay a contractor's salary rate for that work? Are these take home tests results worth $100/hour? $40? Every take home test I've encountered I've wished someone would pay me for that time, based on the opportunity costs of what I could have been doing, but maybe I'm cynical and pessimistic.
[+] [-] CryoLogic|10 years ago|reply
This is happening a lot in Seattle recently - the job market is here is terrible for anyone except senior devs (which it is great for!).
[+] [-] adregan|10 years ago|reply
More than that though: it gives me an idea about what it will be like to work at the company. If my only experience is studying algorithms and coding on a white board, I have a hard time knowing whether or not I will enjoy working at this company for years of my life.
A promise that you can work on "interesting problems" is hardly enough. I'm curious why more companies don't take the opportunity to sell themselves.
Anyhow, I've hung a whiteboard in my room and try to practice for interviews as much as possible. It's incredibly draining and produces a feeling completely opposite to how much fun I have building things. This feels like getting started on the wrong foot.
[+] [-] deciplex|10 years ago|reply
And there aren't, to my knowledge, any really good and widely-used sites that a prospective job-seeker could use to establish his skills, which any random company likely will look at and accept as canon. And so we're all in a bit of a bind.
[+] [-] kanwisher|10 years ago|reply
[+] [-] leeny|10 years ago|reply
[+] [-] tiglionabbit|10 years ago|reply
[+] [-] ChemicalWarfare|10 years ago|reply
[+] [-] kafkaesq|10 years ago|reply
Meanwhile, it really doesn't take long to find out if someone is "worth talking to." Ask them to provide a code sample, and talk to them about for 10 minutes or so. Or if you must use a coding exercise, really, just keep it short and sweet. A good coder can show you a lot about their sensibilities in their solution to even just a slightly non-standard sorting problem.
[+] [-] marknutter|10 years ago|reply
If I had it my way I would do away with the interview process altogether and do something more akin to an internship. Potential employees could start their engagement with a company by working (for pay, mind you) on a very limited basis to solve actual problems that need solving (i.e. "write an algorithm that's 10% more efficient", "create a tooltip that's aware of the viewport in React", etc). Based on their output their engagement could be ramped up until they are brought on as a full time employee. That way it ends up being completely merit based. You can either solve these problems or you can't. And whether or not they ultimately end up becoming an employee doesn't end up mattering because both parties are compensated along the way.
This would obviously put the burden on the company to boil its problems down into smaller, isolated efforts but that's something all companies should be trying to do anyways. In the end, they just want some code written that will end up solving some problem for their customer.
[+] [-] rjzzleep|10 years ago|reply
tptacek summarized this in his hiring post:
http://sockpuppet.org/blog/2015/03/06/the-hiring-post/
My personal conclusion is that most companies don't want this for two reasons:
1. culture fit is more important for people in a rigid hierarchical structure, partly because an out of the box thinker could be dangerous for that structure. too much questioning authority, too much pointing out flaws. It's much easier to have a good worker bee than wondering why you need 40 employees to build an automated gif platform.
2. in most companies everyone is very reluctant to make decisions. for example management struggles with clear direction because it opens them up to the question of liability. if they make a decision and it's wrong they might get fired. HR works the same way, if HR passes a resume along they want it to hit a list of keywords, so they can cover their asses if he turns out to be a bad hire.
Basically everyone is so scared to make a mistake that they make a lot more mistakes trying to avoid them.
The opening of the cracking the coding interview she talks about how they don't really care about false positives and negatives, they just want those to stay below a certain threshold. But consider the hiring scale of google compared to a small company and suddenly those things matter.
One bad hire can be toxic. And basing your hiring strategy on something a huge behemoth with infinite money does is kind of silly imho
[+] [-] richardwhiuk|10 years ago|reply
For example, so I set a test where you have to write some Java, if half the candidates haven't written any Java, they'd all surely do worse on the test than the other half?
Or is their a belief in the industry that there's some scale on which we can absolutely rank all developers - front end, back end, full stack, mobile, desktop, embedded? That sounds like a surprising belief which would require extra-ordinary evidence?
[+] [-] dilemma|10 years ago|reply
[+] [-] mentatseb|10 years ago|reply
1. The population is heterogeneous: interviews test different skills. All interviews don't test the same set of skills, which is mandatory to compare interview scores because scores are aggregates of these skill tests. Different job opportunities means different skills to test, so it seems reasonable to assume that people evaluation vary for different job opportunities, and thus their scores vary for different interviews.
2. The observations are not statistically independent: past interviews may influence future interviews. People may get better at passing interviews or conducting interviews over time. This would impact their score. It would be good to study the evolution of individual scores over time.
While (1) should strongly limit the conclusions of the study, the complete analysis may simply be irrelevant because of (2) if the statistical independence of observations is not demonstrated. Sorry guys but this is Statistics 101 introductory course.
[+] [-] leeny|10 years ago|reply
(2) We actually looked into this and noticed that time didn't really affect performance. Usually, people did their interviews over a pretty short time span and then found a job. Or, people were already experienced interviewers and had kind of hit a plateau. You can see the raw data and how it oscillates wrt time in the footnotes.
[+] [-] Xyik|10 years ago|reply
[+] [-] Gratsby|10 years ago|reply
If you don't have confidence in someone's ability based on their experience and their interview but you did like them, give them a task to accomplish offline. See if their results are anything like your results would be, and bring them in to see how they respond to feedback both negative and positive.
I've seen too many interviews that go along the lines of "How would you rate your Java on a scale of 1-5?" "5" "So how would you fix the problem if your cache hit rate on SomeObscureCommercialProduct went from 94% to 82%?" "Forget that guy. Huge ego. Doesn't know anything."
I did run into one company that had an interesting process for technical validation. They actually hire people for two weeks as contractors and have them work with the team. Then they hold a vote and decide whether to extend an offer.
[+] [-] dms105|10 years ago|reply
1) They usually just measure the amount of effort a person has put into studying interview questions. Whether or not the ability to do this translates to being a better engineer is debatable.
2) An interviewer almost always exercise some form of personal bias, whether it be educational, personal, etc. This doesn't always show up in written feedback, but the interviewers with stronger personalities usually dominate interview debriefs, and often influence others into hire/no-hire decisions. This is especially prevalent in smaller startups where the process is more informal, things move quickly, and decisions are based more on gut feelings.
[+] [-] grillvogel|10 years ago|reply
[+] [-] marknutter|10 years ago|reply
With that, I took a deep breath and went to work furiously scribbling away on the board while everyone watched with anticipation. As I closed the final bracket, and stepped back to examine my work, audible gasps could be heard from around the room. The grizzled old CTO whom everyone was a little scared of broke the silence at last: "God dammit, Mark!" I caught a few nervous glances from some of the less technical folks. You could hear a pin drop when he continued:
"you just saved the god damn company!"
A slow clap started, and soon everyone joined in with a round of applause and cheering. Over the outburst of joy and adoration the CEO shouted to one of the developers who had gathered outside of the conference room, "get this code into production!" Soon the whiteboard was being whisked away by a group of smiling engineers as they navigated with great purpose through a sea of high fives and back slaps.
And then, with a twinkle in his eye, the CEO shook my hand, and pulled a crisp $100 bill out of his wallet. As he handed it to me, he said "I knew from the moment I heard about your interview that you would be a great asset to this company, Mark. Go home and relax for the rest of the day and take your wife out for dinner tonight. You've earned it."
So you see, being able to solve complicated problems on a whiteboard in a crunch time is a valuable skill.
[+] [-] minimaxir|10 years ago|reply
I appreciate there is an appendix addressing this issue, but it does not absolve the issues the analysis, especially since the appendix uses a "Versus Rating" to justify the statistical accuracy of the system, which is also calculated somewhat arbitrarily (since the Versus Rating is derived from the calculated interview score, wouldn't it be expected that the two have a relationship?)
The fact that the results of the non-arbitrary score are centralized around 3 out of a 4 max (instead of the midpoint of 2) implies a potential flaw or bias in the scale criteria. (The post notes that people who get a 3 typically move forward; maybe selection bias is in play since companies would not interview unskilled people in the first place)
That's not to say that the statistical techniques in the analysis themselves are unimpressive though. I particularly like the use of FontAwesome icons with Plot.ly.
[+] [-] leeny|10 years ago|reply
That said, if click on the link in the footnotes to see the original data, you can get an idea of what we're working with.
And lastly, the fontawesome thingy isn't plotly. It was built using http://blockbuilder.org/ by @enjalot
[+] [-] rewqfdsa|10 years ago|reply
This article's headline reveals the spin the HN community is trying to put on it. Many people in SF like to attack the very idea of meritocracy, because if you admit that meritocracy is a good idea, you're implicitly endorsing the idea that people have different levels of merit, and thus that between-group differences in representation might be due to something other than discrimination.
[+] [-] kbd|10 years ago|reply
[+] [-] mirceal|10 years ago|reply
Fundamentals: CS basics. I don't nitpick on details. It's more around if you've heard about it or not and if you could figure out how and when to use it.
Structure: I want to see a structured approach to problem solving. Doesn't matter if your code is perfect. Doesn't matter the programming language you want to use.
Curiosity: You need to be curios about things. Asking the "why".
[+] [-] lostcolony|10 years ago|reply
I've turned down candidates who had impressive technical resumes, who had worked in startups that sold, who had been hired on as consultants at various places, etc, because they were unable to solve simple algorithms in a simple manner, and their code was atrocious. Does this mean they're "bad developers"? No. If we were a consulting firm or a startup they might well be worth it; where the important thing is getting code out the door quickly, and to have something that works, even if it's not easily maintainable. But I was hiring for a position that required someone who would keep solutions simple and maintainable ('craftsmanship' rather than productivity, if you will. Note that the former does not necessarily preclude the latter, but it's the trait that was necessary, and was lacking).
Google optimizes for people with strong algorithmic knowledge. It's debatable whether they need everyone to have that, but certainly, many shops don't. Again, I've hired people with no formal CS background, because most of my job's problems don't require you to have deep algorithmic knowledge (the ones that do we can have others address, or work together on).
We know that people can fail one technical interview, while being radiant in another, and the reality is that what we're looking for, and what others are looking for, are often different. That creates a lot of variance in the data when we compare them.
[+] [-] kearneyandy|10 years ago|reply
I'm curious about the interviewer community. Specifically things like how are they vetted and how often they come to conduct interviews. It would be cool if there was a community of interviewers for the betterment of the process, but I could see their retention for conducting interviews to only be 1 or 2 before they drop out. I see in the appendix that there are those that do more, but no indication about what percent leave quickly.
A better drinking game might be when a candidate offers a data structure they know nothing about. Would a red-black tree work here? No.. I guess not.
[+] [-] siliconc0w|10 years ago|reply
'Programming' is necessary but not sufficient for product engineering and that is what most of these interviews are trying to tease out. Good companies will balance out 'programming' with other rounds like 'technical design' or 'pair programming' or even non-technical rounds with business analysts or product to gauge general ability.
[+] [-] pklausler|10 years ago|reply
[+] [-] Ocerge|10 years ago|reply
[+] [-] hackinthebochs|10 years ago|reply
[+] [-] unknown|10 years ago|reply
[deleted]
[+] [-] unknown|10 years ago|reply
[deleted]