"Can coding interviews work in an ideal world" which is what the article seems to discuss: sure, probably, this seems like a reasonable approach to what that ideal world might look like: a really carefully chosen question and attempting to understand the process the candidate is using to comprehend their skill.
"Do coding interviews work in the actual world in which we live in?" No, fundamentally not. Almost nobody is sufficiently capable to actually analyze code, and writing code with an audience is absolutely a stupid anxiety inducing high stress situation that makes candidates into babbling children who can't define an array - EVEN IF THE CANDIDATE CAN IN FACT PROGRAM JUST FINE. Add in one idiot on the interview panel who can't stop "tsk"-ing or asking obnoxious questions ("Why did you use a foreach and not a for loop?") and you have an hour long anxiety sandwich where the only thing you are getting from the candidate is whether or not they can dance in front of a sufficiently hostile crowd on demand.
I've just interviewed for Google and Meta. In total I went through 8 individual coding interviews.
With an exception of one, which was a bit too much like a puzzle, I think the assignments were completely fair examples of what I might experience on daily basis. Not at all tailored to people doing competition style programming, which was what I expected after being exposed to HN for years.
I have to say I enjoyed both preparing for the interview as well as the interviews themselves. And I believe that overall the interviewers were able to grasp how I would code.
One thing to realize is that interviews are a two-way street. Do you think you want to work with the interviewers? Do you like the interview process? If the interview process is poor, what other kind of hiring decisions is the company making? Are you going to want to work with the people they end up hiring?
You learn a bit about the company during the interview, and if that bit is bad, then move on. If they ask shitty leetcode questions and the interviewers are hostile or hung up on unimportant details - move on to another company.
The false negative is the obvious downside of the coding interview, but I wouldn't say that it means there's no merit to the excercise.
Take for example the "can't define an array" crowd. Are there people virtually sitting in front of me who just forget the basic syntax? Absolutly, but given we a) tell them that correct syntax isn't dreafully important, and b) let them pick the language, a [] would probably suffice, I don't think it's a completely unreasonble ask.
Moreso, when we added a prestep of basic questions that one can google, the "can't define an array" crowd mostly disappeared. Sure, there may be some other causation there, but it's hard to hire a coder that you know, can't demonstrate coding.
Finally, anecdata, since we started coding interviews, the quality of developer is simply better. We have had internal debates about this, but the truth is, everyone is overall happier with what we're doing, and I'm a lot happier that false positives are way down.
> Do coding interviews work in the actual world in which we live in?" No, fundamentally not.
I look forward to seeing you disrupt every major tech company in the world via your superior hiring strategy.
> EVEN IF THE CANDIDATE CAN IN FACT PROGRAM JUST FINE.
You don’t seem to have thought very hard about what employers are optimising for. Everyone realises that programming interviews produce lots of false negatives. Even the strongest programmers have bad days. From the employer’s point of view, that’s okay - minimizing false negatives is not the goal. The goal is minimizing false positives. Failing to hire a strong candidate carries a much lower cost than hiring someone who can’t do the job.
> "Do coding interviews work in the actual world in which we live in?" No, fundamentally not.
See, that's just not right. I mean, we all agree there's a hard problem to solve here, right? The world is populated by engineers of widely varying skill levels. It just is.
So how do you detect them? Well, the ability to bang out a correct solution on a whiteboard is a verifiably correct way to do that. It may not be optimal (passing candidates with other issues that make them bad fits), and it may have undesired failure modes (rejecting people who have trouble performing in the high pressure environment). And those are problems worth discussing.
But... it works. It clearly works. We've all seen it work. Companies that use these techniques aggressively tend strongly to be organizations with a long track record of producing working software. The FAANG employers are desirable because FAANGs ship code and make money!
So I just don't get the absolutism here. If there's a clearly better way that produces a more effective workforce, where's the darwinian example? Who's winning by hiring better people than Apple or Meta? I just don't see it.
My favourite is "I'll be your google, I'll be your Stack Overflow, when you would usually search something, ask me instead". So you eventually ask them, and they give you a cryptic answer back.
I'm sure Google wouldnt be as successful as they are if their answer to "how do you define an array" was "well think about what you're trying to do here"
This! Thank you for writing it. And let’s not forget about the wasted nights ruminating about the poor performance and putting up with people who get their dopamine when they are given authority over a peer. Most developers don’t study psychology and don’t have the ability or interest of creating a safe space for conducting the kind of interviews that their employer demands.
The dirty secret is that it’s never about the coding skills.
It’s about the person’s behavior, the way he/she interacts with you. It’s about the person’s culture, tabs vs space, and that kind of things. And it’s mostly about whether you like the person or not.
And I think it’s fine. You mainly need to be sure that the person knows what a for loop is, other than that, most people are ok at programming. What really matters is that you can work with that person.
Unfortunately interviewers who don’t realize this will unconsciously tweak the interview to make it harder for people they don’t like and easier for people they do like. And then they have an “objective” reason to not hire the person: “oh my god, he didn’t know merge sort”.
I give all kinds of random questions. And many times I recommend hiring people who bombed it, and many times I recommended bailing on ones who nailed it (overconfidence, poor communication, etc).
> It’s about the person’s behavior, the way he/she interacts with you. It’s about the person’s culture, tabs vs space, and that kind of things. And it’s mostly about whether you like the person or not
Not in FAANG or famous startups it's not. You're either gonna solve 3-6 medium hard Leetcode questions perfectly or you're out. Sure, liking you will help you better not come off as a douche - but you're not gonna get an offer without being able to solve (unless it's some kind of a diversity hire).
It is definitely about the coding skills. It is too easy to bullshit and there is a financial incentive to do so. Also many people with genunine 10 years of experience are not too good at coding.
i do sort of pair programming with candidates specifically to evaluate the interaction. i let them drive and navigate, acting more like a passenger with the occasional input. that way i can see coding skills and ability to communicate.
of course i interview for my specific situation. the candidate will be working with me directly, so they should at least be able to interact with me. if i don't like them then that's almost a showstopper, but if the candidate otherwise qualifies i should try to figure out what my problem is, and see if i can get over it.
The analogs of the traditional coding interview in the law school world are the LSAT and classroom timed memory tests. Both have absolutely nothing whatsoever to do with being a lawyer or the kinds of activities that lawyers actually do.
What they are are intelligence tests. They are easy ways for judges and top tier law firms to tell if you are "one of them." You can get your foot in the door if you can perform, at a minimum, at a certain level. There are plenty of things you can do without 97+ percentile LSAT and A's in law school, but you are screened from the top echelon of work.
People object that this isn't fair and screens good people out -- false negatives. But experience has shown that it is effective and easy. Perhaps there is a better way, but this way seems to work well enough. There is little incentive, beyond some people complaining about it, to change a system that is working well.
I see the traditional leetcode interview in the same light. The point is to see if you can perform at a certain minimal level. Yes, it's arbitrary, and often unconnected to real life. But it is an effective way to gauge a minimum level of intelligence and competence. Not perfect, but easy and effective.
Finally, there is ego element to this. People who can perform at this level see these skills as a minimum qualification. They are uncomfortable seeing themselves in relation to others who cannot perform at this level. Knowing that their peers have similar minimum qualifications makes them feel safe and secure about their status. This is not a criticism, just an observation about human nature. People in a club need a shared identity.
> You mainly need to be sure that the person knows what a for loop is, other than that, most people are ok at programming.
When I'm interviewing a senior developer that will guide other juniors, they will need to prove they know more than a for loop. They will need to show they understand the underlying concepts and technologies. Else it's a no hire, tough luck.
We also fired plenty of people that weren't up to the quality we expect.
Statements like this make me distinctly angry because I do everything I can to remove myself from the interview, of sorts.
If they come up with, or start walking with an interesting solution, I listen to them.
Tabs and spaces don't matter.
I actively work against the biases of "whether I like the person or not" because I'm aware of that as a factor.
We can absolutely minimize all of those peripherals when we explicitly train against it and a good programming interview doesn't have to be about that.
Yup this is true for the most part, although someone bad will be harder to justify.
It’s hilarious listening to people review candidates on third coding interviews. Usually it’s “this person didn’t know this one thing I think makes me really smart”.
This is why I lean so heavy in OSS to review candidates. I can see how they interact with people, and whether they are actually able to build software that _people want_
I find with this approach it’s a lot harder to justify nonsense.
I hear people say maybe they don’t have time for OSS and then hand them a Leetcode interview, which as we all know is just about memorizing problems. OSS is such a better metric and has the added benefit of improving the world.
The best coding interviews for 90% of tech jobs are ones that are heavy on coding and light on theory. Very few jobs are particularly well served by somebody with strong theoretical foundations, while most are well served by somebody who can pump out high quality code quickly.
Yet most companies interview as if they are inventing novel storage/processing mechanisms. Theory is important to understand which tools to leverage when, of course, but not to the extent that it's typically prioritized in the typical interview.
I've had plenty of people crush the theory portion of the interview and be poor performers on the job (slow coding, low volume output), but I've never had somebody crush the coding portion (implement very fast and proficiently), and perform poorly on the job
Not for nothing but we have plenty of slow coders that are “low volume” but some of the most diligent and edge case counting engineer’s I’ve ever met. Even then, if the business doesn’t depend on everyone working like a banshee, my team may as well take their time.
Even as their manager it’s not my job to lead a death march.
> I've had plenty of people crush the theory portion of the interview and be poor performers on the job (slow coding, low volume output)
I think the reason for that observation relates to why these interview tests are generally a waste of time.
Based on the tests that I've seen, they are 'recall lots of useless information' tests. That means it's actually possible to study for these interview tests (if you have the desire) and if you can ace the test by just spending time memorizing answers.
However, 'slow coding' has very little to do with coding and everything to do with problem solving. You only learn how to problem solve through lived experience and many people never learn that skill.
These interview tests generally fail only because they're not actually testing for the skills needed to do the job.
Before we criticize the current interview format and propose alternatives, we need to understand how we got here first. This is my understanding of what happened (I wasn't there for most of this!).
Leetcode-style interviews became popular in the mid 00s, primarily because they were used by hot tech companies of the time. The thing to understand is that at that time, the idea of asking people to write code during an interview what sort of revolutionary. Prior to that interviews had little structure. It wasn't unusual for the hiring manager to make the decision, some times based on credentials, recommendations, or trivial-like questions.
This type of interview became wildly popular because it allowed budding unicorns to hire programmers of high quality at scale. The process was less biased than the alternative, reproducible and scalable. Here you have two blog posts [1][2] that show the line of thought at the time.
The reality is that big tech has elevated leetcode type interview to an art. They have reached a near local optimal through years of experiments and refinements. It is working well for them so they don't have the need to take big risks such as completely revamping their hiring process.
I love the topic of hiring and interviewing and I'd love to truly get at the bottom of which method works best. I like this article because it explicitly calls out shortcomings with typical alternatives that are not usually mentioned. I hope in the future a new crop of unicorns can take these practices to the next level and do a fair comparison.
> the idea of asking people to write code during an interview what sort of revolutionary
And I think it's a great idea, personally I would never consider working anywhere that hired devs without seeing them write some code. But my problem is with the types of questions asked at many companies, specifically the types that require weeks (or months!) of prepping.
During my last job search one interview that stood out (positively) was a problem around parsing some HTTP headers (it started simple and then had layers of complexity added as I solved each one). It was honestly one of the best questions I've seen in an interview as it requires the candidate to be able to write code and solve a problem but without requiring/expecting the candidate to have prepped beforehand to learn (or brush up on) theoretical stuff far removed from the types of problems we actually solve on a daily bases (evidence of this disconnect is demonstrated by the fact people need to prep).
I had two jobs at the turn of the decade and they were conversations about topics such as “where would I go to find authoritative information on x” (google is not the answer) and approaches to troubleshooting.
Granted, I’m a script monkey (sysadmin) but some level of automation has always been part of the job.
I’ve had precisely 1 leet-code style interview (2hours for 16 problems) and I couldn’t tolerate the pressure.
The interview I had at google was much more humane than what people are attempting to cargo cult.
This is probably the best take in the entire comment section.
I see whole bunch of people complaining about Leetcode (LC) style coding questions, but I don't really see any alternative that is as good as LC. LC interviews solve all the following criteria better than other interview formats in aggregate.
- Objectivity: can you evaluate candidates as objectively as possible as opposed to subjectivity?
- Scalability: can you evaluate many candidates fast with high throughput?
- Accuracy: can you filter out bad candidates? Yes, the filter is often restrictive to the point where good candidates also get filtered out unfortunately.
- Meritocratic: can you make sure the process is assessed based on the merit of solving LC questions well as opposed to having connection to someone you know at the company or having gone to some prestigious school or other companies?
Leetcode format does good in all criteria above, or at least better than other formats below in aggregate.
1. Take home assignments.
- Objectivity: bad
- Scalability: bad. Once the question leaks, it's harder to come up with another question easily. Also doesn't scale for candidate side either. As a rule of thumb, I always decline interviews that require take home because it's useless to try hard as this is not applicable to interview at other companies. ROI is extremely low for candidates.
- Accuracy: good.
- Meritocratic: good
2. Coding questions with heavy emphasis on practicality (Stripe & Square)
I was pretty impressed with questions from Stripe and Square. They created problems that are heavily related to string, array, hashmap manipulation, searching, replacing etc using regex and such. If you are not into LC questions, you might prefer this, but I actually found some of these questions harder than dynamic programming questions I got from Google.
- Objectivity: good
- Scalability: bad. Once the question leaks, it's harder to come up with another question easily.
- Accuracy: good.
- Meritocratic: good
3. Knowledge / Trivia questions about programming language or framework
This might make sense a bit if you are specifically looking for iOS, Android or web developer but it still will need to be paired with LC questions.
- Objectivity: good
- Scalability: good
- Accuracy: bad. Candidates can memorize and recite these answers to the questions. Also too many good candidates actually do not memorize the details about programming language or framework. Also it can easily obtained from just Googling so people can cheat easily.
- Meritocratic: good
4. Pure behavioral
- Objectivity: bad
- Scalability: good
- Accuracy: bad
- Meritocratic: bad
5. Ex-coworker
- Objectivity: bad
- Scalability: bad
- Accuracy: good. If you've worked with the candidate before, you'll have a pretty good grasp of whether or not he/she can do the job well.
If I have my Jr. Dev a hard problem to work on and they said "breathe over my shoulder for 30 minutes and I'll have the answer for you" I would be having a serious corrective chat with them at our next meeting. No one works that way. Coding interviews are 10 percent about coding and 90 anxiety management... Which to be clear is important, but let's be honest about what we're filtering for in these things.
I don't even think it's as clever as testing for anxiety. I think they have to be there watching your back to make sure you're not cheating - the problems they give are easily searchable online.
As a by product they will also test for candidates who are super good under stress (or drilled Leetcode so hard there's no reason to be in stress).
I used to defend coding interviews, until I had an interview about a month ago, and the leetcode questions came off as kind of insulting.
I have a decade of experience, working as a senior and staff engineer at megacorporations, have experience as a research scientist, am in a PhD program for computer science research, but lets just double check that I know how to use a hash table.
I can tell you right now I have seen 3-4 principal “architects” who don’t know how to write simple tests and all of them were external hires. I absolutely respect any company that requires experienced hires to at a minimum understand how a hash table works. It’s embarrassing when a principal engineer can’t evaluate a simple design because they’ve been “leading” for the past few years. Finally I know at least one person who gets hired on at a principal level, immediately seeks out forward looking projects with no defined success criteria and leaves a year or two in, rinse and repeat at various companies. It’s a waste of a money (400k-500k/year) and a waste of everyone’s time who has to onboard him.
Not that this would apply in your case (I don't know you), but in general there are plenty of well-credentialed people that are poor engineers (imo), or who are a good engineer but a poor fit for that company/team/position. And there are people that simply lie on their resume.
If it is standard at the company to give all candidates that same question, this is actually a good way to reduce bias in hiring decisions. Otherwise, you might bias toward hiring, say, people with college degrees or people who worked at large corporations, for example. And this might unintentionally bias against certain demographics that obtain college degrees at lower rates (for example).
Yeah I can see why it seems insulting but have you ever interviewed a lot of people.
When I started interviewing I started off by jumping into what I considered to be not insulting questions, but I very quickly learnt how awkward that can be if the candidate just had no clue.
Now I start with for loops and preface it with "apologies if this seems too simple" or something like that.
You would be surprised how many people can't do a simple for loop. I mean really simple.
Instead of endlessly rehashing the controversy of coding interviews, has anyone considered just how engineers are expected to be able to understand the material on the interviews?
Should it be from university CS programs? Which are supposed not to be vocational? And would discriminate against those who are not degree holders?
Should it be from internships, which are nowadays subject to the same Leetcode examinations as actual work?
Should it be from junior developer roles, which are an endangered species, as every business in desperate need for seniors, and lack the patience to train?
Maybe open source can be a sort of free apprenticeship to teach developers these good practices?
Or maybe grinding away at Leetcode, Project Euler, and arbitrary coding puzzles is really the only pedagogical solution.
We have no issue whatsoever hiring juniors (we hire about as many as mid-levels), but we don't have to settle for someone who doesn't have internships or projects and the openings can be closed in 2 weeks so it seems like nobody is hiring, but actually the bar has risen (and there are plenty of candidates who clear the bar).
> has anyone considered just how engineers are expected to be able to understand the material on the interviews?
I am confused - how is this question materially different from asking how engineers are expected to know what they will need in order to properly do their jobs? Just how are engineers expected to know programming languages, or domain-specific languages, or standard libraries, or platform apis? Is this what you are asking?
I used to hate leetcode-style coding interviews, but now I don't. The reason for this is a comment from a Google engineer. That comment changed my mind. In that comment they explained the motivation behind asking these type of questions and it all finally clicked for me.
Corporations are not testing your knowledge of algorithms per se. While that knowledge is obviously important, what's more important is how dedicated you are when it comes to reaching a goal. Essentially, they are testing your will power. You know the rules. Can you achieve success and not give up in the middle of the road by skipping, say, Dynamic Programming? They could've tested your level of dedication by asking to bake some fancy cake, but throwing CS-related questions just makes much more sense.
That reasoning completely changed my perspective about algo-heavy coding interviews.
> what's more important is how dedicated you are when it comes to reaching a goal. Essentially, they are testing your will power. You know the rules. Can you achieve success and not give up in the middle of the road by skipping, say, Dynamic Programming?
That's an interesting theory. And they pick the goal that happens to be best suited to fresh/current undergrads from Stanford who weren't working a draining job to put themselves through school, so can afford the additional person-months to achieve the gatekeeping that was set up by people like them, for people like them.
I and many other experienced software engineers have demonstrated perseverance and rising to the occasion of extremely challenging goals. But that theory suggests they're not interested in that. Maybe they're interested in the class shibboleth that was established when computers became less of a meritocracy for people with a passion for it, and more of a go-to status job for the upper middle class?
How dedicated you are to do a task with no intrinsic value besides getting hired at companies with leetcode-style interviews. I don't think google is better off when everyone they hire has wasted hundreds of hours on a fundamentally useless skill rather than having used that time to learn different skills or just enjoy time with family and minimize the chance they burn out on the job at a later date.
There's other ways to tease out if someone is "dedicated to reaching a goal" like actually asking them questions about their life, daily routine, accomplishments etc. I think these are much better signals than "this person can implement many sorting algorithms and solve towers of hanoi or the egg drop puzzle without a google search to refresh their memory" with an N=1 sample size used to gauge how reliably they can do that. How many people if asked to do this on a job rather than an interview a year later would then go and implement it from memory without double checking they didn't forget some edge case on stackoverflow?
I know grinding leetcode is fundamentally useless because you will immediately start losing your ability to interview once you get hired. If you don't change jobs within a year you'll need to start studying all that crap again for the next interview.
Sure, but just make sure you are Google before you try this.
If you are a no-name company paying whatever your HR department defines as "market rate" and you don't get hundreds of very qualified candidates for every role then interviewing like Google might not be a great idea.
They are testing also how docile you are. If you are ready to do useless tasks and go through hundreds of pages of the X-corp way of life then you have a chance to stay long and prosper.
> what's more important is how dedicated you are when it comes to reaching a goal.
That may sound good in theory, but in practice I've been rejected quite a few times for not delivering the correct or most optimal solution despite trying my best.
One reason I'm self-employed now is to avoid this whole flaming hoop audition game. Thankfully my customers don't want to test me. At this point I'm tired of arguing against coding interviews, because the same arguments have already been repeated ad nauseam, and the linked article doesn't seem to me to add anything new. I just don't want to hear about any "talent shortage", because too many programmers are being systematically excluded out of fear of the dreaded "false positive".
I've been asked to interview. I'm not a great interviewer, but maybe OK. I try to figure out if they can work the tools. I'm pretty lax - interviews are stressful and missing a comma can take a minute, I don't like people twisting in the wind over a syntax error and will point it out if I'm pretty sure they'd get there. I try to decide if I can work with this person. I try to figure out if they'd be ok with how things work at my current employer.
I have two very memorable no's. They were for pretty senior/advanced IC roles. Juniors would get far more slack. I usually ask an expression evaluator question.
One person, used eval(string). Brilliant, absolutely brilliant. full points for answering, but how do we make sure the string is safe to eval? Long frustrating discussion around regexes that that maybe a junior dev is responsible for maintaining. Man, just write the parser. I know you can do it. Give me something that's something that's not _crazy_ to put into prod. Give me something that won't be a bug tarpit. Sorry man, I loved your answer. I hated your response to how this can be operationalized. Clearly super smart. Maybe I should have given the green light, but the response to criticism was so bad. There are lots of people I disagree with, that I'm happy to work with. I think I made the right choice, but I wonder from time to time.
Another person I absolutely would have loved to work with, and would have learned a ton. Very into formal methods, or slightly relaxed versions of verifying reliability. My organization at the time was pretty fast and loose. I believed they would be miserable. In the interview I told them I'd give them the green light, but they'd have a huge uphill battle.
it's so hard to tell. People are adaptable, small mismatches can be papered over, but big mismatches - it's so tough to say. Relaxing and being vulnerable on both sides is so hard to do well. Maybe I was just a jerk. I think I made the right calls, but it haunts me.
It is nice to hear a defense of this style since there’s often so much complaining about it, and the tips to mitigate the downsides are useful.
Personally I don’t mind a whiteboard session or two during a loop but what’s wild to me is how, especially at big companies, you’re expected to do four or five of these to get an offer.
How often do these companies decide “well they understand when to use DFS and they can merge-sort a linked-list and they knew to use dynamic programming but we can’t hire them because they couldn’t remember how to implement a heap”?
The problem is that these problems are not even whiteboard problems anymore. They’re often strictly timed coding exercises on Coderpad or Karat with actual mock IDEs and REPLs expecting running code under strict expectations.
The more casual, flexible, subjective coding interview described by the OP is not the prevalent form of whiteboarding interviews, which has been superseded by the weeder version.
The idea of observing someone writing code is a freaking joke. It puts an enormous amount of pressure on the candidate, and it forces the candidate to think and talk deeply at the same time, which in and of itself is a skill that is not never for the job.
Coding interviews are a joke and the defense of them is even worse.
Edit: on further thought. I think I’d be ok with coding interviews if the interviewee got to bring in their own coding question and the interviewer had to do it as well. That way the they also knows the people they are working with can live up to their ideal of a good coder.
I upvoted this because it's well written, even though I disagree with the conclusions. I'm not sure the suggested alternative coding challenges are any better. The text preview one starts ok, but introducing HTML raises the complexity through the roof!
I'll summarize my conclusion about live coding challenges, which is that I'm not convinced that my performance on these challenges reflected any abstract skill I have. Instead, they mostly reflected the fact that the problems I was given were either nearly identical to work I've done in the past, or were similar enough to things I'd done recently that they felt pretty easy.
I guess there's _some_ signal in that, but I don't know if it really says anything about how good I am at coding in general.
If you're augering for a $250,000 job, and the gate is a leetcode interview, just spend a couple weeks prepping for it. It'll be the biggest bang for the buck effort you'll ever make.
Here's some data models, and here's how you can reference eachother by properties. Cool. Let's do 3 levels of nested loops to extract the relationships and map them. That's like 99% of what they'll be doing.
The signals I'd look for are "do they start the entire thing by looking at structures or not", etc.
But unfortunately this breaks down when you're interviewing 10-year+ candidates. It doesn't yield anything useful. So sticking for those positions with a really really basic problem and pseudocoding it, and using the remaining time to instead have them map out solutions they did in the past seems to be useful.
Most things leet code interviews screen is "does this person know how to cram" because I tell you, everyone I know who aced leet code interviews crammed for 3 months and literally knew any one you can think of, and every variation. Juniors tend to do better in those, ironically enough.
That's an interesting observation: if you are hiring someone whose role is largely high level decision making, like a principal engineer, that skillset is orthogonal to anything you could get them to show you on a keyboard. Almost by definition: the role is simply more than what a single person can do at the keyboard in X amount of time, so testing for that would be an incomplete test.
> Most things leet code interviews screen is "does this person know how to cram" because I tell you, everyone I know who aced leet code interviews crammed for 3 months and literally knew any one you can think of, and every variation.
As I commented above, I successfully passed multiple leetcode interviews in the last four years, including FAANG. I did not spend a single minute on prep.
A) SWEs are shit behavioral interviewers in any other way. You are not about to chnage that even if you are google.
B) It is crucial that newcomers be hired by their peers and by their direct manager. This has solid research. People hate it when they get an hire and they hadn't a meaningful part in the decision... The new hire is more likely to fail.
A+B means it is better to have a terrible process lead by SWEs (e.g. coding interviews), than a good one lead by professional recruiters (which will fail, no matter their capabilities and methodology) (and yes, I do believe that without considering A, some non technical recuiters trained at workplace psychology will have a much better success rate without needing any technical interview. They will need someone technical to have a talk with the candidate, but definitely not to ask leetcode etc).
Mind sharing the solid researching on that hiring by the team comment? My company is switching to pooled hiring and it irks me, so I’d love to read up…
Software development is really weirdly unique in the way we interview candidates. In no other profession does a job candidate with work experience have to “study” or “practice” for an interview other than reading up about the company. Unless they’re interviewing for their first job, talk to candidates about their experience and dig in to what they actually accomplished.
My view is that a coding interview is not an interview of whether you can code.
It is an interview about your communication, whether the code you write can be maintained, whether you can inspire/build trust (can you explain what you're doing? does that feel reasonable to the other person?).
Where people fail is to think that it's a test of experience, or a "heads down I must make every test case pass", that the outcome matters more than the process.
Very well written article despite the fact that I completely disagree with the conclusions logic of the author. I'm on the "reading code" and having a more personal connection side of the road. Partially on the debugging aspect for some cases. Pretty much the process explained here: https://talktotheduck.dev/debugging-the-technical-interview-...
The "write code" types of interviews fit for juniors who might not know how to even write code properly. If a person doesn't know how to do that and is a junior they usually have the right energy and can put 100% into the job to adapt/learn. I hired such juniors who had no experience in the languages/platforms and were productive within a month. They had good character and discipline.
Distilling a question like that for an advanced coder is impossible. No wonder the article didn't provide any sample of a "good interview". We'd all burn him at the stake because we'd all hate it. There's no such piece of code that would provide a valuable data point about a candidate. So this is a waste of your time when dealing with a junior and with an advanced coder.
Coding interviews are difficult. You will make mistakes, this is unavoidable. I think the best mistakes to make are the ones where you hire people who are part of your team and can grow with it. Even if they aren't the best coders around, being part of the team will help them hone those skills and evolve. If I had to pick a 10/10 coder or a team player I'd pick the latter every time.
Coding questions are a waste of time. A better interview technique is asking about a problem they solved and drill down on it. This actually is better and allows for a conversation and discussion. I know you can program so don't need to trick you up on it. If you can't then you'll quickly be found out on the job.
On the last type, I had one where I had to write code for a checkout system with the developer sitting next to me. I hated the pressure and felt very stressed, halfway through the problem I hit a bug and my mind went literally blank, all the map in my mind of variables, flow etc literally went black. I remember it happening and I totally lost myself, ended up looking out the window for a while at the sun shining on a building. Somehow got it back and ended up finishing the task and I remember basically shouting I'd finished and that that was it, I wouldn't do anymore as I was so relieved it was over.
I don’t know how to feel about coding interviews. I never had a serious coding interview in my 24 year job history between 7 companies from 1996-2020. They were all your standard “dark matter” [1] development enterprise dev jobs.
Then I did a slight pivot to get into BigTech via cloud app dev consulting - “application modernization” [2] - still with no coding interview although I still do the same type of enterprise/cloud app dev I did in the latter part of my career in the “real world” - and a shit ton of yaml.
Even in 2020, if I couldn’t have gotten a job at either Azure or AWS, I probably would have done what it takes and practiced to increase my income by an easy six figures or more.
Coding interviews are a “gravity problem”. You can not like either. But they still exist.
That being said, if I were starting out today and could break the can’t get a job <-> don’t have experience cycle by “grinding leetcode” for few months and and end up in the top 10% of income earners in the US right out of college, I would have definitely done it and I encourage my younger relatives graduating in CS to do it.
However, if you’re just hiring for your standard yet another senior CRUD developer and paying as such ($90K-$160K) - don’t expect me to do the monkey dance.
I think algorithm style coding interviews are the great equalizer. Even though I at 48 probably couldn’t pass one without some serious practice.
I don't really have an opinion about whether or not programming interviews are useful. I think they probably are useful in most situations. Nevertheless, I suck at them. I don't think I've ever passed one. I'm not even a good programmer. When I ship code, I'm slow. If I get hired it's because someone has seen code I've written in the wild, and it works, and it is clean and readable and well tested -- and because the person hiring can see that I love building stuff.
I'm finishing up a personal open-source project and am looking for a new position focused on ruby or rails, so if you're hiring and are open to evaluating me based on something cool I've written, hit me up.
Other professions handle this via professional qualifications. Architects - and please do contradict me actual architects! - are not tested on their knowledge of basic architectural concepts as part of interviewing at a firm, because that's what the professional qualification is for. You have done the test already.
And here's where our industry isn't a "real" profession yet. There are plenty of qualifications one can get, but none that gates the ability to call oneself a software engineer. The result: we have to prove it in interviews.
And I don't know whether this is better or not. It is different though. The price of not having to sit through exams is to have to jump through hoops.
That system is basically "sit through one exam when you're young". This can work fine in something that's not as detailed as coding.
The thing with software is people tend to think of it as something that needs maintenance to stay fresh, perhaps a bit like piloting an airplane.
Airplanes have logs and pilots recertify on an ongoing basis, but software doesn't work like that. Someone might have stopped coding but still have a title.
I'm not saying software actually does work like that, but I'm sure it's a view some people subscribe to.
Is there anyone who has never had a bad experience with a licensed qualified professional? Bad doctors, dentists and lawyers are easy to find. I wish I had a way of gauging knowledge before letting someone drill my teeth
Programming interviews are really important but the questions should not be those about tricky data structure or based on obsecure mathematical trick. IMO the question should be presented in terms of real world concepts so that the candidate can be tested on what data structure they use to model the data related to the problem statement and the problem statement should be simple enough that you can easily come up with a very high level approach yet the implementation would require understadning concepts of conditionals/iteration/recursion. And you should also check how the candidate break the problem into smaller problems (instead of writing a single big function)
I agree with this articles ending, what you offer though is a hybrid of pair programming, what we ought to say is that effectively we need to have a conversation, trust me i wont do a coding game with you, ill just ignore you, but a conversation like a human being, that i can do and i'm more than willing, and yes ask me anything because i will ask you too, the interview is a two way process, many of us learned that the hard way, therefore we wont repeat the same mistakes.
Perhaps I’m biased as someone with a better than average sense of algorithms and discrete math, but I think such interview questions have a lot of utility, provided the job will require some analysis of algorithms, eg you’re working on the platform, or a solver/optimizer.
Obviously it’s pants on head stupid to do that if you want a web dev to make a pretty app, tie frameworks together and liaise with 3rd party integrators. However, I think many employers want the crème of such devs who are also good at algos.
> However, I think many employers want the crème of such devs who are also good at algos.
Which is not going to work for the vast majority of devs and jobs.
Most devs either don’t care or need to care about algorithms, and the few devs who do, spend most days not using them.
This would be akin to asking a civil engineer to build a wooden road bridge. Having the skills is great, but how often is the company going to need them?
I think in-person coding interviews (so both sides have skin in the game) are good but the problems should actually be applicable to the task at hand. In practice, this is rarely the case - I'm not sure if it's because there's a shortage of non-algorithm-focused questions or if because every bullshit startup wants to come across as some super advanced software R&D operation while what they ultimately need is a variant of CRUD.
In my 25 years as a developer I have been sitting on both ends of a coding interview a good number of times. One thing I find coding interviews helpful for is to weed out people that apply for a specific job description (e.g. "looking for experienced python developer", who's resume shows 7 years of recent python experience, but that during a simple coding test have problems getting the syntax right.
I wonder how well the following exam would correlate to job performance:
A completely fictitious language is developed. Simple enough that you can become fluent within an hour. From this you read a bunch of problems in said language and must solve them.
A set of assertions about the problems are written down and you must verify them. Finally a full program, created from a dialect of the given language is given and you try to understand
Please provide proof that a coding interview makes a statistically significant effect on predicting success in a position, all other things being equal.
My experience is obviously just anecdotal, but I can say that in the severely limited candidate pool I worked with in the past decade, there has been no correlation between their proficiency at code interviews and their performance at their day to day job.
I’m also not familiar with other fields, but are doctors or lawyers asked to do anything analogous to code assessments during interviews?
How could you reasonably prove this? “All other things being equal” seems harder to evaluate than a coding interview.
Drive, experience, intelligence, communication skills, teamwork, emotional intelligence, grit, curiosity, physical health, mental stamina, family/home situation, and dozens of other factors play into job performance, many of which you can’t even ask about to generate your test dataset. Many of these traits are what you’re trying to measure the effects of in the interview.
The academic qualification system is pretty broken in the computer science world. One core problem is a lot of teachers use automated tests to judge submitted code because they're either lazy or overworked, and it's very easy to game that setup.
I'd summarise the article as such. Fizz buzz like coding coding interviews are best. Looks at how the interviewee codes not just now, but in the future as we.
> Don’t ask “trick” questions, like anything where you have to use an obscure data structure or algorithm in a clever way (or dynamic programming).
I just want to put this out there to anyone who gets involved in interviews that this is a very important rule -- if your question can basically be redefined as some really clever solution you found on a problem you put > 30 minutes into, it's probably not a good question.
A lot of the seniors I work with would come up with questions like this, and I get their logic:
1. I'm a senior and this was a challenge for me
2. The logic is clear when you see it and look at it
3. I want people who can do work similar to me
4. I know how to get to the conclusion, so I can judge based on the path the candidate takes
Conclusion: It's a good question
While I understand how we can get to such a conclusion, I think that in most cases the only takeaway you can get is "can this person think like I did in this situation?", which maybe has some value for judging workplace compatibility, but I don't think it's a fair assessment of someone's technical aptitude. If you yourself required a lot of time and brainpower to sort through a tricky issue like this with the benefit of the problem scope and system needs that led to this issue, it's very unlikely you'll be able to judge any assessment
Point 4 seems good at first because we can say "the goal is to just see them think and use their experience"; there is truth to it but without the rest of the context it's not really a good assessment because ultimately, you're looking to see "did you get the same conclusion I did? Did you avoid the same pitfalls I hit?"
Instead, focus on a general scenario that tests their understanding of basic principles. How do they use the fundamental knowledge for the subject matter to work through a new problem that you yourself haven't really dealt with. It removes your bias a bit and opens you up to those "wow why didn't I think of that?" moments that you'd have working on tough projects together.
Try your best to give a good faith interpretation to any path the candidate takes and help take it to natural conclusions, and see how the candidate reacts; if you can see they've made a workable but problematic solution, nudge them towards the problems you see and see how they discuss it.
For me a candidate that can really process their own thoughts and do a good analysis of their own solution (its benefits and shortcomings) is extremely valuable; I usually lead off with "I know how I'd do it, but that doesn't mean it's the right answer, so focus on your strategy and walk me through it." If you want tot test compatibility or you feel you have a stronger solution, accept their solution (if it's valid) then go ahead and discuss your thoughts a bit and see how they respond and how they discuss it. The more you make it a technical conversation against peers and less a university exam, the more you have to see how they think and operate and the better you can understand them.
Stop defending whiteboard interviews. We know [1] that they are inherently biased against certain groups and have a very large false negative rate.
It amazes me how many people will defend the current interviewing process and then openly admit that the majority of their engineers could not pass the same process without spending dozens of hours preparing.
That study lacks data. The entire result disappears if you exclude the women. And womens results looks strange, every woman does perfectly in the private version and every woman does horribly in the observed version. It is possible the interviewer either was a hot man who distracted the women, or he was a sexist creep who distracted them in other ways, or it was just random which is possible since there were so few women.
But other than that the study didn't find any evidence that men have any issues with being observed while coding. So unless a better study comes around we can dismiss complaints about observed interviews being stressful unless it comes from a woman, it is in the study you linked after all. (I do believe there is a stress effect, but that study isn't showing it, need a bit more samples and probably several different observers to get better data)
Some comments were deferred for faster rendering.
vorpalhex|3 years ago
"Do coding interviews work in the actual world in which we live in?" No, fundamentally not. Almost nobody is sufficiently capable to actually analyze code, and writing code with an audience is absolutely a stupid anxiety inducing high stress situation that makes candidates into babbling children who can't define an array - EVEN IF THE CANDIDATE CAN IN FACT PROGRAM JUST FINE. Add in one idiot on the interview panel who can't stop "tsk"-ing or asking obnoxious questions ("Why did you use a foreach and not a for loop?") and you have an hour long anxiety sandwich where the only thing you are getting from the candidate is whether or not they can dance in front of a sufficiently hostile crowd on demand.
vletal|3 years ago
With an exception of one, which was a bit too much like a puzzle, I think the assignments were completely fair examples of what I might experience on daily basis. Not at all tailored to people doing competition style programming, which was what I expected after being exposed to HN for years.
I have to say I enjoyed both preparing for the interview as well as the interviews themselves. And I believe that overall the interviewers were able to grasp how I would code.
quacker|3 years ago
You learn a bit about the company during the interview, and if that bit is bad, then move on. If they ask shitty leetcode questions and the interviewers are hostile or hung up on unimportant details - move on to another company.
ownagefool|3 years ago
Take for example the "can't define an array" crowd. Are there people virtually sitting in front of me who just forget the basic syntax? Absolutly, but given we a) tell them that correct syntax isn't dreafully important, and b) let them pick the language, a [] would probably suffice, I don't think it's a completely unreasonble ask.
Moreso, when we added a prestep of basic questions that one can google, the "can't define an array" crowd mostly disappeared. Sure, there may be some other causation there, but it's hard to hire a coder that you know, can't demonstrate coding.
Finally, anecdata, since we started coding interviews, the quality of developer is simply better. We have had internal debates about this, but the truth is, everyone is overall happier with what we're doing, and I'm a lot happier that false positives are way down.
I'm sure there's other ways of course. :)
dlkf|3 years ago
I look forward to seeing you disrupt every major tech company in the world via your superior hiring strategy.
> EVEN IF THE CANDIDATE CAN IN FACT PROGRAM JUST FINE.
You don’t seem to have thought very hard about what employers are optimising for. Everyone realises that programming interviews produce lots of false negatives. Even the strongest programmers have bad days. From the employer’s point of view, that’s okay - minimizing false negatives is not the goal. The goal is minimizing false positives. Failing to hire a strong candidate carries a much lower cost than hiring someone who can’t do the job.
leeoniya|3 years ago
you should be able to explain why you wrote something the way you did; that's not an absurd ask.
perf characteristics of different iteration styles are often significant to the task at hand.
ajross|3 years ago
See, that's just not right. I mean, we all agree there's a hard problem to solve here, right? The world is populated by engineers of widely varying skill levels. It just is.
So how do you detect them? Well, the ability to bang out a correct solution on a whiteboard is a verifiably correct way to do that. It may not be optimal (passing candidates with other issues that make them bad fits), and it may have undesired failure modes (rejecting people who have trouble performing in the high pressure environment). And those are problems worth discussing.
But... it works. It clearly works. We've all seen it work. Companies that use these techniques aggressively tend strongly to be organizations with a long track record of producing working software. The FAANG employers are desirable because FAANGs ship code and make money!
So I just don't get the absolutism here. If there's a clearly better way that produces a more effective workforce, where's the darwinian example? Who's winning by hiring better people than Apple or Meta? I just don't see it.
ozim|3 years ago
Once you get into scale where you need to hire 20 devs a year and get 100s of CVs per week it all breaks down.
koonsolo|3 years ago
If you have a better way on how to assess a software developer, I would love to hear it.
bodge5000|3 years ago
I'm sure Google wouldnt be as successful as they are if their answer to "how do you define an array" was "well think about what you're trying to do here"
mihaitodor|3 years ago
robertlagrant|3 years ago
peterhunt|3 years ago
d--b|3 years ago
It’s about the person’s behavior, the way he/she interacts with you. It’s about the person’s culture, tabs vs space, and that kind of things. And it’s mostly about whether you like the person or not.
And I think it’s fine. You mainly need to be sure that the person knows what a for loop is, other than that, most people are ok at programming. What really matters is that you can work with that person.
Unfortunately interviewers who don’t realize this will unconsciously tweak the interview to make it harder for people they don’t like and easier for people they do like. And then they have an “objective” reason to not hire the person: “oh my god, he didn’t know merge sort”.
I give all kinds of random questions. And many times I recommend hiring people who bombed it, and many times I recommended bailing on ones who nailed it (overconfidence, poor communication, etc).
weatherlite|3 years ago
Not in FAANG or famous startups it's not. You're either gonna solve 3-6 medium hard Leetcode questions perfectly or you're out. Sure, liking you will help you better not come off as a douche - but you're not gonna get an offer without being able to solve (unless it's some kind of a diversity hire).
quickthrower2|3 years ago
em-bee|3 years ago
of course i interview for my specific situation. the candidate will be working with me directly, so they should at least be able to interact with me. if i don't like them then that's almost a showstopper, but if the candidate otherwise qualifies i should try to figure out what my problem is, and see if i can get over it.
jordanpg|3 years ago
The analogs of the traditional coding interview in the law school world are the LSAT and classroom timed memory tests. Both have absolutely nothing whatsoever to do with being a lawyer or the kinds of activities that lawyers actually do.
What they are are intelligence tests. They are easy ways for judges and top tier law firms to tell if you are "one of them." You can get your foot in the door if you can perform, at a minimum, at a certain level. There are plenty of things you can do without 97+ percentile LSAT and A's in law school, but you are screened from the top echelon of work.
People object that this isn't fair and screens good people out -- false negatives. But experience has shown that it is effective and easy. Perhaps there is a better way, but this way seems to work well enough. There is little incentive, beyond some people complaining about it, to change a system that is working well.
I see the traditional leetcode interview in the same light. The point is to see if you can perform at a certain minimal level. Yes, it's arbitrary, and often unconnected to real life. But it is an effective way to gauge a minimum level of intelligence and competence. Not perfect, but easy and effective.
Finally, there is ego element to this. People who can perform at this level see these skills as a minimum qualification. They are uncomfortable seeing themselves in relation to others who cannot perform at this level. Knowing that their peers have similar minimum qualifications makes them feel safe and secure about their status. This is not a criticism, just an observation about human nature. People in a club need a shared identity.
koonsolo|3 years ago
When I'm interviewing a senior developer that will guide other juniors, they will need to prove they know more than a for loop. They will need to show they understand the underlying concepts and technologies. Else it's a no hire, tough luck.
We also fired plenty of people that weren't up to the quality we expect.
Were are you working? At some government?
t-writescode|3 years ago
If they come up with, or start walking with an interesting solution, I listen to them.
Tabs and spaces don't matter.
I actively work against the biases of "whether I like the person or not" because I'm aware of that as a factor.
We can absolutely minimize all of those peripherals when we explicitly train against it and a good programming interview doesn't have to be about that.
mountainriver|3 years ago
It’s hilarious listening to people review candidates on third coding interviews. Usually it’s “this person didn’t know this one thing I think makes me really smart”.
This is why I lean so heavy in OSS to review candidates. I can see how they interact with people, and whether they are actually able to build software that _people want_
I find with this approach it’s a lot harder to justify nonsense.
I hear people say maybe they don’t have time for OSS and then hand them a Leetcode interview, which as we all know is just about memorizing problems. OSS is such a better metric and has the added benefit of improving the world.
1270018080|3 years ago
If you don't answer the 2 leetcode mediums in optimal complexity within 45 minutes then you don't pass. It's about coding skills.
adam_arthur|3 years ago
Yet most companies interview as if they are inventing novel storage/processing mechanisms. Theory is important to understand which tools to leverage when, of course, but not to the extent that it's typically prioritized in the typical interview.
I've had plenty of people crush the theory portion of the interview and be poor performers on the job (slow coding, low volume output), but I've never had somebody crush the coding portion (implement very fast and proficiently), and perform poorly on the job
xkqd|3 years ago
Even as their manager it’s not my job to lead a death march.
RavingGoat|3 years ago
jussij|3 years ago
I think the reason for that observation relates to why these interview tests are generally a waste of time.
Based on the tests that I've seen, they are 'recall lots of useless information' tests. That means it's actually possible to study for these interview tests (if you have the desire) and if you can ace the test by just spending time memorizing answers.
However, 'slow coding' has very little to do with coding and everything to do with problem solving. You only learn how to problem solve through lived experience and many people never learn that skill.
These interview tests generally fail only because they're not actually testing for the skills needed to do the job.
weatherlite|3 years ago
Patrol8394|3 years ago
What does that even mean?
angarg12|3 years ago
Leetcode-style interviews became popular in the mid 00s, primarily because they were used by hot tech companies of the time. The thing to understand is that at that time, the idea of asking people to write code during an interview what sort of revolutionary. Prior to that interviews had little structure. It wasn't unusual for the hiring manager to make the decision, some times based on credentials, recommendations, or trivial-like questions.
This type of interview became wildly popular because it allowed budding unicorns to hire programmers of high quality at scale. The process was less biased than the alternative, reproducible and scalable. Here you have two blog posts [1][2] that show the line of thought at the time.
The reality is that big tech has elevated leetcode type interview to an art. They have reached a near local optimal through years of experiments and refinements. It is working well for them so they don't have the need to take big risks such as completely revamping their hiring process.
I love the topic of hiring and interviewing and I'd love to truly get at the bottom of which method works best. I like this article because it explicitly calls out shortcomings with typical alternatives that are not usually mentioned. I hope in the future a new crop of unicorns can take these practices to the next level and do a fair comparison.
[1] https://www.joelonsoftware.com/2006/10/25/the-guerrilla-guid... [2] https://sites.google.com/site/steveyegge2/five-essential-pho...
frostmatthew|3 years ago
And I think it's a great idea, personally I would never consider working anywhere that hired devs without seeing them write some code. But my problem is with the types of questions asked at many companies, specifically the types that require weeks (or months!) of prepping.
During my last job search one interview that stood out (positively) was a problem around parsing some HTTP headers (it started simple and then had layers of complexity added as I solved each one). It was honestly one of the best questions I've seen in an interview as it requires the candidate to be able to write code and solve a problem but without requiring/expecting the candidate to have prepped beforehand to learn (or brush up on) theoretical stuff far removed from the types of problems we actually solve on a daily bases (evidence of this disconnect is demonstrated by the fact people need to prep).
dijit|3 years ago
I had two jobs at the turn of the decade and they were conversations about topics such as “where would I go to find authoritative information on x” (google is not the answer) and approaches to troubleshooting.
Granted, I’m a script monkey (sysadmin) but some level of automation has always been part of the job.
I’ve had precisely 1 leet-code style interview (2hours for 16 problems) and I couldn’t tolerate the pressure.
The interview I had at google was much more humane than what people are attempting to cargo cult.
jsnk|3 years ago
I see whole bunch of people complaining about Leetcode (LC) style coding questions, but I don't really see any alternative that is as good as LC. LC interviews solve all the following criteria better than other interview formats in aggregate.
- Objectivity: can you evaluate candidates as objectively as possible as opposed to subjectivity?
- Scalability: can you evaluate many candidates fast with high throughput?
- Accuracy: can you filter out bad candidates? Yes, the filter is often restrictive to the point where good candidates also get filtered out unfortunately.
- Meritocratic: can you make sure the process is assessed based on the merit of solving LC questions well as opposed to having connection to someone you know at the company or having gone to some prestigious school or other companies?
Leetcode format does good in all criteria above, or at least better than other formats below in aggregate.
1. Take home assignments.
- Objectivity: bad
- Scalability: bad. Once the question leaks, it's harder to come up with another question easily. Also doesn't scale for candidate side either. As a rule of thumb, I always decline interviews that require take home because it's useless to try hard as this is not applicable to interview at other companies. ROI is extremely low for candidates.
- Accuracy: good.
- Meritocratic: good
2. Coding questions with heavy emphasis on practicality (Stripe & Square)
I was pretty impressed with questions from Stripe and Square. They created problems that are heavily related to string, array, hashmap manipulation, searching, replacing etc using regex and such. If you are not into LC questions, you might prefer this, but I actually found some of these questions harder than dynamic programming questions I got from Google.
- Objectivity: good
- Scalability: bad. Once the question leaks, it's harder to come up with another question easily.
- Accuracy: good.
- Meritocratic: good
3. Knowledge / Trivia questions about programming language or framework
This might make sense a bit if you are specifically looking for iOS, Android or web developer but it still will need to be paired with LC questions.
- Objectivity: good
- Scalability: good
- Accuracy: bad. Candidates can memorize and recite these answers to the questions. Also too many good candidates actually do not memorize the details about programming language or framework. Also it can easily obtained from just Googling so people can cheat easily.
- Meritocratic: good
4. Pure behavioral
- Objectivity: bad
- Scalability: good
- Accuracy: bad
- Meritocratic: bad
5. Ex-coworker
- Objectivity: bad
- Scalability: bad
- Accuracy: good. If you've worked with the candidate before, you'll have a pretty good grasp of whether or not he/she can do the job well.
- Meritocratic: bad
dccoolgai|3 years ago
weatherlite|3 years ago
IshKebab|3 years ago
Best I can think of is take home tests but I wouldn't want to rely entirely on that, and they have their own problems too.
tombert|3 years ago
I have a decade of experience, working as a senior and staff engineer at megacorporations, have experience as a research scientist, am in a PhD program for computer science research, but lets just double check that I know how to use a hash table.
mlazos|3 years ago
quacker|3 years ago
If it is standard at the company to give all candidates that same question, this is actually a good way to reduce bias in hiring decisions. Otherwise, you might bias toward hiring, say, people with college degrees or people who worked at large corporations, for example. And this might unintentionally bias against certain demographics that obtain college degrees at lower rates (for example).
ma2rten|3 years ago
Also, typically the point of these questions is to see if the candidate can solve a novel problem not if they can use a hash table.
ng12|3 years ago
IshKebab|3 years ago
When I started interviewing I started off by jumping into what I considered to be not insulting questions, but I very quickly learnt how awkward that can be if the candidate just had no clue.
Now I start with for loops and preface it with "apologies if this seems too simple" or something like that.
You would be surprised how many people can't do a simple for loop. I mean really simple.
wiseowise|3 years ago
Apocryphon|3 years ago
Should it be from university CS programs? Which are supposed not to be vocational? And would discriminate against those who are not degree holders?
Should it be from internships, which are nowadays subject to the same Leetcode examinations as actual work?
Should it be from junior developer roles, which are an endangered species, as every business in desperate need for seniors, and lack the patience to train?
Maybe open source can be a sort of free apprenticeship to teach developers these good practices?
Or maybe grinding away at Leetcode, Project Euler, and arbitrary coding puzzles is really the only pedagogical solution.
mr_toad|3 years ago
It is how people pass these tests, but it’s pretty poor pedagogy.
isbvhodnvemrwvn|3 years ago
We have no issue whatsoever hiring juniors (we hire about as many as mid-levels), but we don't have to settle for someone who doesn't have internships or projects and the openings can be closed in 2 weeks so it seems like nobody is hiring, but actually the bar has risen (and there are plenty of candidates who clear the bar).
azangru|3 years ago
I am confused - how is this question materially different from asking how engineers are expected to know what they will need in order to properly do their jobs? Just how are engineers expected to know programming languages, or domain-specific languages, or standard libraries, or platform apis? Is this what you are asking?
andreygrehov|3 years ago
Corporations are not testing your knowledge of algorithms per se. While that knowledge is obviously important, what's more important is how dedicated you are when it comes to reaching a goal. Essentially, they are testing your will power. You know the rules. Can you achieve success and not give up in the middle of the road by skipping, say, Dynamic Programming? They could've tested your level of dedication by asking to bake some fancy cake, but throwing CS-related questions just makes much more sense.
That reasoning completely changed my perspective about algo-heavy coding interviews.
neilv|3 years ago
That's an interesting theory. And they pick the goal that happens to be best suited to fresh/current undergrads from Stanford who weren't working a draining job to put themselves through school, so can afford the additional person-months to achieve the gatekeeping that was set up by people like them, for people like them.
I and many other experienced software engineers have demonstrated perseverance and rising to the occasion of extremely challenging goals. But that theory suggests they're not interested in that. Maybe they're interested in the class shibboleth that was established when computers became less of a meritocracy for people with a passion for it, and more of a go-to status job for the upper middle class?
qz_kb|3 years ago
There's other ways to tease out if someone is "dedicated to reaching a goal" like actually asking them questions about their life, daily routine, accomplishments etc. I think these are much better signals than "this person can implement many sorting algorithms and solve towers of hanoi or the egg drop puzzle without a google search to refresh their memory" with an N=1 sample size used to gauge how reliably they can do that. How many people if asked to do this on a job rather than an interview a year later would then go and implement it from memory without double checking they didn't forget some edge case on stackoverflow?
I know grinding leetcode is fundamentally useless because you will immediately start losing your ability to interview once you get hired. If you don't change jobs within a year you'll need to start studying all that crap again for the next interview.
slyall|3 years ago
If you are a no-name company paying whatever your HR department defines as "market rate" and you don't get hundreds of very qualified candidates for every role then interviewing like Google might not be a great idea.
ta988|3 years ago
jackblemming|3 years ago
Cd00d|3 years ago
"Great" filtering for parents or people caring for ailing family or any other non-work commitment.
I guess it's not ageism at all, right?
hiyer|3 years ago
That may sound good in theory, but in practice I've been rejected quite a few times for not delivering the correct or most optimal solution despite trying my best.
alok-g|3 years ago
lapcat|3 years ago
jfoutz|3 years ago
I have two very memorable no's. They were for pretty senior/advanced IC roles. Juniors would get far more slack. I usually ask an expression evaluator question.
One person, used eval(string). Brilliant, absolutely brilliant. full points for answering, but how do we make sure the string is safe to eval? Long frustrating discussion around regexes that that maybe a junior dev is responsible for maintaining. Man, just write the parser. I know you can do it. Give me something that's something that's not _crazy_ to put into prod. Give me something that won't be a bug tarpit. Sorry man, I loved your answer. I hated your response to how this can be operationalized. Clearly super smart. Maybe I should have given the green light, but the response to criticism was so bad. There are lots of people I disagree with, that I'm happy to work with. I think I made the right choice, but I wonder from time to time.
Another person I absolutely would have loved to work with, and would have learned a ton. Very into formal methods, or slightly relaxed versions of verifying reliability. My organization at the time was pretty fast and loose. I believed they would be miserable. In the interview I told them I'd give them the green light, but they'd have a huge uphill battle.
it's so hard to tell. People are adaptable, small mismatches can be papered over, but big mismatches - it's so tough to say. Relaxing and being vulnerable on both sides is so hard to do well. Maybe I was just a jerk. I think I made the right calls, but it haunts me.
drewrv|3 years ago
Personally I don’t mind a whiteboard session or two during a loop but what’s wild to me is how, especially at big companies, you’re expected to do four or five of these to get an offer.
How often do these companies decide “well they understand when to use DFS and they can merge-sort a linked-list and they knew to use dynamic programming but we can’t hire them because they couldn’t remember how to implement a heap”?
Apocryphon|3 years ago
The more casual, flexible, subjective coding interview described by the OP is not the prevalent form of whiteboarding interviews, which has been superseded by the weeder version.
ergocoder|3 years ago
I agree that this is unreasonable but not for the reason that you mention.
Once you can pass this kind of interview, you can throw me 10 more and I will still pass them.
s1k3|3 years ago
Edit: on further thought. I think I’d be ok with coding interviews if the interviewee got to bring in their own coding question and the interviewer had to do it as well. That way the they also knows the people they are working with can live up to their ideal of a good coder.
drewcoo|3 years ago
It's saying coding interviews are not spectacularly awful if you (thoughtfully) change them.
Great! That's what everyone complaining about them has been saying in one way or another all along, change them.
autarch|3 years ago
I wrote about my own experiences with coding challenges during my recent job search on my blog at https://blog.urth.org/2022/04/19/software-job-search-2022-re...
I'll summarize my conclusion about live coding challenges, which is that I'm not convinced that my performance on these challenges reflected any abstract skill I have. Instead, they mostly reflected the fact that the problems I was given were either nearly identical to work I've done in the past, or were similar enough to things I'd done recently that they felt pretty easy.
I guess there's _some_ signal in that, but I don't know if it really says anything about how good I am at coding in general.
WalterBright|3 years ago
s1k3|3 years ago
Justsignedup|3 years ago
Here's some data models, and here's how you can reference eachother by properties. Cool. Let's do 3 levels of nested loops to extract the relationships and map them. That's like 99% of what they'll be doing.
The signals I'd look for are "do they start the entire thing by looking at structures or not", etc.
But unfortunately this breaks down when you're interviewing 10-year+ candidates. It doesn't yield anything useful. So sticking for those positions with a really really basic problem and pseudocoding it, and using the remaining time to instead have them map out solutions they did in the past seems to be useful.
Most things leet code interviews screen is "does this person know how to cram" because I tell you, everyone I know who aced leet code interviews crammed for 3 months and literally knew any one you can think of, and every variation. Juniors tend to do better in those, ironically enough.
lapcat|3 years ago
Is it ironic or intentional? Seems like a great way to hide age discrimination under the guise of "fairness" (everyone gets the same test).
wespiser_2018|3 years ago
lelanthran|3 years ago
As I commented above, I successfully passed multiple leetcode interviews in the last four years, including FAANG. I did not spend a single minute on prep.
amitport|3 years ago
A) SWEs are shit behavioral interviewers in any other way. You are not about to chnage that even if you are google.
B) It is crucial that newcomers be hired by their peers and by their direct manager. This has solid research. People hate it when they get an hire and they hadn't a meaningful part in the decision... The new hire is more likely to fail.
A+B means it is better to have a terrible process lead by SWEs (e.g. coding interviews), than a good one lead by professional recruiters (which will fail, no matter their capabilities and methodology) (and yes, I do believe that without considering A, some non technical recuiters trained at workplace psychology will have a much better success rate without needing any technical interview. They will need someone technical to have a talk with the candidate, but definitely not to ask leetcode etc).
hawaiianbrah|3 years ago
zainhoda|3 years ago
buro9|3 years ago
It is an interview about your communication, whether the code you write can be maintained, whether you can inspire/build trust (can you explain what you're doing? does that feel reasonable to the other person?).
Where people fail is to think that it's a test of experience, or a "heads down I must make every test case pass", that the outcome matters more than the process.
It's all about communication.
invalidname|3 years ago
The "write code" types of interviews fit for juniors who might not know how to even write code properly. If a person doesn't know how to do that and is a junior they usually have the right energy and can put 100% into the job to adapt/learn. I hired such juniors who had no experience in the languages/platforms and were productive within a month. They had good character and discipline.
Distilling a question like that for an advanced coder is impossible. No wonder the article didn't provide any sample of a "good interview". We'd all burn him at the stake because we'd all hate it. There's no such piece of code that would provide a valuable data point about a candidate. So this is a waste of your time when dealing with a junior and with an advanced coder.
Coding interviews are difficult. You will make mistakes, this is unavoidable. I think the best mistakes to make are the ones where you hire people who are part of your team and can grow with it. Even if they aren't the best coders around, being part of the team will help them hone those skills and evolve. If I had to pick a 10/10 coder or a team player I'd pick the latter every time.
sys_64738|3 years ago
jwmoz|3 years ago
Got the job.
nowherebeen|3 years ago
zach_garwood|3 years ago
unknown|3 years ago
[deleted]
scarface74|3 years ago
Then I did a slight pivot to get into BigTech via cloud app dev consulting - “application modernization” [2] - still with no coding interview although I still do the same type of enterprise/cloud app dev I did in the latter part of my career in the “real world” - and a shit ton of yaml.
Even in 2020, if I couldn’t have gotten a job at either Azure or AWS, I probably would have done what it takes and practiced to increase my income by an easy six figures or more.
Coding interviews are a “gravity problem”. You can not like either. But they still exist.
That being said, if I were starting out today and could break the can’t get a job <-> don’t have experience cycle by “grinding leetcode” for few months and and end up in the top 10% of income earners in the US right out of college, I would have definitely done it and I encourage my younger relatives graduating in CS to do it.
However, if you’re just hiring for your standard yet another senior CRUD developer and paying as such ($90K-$160K) - don’t expect me to do the monkey dance.
I think algorithm style coding interviews are the great equalizer. Even though I at 48 probably couldn’t pass one without some serious practice.
[1] https://www.hanselman.com/blog/dark-matter-developers-the-un...
[2] yes it’s a full time position with the same compensation structure.
fdschoeneman|3 years ago
I'm finishing up a personal open-source project and am looking for a new position focused on ruby or rails, so if you're hiring and are open to evaluating me based on something cool I've written, hit me up.
mrwh|3 years ago
And here's where our industry isn't a "real" profession yet. There are plenty of qualifications one can get, but none that gates the ability to call oneself a software engineer. The result: we have to prove it in interviews.
And I don't know whether this is better or not. It is different though. The price of not having to sit through exams is to have to jump through hoops.
lordnacho|3 years ago
The thing with software is people tend to think of it as something that needs maintenance to stay fresh, perhaps a bit like piloting an airplane.
Airplanes have logs and pilots recertify on an ongoing basis, but software doesn't work like that. Someone might have stopped coding but still have a title.
I'm not saying software actually does work like that, but I'm sure it's a view some people subscribe to.
schwartzworld|3 years ago
ankurdhama|3 years ago
unknown|3 years ago
[deleted]
NickChasanis|3 years ago
gunfighthacksaw|3 years ago
Obviously it’s pants on head stupid to do that if you want a web dev to make a pretty app, tie frameworks together and liaise with 3rd party integrators. However, I think many employers want the crème of such devs who are also good at algos.
manuelabeledo|3 years ago
Which is not going to work for the vast majority of devs and jobs.
Most devs either don’t care or need to care about algorithms, and the few devs who do, spend most days not using them.
This would be akin to asking a civil engineer to build a wooden road bridge. Having the skills is great, but how often is the company going to need them?
mr_toad|3 years ago
I think that many employers don’t have a clue what makes a good dev, in general, or in the positions they’re trying to fill.
Nextgrid|3 years ago
bachmitre|3 years ago
endisneigh|3 years ago
A completely fictitious language is developed. Simple enough that you can become fluent within an hour. From this you read a bunch of problems in said language and must solve them.
A set of assertions about the problems are written down and you must verify them. Finally a full program, created from a dialect of the given language is given and you try to understand
esoterae|3 years ago
manuelabeledo|3 years ago
I’m also not familiar with other fields, but are doctors or lawyers asked to do anything analogous to code assessments during interviews?
sokoloff|3 years ago
Drive, experience, intelligence, communication skills, teamwork, emotional intelligence, grit, curiosity, physical health, mental stamina, family/home situation, and dozens of other factors play into job performance, many of which you can’t even ask about to generate your test dataset. Many of these traits are what you’re trying to measure the effects of in the interview.
andreygrehov|3 years ago
SamReidHughes|3 years ago
unknown|3 years ago
[deleted]
unknown|3 years ago
[deleted]
bfrog|3 years ago
parentheses|3 years ago
https://news.ycombinator.com/item?id=31544634
How do you test knowledge indeed?
WalterBright|3 years ago
https://news.ycombinator.com/item?id=31544634
photochemsyn|3 years ago
langsoul-com|3 years ago
eyelidlessness|3 years ago
Lol be careful not to ask for a loop, nobody uses loops in real programming problems
mouzogu|3 years ago
dont mind talking over code casually but dont put me on the spot or give me homework. not doing that.
danamit|3 years ago
csydas|3 years ago
I just want to put this out there to anyone who gets involved in interviews that this is a very important rule -- if your question can basically be redefined as some really clever solution you found on a problem you put > 30 minutes into, it's probably not a good question.
A lot of the seniors I work with would come up with questions like this, and I get their logic:
1. I'm a senior and this was a challenge for me 2. The logic is clear when you see it and look at it 3. I want people who can do work similar to me 4. I know how to get to the conclusion, so I can judge based on the path the candidate takes Conclusion: It's a good question
While I understand how we can get to such a conclusion, I think that in most cases the only takeaway you can get is "can this person think like I did in this situation?", which maybe has some value for judging workplace compatibility, but I don't think it's a fair assessment of someone's technical aptitude. If you yourself required a lot of time and brainpower to sort through a tricky issue like this with the benefit of the problem scope and system needs that led to this issue, it's very unlikely you'll be able to judge any assessment
Point 4 seems good at first because we can say "the goal is to just see them think and use their experience"; there is truth to it but without the rest of the context it's not really a good assessment because ultimately, you're looking to see "did you get the same conclusion I did? Did you avoid the same pitfalls I hit?"
Instead, focus on a general scenario that tests their understanding of basic principles. How do they use the fundamental knowledge for the subject matter to work through a new problem that you yourself haven't really dealt with. It removes your bias a bit and opens you up to those "wow why didn't I think of that?" moments that you'd have working on tough projects together.
Try your best to give a good faith interpretation to any path the candidate takes and help take it to natural conclusions, and see how the candidate reacts; if you can see they've made a workable but problematic solution, nudge them towards the problems you see and see how they discuss it.
For me a candidate that can really process their own thoughts and do a good analysis of their own solution (its benefits and shortcomings) is extremely valuable; I usually lead off with "I know how I'd do it, but that doesn't mean it's the right answer, so focus on your strategy and walk me through it." If you want tot test compatibility or you feel you have a stronger solution, accept their solution (if it's valid) then go ahead and discuss your thoughts a bit and see how they respond and how they discuss it. The more you make it a technical conversation against peers and less a university exam, the more you have to see how they think and operate and the better you can understand them.
roberthanry79|3 years ago
[deleted]
woeirua|3 years ago
It amazes me how many people will defend the current interviewing process and then openly admit that the majority of their engineers could not pass the same process without spending dozens of hours preparing.
[1] https://news.ncsu.edu/2020/07/tech-job-interviews-anxiety/
Jensson|3 years ago
But other than that the study didn't find any evidence that men have any issues with being observed while coding. So unless a better study comes around we can dismiss complaints about observed interviews being stressful unless it comes from a woman, it is in the study you linked after all. (I do believe there is a stress effect, but that study isn't showing it, need a bit more samples and probably several different observers to get better data)