I won't attempt to generalize from my case, but let me offer a personal anecdote.
I'm now a successful, self-employed indie developer. One of the main reasons I stuck with indie development through the hard times—the maxim that it takes 5 years to become an overnight success was true for me—was that I became practically unhireable. There are multiple strikes against me: I'm middle age in an industry rife with age discrimination, I don't have a computer science degree, and I experience brain freeze during live coding interviews.
I would note that not all stress is the same. Firefighters rush into burning buildings for a living, and what could be more stressful, yet many of them would panic at the idea of giving a speech to a non-burning roomful of strangers. I have no problem with stress on the job, and I've successfully navigated many work emergencies during my career, but something about strangers standing over my shoulder judging me, determining my financial future by providing or withholding a job, like the sword of Damocles, turns my stomach inside out. And like the article author, I can revisit the coding problems and solve them afterward, as soon as the interview is over. Interviewers must think I'm a fraud who can't code, yet all evidence from my career, now almost 2 decades long, suggests otherwise.
A lot of commenters causally speak of "false negatives" as if they were random, but some people, myself included, are always the false negative. I am consistently filtered out by audition-style interviews. I'm not a stage performer.
[Edit:] I didn't expect my comment to rise to the top of an already crowded discussion. I feel a little self-conscious about it. ;-)
Just this week I interviewed a candidate for a Data Engineering role. I gave him four simple SQL statements and he got it instantly. He read the instructions out loud and typed the solutions immediately with no hesitation, perfect syntax. The last one increased the difficulty slightly and he hit a snag. I asked him to "check his work" and he got baffled and defensive and kept repeating "what?" I said "check the table" and he repeated "What?" I finally said "just dump the first five lines of your table" and he couldn't. He then started yammering and giving excuses. Then he pasted some SQL that included [redacted].ai in the output. I suspect he read the instructions into an AI for the first problems. When I asked him to "show the work" he did not understand how to prompt the AI to do that. I'm so glad I gave him that tech challenge, otherwise I would not have caught the cheating.
AI interview cheating tools are becoming very popular among younger people. Some times it’s easy to spot, but others are getting very practiced at using the AI and covering the pauses with awkwardness or “you’re cutting out” tricks.
It has become the most common topic in the hiring sub forum of a manager peer group I’m in.
The companies who can afford it have added in-person final stage interviews. Candidates who do okay in simple remote technical screens but can’t answer basic questions in person are rejected. It’s a huge waste of time and money, but it’s better than the cost of a bad hire.
The A.I. use isn’t limited to technical screens. The people who use gen AI use it everywhere: Their resume, behavioral questions, and even having ChatGPT write S.T.A.R. style responses that are wholly fabricated for them to repeat later.
Verified reference checks are more important than ever. Few people will actually come out and give a negative reference check, but talking to someone can reveal very different pictures about the person’s work. I’ve had calls where former managers of a person told me they worked on something completely different than what was claimed on the resume. Sadly I would have probably hired the person if they had been honest about not having direct experience in our domain, but once you catch someone lying so directly it’s hard to trust them for anything else.
I am currently interviewing candidates and so far about 50% of them used live GenAI to answer questions. I think so far it has been trivial to notice who was doing that. It takes very little to figure out if people know what they are talking about in a natural language conversation.
Ironically, the last candidate I interviewed 2 days ago repeated all the questions back as well, and also needed 10-15 seconds to think after each and every question.
All of this to say, I don't think these tests are an optimal solution to this problem, since they also introduce new problems and cause good candidates to be discarded.
If you look at this through the lens of "using AI isn't cheating, because it's what they would be doing on the job", your interview was actually very effective, and a solid, tiny-bite-sized example of translating requirements.
I think this is a relatively positive direction of exploration in interviewing - let people use the tools that they will have on the job, but also ask them to do the kinds of things they'd need to do on the job, with the appropriate language. If they know how to get the AI to help them with it, more power to them.
I suppose this is just a rephrasing of "point-blank Leetcode questions are a bad interview technique and we should do better", though.
As with everything, it depends. Live coding interviews work. They’re not the best candidate experience, but they work at Meta, Google scale, minimizing false positives better than most other formats. What makes them stressful is the lack of interviewer training and the abstract, puzzle-like nature of the problems, which you can really only solve if you’ve spent time studying (e.g., LeetCode) or you’re fresh out of college or academia.
I’ve worked in the assessment space for 6 years and have seen many hiring processes, from Fortune 10 companies to startups hiring their first engineers. The range of signal required, "how much time can my engineering team spend with a candidate", and how much candidate experience you can get away with, is huge. I’ve also been a candidate myself and failed many live coding interviews. It made me feel terrible about myself. The last time for a role at Ycombinator (the interviewer was super nice).
When I work on my product, I try to view it through the lens of empowering candidates to show their skills and potential. I encourage our customers to use assessments that somewhat resemble on-the-job skills. I don’t like the phrasing “real work” anymore. An assessment shouldn’t be unpaid labor, it should be a way for candidates to demonstrate that they can do the job and handle future work thrown at them, and for hiring managers to feel confident extending what are often very high salaries in tech.
With AI, unfortunately, short take-homes (what I prefer as a candidate, using my own tools and editor) are becoming harder to maintain as a fair signal due to AI assistance. I’ve seen companies move back to onsite, and competitors deploy all kinds of proctoring and invasive monitoring.
The perfect solution, in my view, would be an assessment where the candidate feels relaxed and able to perform at their best, with their own editor and configuration, knowing that every other candidate in the pool has the same constraints in terms of time and tooling. It’s a tough problem to solve. I think about it daily and have not come up with a solution.
A brick layer isn't asked to build a wall prior to getting a job. A certificate confirming he did learn the job is enough for companies to employ them. Same goes for a lot of other jobs. Just hire devs and kick them out if they dont fit ur needs. Why go through these humiliating corpo processes?
> They’re not the best candidate experience, but they work at Meta, Google scale, minimizing false positives better than most other formats.
Do they? That is an assertion with out any data. I've worked with F tier developers who worked at Facebook, and Google. They're trying to minimize their false positive rate, but they don't realistically publish it other than laying off large portions of their work force. Presumably because they weren't great, but passed the interview process. If you lay off 3-5% of your staff in a year you had a 3-5% false positive rate. Maybe it's minimized by these in person interviews, but that seems like an unacceptably high false positive rate given how much time the interviewers spend on the process.
3% is probably about as accurate as 'solve fizz buzz in python' prior to AI, and that's where they were in 2022.
> As with everything, it depends. Live coding interviews work. They’re not the best candidate experience, but they work at Meta, Google scale, minimizing false positives better than most other formats. What makes them stressful is the lack of interviewer training and the abstract, puzzle-like nature of the problems, which you can really only solve if you’ve spent time studying (e.g., LeetCode) or you’re fresh out of college or academia.
They work (for some) and leet-code weeds out the frauds that really cannot problem solve and assesses those who have not built anything to show to justify not doing it and can be applied to companies that are joining from an acquisition.
> The perfect solution, in my view, would be an assessment where the candidate feels relaxed and able to perform at their best, knowing that every other candidate in the pool has the same constraints in terms of time and tooling. It’s a tough problem to solve.
And that would be the fairest one which is to do the leetcode interview in person on a projected whiteboard and pair programming with the interviewer.
This has not been my experience at all. Years ago being ex-FAANG was a pretty good signal, but as the filter has increasingly become "regurgitate a bunch of leetcode" the quality of FAANG engineers has plummeted.
The teams I've worked that have used live coding filters all had far worse devs than all the startups I've worked with that didn't require live coding.
In the old "algorithm whiteboarding" days, I think it was a decent signal, but now it's just a sad parody of this and the results show.
Yeah, in my admittedly limited experience the grade I assigned in a live code test (slightly more than fizzbuzz, but no tricks or algorithms required) matched up very closely with real world performance.
I even have knowledge of some of the fails as people higher up the chain decided to hire them anyway. They didn't do well from the feedback I received.
- What makes them stressful is the lack of interviewer training and the abstract, puzzle-like nature of the problems, which you can really only solve if you’ve spent time studying (e.g., LeetCode) or you’re fresh out of college or academia.
Some of the best engineers I've ever met are always false positives. They get nervous in live interviews. I don't think one can say live coding interviews work so matter of factly when it's eliminating some top, top computer scientists.
I don't think those explanations are mutually exclusive.
Yes, there's a large cohort of "senior" software engineers who can't actually code. They bullshit their way into jobs until they're fired and then apply for the next one. These are the people you want to filter out with live coding.
But also, someone can fail a live coding interview for reasons other than belonging to that group.
I’ve found in my interviewing that since having a child I am way worse at coding interviews—in ways I never was before. Perhaps this is due to being at a higher level of stress all the time—worrying about a young child’s wellbeing is a full time stressor (it doesn’t even stop when they go to sleep!)—but it also seems like there is so much more riding on an interview. When it was just me, and I was younger, if I failed, “oh well, I’ll get it next time.” Now there are concerns about health insurance, a mortgage, retirement &c.
I get stuck in ways I’ve never experienced in my life; my brain feels like it stopped working (but has reserved the bandwidth to shout at me the whole time “you’re blowing this!”). Then shortly after the call ends, sitting quietly, a path forward opens and suddenly I’m working on a solution, which it’s own special kind of hell—to know you could do it but didn’t and now some stranger out there thinks you’re a real idiot.
Studying/practicing has mixed results as you are taking away time from your partner and kid, this causes feelings of guilt. I found the more time I spent practicing, the more leetcode hards I solved, the worse I got. It was paradoxical! It was additionally demoralizing!
I only share all of this to say that circumstances change; you change. What once was easy becomes hard—maybe just for now, maybe forever, I don’t know. What I do know from my time at Big Co., is that this style of interview (often aped at smaller co.) is employed as one of the stated goals is to remove opportunity for biases (test everyone the same "quantifiable" way). However, my experiences lead me to believe that it falls short in that regard. As I said, people change, and a candidate who becomes a coworker with a high tolerance for stress today might not have the same appetite in a few years. Surely removing biases is about seeking balance, but I think certain types of interviews are blind to the imbalances they might cause.
I firmly believe that the only good proxy for how well one can do the job is doing the job. Even if there are good proxies for doing the work, why would you choose to use a proxy when you can just do the work? And if you're work is some complicated that you can't break off some piece of it and do it in an interview, maybe you're making stuff too hard for no particular reason.
Let's say you need someone who can lift 10 kilograms:
- Good interview: "Please lift this 10 kilogram bucket by the handle."
- Not Good interview: "As a proxy for your overall strength, please take off your pants, squat, pick up this 1 kg bucket by clenching the handle with your butt cheeks, and then stand. We know this isn't a real test of your strength, but we want to see how you perform under pressure."
EDIT: what I mean by doing the job, I mean test the skills used on the job. See if a chef knows their way around a kitchen & actually cook something. See if a customer support rep has good written & verbal communication skills in a mock support interaction. See if a phone screening can do phone screens. Stuff like that.
> why would you choose to use a proxy when you can just do the work
"What a useful thing a pocket-map is!" I remarked.
"That's another thing we've learned from your Nation," said Mein Herr, "map-making. But we've carried it much further than you. What do you consider the largest map that would be really useful?"
"About six inches to the mile."
"Only six inches!" exclaimed Mein Herr. "We very soon got to six yards to the mile. Then we tried a hundred yards to the mile. And then came the grandest idea of all! We actually made a map of the country, on the scale of a mile to the mile!"
"Have you used it much?" I enquired.
"It has never been spread out, yet," said Mein Herr: "the farmers objected: they said it would cover the whole country, and shut out the sunlight! So we now use the country itself, as its own map, and I assure you it does nearly as well."
Because that takes too much time. They are using proxies biased towards false negative rather than false positive to filter large numbers of candidates.
Careful there- I believe a number of jurisdictions will consider using someone's work before you've hired them to be very illegal.
You could take this to the logical extreme and just not hire anyone, instead building your entire product off of the work done in interviews. Many would consider this to be a form of wage theft.
Hire them for a day/week/month, see how they do the job. qualified? ok, job
bonus is you get to try different jobs, don't like it? you know you can get another one to try out easily. employer also gets to try different candidates easily with little vested resources. able to find canidtes that actually/really like/enjoy the job, better productivity
(of course this is not compatible with employer based healthcare)
It usually takes significant time for someone to get up to their base level of effectiveness in a new organization, so I don't really think this is better, in fact it's much worse because "doing the job" includes a lot of ancillary things like familiarizing oneself with the tools, metrics, codebase, etc.
Much better to just have a live coding test that tries to measure your communication, effectiveness at working on a problem, and your raw coding skills.
I'm curious how you would respond to the folks who are concerned that asking people to 'just do the work' in an interview are asking for unpaid labor, and that's unfair?
Very well written and so true. It's not even normal stress, which is fine, it's high stakes stress, plus sometime working under the duress of being insulted.
I once went to a job interview with Google. I built one of the first local (Global to the Netherlands) search engines of the Netherlands, but the guy in the cowboy hat at Google asked me to write a binary search with a marker and a whiteboard. I never write with my hand, I always use keyboards. Plus I'm being insulted to write a binary search when I designed and build a search index and retrieval engine.
[I did the binary search but was not happy with the whole process that did not want to even look at what you had actually done before, because that would take away the baseline they wanted].
I guess they must have been looking for cowboys. Tip for interviews, take your cowboy hat, just in case..
I've started to treat live coding screening sessions as a "conversation about code." I make sure that the candidate knows that I'm trying to make sure we can discuss code.
Why?
Purely being a "good programmer" isn't just enough for the job. It's important that we can develop a good working rapport, and communicate.
For example: Someone who's an expert programmer might misinterpret a discussion and build the wrong thing; or someone might be a great programmer but otherwise be impossible to actually conduct a technical discussion.
IE, this is why "Fizzbuz"-style questions work so well: It's not really screening that you're a great programmer, it's screening your ability to have a technical discussion.
Add me to the list of people who acknowledge the problem but haven't heard any alternative. Every job opening, you encounter people who can talk and act like software engineers, but can't write code. I've been in the business for about a quarter century now, and looking back at times that teams decided to overlook someone's failure to demonstrate coding skills in interviews, sometimes it has turned out they can write code, but often not.
Younger engineers trying to find jobs in the industry should consider how much the other parts of the hiring process privilege older engineers regardless of ability. I'm sure many of the people I worked with who lacked basic coding competence twenty years ago still lack competence and are employed right now in jobs that require it, jobs that could go to younger, more capable people except that their resume, their experience, and their ability to say the right-sounding thing don't match up.
Hiring someone to write code without ever seeing them do it is like hiring a guitar player for a band by verifying their MFA in music theory, talking to them about music for an hour, and checking their references. All that is fine, but can they play the guitar? You can acquire everything else, the lingo, the knowledge of music theory, the pros and cons of competing techniques, the familiarity with current and past great performers and performances, without ever putting your hands on the instrument. And some people are built like that. Some people are guitar connoisseurs: they gave up their hopes of playing the instrument long ago, but they're huge fans and nerd out over the art and science of it. If those folks needed to be in a band to have health insurance, and they didn't have to audition, you can bet some of them could talk their way into it.
Finally, when a team interviews someone for a position that requires writing code, they are inevitably making a judgment about whether they think they can do it. If you ask someone to guess without any direct evidence, you're inviting them, almost forcing them, to fill the gap with bias. Do they collect subtle little hints and make fragile deductions like they're Sherlock Holmes? Do they trust their instincts about whether the person in front of them looks and acts like the kind of person who could code? Either one is guaranteed to be rife with bias and problematic preconceptions.
I’ve seen this “stress ≠ skill” gap play out again and again. The Microsoft study the OP cites is brutal. Yet most prep advice (“just grind LeetCode”) still ignores the cortisol factor.
One thing that’s helped me, as both an occasional interviewer and a frequent interviewee, is recreating the stress loop on my own terms. Friends rarely push hard enough, and a paid coach is $150+ an hour. Lately I’ve been working on a little side-project Tough Tongue AI: a voice-driven agent that drops you into a live code editor, throws follow-up questions, even interrupts when you go off-road, then gives a feedback at end. It’s not magic, but after a few sessions the “somebody’s watching me” adrenaline spike starts to feel familiar.
If live coding interviews are here to stay, a repeatable way to train the physiological side of the test, not just the algorithms would be helpful!
one thing that helps me psychologically is not caring. Or tricking myself into not caring.
I learned this in my 20’s. I made it to about 10 different on site interviews over a summer, and was totally burnt out. I was spending hours prepping before each interview, and really putting in the effort! At the last minute a company tried to squeeze me in for an interview before a vacation i had planned. I reluctantly agreed, and i completely winged it. Zero prep, i showed up ready to (mentally) go on vacation lol. Guess what? I aced most of the interview and ended up getting an offer 3 days later.
Of all the interviews i performed the best when i went in with 0 expectations and 0 stress on myself. That’s when it clicked for me.
Easier said than done though. It’s not always easy to get into that mindset.
I am working on a documentary film on the software engineering interview process, Computer Silenced, which is feature length extension of my project Whiteboard Challenge(R). This film focuses on this exact issue, but more specifically how it affects people diagnosed with hidden disabilities, e.g. Autism, ADHD, and Anxiety as well as PTSD and other stress related symptoms.
I have 25+ years in software engineering and have been filtered through the non-job related interview process described here. I also have a medical diagnosis of ASD L1 and ADHD along with stress induced dyslexia.
Can others here please read the following federal law and confirm it applies. Through the film, I am trying to amend the definition of a disability, so we have equal rights.
Title 42 U.S. Code § 12112(b)(6)
"using qualification standards, employment tests or other selection criteria that screen out or tend to screen out an individual with a disability or a class of individuals with disabilities unless the standard, test or other selection criteria, as used by the covered entity, is shown to be job-related for the position in question and is consistent with business necessity"
I do simple questions similar to the quoted LinkedIn post. (So less leetcode and more "do you know how to code anything".) While I can agree it is harder under stress, what is the alternative to knowing if someone can code at all? (My pass rate is similar to the quoted LinkedIn post.)
I have done the entry interview as well. For example have had very good conversations with managers who were applying for senior IC roles. They then go and fail the tech interview basic "can you do write and execute a python hello world script from the command line".
I was a contestant on Jeopardy, and led going into Final Jeopardy 2 out of the 3 matches I played. So I'm an above average Jeopardy competitor. My edge is not in pure trivia. I'm probably below average, by the standards of people who have won a match on Jeopardy. I suspect I'm above average in dealing with the competitive aspects of it and the stress management. It makes a big difference in a fast-paced game. The reality is that Jeopardy is not just a trivia game, but also a game of reflexes, decisionmaking, and stress management.
There are certainly jobs where you need someone who is cool under intense pressure, where what happens inside an hour matters. That's what these interviews are revealing. But I think this is a tiny minority of dev jobs.
However, I also think when people talk about hiring practices, the elephant in the room is that the true purpose our interview processes have evolved to serve is taking the candidate pool from a lot of people to something that is decidable. We can't fully evaluate 500 candidates, but we can evaluate 5. The process of weeding out 99% is designed to be inexpensive, at the cost of accuracy. The hope is that you'll have one person squeeze through the filters that is a good fit. If that happens, then all the false negatives are tolerable.
But we really, really don't like to admit this. We tell ourselves that we are running interview processes that are predictive of performance when there is actually very little evidence to support that, and plenty of reason to doubt that it's true.
>Here’s what they found: The participants being watched scored half compared to those who were alone.
I once had a coding interview where I was given a laptop with no internet access and 30 minutes by myself to solve a couple questions a step or two above FizzBuzz, in pseudocode. The interviewer then came back into the room and we discussed my solutions, using them as a jumping-off point into a more open-ended discussion (e.g. "why did you use a hash table here? can you think of another approach?" which then led to a general discussion on the suitability of hash tables versus other approaches for related problems.)
It was one of the best technical interviews I've had. I'm surprised more places don't do this. It's a win-win-win: it relieves stress on the interviewee, gives the interviewer back half an hour of valuable time, and makes for a much more productive assessment of the candidate — it's a lot easier for a candidate to explain code they've already written, and thus a lot easier to an interviewer to assess the candidate's thought process.
I used to do tech screening in my previous job and not only it measured stress, it was awfully biased against older people. We lost so many senior people that I really wished I could work with because they weren’t used to jumping through the hoops and loops of leet code questions (in my country leetcoding is fairly new). Then there were other younger candidates that were pretty mid but knew all the answers to leet code and design questions and ended up getting the job.
I thrive in high-stress situations (for short periods of time). Examples include hardware validation before a large production run, putting out literal fires in manufacturing sites, and working in foreign countries to troubleshoot/rework bad hardware. I do fine in live coding interviews, they don't feel much different than being alone at an editor for me.
I was interested by the author's statement: "Working memory is the most reliable proxy (I know of) for fluid intelligence, your ability to reason, solve novel problems, and think abstractly." and the linked study (https://pubmed.ncbi.nlm.nih.gov/21037165/). My working memory is not so great, but it degrades less under stress.
Question worth considering for hiring managers: do you prefer stress-capable employees, or greater working-memory employees? Is my model a false dichotomy?
Having gone through a round of interviews, I completely agree that live coding doesn’t measure anything of substance.
There was one company whose process I really appreciated:
- A take-home test, more of a puzzle-style challenge.
- The interviewer does a quick review of your submission.
- If they like what they see, you're invited to a follow-up session where you explain your code and make a few simple changes to it.
This approach proves that the work is genuinely yours, shows your thought process, and still gives the opportunity to demonstrate live coding on code you're already familiar with.
Stress or not, if you can't write the following code in 10+ minutes in your language of choice in an interview setting you are going to get rejected, period. Complaining and writing blog posts about it isn't going to make a difference. Instead spend your energy towards learning, and see a therapist if necessary.
Absolutely. Some of my favorite coworkers would never pass a modern, top company interview which I can pass, and not because they are worse in general, but due to stress management. When I am working in one of those that has no hiring shortcuts, I just can't recommend them, because they will fail. In a small enough company, I can say "hire this person sight unseen, and if they aren't good, it's 100% on me", and then they are successful. My time working with them is far more valuable at skill matching than any interview.... but that's not how we do things in most places, because the fear that people will recommend the otherwise incompetent is high, and for good reason. There would have to be consequences for recommending a bad candidate like that.
Now, that doesn't mean that stress management cannot be part of the job: I've worked in the hellish places where an on-call rotation meant at least 4 calls off-hours, and some which needed resolutions in 10 minutes or less from the pagerduty call. Someone with bad stress response is not going to be doing great at that kind of live diagnosing, with possibly many millions for the company on the line. But the number of positions like that, where you are a cross between a developer and an ER doctor are much less common than places that have none of those demands, yet give you a series of leetcode mediums and hards. They might as well be testing for height, or how much you can bench press. It filters, but it does not help.
> When you’re placed in a high-stakes, time-pressured situation, like live coding, your brain reacts exactly like it would to any other threat. The amygdala gets activated. Cortisol levels spike. [...] You forget what you just typed a few seconds ago. It feels like your IQ dropped by 30 points. In fact, it feels like you’re a completely different version of yourself; a much dumber one.
Did y'all not have to take a load of exams in high school and college?
I'm sure there are variations between education systems, but by the time I graduated college I'd done at least a hundred high-stakes, time-pressured hour long written tests. All of them substantially more difficult than summing the even entries in a list.
Is this not what everyone experiences, in every education system?
Not while performing in front of an audience, no, very, very little of that.
You know the trope of almost all students dreading going up in front of the class to solve a problem on the blackboard/whiteboard? A thing that I think I personally only actually did a countable-on-fingers number of times ever, and only in lower grades?
It's a trope for a reason, and it's because the vast majority of people do in fact hate having to get in front of an audience and perform. Hell, it's often a component of recurring nightmares.
It's like that but way worse because it's also far more high-stakes than that, the problems are far more complex, the point is explicitly for everyone in the audience to judge you and not just your assumption that they all are causing the stress, et c.
It's more like the stress from getting up at an open mic night, than any kind of stress I've ever actually encountered on the job. Even the dynamic in a meeting with clients where you have to diagram something out on a whiteboard or something is totally different—for one thing, you generally have people in there who are very much "on your side", and you don't have to worry if you misspell something that you're personally about to be/remain unemployed, or lose out on a huge raise, or whatever (nobody generally cares about minor mistakes on a whiteboard in an ordinary context, and maybe they don't in some interviews, but they do in others, and candidates worry they might in all of them).
I can't really explain why, but to me they don't feel the same at all.
I've always finished exams absurdly quickly. I used to finish 60 min exams in like 20 min and sleep the rest of the time when the professor didn't allow us to leave, because I had likely spent the previous night drinking and partying my way through college. I'd usually ace most of them.
Thing is, at those times we were working with freshly acquired knowledge that you've been practicing a lot. That's not the case with most leetcode interviews. As a senior/staff engineer, I'm not using double pointers to perform little math tricks on a daily basis, I'm troubleshooting cascading timeouts on distributed systems. I'm not worried about how fast I'm going to iterate over a batch of a couple thousand records on a list I'm worried about how much latency I'll accrue if I rely on a primary source of truth instead of hitting a cached answer.
Code interviews don't measure experience or competence. I don't even think they measure stress as the article mentions. To me, they just measure how much leetcode you practiced for that interview. Nothing more.
I've had only two oral exams, and they were awful, but even then they were far more conversational in nature, as opposed to a coding interview.
I do well on written tests.
I also do well in interviews, but I detest having people look over my shoulder while I code, and it absolutely tanks my performance. I only pass them because I'm experienced enough that I can be stressed out like crazy and perform really horribly compared to what I normally do and still do okay.
I've seen fantastic coders totally freeze up and be unable to do anything at all when being asked to write code in front of people, even though they have no problem talking through problem solving.
And I'm similar to that. I've held speeches in front of thousands, been on TV, held plenty of presentations, and can talk the ears of an interviewer with passion about complicated technical concepts, but have me write code while you watch, and I'd frankly much rather have a root canal treatment (I've been known to fall asleep during those).
[+] [-] lapcat|7 months ago|reply
I'm now a successful, self-employed indie developer. One of the main reasons I stuck with indie development through the hard times—the maxim that it takes 5 years to become an overnight success was true for me—was that I became practically unhireable. There are multiple strikes against me: I'm middle age in an industry rife with age discrimination, I don't have a computer science degree, and I experience brain freeze during live coding interviews.
I would note that not all stress is the same. Firefighters rush into burning buildings for a living, and what could be more stressful, yet many of them would panic at the idea of giving a speech to a non-burning roomful of strangers. I have no problem with stress on the job, and I've successfully navigated many work emergencies during my career, but something about strangers standing over my shoulder judging me, determining my financial future by providing or withholding a job, like the sword of Damocles, turns my stomach inside out. And like the article author, I can revisit the coding problems and solve them afterward, as soon as the interview is over. Interviewers must think I'm a fraud who can't code, yet all evidence from my career, now almost 2 decades long, suggests otherwise.
A lot of commenters causally speak of "false negatives" as if they were random, but some people, myself included, are always the false negative. I am consistently filtered out by audition-style interviews. I'm not a stage performer.
[Edit:] I didn't expect my comment to rise to the top of an already crowded discussion. I feel a little self-conscious about it. ;-)
[+] [-] runamuck|7 months ago|reply
[+] [-] Aurornis|7 months ago|reply
It has become the most common topic in the hiring sub forum of a manager peer group I’m in.
The companies who can afford it have added in-person final stage interviews. Candidates who do okay in simple remote technical screens but can’t answer basic questions in person are rejected. It’s a huge waste of time and money, but it’s better than the cost of a bad hire.
The A.I. use isn’t limited to technical screens. The people who use gen AI use it everywhere: Their resume, behavioral questions, and even having ChatGPT write S.T.A.R. style responses that are wholly fabricated for them to repeat later.
Verified reference checks are more important than ever. Few people will actually come out and give a negative reference check, but talking to someone can reveal very different pictures about the person’s work. I’ve had calls where former managers of a person told me they worked on something completely different than what was claimed on the resume. Sadly I would have probably hired the person if they had been honest about not having direct experience in our domain, but once you catch someone lying so directly it’s hard to trust them for anything else.
[+] [-] Wilder7977|7 months ago|reply
All of this to say, I don't think these tests are an optimal solution to this problem, since they also introduce new problems and cause good candidates to be discarded.
[+] [-] nlawalker|7 months ago|reply
I think this is a relatively positive direction of exploration in interviewing - let people use the tools that they will have on the job, but also ask them to do the kinds of things they'd need to do on the job, with the appropriate language. If they know how to get the AI to help them with it, more power to them.
I suppose this is just a rephrasing of "point-blank Leetcode questions are a bad interview technique and we should do better", though.
[+] [-] domrdy|7 months ago|reply
I’ve worked in the assessment space for 6 years and have seen many hiring processes, from Fortune 10 companies to startups hiring their first engineers. The range of signal required, "how much time can my engineering team spend with a candidate", and how much candidate experience you can get away with, is huge. I’ve also been a candidate myself and failed many live coding interviews. It made me feel terrible about myself. The last time for a role at Ycombinator (the interviewer was super nice).
When I work on my product, I try to view it through the lens of empowering candidates to show their skills and potential. I encourage our customers to use assessments that somewhat resemble on-the-job skills. I don’t like the phrasing “real work” anymore. An assessment shouldn’t be unpaid labor, it should be a way for candidates to demonstrate that they can do the job and handle future work thrown at them, and for hiring managers to feel confident extending what are often very high salaries in tech.
With AI, unfortunately, short take-homes (what I prefer as a candidate, using my own tools and editor) are becoming harder to maintain as a fair signal due to AI assistance. I’ve seen companies move back to onsite, and competitors deploy all kinds of proctoring and invasive monitoring.
The perfect solution, in my view, would be an assessment where the candidate feels relaxed and able to perform at their best, with their own editor and configuration, knowing that every other candidate in the pool has the same constraints in terms of time and tooling. It’s a tough problem to solve. I think about it daily and have not come up with a solution.
[+] [-] zwnow|7 months ago|reply
[+] [-] timeinput|7 months ago|reply
Do they? That is an assertion with out any data. I've worked with F tier developers who worked at Facebook, and Google. They're trying to minimize their false positive rate, but they don't realistically publish it other than laying off large portions of their work force. Presumably because they weren't great, but passed the interview process. If you lay off 3-5% of your staff in a year you had a 3-5% false positive rate. Maybe it's minimized by these in person interviews, but that seems like an unacceptably high false positive rate given how much time the interviewers spend on the process.
3% is probably about as accurate as 'solve fizz buzz in python' prior to AI, and that's where they were in 2022.
[+] [-] gabrieledarrigo|7 months ago|reply
You cannot start your reasoning with "it depends" and then continue with an absolute. I could do the same:
As with everything, it depends. Live coding interviews don't work.
What's the difference?
[+] [-] pelcg|7 months ago|reply
They work (for some) and leet-code weeds out the frauds that really cannot problem solve and assesses those who have not built anything to show to justify not doing it and can be applied to companies that are joining from an acquisition.
> The perfect solution, in my view, would be an assessment where the candidate feels relaxed and able to perform at their best, knowing that every other candidate in the pool has the same constraints in terms of time and tooling. It’s a tough problem to solve.
And that would be the fairest one which is to do the leetcode interview in person on a projected whiteboard and pair programming with the interviewer.
Very relaxing.
[+] [-] nailer|7 months ago|reply
On the job will I constantly have to craft a narrative for another person while I code?
Do you know how many autistic people who are great programmers you're throwing away by asking them to multitask rather than letting them deep focus?
[+] [-] belter|7 months ago|reply
[+] [-] crystal_revenge|7 months ago|reply
This has not been my experience at all. Years ago being ex-FAANG was a pretty good signal, but as the filter has increasingly become "regurgitate a bunch of leetcode" the quality of FAANG engineers has plummeted.
The teams I've worked that have used live coding filters all had far worse devs than all the startups I've worked with that didn't require live coding.
In the old "algorithm whiteboarding" days, I think it was a decent signal, but now it's just a sad parody of this and the results show.
[+] [-] quails8mydog|7 months ago|reply
I even have knowledge of some of the fails as people higher up the chain decided to hire them anyway. They didn't do well from the feedback I received.
[+] [-] unknown|7 months ago|reply
[deleted]
[+] [-] throwzasdf|7 months ago|reply
This allows ageism without repercussions.
[+] [-] sotix|7 months ago|reply
[+] [-] lesuorac|7 months ago|reply
That's the issue though. If you're paying top 1% wages then you should get top 1% performers.
If you want to pay bottom 20% wages then how do you select them using a whiteboard test?
[+] [-] lubesGordi|7 months ago|reply
[+] [-] exabrial|7 months ago|reply
do they?
[+] [-] codeflo|7 months ago|reply
Yes, there's a large cohort of "senior" software engineers who can't actually code. They bullshit their way into jobs until they're fired and then apply for the next one. These are the people you want to filter out with live coding.
But also, someone can fail a live coding interview for reasons other than belonging to that group.
[+] [-] adregan|7 months ago|reply
I get stuck in ways I’ve never experienced in my life; my brain feels like it stopped working (but has reserved the bandwidth to shout at me the whole time “you’re blowing this!”). Then shortly after the call ends, sitting quietly, a path forward opens and suddenly I’m working on a solution, which it’s own special kind of hell—to know you could do it but didn’t and now some stranger out there thinks you’re a real idiot.
Studying/practicing has mixed results as you are taking away time from your partner and kid, this causes feelings of guilt. I found the more time I spent practicing, the more leetcode hards I solved, the worse I got. It was paradoxical! It was additionally demoralizing!
I only share all of this to say that circumstances change; you change. What once was easy becomes hard—maybe just for now, maybe forever, I don’t know. What I do know from my time at Big Co., is that this style of interview (often aped at smaller co.) is employed as one of the stated goals is to remove opportunity for biases (test everyone the same "quantifiable" way). However, my experiences lead me to believe that it falls short in that regard. As I said, people change, and a candidate who becomes a coworker with a high tolerance for stress today might not have the same appetite in a few years. Surely removing biases is about seeking balance, but I think certain types of interviews are blind to the imbalances they might cause.
[+] [-] devoutsalsa|7 months ago|reply
Let's say you need someone who can lift 10 kilograms:
- Good interview: "Please lift this 10 kilogram bucket by the handle."
- Not Good interview: "As a proxy for your overall strength, please take off your pants, squat, pick up this 1 kg bucket by clenching the handle with your butt cheeks, and then stand. We know this isn't a real test of your strength, but we want to see how you perform under pressure."
EDIT: what I mean by doing the job, I mean test the skills used on the job. See if a chef knows their way around a kitchen & actually cook something. See if a customer support rep has good written & verbal communication skills in a mock support interaction. See if a phone screening can do phone screens. Stuff like that.
[+] [-] paulcole|7 months ago|reply
"What a useful thing a pocket-map is!" I remarked.
"That's another thing we've learned from your Nation," said Mein Herr, "map-making. But we've carried it much further than you. What do you consider the largest map that would be really useful?"
"About six inches to the mile."
"Only six inches!" exclaimed Mein Herr. "We very soon got to six yards to the mile. Then we tried a hundred yards to the mile. And then came the grandest idea of all! We actually made a map of the country, on the scale of a mile to the mile!"
"Have you used it much?" I enquired.
"It has never been spread out, yet," said Mein Herr: "the farmers objected: they said it would cover the whole country, and shut out the sunlight! So we now use the country itself, as its own map, and I assure you it does nearly as well."
https://en.wikipedia.org/wiki/On_Exactitude_in_Science
[+] [-] monkeyelite|7 months ago|reply
[+] [-] zdragnar|7 months ago|reply
Careful there- I believe a number of jurisdictions will consider using someone's work before you've hired them to be very illegal.
You could take this to the logical extreme and just not hire anyone, instead building your entire product off of the work done in interviews. Many would consider this to be a form of wage theft.
[+] [-] aspbee555|7 months ago|reply
bonus is you get to try different jobs, don't like it? you know you can get another one to try out easily. employer also gets to try different candidates easily with little vested resources. able to find canidtes that actually/really like/enjoy the job, better productivity
(of course this is not compatible with employer based healthcare)
[+] [-] rockemsockem|7 months ago|reply
Much better to just have a live coding test that tries to measure your communication, effectiveness at working on a problem, and your raw coding skills.
[+] [-] tediousgraffit1|7 months ago|reply
(post made me lol, thanks)
[+] [-] paxys|7 months ago|reply
[+] [-] hcfman|7 months ago|reply
I once went to a job interview with Google. I built one of the first local (Global to the Netherlands) search engines of the Netherlands, but the guy in the cowboy hat at Google asked me to write a binary search with a marker and a whiteboard. I never write with my hand, I always use keyboards. Plus I'm being insulted to write a binary search when I designed and build a search index and retrieval engine.
[I did the binary search but was not happy with the whole process that did not want to even look at what you had actually done before, because that would take away the baseline they wanted].
I guess they must have been looking for cowboys. Tip for interviews, take your cowboy hat, just in case..
[+] [-] gwbas1c|7 months ago|reply
Why?
Purely being a "good programmer" isn't just enough for the job. It's important that we can develop a good working rapport, and communicate.
For example: Someone who's an expert programmer might misinterpret a discussion and build the wrong thing; or someone might be a great programmer but otherwise be impossible to actually conduct a technical discussion.
IE, this is why "Fizzbuz"-style questions work so well: It's not really screening that you're a great programmer, it's screening your ability to have a technical discussion.
[+] [-] dkarl|7 months ago|reply
Younger engineers trying to find jobs in the industry should consider how much the other parts of the hiring process privilege older engineers regardless of ability. I'm sure many of the people I worked with who lacked basic coding competence twenty years ago still lack competence and are employed right now in jobs that require it, jobs that could go to younger, more capable people except that their resume, their experience, and their ability to say the right-sounding thing don't match up.
Hiring someone to write code without ever seeing them do it is like hiring a guitar player for a band by verifying their MFA in music theory, talking to them about music for an hour, and checking their references. All that is fine, but can they play the guitar? You can acquire everything else, the lingo, the knowledge of music theory, the pros and cons of competing techniques, the familiarity with current and past great performers and performances, without ever putting your hands on the instrument. And some people are built like that. Some people are guitar connoisseurs: they gave up their hopes of playing the instrument long ago, but they're huge fans and nerd out over the art and science of it. If those folks needed to be in a band to have health insurance, and they didn't have to audition, you can bet some of them could talk their way into it.
Finally, when a team interviews someone for a position that requires writing code, they are inevitably making a judgment about whether they think they can do it. If you ask someone to guess without any direct evidence, you're inviting them, almost forcing them, to fill the gap with bias. Do they collect subtle little hints and make fragile deductions like they're Sherlock Holmes? Do they trust their instincts about whether the person in front of them looks and acts like the kind of person who could code? Either one is guaranteed to be rife with bias and problematic preconceptions.
[+] [-] ajabhish|7 months ago|reply
One thing that’s helped me, as both an occasional interviewer and a frequent interviewee, is recreating the stress loop on my own terms. Friends rarely push hard enough, and a paid coach is $150+ an hour. Lately I’ve been working on a little side-project Tough Tongue AI: a voice-driven agent that drops you into a live code editor, throws follow-up questions, even interrupts when you go off-road, then gives a feedback at end. It’s not magic, but after a few sessions the “somebody’s watching me” adrenaline spike starts to feel familiar.
If live coding interviews are here to stay, a repeatable way to train the physiological side of the test, not just the algorithms would be helpful!
[+] [-] lurking_swe|7 months ago|reply
I learned this in my 20’s. I made it to about 10 different on site interviews over a summer, and was totally burnt out. I was spending hours prepping before each interview, and really putting in the effort! At the last minute a company tried to squeeze me in for an interview before a vacation i had planned. I reluctantly agreed, and i completely winged it. Zero prep, i showed up ready to (mentally) go on vacation lol. Guess what? I aced most of the interview and ended up getting an offer 3 days later.
Of all the interviews i performed the best when i went in with 0 expectations and 0 stress on myself. That’s when it clicked for me.
Easier said than done though. It’s not always easy to get into that mindset.
[+] [-] ingend88|7 months ago|reply
[+] [-] mkureth|7 months ago|reply
I have 25+ years in software engineering and have been filtered through the non-job related interview process described here. I also have a medical diagnosis of ASD L1 and ADHD along with stress induced dyslexia.
Can others here please read the following federal law and confirm it applies. Through the film, I am trying to amend the definition of a disability, so we have equal rights.
Title 42 U.S. Code § 12112(b)(6) "using qualification standards, employment tests or other selection criteria that screen out or tend to screen out an individual with a disability or a class of individuals with disabilities unless the standard, test or other selection criteria, as used by the covered entity, is shown to be job-related for the position in question and is consistent with business necessity"
[+] [-] apwheele|7 months ago|reply
I have done the entry interview as well. For example have had very good conversations with managers who were applying for senior IC roles. They then go and fail the tech interview basic "can you do write and execute a python hello world script from the command line".
[+] [-] acjohnson55|7 months ago|reply
I wrote about this here: https://acjay.com/2023/11/12/everything-all-at-once-inside-a...
There are certainly jobs where you need someone who is cool under intense pressure, where what happens inside an hour matters. That's what these interviews are revealing. But I think this is a tiny minority of dev jobs.
However, I also think when people talk about hiring practices, the elephant in the room is that the true purpose our interview processes have evolved to serve is taking the candidate pool from a lot of people to something that is decidable. We can't fully evaluate 500 candidates, but we can evaluate 5. The process of weeding out 99% is designed to be inexpensive, at the cost of accuracy. The hope is that you'll have one person squeeze through the filters that is a good fit. If that happens, then all the false negatives are tolerable.
But we really, really don't like to admit this. We tell ourselves that we are running interview processes that are predictive of performance when there is actually very little evidence to support that, and plenty of reason to doubt that it's true.
If quality were paramount, I think we'd do hiring completely differently: https://acjay.com/2019/05/16/hiring-developers-is-easy/
[+] [-] MontyCarloHall|7 months ago|reply
I once had a coding interview where I was given a laptop with no internet access and 30 minutes by myself to solve a couple questions a step or two above FizzBuzz, in pseudocode. The interviewer then came back into the room and we discussed my solutions, using them as a jumping-off point into a more open-ended discussion (e.g. "why did you use a hash table here? can you think of another approach?" which then led to a general discussion on the suitability of hash tables versus other approaches for related problems.)
It was one of the best technical interviews I've had. I'm surprised more places don't do this. It's a win-win-win: it relieves stress on the interviewee, gives the interviewer back half an hour of valuable time, and makes for a much more productive assessment of the candidate — it's a lot easier for a candidate to explain code they've already written, and thus a lot easier to an interviewer to assess the candidate's thought process.
[+] [-] DanielVZ|7 months ago|reply
[+] [-] garphunkle|7 months ago|reply
I was interested by the author's statement: "Working memory is the most reliable proxy (I know of) for fluid intelligence, your ability to reason, solve novel problems, and think abstractly." and the linked study (https://pubmed.ncbi.nlm.nih.gov/21037165/). My working memory is not so great, but it degrades less under stress.
Question worth considering for hiring managers: do you prefer stress-capable employees, or greater working-memory employees? Is my model a false dichotomy?
[+] [-] mmusc|7 months ago|reply
There was one company whose process I really appreciated:
- A take-home test, more of a puzzle-style challenge. - The interviewer does a quick review of your submission. - If they like what they see, you're invited to a follow-up session where you explain your code and make a few simple changes to it.
This approach proves that the work is genuinely yours, shows your thought process, and still gives the opportunity to demonstrate live coding on code you're already familiar with.
[+] [-] paxys|7 months ago|reply
int total = 0;
for (int i=0;i<arr.length;i++) {
}return total;
[+] [-] hibikir|7 months ago|reply
Now, that doesn't mean that stress management cannot be part of the job: I've worked in the hellish places where an on-call rotation meant at least 4 calls off-hours, and some which needed resolutions in 10 minutes or less from the pagerduty call. Someone with bad stress response is not going to be doing great at that kind of live diagnosing, with possibly many millions for the company on the line. But the number of positions like that, where you are a cross between a developer and an ER doctor are much less common than places that have none of those demands, yet give you a series of leetcode mediums and hards. They might as well be testing for height, or how much you can bench press. It filters, but it does not help.
[+] [-] michaelt|7 months ago|reply
Did y'all not have to take a load of exams in high school and college?
I'm sure there are variations between education systems, but by the time I graduated college I'd done at least a hundred high-stakes, time-pressured hour long written tests. All of them substantially more difficult than summing the even entries in a list.
Is this not what everyone experiences, in every education system?
[+] [-] radiofreeeuropa|7 months ago|reply
You know the trope of almost all students dreading going up in front of the class to solve a problem on the blackboard/whiteboard? A thing that I think I personally only actually did a countable-on-fingers number of times ever, and only in lower grades?
It's a trope for a reason, and it's because the vast majority of people do in fact hate having to get in front of an audience and perform. Hell, it's often a component of recurring nightmares.
It's like that but way worse because it's also far more high-stakes than that, the problems are far more complex, the point is explicitly for everyone in the audience to judge you and not just your assumption that they all are causing the stress, et c.
It's more like the stress from getting up at an open mic night, than any kind of stress I've ever actually encountered on the job. Even the dynamic in a meeting with clients where you have to diagram something out on a whiteboard or something is totally different—for one thing, you generally have people in there who are very much "on your side", and you don't have to worry if you misspell something that you're personally about to be/remain unemployed, or lose out on a huge raise, or whatever (nobody generally cares about minor mistakes on a whiteboard in an ordinary context, and maybe they don't in some interviews, but they do in others, and candidates worry they might in all of them).
[+] [-] guhcampos|7 months ago|reply
I've always finished exams absurdly quickly. I used to finish 60 min exams in like 20 min and sleep the rest of the time when the professor didn't allow us to leave, because I had likely spent the previous night drinking and partying my way through college. I'd usually ace most of them.
Thing is, at those times we were working with freshly acquired knowledge that you've been practicing a lot. That's not the case with most leetcode interviews. As a senior/staff engineer, I'm not using double pointers to perform little math tricks on a daily basis, I'm troubleshooting cascading timeouts on distributed systems. I'm not worried about how fast I'm going to iterate over a batch of a couple thousand records on a list I'm worried about how much latency I'll accrue if I rely on a primary source of truth instead of hitting a cached answer.
Code interviews don't measure experience or competence. I don't even think they measure stress as the article mentions. To me, they just measure how much leetcode you practiced for that interview. Nothing more.
[+] [-] Palomides|7 months ago|reply
[+] [-] vidarh|7 months ago|reply
I've had only two oral exams, and they were awful, but even then they were far more conversational in nature, as opposed to a coding interview.
I do well on written tests.
I also do well in interviews, but I detest having people look over my shoulder while I code, and it absolutely tanks my performance. I only pass them because I'm experienced enough that I can be stressed out like crazy and perform really horribly compared to what I normally do and still do okay.
I've seen fantastic coders totally freeze up and be unable to do anything at all when being asked to write code in front of people, even though they have no problem talking through problem solving.
And I'm similar to that. I've held speeches in front of thousands, been on TV, held plenty of presentations, and can talk the ears of an interviewer with passion about complicated technical concepts, but have me write code while you watch, and I'd frankly much rather have a root canal treatment (I've been known to fall asleep during those).