top | item 34999464

A senior engineer's guide to the system design interview

909 points| leeny | 3 years ago |interviewing.io

376 comments

order
[+] rednerrus|3 years ago|reply
We've gone away from system design interview questions on my team. We ask people to diagram something technical they understand well and the team digs in and asks questions to understand depth and breadth of the candidates understanding. For us it works much better. It's a chance to see how well candidates do in following instructions. It gives you a chance to explore depth and breadth of their knowledge on something technical that they claim to understand. Our philosophy is how well you understand something you claim to know well is indicative of the depth and breadth of your technical knowledge in general. There are plenty of opportunities in this to ask about design and get to know how people think when it comes to design. I think it helps to eliminate false negatives and false positives.
[+] azornathogron|3 years ago|reply
I like this... in theory. But I don't think I'm allowed to give you a system diagram and detailed explanation of systems I've worked on, because obviously they're proprietary. That seems like a problem.

How do your candidates normally work around this? Does everyone just talk about hobby or open-source systems they've worked on?

[+] spike021|3 years ago|reply
One place I interviewed with last year did something similar in that it was more of a BYOSD (bring your own system design). So it meant I got a chance to think through complex systems I've worked on, mock up a diagram beforehand, and then present it to the interviewer. They then drilled down on a lot of components, why decisions may have been made, etc. Sort of like what you're saying.

Out of all the similar interviews I did last year for that type of round, I enjoyed that style the most. Rather than your typical "build me an {ecommerce site, social network, video streaming site, url shortener}".

[+] angarg12|3 years ago|reply
I work for a big tech co, and our interview training explicitly says to not ask candidates about systems they have built in the past, but to ask them to build brand new systems from scratch.

This seems completely backwards to me. Is like saying "hey, do you know that relevant on the job experience that you have? we don't want to hear anything about it, here you have a made up scenario".

Ok ok, that is a bit cynical. Asking to design novel systems can give you some good insights about the candidate, or help with people who don't have experience building systems yet. Still it seems to me that asking candidates to describe real world systems that they have actually built is much more useful at checking their skills than building imaginary systems.

One argument against this is that candidates can "cheat" by preparing in depth a made up example. But how is that much different than the current approach?

[+] bluetomcat|3 years ago|reply
Few people at software companies are designing systems from scratch nowadays. If they are, they'll be gluing together third-party middleware applications like database systems, queues, proxies, load balancers, etc.

A general understanding of computer architecture, programming language theory, networking and common protocols and formats is more valuable. At that fundamental level, they won't be using recent fancy buzzwords to sound cool, but could be using their brain to come up with something original.

[+] jason-phillips|3 years ago|reply
> We ask people to [discuss] something technical they understand well and the team digs in and asks questions to understand depth and breadth of the candidates understanding.

This is how I conduct all of my interviews. I can't stand the gotcha-centric, pitfall-laden Jeopardy contest format of interview interrogations.

[+] onlyrealcuzzo|3 years ago|reply
> It gives you a chance to explore depth and breadth of their knowledge on something technical that they claim to understand.

Assuming you know enough about what they're doing to actually dig in - which, granted, maybe you're only considering interviewing people who are working things you're somewhat familiar with.

Ultimately, I don't see how this is different from a technical design interview - in that you can be susceptible to hiring charlatans, and pass on people who'd rather say they don't know something than try to BS most of it on the spot to sound impressive.

The two people could have the same knowledge. You're more likely to hire the charlatan. Maybe that's what you want :shrug:

[+] gautamdivgi|3 years ago|reply
You’re ruling out all candidates who work on classified systems and those under an NDA
[+] mdm12|3 years ago|reply
I have seen this process described elsewhere as 'reverse system design', and it is my preferred approach to evaluating senior candidates as well.
[+] vsareto|3 years ago|reply
Feels like an Architect interview more than a Senior interview though
[+] slytherone|3 years ago|reply
Hey, Creator of the guide here

Reverse system design interviews have a a lot of untapped potential. They just started getting a bit more popular in the last year or so (however, most companies haven't caught on to the trend yet.)

You're one of the trendsetters @rednerrus

[+] emodendroket|3 years ago|reply
To be honest, since I have to go through preparing for the standard style of interview before a job search anyway, all these novel forms seem like annoyances.
[+] Jiocus|3 years ago|reply
This sounds like an enjoyable interview, even from the applicants point of view.

Do you follow up on your decisions to see how it held up regarding false positives you did accept, if any? I mean as related to the heuristics, not performance review generally.

[+] xen2xen1|3 years ago|reply
When doing IT people often say "Well, I don't know anything about computers", which I generally found pointless, as with about 30 seconds of talking I can tell if you know anything or not. Your story passes my smell test, as getting someone talking is 90% of telling what they know.
[+] ramraj07|3 years ago|reply
How many rounds of interviews do you typically do? What other interviews if any? Curious what good orgs are doing in this regard!
[+] bosch_mind|3 years ago|reply
This is pretty cool. I’d probably have fun walking through some systems I’ve designed
[+] bhvangoo|3 years ago|reply
hmm interesting approach, really like it. Where do you work? and are you hiring now?
[+] zabzonk|3 years ago|reply
best way of doing interviews imho. apart from anything else, it gives you a chance to show that you know how to design/implement a system that is perhaps more complex than the one you are being interviewed for
[+] jstx1|3 years ago|reply
Still sounds like a system design interview to me, just a bit less structured.
[+] posharma|3 years ago|reply
I don't know if you work in FAANG or no, but most FAANG interviews just don't work this way. So, as much as I like this style, it's just 1 company and definitely not the norm. The rest of us are stuck with design Uber, Whatsapp, crap...
[+] jahewson|3 years ago|reply
I don’t like this approach because it doesn’t provide measurable or repeatable criteria for comparing candidates. It also suffers from a sort of Gell-Mann amnesia where a candidate who spouts good-sounding BS about a topic you’re not familiar with is assumed to be competent across the board.
[+] mgraczyk|3 years ago|reply
I am on the interviewer side of ~1 system design interview per week and have done probably around 100 of them at Google and other companies.

Most of the advice here is good, although it could be condensed quite a bit.

The biggest red flag for me as an interviewer is when candidates list off concepts or technologies they don't understand. DO NOT say things like "we can't do this because of the CAP theorem" or "we should add a cache to reduce latency" unless what you are saying is true and you can explain why this is true (the guide mentions this).

The biggest green flag is when the candidate obviously has thought deeply about related problems and made design decisions based on real world constraints and requirements. If you bring up a lot of related things you have worked on, explain why you made the decisions you did in those scenarios, if its obviously related to the question, and if you don't spend too much doing it, then you will definitely get a positive recommendation from me.

The purpose of the system design interview is to convince me that you could build, or lead a team to build, the thing that we are discussing at a company like the one where you are interviewing. I feel like I can easily see through interview practice or coaching, so I don't recommend spending more than an hour or so "preparing" for system design interviews. Unless you are interviewing somewhere with very inexperienced interviewers...

[+] omniglottal|3 years ago|reply
I got such fatigue from scrolling through so very much prologue, introduction, expectation-setting, and general "here is what we are going to say and here is how you may expect we will say it" fluff that I gave up, pages in, before getting to any apparent content.
[+] brunooliv|3 years ago|reply
I will read this as it seems really good. However, it always feels like a catch-22 problem to me, in the sense that, if you've never worked at a large company before, you can't claim to have truly used or understood large scale systems properly to work on them or design them in production.

I got rejected from a large tech company after failing system design and this was exactly their reasoning: "you seem to know the concepts and the theory really well, but, it was obvious to the interviewers that you lack the real world experience and hiring you would be a risk".

Obviously I... knew or at least tried to know the theory because I PREPARED. I read books, articles, practiced mock problems, watched YouTube for a few weeks, etc, etc. I just tried to "drill" the knowledge that was not there into my brain for the purpose of passing the interview. I guess that's what a lot of people do.

So yeah, it left me really down for a few months and feeling bad about myself and like the world sucks. So, how are you all out there "faking it till you make it" in a convincing way?

Because I bet that no engineers at a FAANG will alone design a full scale system on their own ever, so I doubt the usefulness of the "signal" these interviews give, but, ofc, that's just me, someone who prepared for months and failed

[+] lampshades|3 years ago|reply
I took a downlevel to get my foot in the door.
[+] John23832|3 years ago|reply
I think to be frank, this gamification of interviews is BS.

I’ve built actual production systems at scale. The fact that I need to follow a guide to “demonstrate” my ability to someone who 9/10 hasn’t built (or couldn’t build) anything at scale in production shows where we are in the absurdity matrix.

[+] eternalban|3 years ago|reply
"You can pass system design interviews even if you’ve never designed distributed systems before. If you have copied files between machines with drag-and-drop, you are halfway there. If you implemented clients or servers or have opened network connections, you’ve got this. This guide will teach you the most important 20% of information that will appear 80% of the time in system design interviews. By the end of this guide you won’t be an expert, but you’ll be well on your way to being a better engineer and a much better interview candidate."

A better engineer, no less! Wow, this must be some magical guide. I have to read it now. Also what does this say about our industry. "You want to be a surgeon but don't have the experience? Ever cut a tomato? You're halfway there. If you've ever made a sandwich, you've got this. Nurse!"

[+] thdc|3 years ago|reply
A senior interviewer's guide to the system design interview*.

I've speed read over the existing two sections and while it's informative, I'd argue that the maybe 10+/12 of the core design concepts are things that someone with a formal CS education should have picked up - you don't need to be a senior software engineer.

The barrier is knowing how these things can be pieced together to make a system from scratch which is the more difficult part as it is a much rarer occurrence in most people's experiences. Even seniors may not have much experience in setting up large scale systems (depending on your definition of senior), so at the end of the day anyone that studied or memorized the material is good enough to pass - practical experience or not.

I'd much rather have a high level view of an existing or theoretical system, be provided with some issue that occurs, and be asked for ways to diagnose and remediate said issue. Forget the dance around setting the system up. This is similar to the practice of providing existing code in an interview, describing a bug with the code, and watching the interviewee debug and fix it - but with systems. It mirrors actual work more closely.

[+] debacle|3 years ago|reply
I remember once I was interviewing for a senior position where the ask was something like "design a general purpose system for a REST layer on top of an ORM."

I had just finished implementing an OSS solution that did exactly that, including some upstream changes we made to improve the system, so I walked through exactly what we did, challenges we faced, etc.

I walked out of the interview feeling as though the interviewers and I didn't speak the same language; they might've said the same. Not only did I not get the job, I never got another communication from the company.

To me, senior level interviewing, especially at smaller companies, is fraught. A full 1/3rd of the companies I've interviewed at would not have been a good "culture fit," and I'm a picky interviewer. I expect, like coding tests etc, these interviews are not aligned with the real travails of the position - growing talent, managing time, triaging, and ensuring stability/capability.

[+] Aperocky|3 years ago|reply
The interviewer usually can only rise to the level of themselves, especially when assessing skills.

System design is hard, necessary skills can only be earned, and then there are many avenue down to bad system design masquerading as good that people who implemented it just don't realize.

It takes a combination of humility and a degree of the skill in question to recognize the interviewee is more proficient at the said skill, and many are lacking in either or both.

[+] theteapot|3 years ago|reply

    .----------.  .------.  .-------.
    | REST API |->| ORM  |->| RDMS  |
    '----------'  '------'  '-------'
Too easy. Next question.
[+] nfw2|3 years ago|reply
I’ve got an interview coming up for a senior role on a UX engineering team (design system, component libraries, etc). The interview is just the standard set of backend system design and leetcode problems.

It’s a pretty big red flag to me that the skills being evaluated are so tangential to the actual job. It doesn’t give me any confidence that the people I will be working with have the skillset I consider important, or that the team and I will approach problems in a compatible way.

[+] hot_gril|3 years ago|reply
> on top of an ORM

My answer would be no.

[+] grufus|3 years ago|reply
Issue with system design is that our industry is so highly opinionated about things. You design something to the best of your abilities, it works (though may have its pros and cons) and then you go present it to a crowd of engineers (HN, say) ... and you get destroyed. There will always be people telling you that these were dumb choices and that that's obvious. And if you go and do what they say is the obviously better solution, then the exact same thing will happen with another crowd.

Our field is too immature to actually agree on things being good or bad.

[+] ed25519FUUU|3 years ago|reply
> Interviewers want to engage you in a back-and-forth conversation about problem constraints and parameters, so avoid making assumptions about the prompt.

As someone who has conducted countless interviews at a FANG, I can not stress this point enough. The ability to just talk about a problem makes a big difference between a mediocre interview and a great interview. Besides, it's what makes an interview fun or lame.

[+] dakiol|3 years ago|reply
Problem is: as candidate you don't know if "engaging in a back-and-forth conversation about problem constraints" is something the interviewers want. Some interviewers want it, but it's not always like that. And, no, usually discarding an entire company just because the interviewer on duty doesn't like "engaging" is BS.
[+] gwbas1c|3 years ago|reply
Are these interviews turning into a "can you regurgitate some memorized information" formality?

As an interviewer: This is always a red flag to me, because being a successful software engineer is much more than memorizing common truisms. (I spend a lot of time cleaning up misinterpreted "truisms.")

As a candidate: If I'm expected to regurgitate truisms, it's a flag that trying to do things "right" will be opposed by people who don't understand how computers / information work; and a lot of friction will come from trying to make something work versus make something fit an inappropriate ideal.

[+] FlyingSnake|3 years ago|reply
“When a measure becomes a target, it ceases to be a good measure”.

The interviewing cottage industry has become a sad joke at this point.

[+] ZephyrBlu|3 years ago|reply
I do believe this is probably useful for people interviewing, but on the other hand it reads like mostly complete bullshit and largely social signalling rather than measuring any sort of skills.

The fact that you can be coached to pass these interviews with zero practical experience designing systems (Even toy ones or hobby projects) shows that there is very little technical skill involved.

Someone who has designed a real production system is still likely to fail these interviews unless they understand the social nuances that interviewers are looking for in a system design round.

[+] nitwit005|3 years ago|reply
This is, unfortunately, painting an overly rosy view of the interviewer.

A lot of the time your interviewer is just going to be some senior person with an interview guide. They'll be asking you to design Netflix, without any experience with video or streaming themselves.

Rather than "trying crazy stuff", I've found that an important step is asking a few questions early in the interview to see if they understand relevant concepts. You can't take it for granted.

[+] ldjkfkdsjnv|3 years ago|reply
Yeah they want a canned answer, like they want some standard algorithm for leetcode. You need to be as standard as possible so they can check their boxes
[+] paxys|3 years ago|reply
Interviewing is a two way street. You can judge the company as much as they judge you. If you find yourself in front of such an interviewer then move on somewhere else.
[+] tootie|3 years ago|reply
I think it's funny how they say you don't need experience to pass one of the interviews. Last time I went in to one of these as an experienced engineer I probably said all sorts of things they didn't want to hear like there's no point building a scalable system until you're sure your product has traction. That and I kept trying to extract imaginary requirements. I still have no idea what they wanted to hear so I just name-dropped architecture paradigms. Didn't get the job.
[+] jefQuery|3 years ago|reply
I read all 4 parts in the pre-released version on the interviewing.io discord. Here's my feedback:

Loved how much they go into the actual interview dynamics and phrases to say, where as other resources wave away interacting with the interviewer as an implementation detail. Haven't seen that anywhere else on the web and it's super helpful. The framework is also much more fleshed out than, for instance, the HiredInTech system design guide.

However, the 12 tech ideas section was a little too dumbed down for me, though it might be helpful for someone with less experience. I also noticed a few typos (MangoDB, cache vs hash) and told them, and they said they'll fix it in the next version of the guide.

3.5/4 stars; would read again

[+] lazyant|3 years ago|reply
Quick thoughts:

  - System design interview changes depending on the company. At Facebook the interviewer said one sentence in the whole interview to me, no feedback or anything, so be prepared for nasty ones. At Google they hate if you use a particular product to solve your problem and everything needs to be backed up by numbers; they'll ask you how many servers you'd need for your solution ("bill of materials").   
  - Expanding on last comment, it's common to see in books and courses a basic math of number users -> estimate usage -> requests per second and storage needs but I've never seen anyone take one estimates about servers by computing processing (CPU/RAM) needed.  
  - This guide, like almost all resources in particular takes on the most common case of web design: client-server request/response (plus queuing). There are other completely different paradigms that show up less frequently and are harder to have experience on, like streaming and batch processing.  
  - As all guides given space/time limitations, some key concepts are hand-waved with a recipe-based approach.
[+] strus|3 years ago|reply
I develop desktop and embedded applications, so do a different work than distributed systems engineers, but I wonder - do you often build systems from scratch and therefore need to have all this knowledge?

Because in every job I've worked in the past 10 years, I landed in an established project, that was developed years ago, and my work was to maintanace and add new features. Most of these projects were too big and complicated for one person to know how they work in every detail - even the ones that were there for the very beginning always said something like "after all these years I don't know how half of the things are implemented".

So, from my perspective, an interview question where someone asks me to design a complex system entirely on my own, in details, is just stupid, as I am pretty sure I will never do anything like that in my life. But maybe webdev is a different story?

[+] shahsyed|3 years ago|reply
I've only read up to Part Two so far but find the guide is extremely helpful and resonates with other resources (i.e Grokking the System Design Interview) and with my own experiences for what I believe is the very broken interview process that we face today.

Highly recommend. Interviewing.io is one of the best resources for those doing prep today and the team behind it is awesome.

[+] joshSzep|3 years ago|reply
I love the content that interviewing.io puts in this category on their YouTube channel. What was especially eye opening is seeing two senior engineers that normally conduct systems design interviews taking the driver's seat. https://youtu.be/Zi0pPkiFemE

TLDW; they struggle! big time!