WingNews logo WingNews GitHub [2]
top | item 32014658

Ask HN: Resources on learning System Design (back end/data engineering)?

116 points| LewisVerstappen | 3 years ago | reply

I'm looking for resources/tips on how to learn concepts in system design. Things around backend engineering and all the various components that go into building large scale systems (load balancing, message queues, CDNs, etc.). I'd also like to learn concepts around data like database sharding, non-relational database models, streaming/batch (lambda architecture, kappa architecture, etc.) and more.

In terms of what I've done so far...

I've read

- Designing Data Intensive Applications (https://dataintensive.net/) - This book has been recommended a ton on HN/twitter/reddit and I found it extremely useful to get a better idea of how data systems work, different types (OLAP vs. OLTP, data warehouse vs. lake, etc.) and get a better understanding of distributed systems.

- Quastor (https://www.quastor.org/) - This is a newsletter that sends out summaries of system design blog posts from Big Tech engineering blogs. It's been helpful to me for getting an understanding of the different components involved in the tech stacks at various companies and how companies think about building scalable systems.

- MIT Distributed Systems (https://www.youtube.com/playlist?list=PLrw6a1wE39_tb2fErI4-W...) - This is a series of lectures by Robert Morris (co-founder of YC) on distributed systems and their properties. The lectures pick a specific tool/technology (Google File System, ZooKeeper, Apache Spark, etc.) and then discusses it. I've really enjoyed reading the papers and watching the lectures.

I'd really love any recommendations from the HN community on how else I should be learning about system design.

Thank you very much!

19 comments

order
[+] eveningtree|3 years ago|reply
Beyond the theory, if you are having trouble getting your hands dirty, here is the path I had taken for data engineering:

(It is pretty hands on.)

After you are done with the initial learning, find an academic machine learning discord or something similar. There will always be people there who will be very happy to find someone to clean their data. It's a great way of getting hands on with data engineering.

System design is best learnt through fires.

A good angle of attack is: pick a certification like AWS SAA or equivalent(AWS, azure, Google cloud. Doesn't really matter. Just pick a mid level certification). Then do the labs. They will quickly point out the holes in your knowledge/understanding. The free tier will take care of your needs, and cloud providers most of the time forgive the first surprise bill(it happens to everyone).

Soon data engineering and systems design merge anyways. This path is like a cheat-code for forcing convergence. Otherwise, the mind will quickly forget most theory.

---

Adding to this:

When you get hands on, you will find how the mind lies to you about how much you know. Mind maps are a nice way to reliably detect holes in understanding. Sit down with paper & pen once a week or so, and make a big map of everything you know.

[+] hardware2win|3 years ago|reply
But why data engineering here?
[+] LewisVerstappen|3 years ago|reply
Really fantastic advice. Thank you very much!
[+] smoe|3 years ago|reply
I like the series "Software Architecture Monday" by Mark Richards

https://www.youtube.com/playlist?list=PLdsOZAx8I5umhnn5LLTNJ...

Currently 140 videos, each a 5 to 15 minutes bite sized introduction to different software architecture topics from specific patterns to more conceptual and management things. I especially like, that it is pretty much completely free from the usual YouTube clickbait nonsense ala "If you DONT use X NOW, you are DOOMED!!!"

[+] oxff|3 years ago|reply
You've read about most of the concepts if you've gone through that material.

> How else I should be learning about system design.

I'd recommend putting that reading to use and make some kind of an architecture astronaut system with it to see if you truly understand it.

[+] vp8989|3 years ago|reply
DDIA is a good book and I've read it at least 3 times, but there is no substitute for real world experience.

I also think a lot of system design material is oddly positioned in that it caters towards synthetic interview scenarios where you are designing mega scale systems but somehow are also the single person responsible for all of load balancing, CDNs, message queuing, databases etc ...

In the real world it doesn't really work for that way for product engineers.

It's good to know about that stuff but there's lots of other topics that would have higher ROI like learning how to plan and sequence work properly, how to reduce different kinds of risk during build and deploy, how to migrate complex, existing systems towards better architectures without disrupting the business, how to actually become someone who gets to design systems ...

[+] iman453|3 years ago|reply
Are there any resources you’d recommend for learning the high ROI items you mentioned?
[+] ivanmontillam|3 years ago|reply
I find the book System Design Interview by Alex Xu very helpful.

It's focused on passing System Design Interviews, but I found it to be a great primer on System Design in general.

He also has a new website to teach you System Design Fundamentals at ByteByteGo[0].

--

[0]: https://bytebytego.com/

[+] mabbo|3 years ago|reply
I think the next step after reading that material is to just get experience in it.

Consider applying to some software companies that are massive in scale. AWS, GCloud, Azure, Netflix. People whose systems need good design.

Once you get into a place like that, go after positions on teams that own huge systems, and get mentorship.