I’ve found this HN comment to be the single best explanation of the purpose of Kafka for engineers (like me) who never truly appreciated why you’d want it in the first place: https://news.ycombinator.com/item?id=35160555
Wow that’s such an amazing comment. And explains very well what kafka does and why it’s valuable from a technical perspective, but I wanted to add why it’s valuable from “corporate political” perspective. This is how I usually “sell” kafka to new recruits.
Imagine a big corporation - lots and lots of various teams all doing their own thing - business processes, auth, customer admin etc.
When all those micro or macro services want to talk to each other, you definitely don’t want them to all dip their fingers into one large shared database - you want some contractual guarantees between them - graphql / rest with openapi, that kind of thing. And it all works up to a certain scale.
But now imagine this intricate web of network requests … and then one service fails… This can cause a cascade that’s really hard to track.
Or … you have one very reliable service being used by various clients, but suddenly there is one more client that sends 10 times more requests than all the other clients combined - happens all the time in large corps.
And sure there are ways to mitigate both with technical or policy ways, but what kafka offers is a single, ready made solution for all of it.
If the data flowing through a corp does so through kafka, you end up with very strong contractual guarantees about shape of the data both for consumers and producer. Scaling your services becomes mostly a solved problem. Producers of data don’t care too much who or how many consumers they have handling 10x, 100x scale changes becomes if not trivial, at least fairly easy.
What that means in practice is you trade every team being experts in scale and deployment of their services, to needing one really good team to shepherd your kafka cluster and the rest don’t need to deal with it too much.
But hosting kafka at scale is quite tricky - haven’t done it myself, but knew the team that handled it at my previous org and those were top notch guys that still struggled with various things.
Anyway, kafka kinda allows for micro services to scale at big orgs, and that to me is just amazing.
[+] [-] dang|2 years ago|reply
I wrote a children's book / illustrated guide to Apache Kafka - https://news.ycombinator.com/item?id=27541339 - June 2021 (226 comments)
(Reposts are fine after a year or so and links to past threads are just to satisfy extra-curious readers!)
[+] [-] warrenm|2 years ago|reply
[+] [-] ramraj07|2 years ago|reply
[+] [-] seer|2 years ago|reply
Imagine a big corporation - lots and lots of various teams all doing their own thing - business processes, auth, customer admin etc.
When all those micro or macro services want to talk to each other, you definitely don’t want them to all dip their fingers into one large shared database - you want some contractual guarantees between them - graphql / rest with openapi, that kind of thing. And it all works up to a certain scale.
But now imagine this intricate web of network requests … and then one service fails… This can cause a cascade that’s really hard to track.
Or … you have one very reliable service being used by various clients, but suddenly there is one more client that sends 10 times more requests than all the other clients combined - happens all the time in large corps.
And sure there are ways to mitigate both with technical or policy ways, but what kafka offers is a single, ready made solution for all of it.
If the data flowing through a corp does so through kafka, you end up with very strong contractual guarantees about shape of the data both for consumers and producer. Scaling your services becomes mostly a solved problem. Producers of data don’t care too much who or how many consumers they have handling 10x, 100x scale changes becomes if not trivial, at least fairly easy.
What that means in practice is you trade every team being experts in scale and deployment of their services, to needing one really good team to shepherd your kafka cluster and the rest don’t need to deal with it too much.
But hosting kafka at scale is quite tricky - haven’t done it myself, but knew the team that handled it at my previous org and those were top notch guys that still struggled with various things.
Anyway, kafka kinda allows for micro services to scale at big orgs, and that to me is just amazing.
[+] [-] rad_gruchalski|2 years ago|reply
[+] [-] candiddevmike|2 years ago|reply
> Starts talking about scaling via partitions
I think this book could've been shortened or put more details around message creation and retrieval.
[+] [-] SQueeeeeL|2 years ago|reply
[+] [-] AugustoCAS|2 years ago|reply
The author forgot to say that teenage otters set `acks=0` when producing messages :D.
[+] [-] xhkkffbf|2 years ago|reply
[+] [-] deserialized|2 years ago|reply
[+] [-] thimkerbell|2 years ago|reply