How Netflix Accurately Attributes eBPF Flow Logs

[nikolay_sivko]

At Coroot, we solve the same problem, but in a slightly different way. The traffic source is always a container (Kubernetes pod, systemd slice, etc.). The destination is initially identified as an IP:PORT pair, which, in the case of Kubernetes services, is often not the final destination. To address this, our agent also determines the actual destination by accessing the conntrack table at the eBPF level. Then, at the UI level, we match the actual destination with metadata about TCP listening sockets, effectively converting raw connections into container-to-container communications.

The agent repo: https://github.com/coroot/coroot-node-agent

[slt2021]

Question to the Netflix folks: I saw a lot of in-house developed tools being quoted, do you guys have service mesh like linkerd ?

Have you guys evaluated vendors like Kentik?

I would love to get more insight into what do you guys actually do with flow logs? for example if I store 1 TB of flow logs, what value can I actually derive from them that justify the cost of collection, processing, and storage.

[retiredpapaya]

I think Netflix does use an Envoy-based Service Mesh [1], and they roll their own control plane.

https://netflixtechblog.com/zero-configuration-service-mesh-...

[madduci]

Exactly my thought. Maybe it's the "not invented here" syndrome?

We use Istio as Service Mesh and get the same result, using the same architecture as shown in the blog post (especially the part where each workload has a sidecar container running Flow).

[__turbobrew__]

Maybe Im missing something but can’t you run workloads in separate network namespaces and then attach a bpf probe to the veth interface in the namespace? At that point you know all flows on that veth are from a specific workload as long as you keep track of what is running in which network namespaces?

I wonder if it is possible with ipv6 to never (or you roll through the addresses so reuse is temporally distant) re use addresses which removes the problems with staleness and false attribution.

[VaiTheJice]

I think thats pretty reasonable tbf and probably at a more 'simpler' scale and i use simple loosely because Netflix’s container runtime is Titus, which is more bare metal oriented than, say, Kubernetes. It doesn’t always isolate workloads as cleanly in separate netns per container, especially for network optimisation purposes like IPv6-to-IPv4 sharing.

"I wonder if it is possible with ipv6 to never... re use addresses which removes the problems with staleness and false attribution."

Most VPCs (also AWS) don’t currently support "true" IPv6 scaleout behavior. Buttt!! if IPs were truly immutable and unique per workload, attribution becomes trivial. It’s just not yet realistic... maybe something to explore with the lads?

[thewisenerd]

so they didn't want to pay for AWS CloudWatch [1]; decided to roll their in-house network flow log collection; and had to re-implement attribution?

i wonder how many hundreds of thousands of dollars network flow logs cost them; obviously at some point it is going to be cheaper to re-implement monitoring in-house.

[1]: https://youtu.be/8C9xNVYbCVk?feature=shared&t=1685

[Hikikomori]

Because vanilla flowlogs that you get from VPC/TGW are nearly useless outside the most basic use cases. All you get is how many bytes and which tcp flags were seen per connection per 10 minutes. Then you need to attribute ip addresses to actual resources yourself separately, which isn't simple when you have containers or k8s service networking.

Doing it with eBPF on end hosts you can get the same data, but you can attribute it directly as you know which container it originates from, snoop dns, then you can get extremely useful metrics like per tcp connection ack delay and retransmissions, etc.

AWS recently released Cloudwatch Network Monitoring that also uses an agent with eBPF, but its almost like a children's toy compared to something like Datadog NPM. I was working on a solution similar to Netflix's when NPM was released, was no point after that.

[DadBase]

I recall a time when we managed network flows by manually parsing /proc/net/tcp and correlating PIDs with netstat outputs. eBPF? Sounds like a fancy way to avoid good old-fashioned elbow grease.

[zX41ZdbW]

If you are interested in network monitoring in Kubernetes, it's worth looking at Kubenetmon: https://github.com/ClickHouse/kubenetmon - an open-source eBPF-based implementation from ClickHouse.

[thewisenerd]

i mean.. from your blog post linked in the repo; this isn't eBPF based?

https://clickhouse.com/blog/kubenetmon-open-sourced

the data collection method says: "conntrack with nf_conntrack_acct"

[philsnow]

Is it necessary to rely on ip address attribution? If FlowExporter uses ebpf and tcp tracepoints, could each workloads be placed in its own cgroup and could FlowExporter directly introspect which cgroup (and thus, workload) a given tcp socket event should be attributed to?

[nptr]

That may help identify the local IPs but not the remote IPs.

[meltyness]

I wonder how much of Netflix infra is on AWS. Feels like building a castle on someone else's kingdom at that scale; in light of the Prime Video investment, and I guess twitch too.

[r3trohack3r]

Netflix serves nearly all of its video from a server down the street from you via its OpenConnect infrastructure. AWS only hosts its microservice graph that does stuff like determining which videos and qualities you should be offered.

That being said, its core product has been nearly comoditized. When Netflix entered the market, delivering long form high quality video over the public internet was nascent. Now everyone and their grandma can spin up a video streaming service from a vendor.

[ilrwbwrkhv]

And more importantly none of this is required. Pornhub pushes more video data on far more unreliable connections without any of this madness. This is purely a play to get the tech company multiple. Nothing else. This is Wework style coverup.

[mmckeen]

https://retina.sh/ is a similar open source tool for Kubernetes.

It's early and has some bugs but seems promising.

[roboben]

Tried it, had some issue, opened a bug report, no response. I think it is dead.

[r3tr0]

we are working on a similar product that is eBPF powered and can extract flow logs:

https://yeet.cx

[nimbius]

i refuse to believe a company that wasted $320 million dollars on "the electric state" could ever manage to do anything correctly. stripe the parking lot? stock the breakroom? clean the toilets? simply not possible.

[autoexec]

Badly managed companies often hire some very good talent which can allow them to sometimes do very impressive things in spite of themselves.

[jandrese]

The scriptwriters aren't managing the network flows across the backend of their infrastructure. Netflix's trouble with scripts doesn't affect their ability to move bits around.

[ZeWaka]

Beautiful art and book, but what a unfaithful travesty of a production that absolutely trodded on the original work.

[butlike]

All that logging and they cant figure out why people are going to other streaming services

[ASalazarMX]

Despite their awful UX, I'm always impressed with how reliable their service is, technically speaking. Video is always good and responsive even on less-than-stellar connections, you can leave a show paused for hours, and resume it almost instantly. Their fast.com speed test is always much faster than your regular internet access, I guess thanks to their Open Connect Appliances.

It must be great to work for them in infrastructure and backends.

[neogodless]

I mean, I haven't been a subscriber for a couple years but are they losing a lot of subscribers?

https://backlinko.com/netflix-users

I see some slowing growth particularly across 2021/2022 but as of this report (April 2024) they were still growing through 2023.

260M subscribers. They aren't exactly hurting.

[blinded]

lol wat?

They make 10 billion+ a <b>quarter</b>.

[seneca]

No one I know uses Netflix anymore, and I haven't for a while, but from what I've seen their subscriber numbers are actually doing quite well.