Packet, where are you? – eBPF-based Linux kernel networking debugger

[ralgozino]

Asking as someone with almost no experience with eBPF.

How do you read the otuput of the tool? In the README's example GIF, If I didn't know there's an IPTables rule dropping the packages I would not know that the package is being dropped.

[bazzargh]

if you have a recent enough kernel, this change https://github.com/cilium/pwru/pull/148 means that it will print the reason the packet was dropped in the output, as logged by kfree_skb_reason - see https://lwn.net/Articles/885729/

There's a whole heap of reasons a packet can be dropped: https://github.com/torvalds/linux/blob/76f598ba7d8e2bfb4855b...

In any case, this makes it less painful than going back to read the kernel source. I don't think that the gif in the README is up to date with this change - it's 2 years old and this feature only appeared 2 months ago.

[zokier]

You'll want to have kernel sources handy. In this case you can see the packet ending up in nf_hook_slow function: https://elixir.bootlin.com/linux/latest/source/net/netfilter...

from there you can see only one branch leading to kfree_skb, so you can make a guess that some netfilter "hook" returned NF_DROP. From that you might already be able to make educated guesses that it might be iptables rule, or continue delving deeper to figure out what those hooks are etc

[nicevalue]

What is the difference b.w this tool and Dropwatch [https://linux.die.net/man/1/dropwatch?]

[suprjami]

Dropwatch just tells you where packets were freed from memory by the "bad" path of `kfree_skb()`. It relies on the "good" packet free path to use `consume_skb()` which isn't always true, so you get a number of false positives with dropwatch. It's hard to spot small individual packet loss among all the noise.

pwru offers src/dst IP/port filtering, so it's easier to exclude uninteresting traffic, and it tracks an skb's journey through the entire network stack, printing each function the skb touches over its lifetime.

They are quite different tools, they probably have some overlap depending on situation.

Also the actual dropwatch command is obsolete, it's more common and useful to use the SystemTap script, or perf, or bpftrace.

[tptacek]

dropwatch is much more limited, and effectively obsolete. It's a purpose-designed kernel feature with its own netlink protocol that has one serious consumer (the accursed dropwatch.c client). Leave pwru aside for a minute, and everything dropwatch can do, you can do better just with bpftrace.

[bmo-at]

This is a great codebase to read for me currently, as I am working on an ebpf based packet aquisition mechanism for a network monitoring program. Anyone got other good projects to look at?

[tinus_hn]

I wish there was a way to describe a packet and then trace how it is handled by the firewall, what rules it matches and why it is allowed, dropped or blocked.

[martinohansen]

Love the name!

[totetsu]

Is 'tracing network packets in the kernel' also what tcpdump/snoop/nettl dp?

[unmole]

No. Tcpdump and others are packet sniffers. They clone network packets so the contents can be examined.

pwru on the other hand is used to trace what the kernel is doing with your packets.

[aeonik]

I can speak for tcpdump and the answer is no. It only looks at the network interface. Something I often want to do is catch which process sent a few UDP packets. Netstat and ss won't catch it because it's too short of a time frame, and tcpdump doesn't contain any information about the kernel.

I've been using opensnitch which uses eBPF rules to track this information lately, but I'm looking for something more flexible.

[suprjami]

I have dreamed about a tool like this for at least 10 years.

[cpufry]

this is pretty neat