A Beginner's Guide to eBPF

[Paul-Craft]

This may prove useful:

> eBPF (often aliased BPF)[2][5] is a technology that can run sandboxed programs in a privileged context such as the operating system kernel.[6] It is used to safely and efficiently extend the capabilities of the kernel at runtime without requiring to change kernel source code or load kernel modules.[7] Safety is provided through an in-kernel verifier which performs static code analysis and rejects programs which crash, hang or otherwise interfere with the kernel negatively.[8][9] Examples of programs that are automatically rejected are programs without strong exit guarantees (i.e. for/while loops without exit conditions) and programs dereferencing pointers without safety-checks.[10] Loaded programs which passed the verifier are either interpreted or in-kernel JIT compiled for native execution performance. The execution model is event-driven and with few exceptions run-to-completion,[2] meaning, programs can be attached to various hook points in the operating system kernel and are run upon triggering of an event. eBPF use cases include (but are not limited to) networking such as XDP, tracing and security subsystems.[6] Given eBPF's efficiency and flexibility opened up new possibilities to solve production issues, Brendan Gregg famously coined eBPF as "superpowers for Linux".[11] Linus Torvalds expressed that "BPF has actually been really useful, and the real power of it is how it allows people to do specialized code that isn't enabled until asked for".[12] Due to its success in Linux, the eBPF runtime has been ported to other operating systems such as Windows.[4]

https://en.wikipedia.org/wiki/EBPF

[wittjeff]

A paragraph like this one should be the first thing in the Readme.

[awesomegoat_com]

Haven't sandboxed programs in a privileged context been the root cause of me seeing BSOD so often in the late 90ties?

[DethNinja]

I’m new to Linux kernel programming & eBPF (just started last week) and I’m having major troubles with eBPF verifier. I honestly feel like it would be easier for me to write a kernel module than eBPF code.

I do wonder if this is the case for many people. It seems verifier is a bit unpredictable and makes eBPF programming quite painful.

[cookiengineer]

Can confirm, it is quite painful.

The bpftools maintainers tell you to learn the bytecode format when you ask them what the errors mean, because they expect you to understand what the verifier means when it tells you "unknown scalar" on every single goddamn line of code.

Something like "ebpf coding rules, what to use and what not" would be very helpful.

All ebpf examples that are older than say, 3 months, already don't work anymore. Not even the official ones from the XDP tutorial project (and the libxdp maintainers because the kinda are splitting off a lot of headers into a separate xdp library as it seems).

Most userspace code still relies on the 5 years old bpf-helpers.h, which meanwhile is not supported anymore because it doesn't use the __helper methods from the kernel (they also refactored the kernel in the meantime, and force you to use e.g. __u128 instead of native data types).

Oh boi, did I underestimate what "bytecode vm" means when kernel developers talk about it.

Also, always use llvm, and remember to build two bpf files for each endianness, and use -g for debug symbols. Otherwise you will try to find out what the bpftool errors mean for days, because of shitty mailing list answers.

[tptacek]

It would be much easier to write a kernel module than an eBPF program. But the eBPF program is unlikely to panic your machine, and the kernel module is almost certain to.

[diarrhea]

Reading more about that… How does the verifier detect infinite loops anyway? Halting problem and all. It must use some rather crude heuristics, no?

[kqr]

Ever since Brendan Gregg started using eBPF for observability back in 2015 I've had the sense that eBPF is an extremely underrated tool of the future. I really would like to learn it, but beyond some improvised bpftrace scripting and the tools that come with bcc, I've not really had the need.

What custom usage do you have for it?

[blipvert]

I’ve used it to write a layer 4 load balancer to replace old hardware appliances. Similar to Facebook’s Katran, but switches traffic at layer 2 (rather than L3) for compatibility and in Go rather than C++,’cos I’m not a great coder ;-)

[wittjeff]

The entire first page begs the question -- "what is eBPF?"

[CoastalCoder]

It's actually answered (indirectly) on that page:

"My report "What is eBPF?" and in-depth book "Learning eBPF" are both available for download [0] from Isovalent or with your subscription to O'Reilly's learning platform. You can buy "Learning eBPF" from any good bookstore (support your local bookshop by ordering it there!)"

IIUC, you need to give contact info on that page to get the PDF. So this [1] might be a better starting point.

[0] https://isovalent.com/ebpf/

[1] https://ebpf.io/

[noduerme]

After looking at the link first, this comment cracked me up. I skimmed most of the GH content without figuring out the answer; I wasn't going to click an O'Reilly link. So I gave up. Then my OCD kicked in and I got annoyed with myself for giving up, since if I'm going to waste time on HN I should really get to the bottom of things. So searched it. I found this:

https://ebpf.io/what-is-ebpf/

It runs sandboxed kernel extensions? Or it's a VM? Something like that? The writing of the page itself doesn't inspire a lot of confidence, but then maybe it's just over my head.

[Dylan16807]

Is a joke going over my head here, because the first sentence links to a document titled "what is eBPF?"

[zquestz]

https://ebpf.io/what-is-ebpf/ is quite useful.

[kjuulh]

Just read the book, I like the way she did it, it starts out easy with what eBPF is, how it works, using short examples for both the low-level under the hood stuff, but also how to actually use it. It doesn't deviate much from this pattern, which is a good thing for such a relatively short book, and she gets to cover a lot of ground that way. It is still a "beginners" eBPF book, but sets the stage for further development through the references or alternative books.

I can highly recommend it if you're eBPF curious =D I guess my only gribe is that the latter parts of the book, is a bit product heavy, but done in the most tasteful manner it could probably have been done, i.e. using it as examples on how to use eBPF.

[qubit0ne]

In case anyone is interested in writing eBPF programs in Rust https://github.com/vishpat/oxidize-ebpf

[jrpelkonen]

I don’t have any experience with this project, but Aya seems more vibrant, https://aya-rs.dev/

[tomgs]

If you’re looking for a slightly higher level look, Unzip does it well: http://unzip.dev/0x00c-ebpf/

[funerr]

Thanks for the shout-out!<3

[anonymousDan]

As I understand it, eBPF is primarily an observation tool and thus is quite limited in the modifications it can make to kernel memory. Does it have any generic way to make arbitrary modifications to kernel memory? Obviously this would invalidate any verification guarantees, but I would expect this to be very minor modifications in practice. For example, if I wanted to hook a page fault handler to change the behaviour without having to pay the cost of signal handlers.

[madog]

What you can do is quite limited. You're restricted to some preset eBPF program types, and each program type has a restricted set of operations it can perform (eBPF helper methods). So arbitrary modifications, absolutely not without adding a helper method and/or program type for this purpose. More program types and helper methods are being added all the time but overall it's pretty limited in use cases and operations.

If you want full control then kernel module is the way to go, but this doesn't have the same security and stability guarantees.

[tptacek]

eBPF has pretty significant latitude with network traffic, but for everything else the idiom is to use it to do fast kernel telemetry to a userland process that does the actual acting.

[zero-dark]

The GitHub page linked here mentions the word “eBPF” 18 times and yet not once does it expand on what this abbreviation means.

[Palomides]

while originally it's "extended Berkeley Packet Filter", at this point it does so much more than packet filtering that spelling out the words probably increases confusion rather than clarifying anything

it's like a reverse backronym

[phillipcarter]

It's extremely berkeley packet fun.

[mooreds]

If you are into podcasts, she was on "Screaming in the Cloud" recently, and talks about what eBPF enables, how to teach technologies and more.

https://www.lastweekinaws.com/podcast/screaming-in-the-cloud...

[ilitirit]

Her implementation of a rudimentary load balancer (in C) demonstrates eBPF in practice:

https://www.youtube.com/watch?v=L3_AOFSNKK8

[jooz]

compared to making a module, apart the fact that the module can end in kernel panic (thats my problem), is there any significance performance difference between BPF and kernel modules ?

If my goal is to reduce the cpu cloud bill, is BPF good enough compared to making a kernel module ?

[jcwayne]

The parody here is so perfect, they even created the technology they don't bother to define.

[witx]

How so?

[JohnDeHope]

This is not eBNF. That is something completely different.

[johnea]

Looks like a "buy my book" ad to me...

[tptacek]

Hey. Please be more careful. The author of this repository isn't the story submitter. People write things that end up on Hacker News all the time with no idea that it's even happened, let alone any intention to promote things to you.

[philpearl1]

Yay!

[upghost]

Something’s off here. I’m reasonably well read and literate on computer topics, I’ve worked in cyber security for over 5 years now, and extremely open-minded to new ideas — this reads at best like derivative marketing jargon and little in the way of technical.

[cookiengineer]

The game changing idea behind eBPF is XDP in my opinion.

Lots of network drivers and NICs support offloading XDP programs ("xdp_prog") to the network controller's chipset, which results in zero CPU i/o interrupts if you e.g. use an XDP_DROP to block traffic.

Being able to block network traffic _before_ it reaches even kernelspace is a game changer.

[tptacek]

eBPF is an extremely big deal in computer and network security, so, I assure you, this isn't "derivative marketing jargon", and it is very technical.

[PhilipRoman]

I've found kernel documentation under Documentation/bpf to be the best resource available. Clear, concise and no marketing-speak

As for this repository - the README triggered a false alarm in my bullshit sensors, but the code example is pretty nice.

[_def]

welcome to cloud engineering.