DeepFlow – open-source eBPF Distributed Tracing

[bigcat12345678]

Nice project, have been following this project casually for a while. The standout feature is to trace RPC flow across network connections, through packet tracing.

[nimrody]

How can it tie requests arriving at a service and generating additional downstream requests?

Distributed tracing needs some common token all requests share to identify all RPCs that should be associated with a specific incoming request.

[javierhonduco]

Haven’t checked the source code yet, wondering if profiling of code without frame pointes is supported. Curious on their approach.

[reactordev]

It uses eBPF to provide instrumentation of the kernel calls up as well as hooking into networking for http2 pgsql etc. Since it’s running the process in eBPF it’s essentially sandboxed and all memory, kernel function calls, and even profiling, is an option. They have an agent that collects this information and sends to the server over RPC (protobuf/grpc). You should check it out (however, some of the docs are in Chinese).

[progbits]

> DeepFlow can even analyze code performance through network profiling under old version kernels (2.6+).

This is an interesting call-out, the last release of 2.6 is from 2011. I wonder who is still running that in production.

[sharangxy]

Some of our users’ ancestral processes are running on kernel 2.6, and the operations staff dare not upgrade the kernel. Indeed, there are many limitations in 2.6, but the simple traffic analysis has brought surprising insights to users. However, this also brings some troubles: even if a problem is known, no one dares to easily modify the code to fix it, unless absolutely necessary :)

[robertheadley]

I don't have any idea what this is, but the graphs are beautiful.

[jinxiao2010]

good project. I'll take a try. ebpf is so popular, but we're still using traditional network plugins.

[bobberkarl]

How are you different from Pixie?

[sharangxy]

The biggest difference: DeepFlow enables *Distributed* Tracing.

In addition, DeepFlow combines the capabilities of eBPF and cBPF to achieve full-stack tracing of syscall + network_forward. You can take a look at our documentation: https://deepflow.io/docs/about/features/

[ilovesnow]

nice

[mdaniel]

Apache 2, if that interests you: https://github.com/deepflowio/deepflow/blob/v6.4.7/LICENSE

heh, GitHub also has "symbol navigation" turned on for that license file but I didn't dig into it to find out what source language it thinks the file is