Tracing HTTP Requests with tcpflow

[imiric]

For a tool that understands TCP and HTTP conversations, the filter syntax is awfully similar to tcpdump's, and having to use `grep -A 15` to filter specific requests seems clunky.

It's good knowing this tool exists, but I think I'll stick to tcpdump and Wireshark. In Wireshark it's trivial to use the `http.request` filter to do this, and following the TCP conversation with decoded bodies, or specifying a TLS cert, is equally simple.

[denysvitali]

For a CLI Wireshark, use tshark

[jimmyed]

There is also netpeek[1] which has better filtering capabilities and UX overall. It supports ngrep like bpf filters which we found useful.

1: https://github.com/darshanime/netpeek

[thegabez]

This assumes you have shell and those tools installed via your Dockerfile. Best practice is to have a multi stage build and just have the working binary, keeping the image as light as possible.

So when do you make a fat image with debug tools and when do you keep it skinny?

[mozman]

Or you’re running on a VM and can install these tools for debugging and then blow it away when you’re done

[ystad]

A thought. I would usually start with my application logs. Reverting to a tcp tool for a http app problem seems cobtrived

[0x445442]

Yeah the app looked like it was delegating to another service and needed a bearer token for that. The app logs should have showed this straight away. Furthermore, I’ve been burned enough to know that if I have a service that’s dependent on other services then I setup tasks in my service to ping the health of those services periodically and log warnings/errors if they’re unreachable. This allows me to divert blame as quickly as possible if alerted in the middle of the night.

[nhoughto]

This looks handy, better ergonomics than strace to just hook into http requests of a running process.

I wonder how it works with TLS if it’s working at the socket level..

[xtreak29]

There seems to be an issue open for this https://github.com/simsong/tcpflow/issues/58

[youngtaff]

I’m guessing it doesn’t work with TLS as it’d need the keys to decrypt the traffic

[_ZeD_]

in this space I have successfully used mitmproxy, charles and, with the best experience, fiddler

mitmproxy.org www.charlesproxy.com https://www.telerik.com/fiddler/fiddler-classic

[rwestergren]

+1 for Fiddler, though it doesn't sound like the author was in a position to forward traffic to an HTTP proxy at the time in prod

[29athrowaway]

There's also termshark, a TUI for tshark inspired by Wireshark.

You also have mitmproxy which can be useful if TLS is involved.

[aftbit]

Just because I know them better, I would have used `netstat -putln` and `tcpdump -i lo -n -A port 8000` to do the same thing. I'll take a look at tcpflow and ss though - it's always nice to know more tools.