top | item 12442878

Linux debugging tools you'll love

400 points| rollulus | 9 years ago |jvns.ca | reply

63 comments

order
[+] partycoder|9 years ago|reply
I think it's a good contribution. Couple of things though:

- lsof can also print the list of files open by a process. fuser can tell you which process open a specific file. lsof is mentioned later in the zine though.

- sysdig is not mentioned but is great.

- wireshark/tcpdump/ngrep/etc might be challenging to practically use in the presence of SSL/TLS. you can mitigate this problem by setting up SSL termination at your network perimeter so you can monitor unencrypted traffic.

- additionally to netcat you have curl, httpie and many other clients for similar purposes. in the Chrome developer tools network tab, you can select a request sent when visiting a website and click "Copy as cURL". this will quickly export any request you were making as a cURL command. likewise, proxies like burp also implement this same feature.

sometimes you need to debug DNS problems, that is not covered sadly. I use dig for this purposes but I wonder if there's something better.

if you are going to be using command line utilities, getting acquainted with cat, grep, cut, sed, awk, tail, head, colrm, tr, sort, uniq, comm, wc, etc. is very recommended. ministat is also cool.

then, about java and node... java has excellent profiling tools, including the free jvisualvm tool shipped with Java that does the job for the most part. profiling and debugging node in runtime can be really challenging. specially analyzing node coredumps with mdb_v8 is not for the faint of heart... you need to set up a VM with joyent's SmartOS for this. there's an npm package that simplifies this process called "autopsy". now, flamegraphs might be fine, but i strongly prefer nodegrind + qcachegrind.

[+] avtar|9 years ago|reply
> wireshark/tcpdump/ngrep/etc might be challenging to practically use in the presence of SSL/TLS. you can mitigate this problem by setting up SSL termination at your network perimeter so you can monitor unencrypted traffic.

https://mitmproxy.org/ is pretty awesome for that type of work.

[+] JakeTheAndroid|9 years ago|reply
Additionally, Chrome offers chrome://net-internals which let's you inspect a good deal of information about the request being sent.
[+] wyldfire|9 years ago|reply
> ways I've changed how I think about debugging:

> bug is happening for a logical reason. there's no magic

> be confident I can fix it

> talk to someone

Those are good steps. IMO "talk to someone" is a critical step but you'll get wildly different results based on the quality of your reference resource. Ms Evans likely has access to someone (or is someone) who clearly knows their stuff. eBPF and other debugging features enabled by modern kernels are not well known by many developers.

But "be confident I can [diagnose] it" is a great first step and if you're persistent you will find the help you need. Sometimes it may take plumbing the depths of SO or IRC to find that help, but it's out there.

[+] phil21|9 years ago|reply
Finding someone smart to talk to is important, and a lot of people fail at this step. I like to consider myself decently good at Linux troubleshooting and get asked a lot of questions most days.

My sole piece of advice to folks looking for help: Be specific and help yourself. This means don't come and ask "why is my server slow?!" - it's coming into the conversation telling me the exact symptoms and what you've already tried and where you think the next steps are.

If you are not at that point, you haven't put enough effort in for me to bother. Those are called paid consulting engagements.

I would say most people get this wrong, and end up being ignored over time. The few folks who consistently give me interesting (even if they are trivial and I've seen it before) problems to help them with that they just need a bit of specific knowledge to solve? I look forward to them contacting me.

[+] baldfat|9 years ago|reply
> Write out a good logical question for stackoverflow and answer my own question.

90% of the time!

[+] AlexB138|9 years ago|reply
The cutesy/manic writing style is a little jarring, but it seems like a lot of effort was put in to making a useful resource. Worth taking a peak at, don't let the style scare you off.
[+] xbryanx|9 years ago|reply
The approachable/playful writing style makes it so much more engaging for me. It's refreshing to see some of this stuff explained in a conversational and enthusiastic manner. Obviously, just personal taste.
[+] gricardo99|9 years ago|reply
To be fair, this is a spin-off (or elaboration) on an earlier post, where (if you follow the Author's blog) she specifically talked about how she'd like to write her own zine. The style is the point of it, I think.

If you want less cutesy, here's the related post: https://news.ycombinator.com/item?id=12059156

[+] mVChr|9 years ago|reply
I like the change of pace. So many of these types of guides seem like cookie cutters of whatever the latest writing and design trends are. I find it nice to see indie zine style (stuff I like to read on my off time) fused with a practical technical resource (stuff I like to read on my on time).
[+] blub|9 years ago|reply
You weren't kidding.

The content is good (for the 7 pages I was able to follow it), but the comic book delivery is not working as well as her previous blog posts. Not searchable, more difficult to follow.

[+] ryancnelson|9 years ago|reply
This 'zine (and really, much of the author's entire website) are pretty much pure, concentrated joy and tech-geek happiness. I hope my kid grows up to have the sort of attitude about solving problems and going about life that she does.

Bravo, Ms. Jvns!

[+] jcoffland|9 years ago|reply
While we're on the topic of simple debugging tools. One I find I use all the time is top. Not classically a debugging tool but it sure solves a debugging problem and I find myself using it very often to check CPU and memory usage.

On the topic of more sophisticated debugging tools I like valgrind and Google's gperftools. These are definitely more difficult to use but I recommend not giving up on them because the pay off is huge. The trick is a) knowing how to run these tools and b) knowing how to read and understand the output. Both can be achieved through practice and RTFM.

[+] sametmax|9 years ago|reply
I prefer htop to top (or glances if Python is an option), but it's true it's not installed by default.
[+] wyldfire|9 years ago|reply
Regarding "valgrind" -- I've had better results with electric fence and ASan/TSan/UBSan, especially since most of them work on non-x86. And I've had issues with valgrind when investigating a program that leverages newer x86 instructions than it was built to interpret.
[+] oopsies49|9 years ago|reply
Anyone have more details on the netcat file transfer trick on page 9?

You would want to confirm the data received matches the data sent somehow right?

[+] 1_player|9 years ago|reply
You start a listening nc on the destination server, and push the file from the source server.

    Destination: nc -l -p 1234 > foo
    Source: nc destination 1234 < foo
Protip #1: You can also send directories:

    Destination: nc -l -p 1234 | tar xf -
    Source: tar czf - directory | nc destination 1234
Protip #2: pv for progress indicators (pv on one side is enough)

    Destination: nc -l -p 1234 | pv > foo
    Source: pv foo | nc destination 1234
Note: netcat (BSD or GNU variants) syntax varies across unixes and distros. Sometimes it's `nc -l -p 1234`, other times `nc -l 0.0.0.0 1234`. Check your man.

You can check for data corruption with md5sum or similar checksumming tools.

[+] helper|9 years ago|reply
This is by far best enjoyed on paper. Print one out and share it with your office!
[+] lucb1e|9 years ago|reply
Wow, that is a different style than I'm used to. Must say I'm happily surprised by good content, despite the playful look. It makes it seem very accessible, nicely done.
[+] tbarbugli|9 years ago|reply
I appreciate the effort and enjoyed this paper. One small remark: the first example is ridiculous. You don't need such a low level tool to find out someone is doing crazy database queries, just analyze your slow log!
[+] garaetjjte|9 years ago|reply
netstat is obsolete.
[+] hobr|9 years ago|reply
What do you recommend as a replacement? I'd love to get something that wasn't quite so opaque.