[tie_]

Install atop and configure it to sample every second.

I can't count the number of times this has helped solve a mysterious behavior. Atop is king.

[jeffbee]

It's great but it's very expensive so be aware that if you are down to the level of caring about things in the single-digit percentages of your budget, something like atop once per second could easily consume that amount of resources. A lot of these stats are really not cheap at all. E.g. you may find a substantial difference in the amount of CPU time needed to produce /proc/pid/stat as opposed to /proc/pid/statm. And some of the system-wide stats are even worse.

[geocrasher]

Yes, atop is great for this! There's also "sar" (system activity report) which can do similar things. Both are quite helpful.

[winrid]

Be careful running atop with the -R flag on machines that create lots of short lived threads or do lots of IO, like many databases.