[dexen]

>printing also will likely impact timing and can change concurrent behaviour as well.

I've had a bug like that and the intuitive way to handle it turned out to be entirely sufficient.

The bug (deep in networking stack, linux kernel on embedded device) was timing sensitive enough that printk() introduced unsuitable shifts. Instead I appended single-character traces into pre-allocated ring buffer memory. The overhead was down to one memory read and two memory writes, plus associated TLB misses if any; not even a function call. Very little infra was needed, and the naive, intuitive implementation sufficed.

An unrelated process would read the ring buffer (exposed as /proc/ file) at opportune time and hand over to the developer.

tl;dr know which steps introduce significant processing, timing delays, or synchronization events and push them out of critical path

[jffhn]

>I appended (...) traces into (...) memory. (...) An unrelated process would read (...) at opportune time and hand over to the developer.

I did something similar to debug concurrent treatments in Java, that allows to accumulate log statements in thread-local or instance-local collections and then publish them with possibly just a lazySet():

https://github.com/jeffhain/jolikit/blob/master/src/main/jav...