[cdavies]

The most exciting defect that I ever had to debug was a dormant async bug that had been in the code for many years. I had checked in what I thought was going to be a fairly innocuous change in a component in our team's subsystem, but it ended up breaking the full build at the end of the day and was going to be backed out the next day if I didn't prove I wasn't responsible for it.

It was about 6AM after a night of constantly building debug ROMs with different instrumentation to try and get one that I could get both meaningful debug information out of but still suffered from the defect that I finally tracked down the problem. It turned out that a circular pseudo-dependency involving 3 different subsystems and five different components was deadlocking, and all I'd done is change the timing a fraction of a second to trigger the deadlock.

Asynchronicity is hard to test, and you'll always, always get it wrong. Don't sweat it too much. Unless you're building flight control software, just make sure the common use cases work 100% of the time and if broken edge cases turn up, fix them as and when.