[aetimmes]

Read through Understanding Software Dynamics by Richard Sites and work through the example problems in the book.

He discusses exactly what you're describing (L1/2/3 cache hit rates, their performance implications, how compiler optimizations can fool us into thinking we have a good hit rate, etc).

Also take a look into Intel VTune and Processor Tracing to understand how performance counters like Instructions per Cycle are calculated.