From CPU’s perspective, the most important metric to measure cache performance is average memory access time.
Memory Access Time Definition 1
Average memory access time = cache hit rate x hit latency + (1 – cache hit rate) x miss penalty
The “4C” we discussed before determines the cache hit rate directly.
The cache read policy decides the cache hit latency. On a cache hit, we first have to check whether the data is in the cache, find where the data is stored in cache, and then read data from cache.
If you recall the reason why we need cache, the main memory access time determines the cache miss penalty. On a cache miss, we first need to determine that the data is not in cache, and get the data from main memory.
Usually, we want high hit rate, so that the miss penalty can be neglected.
Memory Access Time Definition 2
Average memory access time = % of instructions x average instruction cache access time + % of data x average data cache access time
Usually, L1 instruction cache and L1 data cache are separated, thus the total average memory access time is determined by both instruction cache performance and data cache performance.