Module 8 — Cache Memory (L1, L2, L3)

Learning Objectives

After completing this module, you should be able to answer:

• Why do we need cache?
• Why is RAM slow?
• What is cache memory?
• What are L1, L2, and L3 caches?
• What is cache locality?
• What is a cache line?
• How does the CPU search the cache?
• What is a cache hit?
• What is a cache miss?
• What are direct-mapped, set-associative, and fully associative caches?
• What are write-through and write-back caches?
• What is cache coherence?
• Why is cache important in embedded systems?

1. Why Do We Need Cache?

Imagine a CPU running at 3 GHz.
That means:
3 Billion cycles/second

Now imagine RAM takes about 60–100 nanoseconds to return data (actual values vary by system).
At 3 GHz:
1 clock cycle ≈ 0.33 ns

If RAM access takes 90 ns:
90 / 0.33 ≈ 270 CPU cycles

The CPU may have to wait hundreds of cycles for a RAM access.
Without cache:

Most of the CPU would be idle.

Real-Life Analogy

Imagine you are cooking.

• Warehouse = SSD
• Kitchen Cabinet = RAM
• Kitchen Table = Cache
• Your Hands = Registers

Would you go to the warehouse for every spoon?
No.
You keep frequently used items on the table.
That's exactly what cache does.

Memory Hierarchy

The closer the memory is to the CPU, the faster—and usually smaller—it is.

2. What is Cache?

Cache is small, very fast memory located on or very close to the CPU.
Purpose:
Store recently or frequently used data.

Instead of

we have

If the data is found in a cache, the CPU avoids a slower RAM access.

3. L1 Cache

The fastest cache.

Characteristics:

• Smallest
• Fastest
• Private to each CPU core

Example:

Typical size (varies by processor):

• 32 KB Instruction Cache
• 32 KB Data Cache

Often split into:

• L1I (Instruction Cache)
• L1D (Data Cache)

4. L2 Cache

Larger than L1.
Slightly slower.
Often private to each core (architecture-dependent).

Example:

Typical size:

• 256 KB
• 512 KB
• 1 MB

(varies widely)

5. L3 Cache

Largest on-chip cache.
Usually shared by multiple cores.

Example

Typical size:

• 8 MB
• 16 MB
• 32 MB
• or more.

Complete Cache Hierarchy

6. How Cache Works

Suppose CPU wants

a = array[100];

Step 1
CPU checks
L1
If found
Done.

If not
↓
Check
L2

If not
↓
Check
L3

If not
↓
RAM

This sequence is automatic and managed by hardware.

7. Cache Hit

Suppose
CPU wants
Variable X

Already present
L1 Cache

Result

This is a cache hit.
Fast.

8. Cache Miss

Suppose
Data is not in L1.

This is a cache miss.
Slow.

The fetched data is then typically copied into one or more cache levels for future accesses.

Cache Hit vs Miss

Cache Hit	Cache Miss
Fast	Slow
No RAM access	RAM access required (eventually)
Better performance	Higher latency

9. Why Does Cache Work?

Programs usually access memory with predictable patterns.
Two important principles explain this.

Temporal Locality

"If you used data recently, you'll probably use it again."

Example

sum += x;
sum += y;
sum += z;

Variable
sum
is reused repeatedly.
Keep it in cache.

Spatial Locality

"If you accessed one memory location, you'll probably access nearby locations."

Example

array[0]
array[1]
array[2]
array[3]

Since they are contiguous, fetching one often brings nearby elements into the cache.

10. Cache Line

The CPU does not fetch one byte at a time.
It fetches a cache line.

Typical cache line size:
64 Bytes

Suppose you need
Address 1000

CPU loads
1000, 1001, 1002, ..., 1063
All 64 bytes.
This exploits spatial locality.

Visualization

11. Cache Mapping

The cache cannot hold all of RAM.
The CPU must decide where to place each memory block.
Three common approaches:

Direct-Mapped Cache

Each memory block has exactly one possible cache location.

Example

Simple.
Fast.
Can suffer from frequent conflicts.

Fully Associative Cache

Any memory block can go into any cache slot.

Example

Most flexible.
Most expensive hardware.

Set-Associative Cache

A practical compromise.

Example

A memory block maps to one set but can occupy any way within that set.

Common choices:

• 2-way
• 4-way
• 8-way
• 16-way

Most modern CPUs use set-associative caches.

12. Cache Replacement Policy

What happens when the cache is full?
One line must be evicted.

Common policies:

• Least Recently Used (LRU): Remove the line not used for the longest time.
• FIFO: Remove the oldest line.
• Random: Choose a random line.

Modern CPUs often use approximations to LRU for efficiency.

13. Write Policies

Write-Through

Every write updates both cache and RAM.

Advantages:

• RAM always contains the latest data.

Disadvantages:

• More memory traffic.

Write-Back

RAM is updated only when the modified cache line is evicted.

Advantages:

• Faster writes.
• Lower memory bandwidth usage.

Disadvantages:

• More complex.
• Requires tracking modified ("dirty") cache lines.

14. Cache Coherence (Multicore)

Suppose
Core 1
Counter = 10

Core 2
also has
Counter = 10

Core 1 updates
Counter =20

Core 2 still has
10

Wrong.
The hardware must keep caches synchronized.
This is called cache coherence.

Modern processors commonly use protocols such as MESI (Modified, Exclusive, Shared, Invalid).

15. DMA and Cache (Very Important for Embedded)

This is highly relevant to your ADAS work.

CPU
↓
Reads Image

Problem:
CPU may still have an old copy of the image in its cache.
DMA updated RAM directly.
CPU cache wasn't updated.

Result:
Old image processed.
Wrong ADAS output.

Solution:
Typical embedded software performs cache maintenance operations:

Or, before DMA reads data written by the CPU:

The exact operations depend on the CPU architecture and DMA direction.

ADAS Example

If cache maintenance is forgotten:

• Stale image data
• Corrupted frames
• Flickering
• Incorrect detections

This is a common source of embedded bugs.

Why Cache Matters for Operating Systems

The OS:

• Manages memory mappings.
• Coordinates DMA with device drivers.
• Schedules tasks across CPU cores.
• Relies on hardware cache coherence mechanisms.
• Uses cache-aware data structures for performance.

Interview Questions

Beginner

• Why do we need cache?
• Difference between cache and RAM?
• What are L1, L2, and L3 caches?
• What is a cache hit?
• What is a cache miss?

Intermediate

• Explain temporal locality.
• Explain spatial locality.
• What is a cache line?
• Difference between write-through and write-back.
• What is cache coherence?

Senior

• Explain DMA cache coherency problems.
• Why are cache maintenance operations required in embedded systems?
• Direct-mapped vs set-associative cache.
• Explain MESI at a high level.
• Why does cache improve CPU performance?
• How can poor data layout reduce cache efficiency?

Common Mistakes

• ❌ Thinking cache is software-managed in general-purpose CPUs. It is primarily managed by hardware (with software involvement for special cases like DMA).
• ❌ Assuming L3 exists on every processor. Many microcontrollers have only L1 or no cache at all.
• ❌ Believing cache stores only instructions. Modern CPUs often have separate instruction and data caches.
• ❌ Thinking DMA automatically updates CPU caches. It updates RAM, not cached copies.

Summary

Cache memory bridges the performance gap between the CPU and RAM. It exploits temporal and spatial locality to keep frequently used data close to the processor. Modern systems use a hierarchy of L1, L2, and L3 caches, along with sophisticated mapping, replacement, write policies, and coherence protocols. For embedded systems—especially ADAS camera pipelines—understanding cache behavior is essential for both correctness and performance.

Mini Quiz

1. Why is cache needed?
2. Why is RAM slower than cache?
3. What is the difference between L1, L2, and L3 caches?
4. What is a cache hit?
5. What is temporal locality?
6. What is spatial locality?
7. Why are cache lines used instead of fetching a single byte?
8. What is the difference between write-through and write-back?
9. Why is cache coherence necessary in multicore systems?
10. Why can DMA create cache coherency issues?