Cache Hit Rate — Context Reuse Efficiency

The formula

The cache hit rate measures the fraction of cache operations that are reads (reusing prior context) versus writes (creating new cache). A hit rate of 0.9 means 90% of your cache activity is reuse — you’re efficiently building on prior context. A hit rate of 0.1 means 90% is writing cache that’s never read back — you’re constantly establishing context that doesn’t compound.

The denominator (cache_read + cache_write) represents your total cache activity. If you never write cache, the rate is undefined (no cache activity at all). If you write cache and always read it back, the rate approaches 1. The metric captures whether your cache investments are paying off.

What prompt caching is

Prompt caching is a feature offered by modern AI platforms — Claude, ChatGPT, Gemini, Copilot, Cursor, and others. When you send a prompt, the platform stores the processed tokens (the cache_write). On subsequent turns, if your prompt begins with the same prefix, the platform reuses the stored tokens (the cache_read) instead of reprocessing them from scratch.

The economics are dramatic. Cache reads typically cost 10x less than fresh input tokens and process significantly faster — the model reads from memory rather than recomputing attention over the full context. An operator with a 90% cache hit rate pays a fraction of what a 10% hit-rate operator pays for the same total context size.

The catch: cache breaks when context changes. If you reorder your prompt, rewrite the prefix, or switch topics mid-session, the cached prefix no longer matches and the platform falls back to fresh processing. Cache hit rate is fundamentally a measure of context stability.

Why cache hit rate is the highest-leverage metric

Cache reuse is the cheapest, fastest source of signal in the entire cascade. Every cache hit replaces an expensive fresh-input token with a cheap cached one. Improving cache hit rate cascades into every other metric:

• Yield goes up — cache_read is in the yield numerator. More cache reads, higher yield.
• SNR goes up — cache_read is signal. More signal, higher SNR.
• Leverage goes up — cache_read / input increases when you reuse more and send less.
• Cost goes down — cache reads cost a fraction of fresh input.
• Latency goes down — cached tokens are served from memory, not recomputed.

No other metric has this many downstream effects. Compression ratio only affects yield through one term. Velocity is secondary. Cache hit rate is the multiplier that amplifies everything else — which is why it’s the first metric to optimize when improving your cascade.

How to improve cache reuse

• 1. Keep context stable. The cache matches on prefix — if your prompt starts the same way each turn, the cache hits. Don’t reorder your context, rewrite the opening, or shuffle file order between turns. Stability is the single most important factor.
• 2. Use structured prefixes. Put stable, structured content at the top of your context: project conventions, file layout, coding standards, system prompt. This “header” gets cached on turn one and reused on every subsequent turn. Put variable content (the specific question or task) at the bottom.
• 3. Maintain conversation continuity. Don’t restart sessions unnecessarily. The cached prefix from turn 5 is free signal on turn 50. Each session restart forces a new cache_write with no prior cache_read — a hit rate of zero until the cache builds again.
• 4. Avoid mid-session context switches. Jumping between unrelated topics in one session dilutes the cache. Each topic switch changes the prefix and breaks the cache. Group related work into focused sessions where the cache stays relevant throughout.
• 5. Reference, don’t re-paste. If a file is already in context, reference it by name. Re-pasting its contents changes the prompt prefix and can break the cache. “In the auth module from earlier” preserves the cache; pasting auth.ts again may invalidate it.

Impact on cost and latency

FAQ

What is cache hit rate in AI coding?

Cache Hit Rate = cache_read / (cache_read + cache_write). It measures the fraction of cache operations that are reads (reusing prior context) versus writes (creating new cache). A hit rate of 0.9 means 90% of your cache activity is reuse.

What is prompt caching?

Prompt caching stores processed tokens from prior turns and reuses them in subsequent turns at reduced cost and latency. Instead of reprocessing your full context from scratch, the model reads the cached prefix and only processes new input. Cache reads are typically 10x cheaper and faster than fresh input.

Why is cache hit rate the highest-leverage metric?

Cache reuse is the cheapest, fastest source of signal. Every cache hit replaces an expensive fresh-input token with a cheap cached one. Improving cache hit rate increases yield, SNR, and leverage while reducing cost and latency — no other metric has this many downstream effects.

How do I improve my cache hit rate?

Keep context stable across turns, use structured prefixes at the top of your prompts, maintain conversation continuity, avoid mid-session context switches, and reference files instead of re-pasting them. Cache breaks when context changes, so stability is everything.

What is the impact on cost and latency?

Cache reads cost a fraction of fresh input and process significantly faster. A 90% cache hit rate means paying a fraction of the cost and getting faster responses compared to a 10% hit rate for the same total context. High cache hit rate is the single biggest cost and latency lever.