Systems on Glass is an AI-automation side project of mine. Every episode is generated by Claude (Anthropic's AI, working through Claude Code in my own video studio): the script, the animations and 3D scenes, the edit, and the narration in an AI clone of my voice. I choose the topics, bring the stories from my own work, and review each episode before it goes up. Not affiliated with Anthropic.
Every time an ad loaded, the ad server made seventeen calls to the API I worked on, each due back in about twenty milliseconds. Working the answers out live was far too slow. So we cheated: we answered every question before anyone asked it.
The short version
- Hits and misses. Check the cache; on a miss read the database and keep a copy. Your code can fill it (cache-aside) or the cache can fetch for you (read-through). Writes go through, behind, or around it.
- Hit rate is the whole game. The last nine, 90% to 99%, cut the average about four times. Lazy caches fill on misses; eager ones are built before traffic arrives. Ours was fully eager: rebuilt weekly by a batch job, with the hottest answers in each server's memory.
- Something has to leave. LRU, LFU, FIFO and TTL are guesses at Belady's perfect but impossible policy, and learned caches now predict it. Then invalidate (expire, delete or version the key), coalesce the herd, cache the "no", and jitter expiries.
- AI has three levels. The model's own KV cache; prompt caching between requests (reads about a tenth of the price, prefix only, so stable things first); and skipping the model with exact or meaning-matched response caches.
- The rules. Cache what's read often and changes rarely, pick how stale is okay, survive without the cache, and key by what a thing is, not what it's called.
As in my post about this API, the numbers about that system (seventeen calls, twenty milliseconds, a weekly rebuild) are illustrative: rounded, not the real figures. Prices are Anthropic's published multipliers at the time of writing; check the docs before you budget.
90% hits → 0.1×0.9 + 5×0.1 ≈ 0.59 ms · 99% hits → ≈ 0.15 ms (≈ 4× faster)
10 prompts, same prefix: no cache = 10 · cached = 1.25 + 9 × 0.1 ≈ 2.2 (≈ 5× cheaper)
The bug I owned up to in the video was real and recent: my video studio cached voice lines by file name, so two different lines with the same name could come back as the same audio. Keying the cache by a hash of the content fixed it, and is the habit I'd suggest for anything you'd pay to make twice.
Further reading
- Caching challenges and strategies — the Amazon Builders' Library on local versus shared caches, herds and coalescing.
- HTTP caching — MDN's guide to Cache-Control, ETags and 304s (the standard is RFC 9111; stale-while-revalidate is RFC 5861).
- Cache replacement policies — LRU, LFU, FIFO, ARC and Belady's optimal algorithm (1966) side by side.
- Learning Relaxed Belady for CDN caching — Song et al., NSDI 2020: the learned cache that cut WAN traffic 5–24%.
- Two Hard Things — Martin Fowler on the Phil Karlton joke, and its variants.
- Teach Yourself Programming in Ten Years — Peter Norvig's table of timings: the half-nanosecond to 150 ms ladder.
- Prompt caching — Anthropic's docs: prefix order, lifetimes and pricing.
- Automatic prefix caching in vLLM — how a serving engine shares a common prompt start across users.
- A Million Questions a Second — my long post on the ad-relevance API, including the weekly precompute.
Next episode: Context is RAM, why more context doesn't make your AI agent smarter. Previously: What Is an Agent? All episodes →