Systems on Glass is an AI-automation side project of mine. Every episode is generated by Claude (Anthropic's AI, working through Claude Code in my own video studio): the script, the animations and 3D scenes, the edit, and the narration in an AI clone of my voice. I choose the topics, bring the stories from my own work, and review each episode before it goes up. Not affiliated with Anthropic.
At Amazon Ads, the ad-relevance API I worked on answered over a million requests a second, each inside twenty milliseconds. That sounds like one number about speed. It's two different problems, and they have two different fixes.
The short version
- Latency is how long one request takes. Throughput is how many you finish per second. A wider highway moves more cars; no single car arrives sooner.
- Throughput, you buy with more servers. Latency, you earn inside every request, by doing less work.
- Little's law ties them together: requests in flight = throughput × latency.
- Busy servers get slow fast. In the simplest queue model, a request takes its work time ÷ (1 − how busy the server is): 2× at 50% busy, 10× at 90%. Headroom isn't waste.
- Averages lie. One ad request fanned out to 17 calls. If each is slow 1% of the time, roughly 1 request in 6 hits a slow one, and users feel the slowest call. Measure p99, not the mean.
As in my post about this API, every number about that system (a million a second, 20 ms, 17 calls, 3× for Prime Day) is illustrative: deliberately rounded, inflated or deflated, not the real figures. The maths holds for any numbers.
slow at least once = 1 − 0.9917 ≈ 16% ≈ 1 in 6
time per request ≈ work ÷ (1 − busy): 50% → 2×, 90% → 10×
Before Prime Day, a traffic team forecast the peak a month ahead, and I owned scaling our systems up to about three times normal, with autoscaling on top for surprises and the capacity warmed early so nothing started cold. Spare capacity is what kept latency flat.
Further reading
- Little's law — the one-line law behind "20,000 in flight" (Little, 1961).
- The M/M/1 queue — where "10× at 90% busy" comes from (Kleinrock, Queueing Systems, 1975).
- Latency Sneaks Up On You — Marc Brooker on why utilisation and latency don't move together.
- The Tail at Scale — Dean and Barroso (2013) on fan-out and tail latency.
- How NOT to Measure Latency — Gil Tene's talk on percentiles and why averages mislead.
- Designing Data-Intensive Applications — Kleppmann; chapter 1 covers describing load and performance.
- A Million Questions a Second — my long post on the ad-relevance API: precompute, Bloom filters, fan-out and failover.
Next episode: What Is an Agent? Then Caching, and how seventeen calls got cheap. All episodes →