Systems on Glass is an AI-automation side project of mine. Every episode is generated by Claude (Anthropic's AI, working through Claude Code in my own video studio): the script, the graphs and 3D scenes, the edit, and the narration in an AI clone of my voice. I choose the topics, bring the stories from my own work, and review each episode before it goes up. Not affiliated with Anthropic.
At 7 AM my Daily Bible service sends the day's scripture to half a million phones. If the provider stumbles for one second, every send in that second fails at once. The obvious fix, retrying right away, sends the same wave straight back at a provider that just fell over.
The short version
- Retries multiply. In a call five services deep with three retries per layer, the database at the bottom sees 243 times the load (Amazon's Builders' Library). In AWS's October 2025 us-east-1 outage, one EC2 subsystem kept re-queuing work until it "entered a state of congestive collapse", and engineers had to throttle incoming work.
- Back off. Wait longer after each failure (1, 2, 4, 8 seconds), up to a cap, so the service gets room to recover.
- Add jitter. Backoff alone keeps a crowd that failed together retrying together. Full jitter waits a random time between zero and the backoff. Marc Brooker showed it in 2015; AWS's 2026 SDK update makes it the default in every SDK:
random(0, 1) × min(20 s, base × 2^retry). - Budget retries. Three attempts per request and a retry quota (the AWS SDKs: 500 tokens, 14 per retry after a timeout). Without one, a client keeps retrying at full rate during an outage.
- Know when to stop. Never retry a permanent failure (a bounced address won't heal). Believe a 429's Retry-After. After five tries, send it to a dead-letter queue with an alarm, fix the cause, and redrive with the same key.
AWS SDKs (2026): delay = random(0, 1) × min(20 s, base × 2retry) · base 50 ms (transient), 1 s (throttling) · 3 attempts · 500-token quota
The half-million wave and the provider's limit are illustrative, as in my post about the design, which has the dead-letter queue and the sweeper.
Further reading
- Announcing updated retry behavior for AWS SDKs and Tools — May 2026: standard retries, full jitter and a retry quota in every AWS SDK, default from November 2026.
- Retry behavior (AWS SDKs and Tools) — the exact formula, base delays and token costs.
- AWS's summary of the October 2025 us-east-1 disruption — congestive collapse, and recovering by throttling work.
- Timeouts, retries and backoff with jitter — the Builders' Library on why retries are selfish, and the 243× stack.
- Exponential Backoff And Jitter — Marc Brooker's 2015 simulation, where it started.
Next episode: Skills. Previously: Give Your Agent a Memory. All episodes →