Reel · Chaos & Reliability · Oct 26, 2026
Retries ne bugs se zyada production maara hai
In short
A service slows down. It has three callers and each retries three times, so 3 × 3 = 9 attempts, and it compounds down the chain like 6 PM at Rajiv Chowk. The service that was only slow is now down, because the retries never let it recover. The fix: add jitter, give retries a budget, or don't retry at that layer at all.
Label from the original post: composite scenario, demo figures
Checklist: what to do about it
- Find every place your service retries, including client libraries and SDK defaults.
- Use exponential backoff with jitter so clients don't retry in lockstep.
- Cap retries per request and across the whole service (a retry budget).
- Retry at one layer, close to the user, rather than at every hop.
- Retry only idempotent operations. Make writes idempotent first.
- Set timeouts shorter than your caller's timeout.
- Test it: inject latency into one dependency in staging and watch whether traffic to it goes up.