Your retries can take down your own service

Retries ne bugs se zyada production maara hai. Haan, sach. Retries are a feature. Unbounded retries are a load test you didn't plan.

How a slow day becomes an outage

A dependency slows down. Every client retries immediately. Now the struggling service gets the original traffic plus every retry, exactly when it has the least capacity to handle it.

It gets worse with layers. Say a service has callers that each try 3 times, and those callers have callers that also try 3 times. That's up to 3 × 3 = 9 attempts at the bottom for one user request. Add a third layer and it's 27. The service that was only a little slow is now fully down, because the retries never give it room to recover.

Four habits that stop it

1. Back off exponentially, and add jitter. Wait longer after each failure, and randomise the wait so clients don't retry in lockstep and hit the service in synchronised waves.
2. Put a budget on retries. A cap per request, and a cap across the whole service, so retries can never become most of your traffic.
3. Retry only what's safe to repeat. Make writes idempotent first. Retrying a non-idempotent payment call is how you charge someone twice.
4. Set timeouts shorter than your caller's timeout. If the caller has already given up, any work you're still doing for it is wasted load.

And sometimes the right number of retries at a given layer is zero. Retry at one layer, close to the user, rather than at every hop.

Test it on purpose

This is a great first chaos experiment: inject latency into one dependency in staging and watch your request rate. If traffic to the slow service goes up while it's struggling, you've found your retry storm before production did.

Send this to the engineer who owns your retry settings.

Further reading

• Amazon Builders' Library: Timeouts, retries and backoff with jitter
• Google SRE book: Handling Overload