The need to retry is instinctive. Something didn’t work, so you try it again. We do it with a card that the payment terminal rejects, a door that sticks, a page that won’t load. So it’s no surprise that code does it too, and almost every codebase I have worked on retries something, somewhere.
It can be as naive as this:
Response callWithRetry(Request request) throws Exception {
Exception last = null;
for (int attempt = 1; attempt <= 3; attempt++) {
try {
return client.send(request);
} catch (Exception e) {
last = e;
Thread.sleep(1000);
}
}
throw last;
}Or as complex as a Resilience4j Retry with exponential backoff and jitter, wrapped around a circuit breaker and a
time limiter, configured per dependency in YAML and reporting to a metrics dashboard. In between there’s the single
@Retryable(maxAttempts = 3) annotation, the SDK that retries on your behalf, and the line in a service mesh config
that nobody on the current team remembers adding. The idea underneath is the same in all of them: if the call fails,
wait a bit and try again.
And it works. The network drops a packet, a pod gets rescheduled, a pooled connection turns out to be stale, and the second attempt goes through. The user never finds out. From the point of view of one client making one call, a retry is pure upside.
That point of view is the trap.
A retry policy runs in every instance of every client, and it activates on failure. Failures are rarely independent. When the database has a bad thirty seconds, it has a bad thirty seconds for everyone, and every caller’s retry loop wakes up at the same moment. Each of those loops was written to make a single request more likely to succeed. Together they send the struggling dependency two or three times its normal traffic, right when it has the least capacity to deal with it.
So the question I keep coming back to is what happens when every caller retries this call at once.
The cases that matter, in the order this article goes through them:
- several layers each retry, and the attempts multiply
- a 400gets retried three times and fails anyway, only slower
- a timeout fires after the server already did the work
- thousands of clients back off on the same schedule and return together
- the downstream is overloaded, and retries add load
- the retry library does something different from what you assumed
- the server sends Retry-Afterand every client obeys it to the second
- a POSTgets retried with a fresh idempotency key on every attempt
- a queue consumer keeps retrying a message that will never succeed
- the original problem goes away, and the retries keep the system down anyway
Each of these gets its own section below with what I’d do about it, including what it looks like in Resilience4j where a library does the work. The tests and the checklist at the end pull the fixes together.
If your retry logic only has an answer for the flaky-packet case, it’s tuned for the easiest failure you will ever have.
Backoff is older than your codebase
None of this is new, and the history is worth a couple of minutes, because it explains why the fixes look the way they do.
The first well-known retry problem was on a radio network. ALOHAnet, built at the University of Hawaii in the early 1970s, let stations transmit whenever they liked. When two transmissions collided, both were lost and both stations had to send again. If they both waited the same fixed time, they collided again, and again. So each station waited a random time. Randomness was part of the fix from the first version.
Ethernet took the idea further. In the 1976 paper describing it, Metcalfe and Boggs explain that after each collision a controller waits a random interval with a mean twice as long as the previous one, an algorithm they call Binary Exponential Backoff.[15] The same paper has a line I like a lot. A station, they write, could “usurp the Ether” by not adjusting its retransmission interval as traffic increases, and “both practices are now prohibited by low-level software in each station.” Backing off was the price of sharing a medium with everyone else. A station that didn’t pay it was misbehaving.
TCP learned the same lesson the hard way. In October 1986 the early internet had its first congestion collapse: throughput between Lawrence Berkeley Laboratory and UC Berkeley, 400 yards apart, dropped from 32 Kbps to 40 bps.[16] Senders kept retransmitting lost packets into a network that was losing them because it was overloaded. The fixes Van Jacobson and Mike Karels published in 1988 included exponential backoff of the retransmit timer. Squint a little and that collapse is a retry storm.
For a long time, application developers didn’t have to think about any of this. TCP retransmitted lost packets underneath them, and many applications were a single process talking to a database in the same data centre. That changed as systems were split into services in the 2000s. One user action now crossed several network hops, each of which could fail on its own, and each hop grew its own retry loop. Those loops were mostly written from scratch, with a fixed delay and no randomness, by people who had never needed to read the Ethernet paper.
The lessons came back slowly, and mostly through outages. Michael Nygard’s Release It! (2007) collected the patterns teams had arrived at the hard way and gave them names that stuck: timeouts, circuit breakers, bulkheads.[17] Netflix later packaged several of them into Hystrix, which made them mainstream on the JVM. It has been in maintenance mode since 2018, and its README now points new projects to Resilience4j.[18] Jitter and retry budgets took longer to catch on. They became common advice in the mid-2010s, when Marc Brooker’s post on jitter[1] and Google’s SRE book[4] explained them to backend engineers who were running into the same problem TCP had hit thirty years earlier.
What stands out to me is that the lower layers got backoff and randomness together, as one idea, along with the sense that backing off is something you owe everyone else on the wire. Application retries often kept the loop and dropped the rest. Most of this article is those older lessons moved up the stack.
Count the attempts at the bottom
Start with arithmetic, because it’s the part people skip.
A mobile app makes up to 3 attempts. It calls an API gateway that also makes up to 3 attempts. The gateway calls an orders service, whose HTTP client makes up to 3 attempts to the payments service. Payments uses a database driver with its own retry setting of 3.
mobile app 3 attempts
API gateway 3 attempts per request it receives
orders service 3 attempts per request it receives
payments service 3 attempts per request it receives
database up to 3 × 3 × 3 × 3 = 81 attempts per tapNobody configured 81. Each team picked 3, which sounds modest. The number at the bottom only exists as a product of decisions made in four different repositories.
In the healthy case this costs nothing, since the first attempt succeeds and no retries happen. The multiplier only shows up when the bottom layer is failing, which is exactly when the bottom layer can least afford it. Marc Brooker puts the single-layer version plainly: with N retries and a downstream that fails every call, the system does 1 + N times the work.[2] Stack the layers and that factor compounds.
Google’s SRE book calls this a combinatorial explosion and lands on a rule I agree with: retry only at the layer immediately above the one that is rejecting requests.[4] More on that later. For now, the exercise is simple. Draw the call path from the user to the deepest dependency, and write the retry setting next to every hop. Include the ones you didn’t write yourself: SDK defaults, HTTP client defaults, database drivers, the service mesh, the load balancer, the message broker. Then multiply.
If you can’t fill in a hop because nobody knows what it does, that’s your first finding.
Some failures should never be retried
The loop at the top catches Exception. That means it retries a 400 Bad Request three times, waits a second between
each attempt, and then fails the way it was always going to fail, only slower.
Before deciding how to retry, a client needs to decide what failed. I find it useful to sort failures into three groups:
- The request was never processed. Connection refused, DNS failure, a connect timeout, an HTTP/2
REFUSED_STREAM. Retrying is safe for any method, because the server never saw the request.
- The request was processed and rejected. Validation errors, auth failures, not found, business rules. Retrying produces the same answer.
- Nobody knows. A read timeout after the request was sent. A connection reset halfway through the response. A
504from a gateway. The server may have done nothing, part of the work, or all of it.
The third group is where the damage happens, and it’s larger than people expect.
A rough default table for HTTP:
The 504 row surprises people. A gateway timeout means the gateway stopped waiting. It says nothing about whether the
upstream service stopped working. The order may have been committed two seconds after the gateway hung up.
HTTP itself is careful here. RFC 9110 says a client SHOULD NOT automatically retry a request with a non-idempotent
method unless it knows the request semantics are actually idempotent, or has some means to detect that the original
request was never applied.[9] HTTP/2 goes a step further and gives servers two ways to say “I didn’t process this
one”: the last stream ID in a GOAWAY frame, and the REFUSED_STREAM error code. Requests covered by either signal can
be retried even if they are POSTs.[10] That’s the transport-level version of group 1, and it’s about the only place
you get that guarantee for free.
gRPC draws the same line. Without a configured retry policy, it only retries transparently when it is certain the server application never saw the call. Anything beyond that needs an explicit policy listing the retryable status codes.[6]
I’d add a fourth group that the server decides: “overloaded, don’t retry.” Google’s SRE book describes backends that
return a distinct error when they can see retries piling up, so callers stop instead of adding more.[4] Most HTTP APIs
don’t have an equivalent, but a stable error code in the body of a 503 is enough to build one.
In Resilience4j, this classification is what retryOnException and retryOnResult are for, and it’s worth setting
both explicitly. Leave the exception predicate out and the default is throwable -> true: every exception is
retryable, including whatever your client throws for a 400.[20] With the JDK HttpClient, I’d end up with two
configurations, one for calls that are safe to repeat and one for calls that aren’t. The examples in this article use
Resilience4j 2.x:
RetryConfig unsafeCallRetry = RetryConfig.<HttpResponse<String>>custom()
.maxAttempts(3)
.retryOnException(e -> e instanceof ConnectException
|| e instanceof HttpConnectTimeoutException)
.retryOnResult(response -> response.statusCode() == 429)
.build();
RetryConfig idempotentCallRetry = RetryConfig.<HttpResponse<String>>custom()
.maxAttempts(3)
.retryOnException(e -> e instanceof IOException)
.retryOnResult(response -> Set.of(429, 502, 503, 504).contains(response.statusCode()))
.failAfterMaxAttempts(true)
.build();The first one only retries the “never processed” group, plus 429, which normally means the server refused the request
before doing anything with it. The second also retries read timeouts, which are HttpTimeoutExceptions and therefore
IOExceptions, and 504, because repeating the call is harmless. failAfterMaxAttempts(true) makes Resilience4j
throw MaxRetriesExceededException when the last attempt still returns a retryable status. Without it, the final 503
comes back as an ordinary return value, and code that doesn’t check the status will treat it as a normal response.
A timeout tells you nothing about the outcome
This is where retries and idempotency meet, and where most duplicate side effects come from.
A client sends POST /payments. The server receives it, validates it, calls the payment provider, writes the row, and
starts serializing the response. The client’s read timeout fires at 5 seconds. The response would have arrived at 5.2.
From the client’s side, the request failed. From the server’s side, it succeeded. The retry loop sends it again.
Without an idempotency key, that’s a second payment. With one, it depends on how well the server handles the second request, which is a long topic of its own and the reason I wrote Idempotency Is Easy Until the Second Request Is Different. The short version: the server needs an atomic way to decide who owns execution, a way to recognise a replay of the same command, and a plan for when the first attempt is still running while the retry arrives.
That last case is the one retries produce most often. A client with a 2-second timeout calling an endpoint that sometimes takes 3 seconds will regularly send a retry while the first request is still in flight. The server now has two concurrent executions of the same operation. If the idempotency check is “look up the key, then insert” instead of an atomic insert, both proceed. The section on who owns execution in that article goes through why.
Two things follow for retry design.
First, a client timeout should come from how long the operation can legitimately take, with some headroom, based on real latency distributions. A timeout set near the median guarantees that a large share of requests that would have succeeded get retried. These timeouts usually get justified as failing fast. In practice they mostly produce duplicate work.
Second, the server keeps working after the client gives up. Most frameworks don’t cancel a handler when the client disconnects, and even the ones that do can’t un-send a call to a payment provider. Every retry after a timeout is potentially a second concurrent copy of the operation, and the server has to be built with that in mind.
Timeouts have to nest
Retries multiply attempts. Timeouts decide whether anyone is still waiting for them.
Take this configuration, which I suspect exists in some form in a lot of systems:
mobile app → API gateway timeout 10s, 3 attempts
API gateway → orders service timeout 30s
orders service → payments service timeout 5s, 3 attempts, backoff up to 2s
payments service → provider timeout 20sThe provider slows down and starts taking 25 seconds to respond.
Payments waits up to 20 seconds per provider call. Orders gives up on payments after 5 seconds, retries, gives up again, and retries again. Each of those abandoned requests is still sitting in payments, waiting on the provider. After 10 seconds the mobile app gives up on the gateway and retries, which starts a fresh chain. The gateway’s 30-second timeout never matters, because nobody is listening by then.
One user tap, and within about half a minute up to nine provider calls are in flight for the same payment. None of their responses will reach the person who tapped. Every one of them has an unknown outcome.
The rules I try to follow:
- Inner timeouts are shorter than outer timeouts.
- A layer’s total retry time (attempts × per-attempt timeout + backoff) fits inside its caller’s timeout. Otherwise the later attempts are work nobody will wait for.
- Better still, pass the deadline down instead of configuring each hop independently.
Deadline propagation means the caller says “I need an answer by 10:00:03.500,” and every hop checks how much time is left before deciding what to do. gRPC supports this directly: the client sets a deadline, servers can check whether it has passed, and a server calling other services can propagate the original deadline to them.[7] Over plain HTTP you carry it yourself, usually as a header containing either an absolute timestamp or a remaining budget in milliseconds.
Once you have a deadline, the retry decision gets much more sensible:
for (int attempt = 1; attempt <= maxAttempts; attempt++) {
Duration remaining = Duration.between(clock.instant(), deadline);
if (remaining.compareTo(minUsefulAttempt) < 0) {
break;
}
Duration attemptTimeout = min(perAttemptTimeout, remaining);
// send with attemptTimeout, classify the failure, back off
}If there’s 80 milliseconds left and the call normally takes 200, don’t send it. Failing now is cheaper for you and for the dependency.
Backoff buys time
The fixed Thread.sleep(1000) in the opening loop has a specific problem. Every client that failed at the same moment
retries at the same moment, one second later, and again one second after that. If the dependency couldn’t handle the
first wave, an identical second wave won’t go any better.
Exponential backoff grows the delay between attempts: 100ms, 200ms, 400ms, 800ms, capped at some maximum. That gives a briefly overloaded dependency room to recover and spreads the retries over a longer window.
It helps less than people assume, though, and Brooker’s post “What is Backoff For?” explains why better than I can.[3] Backoff defers work. It only reduces total work when the clients are a small, fixed group sending requests one after another, like a fleet of workers polling a queue. A worker that is sleeping isn’t sending its next request either, so the system gets real relief.
Most public APIs look different. They have a large, open population of callers, each deciding independently when to send a first request. A page that a million people load once a day doesn’t get less traffic because some clients are backing off; new visitors keep arriving at the same rate. There, backoff flattens short spikes, which is valuable. During a long overload it moves the retries to later, where they pile up behind the next wave of first attempts.
So backoff is necessary, and it handles short blips well. It does nothing to limit how much extra load retries create during a sustained outage. That needs a budget, which comes a couple of sections below.
Jitter breaks up the crowd
Exponential backoff without randomness still keeps clients in step. Ten thousand clients that failed at t=0 will all
retry at t=100ms, then t=300ms, then t=700ms. The load arrives in spikes with quiet gaps between them, and the
spikes are what hurt.
Jitter randomises each delay so clients drift apart. The version I default to is what Brooker’s 2015 AWS post calls “full jitter”: pick a random delay between zero and the capped exponential value.[1]
long backoffMillis(int retry) {
long exponential = BASE_MILLIS << Math.min(retry - 1, 16);
long capped = Math.min(MAX_BACKOFF_MILLIS, exponential);
return ThreadLocalRandom.current().nextLong(capped + 1);
}In his simulations, both full jitter and “decorrelated jitter” cut total client work substantially compared to no jitter. Full jitter did slightly less work, and “equal jitter”, which keeps half the delay and randomises the other half, came out worst of the three.[1] gRPC’s built-in retries apply a narrower jitter of plus or minus 20% around the backoff value.[6] Any of these is dramatically better than none. The exact formula matters much less than having the randomness at all.
Resilience4j’s default Retry has neither backoff nor jitter: it waits a fixed 500ms between attempts.[20] Both come
from an IntervalFunction:
IntervalFunction backoff = IntervalFunction.ofExponentialRandomBackoff(
Duration.ofMillis(100), 2.0, 0.5, Duration.ofSeconds(5));
RetryConfig config = RetryConfig.custom()
.maxAttempts(3)
.intervalFunction(backoff)
.build();Or, with the Spring Boot starter:
resilience4j.retry:
instances:
payments:
maxAttempts: 3
waitDuration: 100ms
enableExponentialBackoff: true
exponentialBackoffMultiplier: 2
exponentialMaxWaitDuration: 5s
enableRandomizedWait: true
randomizedWaitFactor: 0.5Two details are worth knowing, and I only found them by reading the source.[29] The randomisation factor spreads each delay evenly around the exponential value, so with 0.5 a base delay of 800ms becomes anything between 400ms and 1200ms. That’s the plus-or-minus style of jitter, closer to gRPC’s than to full jitter. And the cap is applied after the randomisation. Once the exponential value passes the cap, most randomised delays land above it and get clipped to exactly the cap, which lines clients up again. With a 1-second initial interval, a 5-second cap and a factor of 0.5, the fourth retry has a base of 8 seconds, draws a value between 4 and 12, and comes out as exactly 5 seconds about seven times out of eight.
Either keep the attempt count low enough that you never reach the cap, or pass your own IntervalFunction that caps
first and randomises second, which is full jitter again:
IntervalFunction fullJitter = attempt -> {
long exponential = 100L << Math.min(attempt - 1, 16);
return ThreadLocalRandom.current().nextLong(Math.min(5_000L, exponential) + 1);
};The same synchronisation shows up outside retry loops:
- mobile apps that all reconnect the moment connectivity comes back after an outage
- cron jobs scheduled at 0 * * * *across hundreds of tenants
- cache entries written together during a deploy, with the same TTL, expiring together
- clients that refresh auth tokens at a fixed interval after startup, all started by the same rollout
They all benefit from the same trick: add a random offset so the fleet stops moving in step.
Budgets cap the multiplier
A maximum number of attempts per request is a necessary limit, and it’s the only one most retry code has. It caps what a single request can cost. It says nothing about what the whole client costs the dependency when everything is failing. At a 100% failure rate, “3 attempts” means 3x the load, exactly when you want less.
A retry budget caps retries as a fraction of total traffic. When failures are rare, every failure can be retried. When failures become common, retries dry up, and the client goes back to sending roughly its normal first-attempt traffic.
The token bucket version is small:
final class RetryBudget {
private final double maxTokens;
private final double tokensPerSuccess;
private double tokens;
RetryBudget(double maxTokens, double tokensPerSuccess) {
this.maxTokens = maxTokens;
this.tokensPerSuccess = tokensPerSuccess;
this.tokens = maxTokens;
}
synchronized void recordSuccess() {
tokens = Math.min(maxTokens, tokens + tokensPerSuccess);
}
synchronized boolean hasTokens() {
return tokens >= 1;
}
synchronized boolean tryAcquireRetry() {
if (tokens < 1) {
return false;
}
tokens -= 1;
return true;
}
}With tokensPerSuccess = 0.1, every success adds a tenth of a token and every retry costs a whole one. Over time that
limits retries to about 10% of successful calls. When the downstream falls over, successes stop, the bucket drains after
a handful of retries, and the client sends only first attempts until things recover.
The idea shows up in plenty of places:
- gRPC’s retryThrottlingconfig tracks a token count per server. Failures remove a token, successes addtokenRatio, and retries pause while the count is below half ofmaxTokens.[6]
- The AWS SDKs’ standard retry mode has a retry quota that works the same way. It never delays the initial request, only retries.[8]
- Google’s SRE book describes a per-client budget where a request is only retried while retries make up less than 10% of that client’s traffic, and notes that it brings the growth from retries down from 3x to about 1.1x in the general case.[4]
- Finagle clients retry out of a budget that allows roughly 20% of requests to be retried, plus 10 retries per second so that new or quiet clients can still retry.[19]
- Envoy can limit concurrent retries to a percentage of active requests, 20% by default, once you configure a retry budget on the cluster.[28]
Resilience4j doesn’t include a retry budget, but its predicates and events make it straightforward to add the one above. Check the budget when deciding whether a failure is retryable, spend a token when a retry actually happens, and record successes around the call:
RetryBudget budget = new RetryBudget(10, 0.1);
RetryConfig config = RetryConfig.<HttpResponse<String>>custom()
.maxAttempts(3)
.intervalFunction(fullJitter)
.retryOnException(e -> e instanceof IOException && budget.hasTokens())
.retryOnResult(r -> RETRYABLE_STATUSES.contains(r.statusCode()) && budget.hasTokens())
.build();
Retry paymentsRetry = Retry.of("payments", config);
paymentsRetry.getEventPublisher().onRetry(event -> budget.tryAcquireRetry());
HttpResponse<String> response = paymentsRetry.executeCallable(() -> {
HttpResponse<String> r = http.send(request, BodyHandlers.ofString());
if (!RETRYABLE_STATUSES.contains(r.statusCode())) {
budget.recordSuccess();
}
return r;
});The token is spent in the onRetry event, which only fires when a retry is about to happen. Resilience4j also runs
the predicates after the final attempt, when no retry will follow, so spending tokens there would overcharge. The
Retry instance and its budget should be shared by everything that calls the same dependency. Creating one per request
gives every request a full bucket, and the budget limits nothing.
One caveat from Brooker’s simulations: budgets are estimated per client, and small clients make noisy estimates.[2] A thousand short-lived serverless functions, each starting with its own full bucket, behave a lot like a thousand clients with plain “N retries,” because none of them sees enough failures to drain its bucket. If your callers are numerous and short-lived, a budget in a shared proxy or sidecar sees more traffic and makes better decisions than one inside each function.
Pick one layer to retry
Back to the 81 attempts. The change with the biggest payoff is usually deciding which layer owns retries for each hop and turning retries off everywhere else.
The layer immediately above the failing dependency is usually the right owner. It has the most context: it knows which errors are transient for that dependency, whether the operation is idempotent, and how much of its deadline is left. Layers further up only see “orders returned 503” and have no idea that three retries already happened underneath.
When the owning layer gives up, it should return something that tells its callers to stop. That might be a 503 with
an error code like DEPENDENCY_UNAVAILABLE, documented as non-retryable, or a degraded response if one makes sense.
Without that signal, the next layer up treats the failure as transient and starts its own loop.
Places where retries hide:
- HTTP client libraries. Some retry certain failures on their own, usually connection-level ones. Check the defaults of the one you actually use; they differ between libraries and sometimes between versions of the same library.
- Cloud SDKs. The AWS SDKs’ standard mode defaults to 3 total attempts.[8]
- Service meshes. Mesh-level retries take one YAML block to add and are easy to forget about while the application also retries.
- Load balancers and reverse proxies. nginx’s proxy_next_upstreampasses a failed request to the next upstream server on errors and timeouts. Since 1.9.13 it won’t do that forPOST,LOCKorPATCHonce the request has been sent upstream, unless you add thenon_idempotentoption.[12] That option is worth searching for in your configs.
- Database drivers and ORMs. Some reconnect and re-run statements after a failover. Know whether yours does, and for which statements.
- Framework annotations. A @Retryablemethod that calls another@Retryablemethod.
- Message brokers. Redelivery is a retry policy too. More below.
- People. A user staring at a spinner will press the button again. If the UI doesn’t disable the submit button, or doesn’t reuse the same idempotency key on the second click, the user is an extra retry layer with no backoff at all.
Read the defaults before trusting a library
I’d use a library for all of this. The hand-written snippets in this article are there to show the mechanics, and a maintained library handles edge cases that a loop written on a Friday afternoon won’t. But a library’s retry is still a retry policy, and its defaults are decisions somebody else made for a general case. They vary more than I expected:
Sources for each row are in the references.[20][22][23][24][25][26][27]
A few things jump out.
The same “3” means three attempts in some libraries and four in others. Spring’s two @Retryable annotations disagree
with each other: Spring Retry’s maxAttempts counts the first call, while Spring Framework 7’s maxRetries doesn’t.
Spring Retry is now archived in favour of the Spring Framework 7 annotation, so plenty of teams will migrate from one
to the other and quietly go from three attempts to four.
Most of them retry every exception unless you tell them otherwise. That covers the exception for a 400, a failed
deserialisation, and a NullPointerException in your own code.
Jitter is usually opt-in. The .NET standard handler is the exception: exponential backoff with jitter, a circuit breaker
and per-attempt and total timeouts out of the box. It also retries POST by default, until you call
DisableForUnsafeHttpMethods().[25] go-retryablehttp also retries regardless of method, and when it sees a 429 or
503 with Retry-After, its default backoff returns the server’s value as-is, with no jitter.[27] That’s exactly the
synchronised return described in the next section.
On the positive side, tenacity’s wait_random_exponential implements full jitter, and its documentation links to the
AWS post that named it.[26] Most of these libraries can do the right thing. They just don’t all do it by default.
Order matters in Resilience4j
Resilience4j splits resilience into separate pieces: Retry, CircuitBreaker, TimeLimiter, Bulkhead and
RateLimiter. How you nest them changes what happens during an outage. With the Spring Boot annotations the default
order is Retry ( CircuitBreaker ( RateLimiter ( TimeLimiter ( Bulkhead ( Function ) ) ) ) ), so every attempt goes
through the circuit breaker and gets its own time limit.[21] With the functional style you pick the order yourself, and
each with... call wraps the ones before it:
Supplier<HttpResponse<String>> call = Decorators.ofSupplier(() -> send(request))
.withCircuitBreaker(paymentsBreaker)
.withRetry(paymentsRetry)
.decorate();send here wraps the checked IOException in an UncheckedIOException, since Supplier can’t throw checked
exceptions.
With the retry outside the breaker, an open breaker makes every attempt fail immediately with
CallNotPermittedException. The default exception predicate treats that as retryable, so Retry waits, tries again,
hits the same open breaker, and repeats until it runs out of attempts. Add it to ignoreExceptions, which applies on
top of any predicate you configured, so an open breaker ends the call straight away:
RetryConfig config = RetryConfig.<HttpResponse<String>>custom()
.maxAttempts(3)
.intervalFunction(fullJitter)
.retryOnException(e -> e instanceof IOException || e instanceof UncheckedIOException)
.ignoreExceptions(CallNotPermittedException.class)
.build();A TimeLimiter inside the Retry is a per-attempt timeout. It doesn’t limit the whole call including backoff, so you
still need a total deadline, checked in the retry predicates or enforced by the caller. And when it fires,
cancelRunningFuture cancels the local future. The remote server keeps working, as described in the timeout section.
Retry-After deserves respect, and jitter
When a server returns 429 Too Many Requests or 503 Service Unavailable with a Retry-After header, it’s telling
the client how long to wait.[9][11] Clients should honour it. Plenty don’t, especially hand-written ones, and a client
that answers a 429 by retrying 100ms later is the reason the server needed rate limiting in the first place.
Two details get missed.
Retry-After can be a number of seconds or an HTTP date:[9]
Retry-After: 30
Retry-After: Fri, 25 Sep 2026 10:15:30 GMTA client that only parses integers will treat the date form as garbage, so decide what happens then. Falling back to
normal backoff is reasonable; retrying immediately isn’t. Clamp the value as well. A server bug that sends
Retry-After: 86400 shouldn’t park your worker for a day, and a value longer than your remaining deadline means you
should fail now.
Optional<Duration> retryAfter(HttpResponse<?> response, Instant now) {
return response.headers().firstValue("Retry-After")
.map(String::trim)
.flatMap(value -> parseSeconds(value).or(() -> parseHttpDate(value, now)))
.map(delay -> delay.isNegative() ? Duration.ZERO : delay)
.map(delay -> delay.compareTo(MAX_RETRY_AFTER) > 0 ? MAX_RETRY_AFTER : delay);
}
Optional<Duration> parseSeconds(String value) {
try {
return Optional.of(Duration.ofSeconds(Long.parseLong(value)));
} catch (NumberFormatException e) {
return Optional.empty();
}
}
Optional<Duration> parseHttpDate(String value, Instant now) {
try {
var at = ZonedDateTime.parse(value, DateTimeFormatter.RFC_1123_DATE_TIME);
return Optional.of(Duration.between(now, at.toInstant()));
} catch (DateTimeParseException e) {
return Optional.empty();
}
}The second detail is my favourite. If a server tells ten thousand clients Retry-After: 30 during an incident, and
every client waits exactly thirty seconds, the server has scheduled its own thundering herd for thirty seconds from
now. Clients should add a little jitter on top of Retry-After. Servers can help too, by randomising the value they
send, for example anywhere between 20 and 40 seconds instead of a flat 30.
In Resilience4j, intervalBiFunction receives the attempt number and either the exception or the result, so it can
read Retry-After from the response and fall back to normal backoff for everything else. It replaces
intervalFunction; configuring both throws an IllegalStateException.[20]
IntervalBiFunction<HttpResponse<String>> retryAfterOrBackoff = (attempt, outcome) -> {
long backoff = fullJitter.apply(attempt);
if (outcome.isLeft()) {
return backoff;
}
return retryAfter(outcome.get(), Instant.now())
.map(delay -> delay.toMillis() + ThreadLocalRandom.current().nextLong(1_000))
.orElse(backoff);
};
RetryConfig config = RetryConfig.<HttpResponse<String>>custom()
.maxAttempts(3)
.retryOnResult(r -> (r.statusCode() == 429 || r.statusCode() == 503) && retryAfterFitsDeadline(r))
.intervalBiFunction(retryAfterOrBackoff)
.build();The deadline check belongs in the predicate. By the time the interval function runs, Resilience4j has already decided to retry, which is too late to say “don’t bother, this would take longer than the time I have left.”
The idempotency article suggested 409 Conflict with Retry-After for a request that arrives while an earlier
attempt with the same key is still running. The same advice applies there. Telling the client when to come back beats
letting it spin, and the value you send should be jittered too.
The idempotency key belongs outside the loop
Retrying a POST safely needs an idempotency key, and the key has to stay the same across every attempt of the same
operation. That sounds obvious, and this bug still ships:
for (int attempt = 1; attempt <= maxAttempts; attempt++) {
var request = HttpRequest.newBuilder(paymentsUri)
.header("Idempotency-Key", UUID.randomUUID().toString())
.POST(body)
.build();
// send, classify, back off
}A fresh key per attempt makes every retry look like a new payment. The server’s idempotency layer can be flawless and it still can’t help, because the client has told it these are different operations.
Create the key once, when the client decides to perform the operation:
String idempotencyKey = paymentAttempt.idempotencyKey();
for (int attempt = 1; attempt <= maxAttempts; attempt++) {
var request = HttpRequest.newBuilder(paymentsUri)
.header("Idempotency-Key", idempotencyKey)
.POST(body)
.build();
// send, classify, back off
}Reading the key from paymentAttempt is deliberate. If the key only lives in memory, a client that crashes after the
first attempt comes back without it, and whatever picks the work up after the restart, a scheduled job or a queue
redelivery, will generate a new one. For operations that matter, store the key with the record that represents the
intent, like the order or the payment attempt, before making the first call. Then every retry from every process uses
the same key.
The body has to stay the same too. If a retry re-reads the cart and the total changed between attempts, the second request carries the same key with a different amount. A well-built server rejects that, as described in same key, different command. Build the request once and resend it.
Libraries make this easier to get wrong, because the retry boundary is less visible. With Resilience4j, everything inside the decorated lambda runs again on every attempt, so the request and its key have to be built outside it:
HttpRequest request = HttpRequest.newBuilder(paymentsUri)
.header("Idempotency-Key", paymentAttempt.idempotencyKey())
.POST(BodyPublishers.ofString(body))
.build();
HttpResponse<String> response = paymentsRetry.executeCallable(
() -> http.send(request, BodyHandlers.ofString()));Move UUID.randomUUID() or a fresh read of the cart inside that lambda and both bugs are back. Annotation-based
retries have the same trap in a less obvious place. @Retryable re-runs the whole annotated method, so a key generated
inside that method is a new key on every attempt. Generate it in the caller, or better, load it from the stored payment
attempt.
The same thinking applies one level down, to the calls your service makes on the user’s behalf. When you retry a call to a payment provider, the provider’s idempotency key should be derived from your own operation ID so it’s identical on every attempt.
Queues retry too
HTTP retries are visible in code. Queue retries often aren’t, because the broker does them for you.
When a consumer fails to process a message, most setups make the message available again. An SQS message reappears after its visibility timeout. A Kafka consumer that doesn’t commit its offset sees the record again. A RabbitMQ message that’s nacked with requeue goes back on the queue. Each of these is a retry policy with the same problems as the HTTP one, and usually less supervision.
Two patterns come up again and again.
Poison messages. A message fails every time because of its content: a missing field, an unknown enum value, a reference to something that was deleted. Immediate redelivery turns it into a hot loop. On a Kafka partition processed in order, it also blocks everything behind it. The fix is boring and necessary: cap the number of attempts, then move the message somewhere a person can look at it, usually a dead-letter queue. In the enums article I mentioned a webhook handler that returns 500 on an unexpected enum value. If the sender retries webhooks, that’s the same poison-message loop, just across a network boundary.
Downstream outages. The consumer is fine and the messages are fine, but the service it calls is down. Every message fails, every message gets redelivered, and the consumer hammers the dead service as fast as the broker hands out work. If there’s a max-attempts setting, it’s used up in minutes and perfectly good messages end up in the dead-letter queue. When the downstream recovers, someone redrives the DLQ, which sends the whole backlog at a service that has only just come back.
For outages, the consumer should notice that the dependency is failing and slow down or pause consumption as a whole, then resume gradually. Retrying each message harder makes things worse. Treating “this message is broken” and “the dependency is down” as the same kind of failure is what drains good messages into the DLQ.
Redelivery also means the same message can be processed more than once, so consumers need their own deduplication. The idempotency article’s section on queue consumers covers the inbox-table approach.
When the retries outlive the outage
There’s a failure pattern that catches teams off guard the first time they see it.
The database has a brief problem, say a 20-second failover. Requests slow down, timeouts fire, clients retry. The extra load makes the database slower, so more requests time out, so more retries go out. The failover finishes and the database is healthy again.
The system stays down.
Incoming traffic is now around three times the baseline, because every request is being retried, and the database can’t serve three times the baseline within the clients’ timeouts. Requests keep timing out, which keeps the retry rate high, which keeps the load high. The trigger is gone and the loop sustains itself.
Bronson and colleagues call these metastable failures, in a HotOS paper I’d recommend to anyone running a distributed system.[13] The defining feature is that the system has a stable bad state. Removing the original cause doesn’t bring it back; something has to break the feedback loop. Retries are one of the sustaining effects the paper discusses, and Google’s SRE chapter on cascading failures describes the same positive feedback, including clients retrying requests that missed their deadlines and adding to the overload.[5]
What helps:
- Retry budgets, so the loop can’t grow past a small multiple of normal traffic.
- Load shedding on the server. Rejecting early and cheaply, with a clear “overloaded” signal, is far better than accepting work you’ll time out on. A request that fails in 2ms costs much less than one that fails after holding a connection for 5 seconds.
- A way to turn retries off. During an incident, being able to set retries to zero across a fleet with a config change can be what ends it.
- Gradual recovery. When traffic comes back, let it in in steps.
Circuit breakers come up here too, with a caveat. A client-side breaker that trips on a high error rate can make a partial outage worse, for example when one shard of a dependency is down and the breaker blocks calls to all of them.[2] Brooker’s post also looks at breakers that only stop retries while still letting first attempts through, which avoids the worst of that.
The metrics that show this coming are ratios more than counts:
client.retry.ratio retries / first attempts, per dependency
client.retry.budget_exhausted.count
client.deadline_exceeded_before_send.count
server.requests.attempt_number from a header such as X-Retry-Attempt
consumer.redelivery.count
consumer.dlq.countIf clients send their attempt number in a header, the server can see the retry storm building without guessing. It’s a cheap addition and it makes the “overloaded, don’t retry” decision much easier to automate.
When I wouldn’t retry at all
Retries have a cost, and some calls are better off without them.
- Deterministic failures. Validation errors, permission errors, business rule rejections. Retrying these only makes the failure slower.
- Non-idempotent operations without a key. If you can’t make the retry safe, show the caller the unknown outcome. “We couldn’t confirm your payment. Check your payment history before trying again” is a better experience than a duplicate charge.
- When the deadline is nearly spent. A retry that can’t finish in time is load with no possible benefit.
- Interactive requests where the user can retry. A page that fails fast with a retry button is often better than one that spins for 15 seconds while three attempts happen behind the scenes.
- Deep in the call stack. If a layer above already owns retries, adding more here only multiplies them.
There’s a related technique worth knowing: hedging. The client sends a second copy of a request if the first hasn’t answered after a short delay, and uses whichever response comes back first. gRPC supports it natively.[14] It can cut tail latency nicely for reads. It also deliberately sends duplicates, so it belongs only on idempotent operations, with a tight cap on how many hedged requests can be in flight.
Failure modes worth testing
Retry behaviour is hard to see in unit tests, because a unit test usually covers one client making one call. The tests below need a dependency you can control, either a fake or a fault-injecting proxy, and a way to count the requests that arrive at it.
Downstream returns 503 for everything
Make the dependency return 503 for every request for 60 seconds while normal traffic runs.
Measure the request rate the dependency receives, compared to baseline. With a budget in place, it should stay close to baseline, somewhere around 1.1 to 1.5x. At 3x you only have per-request limits. At 9x or 27x, more than one layer is retrying.
Downstream recovers
Keep the same test running, then make the dependency healthy again.
Traffic should return to baseline quickly, and the success rate should recover with it. If traffic stays elevated after the dependency is healthy, you’re looking at the start of a metastable loop.
Latency just above the timeout
This is nastier than a plain outage. Make the dependency respond successfully, but slightly slower than the client’s timeout: the timeout is 2 seconds, responses take 2.2.
Every attempt does the full work on the server and counts as a failure on the client. Check how many concurrent
executions of the same operation the server sees, and whether any side effect happens twice. For a POST, this is the
test that shows whether idempotency and retries were designed together.
Retry-After
Return 429 with Retry-After: 30. The client should wait at least 30 seconds, plus a little jitter. Then try the
HTTP-date form, a date in the past, a negative number, an empty value, Retry-After: banana, and
Retry-After: 999999. The client should handle each of them without retrying immediately or hanging for days.
Non-retryable errors
Return 400, 403, 404 and 422. Each should produce exactly one request at the dependency. A 401 should produce
at most two, with a credential refresh in between.
Timeout after the server committed
Let a POST succeed on the server, then drop the connection before the response reaches the client.
The retry should carry the same idempotency key and the same body, the server should replay the original result, and exactly one resource should exist afterwards.
Then kill the client process between attempts and restart it. The retry after the restart should still use the original key.
Deadline nearly spent
Send a request with 50ms of deadline remaining to a service whose downstream call normally takes 200ms. There should be no downstream call at all, or at most one, and no retries.
Queue consumer during an outage
Take the consumer’s dependency down for five minutes with messages flowing.
The consumer should slow down or pause, messages should stay on the queue, and the dead-letter queue shouldn’t fill up with messages that were never bad. When the dependency returns, consumption should resume without a burst that knocks it over again.
Count the attempts end to end
Make the deepest dependency fail, send one request at the top of the stack, and count how many attempts arrive at the bottom. Compare that with what you believe your configuration allows. If the numbers differ, there’s a retry somewhere that nobody knew about.
Checklist before shipping
- Classify failures before retrying: never processed, processed and rejected, unknown outcome.
- Retry unknown outcomes only for idempotent operations.
- Don’t retry 400,403,404or422by default.
- Use exponential backoff with jitter. Avoid fixed delays.
- Cap retries with a budget in addition to a per-request attempt limit.
- Pick one layer to own retries for each hop, and turn retries off in the others.
- Inventory the retries you didn’t write: SDKs, HTTP clients, drivers, meshes, proxies, brokers.
- Read your retry library’s defaults: how it counts attempts, which exceptions it retries, whether it jitters, and
whether it retries POST.
- In Resilience4j, set the retry predicates explicitly, and ignore CallNotPermittedExceptionwhen the retry wraps a circuit breaker.
- Make inner timeouts shorter than outer ones, and propagate deadlines where you can.
- Skip the retry when the remaining deadline can’t fit another attempt.
- Honour Retry-After, parse both formats, clamp it, and add jitter.
- Generate idempotency keys once per operation, persist them when the operation matters, and reuse them across attempts and restarts.
- Resend the same request body on every attempt.
- Cap message redelivery, dead-letter poison messages, and pause consumers when a dependency is down.
- Give operators a way to turn retries down or off during an incident.
- Test sustained failure, recovery, latency just above the timeout, and end-to-end attempt counts.
- Monitor retries as a ratio of first attempts, per dependency.
Every retry runs on the whole fleet
The easy version of retries asks one question: did this call fail?
The version that holds up in production asks a few more. What kind of failure was it? Did the server do the work anyway? How much of the deadline is left? Is a layer above or below already retrying this? How many retries has this client sent in the last few seconds? Is the dependency asking me to slow down?
Whatever retry loop you write will run in every copy of your service, triggered by the same outage at the same moment. Design it for that moment. A good retry policy makes a flaky network invisible to users, and when a dependency is truly down, it backs off quickly and lets it recover.
References
- Marc Brooker, AWS Architecture Blog, “Exponential Backoff And Jitter”.
- Marc Brooker, “Fixing retries with token buckets and circuit breakers”.
- Marc Brooker, “What is Backoff For?”.
- Google SRE Book, Chapter 21: Handling Overload.
- Google SRE Book, Chapter 22: Addressing Cascading Failures.
- gRPC, Retry.
- gRPC, Deadlines.
- AWS SDKs and Tools Reference Guide, Retry behavior.
- RFC 9110, HTTP Semantics, §9.2.2 Idempotent Methods and §10.2.3 Retry-After.
- RFC 9113, HTTP/2, §8.7 Request Reliability.
- RFC 6585, Additional HTTP Status Codes, §4 429 Too Many Requests.
- nginx, proxy_next_upstream.
- Nathan Bronson, Abutalib Aghayev, Aleksey Charapko, Timothy Zhu, “Metastable Failures in Distributed Systems”, HotOS 2021.
- gRPC, Request Hedging.
- Robert M. Metcalfe, David R. Boggs, “Ethernet: Distributed Packet Switching for Local Computer Networks”, Communications of the ACM, 1976.
- Van Jacobson, Michael J. Karels, “Congestion Avoidance and Control”, SIGCOMM 1988.
- Michael T. Nygard, Release It!, Pragmatic Bookshelf (first edition 2007).
- Netflix, Hystrix README: Hystrix Status.
- Finagle, Clients: Retries and RetryBudget.
- Resilience4j, Retry.
- Resilience4j, Spring Boot getting started: Aspect order.
- Spring Framework, Resilience Features: @Retryable.
- Spring Retry, README and @Retryable.
- Polly, Retry resilience strategy.
- Microsoft Learn, Build resilient HTTP apps: standard resilience handler.
- tenacity, Documentation.
- HashiCorp, go-retryablehttp.
- Envoy, Circuit breaker thresholds: retry_budget.
- Resilience4j, IntervalFunctionsource.