Resilience: Retries, Concurrency Limits and Circuit Breakers

Every remote call eventually fails: a network blip, a deployment on the other side, a database failover, a rate-limited partner API. A resilient application expects this. It retries transient failures, stops hammering a service that is clearly down, limits how much concurrent load it sends, and degrades gracefully with fallbacks instead of failing completely.

Spring Framework 7 brings retry and concurrency limiting into core Spring with @Retryable and @ConcurrencyLimit, and circuit breakers come from Resilience4j (via Spring Cloud Circuit Breaker). This lesson covers each pattern, when to use it, how to configure it, and how to combine them safely.

Retries with @Retryable

Annotate a method with @Retryable and enable processing with @EnableResilientMethods on a configuration class. On an exception, Spring retries the call with a configurable maxRetries, delay, multiplier (exponential back-off), maxDelay and jitter. Restrict retries to transient failures with includes/excludes. The programmatic RetryTemplate in org.springframework.core.retry offers the same without annotations. This replaces the separate Spring Retry library for most uses.

When Not to Retry

Only retry operations that are safe to repeat (idempotent) and failures that may succeed next time: timeouts, 503, connection resets. Do not retry 400 Bad Request, authentication failures or business rule violations. Beware of retry storms: if every layer retries 3 times, one failing call can become 27. Retry at one layer, with back-off and jitter.

Concurrency Limits and Bulkheads

@ConcurrencyLimit(n) restricts how many threads may execute a method at once — a bulkhead that protects a fragile dependency (or your own resources) from being overwhelmed. It is especially important with virtual threads, where thousands of concurrent requests could otherwise all hit a small downstream service at the same moment.

Circuit Breakers

A circuit breaker watches calls to a dependency. When the failure rate crosses a threshold, it opens and fails calls immediately (usually with a fallback) instead of waiting for timeouts, giving the dependency time to recover. After a wait it goes half-open, lets a few trial calls through, and closes again if they succeed. Use Resilience4j through spring-cloud-starter-circuitbreaker-resilience4j and the CircuitBreakerFactory abstraction, configured with resilience4j.circuitbreaker.* properties.

Timeouts and Fallbacks

Every remote call needs a timeout — otherwise no other pattern helps. Fallbacks should be honest: cached or default data, a "try again later" response, or queuing the work for later — never silently pretending an operation succeeded.

Examples

@SpringBootApplication
@EnableResilientMethods
public class ShopApplication { ... }

@Service
public class ExchangeRateClient {

    private static final Logger log = LoggerFactory.getLogger(ExchangeRateClient.class);
    private final RestClient client;

    public ExchangeRateClient(RestClient.Builder builder) {
        this.client = builder.baseUrl("https://rates.example.com").build();
    }

    @Retryable(includes = {ResourceAccessException.class, HttpServerErrorException.ServiceUnavailable.class},
               maxRetries = 3, delay = 200, multiplier = 2, maxDelay = 2000, jitter = 50)
    public Rates latest(String base) {
        log.info("Fetching rates for {}", base);
        return client.get().uri("/latest/{base}", base).retrieve().body(Rates.class);
    }
}

INFO Fetching rates for INR        -> 503 Service Unavailable
INFO Fetching rates for INR        (after ~200 ms) -> 503
INFO Fetching rates for INR        (after ~400 ms) -> 200 OK
(a 404 or 400 would not be retried)

RetryPolicy policy = RetryPolicy.builder()
    .maxRetries(4)
    .delay(Duration.ofMillis(100))
    .multiplier(2)
    .includes(TransientDataAccessException.class)
    .build();

RetryTemplate retry = new RetryTemplate(policy);

Order saved = retry.execute(() -> orderRepository.save(order));

attempt 1 -> CannotAcquireLockException (transient)
attempt 2 -> success

@Service
public class LegacyInvoiceClient {

    // The legacy system falls over above ~10 parallel requests
    @ConcurrencyLimit(10)
    public Invoice render(String orderId) {
        return legacy.renderInvoice(orderId);
    }
}

200 concurrent requests (virtual threads) -> at most 10 inside render() at any time; the rest wait their turn

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-circuitbreaker-resilience4j</artifactId>
</dependency>
<!-- import spring-cloud-dependencies 2025.1.x BOM -->

# application.yml
resilience4j:
  circuitbreaker:
    instances:
      recommendations:
        sliding-window-size: 20
        failure-rate-threshold: 50           # open when 50% of the last 20 calls fail
        wait-duration-in-open-state: 30s
        permitted-number-of-calls-in-half-open-state: 3
  timelimiter:
    instances:
      recommendations:
        timeout-duration: 800ms

@Service
public class RecommendationsFacade {

    private final CircuitBreaker breaker;
    private final RecommendationsClient client;
    private final PopularCourses popular;

    public RecommendationsFacade(CircuitBreakerFactory<?, ?> factory,
                                 RecommendationsClient client, PopularCourses popular) {
        this.breaker = factory.create("recommendations");
        this.client = client;
        this.popular = popular;
    }

    public List<Course> forStudent(long studentId) {
        return breaker.run(
            () -> client.personalised(studentId),
            throwable -> popular.top10());            // honest fallback: popular courses
    }
}

calls 1-10: recommendations service timing out -> fallback popular.top10()
circuit OPEN  -> next calls return the fallback instantly (no 800 ms wait)
after 30 s: HALF_OPEN -> 3 trial calls succeed -> CLOSED again

GET /actuator/health
{"components":{"circuitBreakers":{"details":{"recommendations":{"status":"CIRCUIT_OPEN", ...}}}}}