Apache Kafka with Spring Boot

Apache Kafka is the backbone of event-driven architectures at companies of every size. It is a distributed, durable log: producers append events to topics, and any number of consumer groups read them at their own pace, even days later. Order placed, payment received, lesson completed — each becomes an event other services react to without calling each other directly.

Spring for Apache Kafka, auto-configured by spring-boot-starter-kafka in Spring Boot 4, gives you KafkaTemplate for sending and @KafkaListener for consuming. This lesson covers Kafka's core concepts, running Kafka locally, producing and consuming JSON events with Jackson 3 serializers, keys and partitions, error handling with retries and dead-letter topics, and testing with Testcontainers.

Kafka Concepts

The vocabulary you need:

Topic — a named stream of events, split into partitions for parallelism.
Key — events with the same key always go to the same partition, so they are processed in order (e.g. all events for order 42).
Offset — the position of an event in a partition; consumers commit offsets to remember progress.
Consumer group — instances sharing a group id split the partitions between them; different groups each get every event.
Retention — events stay for a configured time (days) or forever with compaction, regardless of whether they were consumed.

Producing Events

KafkaTemplate.send(topic, key, value) returns a CompletableFuture<SendResult>. Configure serializers in spring.kafka.producer.*: StringSerializer for keys and, in Spring Boot 4, JacksonJsonSerializer (Jackson 3) for JSON values. For reliability set acks=all and keep idempotence enabled (the default in modern Kafka clients), so retries never create duplicates within a partition.

Consuming Events

@KafkaListener(topics = "orders", groupId = "email-service") on a bean method receives deserialised events. Spring manages polling, threading and offset commits. Increase concurrency to use more consumer threads (up to the number of partitions). Kafka delivers at least once: after a crash, an event may be redelivered, so listeners must be idempotent — processing the same event twice must be harmless.

Error Handling, Retries and Dead Letters

When a listener throws, Spring's DefaultErrorHandler retries with a back-off and, after the last attempt, can hand the record to a DeadLetterPublishingRecoverer that sends it to <topic>-dlt so one bad message does not block the partition forever. @RetryableTopic offers non-blocking retries through separate retry topics. Deserialisation errors need the ErrorHandlingDeserializer wrapper, otherwise a malformed message causes an endless failure loop.

Transactional Outbox

Writing to the database and then sending to Kafka is not atomic: the send can fail after the commit, or succeed before a rollback. The robust pattern is the transactional outbox: store the event in an outbox table in the same database transaction, and let a separate process (a scheduled publisher, Debezium CDC, or Spring Modulith's externalised events) publish it to Kafka.

Examples

Running Kafka locally and configuring Spring Boot

Java

# compose.yaml — single-node Kafka in KRaft mode (no ZooKeeper)
services:
  kafka:
    image: apache/kafka:4.1.0
    ports:
      - "9092:9092"

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-kafka</artifactId>
</dependency>

# application.yml
spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JacksonJsonSerializer
      acks: all
    consumer:
      group-id: webnest-shop
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JacksonJsonDeserializer
        spring.json.trusted.packages: com.webnest.shop.events

Output

Kafka version: 4.1.0
[Producer clientId=producer-1] Cluster ID: 5L6g3nShT-eMCtK--X86sw

Declaring a topic and producing events with a key

Java

public record OrderPlaced(String orderId, String customerEmail, BigDecimal total, Instant placedAt) {}

@Configuration
public class TopicConfig {
    @Bean
    NewTopic ordersTopic() {
        return TopicBuilder.name("orders.placed").partitions(6).replicas(1).build();
    }
}

@Service
public class OrderEventsProducer {

    private static final Logger log = LoggerFactory.getLogger(OrderEventsProducer.class);
    private final KafkaTemplate<String, OrderPlaced> kafka;

    public OrderEventsProducer(KafkaTemplate<String, OrderPlaced> kafka) {
        this.kafka = kafka;
    }

    public void publish(OrderPlaced event) {
        kafka.send("orders.placed", event.orderId(), event)          // key = orderId keeps per-order ordering
            .whenComplete((result, ex) -> {
                if (ex != null) {
                    log.error("Failed to publish {}", event.orderId(), ex);
                } else {
                    RecordMetadata m = result.getRecordMetadata();
                    log.info("Published {} to partition {} offset {}", event.orderId(), m.partition(), m.offset());
                }
            });
    }
}

Output

INFO Published WN-10231 to partition 3 offset 0
INFO Published WN-10232 to partition 1 offset 0
INFO Published WN-10231 to partition 3 offset 1   (same key -> same partition)

Idempotent consumers in two independent consumer groups

Java

@Component
public class ConfirmationEmailConsumer {

    private final ProcessedEventRepository processed;
    private final MailService mail;

    public ConfirmationEmailConsumer(ProcessedEventRepository processed, MailService mail) {
        this.processed = processed;
        this.mail = mail;
    }

    @KafkaListener(topics = "orders.placed", groupId = "email-service", concurrency = "3")
    @Transactional
    public void onOrderPlaced(OrderPlaced event, @Header(KafkaHeaders.RECEIVED_PARTITION) int partition) {
        if (!processed.markIfNew("email:" + event.orderId())) {
            return;                                     // duplicate delivery — already handled
        }
        mail.sendOrderConfirmation(event.customerEmail(), event.orderId(), event.total());
    }
}

@Component
public class AnalyticsConsumer {
    @KafkaListener(topics = "orders.placed", groupId = "analytics-service")
    public void record(OrderPlaced event) {
        System.out.println("Revenue +" + event.total() + " at " + event.placedAt());
    }
}

Output

email-service     : confirmation sent for WN-10231
analytics-service : Revenue +2999.00 at 2026-09-27T10:12:03Z
(both groups receive every event; within a group partitions are shared across 3 threads)

Retries with back-off and a dead-letter topic

Java

@Configuration
public class KafkaErrorConfig {

    @Bean
    DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
        DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template);
        ExponentialBackOff backOff = new ExponentialBackOff(1000, 2.0);
        backOff.setMaxElapsedTime(10_000);                 // ~4 attempts over 10 s
        DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);
        handler.addNotRetryableExceptions(ValidationException.class);   // pointless to retry
        return handler;
    }
}

@KafkaListener(topics = "orders.placed.dlt", groupId = "dlt-monitor")
public void deadLetters(OrderPlaced event,
                        @Header(KafkaHeaders.DLT_EXCEPTION_MESSAGE) String error) {
    log.error("Order event {} dead-lettered: {}", event.orderId(), error);
}

Output

WARN  Retrying record orders.placed-3@7 (attempt 2) after SmtpTimeoutException
WARN  Retrying record orders.placed-3@7 (attempt 3)
ERROR Order event WN-10240 dead-lettered: Listener failed; SmtpTimeoutException: connect timed out
(the partition continues with the next record)

Integration test with a Kafka container

Java

@SpringBootTest
@Testcontainers
class OrderEventsIT {

    @Container
    @ServiceConnection
    static KafkaContainer kafka = new KafkaContainer("apache/kafka:4.1.0");

    @Autowired OrderEventsProducer producer;
    @MockitoBean MailService mail;

    @Test
    void emailIsSentWhenOrderIsPlaced() {
        producer.publish(new OrderPlaced("WN-1", "asha@webnest.in", new BigDecimal("2999"), Instant.now()));

        await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->
            verify(mail).sendOrderConfirmation("asha@webnest.in", "WN-1", new BigDecimal("2999")));
    }
}

Output

Container apache/kafka:4.1.0 started
OrderEventsIT > emailIsSentWhenOrderIsPlaced() PASSED (4.8 s)

Common Mistakes

Depending only on spring-kafka in Spring Boot 4 instead of spring-boot-starter-kafka, losing auto-configuration.
Using the Jackson 2 based JsonSerializer/JsonDeserializer in Boot 4 instead of JacksonJsonSerializer/JacksonJsonDeserializer.
Writing non-idempotent consumers; redelivery after a crash then double-charges or double-emails.
Sending events without a key when ordering per entity matters.
Publishing to Kafka inside a database transaction and assuming both succeed or fail together; use an outbox.

Key Points to Remember

Kafka stores events durably in partitioned topics; consumer groups read independently.
spring-boot-starter-kafka provides KafkaTemplate and @KafkaListener with spring.kafka.* configuration.
Use keys for per-entity ordering and acks=all with idempotent producers for reliability.
Consumers must be idempotent; use DefaultErrorHandler with back-off and dead-letter topics.
Use the transactional outbox for consistent database + Kafka updates, and test with KafkaContainer.

Practice the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

Use your local JDK or project IDE for these examples. Codelab currently runs Python and HTML/CSS/JavaScript; framework examples may need project dependencies.