Course topics

By WebNest Studio Engineering Team

Spring Boot Tutorial

Apache Kafka with Spring Boot

Apache Kafka is the backbone of event-driven architectures at companies of every size. It is a distributed, durable log: producers append events to topics, and any number of consumer groups read them at their own pace, even days later. Order placed, payment received, lesson completed — each becomes an event other services react to without calling each other directly.

Spring for Apache Kafka, auto-configured by spring-boot-starter-kafka in Spring Boot 4, gives you KafkaTemplate for sending and @KafkaListener for consuming. This lesson covers Kafka's core concepts, running Kafka locally, producing and consuming JSON events with Jackson 3 serializers, keys and partitions, error handling with retries and dead-letter topics, and testing with Testcontainers.

Kafka Concepts

The vocabulary you need:

  • Topic — a named stream of events, split into partitions for parallelism.
  • Key — events with the same key always go to the same partition, so they are processed in order (e.g. all events for order 42).
  • Offset — the position of an event in a partition; consumers commit offsets to remember progress.
  • Consumer group — instances sharing a group id split the partitions between them; different groups each get every event.
  • Retention — events stay for a configured time (days) or forever with compaction, regardless of whether they were consumed.

Producing Events

KafkaTemplate.send(topic, key, value) returns a CompletableFuture<SendResult>. Configure serializers in spring.kafka.producer.*: StringSerializer for keys and, in Spring Boot 4, JacksonJsonSerializer (Jackson 3) for JSON values. For reliability set acks=all and keep idempotence enabled (the default in modern Kafka clients), so retries never create duplicates within a partition.

Consuming Events

@KafkaListener(topics = "orders", groupId = "email-service") on a bean method receives deserialised events. Spring manages polling, threading and offset commits. Increase concurrency to use more consumer threads (up to the number of partitions). Kafka delivers at least once: after a crash, an event may be redelivered, so listeners must be idempotent — processing the same event twice must be harmless.

Error Handling, Retries and Dead Letters

When a listener throws, Spring's DefaultErrorHandler retries with a back-off and, after the last attempt, can hand the record to a DeadLetterPublishingRecoverer that sends it to <topic>-dlt so one bad message does not block the partition forever. @RetryableTopic offers non-blocking retries through separate retry topics. Deserialisation errors need the ErrorHandlingDeserializer wrapper, otherwise a malformed message causes an endless failure loop.

Transactional Outbox

Writing to the database and then sending to Kafka is not atomic: the send can fail after the commit, or succeed before a rollback. The robust pattern is the transactional outbox: store the event in an outbox table in the same database transaction, and let a separate process (a scheduled publisher, Debezium CDC, or Spring Modulith's externalised events) publish it to Kafka.

Examples

Running Kafka locally and configuring Spring Boot

Java
# compose.yaml — single-node Kafka in KRaft mode (no ZooKeeper)
services:
  kafka:
    image: apache/kafka:4.1.0
    ports:
      - "9092:9092"

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-kafka</artifactId>
</dependency>

# application.yml
spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JacksonJsonSerializer
      acks: all
    consumer:
      group-id: webnest-shop
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JacksonJsonDeserializer
        spring.json.trusted.packages: com.webnest.shop.events
Output
Kafka version: 4.1.0
[Producer clientId=producer-1] Cluster ID: 5L6g3nShT-eMCtK--X86sw

Declaring a topic and producing events with a key

Java
public record OrderPlaced(String orderId, String customerEmail, BigDecimal total, Instant placedAt) {}

@Configuration
public class TopicConfig {
    @Bean
    NewTopic ordersTopic() {
        return TopicBuilder.name("orders.placed").partitions(6).replicas(1).build();
    }
}

@Service
public class OrderEventsProducer {

    private static final Logger log = LoggerFactory.getLogger(OrderEventsProducer.class);
    private final KafkaTemplate<String, OrderPlaced> kafka;

    public OrderEventsProducer(KafkaTemplate<String, OrderPlaced> kafka) {
        this.kafka = kafka;
    }

    public void publish(OrderPlaced event) {
        kafka.send("orders.placed", event.orderId(), event)          // key = orderId keeps per-order ordering
            .whenComplete((result, ex) -> {
                if (ex != null) {
                    log.error("Failed to publish {}", event.orderId(), ex);
                } else {
                    RecordMetadata m = result.getRecordMetadata();
                    log.info("Published {} to partition {} offset {}", event.orderId(), m.partition(), m.offset());
                }
            });
    }
}
Output
INFO Published WN-10231 to partition 3 offset 0
INFO Published WN-10232 to partition 1 offset 0
INFO Published WN-10231 to partition 3 offset 1   (same key -> same partition)

Idempotent consumers in two independent consumer groups

Java
@Component
public class ConfirmationEmailConsumer {

    private final ProcessedEventRepository processed;
    private final MailService mail;

    public ConfirmationEmailConsumer(ProcessedEventRepository processed, MailService mail) {
        this.processed = processed;
        this.mail = mail;
    }

    @KafkaListener(topics = "orders.placed", groupId = "email-service", concurrency = "3")
    @Transactional
    public void onOrderPlaced(OrderPlaced event, @Header(KafkaHeaders.RECEIVED_PARTITION) int partition) {
        if (!processed.markIfNew("email:" + event.orderId())) {
            return;                                     // duplicate delivery — already handled
        }
        mail.sendOrderConfirmation(event.customerEmail(), event.orderId(), event.total());
    }
}

@Component
public class AnalyticsConsumer {
    @KafkaListener(topics = "orders.placed", groupId = "analytics-service")
    public void record(OrderPlaced event) {
        System.out.println("Revenue +" + event.total() + " at " + event.placedAt());
    }
}
Output
email-service     : confirmation sent for WN-10231
analytics-service : Revenue +2999.00 at 2026-09-27T10:12:03Z
(both groups receive every event; within a group partitions are shared across 3 threads)

Retries with back-off and a dead-letter topic

Java
@Configuration
public class KafkaErrorConfig {

    @Bean
    DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
        DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template);
        ExponentialBackOff backOff = new ExponentialBackOff(1000, 2.0);
        backOff.setMaxElapsedTime(10_000);                 // ~4 attempts over 10 s
        DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);
        handler.addNotRetryableExceptions(ValidationException.class);   // pointless to retry
        return handler;
    }
}

@KafkaListener(topics = "orders.placed.dlt", groupId = "dlt-monitor")
public void deadLetters(OrderPlaced event,
                        @Header(KafkaHeaders.DLT_EXCEPTION_MESSAGE) String error) {
    log.error("Order event {} dead-lettered: {}", event.orderId(), error);
}
Output
WARN  Retrying record orders.placed-3@7 (attempt 2) after SmtpTimeoutException
WARN  Retrying record orders.placed-3@7 (attempt 3)
ERROR Order event WN-10240 dead-lettered: Listener failed; SmtpTimeoutException: connect timed out
(the partition continues with the next record)

Integration test with a Kafka container

Java
@SpringBootTest
@Testcontainers
class OrderEventsIT {

    @Container
    @ServiceConnection
    static KafkaContainer kafka = new KafkaContainer("apache/kafka:4.1.0");

    @Autowired OrderEventsProducer producer;
    @MockitoBean MailService mail;

    @Test
    void emailIsSentWhenOrderIsPlaced() {
        producer.publish(new OrderPlaced("WN-1", "asha@webnest.in", new BigDecimal("2999"), Instant.now()));

        await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->
            verify(mail).sendOrderConfirmation("asha@webnest.in", "WN-1", new BigDecimal("2999")));
    }
}
Output
Container apache/kafka:4.1.0 started
OrderEventsIT > emailIsSentWhenOrderIsPlaced() PASSED (4.8 s)

Common Mistakes

  • Depending only on spring-kafka in Spring Boot 4 instead of spring-boot-starter-kafka, losing auto-configuration.
  • Using the Jackson 2 based JsonSerializer/JsonDeserializer in Boot 4 instead of JacksonJsonSerializer/JacksonJsonDeserializer.
  • Writing non-idempotent consumers; redelivery after a crash then double-charges or double-emails.
  • Sending events without a key when ordering per entity matters.
  • Publishing to Kafka inside a database transaction and assuming both succeed or fail together; use an outbox.

Exercise

A teammate is about to make this mistake: Depending only on spring-kafka in Spring Boot 4 instead of spring-boot-starter-kafka, losing auto-configuration. Use Kafka Concepts to review the decision. Write a corrected implementation or design sketch that follows Kafka stores events durably in partitioned topics; consumer groups read independently., then name one edge case you would check.

Show hints
  • Start with Kafka Concepts and compare your approach with Running Kafka locally and configuring Spring Boot. Identify the decision or behavior the lesson recommends.
  • Walk through your scenario step by step. Confirm that it follows Kafka stores events durably in partitioned topics; consumer groups read independently. and that it would catch the failure described by Depending only on spring-kafka in Spring Boot 4 instead of spring-boot-starter-kafka, losing auto-configuration..

Quick quiz

1. Which choice best describes the recommended approach in Apache Kafka with Spring Boot?

  • A. Depending only on spring-kafka in Spring Boot 4 instead of spring-boot-starter-kafka, losing auto-configuration.
  • B. Using the Jackson 2 based JsonSerializer/JsonDeserializer in Boot 4 instead of JacksonJsonSerializer/JacksonJsonDeserializer.
  • C. Kafka stores events durably in partitioned topics; consumer groups read independently.
Show answer

C. This is the recommended approach: Kafka stores events durably in partitioned topics; consumer groups read independently. Avoid: Depending only on spring-kafka in Spring Boot 4 instead of spring-boot-starter-kafka, losing auto-configuration.

Interview questions

Why does this principle matter in Apache Kafka with Spring Boot: Kafka stores events durably in partitioned topics; consumer groups read independently.

Show answer

Kafka stores events durably in partitioned topics; consumer groups read independently. Apply this in Kafka Concepts; the example Running Kafka locally and configuring Spring Boot shows one way to do it. Avoid Depending only on spring-kafka in Spring Boot 4 instead of spring-boot-starter-kafka, losing auto-configuration. and explain how your implementation or design prevents that failure.

Key Points to Remember

  • Kafka stores events durably in partitioned topics; consumer groups read independently.
  • spring-boot-starter-kafka provides KafkaTemplate and @KafkaListener with spring.kafka.* configuration.
  • Use keys for per-entity ordering and acks=all with idempotent producers for reliability.
  • Consumers must be idempotent; use DefaultErrorHandler with back-off and dead-letter topics.
  • Use the transactional outbox for consistent database + Kafka updates, and test with KafkaContainer.

Practice in the playground

Write and run your own code for this topic, or start from the exercise or an example. Runs plain Java 17. Frameworks such as Spring, and databases, are not available here. Your code for each topic is kept in this browser.

Run the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

These examples need a JDK project with their libraries on your computer, so they can be copied but not run here. Codelab runs Python, FastAPI backends with SQLite, plain Java 17 and HTML/CSS/JavaScript.

WebNest Studio

Written by

WebNest Studio Engineering Team

Practical lessons from developers working with Java, Python, React, databases and production systems.

Led by Vansh Duggal, Founder, WebNest Studio · Full-stack & software engineering