Course topics

By WebNest Studio Engineering Team

Spring Boot Tutorial

Distributed Tracing with OpenTelemetry

In a system of many services, one slow checkout request might pass through the gateway, the order service, the payment service, a database and Kafka. Logs from each service show fragments; metrics show that something is slow but not where. Distributed tracing follows each request across every service and shows a timeline of where the time went.

OpenTelemetry (OTel) is the industry standard for traces, metrics and logs. Spring Boot 4 provides spring-boot-starter-opentelemetry, which configures the OpenTelemetry SDK and Micrometer Tracing and exports over OTLP to backends such as Jaeger, Grafana Tempo, Zipkin, Honeycomb, Datadog or New Relic. This lesson covers traces and spans, setup, context propagation, trace ids in logs, custom spans, and sampling.

Traces, Spans and Context

A trace represents one request's whole journey and has a trace id. It consists of spans — timed operations such as "HTTP GET /api/orders", "SELECT orders" or "send to Kafka" — each with a span id, a parent, attributes and events. When a service calls another, the trace context travels in the W3C traceparent HTTP header (or message headers for Kafka/RabbitMQ), so the downstream spans join the same trace.

Setup in Spring Boot 4

Add spring-boot-starter-opentelemetry and set the OTLP endpoint with management.opentelemetry.tracing.export.otlp.endpoint. Spring instruments incoming and outgoing HTTP (RestClient, WebClient, HTTP service clients built from Spring's builders), JDBC-level observations, Kafka, RabbitMQ and @Observed methods automatically. By default only 10% of requests are sampled; set management.tracing.sampling.probability.

Trace IDs in Logs

When tracing is active, Spring Boot puts traceId and spanId into the logging MDC and includes them in log lines by default. You can jump from an error log to the full trace, or search all logs of one request across services — the single most useful debugging capability in microservices.

Custom Spans and Attributes

Add detail with @Observed, the Observation API, or Micrometer's Tracer for manual spans. Add low-cardinality attributes (payment method, tenant tier) that help you filter traces. Never put secrets or personal data in span attributes.

Sampling in Production

Recording 100% of traces is useful in development but expensive at high traffic. Common strategies: a fixed probability (e.g. 10%), or tail-based sampling in an OpenTelemetry Collector that keeps all error and slow traces and a fraction of the rest.

Examples

Dependencies, configuration and a local Jaeger

Java
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-opentelemetry</artifactId>
</dependency>

# application.yml
spring:
  application:
    name: order-service
management:
  tracing:
    sampling:
      probability: 1.0            # 100% in development
  opentelemetry:
    tracing:
      export:
        otlp:
          endpoint: http://localhost:4318/v1/traces

# compose.yaml — Jaeger with an OTLP receiver and UI on :16686
services:
  jaeger:
    image: jaegertracing/jaeger:latest
    ports:
      - "16686:16686"
      - "4318:4318"
Output
Open http://localhost:16686 -> Service: order-service -> Find Traces

Trace ids in logs across two services

Java
// order-service
@PostMapping("/api/orders")
public OrderDto place(@RequestBody PlaceOrder cmd) {
    log.info("Placing order for {}", cmd.customerEmail());
    PaymentResult payment = paymentClient.charge(cmd.total());   // HTTP service client -> payment-service
    return orders.save(cmd, payment);
}

// payment-service
@PostMapping("/api/payments")
public PaymentResult charge(@RequestBody ChargeRequest req) {
    log.info("Charging {}", req.amount());
    return gateway.charge(req);
}
Output
order-service   INFO [order-service,6f1c2b9e4d3a8f7e1c2b9e4d3a8f7e1c,a1b2c3d4e5f60718] Placing order for asha@webnest.in
payment-service INFO [payment-service,6f1c2b9e4d3a8f7e1c2b9e4d3a8f7e1c,0918f7e6d5c4b3a2] Charging 2999.00
(same trace id in both services)

A custom span with attributes and events

Java
@Service
public class FraudCheckService {

    private final ObservationRegistry registry;

    public FraudCheckService(ObservationRegistry registry) {
        this.registry = registry;
    }

    public FraudScore check(Order order) {
        return Observation.createNotStarted("fraud.check", registry)
            .contextualName("fraud-check")
            .lowCardinalityKeyValue("payment.method", order.paymentMethod())
            .observe(() -> {
                FraudScore score = model.score(order);
                if (score.value() > 0.8) {
                    Span.current().addEvent("high-risk-order");   // OpenTelemetry API
                }
                return score;
            });
    }
}
Output
Jaeger timeline for trace 6f1c2b9e...
POST /api/orders                       order-service     412 ms
 ├─ fraud-check                        order-service      38 ms  payment.method=card  event: high-risk-order
 ├─ POST /api/payments                 order-service →   290 ms
 │   └─ POST /api/payments             payment-service   281 ms
 │       └─ HTTP POST gateway.example  payment-service   255 ms   <- the slow part
 └─ INSERT orders                      order-service      12 ms

Common Mistakes

  • Creating RestClient/WebClient instances with static create() methods instead of Spring's builders, so trace context is not propagated.
  • Sampling 100% of production traffic and overwhelming the tracing backend and budget.
  • Recording personal data, tokens or request bodies as span attributes.
  • Losing trace context in @Async or manually created threads — use context-propagating executors.
  • Setting up tracing but not including trace ids in logs, missing the easiest way to correlate.

Exercise

A teammate is about to make this mistake: Creating RestClient/WebClient instances with static create() methods instead of Spring's builders, so trace context is not propagated. Use Traces, Spans and Context to review the decision. Write a corrected implementation or design sketch that follows A trace follows one request across services; spans are timed operations inside it., then name one edge case you would check.

Show hints
  • Start with Traces, Spans and Context and compare your approach with Dependencies, configuration and a local Jaeger. Identify the decision or behavior the lesson recommends.
  • Walk through your scenario step by step. Confirm that it follows A trace follows one request across services; spans are timed operations inside it. and that it would catch the failure described by Creating RestClient/WebClient instances with static create() methods instead of Spring's builders, so trace context is not propagated..

Quick quiz

1. Which choice best describes the recommended approach in Distributed Tracing with OpenTelemetry?

  • A. Creating RestClient/WebClient instances with static create() methods instead of Spring's builders, so trace context is not propagated.
  • B. Sampling 100% of production traffic and overwhelming the tracing backend and budget.
  • C. A trace follows one request across services; spans are timed operations inside it.
Show answer

C. This is the recommended approach: A trace follows one request across services; spans are timed operations inside it. Avoid: Creating RestClient/WebClient instances with static create() methods instead of Spring's builders, so trace context is not propagated.

Interview questions

Why does this principle matter in Distributed Tracing with OpenTelemetry: A trace follows one request across services; spans are timed operations inside it.

Show answer

A trace follows one request across services; spans are timed operations inside it. Apply this in Traces, Spans and Context; the example Dependencies, configuration and a local Jaeger shows one way to do it. Avoid Creating RestClient/WebClient instances with static create() methods instead of Spring's builders, so trace context is not propagated. and explain how your implementation or design prevents that failure.

Key Points to Remember

  • A trace follows one request across services; spans are timed operations inside it.
  • Spring Boot 4: spring-boot-starter-opentelemetry + management.opentelemetry.tracing.export.otlp.endpoint.
  • Context propagates via W3C traceparent headers through Spring's HTTP clients and messaging.
  • traceId and spanId appear in logs automatically for correlation.
  • Tune management.tracing.sampling.probability and add spans with @Observed or the Observation API.

Practice in the playground

Write and run your own code for this topic, or start from the exercise or an example. Runs plain Java 17. Frameworks such as Spring, and databases, are not available here. Your code for each topic is kept in this browser.

Run the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

These examples need a JDK project with their libraries on your computer, so they can be copied but not run here. Codelab runs Python, FastAPI backends with SQLite, plain Java 17 and HTML/CSS/JavaScript.

WebNest Studio

Written by

WebNest Studio Engineering Team

Practical lessons from developers working with Java, Python, React, databases and production systems.

Led by Vansh Duggal, Founder, WebNest Studio · Full-stack & software engineering