Distributed Tracing with OpenTelemetry

In a system of many services, one slow checkout request might pass through the gateway, the order service, the payment service, a database and Kafka. Logs from each service show fragments; metrics show that something is slow but not where. Distributed tracing follows each request across every service and shows a timeline of where the time went.

OpenTelemetry (OTel) is the industry standard for traces, metrics and logs. Spring Boot 4 provides spring-boot-starter-opentelemetry, which configures the OpenTelemetry SDK and Micrometer Tracing and exports over OTLP to backends such as Jaeger, Grafana Tempo, Zipkin, Honeycomb, Datadog or New Relic. This lesson covers traces and spans, setup, context propagation, trace ids in logs, custom spans, and sampling.

Traces, Spans and Context

A trace represents one request's whole journey and has a trace id. It consists of spans — timed operations such as "HTTP GET /api/orders", "SELECT orders" or "send to Kafka" — each with a span id, a parent, attributes and events. When a service calls another, the trace context travels in the W3C traceparent HTTP header (or message headers for Kafka/RabbitMQ), so the downstream spans join the same trace.

Setup in Spring Boot 4

Add spring-boot-starter-opentelemetry and set the OTLP endpoint with management.opentelemetry.tracing.export.otlp.endpoint. Spring instruments incoming and outgoing HTTP (RestClient, WebClient, HTTP service clients built from Spring's builders), JDBC-level observations, Kafka, RabbitMQ and @Observed methods automatically. By default only 10% of requests are sampled; set management.tracing.sampling.probability.

Trace IDs in Logs

When tracing is active, Spring Boot puts traceId and spanId into the logging MDC and includes them in log lines by default. You can jump from an error log to the full trace, or search all logs of one request across services — the single most useful debugging capability in microservices.

Custom Spans and Attributes

Add detail with @Observed, the Observation API, or Micrometer's Tracer for manual spans. Add low-cardinality attributes (payment method, tenant tier) that help you filter traces. Never put secrets or personal data in span attributes.

Sampling in Production

Recording 100% of traces is useful in development but expensive at high traffic. Common strategies: a fixed probability (e.g. 10%), or tail-based sampling in an OpenTelemetry Collector that keeps all error and slow traces and a fraction of the rest.

Examples

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-opentelemetry</artifactId>
</dependency>

# application.yml
spring:
  application:
    name: order-service
management:
  tracing:
    sampling:
      probability: 1.0            # 100% in development
  opentelemetry:
    tracing:
      export:
        otlp:
          endpoint: http://localhost:4318/v1/traces

# compose.yaml — Jaeger with an OTLP receiver and UI on :16686
services:
  jaeger:
    image: jaegertracing/jaeger:latest
    ports:
      - "16686:16686"
      - "4318:4318"

Open http://localhost:16686 -> Service: order-service -> Find Traces

// order-service
@PostMapping("/api/orders")
public OrderDto place(@RequestBody PlaceOrder cmd) {
    log.info("Placing order for {}", cmd.customerEmail());
    PaymentResult payment = paymentClient.charge(cmd.total());   // HTTP service client -> payment-service
    return orders.save(cmd, payment);
}

// payment-service
@PostMapping("/api/payments")
public PaymentResult charge(@RequestBody ChargeRequest req) {
    log.info("Charging {}", req.amount());
    return gateway.charge(req);
}

order-service   INFO [order-service,6f1c2b9e4d3a8f7e1c2b9e4d3a8f7e1c,a1b2c3d4e5f60718] Placing order for asha@webnest.in
payment-service INFO [payment-service,6f1c2b9e4d3a8f7e1c2b9e4d3a8f7e1c,0918f7e6d5c4b3a2] Charging 2999.00
(same trace id in both services)

@Service
public class FraudCheckService {

    private final ObservationRegistry registry;

    public FraudCheckService(ObservationRegistry registry) {
        this.registry = registry;
    }

    public FraudScore check(Order order) {
        return Observation.createNotStarted("fraud.check", registry)
            .contextualName("fraud-check")
            .lowCardinalityKeyValue("payment.method", order.paymentMethod())
            .observe(() -> {
                FraudScore score = model.score(order);
                if (score.value() > 0.8) {
                    Span.current().addEvent("high-risk-order");   // OpenTelemetry API
                }
                return score;
            });
    }
}

Jaeger timeline for trace 6f1c2b9e...
POST /api/orders                       order-service     412 ms
 ├─ fraud-check                        order-service      38 ms  payment.method=card  event: high-risk-order
 ├─ POST /api/payments                 order-service →   290 ms
 │   └─ POST /api/payments             payment-service   281 ms
 │       └─ HTTP POST gateway.example  payment-service   255 ms   <- the slow part
 └─ INSERT orders                      order-service      12 ms