Jaeger Trace MDC

Simple Demo Code

1. Solution Overview

1.1 Background and Objectives

In distributed systems, cross-service request link tracing is a key means of locating problems. This solution aims to implement deep integration of Jaeger distributed tracing and SLF4J MDC (Mapped Diagnostic Context), enabling:

• Logs automatically carry traceId/spanId, facilitating log aggregation queries
• Support cross-service trace context passing
• Maintain trace context accuracy in asynchronous scenarios like thread pools

1.2 Core Value

• Problem Location Efficiency Improvement: Quickly correlate all relevant logs in distributed systems via traceId
• Zero Intrusiveness: Business code does not need to manually manage MDC, automatic injection and cleanup
• Thread Safety: Support nested Spans and thread pool reuse scenarios

2. Architecture Design

2.1 Overall Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                     Application Layer                        │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐  │
│  │ Controller   │───>│  Service     │───>│   DAO        │  │
│  │ (HTTP Entry) │    │ (Business)   │    │ (Database)   │  │
│  └──────────────┘    └──────────────┘    └──────────────┘  │
│         │                    │                    │          │
│         └────────────────────┴────────────────────┘          │
│                              │                                │
47: ├──────────────────────────────┼────────────────────────────────┤
│          Tracing Layer       ▼                                │
│  ┌───────────────────────────────────────────────┐           │
│  │     CustomMDCScopeManager (Core Component)    │           │
│  │  ┌─────────────────────────────────────┐     │           │
│  │  │  ThreadLocal<CustomMDCScope>        │     │           │
│  │  │  - Manage Scope Lifecycle           │     │           │
│  │  │  - Maintain Span Stack              │     │           │
│  │  └─────────────────────────────────────┘     │           │
│  │                                               │           │
│  │  ┌─────────────────────────────────────┐     │           │
│  │  │  CustomMDCScope (Scope Impl)        │     │           │
│  │  │  - MDC Snapshot and Restore         │     │           │
│  │  │  - Support Nested Span              │     │           │
│  │  └─────────────────────────────────────┘     │           │
│  └───────────────────────────────────────────────┘           │
│                              │                                │
├──────────────────────────────┼────────────────────────────────┤
│         Logging Layer        ▼                                │
│  ┌───────────────────────────────────────────────┐           │
│  │              SLF4J MDC                        │           │
│  │  ┌─────────────────────────────────────┐     │           │
│  │  │  traceId:  04bf92f3577b34da...      │     │           │
│  │  │  spanId:   36bd32b7a5712a1a         │     │           │
│  │  │  parentId: 00f067aa0ba902b7         │     │           │
│  │  └─────────────────────────────────────┘     │           │
│  └───────────────────────────────────────────────┘           │
│                              │                                │
└──────────────────────────────┼────────────────────────────────┘
                               ▼
                    ┌────────────────────┐
                    │  Jaeger Collector  │
                    │  (Trace Storage)   │
                    └────────────────────┘

2.2 Core Component Description

2.2.1 CustomMDCScopeManager

Responsibilities: Implement OpenTracing's ScopeManager interface, manage Span activation and propagation

Key Implementation:

private final ThreadLocal<CustomMDCScope> tlsScope = new ThreadLocal<>();

@Override
public Scope activate(Span span) {
    return new CustomMDCScope(span);
}

@Override
public Span activeSpan() {
    CustomMDCScope scope = tlsScope.get();
    return scope == null ? null : scope.wrapped;
}

Design Points:

• Use ThreadLocal to ensure thread isolation
• Support Scope nesting (maintain previous through linked list structure)
• Implement lazy activation: Inject MDC only when calling activate()

2.2.2 CustomMDCScope

Responsibilities: Implement OpenTracing's Scope interface, manage single Span lifecycle

Core Mechanism: Snapshot-Inject-Restore

CustomMDCScope(Span span) {
    // 1. Save current MDC snapshot
    this.previousTraceId = MDC.get("traceId");
    this.previousSpanId = MDC.get("spanId");
    
    // 2. Establish linked list relationship (support nesting)
    this.previous = CustomMDCScopeManager.this.tlsScope.get();
    CustomMDCScopeManager.this.tlsScope.set(this);
    
    // 3. Inject new trace context
    MDC.put("traceId", span.context().toTraceId());
    MDC.put("spanId", span.context().toSpanId());
}

@Override
public void close() {
    // 4. Restore to previous Scope
    CustomMDCScopeManager.this.tlsScope.set(previous);
    
    // 5. Restore MDC to snapshot state
    restoreMDC("traceId", previousTraceId);
    restoreMDC("spanId", previousSpanId);
}

3. Key Process Design

3.1 Process of Receiving Remote Trace Context

┌─────────────┐
│ HTTP Request│
│  Headers    │
│ uber-trace-id: 4bf92f3577b...│
└──────┬──────┘
       │
       ▼
┌──────────────────────────────────┐
│ 1. Parse HTTP Header             │
│    - Extract traceId (128-bit)   │
│    - Extract parentSpanId (64-bit)│
│    - Extract flags (Sample Tag)  │
└──────┬───────────────────────────┘
       │
       ▼
┌──────────────────────────────────┐
│ 2. Construct JaegerSpanContext   │
│    long[] parts = splitTraceId() │
│    new JaegerSpanContext(        │
│      traceIdHigh,                │
│      traceIdLow,                 │
│      parentSpanId,               │
│      parentOfParentId,           │
│      flags                       │
│    )                             │
└──────┬───────────────────────────┘
       │
       ▼
┌──────────────────────────────────┐
│ 3. Create Child Span             │
│    tracer.buildSpan("child-span")│
│          .asChildOf(parentContext)│
│          .withTag(...)           │
│          .start()                │
└──────┬───────────────────────────┘
       │
       ▼
┌──────────────────────────────────┐
│ 4. Activate Span to Current Thread│
│    try (Scope scope =            │
│      tracer.scopeManager()       │
│            .activate(childSpan)) │
│    {                             │
│      // MDC Auto Injection       │
│      // Business Logic Execution │
│    } // MDC Auto Cleanup         │
└──────────────────────────────────┘

3.2 MDC Management Process for Nested Spans

Timeline ────────────────────────────────────────────>

ThreadLocal Stack:
┌─────────────────────────────────────────────────┐
│  null                                           │
└─────────────────────────────────────────────────┘

activate(spanA)
┌─────────────────────────────────────────────────┐
│  ScopeA: {traceId: xxx, spanId: A, previous: null}│
└─────────────────────────────────────────────────┘
MDC: {traceId: xxx, spanId: A}

  activate(spanB)  // Nested Call
  ┌─────────────────────────────────────────────────┐
  │  ScopeB: {traceId: xxx, spanId: B, previous: ScopeA}│
  └─────────────────────────────────────────────────┘
  MDC: {traceId: xxx, spanId: B}  // spanId Update

    // Business Code Execution
    log.info("Processing...")  // Log carries spanId=B

  close(ScopeB)
  ┌─────────────────────────────────────────────────┐
  │  ScopeA: {traceId: xxx, spanId: A, previous: null}│
  └─────────────────────────────────────────────────┘
  MDC: {traceId: xxx, spanId: A}  // Restore to spanId=A

close(ScopeA)
┌─────────────────────────────────────────────────┐
│  null                                           │
└─────────────────────────────────────────────────┘
MDC: {}  // Completely Clear

3.3 Trace ID Split Algorithm

Jaeger supports 128-bit traceId, but Java long is only 64-bit, so splitting is required:

/**
 * Split hexadecimal traceId string into high 64 bits and low 64 bits
 * 
 * Example:
 * Input: "04bf92f3577b34da63ce929d0e0e4736" (32 hex characters = 128 bit)
 * Output: [0x04bf92f3577b34da, 0x63ce929d0e0e4736]
 */
public static long[] splitTraceId(String traceIdHex) {
    if (traceIdHex.length() <= 16) {
        // Only low 64 bits, high bits are 0
        return new long[]{0L, parseLong(traceIdHex)};
    } else {
        // Split into high 64 bits and low 64 bits
        String highHex = traceIdHex.substring(0, traceIdHex.length() - 16);
        String lowHex = traceIdHex.substring(traceIdHex.length() - 16);
        return new long[]{parseLong(highHex), parseLong(lowHex)};
    }
}

4. Configuration and Integration

4.1 Tracer Initialization Configuration

static Tracer tracer = new Configuration("order-service")
    .withSampler(
        new Configuration.SamplerConfiguration()
            .withType("const")
            .withParam(1)  // Sampling Rate 100%
    )
    .withReporter(
        new Configuration.ReporterConfiguration()
            .withLogSpans(true)  // Enable log output in development environment
            .withSender(
                new Configuration.SenderConfiguration()
                    .withAgentHost("localhost")
                    .withAgentPort(6831)
            )
    )
    .getTracerBuilder()
    .withScopeManager(new CustomMDCScopeManager())  // Key: Inject Custom Manager
    .build();

4.2 Logback Configuration

<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} [traceId=%X{traceId} spanId=%X{spanId}] - %msg%n</pattern>
        </encoder>
    </appender>
    
    <appender name="JSON" class="ch.qos.logback.core.FileAppender">
        <file>logs/app.json</file>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <includeMdcKeyName>traceId</includeMdcKeyName>
            <includeMdcKeyName>spanId</includeMdcKeyName>
            <includeMdcKeyName>parentId</includeMdcKeyName>
        </encoder>
    </appender>
    
    <root level="INFO">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="JSON" />
    </root>
</configuration>

4.3 Spring Boot Integration (Optional)

@Configuration
public class TracingConfig {
    
    @Bean
    public Tracer jaegerTracer() {
        return new Configuration(
                env.getProperty("spring.application.name", "unknown-service")
            )
            .withSampler(samplerConfig())
            .withReporter(reporterConfig())
            .getTracerBuilder()
            .withScopeManager(new CustomMDCScopeManager())
            .build();
    }
    
    @Bean
    public TracingFilter tracingFilter(Tracer tracer) {
        return new TracingFilter(tracer);
    }
}

5. Usage Scenarios and Best Practices

5.1 Receive Upstream Trace Context

@RestController
public class OrderController {
    
    @GetMapping("/order/{id}")
    public Order getOrder(
        @PathVariable String id,
        @RequestHeader(value = "uber-trace-id", required = false) String uberTraceId
    ) {
        JaegerSpanContext parentContext = parseUberTraceId(uberTraceId);
        
        Span span = tracer.buildSpan("get-order")
            .asChildOf(parentContext)  // Key: Link Remote Parent Span
            .withTag(Tags.SPAN_KIND, Tags.SPAN_KIND_SERVER)
            .withTag("order.id", id)
            .start();
        
        try (Scope scope = tracer.scopeManager().activate(span)) {
            log.info("Processing order request");  // Automatically carries traceId/spanId
            return orderService.getById(id);
        } finally {
            span.finish();
        }
    }
}

5.2 Pass Trace Context Downstream

public class PaymentClient {
    
    public void processPayment(String orderId) {
        Span span = tracer.buildSpan("call-payment-service")
            .withTag(Tags.SPAN_KIND, Tags.SPAN_KIND_CLIENT)
            .start();
        
        try (Scope scope = tracer.scopeManager().activate(span)) {
            HttpHeaders headers = new HttpHeaders();
            
            // Inject trace context to HTTP Header
            tracer.inject(
                span.context(),
                Format.Builtin.HTTP_HEADERS,
                new HttpHeadersCarrier(headers)
            );
            
            restTemplate.exchange(
                "http://payment-service/pay",
                HttpMethod.POST,
                new HttpEntity<>(paymentRequest, headers),
                PaymentResponse.class
            );
        } finally {
            span.finish();
        }
    }
}

5.3 Async Scenario Handling

@Service
public class AsyncOrderService {
    
    @Autowired
    private Tracer tracer;
    
    @Autowired
    private ExecutorService executorService;
    
    public void processOrderAsync(String orderId) {
        Span parentSpan = tracer.activeSpan();  // Get Current Span
        
        executorService.submit(() -> {
            // Reactivate Parent Span in New Thread
            Span asyncSpan = tracer.buildSpan("async-process")
                .asChildOf(parentSpan)
                .start();
            
            try (Scope scope = tracer.scopeManager().activate(asyncSpan)) {
                log.info("Async processing order");  // MDC Correctly Injected
                // Business Logic
            } finally {
                asyncSpan.finish();
            }
        });
    }
}

6. Key Design Decisions

6.1 Why not operate MDC directly in business code?

Problem: Manual management is easy to miss cleanup, leading to traceId pollution during thread pool reuse.

Solution: Through ScopeManager lifecycle management, inject at activate() and automatically clean up at close().

6.2 Why need to save MDC snapshot?

Scenario: When nested Span calls, need to restore parent Span's MDC after child Span ends.

Implementation:

// When entering child Span
this.previousTraceId = MDC.get("traceId");  // Save snapshot
MDC.put("traceId", childSpan.context().toTraceId());  // Overwrite

// When exiting child Span
restoreMDC("traceId", previousTraceId);  // Restore snapshot

6.3 Why use ThreadLocal instead of InheritableThreadLocal?

Consideration:

• ThreadLocal: Strict thread isolation, suitable for synchronous scenarios
• InheritableThreadLocal: Child thread inherits parent thread values, but context leakage is prone to occur during thread pool reuse

Recommendation: Explicitly pass Span in asynchronous scenarios, rather than relying on automatic inheritance

7. Monitoring and Debugging

7.1 Log Output Example

21:45:32.123 [http-nio-8080-exec-1] INFO  c.w.OrderService [traceId=04bf92f3577b34da63ce929d0e0e4736 spanId=36bd32b7a5712a1a] - Processing order ORD-001
21:45:32.234 [http-nio-8080-exec-1] INFO  c.w.PaymentClient [traceId=04bf92f3577b34da63ce929d0e0e4736 spanId=7f3a28b9c4d5e6a1] - Calling payment service
21:45:32.456 [http-nio-8080-exec-1] INFO  c.w.OrderService [traceId=04bf92f3577b34da63ce929d0e0e4736 spanId=36bd32b7a5712a1a] - Order processed successfully

7.2 Jaeger UI Query

In Jaeger UI you can:

1. View complete call chain via traceId 04bf92f3577b34da63ce929d0e0e4736
2. View each Span's Tags (such as db.statement, http.status_code)
3. View parent-child relationship and time consumption distribution between Spans

7.3 Common Troubleshooting

Problem Phenomenon	Possible Cause	Troubleshooting Method
traceId is empty in log	Span not activated or ScopeManager not configured correctly	Check activate() call and Tracer construction
traceId crosstalk between different requests	MDC not cleaned up correctly (thread pool reuse)	Ensure MDC.remove() called in finally block
Child Span not linked to Parent Span	asChildOf() parameter error	Verify if parentSpanId is parsed correctly
Span not visible in Jaeger	Sampling rate set to 0 or Reporter configuration error	Check withParam(1) and network connectivity

8. Performance Considerations

8.1 Performance Impact Analysis

Operation	Time Consumption	Impact
MDC.put()	< 1μs	Negligible
ThreadLocal.get()	< 1μs	Negligible
Span.start()	~10μs	Low
Span.finish() + Report	~100μs	Async reporting, small impact on main flow

8.2 Optimization Suggestions

1. Reasonable control of sampling rate: Production environment can be set to 0.1 (10%) to reduce storage costs
2. Batch reporting: Reporter configuration withFlushInterval(1000) batch send Spans
3. Avoid excessive Spans: Do not create Span for every database query, control granularity

9. Extension Directions

9.1 Support Reactive Programming (Reactor/WebFlux)

public class ReactorScopeManager implements ScopeManager {
    @Override
    public Scope activate(Span span) {
        return new ReactorScope(span);
    }
    
    static class ReactorScope implements Scope {
        ReactorScope(Span span) {
            // Use Reactor Context instead of ThreadLocal
            Context.of("span", span);
        }
    }
}

9.2 Integrate Spring Cloud Sleuth

Spring Cloud Sleuth provides out-of-the-box distributed tracing, can replace this solution:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>

9.3 Support OpenTelemetry

OpenTelemetry is the new generation observability standard, migration is recommended in the future:

OpenTelemetry openTelemetry = AutoConfiguredOpenTelemetrySdk
    .initialize()
    .getOpenTelemetrySdk();

10. Summary

This solution implements seamless integration of Jaeger Trace and SLF4J MDC through custom ScopeManager, with the following features:

✅ Automation: Scope lifecycle management, no need to manually operate MDC ✅ Thread Safety: ThreadLocal isolation + Snapshot recovery mechanism ✅ Nesting Support: Linked list structure maintains multi-layer Span relationship ✅ Production Ready: Considered boundary conditions such as thread pool reuse, asynchronous scenarios

This solution is suitable for microservice architectures requiring fine-grained control of trace context propagation, especially scenarios such as service gateways and BFF layers that need to receive upstream trace information.

Agreement

The code part of this work is licensed under Apache License 2.0 . You may freely modify and redistribute the code, and use it for commercial purposes, provided that you comply with the license. However, you are required to:

• Attribution: Retain the original author's signature and code source information in the original and derivative code.
• Preserve License: Retain the Apache 2.0 license file in the original and derivative code.

The documentation part of this work is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License . You may freely share, including copying and distributing this work in any medium or format, and freely adapt, remix, transform, and build upon the material. However, you are required to:

• Attribution: Give appropriate credit, provide a link to the license, and indicate if changes were made.
• NonCommercial: You may not use the material for commercial purposes. For commercial use, please contact the author.
• ShareAlike: If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.