Structured Logging and Tracing

In distributed systems such as LogisticsCore, operational transparency is non-negotiable. Real-time observability into execution flow, latency attribution, and failure root cause analysis demands a rigorous approach to logging and tracing. This section establishes the architectural foundation for structured logging using Logback, integrates Micrometer Tracing for distributed context propagation, and enforces secure, auditable log output through custom instrumentation. All implementations are grounded in Java 21+ semantics, with explicit reliance on Records, Pattern Matching, and Virtual Threads where applicable.

Introduction to Structured Logging

Unstructured text logs are inherently unsuitable for automated analysis in high-throughput, distributed environments. Structured logging—emitting log events in a machine-readable format such as JSON—enables deterministic parsing, indexing, and querying by log aggregation systems. For LogisticsCore, this means every log entry must conform to a standardized schema, embedding contextual metadata including timestamps, service identifiers, and distributed tracing identifiers.

The use of structured formats eliminates the need for fragile regular expression-based log extraction and enables precise filtering at scale. This is not an optional enhancement; it is a prerequisite for reliable system observability.

Logback Configuration for JSON Output

Logback, the de facto logging framework in Spring Boot applications, must be configured to emit JSON-formatted output. The net.logstash.logback.encoder.LogstashEncoder is employed to serialize log events into JSON, enriched with static contextual fields such as application name and deployment environment.

The following logback-spring.xml configuration defines an asynchronous JSON appender, minimizing I/O contention on application threads:

<configuration>
    <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <customFields>{"application":"logistics-core","environment":"${ENV:-dev}"}</customFields>
        </encoder>
    </appender>
    
    <appender name="ASYNC_JSON" class="ch.qos.logback.classic.AsyncAppender">
        <appender-ref ref="JSON" />
        <queueSize>10000</queueSize>
        <discardingThreshold>0</discardingThreshold>
    </appender>
    
    <root level="INFO">
        <appender-ref ref="ASYNC_JSON" />
    </root>
</configuration>

This configuration ensures that all log output is serialized to JSON and written asynchronously, reducing the risk of logging overhead impacting shipment processing latency. The inclusion of customFields guarantees that downstream systems can route and filter logs by application context without parsing message content.

Integrating Micrometer Tracing

Micrometer Tracing provides a vendor-agnostic API for distributed tracing instrumentation, abstracting over backend systems such as Zipkin or OpenTelemetry. The Observation API, introduced in Spring Boot 3, serves as the primary mechanism for span creation and context propagation.

The following implementation demonstrates the use of Observation within the ShipmentTrackingService of LogisticsCore:

package com.logistics.core.tracing;

import io.micrometer.observation.Observation;
import io.micrometer.observation.ObservationRegistry;
import io.micrometer.tracing.Span;
import io.micrometer.tracing.Tracer;
import org.slf4j.MDC;
import org.springframework.stereotype.Service;

import java.util.concurrent.Executors;
import java.util.concurrent.ThreadFactory;

@Service
public class ShipmentTrackingService {
    private final ObservationRegistry observationRegistry;
    private final Tracer tracer;

    public ShipmentTrackingService(ObservationRegistry observationRegistry, Tracer tracer) {
        this.observationRegistry = observationRegistry;
        this.tracer = tracer;
    }

    public void trackShipment(String shipmentId) {
        Observation observation = Observation.start("track-shipment", observationRegistry);
        try (Observation.Scope scope = observation.openScope()) {
            Span currentSpan = tracer.currentSpan();
            if (currentSpan != null) {
                currentSpan.tag("shipment.id", shipmentId);
            }
            processShipmentInternally(shipmentId);
        } finally {
            observation.stop();
        }
    }

    private void processShipmentInternally(String shipmentId) {
        org.slf4j.LoggerFactory.getLogger(ShipmentTrackingService.class)
                .info("Processing shipment {}", shipmentId);
    }
}

Upon entering the observation scope, Micrometer Tracing automatically injects the traceId and spanId into the Mapped Diagnostic Context (MDC), provided the logging configuration includes %X{traceId} and %X{spanId} in the pattern layout. This enables correlation of log entries across service boundaries.

Propagating Context Across Virtual Threads

Java 21’s virtual threads introduce a new execution model that decouples concurrency from OS thread count. However, MDC state is thread-local and does not automatically propagate across virtual thread boundaries. When offloading work to a virtual thread, the tracing context must be explicitly captured and restored.

The following method demonstrates manual context propagation using a ThreadPerTaskExecutor backed by virtual threads:

public void trackShipmentAsync(String shipmentId) {
    Observation observation = Observation.start("track-shipment-async", observationRegistry);
    try (Observation.Scope scope = observation.openScope()) {
        String traceId = MDC.get("traceId");
        Span currentSpan = tracer.currentSpan();
        
        ThreadFactory virtualThreadFactory = Thread.ofVirtual().factory();
        var executor = Executors.newThreadPerTaskExecutor(virtualThreadFactory);
        
        Runnable task = () -> {
            if (traceId != null) {
                MDC.put("traceId", traceId);
            }
            if (currentSpan != null) {
                tracer.withSpan(currentSpan);
            }
            try {
                processShipmentInternally(shipmentId);
            } finally {
                MDC.clear();
            }
        };
        
        executor.submit(task);
    } finally {
        observation.stop();
    }
}

The MDC state and active Span are captured in the platform thread and manually injected into the virtual thread’s execution context. This is not an abstraction provided by Spring; it is a mandatory step to preserve observability in a virtual thread environment. Failure to do so results in orphaned log entries with missing trace identifiers.

Custom Logback Appender for Sensitive Data Masking

Log output may inadvertently contain sensitive data such as personally identifiable information (PII) or authentication credentials. To enforce data protection compliance, a custom Logback appender must be implemented to sanitize log messages before emission.

The following SensitiveDataMaskingAppender uses Java 21 Records and regular expression matching to identify and mask sensitive patterns:

package com.logistics.core.logging;

import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.AppenderBase;
import org.springframework.stereotype.Component;

import java.util.List;
import java.util.regex.Pattern;

@Component
public class SensitiveDataMaskingAppender extends AppenderBase<ILoggingEvent> {
    
    private record MaskingRule(String name, Pattern pattern, String replacement) {}
    
    private final List<MaskingRule> maskingRules = List.of(
        new MaskingRule("CreditCard", Pattern.compile("\\b\\d{4}[ -]?\\d{4}[ -]?\\d{4}[ -]?(\\d{4})\\b"), "[CREDIT_CARD_MASKED]"),
        new MaskingRule("SSN", Pattern.compile("\\b\\d{3}-\\d{2}-\\d{4}\\b"), "[SSN_MASKED]"),
        new MaskingRule("Password", Pattern.compile("(?i)(password|passwd|pwd)[\"']?[:=]?[\"']?([^\"'\\s]+)"), "[PASSWORD_MASKED]")
    );

    @Override
    protected void append(ILoggingEvent eventObject) {
        String message = eventObject.getFormattedMessage();
        String maskedMessage = message;
        
        for (MaskingRule rule : maskingRules) {
            maskedMessage = rule.pattern().matcher(maskedMessage).replaceAll(rule.replacement());
        }
        
        System.out.println(maskedMessage);
    }
}

This appender must be integrated into the Logback configuration pipeline to ensure that no sensitive data reaches the log sink. The use of Records enhances code clarity and immutability, while the pattern-based approach allows for extensible rule definitions.

Conclusion

Observability in LogisticsCore is achieved through a combination of structured logging, distributed tracing, and secure log handling. The integration of Logback with JSON output ensures machine-readability. Micrometer Tracing provides a unified API for context propagation, while manual MDC management is required to maintain continuity across virtual threads. Finally, custom appenders enforce data masking, ensuring compliance with security policies.

These mechanisms are not optional conveniences; they are architectural imperatives for maintaining system integrity, debuggability, and compliance at scale.

References

[1] C. Betts, “Logback Documentation,” QOS.ch, 2023. [Online]. Available: https://logback.qos.ch/documentation.html [2] M. Simons, “Micrometer Documentation,” Micrometer.io, 2023. [Online]. Available: https://micrometer.io/docs [3] J. Goetz et al., Java Concurrency in Practice. Addison-Wesley, 2006, ch. 8. [4] Oracle, “Java 21 Documentation,” Oracle, 2023. [Online]. Available: https://docs.oracle.com/javase/specs/jls/se21/html/index.html