Full-Stack Tracing

The telemetry platform has a k6 baseline (CH1), query profiling (CH2), JVM profiling (CH3), and connection pool metrics (CH4). Each tool answers one question: “How fast is this?” But none answers “Why is this request slow?” across the full stack.

A request to the telemetry API traverses: load balancer, Spring Boot controller, service layer, MongoDB driver, connection pool, network, mongos/mongod, WiredTiger. A slowdown at any layer propagates to the HTTP response. Without tracing, diagnosing which layer caused the slowdown requires checking each tool independently and correlating timestamps manually.

OpenTelemetry (OTel) creates a trace for each request. The trace contains spans for each operation. A span records the start time, end time, and metadata (collection name, query filter, server address). Spans are nested: the HTTP span contains the service span, which contains the MongoDB span.

OpenTelemetry Setup

<!-- pom.xml dependencies -->
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-mongo-3.1</artifactId>
</dependency>

// Configure OpenTelemetry SDK
@Configuration
public class OtelConfig {

    @Bean
    public OpenTelemetry openTelemetry() {
        Resource resource = Resource.getDefault()
            .merge(Resource.create(Attributes.of(
                ResourceAttributes.SERVICE_NAME, "telemetry-api",
                ResourceAttributes.SERVICE_VERSION, "1.0.0"
            )));

        SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
            .addSpanProcessor(BatchSpanProcessor.builder(
                OtlpGrpcSpanExporter.builder()
                    .setEndpoint("http://otel-collector:4317")
                    .build()
            ).build())
            .setResource(resource)
            .setSampler(Sampler.traceIdRatioBased(0.1))  // Sample 10% of traces
            .build();

        return OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .setPropagators(ContextPropagators.create(
                W3CTraceContextPropagator.getInstance()))
            .build();
    }
}

MongoDB Driver Instrumentation

The MongoDB Java driver supports OpenTelemetry via CommandListener. Each MongoDB command (find, insert, aggregate) creates a child span:

// Custom MongoDB span creator using CommandListener
public class MongoOtelCommandListener implements CommandListener {

    private final Tracer tracer;
    private final ConcurrentHashMap<Integer, Span> activeSpans = new ConcurrentHashMap<>();

    public MongoOtelCommandListener(OpenTelemetry otel) {
        this.tracer = otel.getTracer("mongodb-driver");
    }

    @Override
    public void commandStarted(CommandStartedEvent event) {
        Span span = tracer.spanBuilder("mongodb." + event.getCommandName())
            .setSpanKind(SpanKind.CLIENT)
            .setAttribute("db.system", "mongodb")
            .setAttribute("db.name", event.getDatabaseName())
            .setAttribute("db.operation", event.getCommandName())
            .setAttribute("net.peer.name",
                event.getConnectionDescription().getServerAddress().getHost())
            .setAttribute("net.peer.port",
                event.getConnectionDescription().getServerAddress().getPort())
            .startSpan();

        // Extract collection name from command
        BsonDocument command = event.getCommand();
        String commandName = event.getCommandName();
        BsonValue collValue = command.get(commandName);
        if (collValue != null && collValue.isString()) {
            span.setAttribute("db.mongodb.collection", collValue.asString().getValue());
        }

        activeSpans.put(event.getRequestId(), span);
    }

    @Override
    public void commandSucceeded(CommandSucceededEvent event) {
        Span span = activeSpans.remove(event.getRequestId());
        if (span != null) {
            span.setAttribute("db.response_time_ms",
                event.getElapsedTime(TimeUnit.MILLISECONDS));
            span.end();
        }
    }

    @Override
    public void commandFailed(CommandFailedEvent event) {
        Span span = activeSpans.remove(event.getRequestId());
        if (span != null) {
            span.setStatus(StatusCode.ERROR, event.getThrowable().getMessage());
            span.recordException(event.getThrowable());
            span.end();
        }
    }
}

Register the listener with the MongoClient:

MongoClientSettings settings = MongoClientSettings.builder()
    .applyConnectionString(new ConnectionString(uri))
    .addCommandListener(new MongoOtelCommandListener(openTelemetry))
    .build();