Tracing Kafka Message Flows Without Explicit Logging

Tracing Across Message Queue - Kafka, Without Writing a Log and Trace

Asynchronous communication via message queues like Kafka introduces challenges in tracing the complete flow of a request. Traditionally, developers rely on extensive logging, but this can be costly and complex to manage. Bitryon logger offers a novel approach, connecting tracing puzzles into a workflow and stack-trace without requiring explicit log and trace statements.

The ideal model is full visibility into every request’s journey, but the reality is often fragmented data and difficulty correlating events across services. Without a solution like Bitryon, tracing can become a significant engineering effort, especially in high-volume systems, potentially costing valuable developer time and impacting system performance.

Key Insights

• Bitryon logger: A tracing and logging tool designed to correlate events without extensive code changes.
• Step Log ID Propagation: The core concept involves passing a unique ID through message headers, linking related events across services.
• Interceptor Pattern: Utilizing Kafka’s interceptor mechanism allows for transparent addition and extraction of the step log ID without modifying core application logic.

Working Example

public class ProducerLoggerInterceptor implements ProducerInterceptor<String, String> {
private static final Logger logger = LoggerFactory.getLogger();
@Override
public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
// CRITICAL .getNextStepLogId()
String nextSteplogId = logger.getNextStepLogId();
record.headers().add(PreDefinition.HTTP_HEADER_STEP_LOG_ID, nextSteplogId.getBytes(PreDefinition.CharsetEncoding));
return record;
}
// other code
}

public class ConsumerLoggerInterceptor implements ConsumerInterceptor<String, String> {
private static final Logger logger = LoggerFactory.getLogger();
@Override
public ConsumerRecords<String, String> onConsume(ConsumerRecords<String, String> records) {
records.forEach(record -> {
Header header = record.headers().lastHeader(PreDefinition.HTTP_HEADER_STEP_LOG_ID);
if (header!=null) {
String stepLogId = new String(header.value(), PreDefinition.CharsetEncoding);
logger.setStepLogId(stepLogId);
}else {
logger.reset();// reset to decouple each consume
}
});
return records;
}
// other code
}

@Configuration
public class KafkaBeansConfiguration {
// ... (bean definitions for KafkaTemplate, ProducerFactory, ConsumerFactory, and ConcurrentKafkaListenerContainerFactory)
}

Practical Applications

• E-commerce Order Processing: Track an order’s journey from placement to fulfillment, across multiple microservices and Kafka topics.
• Pitfall: Incorrectly configuring interceptors or failing to propagate the step log ID will break the trace, leading to incomplete visibility.

References:

• https://dev.to/franknpc/how-to-trace-across-message-queue-kafka-without-writing-a-log-and-trace-2dci