The Intent Log Architecture

Introduction

The intent log architecture is a design pattern that utilizes Kafka as the immutable source of truth for business intent. This approach provides flexibility and abstraction by specifying desired outcomes, allowing systems to determine optimal execution paths. In this section, we will delve into the core principles and components of the intent log architecture, exploring how Kafka serves as the foundation for this design.

Kafka as the Immutable Source of Truth

Kafka is ideally suited for the intent log architecture due to its ability to capture every business intent as an event. By using Kafka topics, we can ensure that all intents are processed in sequential order within a single partition. This is achieved through the use of Entity-ID as the message key, which guarantees that all intents for a specific entity are routed to the same partition.

Log Compaction

Log compaction is a critical component of the intent log architecture. It allows for the retention of only the latest version of a specific intent, effectively compressing the timeline of intent. This is particularly useful in scenarios where the latest intent is the only relevant one. Kafka’s log compaction feature is ideal for this purpose, as it provides a mechanism for removing older intent versions while preserving the latest one.

Tombstones

Tombstones are special messages with null payloads that are sent to a compacted Kafka topic to signal the deletion of an entity key. This mechanism allows for the removal of intents that are no longer relevant, ensuring that the intent log remains up-to-date and accurate.

Ingestion Layer Design

The ingestion layer is responsible for processing business intents and sending them to Kafka. This layer must prioritize intent durability before acknowledging receipt to the client. Java Records are used in the ingestion layer as shallowly immutable data carriers for intent payloads. Sealed interfaces enable exhaustive pattern matching for different intent types, such as CreateIntent, UpdateIntent, and CancelIntent.

Code Example

public sealed interface BusinessIntent permits CreateOrderIntent, UpdateOrderIntent, CancelOrderIntent {
    String entityId();
    long timestamp(); 
}

public record CreateOrderIntent(String entityId, String customerId, List<String> items, long timestamp) implements BusinessIntent {}
public record UpdateOrderIntent(String entityId, List<String> updatedItems, long timestamp) implements BusinessIntent {}
public record CancelOrderIntent(String entityId, String reason, long timestamp) implements BusinessIntent {}

Configuration and Optimization

To optimize the intent log architecture, several Kafka topic configurations must be carefully considered. These include cleanup.policy, min.insync.replicas, acks, compression.type, and min.cleanable.dirty.ratio. The following table summarizes the recommended values for these configurations:

Conclusion

The intent log architecture provides a robust and scalable solution for managing business intent. By leveraging Kafka as the immutable source of truth, we can ensure that all intents are processed in sequential order and that the latest version of each intent is retained. Through careful configuration and optimization, we can optimize the performance and reliability of the intent log architecture.

Sources

[1] https://www.baeldung.com/ops/kafka-topics-compaction [2] https://docs.confluent.io/kafka/design/log_compaction.html [3] https://mikulskibartosz.name/what-is-kafka-log-compaction-how-does-it-work