Designing a Kafka-Like Message Queue in Java: LLD Best Practices

Java LLD: Designing a Kafka-Like Message Queue for Machine Coding Interviews

Designing a high-performance message queue is a core requirement for senior machine coding rounds that tests thread-safety and decoupled architecture. A primary failure point is using standard Java collections that remove elements upon polling, which prevents multiple consumer groups from accessing the same data stream.

Why This Matters

In high-scale engineering, using destructive queue structures prevents the replayability required for heterogeneous systems to function independently. Technical reality demands an immutable, append-only log where consumers manage their own offsets, ensuring that one lagging consumer does not cause data loss or system-wide interference across the entire pub-sub architecture.

Key Insights

• Using java.util.Queue for pub-sub systems is a common architectural error as it prevents multiple consumer groups from reading the same data.
• The core mental model for a scalable queue is an immutable, append-only log where messages are persisted per topic (2026).
• Implementing ReentrantLock ensures atomic appends to the message log within multi-threaded producer environments.
• Independent offset management via an OffsetManager allows ConsumerGroups to process messages at varying speeds without data loss.

Working Examples

Thread-safe Topic implementation using ReentrantLock for atomic appends and subList for offset-based message retrieval.

public class Topic {
private final List<Message> messages = new ArrayList<>();
private final ReentrantLock lock = new ReentrantLock();
public void addMessage(Message message) {
lock.lock();
try {
messages.add(message);
} finally {
lock.unlock();
}
}
public List<Message> getMessagesFrom(int offset) {
return (offset >= messages.size()) ? List.of() : messages.subList(offset, messages.size());
}
}

Practical Applications

• Use case: Kafka-like systems use append-only logs to allow multiple heterogeneous systems to consume the same stream. Pitfall: Coupling producers directly to consumers violates pub-sub principles and creates scaling bottlenecks.
• Use case: Independent Offset Management enables a ConsumerGroup to recover progress after a lag. Pitfall: Using standard polling that removes elements leads to data loss when multiple groups attempt to read the same topic.

References:

• https://javalld.com/problems/message-queue
• https://dev.to/vishalagg/java-lld-designing-a-kafka-like-message-queue-for-machine-coding-interviews-411l