RabbitMQ’s Queues, Exchanges, and When to Choose Them

The Black Box

The logistics platform dispatches delivery routes to driver apps. A route is a sequence of stops with time windows. The dispatcher service computes a route and puts it on a queue. A driver app picks it up, acknowledges receipt, and begins the route. If the driver app crashes, the route must go to another driver.

The team initially used Kafka for this. The result: routes were delivered to all consumers in the group (because Kafka’s consumer group assigns partitions, not messages). Scaling to 50 driver apps required 50 partitions, and rebalancing during driver app restarts caused all drivers to pause.

The Mechanism

RabbitMQ separates routing from queuing:

Exchange: Receives messages from producers and routes them to queues based on rules. The exchange does not store messages. It is a routing table.

Queue: Stores messages until a consumer acknowledges them. The queue is the storage layer.

Binding: A rule that connects an exchange to a queue, optionally filtered by a routing key.

Exchange Types

// Concept: exchange types and routing patterns

// 1. Direct exchange: route by exact routing key match
channel.exchangeDeclare("logistics", "direct", true);
channel.queueDeclare("route-dispatch", true, false, false, null);
channel.queueBind("route-dispatch", "logistics", "route.new");
// Messages published to "logistics" with routing key "route.new"
// go to the "route-dispatch" queue.

// 2. Topic exchange: route by wildcard pattern
channel.exchangeDeclare("events", "topic", true);
channel.queueDeclare("warehouse-events", true, false, false, null);
channel.queueBind("warehouse-events", "events", "package.*.WH-042");
// Matches: package.scanned.WH-042, package.shipped.WH-042
// Does not match: package.scanned.WH-019

// 3. Fanout exchange: route to all bound queues (broadcast)
channel.exchangeDeclare("notifications", "fanout", true);
channel.queueDeclare("email-notifications", true, false, false, null);
channel.queueDeclare("sms-notifications", true, false, false, null);
channel.queueBind("email-notifications", "notifications", "");
channel.queueBind("sms-notifications", "notifications", "");
// Every message published to "notifications" goes to BOTH queues.

Message Persistence

By default, messages in RabbitMQ are stored in memory. If the broker restarts, messages are lost. For the route dispatch queue, this is unacceptable.

// Concept: durable queue + persistent messages for route dispatch
// Both must be set for messages to survive broker restart.

// Durable queue: queue metadata survives restart
channel.queueDeclare("route-dispatch", /*durable*/ true, false, false, null);

// Persistent message: message body is written to disk
AMQP.BasicProperties props = new AMQP.BasicProperties.Builder()
    .deliveryMode(2)       // 2 = persistent
    .contentType("application/json")
    .build();

channel.basicPublish("logistics", "route.new", props, routeJson.getBytes());

// With both durable=true and deliveryMode=2:
// 1. Message is written to the Mnesia disk store (or quorum queue log)
// 2. Broker restart: queue is recreated, messages are replayed from disk
// Cost: ~10x slower than transient messages (disk I/O per message)

Prefetch and Fairness

The prefetch count controls how many unacknowledged messages a consumer can hold. It determines throughput and fairness.

// Concept: prefetch tuning
// prefetch=1: fair dispatch, but low throughput (consumer waits for ack round trip)
channel.basicQos(1);
// Use when: processing time varies widely (some routes take 5ms, others 500ms)
// Consumer with a long route does not block others from receiving new routes.

// prefetch=20: higher throughput, less fair
channel.basicQos(20);
// Use when: processing time is uniform (all routes take ~50ms)
// Consumer processes 20 messages before needing to wait for more.
// Risk: if a consumer crashes with 20 unacked messages, all 20 are redelivered.

Dead Letter Queues

A message that fails processing repeatedly is a poison message. Without a dead letter queue, it cycles between delivery, rejection, and redelivery forever.

// Concept: dead letter queue for poison messages
Map<String, Object> args = new HashMap<>();
args.put("x-dead-letter-exchange", "logistics-dlx");
args.put("x-dead-letter-routing-key", "route.failed");

channel.queueDeclare("route-dispatch", true, false, false, args);
channel.exchangeDeclare("logistics-dlx", "direct", true);
channel.queueDeclare("route-failures", true, false, false, null);
channel.queueBind("route-failures", "logistics-dlx", "route.failed");

// When a message is rejected (basicNack with requeue=false)
// or exceeds the delivery count limit, it moves to "route-failures"
// for manual inspection instead of cycling in the main queue.

The Observable Consequence

For the logistics platform’s route dispatch with 50 driver apps:

Configuration	Kafka (12 partitions)	RabbitMQ
Max consumers	12 (one per partition)	Unlimited
Message delivery	Per-partition	Per-message
Rebalance on consumer crash	45s group pause (eager)	Instant redelivery
Message replay	Yes	No
Ordering	Per-partition	No global ordering

RabbitMQ delivers routes to individual drivers with instant redelivery on failure. Kafka cannot scale to 50 independent consumers without 50 partitions, and each consumer restart causes a rebalance.

The Decision Rule

Use RabbitMQ when you need per-message delivery to competing consumers, when the message is disposable after processing, and when you need more consumers than is practical with Kafka partitions. Task dispatch, work queues, request-reply patterns.

Use Kafka when you need message retention, replay, and multiple independent consumer groups. Event streams, changelogs, data pipelines.

The logistics platform uses both: Kafka for package events (multiple consumers, replay needed) and RabbitMQ for route dispatch (one consumer per message, immediate redelivery on failure).