Change Data Capture and the Outbox Pattern

Chapter 7 established the single source of truth principle: write to one database, derive everything else. Chapter 8 introduced Kafka as the event streaming layer. This chapter connects the two: how do changes in PostgreSQL get into Kafka without dual writes?

The answer is Change Data Capture (CDC): reading the database’s WAL (Chapter 3) to capture every insert, update, and delete as a stream of change events. Debezium is the tool that does this.

CDC: The WAL as a Data Integration API

The WAL was introduced in Chapter 3 as a crash recovery mechanism. Replication in Chapter 4 used the WAL as a data transfer protocol between primary and replica. CDC is a third use of the same mechanism: reading the WAL to capture changes and publishing them to an external system.

The pattern is elegant: the database already records every change in the WAL for its own purposes. CDC reads the WAL as an external consumer, without modifying the database’s write path. No triggers. No polling queries. No application code changes.

Debezium Architecture

Debezium runs as a Kafka Connect connector. Kafka Connect is a framework for moving data between Kafka and external systems. Debezium is a source connector: it reads from a database and writes to Kafka.

// Concept: Debezium PostgreSQL connector configuration
{
    "name": "logistics-postgres-connector",
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "database.hostname": "postgres-primary",
        "database.port": "5432",
        "database.user": "debezium",
        "database.password": "${file:/secrets/debezium.properties:password}",
        "database.dbname": "logistics",
        "topic.prefix": "logistics",
        "table.include.list": "public.packages,public.package_events,public.inventory",
        "plugin.name": "pgoutput",
        "slot.name": "debezium_slot",
        "publication.name": "dbz_publication",
        "snapshot.mode": "initial",
        "transforms": "route",
        "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
        "transforms.route.regex": "logistics\\.public\\.(.*)",
        "transforms.route.replacement": "cdc.$1"
    }
}

This connector:

1. Creates a PostgreSQL logical replication slot (debezium_slot).
2. Takes an initial snapshot of the configured tables (if snapshot.mode=initial).
3. Starts reading the WAL from the slot, capturing every change.
4. Publishes each change as a Kafka message to a topic named after the table (e.g., cdc.packages, cdc.package_events).

PostgreSQL Logical Replication Slots

A replication slot is a bookmark in the WAL. PostgreSQL guarantees that WAL segments referenced by any active slot will not be recycled, even if no physical replica is reading them. This ensures Debezium never misses a change.

-- Concept: monitoring the Debezium replication slot
SELECT
    slot_name,
    plugin,
    active,
    pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS lag_bytes,
    pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) / 1024 / 1024 AS lag_mb
FROM pg_replication_slots;

-- slot_name     | plugin    | active | lag_bytes | lag_mb
-- debezium_slot | pgoutput  | t      | 2097152   | 2.0

-- The slot is 2MB behind the current WAL position.
-- This is normal during steady-state operation.

The danger: if Debezium stops reading (connector crash, Kafka Connect outage), the slot retains WAL segments. If Debezium is down for hours, WAL segments accumulate and the disk fills. PostgreSQL will not recycle WAL segments that a slot references.

-- Concept: the replication slot WAL retention danger
-- Check WAL retention per slot

SELECT
    slot_name,
    active,
    pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) / 1024 / 1024 / 1024 AS lag_gb
FROM pg_replication_slots
WHERE NOT active;

-- slot_name     | active | lag_gb
-- debezium_slot | f      | 48.2

-- The inactive slot has retained 48 GB of WAL.
-- At 2 MB/s WAL generation, that is 6.7 hours of accumulation.
-- If disk space runs out, PostgreSQL stops accepting writes.
-- Fix: either restart Debezium or drop the slot.
-- SELECT pg_drop_replication_slot('debezium_slot');

The diagram shows the CDC pipeline. PostgreSQL writes changes to its WAL (the same WAL from Chapter 3). Debezium reads the WAL through a logical replication slot, translates each change into a structured event, and publishes it to Kafka. Downstream consumers (ClickHouse, Redis, notification service) read from Kafka topics independently. The database’s write path is unchanged. CDC is a read-only consumer of the WAL.

The Transactional Outbox Pattern

CDC captures every database change. Sometimes, the application needs to publish a specific event that does not correspond directly to a table change. For example: “package PKG-40291 is ready for dispatch” is a business event that the application emits, not a database state change.

The outbox pattern writes the event to a dedicated outbox table within the same database transaction as the data change. Debezium reads the outbox table via CDC and publishes the event to Kafka.

-- Concept: transactional outbox table
CREATE TABLE outbox (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type VARCHAR(255) NOT NULL,
    aggregate_id VARCHAR(255) NOT NULL,
    event_type VARCHAR(255) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- In the same transaction as the data change:
BEGIN;

UPDATE packages
SET status = 'READY_FOR_DISPATCH'
WHERE package_id = 'PKG-40291';

INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)
VALUES (
    'Package',
    'PKG-40291',
    'PackageReadyForDispatch',
    '{"packageId": "PKG-40291", "warehouse": "WH-042", "weight": 2.4}'
);

COMMIT;
-- Both writes succeed or both fail. Atomicity guaranteed by the database transaction.
-- Debezium reads the outbox table insert from the WAL and publishes to Kafka.

The outbox pattern solves the dual write problem from Chapter 7: the application writes to one database, and CDC moves the event to Kafka. No dual writes. No distributed transactions.

The Decision Rule

Use CDC (Debezium) when you need to replicate database changes to Kafka, ClickHouse, Elasticsearch, or any downstream system. CDC is transparent to the application code. No application changes are required to start capturing changes.

Use the transactional outbox pattern when the application needs to publish domain events (not just database row changes) with atomicity guarantees. The outbox table participates in the database transaction. Debezium extracts the event.

Monitor the replication slot lag. Alert when lag_gb exceeds a threshold (e.g., 10 GB). If the slot is inactive and accumulating WAL, either restart the connector or drop the slot to prevent disk exhaustion.

The CDC pipeline is the mechanical connection between the single source of truth (PostgreSQL, established in Chapter 7) and the derived stores (Kafka, ClickHouse, Redis). The WAL from Chapter 3 is the interface. The replication slot from Chapter 4 is the consumption mechanism. CDC is replication to an external system, using the same log that replication to a PostgreSQL replica uses.