Debezium, the WAL, and Streaming Database Changes

The Black Box

The team deploys a Debezium connector and sees change events in Kafka. What they do not see: how Debezium handles the initial snapshot, what happens when a column is added to a captured table, how much WAL Debezium reads per change event, and why the connector periodically falls behind after a schema migration.

The Mechanism

Change Event Structure

Every Debezium change event has a consistent structure:

// Concept: Debezium change event for a package status update
{
    "schema": { ... },
    "payload": {
        "before": {
            "package_id": "PKG-40291",
            "status": "IN_TRANSIT",
            "warehouse_id": "WH-042",
            "updated_at": 1731672000000000
        },
        "after": {
            "package_id": "PKG-40291",
            "status": "DELIVERED",
            "warehouse_id": "WH-042",
            "updated_at": 1731675600000000
        },
        "source": {
            "version": "2.5.0.Final",
            "connector": "postgresql",
            "name": "logistics",
            "ts_ms": 1731675600123,
            "db": "logistics",
            "schema": "public",
            "table": "packages",
            "lsn": 172093440,
            "txId": 58291
        },
        "op": "u",
        "ts_ms": 1731675600234
    }
}

The op field indicates the operation: c (create/insert), u (update), d (delete), r (read, during initial snapshot).

The before field is populated for updates and deletes (requires REPLICA IDENTITY FULL on the table, otherwise only the primary key is included in before).

-- Concept: REPLICA IDENTITY controls what Debezium sees in the "before" image
-- Default: only primary key columns in "before"
-- FULL: all columns in "before" (required for downstream consumers that need the old value)

ALTER TABLE packages REPLICA IDENTITY FULL;

-- Cost: with REPLICA IDENTITY FULL, PostgreSQL writes all column values
-- to the WAL for every UPDATE, not just the changed columns.
-- For a table with 20 columns where only 1 column is updated,
-- WAL volume increases by approximately 20x for that table.
-- Use FULL only on tables where downstream consumers need the old values.

Snapshot Behavior

When Debezium first connects, it can take an initial snapshot of the existing data:

• snapshot.mode=initial: Read all existing rows from captured tables, publish them to Kafka, then start streaming from the WAL. This gives downstream consumers a complete picture.
• snapshot.mode=schema_only: Do not snapshot existing data. Only capture changes from this point forward. Faster startup, but downstream consumers miss historical data.

The initial snapshot of a 10 million row package_events table takes time. Debezium reads the table using a consistent snapshot (SELECT * FROM package_events within a REPEATABLE READ transaction) and publishes each row to Kafka with op: "r".

# Concept: monitoring Debezium snapshot progress
# Check Kafka Connect connector status

curl -s localhost:8083/connectors/logistics-postgres-connector/status | jq .

# {
#   "connector": { "state": "RUNNING" },
#   "tasks": [{
#     "id": 0,
#     "state": "RUNNING",
#     "worker_id": "connect-1:8083",
#     "trace": ""
#   }]
# }

# Check snapshot progress via Debezium metrics (JMX)
# SnapshotCompleted: false
# RemainingTableCount: 2
# TotalTableCount: 3
# NumberOfEventsFiltered: 0
# MilliSecondsSinceLastEvent: 42

The Observable Consequence

Schema Changes

When a column is added to a captured table, Debezium must handle the schema change. The behavior depends on the configuration:

-- Add a new column to the packages table
ALTER TABLE packages ADD COLUMN estimated_delivery TIMESTAMPTZ;

After this DDL:

1. PostgreSQL writes the schema change to the WAL.
2. Debezium detects the schema change.
3. New change events include the estimated_delivery field.
4. Downstream consumers that deserialize Debezium events must handle the new field.

If the downstream consumer uses a strict schema (e.g., a Java class expecting exactly 5 fields), the new field causes a deserialization error. This is the most common Debezium pipeline failure: a schema change in PostgreSQL breaks a downstream consumer.

The fix is schema compatibility. Use Avro serialization with a Schema Registry configured for BACKWARD compatibility. New fields must have a default value. Old consumers can ignore new fields.

The Code

A downstream consumer that processes Debezium events and populates Redis:

// Concept: consuming Debezium CDC events to update Redis cache
// The consumer reads from the Kafka topic that Debezium writes to.

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("cdc.packages"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        JsonNode event = objectMapper.readTree(record.value());
        String op = event.path("payload").path("op").asText();
        JsonNode after = event.path("payload").path("after");

        if ("c".equals(op) || "u".equals(op) || "r".equals(op)) {
            String packageId = after.path("package_id").asText();
            String status = after.path("status").asText();
            redis.setex("pkg:status:" + packageId, 300, status);
        } else if ("d".equals(op)) {
            JsonNode before = event.path("payload").path("before");
            String packageId = before.path("package_id").asText();
            redis.del("pkg:status:" + packageId);
        }
    }
    consumer.commitSync();
}

The Decision Rule

Set REPLICA IDENTITY FULL only on tables where downstream consumers need the old values (the before field). For tables where only the new state matters (the after field), leave the default identity (primary key only) to minimize WAL volume.

Use snapshot.mode=initial for the first deployment to populate downstream stores. After the initial snapshot, subsequent restarts of the connector resume from the WAL position without re-snapshotting (unless the replication slot is dropped).

Monitor the replication slot lag. If the connector falls behind by more than your staleness budget, investigate Kafka Connect worker health, connector errors, and throughput limits. The most common cause of connector lag is a slow Kafka cluster or a misconfigured Kafka Connect worker with insufficient memory or threads.