Kafka 4.0+: Mastering KRaft, Incremental Rebalancing, and Production Python Patterns

The Big Change in 2025: KRaft Replaces ZooKeeper

Apache Kafka has transitioned to KRaft mode as of version 4.0. This architectural shift completely eliminates the requirement for a separate ZooKeeper service for cluster metadata management.

Why This Matters

While theoretical models suggest seamless scaling, legacy Kafka deployments suffered from ‘stop-the-world’ rebalances where all consumers paused processing during group changes. In large consumer groups with dozens of partitions, this created significant operational gaps and latency spikes that disrupted real-time data pipelines.

Key Insights

• KRaft (Kafka Raft Metadata) became the exclusive metadata management system in Kafka 4.0 (released March 2025), rendering ZooKeeper-based setups legacy.
• Incremental rebalancing via KIP-848 (GA in Kafka 4.0) allows only affected partitions to move during consumer group changes rather than revoking all partitions.
• At-least-once delivery is the production standard for data engineering, utilizing manual commits after processing paired with idempotent sinks like PostgreSQL ‘ON CONFLICT’ upserts.
• Producer throughput is optimized using ‘linger.ms’ (e.g., set to 10ms) to batch messages more efficiently and reduce network round trips.

Working Examples

KRaft mode Docker Compose configuration for Kafka 4.1 / Confluent Platform 8.1.

version: '3.8'
services:
  kafka:
    image: confluentinc/cp-kafka:8.1.2
    container_name: kafka
    hostname: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_AUTOCREATE TOPICS ENABLE: 'true'
      CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
    volumes:
      - kafka_{data}:/var/lib/kafka/data

Idempotent Python producer pattern using confluent-kafka.

from confluent_kafka import Producer
import json

KAFKA_CONFIG = {
    'bootstrap.servers': 'localhost:9092',
    'acks': 'all',
    'enable.idempotence': True,
    'compression.type': 'snappy',
    'linger.ms': 10,
}
producer = Producer(KAFKA_CONFIG)
def produce(topic, key, value):	producer.produce(topic=topic, key=key, value=json.dumps(value), callback=delivery报告)	producer poll(0)

Practical Applications

• • Real-time ingestion systems using Kafka Connect + Debezium for Change Data Capture (CDC) from PostgreSQL write-ahead logs.
• • High-throughput DB loaders utilizing a batch consumer pattern that polls messages into a list and performs bulk upserts before calling consumer.commit().
• • Error handling via Dead Letter Queues (DLQ), routing malformed JSON or invalid messages to a separate topic to prevent pipeline blockage.

References:

• https://dev.to/de_{clerke}/kafka-for-data-engineers-core-concepts-kraft-and-the-patterns-that actually work -3d0m