Building Real-Time Streaming Systems with Apache Kafka and Python

Apache Kafka for Beginners: Building Real-Time Streaming Systems with Python

Apache Kafka is a distributed event streaming platform used across banking, e-commerce, and healthcare. It allows organizations to process massive streams of data continuously while maintaining historical logs for replay.

Why This Matters

In traditional distributed systems, tight coupling between producers and consumers often leads to system-wide failures when one component slows down. Kafka solves this by decoupling these layers, allowing producers to continue sending data regardless of consumer state, thereby transforming fragile linear pipelines into resilient, asynchronous architectures that can handle extreme throughput without data loss.

Key Insights

• Exactly-Once Semantics (EOS) prevents duplicate data during failures, which is critical for financial transactions and payment platforms.
• Horizontal scalability is achieved by adding brokers and dividing topics into partitions, enabling parallel processing by multiple consumers.
• KRaft mode eliminates the dependency on ZooKeeper for cluster coordination and metadata management, resulting in faster startup and simpler architecture.
• Idempotent producers ensure a message is never written twice during retries by using specific configurations like ‘acks=all’ and defined retry limits.

Working Examples

A basic Python producer that sends a stream of student records to a Kafka topic.

from kafka import KafkaProducer
import json
import time

producer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

students = [
    {"id": 1, "name": "Samwel", "course": "Data Engineering"},
    {"id": 2, "name": "Alice", "course": "AI Engineering"},
    {"id": 3, "name": "John", "course": "Cloud Computing"}
]

for student in students:
    producer.send("students", value=student)
    print("Message sent:", student)
    time.sleep(1)

producer.flush()
producer.close()

A Python consumer that reads messages from the ‘students’ topic using a group ID for load balancing.

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'students',
    bootstrap_servers='localhost:9092',
    group_id='etl-group',
    auto_offset_reset='latest',
enable_auto_commit=True,
    value_deserializer=lambda x: json.loads(x.decode('utf-8'))
)
e
iprint("Consumer running...")
or message in consumer:
data = message.value
iprint("Received:", data)

Practical Applications

References:

• [type]: ‘text’, ‘text’: ‘```json You are a technical writer for engineers. Write a concise, high-signal article from the provided context. ano’]}