Unbundling the Database

Unbundling the Database: Composing Specialized Tools via Change Logs

The unbundled database architecture trades the operational simplicity of a monolithic system for the flexibility of specialized, independently scalable components. This decomposition is not an optimization—it is a forced choice under scale. The central, immutable log becomes the sole source of truth, and all other systems—caches, indexes, analytics stores—are derived consumers. Their consistency is not guaranteed by design; it is achieved only through deterministic, replayable processing of the log. Failure of any derived system is the expected state; recovery is accomplished by reprocessing from the log’s beginning.

This model inverts traditional data architecture: instead of writing to a database and pushing changes outward, data flows from the log outward to consumers. This is the Outside-In database: the database no longer serves queries directly; it emits events, and external systems materialize state on demand. The log is not a side effect—it is the primary artifact. Queries are answered not by the source of record, but by derived systems whose correctness depends entirely on their ability to consume and interpret the log.

Invariants First: The Foundation of Derived Consistency

Invariant: The output state of any derived data system MUST be a deterministic function of its immutable input log.

This invariant is non-negotiable. Any deviation—non-deterministic processing, stateful transformations, or external side effects—breaks the guarantee of recoverability. To enforce this, two mechanisms are required:

1. Change Data Capture (CDC): The log must capture every state transition, not just final values. This enables reconstruction of history and supports multiple interpretations of the same event stream.
2. Idempotent Writes: Derived systems must apply events in a way that repeated processing does not alter the final state. This is essential for recovery and reprocessing.

Without these, the system cannot tolerate consumer failure, and the unbundled architecture collapses into a fragile, inconsistent collection of caches.

Derived Data Systems: Specialization at the Cost of Coordination

Derived systems are not enhancements—they are necessary compromises. Each optimizes for a specific access pattern (e.g., low-latency lookup, full-text search, aggregation) but introduces operational debt. They are eventually consistent by necessity, not choice.

The following coordination mechanisms enforce consistency across derived systems, each with distinct trade-offs:

Mechanism	Use Case	Trade-off	Failure Mode
Single Log Partition	Key-coordinated event processing	Limits parallelism; creates hot partitions	Backpressure, latency spikes
Distributed Lock	Mutually exclusive access to state	Introduces latency; risk of deadlocks	System-wide stalls
Vector Clock	Detecting concurrent updates	Requires metadata; complex conflict resolution	Inconsistent merges
Idempotent Write	Safe reprocessing	Requires deduplication keys	Storage overhead
Deterministic Replay	Full recovery from log	High replay latency	Extended downtime during rebuild

These are not interchangeable. The choice of mechanism defines the system’s behavior under failure.

Example: Materialized View with Deterministic Replay

A materialized view must satisfy the invariant: its state is a function of the log. The following implementation ensures idempotency and deterministic processing.

from dataclasses import dataclass, field
from typing import Dict, Set
import asyncio

@dataclass(frozen=True)
class StockEvent:
    event_id: str
    product_id: str
    quantity: int
    event_type: str  # 'add' | 'remove'

@dataclass
class StockMaterializedView:
    stock_levels: Dict[str, int] = field(default_factory=dict)
    processed_events: Set[str] = field(default_factory=set)

    async def process_event(self, event: StockEvent) -> None:
        # Idempotency: skip if already processed
        if event.event_id in self.processed_events:
            return

        if event.event_type == 'add':
            self.stock_levels[event.product_id] = self.stock_levels.get(event.product_id, 0) + event.quantity
        elif event.event_type == 'remove':
            new_level = self.stock_levels.get(event.product_id, 0) - event.quantity
            if new_level < 0:
                raise ValueError(f"Insufficient stock for {event.product_id}")
            self.stock_levels[event.product_id] = new_level

        self.processed_events.add(event.event_id)

    async def get_stock_level(self, product_id: str) -> int:
        return self.stock_levels.get(product_id, 0)

# Recovery: replay all events from log
# Failure of the view is irrelevant—state is rebuilt deterministically

Race Conditions: The Inevitable Consequence of Parallel Consumers

Race conditions arise when multiple derived systems process events from the same log without total order guarantees across keys. Consider the following sequence:

1. Event A: remove(5) for product X
2. Event B: add(10) for product X
3. Consumer 1 processes B, then A → final stock: 5
4. Consumer 2 processes A, then B → final stock: 5

While the final state is consistent in this case, the intermediate states differ. If either consumer serves queries during processing, clients observe transient inconsistency. Worse, if events are not idempotent or deduplicated, the final state may diverge.

This problem is not theoretical. It occurs when:

• Events are sharded across consumers by key
• Network partitions delay event delivery
• Consumers restart and reprocess events at different rates

Prevention strategies:

• Single Log Partition per Key: Ensures total order for events affecting the same entity. Limits scalability but guarantees consistency.
• Idempotent Writes with Deduplication Keys: Allows reprocessing without side effects. Required for recovery.
• Monotonic Clocks or Version Vectors: Detect out-of-order delivery, though not sufficient alone for resolution.

The architecture diagram of the Outside-In model illustrates this: the immutable log sits at the center, with multiple derived consumers fanning out. Each consumer reads the same event stream, but their internal state evolves independently. The log does not synchronize them; it only records facts. Synchronization, if required, must be built atop.

Conclusion: Trade-offs, Not Triumphs

The unbundled database is not a superior architecture—it is a different set of trade-offs. It exchanges the predictability of a monolithic database for the scalability of specialized components. It replaces strong consistency with eventual correctness, and simplicity with operational complexity.

The benefit is not elegance, but adaptability: the ability to compose systems that evolve independently, driven by workload demands. The cost is unrelenting: every derived system is a potential point of failure, every query a potential inconsistency. The only certainty is the log. Everything else is derived, temporary, and subject to failure—by design.