Migrating from CRUD to Event Sourcing: The Incremental Strategy
Migrating from CRUD to Event Sourcing
The incremental migration has four phases. Phase 1 introduces shadow events from CRUD mutations via a bridge. Phase 2 builds new features on the event store while old features remain on CRUD. Phase 3 backfills historical data into synthetic event streams. Phase 4 switches all reads and writes to the event store and retires the CRUD tables. Continuous verification runs throughout, comparing both data stores.
The decision has been made. The order management system, currently a CRUD application with a PostgreSQL database, needs event sourcing. The audit requirements are increasing. The business wants real-time dashboards, full-text search, and the ability to replay historical order data for analytics. Chapter 16’s decision framework has been applied, and event sourcing is justified for this bounded context.
The question is how to get there without shutting down the system for six months.
A big-bang rewrite replaces the entire application at once. The team builds the event-sourced version in parallel, then switches over on a weekend. This approach fails for three reasons. First, the rewrite takes longer than estimated because the team discovers edge cases that the original system handles but no one documented. Second, the parallel codebase diverges from the production system as the production system continues to receive bug fixes and feature changes. Third, the cutover is a single point of failure: if the new system has a critical bug, there is no gradual rollback.
The incremental strategy migrates one piece at a time. The CRUD system continues to run. Event sourcing is introduced alongside it. The two systems coexist until the migration is complete.
Phase 1: Shadow Events
The Problem
The CRUD system processes orders. Each mutation (insert, update) changes rows in the orders table. No events exist. The first step is to generate events from CRUD mutations without changing the CRUD system’s behavior.
The Mechanism
A migration bridge intercepts CRUD mutations and writes corresponding events to the event store. The CRUD database remains the source of truth. The events are shadows: they mirror state changes but do not drive them.
// FROM SCRATCH
public class MigrationBridge {
private final DataSource crudDataSource;
private final EventStore eventStore;
private final ObjectMapper mapper;
public MigrationBridge(DataSource crudDataSource, EventStore eventStore,
ObjectMapper mapper) {
this.crudDataSource = crudDataSource;
this.eventStore = eventStore;
this.mapper = mapper;
}
public void bridgeOrderInsert(String orderId) {
// Read the current state from CRUD
CrudOrder order = loadCrudOrder(orderId);
if (order == null) return;
// Generate the corresponding event
OrderPlaced event = new OrderPlaced(
order.orderId(),
order.customerId(),
order.lineItems(),
order.total(),
order.deliveryAddress(),
order.createdAt(),
"",
1
);
String streamId = "order-" + orderId;
try {
String payload = mapper.writeValueAsString(event);
eventStore.append(streamId, 0,
List.of(new EventData("OrderPlaced", payload, order.createdAt())));
} catch (JsonProcessingException e) {
throw new MigrationException("Failed to serialize event for " + orderId, e);
}
}
public void bridgeOrderUpdate(String orderId, String previousStatus,
String newStatus) {
String streamId = "order-" + orderId;
long currentVersion = eventStore.streamVersion(streamId);
String eventType = deriveEventType(previousStatus, newStatus);
if (eventType == null) return;
Object event = createTransitionEvent(orderId, previousStatus, newStatus);
try {
String payload = mapper.writeValueAsString(event);
eventStore.append(streamId, currentVersion,
List.of(new EventData(eventType, payload, Instant.now())));
} catch (JsonProcessingException e) {
throw new MigrationException("Failed to bridge update for " + orderId, e);
}
}
private String deriveEventType(String from, String to) {
return switch (to) {
case "CONFIRMED" -> "OrderConfirmed";
case "PAYMENT_AUTHORIZED" -> "PaymentAuthorized";
case "SHIPPED" -> "FulfilmentDispatched";
case "CANCELLED" -> "OrderCancelled";
default -> "OrderStatusChanged";
};
}
private Object createTransitionEvent(String orderId, String from, String to) {
// Create a generic transition event for status changes
return Map.of(
"orderId", orderId,
"previousStatus", from,
"newStatus", to,
"occurredAt", Instant.now().toString()
);
}
private CrudOrder loadCrudOrder(String orderId) {
try (Connection conn = crudDataSource.getConnection();
PreparedStatement stmt = conn.prepareStatement(
"SELECT * FROM orders WHERE order_id = ?")) {
stmt.setString(1, orderId);
try (ResultSet rs = stmt.executeQuery()) {
if (rs.next()) {
return mapToCrudOrder(rs);
}
}
} catch (SQLException e) {
throw new MigrationException("Failed to load CRUD order " + orderId, e);
}
return null;
}
private CrudOrder mapToCrudOrder(ResultSet rs) throws SQLException {
return new CrudOrder(
rs.getString("order_id"),
rs.getString("customer_id"),
List.of(), // Line items loaded separately
rs.getBigDecimal("total"),
null, // Address loaded separately
rs.getTimestamp("created_at").toInstant()
);
}
private record CrudOrder(String orderId, String customerId, List<LineItem> lineItems,
BigDecimal total, Address deliveryAddress, Instant createdAt) {}
}What the Implementation Reveals
The bridge has a fundamental problem: it generates events after the CRUD mutation, not atomically with it. If the application crashes between the CRUD write and the event write, the event is lost. This is acceptable for shadow events because the CRUD database is still the source of truth. The events are supplementary. Missing a shadow event means the event store is temporarily inconsistent with the CRUD database, but the CRUD database is correct.
For stronger guarantees, use a database trigger or Change Data Capture (CDC) to generate events.
CDC-Based Shadow Events
// FROM SCRATCH
// PostgreSQL trigger approach: capture changes in an outbox table
// The trigger fires on every INSERT/UPDATE to the orders table-- FROM SCRATCH
CREATE TABLE migration_outbox (
id BIGSERIAL PRIMARY KEY,
table_name VARCHAR(100) NOT NULL,
operation VARCHAR(10) NOT NULL, -- INSERT, UPDATE, DELETE
record_id VARCHAR(255) NOT NULL,
old_data JSONB,
new_data JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
processed BOOLEAN NOT NULL DEFAULT FALSE
);
CREATE OR REPLACE FUNCTION capture_order_changes()
RETURNS TRIGGER AS $$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO migration_outbox (table_name, operation, record_id, new_data)
VALUES ('orders', 'INSERT', NEW.order_id, row_to_json(NEW)::jsonb);
ELSIF TG_OP = 'UPDATE' THEN
INSERT INTO migration_outbox (table_name, operation, record_id, old_data, new_data)
VALUES ('orders', 'UPDATE', NEW.order_id,
row_to_json(OLD)::jsonb, row_to_json(NEW)::jsonb);
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER orders_capture_changes
AFTER INSERT OR UPDATE ON orders
FOR EACH ROW EXECUTE FUNCTION capture_order_changes();The CDC approach captures every mutation at the database level. A relay process reads the migration outbox and writes events to the event store. This is more reliable than application-level bridging because it catches mutations from all sources: the application, database migrations, manual fixes, and background jobs.
Phase 2: Event Store for New Features
The Mechanism
New features are built on the event store. Old features continue to use the CRUD database. The application has two data paths that coexist.
// FROM SCRATCH
public class HybridOrderService {
private final OrderRepository crudRepository; // Existing CRUD
private final EventStore eventStore; // New event store
private final UpcastingEventTypeRegistry registry;
private final MigrationBridge bridge;
// Existing operations: use CRUD, bridge to event store
public void placeOrder(PlaceOrderCommand command) {
Order order = new Order(command);
crudRepository.save(order); // CRUD write
bridge.bridgeOrderInsert(order.getId()); // Shadow event
}
public void confirmOrder(String orderId) {
Order order = crudRepository.findById(orderId);
String previousStatus = order.getStatus();
order.confirm();
crudRepository.save(order); // CRUD write
bridge.bridgeOrderUpdate(orderId, previousStatus, "CONFIRMED"); // Shadow event
}
// New operations: use event store directly
public OrderAnalytics getOrderAnalytics(String orderId) {
// New feature, built on event store
String streamId = "order-" + orderId;
List<StoredEvent> events = eventStore.readStream(streamId, 0);
return OrderAnalyticsProjection.compute(events, registry);
}
public List<OrderTimelineEntry> getOrderTimeline(String orderId) {
// New feature, built on event store
String streamId = "order-" + orderId;
List<StoredEvent> events = eventStore.readStream(streamId, 0);
return events.stream()
.map(e -> new OrderTimelineEntry(
e.eventType(),
e.occurredAt(),
e.payload()))
.toList();
}
}What the Implementation Reveals
The hybrid service has dual data paths. This is temporary but necessary. The team ships new features (analytics, timeline) on the event store while the existing features (place order, confirm order) continue to use CRUD. The bridge ensures the event store stays synchronized.
The risk during this phase is data inconsistency between CRUD and the event store. The verification test (below) mitigates this risk. Run it continuously in production to detect divergence.
Phase 3: Backfilling Historical Data
The Problem
The event store has shadow events for all orders created after the bridge was deployed. Orders created before the bridge have no events. To rebuild projections from the event store (and eventually retire the CRUD tables), historical orders must be backfilled.
The Mechanism
Read every order from the CRUD database and generate a synthetic event stream for it. The synthetic stream contains the events that would have been produced if the system had been event-sourced from the start.
// FROM SCRATCH
public class HistoricalBackfiller {
private final DataSource crudDataSource;
private final EventStore eventStore;
private final ObjectMapper mapper;
public HistoricalBackfiller(DataSource crudDataSource, EventStore eventStore,
ObjectMapper mapper) {
this.crudDataSource = crudDataSource;
this.eventStore = eventStore;
this.mapper = mapper;
}
public BackfillResult backfill(int batchSize) {
int processed = 0;
int skipped = 0;
int errors = 0;
String sql = """
SELECT order_id FROM orders
WHERE order_id NOT IN (
SELECT REPLACE(stream_id, 'order-', '')
FROM event_store
WHERE event_type = 'OrderPlaced'
)
ORDER BY created_at
LIMIT ?
""";
try (Connection conn = crudDataSource.getConnection();
PreparedStatement stmt = conn.prepareStatement(sql)) {
stmt.setInt(1, batchSize);
try (ResultSet rs = stmt.executeQuery()) {
while (rs.next()) {
String orderId = rs.getString("order_id");
try {
backfillOrder(orderId);
processed++;
} catch (Exception e) {
errors++;
}
}
}
} catch (SQLException e) {
throw new BackfillException("Backfill query failed", e);
}
return new BackfillResult(processed, skipped, errors);
}
private void backfillOrder(String orderId) throws JsonProcessingException {
CrudOrderWithHistory order = loadOrderWithHistory(orderId);
String streamId = "order-" + orderId;
List<EventData> events = new ArrayList<>();
// Generate OrderPlaced
OrderPlaced placed = new OrderPlaced(
order.orderId(), order.customerId(), order.lineItems(),
order.total(), order.deliveryAddress(), order.createdAt(), "", 1
);
events.add(new EventData("OrderPlaced",
mapper.writeValueAsString(placed), order.createdAt()));
// Generate status transition events from audit log if available
for (StatusChange change : order.statusChanges()) {
String eventType = deriveEventType(change.toStatus());
Map<String, Object> payload = Map.of(
"orderId", orderId,
"occurredAt", change.changedAt().toString()
);
events.add(new EventData(eventType,
mapper.writeValueAsString(payload), change.changedAt()));
}
eventStore.append(streamId, 0, events);
}
private String deriveEventType(String status) {
return switch (status) {
case "CONFIRMED" -> "OrderConfirmed";
case "SHIPPED" -> "FulfilmentDispatched";
case "CANCELLED" -> "OrderCancelled";
default -> "OrderStatusChanged";
};
}
private CrudOrderWithHistory loadOrderWithHistory(String orderId) {
// Load order + status changes from CRUD tables
// Implementation depends on existing schema
return null;
}
public record BackfillResult(int processed, int skipped, int errors) {}
private record StatusChange(String fromStatus, String toStatus, Instant changedAt) {}
private record CrudOrderWithHistory(String orderId, String customerId,
List<LineItem> lineItems, BigDecimal total,
Address deliveryAddress, Instant createdAt,
List<StatusChange> statusChanges) {}
}What the Implementation Reveals
Backfilling is imprecise. The CRUD database stores current state, not history. If the order is currently “shipped,” you know it was placed and shipped, but you may not know the exact timestamps of intermediate states (confirmed, payment authorized). If the CRUD system has an audit log, you can reconstruct the state changes. If it does not, you generate synthetic events with estimated timestamps.
The synthetic events are marked with metadata indicating they are backfilled, not original. This prevents confusion when debugging.
// Metadata for backfilled events
Map<String, Object> metadata = Map.of(
"source", "backfill",
"backfilledAt", Instant.now().toString(),
"originalSource", "crud_database"
);Backfilling is idempotent. The query filters out orders that already have an OrderPlaced event. Running the backfiller multiple times processes only orders that have not been backfilled yet.
Phase 4: Retiring CRUD Tables
The Mechanism
Once all features read from the event store (or from projections built from the event store), and the backfill is complete, the CRUD tables can be retired. This is the final phase and the most dangerous.
The retirement is gradual:
-
Read switchover. All read queries now go to projections built from the event store. The CRUD tables are no longer queried for reads. Verify that projection-based reads produce the same results as CRUD reads.
-
Write switchover. Commands now produce events directly instead of CRUD mutations. The migration bridge is reversed: instead of CRUD-to-events, events drive projections that replace the CRUD tables.
-
CRUD tables become read-only. The tables are kept for rollback but no longer receive writes.
-
CRUD tables are archived and dropped. After a confidence period (weeks, not days), the tables are archived and dropped.
// FROM SCRATCH
// Read switchover verification
public class ReadSwitchoverVerifier {
private final DataSource crudDataSource;
private final DataSource projectionDataSource;
public ReadSwitchoverVerifier(DataSource crudDataSource,
DataSource projectionDataSource) {
this.crudDataSource = crudDataSource;
this.projectionDataSource = projectionDataSource;
}
public VerificationResult verify(String orderId) {
OrderView crudView = loadFromCrud(orderId);
OrderView projectionView = loadFromProjection(orderId);
if (crudView == null && projectionView == null) {
return VerificationResult.match("Both null");
}
if (crudView == null || projectionView == null) {
return VerificationResult.mismatch(
"One side null: crud=" + crudView + ", projection=" + projectionView);
}
List<String> differences = new ArrayList<>();
if (!Objects.equals(crudView.status(), projectionView.status())) {
differences.add("status: crud=" + crudView.status() +
", projection=" + projectionView.status());
}
if (crudView.total().compareTo(projectionView.total()) != 0) {
differences.add("total: crud=" + crudView.total() +
", projection=" + projectionView.total());
}
if (differences.isEmpty()) {
return VerificationResult.match("All fields match");
}
return VerificationResult.mismatch(String.join("; ", differences));
}
public VerificationSummary verifyBatch(int sampleSize) {
List<String> orderIds = sampleOrderIds(sampleSize);
int matches = 0;
int mismatches = 0;
List<String> mismatchDetails = new ArrayList<>();
for (String orderId : orderIds) {
VerificationResult result = verify(orderId);
if (result.matches()) {
matches++;
} else {
mismatches++;
mismatchDetails.add(orderId + ": " + result.details());
}
}
return new VerificationSummary(matches, mismatches, mismatchDetails);
}
private List<String> sampleOrderIds(int size) {
List<String> ids = new ArrayList<>();
try (Connection conn = crudDataSource.getConnection();
PreparedStatement stmt = conn.prepareStatement(
"SELECT order_id FROM orders ORDER BY RANDOM() LIMIT ?")) {
stmt.setInt(1, size);
try (ResultSet rs = stmt.executeQuery()) {
while (rs.next()) {
ids.add(rs.getString("order_id"));
}
}
} catch (SQLException e) {
throw new VerificationException("Failed to sample orders", e);
}
return ids;
}
private OrderView loadFromCrud(String orderId) { return null; /* implementation */ }
private OrderView loadFromProjection(String orderId) { return null; /* implementation */ }
private record OrderView(String orderId, String status, BigDecimal total) {}
public record VerificationResult(boolean matches, String details) {
static VerificationResult match(String details) { return new VerificationResult(true, details); }
static VerificationResult mismatch(String details) { return new VerificationResult(false, details); }
}
public record VerificationSummary(int matches, int mismatches, List<String> mismatchDetails) {}
}The Production Path
// PRODUCTION
@Component
public class MigrationVerificationJob {
private final ReadSwitchoverVerifier verifier;
@Scheduled(fixedRate = 300000) // Every 5 minutes
public void verifyConsistency() {
var summary = verifier.verifyBatch(100);
if (summary.mismatches() > 0) {
log.warn("Migration verification found {} mismatches out of {} samples",
summary.mismatches(), summary.matches() + summary.mismatches());
for (String detail : summary.mismatchDetails()) {
log.warn("Mismatch: {}", detail);
}
// Emit metric for alerting
meterRegistry.counter("migration.verification.mismatches")
.increment(summary.mismatches());
} else {
log.info("Migration verification: all {} samples match", summary.matches());
}
}
}The Test
// FROM SCRATCH
@Testcontainers
class MigrationBridgeTest {
@Container
static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:16")
.withDatabaseName("migration_test");
@Test
void crudAndEventStoreProduceSameState() throws Exception {
var ds = createDataSource();
createCrudTables(ds);
createEventStoreTables(ds);
var mapper = new ObjectMapper();
mapper.registerModule(new JavaTimeModule());
mapper.configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false);
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
var eventStore = new EventStore(ds, mapper);
var bridge = new MigrationBridge(ds, eventStore, mapper);
// Create order in CRUD
try (Connection conn = ds.getConnection();
PreparedStatement stmt = conn.prepareStatement(
"INSERT INTO orders (order_id, customer_id, total, status, created_at) " +
"VALUES (?, ?, ?, ?, NOW())")) {
stmt.setString(1, "order-100");
stmt.setString(2, "customer-1");
stmt.setBigDecimal(3, new BigDecimal("49.99"));
stmt.setString(4, "PENDING");
stmt.executeUpdate();
}
// Bridge the insert
bridge.bridgeOrderInsert("order-100");
// Verify event was created
List<StoredEvent> events = eventStore.readStream("order-order-100", 0);
assertEquals(1, events.size());
assertEquals("OrderPlaced", events.get(0).eventType());
// Verify event payload matches CRUD state
JsonNode payload = mapper.readTree(events.get(0).payload());
assertEquals("order-100", payload.get("orderId").asText());
assertEquals("customer-1", payload.get("customerId").asText());
assertEquals(new BigDecimal("49.99"),
payload.get("total").decimalValue());
}
@Test
void backfillerProcessesUnmigratedOrders() throws Exception {
var ds = createDataSource();
createCrudTables(ds);
createEventStoreTables(ds);
var mapper = new ObjectMapper();
mapper.registerModule(new JavaTimeModule());
mapper.configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false);
var eventStore = new EventStore(ds, mapper);
var backfiller = new HistoricalBackfiller(ds, eventStore, mapper);
// Create orders in CRUD (simulating pre-migration data)
try (Connection conn = ds.getConnection();
Statement stmt = conn.createStatement()) {
stmt.executeUpdate(
"INSERT INTO orders (order_id, customer_id, total, status, created_at) " +
"VALUES ('old-1', 'c1', 25.00, 'SHIPPED', '2025-06-01 10:00:00')");
stmt.executeUpdate(
"INSERT INTO orders (order_id, customer_id, total, status, created_at) " +
"VALUES ('old-2', 'c2', 50.00, 'CANCELLED', '2025-07-15 14:30:00')");
}
// Backfill
var result = backfiller.backfill(100);
assertEquals(2, result.processed());
// Verify events exist for both orders
assertFalse(eventStore.readStream("order-old-1", 0).isEmpty());
assertFalse(eventStore.readStream("order-old-2", 0).isEmpty());
}
private DataSource createDataSource() {
var ds = new org.postgresql.ds.PGSimpleDataSource();
ds.setUrl(postgres.getJdbcUrl());
ds.setUser(postgres.getUsername());
ds.setPassword(postgres.getPassword());
return ds;
}
private void createCrudTables(DataSource ds) throws SQLException {
try (Connection conn = ds.getConnection();
Statement stmt = conn.createStatement()) {
stmt.execute("""
CREATE TABLE IF NOT EXISTS orders (
order_id VARCHAR(255) PRIMARY KEY,
customer_id VARCHAR(255) NOT NULL,
total DECIMAL(10,2) NOT NULL,
status VARCHAR(50) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
)
""");
}
}
private void createEventStoreTables(DataSource ds) throws SQLException {
try (Connection conn = ds.getConnection();
Statement stmt = conn.createStatement()) {
stmt.execute("""
CREATE TABLE IF NOT EXISTS event_store (
global_position BIGSERIAL PRIMARY KEY,
stream_id VARCHAR(255) NOT NULL,
sequence_number BIGINT NOT NULL,
event_type VARCHAR(255) NOT NULL,
payload JSONB NOT NULL,
metadata JSONB DEFAULT '{}',
occurred_at TIMESTAMPTZ NOT NULL,
stored_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (stream_id, sequence_number)
)
""");
}
}
}The test creates both CRUD tables and event store tables in the same Testcontainers PostgreSQL instance. It inserts orders through CRUD, bridges them to the event store, and verifies that the event payloads match the CRUD state. This is the critical invariant during migration: both data stores represent the same business state.
Migration from CRUD to event sourcing is not a weekend project. It is a multi-month effort that requires careful planning, continuous verification, and the discipline to keep both systems running correctly during the transition. The incremental approach described in this chapter minimizes risk by ensuring that at every phase, the system can fall back to the CRUD database if the event store has issues.
This chapter completes the book. The seventeen chapters have covered event sourcing from first principles through production operations and strategic decision-making. The patterns, implementations, and trade-offs presented here provide a foundation for building event-sourced systems that work in practice, not just in theory.