Redesigning a Failing Data Pipeline to Eliminate Cascading Failures

Redesigning a Failing Data Pipeline to Eliminate Cascading Failures

Marwa Ahmed redesigned a failing data pipeline for an activity tracking system, which was breaking under load due to a tightly coupled Lambda architecture. The new design achieved a 99.7% ingestion success rate and eliminated cascading failures during traffic spikes. The redesign used AWS managed services, including DynamoDB, EventBridge Pipes, SQS, and Lambda, and was provisioned using Terraform.

Why This Matters

The original architecture was prone to cascading failures, with a blast radius that affected the entire system, resulting in a 2-hour delay in leaderboard updates and significant operational costs. The redesigned architecture prioritized decoupling, durability, and observability, enabling the system to handle burst traffic patterns and reducing the mean time to recovery from hours to minutes.

Key Insights

• The original architecture had a 95% ingestion success rate, with Lambda throttling errors spiking during peak hours: “During peak hours, employee activity submissions would time out, retry storms would cascade through the system” (Marwa Ahmed, 2026)
• The redesigned architecture used SQS to introduce durable buffering and managed retries, reducing cascading failures: “Amazon SQS introduces durable buffering between ingestion and processing” (Marwa Ahmed, 2026)
• Terraform enabled safe migration and reproducible environments across dev/staging/prod: “I provisioned all components using Terraform, including DynamoDB tables and streams” (Marwa Ahmed, 2026)

Working Example

# Configure the AWS provider
provider "aws" {
  region = "us-west-2"
}

# Create a DynamoDB table
resource "aws_dynamodb_table" "activity_events" {
  name           = "activity-events"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "id"
  attribute {
    name = "id"
    type = "S"
  }
}

# Create an SQS queue
resource "aws_sqs_queue" "activity_queue" {
  name                        = "activity-queue"
  delay