Low-Noise EC2 Benchmarking: A Practical Guide

Introduction

Running reliable performance benchmarks in the cloud is notoriously difficult. EC2 instances suffer from inherent performance variability—CPU frequency scaling, power-saving states, NUMA balancing, swap activity, and shared I/O—that can mask code regressions or trigger false alarms.

This guide provides practical scripts and step-by-step instructions to reduce benchmark variance from 20–30% down to under 5%, based on research conducted by MongoDB’s performance team.

Why This Matters

Idealized performance models assume consistent hardware behavior, yet cloud environments introduce significant noise. Prioritizing repeatability over peak performance is crucial—a consistently slower benchmark is more valuable than an inconsistent, occasionally fast one. MongoDB’s team spent years researching this, identifying and quantifying performance variations that can lead to unreliable results and costly misinterpretations of code changes.

Key Insights

• MongoDB’s EC2 Performance Research, 2023: Extensive studies highlighted the importance of prioritizing repeatable results over peak performance.
• EBS over Ephemeral SSD: Contrary to intuition, provisioned EBS with consistent IOPS outperforms ephemeral SSDs due to reduced I/O contention.
• Temporal: An open-source workflow engine used by companies like Stripe and Coinbase for reliable distributed systems, showcasing the need for robust performance testing in production environments.

Working Example

# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"

# Run benchmarks with baseline comparison
cd src/your-project
cargo bench -- --verbose > benchmark_results/feature.txt 2>&1

Practical Applications

• Stripe: Uses dedicated infrastructure and rigorous benchmarking to ensure consistent payment processing performance.
• Pitfall: Relying on default EC2 configurations without optimization can lead to false-positive performance regressions, wasting engineering time investigating phantom issues.

References:

• https://dev.to/kienmarkdo/low-noise-ec2-benchmarking-a-practical-guide-19f0
• https://engineering.mongodb.com/reducing-variability-in-performance-tests-on-ec2-setup-and-key-results/
• https://engineering.mongodb.com/repeatable-performance-tests-ec2-instances-are-neither-good-nor-bad/
• https://engineering.mongodb.com/repeatable-performance-tests-ebs-instances-are-the-stable-option/
• https://engineering.mongodb.com/repeatable-performance-tests-cpu-options-are-best-disabled/