Database Observability: An Engineer's Guide to Full-Stack Monitoring Across SQL, NoSQL, and Cloud Databases

Database Observability: An Engineer’s Guide to Full-Stack Monitoring Across SQL, NoSQL, and Cloud Databases

Modern database environments often grow into fragmented three-dashboard setups that fail to correlate data during critical incidents. Observability bridges this gap by connecting application services, SQL statements, and host disk I/O into a single causal chain. By monitoring specific signals like p99 latency and cache hit ratios, engineers can identify root causes before they trigger SLA breaches.

Why This Matters

Metric collection merely identifies threshold breaches, whereas observability provides the distributed trace needed to answer why a system is failing without adding instrumentation mid-incident. In production environments running mixed stacks of PostgreSQL, MongoDB, and Aurora, a lack of unified telemetry results in high operational costs and delayed response times. Ideal models often overlook the reality of engine-specific telemetry behaviors, such as WiredTiger cache eviction patterns in MongoDB or CPU credit exhaustion in burstable cloud instances. These hidden triggers can cause non-linear latency spikes that infrastructure-only monitoring fails to capture.

Key Insights

• PostgreSQL and MySQL cache hit ratios should be maintained at 99% or higher; a ratio below 95% indicates significant performance trouble (Sanoja, 2026).
• MongoDB operation latency must be split by read, write, and command types because asymmetric workloads can hide spikes in combined averages.
• AWS Performance Insights console functionality is migrating to CloudWatch Database Insights with a scheduled EOL of June 30, 2026.
• OpenTelemetry (OTel) Collector receivers for PostgreSQL, MySQL, and MongoDB normalize metrics into shared semantic conventions for vendor-neutral observability.
• Dynamic baselining reduces alert fatigue by firing only when metrics deviate from rolling historical patterns for specific time windows, such as a Tuesday 3am batch job.
• WiredTiger cache utilization in MongoDB is a critical signal; high eviction pressure can cause disk-bound behavior that is invisible to host-level memory metrics.

Working Examples

Calculates the PostgreSQL cache hit ratio while guarding against division-by-zero.

SELECT round(sum(heap_blks_hit)::numeric / nullif(sum(heap_blks_hit + heap_blks_read), 0), 4) AS hit_ratio FROM pg_statio_user_tables;

Surfaces the top 15 PostgreSQL queries by cumulative execution time using pg_stat_statements.

SELECT left(query, 80) AS query_preview, calls, round((total_exec_time / 1000)::numeric, 2) AS total_time_sec, round((mean_exec_time)::numeric, 2) AS avg_ms, rows FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 15;

Aggregates MySQL query fingerprints and execution statistics from the Performance Schema.

SELECT LEFT(DIGEST_TEXT, 120) AS query_digest, COUNT_STAR AS exec_count, ROUND(SUM_TIMER_WAIT / 1e12, 3) AS total_sec, ROUND(AVG_TIMER_WAIT / 1e12, 3) AS avg_sec, SUM_ROWS_EXAMINED, SUM_ROWS_SENT FROM performance_schema.events_statements_summary_by_digest ORDER BY SUM_TIMER_WAIT DESC LIMIT 15;

Extracts operation latency, queue depth, and WiredTiger cache utilization from MongoDB.

const s = db.runCommand({ serverStatus: 1 }); printjson(s.opLatencies); print("Queued ops:", s.globalLock.currentQueue.total); const used = s.wiredTiger.cache["bytes currently in the cache"]; const max = s.wiredTiger.cache["maximum bytes configured"]; print("Cache fill:", (used / max).toFixed(3));

A minimal OpenTelemetry Collector configuration for PostgreSQL metric collection.

receivers:
  postgresql:
    endpoint: localhost:5432
    username: otel_reader
    password: "${env:PGMON_PASS}"
    databases:
      - app_prod
    collection_interval: 20s

Practical Applications

• Application Performance: Correlate APM traces with ‘db.query.text’ attributes to identify specific SQL statements causing p95 latency spikes. Pitfall: Relying on mean latency, which hides outliers impacting real user sessions.
• Cloud Management: Monitor ‘CPUCreditBalance’ on AWS burstable instances (T3/T4g) to detect exhaustion-driven slowdowns. Pitfall: Standard 1-minute CloudWatch intervals may miss transient engine-level anomalies.
• Incident Triage: Use ‘pg_blocking_pids()’ in PostgreSQL to pinpoint the exact session holding a lock during contention events. Pitfall: Implementing static alert thresholds that trigger false positives during predictable cyclical workloads.
• Capacity Planning: Track MongoDB replication oplog windows in hours to ensure lagging secondaries have sufficient buffer for recovery. Pitfall: Maintaining a window under 4 hours on write-heavy deployments, risking full resync requirements.

References:

• https://dev.to/damasosanoja/database-observability-an-engineers-guide-to-full-stack-monitoring-across-sql-nosql-and-cloud-1b3o