Scatter-Gather Operations

When mongos receives a query that includes the shard key in the filter, it routes the query to the specific shard that owns the matching data. This is a targeted query. When the filter does not include the shard key, mongos must send the query to every shard, collect results, and merge them. This is a scatter-gather query.

Scatter-gather queries have two costs: latency (bounded by the slowest shard’s response) and resource consumption (every shard performs the query, even if most return zero results).

Identifying Scatter-Gather in explain()

// Targeted query: includes shard key
db.readings.explain("executionStats").find({
  sensorId: "sensor-00042",
  ts: { $gte: ISODate("2024-06-01"), $lt: ISODate("2024-06-02") }
})

// explain output includes:
// "queryPlanner": {
//   "winningPlan": {
//     "stage": "SINGLE_SHARD",        <- targeted
//     "shards": [{ "shardName": "shard-2", ... }]
//   }
// }

// Scatter-gather query: no shard key in filter
db.readings.explain("executionStats").find({
  temperature: { $gt: 100 }
})

// explain output includes:
// "queryPlanner": {
//   "winningPlan": {
//     "stage": "SHARD_MERGE",          <- scatter-gather
//     "shards": [
//       { "shardName": "shard-1", ... },
//       { "shardName": "shard-2", ... },
//       { "shardName": "shard-3", ... },
//       { "shardName": "shard-4", ... }
//     ]
//   }
// }

The SHARD_MERGE stage in the explain output is the indicator of a scatter-gather query. Every shard executes the query independently, and mongos merges the results.

The Cost Model

For a 4-shard cluster where each shard can answer a query in 5ms:

• Targeted query: 5ms (1 shard) + 1ms (routing) = 6ms
• Scatter-gather: max(5ms, 5ms, 5ms, 5ms) + 1ms (routing) + 2ms (merge) = 8ms

The difference seems small. On a 16-shard cluster:

• Targeted query: 5ms + 1ms = 6ms
• Scatter-gather: max(5ms across 16 shards, with variance) + 1ms + 5ms (merge 16 result sets) = 15ms

The merge cost grows linearly with shard count. Worse, the slowest shard determines the latency. If shard 7 is experiencing a GC pause and responds in 200ms, the scatter-gather query takes 200ms+ even though 15 shards responded in 5ms.

// Measure scatter-gather impact in the application
@BenchmarkMode(Mode.SampleTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
public class ScatterGatherBenchmark {

    // SLOW: Scatter-gather query (no shard key)
    @Benchmark
    public List<Document> scatterGather() {
        return collection.find(Filters.gt("temperature", 100))
            .limit(100)
            .into(new ArrayList<>());
    }

    // FAST: Targeted query (includes shard key)
    @Benchmark
    public List<Document> targeted() {
        return collection.find(
            Filters.and(
                Filters.eq("sensorId", "sensor-00042"),
                Filters.gt("temperature", 100)
            ))
            .limit(100)
            .into(new ArrayList<>());
    }
}