Stop Tolerating Random LLM Judge Scores: How to Build a Reliable AI Evaluation Gate

LLM-as-judge disagrees with itself between runs

A production faithfulness judge gate failed at 0.79, then passed at 0.82 on the identical input with no code change. Running the same job a third time yielded exactly 0.80, exposing a 0.03 swing from a single floating-point threshold.

Why This Matters

An LLM-as-judge gate that returns different verdicts on identical inputs is worse than no gate—engineers learn to re-run until green, letting real regressions sail through on a lucky pull. In production, a single judge call at non-zero temperature can swing scores by 0.03, meaning a threshold like 0.80 lives inside the noise rather than marking a true regression.

Key Insights

• Floating aliases like ‘gpt-4o’ cause score drift overnight when providers silently update snapshots. Pin the exact dated model version string in the judge call.
• Temperature above zero introduces sampling noise that can flip borderline scores by 1 point on a 5-point scale. Setting temperature to 0 eliminates the largest source of jitter.
• Ambiguous rubrics without anchor definitions for each score level amplify variance. A rubric defining what 3 vs 4 means in concrete terms reduces ambiguity.
• Averaging k=5 judge calls at temperature 0 shrinks variance by approximately sqrt(k). This reduced run-to-run swing on a faithfulness metric from 0.03 to under 0.01.
• Quantizing continuous scores to a coarse grid (0.0, 0.25, 0.5, 0.75, 1.0) prevents sub-threshold jitter from moving the aggregate across the gate boundary.

Working Examples

Implements a stable LLM-as-judge evaluation with k=5 runs, score quantization, and a noise-aware gate that fails only when the mean is below threshold by more than the measured standard deviation.

import statistics
from typing import Callable

def stable_judge_score(
    judge: Callable[[str, str], float],
    output: str,
    reference: str,
    k: int = 5,
    quantize_to: float = 0.25,
) -> tuple[float, float]:
    scores = []
    for _ in range(k):
        raw = judge(output, reference)
        snapped = round(raw / quantize_to) * quantize_to
        scores.append(snapped)
    mean = statistics.fmean(scores)
    stdev = statistics.pstdev(scores) if k > 1 else 0.0
    return mean, stdev

def gate(mean: float, stdev: float, threshold: float = 0.80) -> bool:
    return mean >= (threshold - stdev)

if __name__ == "__main__":
    def fake_judge(out: str, ref: str) -> float:
        return 0.79 if "borderline" in out else 0.9

    mean, stdev = stable_judge_score(fake_judge, "a borderline answer", "ref", k=5)
    passed = gate(mean, stdev, threshold=0.80)
    print(f"mean={mean:.3f} stdev={stdev:.3f} pass={passed}")
    raise SystemExit(0 if passed else 1)

Practical Applications

• CI/CD gate for faithfulness evaluation: averaging k=5 judge calls with quantization reduces the run-to-run swing from 0.03 to under 0.01, enabling reliable pass/fail decisions.
• Threshold-based scoring: applying a noise band around the threshold that passes when mean is within one standard deviation prevents flapping gates that engineers learn to re-run.
• Version-controlled judge prompt: storing the rubric as code with a version string allows bisecting score changes and prevents silent modifications made in a UI from triggering regressions.

References:

• https://dev.to/ethanwritesai/llm-as-judge-disagrees-with-itself-between-runs-2dp4