She Replaced Vibes With Metrics How One Team Cuts Hallucinations By Automating LLM Evaluations In Production

The Problem Why Vibe Checks Fail in Production

The team shipped a RAG based customer support assistant that confidently cited nonexistent policies to hundreds of users before anyone caught it. The post mortem revealed zero automated evaluation their process was literally ask questions read answers thumbs up.

Why This Matters

The gap between demo ready chatbot vibes based testing versus reliable production behavior caused this team s RAG system hallucinate non existent billing policies reach customers before detection creating trust deficit hard repair Real world failures aren t captured academic benchmarks MMLU HellaSwag demand domain specific continuous quantitative measurement integrated directly development workflow else risk similar incidents scale.

Key Insights

• A golden dataset should start with real production cases not thousands of synthetic ones stratified as percent basic happy path percent edge case percent adversarial intent and percent multilingual long context per the article source.
• A judge ensemble combines multiple specialized evaluators beyond generic RAGAS faithfulness instruction following JSON schema safety and domain expert each with explicit thresholds like for faithfulness per the article source.
• A custom LLM judge can be built using structured output libraries like instructor from openai to produce typed scores and reasoning grounded in domain specific criteria with few shot examples per the article source.
• CICD integration enforces that every prompt change triggers an automated evaluation blocking merges that degrade quality by more than five percent mean score drop compared to a git tracked baseline per the article source.

Working Examples

Core abstractions for an LLM evaluation pipeline including TestCase, EvaluationResult, Judge (abstract), and EvaluationHarness.

# eval/base.py
@dataclass(frozen=True)
class TestCase:
    id: str
    input: dict[str, Any]
    expected: dict[str, Any] | None = None
    tags: list[str] = field(default_factory=list) # ["edge-case", "long-context"]

@dataclass(frozen=True)
class EvaluationResult:
    test_case_id: str
    judge_name: str
    score: float
    passed: bool
    reasoning: str

class Judge(ABC):
    @abstractmethod
    async def evaluate(self, test_case: TestCase, response: Any) -> EvaluationResult: ...

class EvaluationHarness:
    def __init__(self, judges: list[Judge]):
        self.judges = judges

    async def evaluate_all(self, test_cases, generate_fn, concurrency=10):
        # Runs all cases through all judges with controlled concurrency
        ...

Custom LLM judge implementation with structured output validation using instructor and OpenAI.

# eval/judges.py
class LLMJudge(Judge):
    def __init__(self, name, criteria, model="gpt-4o-mini", few_shot_examples=None):
        self.name = name
        self.criteria = criteria
        self.model = model
        self.few_shot = few_shot_examples or []

async def evaluate(self, test_case, response):
        client = instructor.from_openai(AsyncOpenAI())
        class Output(BaseModel):
            score: float = Field(ge=0, le=1)
            reasoning: str
            passed: bool
        result = await client.chat.completions.create(
            model=self.model,
            response_model=Output,
            messages=[
                {"role": "system", "content": self._system_prompt()},
                *self._few_shot_messages(),
                {"role": "user", "content": self._build_prompt(test_case, response)},
            ],
            temperature=0.0,
        )
return EvaluationResult(...)

# .github/workflows/llm-eval.yml
name: LLM Evaluation
on:
pull_request:
dirs_paths**: ['prompts/**', 'eval/**']
schedule:
cron**: '0 2 * * *' # Nightly

jobs:
evaluate:
runs-on**: ubuntu-latest

steps**:**

- uses**: actions/checkout@v4

- uses**: actions/setup-python@v5

with**: {python-version:'3.11'}

CICD integration for automated LLM evaluation with GitHub Actions triggering on pull requests to promptseval paths and nightly runs including regression checks and PR commenting.

- name**: Install deps run:pip install -e .[dev] - name:**Run evaluation env:** OPENAI_API_KEY:${{ secrets.OPENAI_API_KEY }} run:** python -m eval.run_suite --config config.yaml --output results.json - name:**Check regressions run:** python -m eval.check_regression --baseline baseline.json --current results.json - name:**Comment PR if:** github.event_name== 'pull_request' uses:**actions/github-script@v7 with:** script:| const results=JSON.parse(fs.readFileSync('results.json'));const body=`##LLM Eval Results|Judge|Pass Rate|Mean Score||-------|-----------|------------|${Object.entries(results.summary).map(([k,v])=>`|${k}|${(v.pass_rate*100).toFixed(1)}%|${v.mean_score.toFixed(3)}|`).join('
in')}`;github.rest.issues.createComment({issue_number:context.issue.number owner context.repo.owner repo context.repo.repo body});``` }}}}}}%                             

Practical Applications

• Catch hallucinations automatically before deployment by running an ensemble of faithfullness safety and instruction following judges against a golden dataset every time prompts change as described in the article source failing a CI check when any judge drops below its threshold such as faithfulness below point eight per the article source.
• Iterate on prompts times faster by reducing manual review cycles from two hours to fifteen minutes using automated regression detection that flags any five percent mean score drop between baseline and current results allowing teams to tune prompts without waiting for human reviewers per the article source.
• A common pitfall is starting evaluation with thousands of synthetic test cases which lack real world failure modes instead teams should begin with just fifty real production logs grow the set by converting every production incident into a new golden test case stratifying across categories like basic edge case adversarial multilingual per the article source.

References:

• Evaluation is infrastructure not afterthought Treat judges as first class code versioned tested reviewed Golden dataset your most valuable IP curate it religiously Every prompt change eval run enforced by CI Regression alerts paging alerts not email digests Your users don t care about your prompt engineering cleverness They care that answer is right Automated evaluation how you guarantee at scale Code github com yourname llm eval harness Discussion Hacker News Follow yourname