Validating AI for CAPA Triage: A Pragmatic Pattern Under 21 CFR 820 and ISO 13485

Letting AI touch CAPA: the validation question nobody actually answers

James Whitfield prototyped an AI assistant to triage CAPA backlogs by reading nonconformance reports and suggesting root-cause categories. A shadow-mode pipeline logged AI suggestions against human outcomes to validate the probabilistic assistant.

Why This Matters

Medical device companies face mounting CAPA backlogs that bury QA teams, but regulators require documented, verifiable corrective/preventive processes under 21 CFR 820 and ISO 13485. Probabilistic AI models challenge the deterministic validation paradigm, yet a structured approach—using shadow-mode pipelines, golden-test corpora, and human-in-the-loop controls—can produce auditor-acceptable evidence without halting innovation.

Key Insights

• Traditional software validation expects deterministic input-output pairs, but LLM outputs are probabilistic and update-sensitive, requiring a ‘controlled assistance’ model, according to James Whitfield’s 2026 prototype.
• A golden-test corpus of de-identified historical CAPA cases, with human-accepted root causes and references, enables measurable acceptance criteria like top-1 root-cause agreement and hallucination rate per 100 suggestions.
• Continuous monitoring requires cataloging model versions, prompt templates, and embedding index snapshots, with periodic re-tests against the golden corpus and live-decision audits to trigger change control if performance drops.
• Human-in-the-loop controls are essential: AI drafts are saved with provenance (model version, timestamp, prompt) and require a named reviewer to accept or edit before any action is taken.
• Treat prompts, test suites, and evaluation code as source-controlled artifacts, running CI jobs to test against the golden corpus when prompts or models change, Whitfield reported.

Practical Applications

• Administrative triage – Use the AI assistant to categorize incoming nonconformances and surface related technical file documents, reducing QA team screening time.
• Manufacturing hold decisions – Never allow the AI to directly change product disposition without a human sign-off, as probabilistic outputs could lead to incorrect decisions.

References:

• https://dev.to/jwithfield_qa/letting-ai-touch-capa-the-validation-question-nobody-actually-answers-l0k