Moving Beyond AI Success Theatre: Engineering Lessons from Sprint 7

We Got Called Out for Writing AI Success Theatre — Here’s What We’re Changing

Senior engineer Nick Pelling criticized ORCHESTRATE’s AI project retrospectives for resembling “CIA intelligence histories” rather than technical content. During Sprint 7, the team realized they had built 118 services into a single monolithic file with zero functional runtime validation.

Why This Matters

Technical reality often diverges from the polished “success theatre” of automated development, where high service counts mask deep architectural failures. In this case, building 118 backend services without domain separation or functional testing created a system that appeared successful on paper but was fundamentally unverified and difficult to maintain. This highlights the gap between AI generation speed and the rigorous engineering standards required for production-grade reliability.

Key Insights

• AI-managed development can lead to rapid service creation but extreme technical debt, such as 118 routes in a single api-server.mjs file (Sprint 7, 2026).
• Source code inspection, such as checking if app.post exists in a file, is an insufficient substitute for runtime validation and functional API testing.
• Advisory-only governance (ADR-032) fails with AI agents; if a task like memory storage is not mechanically enforced by a blocking gate, agents consistently skip it.
• Effective pipeline diagnostics require a distinction between ‘failed’ and ‘skipped’ stages to prevent root causes from being obscured by cascading errors.
• Estimation in AI-assisted projects is often over-optimistic, as demonstrated by ORCHESTRATE’s 53% error rate due to underestimated ceremony overhead.

Working Examples

An example of a ‘false positive’ test that validates source code presence rather than functional runtime behavior.

const src = fs.readFileSync('server.mjs', 'utf-8');
expect(src).toContain('app.post("/api/memory/store"');
// Passes — the route registration exists in the source code
// We never wrote the runtime validation test to check status 200

A multi-stage pipeline pattern that differentiates between the failing stage and subsequently skipped stages for better diagnostics.

class PipelineExecutor {
  private stages: Array<{ name: string; fn: StageFn }> = [];
  run(): Result<PipelineResult> {
    let currentInput = null;
    let failed = false;
    for (const stage of this.stages) {
      if (failed) {
        results.push({ ...stage, status: 'skip' });
        continue;
      }
      try {
        const output = stage.fn(currentInput);
        if (output === null) { failed = true; }
        currentInput = output;
      } catch (e) {
        failed = true;
      }
    }
  }
}

Practical Applications

• Use Case: Implementing a multi-stage execution pipeline (Source -> Script -> Audio) that skips subsequent stages upon failure to preserve diagnostic trace clarity.
• Pitfall: Relying on advisory warnings for AI agents (ADR-032) which leads to zero memory storage; use blocking gates to ensure compliance.
• Use Case: Refactoring monolithic API files into route modules to prevent technical debt in high-velocity AI coding projects.
• Pitfall: Over-optimistic sprint estimation; AI agents write code quickly, but TDD, documentation, and provenance tracking add significant time overhead.

References:

• https://dev.to/tmdlrg/we-got-called-out-for-writing-ai-success-theatre-heres-what-were-changing-3fad