Why Implicit Glue Code Fails: Moving Toward Explicit Workflow State Machines

The Workflow Problem That Made Me Stop Trusting Glue Code

Engineer Brock Claussen observed a webhook handler charge a customer twice in one minute due to uncoordinated success and retry paths. This incident highlights how informal workflow models fail when systems scale beyond simple application code.

Why This Matters

In technical reality, workflows often grow into complex state machines scattered across database columns and queue messages rather than being explicitly defined. This fragmentation leads to expensive production incidents because the runtime cannot distinguish between repeatable transforms and non-idempotent side effects like charging a card or sending notifications.

Key Insights

• A 2026 production incident demonstrated that success and retry paths written six months apart can cause double-billing without explicit state management.
• The concept of ‘Side-Effect Boundaries’ is critical to distinguish between safe operations like JSON parsing and committed actions like API calls.
• Implicit state machines often live in status columns and conditionals, leading to terminal state errors where ‘charged’ status is not properly enforced.
• Durable-execution systems provide a platform for orchestration, but the author argues the workflow contract must be the source of truth over the runtime.

Practical Applications

• Use case: Webhook handlers enriching data with customer records and calling external APIs. Pitfall: Treating workflows as implicit glue code, leading to state and boundary bugs during retries.
• Use case: Distributed systems requiring approval steps and cancellation logic. Pitfall: Defining states via convention rather than explicit transitions, resulting in illegal states that surface during 2 AM outages.

References:

• https://dev.to/dieshen/the-workflow-problem-that-made-me-stop-trusting-glue-code-bla