Automated Future: Scaling Test Results Beyond Ephemeral CI Logs

A better home for your automated test results

Developer Steve Pryde transitioned a SaaS startup from zero to a platform running 30,000 tests per month. This system integrated JIRA and auto-scaling runners to treat test results as persistent data rather than transient logs.

Why This Matters

Standard CI pipelines treat test results as secondary artifacts, leading to expired logs and scattered data that lack searchable history or trend analysis. In a high-scale environment, the inability to distinguish between known failures and new regressions creates significant diagnostic overhead and technical debt.

Key Insights

• Automated Future decouples test data from build processes, establishing a dedicated repository for long-term health metrics (2026).
• Visual regression tools can achieve high ROI with minimal complexity through single-endpoint screenshot comparison and reference image scoring (SaaS Startup Case).
• JIRA integration reduces redundant work by mapping recurring failure modes to existing tickets or pre-populating new ticket flows with error context.
• The Automated Future CLI uses a prefix-based execution model to auto-discover JUnit XML files without requiring changes to existing test logic.

Working Examples

The af run command wraps existing test execution to parse and upload JUnit XML results to the platform.

af run -- npm test

Practical Applications

• High-volume engineering teams can use JQL-style syntax like ‘status = Failed AND duration > 5000’ to identify performance bottlenecks. Pitfall: Relying on terminal output prevents historical trend analysis and pattern recognition.
• UI development teams can implement screenshot comparison with pan-and-zoom UIs for image-based regression testing. Pitfall: Storing visual artifacts in CI logs makes it difficult to manage approvals or leave granular feedback.

References:

• https://dev.to/stevepryde/a-better-home-for-your-automated-test-results-3951
• https://dashboard.automatedfuture.co