Scaling Agent Workflows to Production: How thingd.cloud Handles Thousands of Concurrent AI Agents

The Production Challenge: Multi-Tenancy for Machines

Sayan Mohsin and the thingd.cloud team are moving agent-native infrastructure from local-first Rust prototypes to a globally scalable cloud platform. The core challenge is that AI agents cannot tolerate even a single second of latency without stalling execution loops and inflating inference costs.

Why This Matters

Key Insights

• Multi-tenancy for machines: Unlike human SaaS users who tolerate ~1.5s load times, AI agents have zero patience; any latency spike stalls execution loops, causing context window timeouts and skyrocketing inference costs.
• Absolute tenant isolation: Agent A’s memory structures must never bleed into Agent B’s data assets—a hard architectural requirement for the production cloud layer.
• High-concurrency orchestration: The system must handle thousands of active tool invocations per second without creating database deadlocks, requiring a completely different playbook from local-first workflows.

Practical Applications

• • Use case: Agent-native SaaS providers relying on low-latency data engines. Pitfall: Using human-tolerant cloud APIs (e.g., cold starts >500ms) can cause agent execution stalls and increased inference costs.
• • Use case: Platforms requiring strict data isolation between autonomous agents. Pitfall: Shared database pools without tenant-aware connection management lead to cross-tenant data bleed or deadlocks at scale.

References:

• From internal analysis