Building Spectrion: A 57-Tool Autonomous AI Agent Architecture for iOS

I built an AI agent with 57 tools that actually does stuff on your iPhone

Denis Babkevich has developed Spectrion, an autonomous AI agent for iOS that utilizes 57 distinct tools to perform real-world tasks. The system shifts from simple chatbots to true agents by employing a persistent execution loop that handles up to 200 iterations per task.

Why This Matters

Most mobile AI applications are limited to text-box wrappers that cannot interact with the device’s native capabilities. Spectrion solves the problem of model ‘stalling’ during multi-step tasks by implementing a todo system and a hierarchy of nudges that force the LLM to complete pending items. This architecture addresses technical friction in agentic workflows by providing self-healing mechanisms like context overflow recovery and repetition detection, ensuring reliability in complex, real-world environments.

Key Insights

• The core agent loop utilizes a generate-execute-append cycle, allowing the model to see tool results and decide subsequent actions without user intervention.
• A hierarchical nudge system prevents failures by detecting ‘action-promises’ where the model claims it will act but fails to call a tool.
• Spectrion’s Device Mesh uses Curve25519 ECDH and AES-256-GCM for encrypted cross-device tool execution, such as a Mac agent calling a remote iPhone camera.
• The Evolution Engine performs autonomous self-improvement every 24 hours by analyzing tool success rates and auto-creating new JavaScript tools via an LLM.
• Parallel tool execution via Swift TaskGroups reduces latency by 2-3x for multi-tool turns, such as simultaneous web searches and calendar checks.

Working Examples

The basic agentic loop that powers tool selection and execution.

messages = [userMessage]
loop {
  response = llm.generate(messages)
  if response.hasToolCalls {
    results = execute(response.toolCalls)
    messages.append(response)
    messages.append(results)
    continue
  } else {
    display(response.text)
    break
  }
}

Example of a custom tool created by the agent at runtime in the JavaScript sandbox.

const response = http({
  url: `https://api.github.com/repos/${owner}/${repo}`,
  method: 'GET',
  headers: { 'Accept': 'application/vnd.github.v3+json' }
});
const data = JSON.parse(response.body);
return `${data.full_name}: ${data.stargazers_count} stars`;

Practical Applications

• Use Case: Executing compound requests like ‘research competitors and build a tracker’ using a todo system to manage subtasks. Pitfall: Model outputting repeated text or hanging, mitigated by a 600-second watchdog and repetition detector.
• Use Case: Cross-device workflow where an agent on a Mac reads files from an iPhone’s filesystem via an encrypted WebSocket tunnel. Pitfall: Conflict-prone data merging, which is resolved using a Hybrid Logical Clock (HLC).
• Use Case: Deploying long-running agents to Linux servers as Docker containers via the ‘remote_cli’ tool. Pitfall: Burning premium tokens on simple tasks, solved by routing sub-agents to cheaper, specific model tiers.

References:

• https://dev.to/denisbabkevich/i-built-an-ai-agent-with-57-tools-that-actually-does-stuff-on-your-iphone-3345