Engineering Autonomous AI Pipelines: A Guide to Cron-Scheduled Agents

How to Schedule AI Agent Tasks with Cron (The Missing Guide)

Nathaniel Hamlett operates 23 autonomous cron jobs that discover, research, and apply to opportunities without human intervention. This system relies on external state rather than internal session memory to maintain coordination across isolated tasks.

Why This Matters

AI frameworks typically ignore the complexities of scheduling, leading to session isolation issues where agents lack context of previous runs. By externalizing state into a database and implementing lock files, developers can prevent process collisions and ensure 24/7 reliability in complex multi-stage pipelines.

Key Insights

• 23 autonomous cron jobs manage a full job pipeline including discovery, research, and submission (Hamlett, 2026).
• Session isolation through external databases prevents context loss and coordination failures in fresh cron starts.
• SQLite with WAL mode and busy_timeout serves as the coordination layer for concurrent agent sessions.
• File-based locks in /tmp/agent-locks prevent race conditions between research and packet-building tasks.
• Logrotate with copytruncate allows continuous logging without requiring process restarts for autonomous agents.

Working Examples

Initializing external state with SQLite to manage session context.

import sqlite3\ndb = sqlite3.connect('/path/to/agent.db')\ndb.execute('PRAGMA journal_mode=WAL')\ndb.execute('PRAGMA busy_timeout=5000')\nitems = db.execute('SELECT * FROM opportunities WHERE stage = "discovered" AND fit_score >= 7.0 ORDER BY fit_score DESC LIMIT 5').fetchall()

File-based lock mechanism to prevent concurrent execution of related tasks.

def acquire_lock(lock_type: str) -> bool:\n    lock_file = os.path.join(LOCK_DIR, f'{lock_type}.lock')\n    if os.path.exists(lock_file):\n        try:\n            with open(lock_file) as f:\n                pid = int(f.read().strip())\n            os.kill(pid, 0)\n            return False\n        except (ProcessError, ValueError, FileNotFoundError):\n            pass\n    with open(lock_file, 'w') as f:\n        f.write(str(os.getpid()))\n    return True

Pipeline schedule ensuring tasks flow from discovery to submission.

0 7 * * * agent scan\n0 8 * * * agent research\n0 9 * * * agent packets\n0 11 * * * agent submit\n0 23 * * * agent review

Practical Applications

• System: Autonomous Job-Hunting Agent. Pitfall: Context pollution where an LLM with too many tools becomes confused and selects the wrong action.
• System: High-volume Data Discovery Pipeline. Pitfall: Batch processing without item-level commits results in total data loss upon API failure.

References:

• https://dev.to/nathanhamlett/how-to-schedule-ai-agent-tasks-with-cron-the-missing-guide-4kgn