The Bug That Taught Me How to Run One Process Per User

“The bug that taught me how to run one process per user”

Giulio Ritfeld built a platform where each user gets a long-lived background process.
A single manager restart lost its internal dictionary of running workers – making every worker appear stopped while they were still alive.

Why This Matters

In theory, process managers should survive restarts cleanly; in practice, naive dependency on in-memory state leads to orphaned duplicates and unresolvable zombies.
This overlooked plumbing cost developers hours debugging issues that are invisible until deployment day – exactly when reliability matters most.

Key Insights

• **Fact:** After deploying an update, every worker appeared stopped because manager memory resets – despite workers running normally (Giulio Ritfeld’s experience).
  **Concept:** Persistent registry using PID + start time prevents mistaken adoption of recycled PIDs.
  **Tool:** WorkerDeck library implements this check along with graceful shutdown via SIGTERM→SIGKILL.

• **Fact:** Clicking “start” on an apparently-stopped worker spawned a second copy fighting for same port (“address already in use”).
  **Concept:** Zombie processes linger when original parent disappears; naive liveness checks see defunct entries as alive.
  **Tool:** Reading Linux /proc/[pid]/status state field distinguishes zombies from running processes.

• **Fact:** “Stop” button did nothing because new manager had no record of old workers (empty dictionary lookup).
  **Concept:** Process group signalling ensures child processes also receive stop signals instead of becoming orphans.
  **Tool:** Exponential backoff + circuit breaker prevents hot-loops on crash-start cycles (WorkerDeck feature).

• **Fact:** Workers keep running across manager restart because they are separate OS processes – only tracking data was lost.
  **Concept:** Re‑adoption by recorded PID plus start time gives near-certain identity verification even after reboot.​
     ​
    ​​**Tool**: Procinfo libraries (e.g., Python `psutil`) provide start time comparison across platforms.​​
     ​​​’s identity verification even after reboot.

•   		**None.**

Practical Applications

•   **Use case**: Platform managing per-user agents/pipelines – WorkerDeck adopts orphaned workers after manager restart via PID+start time check.

• Pitfall: Storing handles only in memory (dict) loses all state on deploy – leads silent duplicates until ports conflict.

•   **Use case**: Stopping a specific user’s worker – proper signalling uses SIGTERM then SIGKILL for whole process group.

• Pitfall: Ignoring zombie detection causes “stop” commands hang indefinitely waiting for dead child.

•   **Use case**: Self-healing crashed workers – exponential backoff + circuit breaker prevents infinite restart loops.

• Pitfall: Allowing immediate retry without limit can thrash system resources during persistent failures.

References:

• From internal analysis