The LLM Is an ALU

The LLM Is an ALU

While explaining why his own agent was wasteful, Matt watched it burn four consecutive tool calls just to store one fact—a robo message followed by three calls and a summary before anything hit storage.

Why This Matters

Treating an LLM as a slow but fixable processor misses its fundamental nature—it’s stateless and non‑sequential by design. No amount of parameter scaling adds a program counter or persistent memory; every forward pass starts from scratch with whatever input you provide today.

Key Insights

• A CPU has registers and a fetch‑decode‑execute cycle; an LLM has neither—it transforms input once per call with no internal loop.
• The robo‑message hack injected fake user messages into conversation history because APIs originally lacked tool injection—analogous to self‑modifying trampolines in ROM‑bound computers.
• matbot’s contextual_search tool works like demand paging in MMUs—skills are fetched only when requested via page fault rather than loaded upfront into precious context window.
• The ARM7 full context switch took six instructions microsecond speed (Pogo phone kernel circa 2001) proving cheap save/restore enables reliable sequencing around any compute core.

Practical Applications

• [Use case] Agent skill “Remember This” performed repeated model calls when harness could have written directly – [Pitfall] Relying on expensive model loops for purely mechanical operations wastes tokens and latency.
• [Use case] Loading every skill into system prompt at conversation start – [Pitfall] Bank‑switching without eviction permanently occupies limited context window leading to bloat.
• [Use case] Compiling procedural markdown skills into TypeScript plugins – [Pitfall] Overcompiling narrative or reference tables where the model’s generation value outweighs any efficiency gain.

References:

• https://dev.to/matatbread/the-llm-is-an-alu-12c8