Beyond SQL Injection: The Critical Risk of Writable System Prompts in LLM Apps

The McKinsey AI Breach Isn’t About SQL Injection. It’s About Writable System Prompts.

Red-team security startup CodeWall gained read-write access to McKinsey’s Lilli AI platform in two hours. The researchers accessed tens of millions of messages and successfully modified system prompts via a single SQL UPDATE statement.

Why This Matters

In traditional software, behavior is defined in code and governed by versioned deployment pipelines, whereas LLM applications often treat prompts as dynamic database configurations. This architectural pattern creates a critical vulnerability where a data-layer breach results in a complete takeover of the application’s behavioral control plane. Because the model still produces plausible text, these subtle shifts in safety or confidentiality policies are significantly harder to detect than traditional system failures, allowing for persistent and scalable manipulation of the entire user base.

Key Insights

• Fact: CodeWall researchers accessed tens of millions of internal messages from McKinsey’s Lilli platform in a 2026 red-team engagement.
• Concept: Prompt tampering vs. leakage; tampering allows persistent behavioral control by modifying the instructions that steer model policies and responses.
• Tool: Aguardic is a policy-as-code platform used to enforce organizational rules across AI outputs, code, and documents when prompts fail.
• Fact: The vulnerability allowed researchers to change application behavior without a code deployment or deployment pipeline review.
• Concept: Control plane protection; LLM security requires securing the artifacts that define behavior, including prompts, tool configurations, and retrieval settings.

Practical Applications

• Use Case: Implementing immutable production prompts where the application runtime has read-only access to prevent database-driven prompt modification.
• Pitfall: Managing system prompts via unprotected Admin UIs or dynamic database fields, which bypasses the rigor of version control and code review.
• Use Case: Deploying output evaluation layers to detect sensitive data exposure as a defense-in-depth measure against compromised system instructions.
• Pitfall: Treating prompts as configuration rather than production code, leading to unauthorized behavioral drift that is difficult to monitor.

References:

• https://dev.to/aguardic/the-mckinsey-ai-breach-isnt-about-sql-injection-its-about-writable-system-prompts-lb4