Prompt injection

Prompt injection is a class of attack against language-model systems in which an adversary embeds instructions in content the model processes, causing it to disregard its intended instructions and act on the attacker’s instead. Because a language model treats the text it is given as a single stream, it does not inherently distinguish trusted instructions from untrusted data — so instructions hidden inside a document, web page, email or other input can be interpreted as commands.

The risk is greatest for systems that read external or user-supplied content, and especially for those with tools or actions available to them. Indirect prompt injection — where the malicious instruction is planted in a document or web page the model later retrieves, rather than typed by the user — is particularly dangerous, because a benign user can trigger the attack simply by asking the system to summarise or act on poisoned content. Consequences range from leaking data the model can access to misusing connected tools.

There is no single complete defence; prompt injection is mitigated in layers. Practical measures include constraining what a model is permitted to do, keeping least-privilege boundaries around any tools and data it can reach, separating and treating retrieved content as untrusted, validating outputs before they trigger consequential actions, and keeping a human in the loop for high-impact decisions. Reducing the blast radius — what an injected instruction could actually cause — matters as much as trying to detect the injection itself.