Prompt Injection Defense Before an Agent Acts

An agent that reads a document and then sends, deletes, or exports something isn't just answering a question. It's exercising authority, often on material written by someone outside your organisation. Prompt injection defense therefore isn't primarily about teaching a model to spot a suspicious sentence. It's about deciding which actions remain reachable if that sentence works, who can authorise them, and what evidence shows why an action was allowed or stopped. In our analysis, that's the accountable question for a CISO, counsel, or board member: not whether a filter claims to block attacks, but whether the surrounding workflow can contain a successful one.

How Often Does Prompt Injection Still Succeed?

It succeeds often enough in controlled evaluations that you shouldn't build a workflow on the assumption it'll be caught. In 2025, the NIST technical blog on strengthening AI agent hijacking evaluations reported that the average attack success rate across five indirect prompt-injection tasks was 57%, rising to 80% after repeated attempts. That second figure matters operationally. An attacker doesn't need the first phrasing to work if the system allows retries, varied wording, or repeated exposure to similar content.

Published prompt-injection attack success

• InjecAgent, ReAct-style GPT-4 — Year: 2024; What was measured: Malicious instructions in external content the agent was expected to read; Reported success: 24 percent; 47 percent with a reinforcing hacking prompt
• NIST indirect-injection evaluation — Year: 2025; What was measured: Average across five indirect prompt-injection tasks; Reported success: 57 percent; 80 percent after repeated attempts
• NAACL Findings, eight defenses — Year: 2025; What was measured: Adaptive attacks against indirect prompt injection; Reported success: Consistently above 50 percent; all eight defenses bypassed
• Agentic AI coding editors — Year: 2025; What was measured: Prompt injection against coding editors; one command-execution scenario; Reported success: 66.9 to 84.1 percent; 89 percent of 314 payloads

The InjecAgent study evaluated 30 LLM agents and found that a ReAct-style GPT-4 agent followed an injected instruction in 24% of attack cases, rising to 47% with a separate reinforcing hacking prompt. The 2025 study of agentic AI coding editors measured attack success rates between 66.9% and 84.1%, and in one scenario 89% of 314 attack payloads triggered command execution.

These populations, models, and tasks differ, so they mustn't be blended into one organisational rate. Each denominator is the study's own set of cases, not your traffic, your documents, or your agents. A vendor who averages them into a single headline percentage is giving you a number without a defensible scope.

Residual risk isn't the attack percentage. It's the set of consequential actions still reachable after your controls are in place.

That definition changes the procurement conversation. Instead of asking how often a model refuses a malicious instruction, ask what happens on the attempt that gets through. Can it reach a mailbox, a file export, a permission change, or only a draft that a person reviews? The answer depends on controls that sit around the model, not inside its prompt.

Which Controls Belong Outside the Model?

The controls that decide permissions, destinations, and irreversible actions belong outside the model, in deterministic application code. The OWASP guidance on prompt injection recommends least privilege, human-in-the-loop controls, trust boundaries, and output validation, treats the model as an untrusted user, and states that no foolproof prevention exists inside the model itself. That framing is the right starting point for accountable design.

The pipeline below shows where those controls sit relative to the model.

A seven-step control pipeline

1. Mark everything inbound as untrusted data. That includes the user prompt, retrieved documents, web pages, emails, tool outputs, and earlier conversation turns. None of it inherits authority because it arrived in the same session.

1. Separate instructions from content structurally. Keep system and developer instructions in typed fields that external content can't write to. Wrap supplied material in explicit delimiters so the workflow, not the prose, carries the distinction.

1. Normalise before you filter. Standardise case, Unicode, confusables, invisible characters, whitespace, and encodings, then cap input length. Matching against raw text lets trivial obfuscation walk past the filter.

1. Grant only the tools the task needs. Check destination, permission, and data scope in application code for every proposed call. In this model, least-privilege runtime control for AI agents becomes an authorisation decision rather than a prompt instruction.

1. Hold consequential actions for a person. Sending, deletion, permission changes, and confidential exports wait for human approval of the exact action proposed.

1. Validate arguments and output. Check tool arguments against a schema, scan for sensitive values, and test whether each claim in a draft stays inside the supplied context. Log the complete decision trail.

1. Stop when evidence is incomplete. If independent reviewers disagree or the basis for an action can't be identified, the gate doesn't release the action.

These steps align with broader secure-by-design patterns for AI agents from Cyndra, which are useful context for teams structuring agent permissions from the start rather than retrofitting them.

The second-model trap

Asking a second model to approve an action doesn't create an authorisation boundary if the same unrestricted tool path remains open. You've added review, but the final decision still rests on model compliance. Independent reviewers can challenge a proposed action and surface disagreement, which is valuable evidence. A non-model policy engine should still be the final gate.

Say a matter file contains a hidden line asking the agent to email the file to an external address. The draft summary may look entirely helpful. The defense doesn't depend on the model noticing the line, because the send tool checks the destination against an approved list in code, finds no match, and refuses the call. The attack can influence the text without gaining the capability.

How Should Untrusted Content Stay Data?

Untrusted content stays data when the system separates what the model may read from what it's allowed to do. Screening and labels reduce obvious instruction smuggling, but the key boundary is the separation of data from permissions. The OWASP LLM Prompt Injection Prevention Cheat Sheet identifies retrieved documents, RAG context, tool outputs, web pages, email bodies, and conversation history as channels that may contain instructions, and recommends screening both the user prompt and fetched context before the primary model processes them.

This diagram shows that separation between the data tier and the permissions tier.

Two worked cases

In a legal intake workflow, an incoming email body is normalised first, covering case, Unicode confusables, invisible characters, whitespace, and encodings, and then length-capped before any pattern matching runs. If a line reads like an instruction to the agent, the workflow doesn't delete it. It keeps the line in an evidence log with its original span, so a reviewer can later see exactly what arrived and what the system did about it.

In a retrieved medical PDF, the relevant passage is wrapped in delimiter spotlighting that marks it as data the model may quote, summarise, or compare against other sources. It isn't a new task definition. If the passage contains wording that resembles a directive, the model can report that wording as content. It can't promote that wording into a tool call.

These patterns connect directly to the risks described in our analysis of hidden AI instructions in external documents and retrieval, where the carrier looks like ordinary working material.

The following walkthrough illustrates how instruction-like content can hide inside routine material an agent is asked to process.

Test strings worth building

Describe test cases by structure rather than circulating reusable payloads. Three patterns cover common smuggling routes:

• Buried instruction: an instruction placed after a long, benign summary in a supplied document.
• Obfuscated instruction: the same instruction with unusual Unicode spacing or confusable characters.
• Tool-result escalation: a tool response that asks the agent to widen its own file access.

One gap remains poorly solved. Implicit passages and image content can shift model behaviour without containing any explicit instruction, and detectors perform poorly on them because there's no directive sentence to match. Treat that influence as unclassified. The gate fails closed rather than guessing.

A label doesn't enforce a trust boundary. Code that checks permissions before a tool runs is what enforces it.

As build tickets, the requirement is concrete: typed untrusted fields for every external source, normalisation before matching, and no tool call whose arguments came only from unmarked external text.

When Should the Workflow Fail Closed?

The workflow should fail closed when a proposed action is irreversible, privileged, or influenced by content whose effect can't be identified. A suspected injection is decided by consequence, reversibility, data sensitivity, and whether the influencing span can be named, not by a single refusal score. The 2025 NAACL Findings study tested eight defenses against indirect prompt injection and bypassed all of them with adaptive attacks, achieving attack success rates consistently above 50%. A passed classifier therefore isn't permission to continue.

This comparison sets out the four factors that drive the decision.

Stop, escalate or allow after suspected injection

• Read inside an already authorised file — Sensitivity and reversibility: Reversible, no new destination; Decision: Allow if logged and no secret leaves scope; What to record: Action, file scope, timestamp
• Draft that only quotes supplied text — Sensitivity and reversibility: No tool call; Decision: Escalate for human review; What to record: Quoted span and open questions
• Send, delete, permission change or confidential export — Sensitivity and reversibility: Irreversible or privileged; Decision: Stop until a person approves the exact arguments; What to record: Arguments presented for approval
• Reviewers disagree or the influencing span cannot be named — Sensitivity and reversibility: Uncertainty, including implicit or image influence; Decision: Fail closed; What to record: Disagreement and missing evidence

A reversible read inside an already authorised file can proceed with logging, provided no new destination and no secret are involved. A draft that quotes untrusted text but proposes no tool call goes to a person, because quotation can still carry a corrupted fact into a filing, a clinical note, or a board paper.

Sending, deleting, changing permissions, exporting confidential material, or calling a new tool stops until a person approves the exact arguments, not a general description of intent. Approval of send this summary isn't approval of a recipient list the reviewer never saw.

If independent reviewers disagree, the source span can't be identified, or an image or implicit passage may be steering the action, the gate fails closed and records why. Disagreement is evidence, not noise to be averaged away.

A low residual rate can still be unacceptable in legal, clinical, compliance, or political work. One unauthorised send or one corrupted fact in a relied-upon document matters even when the attacker's full goal fails. Partial compromise isn't a near miss if the partial result enters a record someone acts on.

For audit purposes, every gated decision should leave five fields: the proposed action, the tool arguments, the content span relied on, the reviewer outcome, and whether the gate stopped or released the action. Without those fields, you can't reconstruct accountability after an incident. You can only assert that a control existed.

How Do You Test Without Only Counting Refusals?

You test by measuring attack success and task utility together, on a repeatable corpus, rather than counting how often the model says no. The evaluation approach described in the multi-layer prompt-injection defense benchmark builds cases across direct instruction override, malicious retrieved content, context manipulation, data exfiltration, and cross-context contamination, then adds controls progressively and records both security and task-preservation outcomes. Report results from your own corpus runs, not headline figures from someone else's benchmark.

Build the corpus in five families

Within each family, include obfuscation such as Unicode confusables, invisible characters, encoded candidates, and instruction-like text inside ordinary documents. Add benign but influential passages as well. Without them, overblocking stays invisible until a legitimate user hits it in production.

Run the same corpus against a baseline configuration first. Then add one layer at a time: normalisation and length bounds, content filtering, hierarchical instruction handling, least-privilege tools, and independent response verification. Adding everything at once tells you the combined result but not which layer earned it, which matters when a later change removes the effective one without notice. This layered method also supports structured red teaming of AI agent prompt-injection chains across real tool paths.

Test case families and release metrics

• Direct instruction override — Test case to build: User turn asks the agent to ignore its brief and reveal restricted content; Metric to record: Attack-success rate
• Malicious retrieved content — Test case to build: Instruction placed after a long benign summary in a supplied PDF; Metric to record: Attack-success rate and unauthorised tool calls
• Context manipulation — Test case to build: Earlier turn that redefines the scope of the task; Metric to record: Benign-task completion
• Data exfiltration — Test case to build: Retrieved text asks for a file to be sent to an external address; Metric to record: Secret exposure and unauthorised tool calls
• Cross-context contamination — Test case to build: Fact from one matter file introduced into another matter; Metric to record: Reviewer disagreement and false-positive rate

Record seven metrics for every run: attack-success rate, false-positive rate, benign-task completion, unauthorised tool calls, secret exposure, abstention, and reviewer disagreement. Each one catches a failure the others miss. A configuration can post a strong refusal rate while still leaking through a tool call it never refused, because nobody asked it a question it recognised as an attack.

Simple phrase filters aren't sufficient on their own, since normalisation choices determine what the filter can even see. But the opposite failure is just as real. Maximising refusals can destroy usefulness in legal, medical, or investigative work, where legitimate documents routinely contain imperative language, quoted threats, or descriptions of wrongdoing.

A release that reports only blocked attacks hasn't demonstrated safety. It may have demonstrated that the system blocks work.

Close each release cycle with a short checklist: security and utility numbers side by side, the corpus version used, and the specific layer added since the last run. If utility fell while attack success barely moved, that trade-off should be visible to the person accountable for accepting it, in writing, before deployment.

What Should You Require Before an Agent Acts?

Require four things in writing before an agent acts on untrusted content in a professional workflow. First, untrusted content kept structurally separate from instructions, so supplied material can't redefine the task. Second, tool permissions enforced in application code, so a manipulated model output can't widen its own authority. Third, a human gate for irreversible or privileged actions, with approval given on exact arguments. Fourth, a release record showing attack success beside task completion, false positives, and abstention.

The evidence question

For every disputed action, ask which document span, tool result, or image region influenced it, and what happens when that influence can't be classified. If the answer is that the workflow proceeds anyway, the control environment has a named gap. The defensible answer is to fail closed and leave the final judgement with the user.

A verification layer can sit beside the gate rather than replace it. Verification supports review and control. It doesn't ensure correctness, compliance, or that an injection will be caught.

The same limit applies to privacy controls. It doesn't detect every indirect identifier and can't make re-identification impossible. Stating that limit plainly is part of a defensible position, not a concession.

Those measures protect the workflow around the model. They don't turn any single model output into a trusted instruction.

More articles on this subject are collected in the Agentic AI and AI agents overview.

Sources: Attack success figures are attributed to NIST's 2025 agent hijacking blog, the InjecAgent benchmark, a 2025 study of agentic coding editors, and NAACL Findings; control guidance is attributed to OWASP's prompt injection guidance and Cyndra's operator guide.