LLM Red Teaming: Test Workflow and Scoring Guide

The most common advice on LLM red teaming is to collect jailbreak prompts and count how many the model refuses. That advice produces a score without evidence. A clean refusal tally does not show whether a legal intake assistant fabricates authority, whether a retrieved clause rewrites the system's limits, or whether a pass in English says anything about behaviour in another language. A defensible program treats red teaming as structured evidence of professional reliability rather than a single headline number. Findings become controls, regression tests and a written decision on residual risk, so the work can be reviewed, repeated and defended later. The goal is not to prove the model is safe once, but to show how each weakness was found, what was changed in response, and which risks you have knowingly chosen to accept.

What Should an LLM Red Team Test First?

An LLM red team starts with the use case, the sector, the users, the intended outcomes and the unacceptable impacts. It does not start with a prompt list. The owner names who will use the system, what a correct result looks like, and which harms are unacceptable in that setting. Testers then receive scenario instructions that name the violative information they should try to elicit. Those instructions become harm categories with explicit pass or fail criteria, so a later reader can see what was tested and what counted as a failure. Red teaming, in this sense, is structured probing for inaccurate, harmful or discriminatory behaviour in a controlled setting. This article is that controlled exercise, not a general model bake-off and not a live field trial.

Say a firm tests a legal-intake assistant. A direct request for prohibited advice may be refused, while the same assistant, given a retrieved file and a document tool, still drafts a filing the lawyer did not authorise. The model score and the deployed behaviour are different questions.

The OWASP GenAI Red Teaming Guide splits an engagement into four assessment areas, so accountability does not stop at the model.

• Model evaluation — What it examines: Model behaviour and safety boundaries; Example failure to look for: Harmful answer to a direct request that the model should refuse
• Implementation testing — What it examines: How the application configures and uses the model; Example failure to look for: Safety instruction overridden by the app wrapper
• Infrastructure assessment — What it examines: Surrounding systems and integrations; Example failure to look for: Over-permissive credentials reachable through a model-connected tool
• Runtime behaviour analysis — What it examines: Behaviour in live operation; Example failure to look for: Gradual escalation that appears only late in a long session

A customer instance, account or document is in scope for testing only where the tester owns it or has explicit written authorisation to test it. I would not treat a vendor demo tenant, a colleague's matter file, or a copied production prompt as fair game.

The scope record

Before the first run, write a scope record that names the owner of each assessment area, the languages in scope, and the user groups covered. Map each confirmed finding to MITRE ATLAS, so the owner sees the adversary technique, not only the prompt that failed. A model can pass isolated safety tests while the deployed application still fails through its retrieval layer, tool permissions, infrastructure or runtime controls.

How Do You Run an LLM Red-Team Cycle Step by Step?

Limit the first cycle to one use case and one pinned model. NIST's evaluation guidance separates model testing, red teaming and field testing, and treats red teaming as structured probing in a controlled setting. The cycle below is that controlled exercise, not a general sweep for clever jailbreaks.

1. Fix the scenario. Record the use case, sector, users, intended outcome, harm category, the violative information testers must try to elicit, and a fixed attack budget. Include the procedural failure that would make a wrong answer dangerous: a filing sent unchecked, a tool call with the wrong scope, a confidential value copied into a reply. The budget stops a long session from being scored as if it were a short one.
2. Pin the configuration. Store the model version, system prompt, tool permissions, temperature and the full conversation history. Another tester should be able to repeat the run without asking what was changed.
3. Split the seeds. Take seed attacks from the agreed suite. Keep a working set, and seal a holdout that prompt authors do not tune against.
4. Change one factor, then store the state. Mutate one factor at a time, execute the attempt, and keep the complete conversation with those settings. Mixed changes make a later fix impossible to attribute.
5. Label, then adjudicate. An evaluator labels the output. A human then adjudicates, recording partial disclosure, transformed detail and tool side effects separately from a clean refusal. Severity is labelled independently of the wording.
6. Break out the results. Publish results by harm category, attack family, model version and conversation length. Report attack-success rate, meaning successful unsafe outcomes divided by attempts, plus a severity-weighted unsafe-output rate, refusal quality and the false-positive refusal rate. One average hides the category that matters.
7. Regress, then rerun the holdout. Add confirmed failures to the regression suite and rerun the holdout after every model or prompt change.

Keep the holdout sealed and rerun it on the pinned configuration; that is what makes a later comparison meaningful. A similar rerun-after-change habit is described for a different control domain in Faberwork insights on expense tracking automation, but in LLM work the point is narrower: prove the fix did not reopen the same path.

Practical rule: A response that says it will not help, then still gives a usable detail, is not a pass.

A second slip is the judge. A judge model can share the target's blind spots, so human adjudication stays in the record for borderline and high-severity cases. Do not widen the first cycle to every product surface. One use case, fully evidenced, is more useful in a sign-off meeting than a thin pass across ten assistants.

Which Adversarial Prompts Belong in Your Test Suite?

Direct prompt injection is a user attempt to override instructions. Indirect prompt injection hides the instruction in a document, web page, email or tool result. The OWASP Top 10 for LLM Applications, 2025 draws that line and treats retrieved content as untrusted. Every retrieval path and every tool path needs indirect cases, including the document workflows discussed in our note on indirect prompt injection in LLM workflows.

Say counsel uploads a contract clause and asks for a summary. The visible request is harmless. The clause itself tells the model to drop confidentiality limits and to quote internal notes. A suite that only attacks the chat box will record a pass.

• Paraphrase — What it changes: Wording of the same request; What a pass can still miss: A refusal that holds for only one phrasing
• Role-play — What it changes: Persona or scenario wrapper; What a pass can still miss: Safety that depends on the user sounding malicious
• Encoding — What it changes: Obfuscated instructions; What a pass can still miss: Filters that read only plain text
• Multilingual variants — What it changes: Language or cultural framing; What a pass can still miss: An English-only safety pass
• Multi-turn escalation — What it changes: Gradual requests across a session; What a pass can still miss: A single-turn refusal benchmark
• Direct prompt injection — What it changes: A user attempt to override instructions; What a pass can still miss: Any attack hidden outside the visible prompt
• Indirect prompt injection — What it changes: Instructions in a document, page, email or tool result; What a pass can still miss: Tests that use only a harmless user prompt

Each row changes one factor. That is how a failure stays attributable.

Automation and attacker strength

A Crucible analysis covering 214,271 attack attempts, 1,674 users and 30 LLM challenges reported a 69.5% success rate for automated approaches versus 47.6% for manual techniques. Only 5.2% of users employed automation. Automation widens coverage. Human testers still build the creative chains that a mutator has not seen. The same analysis shows that success rates vary substantially across challenge types and methods, which is why one suite score can conceal a weak category.

A separate study of more than 600 attacker-target pairs reported a correlation above 0.88 between attacker success and general capability, a sharp drop when the target was more capable than the attacker, and a capability gap that can be modelled as a sigmoid, as set out in that attacker-target study. I would not rely on one attacker model or one fixed prompt set.

Keep the holdout unseen by the people who write prompts. Retire an attack family when new seeds stop producing new confirmed failures, and spend the next cycle on the family that still does.

Where Do Standard Benchmarks Miss Real Failures?

A clean English benchmark score is not evidence that the system is safe for the people who will use it.

Do-Not-Answer, HarmBench, AdvBench, HARMFULQA and RedBench are English-only. Passing them does not establish comparable safety in other languages or cultural settings. A Yoruba and Nigerian Pidgin case study reports that multilingual training alone does not produce safe behaviour. Some multilingual models produce more toxic outputs in low-resource languages, and safety norms may encode predominantly Western assumptions.

The same study reported a notable Yoruba safety discrepancy for Aya 101 and found that weak language comprehension partly explained apparent safety results. It identified Gemini 2.0 Flash as the most consistent model across languages. Weak comprehension recorded as a safe refusal is a scoring error, not a control.

What to report instead of one pass

Report comprehension, refusal, toxicity and helpfulness separately. The practical question is not only whether the model refuses a harmful prompt. It is whether the model recognises harm, ambiguity, protected-group context and culturally specific risk, without confusing poor comprehension with safe behaviour.

For a global professional system, that means language-specific reviewers and a harm taxonomy grounded in the communities served. Our discussion of benchmark validity in task-focused evaluation sets out why a benchmark that does not match the task cannot carry a release decision.

Before sign-off, the suite should show two gaps in writing.

• Language gap: every language in which affected users will read or file an output, not only the language of the benchmark.
• Failure gap: harms that are plausible and well written, including unsafe or misleading output in a language the English suite never touched.
• Comprehension gap: cases where the model did not understand the request, scored apart from cases where it understood and refused.

Legal, healthcare, journalism and public-sector workflows are where an English pass is most likely to mislead. A model can look safe in the benchmark language and still produce unsafe or misleading output in a language used by the people affected.

Which Scores Should You Report for Each Test?

Read attack-success rate with the other five measures from the same pinned run. A single rate cannot support a release decision, and every metric is broken out by harm category, attack family, model version and conversation length.

• Attack-success rate — How to calculate it: Successful unsafe outcomes divided by attempts; What it reveals: How often attacks get through in a scenario
• Severity-weighted unsafe-output rate — How to calculate it: Unsafe outputs weighted by independently labelled severity; What it reveals: Whether serious failures hide behind many minor ones
• Refusal quality — How to calculate it: Refusals that disclose no actionable detail, divided by all refusals; What it reveals: Refusals that still leak partial or transformed content
• False-positive refusal rate — How to calculate it: Benign prompts refused, divided by benign prompts tested; What it reveals: Over-blocking that harms legitimate users
• Discovery efficiency — How to calculate it: Unique confirmed vulnerabilities per tester-hour or per 1,000 calls; What it reveals: Cost of finding new failures
• Evaluator disagreement — How to calculate it: Share of outputs where evaluator and human labels differ; What it reveals: Reliability of the scoring method itself

Severity is labelled independently of the response, so a rare serious failure is not drowned by minor ones. Refusal quality fails when a nominal refusal still gives a partial or transformed detail. A plausible procedural harm counts as a failure even when the wording is not toxic.

What a toxicity score leaves out

Earlier research released 38,961 red-team attacks and found that reinforcement-learning-from-human-feedback models become increasingly difficult to red-team as they scale. That pattern, set out in the 2022 red-teaming study, can reflect stronger resistance to known attack styles rather than broad coverage. The same body of work notes that automated methods generate only a small number of effective cases, and that no participant in one study exceeded a 0.5 toxicity probability. Many evaluations also remain single-turn, while real use is conversational.

The failures that matter in professional work are often well written. Fabricated legal support, selective citation, omission of contraindications, a confident misreading of a confidential document, and injection that appears only late in a long session will not show up as overt toxicity. Coverage for those cases combines adversarial dialogue, source and claim checks, uncertainty calibration, document-level tests, and an independent review of whether the evidence supports the conclusion.

Evaluator disagreement belongs in the pack. A method that conflicts with the human label cannot support sign-off, a point developed in our note on LLM judges, bias and output variation.

For the meeting, bring three figures. The severity-weighted rate for the highest harm category. Refusal quality. And the categories that have no tests.

How Do Red-Team Findings Feed a Verification Chain?

A finding that is filed and not retested is not a control. Each confirmed failure produces three outputs: a regression test, a control change and a review step. The three stay linked, so a later model update cannot drop the failure unnoticed.

A confirmed injection through a retrieval layer becomes a holdout test, plus a rule that retrieved content is treated as untrusted. A fabricated citation becomes a check that every cited claim links to a source a reviewer can open. The control is specific to the failure. A generic "be careful" instruction is not a control change.

Separate reviewers, visible disagreement

Generation, factual audit, adversarial challenge and source verification are performed by separate reviewers, so no single model has the final word. Our chain assigns those roles to independent models. Claude produces the first answer, another model challenges the reasoning, and live source verification checks whether claims can be grounded. Disagreement stays visible in the final synthesis instead of being smoothed into one answer.

Verification across several models reduces the risk of a confident but wrong answer but does not eliminate it. Running several models is a check, not a proof. Judgement and the final decision remain with the user.

Record the trail as test, model, fix and re-test. A reviewer should be able to see which configuration failed, which change was made, and whether the holdout still fails.

• Rerun the holdout after each model update, including a change that looks unrelated to safety.
• Keep inspectable logs so a reviewer can follow the run without asking the team to reconstruct it.
• Leave unresolved disagreements in the synthesis, marked as unresolved, rather than averaging them away.

A control that cannot be retested is a note in a report. A control that fails the next holdout is still an open finding.

Who Should Sign Off Before a Model Goes Live?

Before a model goes live, a decision-maker should ask for five things, and should not accept a single pass rate in their place.

• Scope record: use case, languages and harm categories, with the owner of each assessment area.
• Per-category results: the core metrics, broken out rather than averaged.
• Untested surface: assessment areas and languages that were not tested.
• Open failures: severity, owner and deadline for each one still open.
• Re-test evidence: the holdout result after each fix, on the pinned configuration.

Ask who accepts residual risk in writing. That person should be able to see the untested languages and the open failures, not only the categories that passed. Customers must review output before relying on it, acting on it, filing it or sending it to a third party, so human review stays in the approval conditions. A growing company can use a practical split of ownership, as in governing AI with Rite NRG, so the person who accepts the risk is not the person who wrote the prompts.

Three decisions are available, and each one states what is still uncertain.

Approve fits when the in-scope categories have been tested, open failures are closed on re-test, and the remaining uncertainty is named, for example a language that is out of scope for this release. Approve with named conditions fits when a control is in place but a deadline, a language limit or a human review step is still attached. Hold fits when a high-severity failure is open, a required language was not tested, or the re-test was run on a different configuration from the one that failed.

I would run one complete cycle on a single high-impact use case and bring that pack to the meeting. A thin pass across every assistant is a weaker basis for a signature.

Approve, approve with named conditions, or hold, and write the remaining uncertainty beside the decision.

It routes an answer through separate factual, adversarial and source checks, and it keeps disagreements visible, which is the same discipline a red-team finding needs before anyone relies on the output.

More articles on this subject are collected in the AI security overview.

Sources: Assessment areas and application-layer testing draw on the OWASP GenAI Red Teaming and OWASP Top 10 for LLM Applications guidance; the separation of model testing, red teaming and field testing is attributed to NIST evaluation guidance, with a cross-domain rerun-after-change example from Faberwork.