AI agent security testing is a systems pentest of the model, its tools, memory, execution identity and control loop. The finding is attacker-controlled content crossing a trust boundary into an over-privileged tool and producing a verifiable impact. A jailbreak screenshot is not a finding.
The short answer
- AI agent security testing is a systems pentest of the model, tools, memory, control loop and execution identity, not a jailbreak exercise.
- Run seven phases in order: scope, recon, identity mapping, threat modeling, adversarial testing, impact chaining, reporting.
- A finding needs a verified sink (exfiltration, an unauthorized action, a cross-tenant read or cost) plus trial counts and a confidence interval.
- The fixes that hold are architectural: remove a lethal trifecta leg, enforce authorization in code, and allowlist egress.
Key takeaways
- AI agent security testing targets the deployed system: model, tools, memory, control loop and execution identity, not the model alone.
- Identity mapping is the highest-yield recon step; a shared service account behind a per-user agent is a severe finding that needs no prompt injection.
- Prompt injection is a delivery primitive; the finding is a verified sink such as exfiltration, an unauthorized action, a cross-tenant read or cost amplification.
- Report trial counts, success definitions and confidence intervals; 0 of 30 is not proof that a control works.
- Architectural fixes that remove a lethal trifecta leg outrank classifiers and prompt instructions.
What is AI agent security testing, and how is it different from LLM red teaming?
AI agent security testing asks what an attacker can make a deployed agent do. LLM red teaming asks what they can make a model say. The two need different targets, success criteria and evidence.
An LLM agent has five parts: the model, a tool or function-calling surface, memory or state, a control loop that picks the next step, and an execution identity that tool calls run as. Probing only the model in a playground covers about one fifth of that surface. It also misses where impact happens: tool implementations, retrieval ACLs, credential scopes, the approval dialog and egress.
| Dimension | Agent security testing | LLM safety red teaming | Web/API pentest |
|---|---|---|---|
| Target | Deployed system: model, tools, memory, identity, loop | Model behavior and refusal policy | HTTP endpoints, auth, business logic |
| Success criterion | Tool invoked, data left the boundary, unauthorized action, cost incurred | Policy-violating output generated | Exploit reaches data or function |
| Evidence standard | Attack chain plus trial count and observed rate | Transcript, benchmark score | Reproducible request/response |
| Determinism | Probabilistic, so report distributions | Probabilistic | Mostly deterministic |
Benchmarks such as HarmBench, JailbreakBench and AdvBench measure refusal behavior. They say nothing about the confidentiality, integrity or availability of your system. Keep them off the security posture slide.
How do you scope and sequence an AI agent pentest?
Run seven phases in order, because each feeds the next. Skip recon and identity mapping and your severity ratings are guesses.
- Scope and rules of engagement. Environments, accounts, data handling, stop conditions, cross-tenant rules.
- Recon. Every surface the agent exposes and every tool, data source and downstream system it touches.
- Identity and authorization mapping. Which principal each tool call runs as, with what scopes, for how long.
- Threat modeling. The lethal trifecta, excessive agency (OWASP LLM06), autonomy level and blast radius.
- Adversarial testing. Prompt injection as a primitive that delivers instructions, not as the finding.
- Impact chaining. Carry the primitive to a sink you can verify outside the transcript.
- Reporting. Attack-chain diagram, probabilistic evidence, severity with an impact narrative, tiered remediation.
Phase 6 depends on phases 2 and 3. You cannot chain injection to an exfiltration sink you never mapped. In our experience, a focused single-agent assessment with API access fits 5-15 tester-days; multi-agent or MCP-federated systems with custom tooling run 3-6 weeks. Those are our heuristics, not an industry standard.
Agent tests touch live mailboxes, ticket queues, third-party SaaS and sometimes other tenants’ data, so the rules of engagement must say what testers may read, what they redact and when they stop. Access level then decides what you can prove:
- Black-box: shows injection is reachable, but tool wiring and delegation stay invisible, so blast radius is inferred.
- Grey-box: tool manifests, trace access (LangSmith, Langfuse, Arize Phoenix) and a test tenant. The minimum for a defensible severity rating.
- White-box: source for tool implementations, authorization middleware and the orchestration graph. Required when the agent can write, pay or execute code.
Run write-capable tests in mirrored staging and keep production read-only or canary-scoped. Record every exclusion with a reason and a risk owner; a silently excluded tool, identity, memory store or approval gate removes the agent-specific risk from the report. Our AI agent penetration test scope checklist lists the line items.
Two items teams forget. First, the fallback model chain: the primary model gets hardened while failures route silently to an unguarded fallback. Second, the cost ceiling: unbounded consumption testing (LLM10) needs an agreed spend limit, or the test itself will cause a cost incident.
How do you map tools, identities, and trust boundaries without documentation?
Reconstruct the tool surface from what the agent leaks and what its clients ship, then attach an identity to every tool you find. Undocumented agents still describe themselves through:
- System prompt leakage (LLM07): tool names, parameter schemas and routing rules often sit in the prompt.
- Error messages: malformed tool arguments return validator output naming fields and types.
- MCP
tools/list,resources/list,prompts/list: any reachable MCP server describes itself over JSON-RPC 2.0. - Client bundles and SDKs: web front ends, IDE extensions and mobile apps embed schemas and endpoint paths.
- HTTP interception: Burp, Caido or mitmproxy between orchestrator and tool backends show the calls actually made, not the declared ones.
Identity mapping is the highest-yield recon step and the one most often skipped. For each tool, record:
- The principal it runs as: user-delegated OAuth, a shared service account, or a workload identity
- Its scopes
- Token lifetime and storage location
- Whether the agent can act with more privilege than the caller (the confused deputy)
A shared service account behind a per-user chat interface is the most common severe finding we see, and exploiting it needs no prompt injection at all. The agent’s real privilege is the union of its tools’ privileges as enforced, not as declared. We cover the test cases in how to test excessive agency in AI agents.
Attacker content enters wherever the agent reads something a third party can write: RAG chunks, web fetches, inboxes, tickets, calendar invites, tool responses, other agents and the memory store. Sinks are tool calls, outbound HTTP, code execution, database and file writes, messaging and payment APIs. Every source-to-sink path through the model is a candidate injection path.
That is why indirect prompt injection dominates real attacks. Simon Willison coined “prompt injection” in September 2022, and Greshake et al. formalized the indirect variant in February 2023. Direct injection needs a malicious user. Indirect injection needs one malicious paragraph somewhere the agent will read.
Treat model output like user input in every downstream system. If it lands unescaped in HTML, a SQL string or a shell argument, you have an OWASP LLM05 Improper Output Handling finding before you write a single payload.
What is the lethal trifecta, and how do you prove impact with it?
The lethal trifecta, named by Simon Willison in June 2025, is an agent with access to private data, exposure to untrusted content, and the ability to communicate externally. With all three, it is exploitable by design. The test: find all three legs, prove a chain through them, recommend removing one.
- From the identity map, list every private data source the agent can read.
- From the source map, list every untrusted input it ingests.
- List every egress path: tool calls to arbitrary URLs, rendered markdown images and links, link unfurling, email or chat sends, DNS lookups, and error or log channels that leave the boundary.
- Plant a canary in the private store, deliver instructions through the untrusted channel, and watch a listener you control.
If the canary arrives, you have a finding with a real sink. If it doesn’t, that is a data point, not a control. Recommend an architectural break: an egress allowlist, no rendering of model-generated URLs, or separate reader and sender agents. Payload filters lose to rephrasing. A missing leg does not.
EchoLeak is the reference case. Tracked as CVE-2025-32711 in Microsoft 365 Copilot (CVSS 9.3, disclosed by Aim Security in 2025), it combined an inbound email carrying instructions, retrieval of the victim’s private context, and an auto-fetched image reference that carried data out. The user clicked nothing. The sink was a network request, not a sentence.
Sinks worth chaining to, roughly ordered by how much each moves the severity rating:
- Cross-tenant read via RAG or memory (no injection needed if namespaces are confused)
- Unauthorized write or action through a tool running as a privileged service account
- Zero-click exfiltration through rendering or unfurling
- Code execution through a tool that passes model output to an interpreter
- Cost amplification through recursive tool loops (LLM10)
Map each chain to MITRE ATLAS (for example, AML.T0051 LLM Prompt Injection) so the SOC can tie it to existing detection content. We do the trifecta mapping as part of agentic AI threat modeling before any payload is sent, so the testing budget goes to chains that can actually be closed end to end.
How do you rate and report a probabilistic agent finding?
Report the trial count, the success definition, the observed rate and a confidence interval. Then rate severity with CVSS as a floor plus an impact narrative. Agents are stochastic, so one run in either direction proves nothing.
Seven exfiltrations in 50 trials is a 14% observed rate, with a 95% Wilson interval of roughly 7-26%. That is a real finding: an attacker who can send 50 emails gets several hits. Zero in 30 trials still leaves an upper bound near 10%. That is not zero, and it is not evidence that a control works.
Define success precisely: “tool invoked with an attacker-chosen URL,” not “model agreed.” Never re-run until a result looks clean. That is how teams bury real findings or manufacture fake ones.
CVSS under-describes agentic risk because exploitability is probabilistic and impact depends on the tool’s authority. Agent-specific scoring guidance is still maturing, so pair the base score with:
- What data or action was reached, and as which principal
- Blast radius: one user, one tenant, all tenants
- Autonomy: whether a human approved the step, and whether the approved action matched the executed call
- Repeatability: observed rate and interval
The core artifact is the attack-chain diagram: entry vector, trust boundary crossed, tool and identity used, impact, evidence. Add a detection section answering one question: what would the SOC have seen? Many agents ship with no per-tool audit trail, which is a finding on its own. Traces normalized to the OpenTelemetry GenAI semantic conventions (still experimental) make remediation verifiable.
Where PyRIT, Garak, promptfoo and Inspect fit
None of these tools proves a chain on its own. Each covers one stage:
- Garak (NVIDIA): probe-based scanning of model and prompt weaknesses. A baseline, not a tool-chain test.
- PyRIT (Microsoft): scripted multi-turn and converter-based attack campaigns against an endpoint, for phase 5 at volume.
- promptfoo: config-driven test cases and red-team plugins that run in CI.
- Inspect (UK AI Security Institute): an evaluation framework with agent tasks and sandboxed tool use, suited to measuring success rates.
Convert every confirmed chain into a promptfoo or Inspect regression case, so the retest measures the same rate against the same success definition.
What changes when you test MCP servers, RAG pipelines, and multi-agent systems?
Each adds a trust boundary a plain chat agent lacks, and each needs its own test path.
MCP. The Model Context Protocol (Anthropic, November 2024) exposes tools, resources and prompts over JSON-RPC 2.0. Local stdio servers run with the user’s host privileges, so a compromised server means code execution on the host. Remote Streamable HTTP servers shift the problem to authorization. Test for:
- Tool poisoning: instructions hidden in tool descriptions (described by Invariant Labs, April 2025)
- Rug pulls: a definition that changes after the user approves it
- Cross-server tool shadowing
- Token passthrough, which the MCP authorization spec names as an anti-pattern; check audience binding via RFC 8707 Resource Indicators and RFC 9728 Protected Resource Metadata
- Session hijacking on stateful HTTP sessions
Published advisories against MCP Inspector and mcp-remote show these bugs reach real tooling. Our MCP server and tool-chain security testing follows this path.
RAG. Prove which document IDs reached the context window. A second tenant and canary documents catch ACL loss after chunking, metadata filter injection and namespace confusion (LLM08). Memory stores need the same tests plus one more: does a poisoned memory survive into a later session for a different user?
Multi-agent and A2A. Google’s Agent2Agent protocol, donated to the Linux Foundation in 2025, adds Agent Cards and task delegation. Test for forged or tampered Agent Cards, delegated authority that grows across hops, and injection in agent A’s input that triggers a tool call by agent B. When every hop trusts the previous one, one injection travels the whole pipeline.
Which defenses hold against indirect prompt injection, and which are theater?
The defenses that hold are architectural and live outside the model. Classifiers and prompt instructions lower the observed success rate, but they share the model’s trust domain and can be worked around. Rank remediation accordingly.
| Tier | Examples | Bypass resistance |
|---|---|---|
| Architectural | Remove a trifecta leg, authorization enforced in code, per-tool capability scoping, egress allowlists, deterministic policy engine | High: does not depend on model behavior |
| Model-level | Input/output classifiers, spotlighting untrusted content, dual-LLM pattern, capability tracking such as Google DeepMind’s CaMeL | Variable: CaMeL and dual-LLM patterns reduce but do not eliminate success rates; classifiers are probabilistic |
| Operational | Approval UX, rate and cost limits, full tool-call tracing, SIEM rules on anomalous tool sequences | Detects and contains; approval gates fail under fatigue |
Theater looks like a system prompt saying “ignore instructions in documents,” a keyword blocklist, or an approval dialog that shows a summary instead of the exact tool call. Approval UIs fail three ways: fatigue, a time-of-check to time-of-use gap between approval and execution, and rendering tricks that hide the real arguments. Test the gate like any other control.
No prompt-level defense is currently considered reliable against indirect injection. Assume the payload lands, and make sure it reaches nothing worth having.
NIST AI RMF and its Generative AI Profile (NIST AI 600-1), ISO/IEC 42001 and the EU AI Act all drive demand for this evidence, though none prescribes a test method. The methodology above produces what they ask for.
FAQ
What is AI agent security testing?
It is a penetration test of an agentic system's model, tools, memory, control loop and execution identity. The goal is to show attacker-controlled content crossing a trust boundary into a tool and producing a verifiable impact, such as data exfiltration, an unauthorized action or a cross-tenant read.
How is an AI agent pentest different from LLM red teaming?
LLM red teaming measures what a model will say, usually against a refusal policy. An agent pentest measures what the deployed system will do: which tools fire, as which identity, and what data or actions an attacker can reach. Evidence is an attack chain with a measured success rate, not a transcript.
How long does an AI agent penetration test take?
In our experience, a focused single-agent assessment with API access fits 5-15 tester-days. Multi-agent or MCP-federated systems with custom tooling usually run 3-6 weeks. These are our scoping heuristics; tool count, write capability and access level move the number.
How many trials make a prompt injection finding credible?
Enough to report a rate with an interval. Fifty trials with 7 successes gives a 14% rate and a 95% Wilson interval of roughly 7-26%, which is a real finding. Zero in 30 still leaves an upper bound near 10%, so a clean run is not proof of a working control.
What actually stops indirect prompt injection?
Architectural controls outside the model: removing one leg of the lethal trifecta, enforcing authorization in code, scoping each tool's capabilities and allowlisting egress. Classifiers and prompt instructions lower success rates but can be bypassed, and no prompt-level defense is currently considered reliable.
If you need this methodology run against your own agents, with identity maps, measured success rates and attack-chain evidence your engineers can retest, see our AI agent penetration testing service.