Multi-Agent System Security: Test Trust Per Hop

Multi-agent system security fails at the message, not the model. Orchestrators paste worker output, tool results, and retrieved documents into the next prompt as ordinary text, so any agent that can influence that text can borrow another agent's tools and credentials.

What is multi-agent system security, and how is it different from one LLM app?

Multi-agent system security is the control of trust between two or more LLM-backed components with distinct prompts, tools, and credentials. Those components exchange natural-language or semi-structured messages.

A single LLM app has one context window and one tool set. A multi-agent system adds hops. Each hop is a place where untrusted text can be promoted into an instruction.

Four topologies show up in production. An orchestrator/worker pair routes tasks and merges results. A hierarchy puts managers between the planner and the leaf workers. A peer handoff transfers the live task between specialists. A shared blackboard or vector store lets every agent read what any agent wrote.

Authority sits in a different place in each topology. A finding in one does not transfer cleanly to another.

Topology	Who holds authority	Where injection lands	What is easiest to bound
Orchestrator/worker	Orchestrator tools and parent credentials	Worker output copied into the next planner turn	Tool allow-list on the orchestrator, scoped worker tokens
Hierarchical	Each manager’s tool set, often inherited downward	Any intermediate summary the parent treats as a plan	Per-level capability attenuation
Peer handoff	Whoever currently owns the task	The handoff payload and the claimed sender role	Signed envelopes and a fixed receiver policy
Shared blackboard	Every future reader of the store	A poisoned document or memory entry that persists	Write ACLs, provenance tags, ingestion checks

The shared defect is channel collapse. The system prompt, the user turn, tool output, a retrieved chunk, and a peer message all arrive as tokens in one context window.

Textual framing such as “the following is data” is the model’s only separator. Framing is not an enforcement mechanism.

Greshake and colleagues showed this in February 2023 (arXiv:2302.12173): content an integrated LLM retrieves can carry instructions. In a multi-agent system that pattern is the default path, because every agent is a data source for the next one.

How does agent-to-agent prompt injection propagate?

Prompt injection propagates when the orchestrator forwards a worker’s output, a tool result, or a retrieved document into the next prompt without a separate data channel.

Direct injection is the user attacking their own session. Indirect injection is a third party planting instructions in data the system later processes: a web page, a PDF, a ticket, a connector record, or a tool description. Multi-agent designs make the indirect case the common one.

A typical path has four steps. None of them require the attacker to talk to the privileged agent.

• Untrusted content enters a low-privilege worker through a tool, a browser fetch, or a retrieval hit.
• The worker’s completion, or the raw tool payload, is appended to the orchestrator’s context as the research result.
• The orchestrator treats that text as guidance and selects a tool the worker was never allowed to call: send mail, write CRM, read object storage, open a ticket.
• If the framework loops, the same instruction is retried and rephrased. Propagation follows the message graph.

Simon Willison’s lethal trifecta is the triage test: private data, exposure to untrusted content, and an outbound channel. Teams often split those three legs across agents and call that isolation. A shared message bus puts the legs back together. Splitting the model calls does not split the authority.

Exfiltration does not have to be an obvious “send this file” tool. Test markdown images whose URL carries data, link unfurling, webhook arguments, DNS lookups from a code interpreter, calendar invites, and comments written back into a ticket the attacker can read.

OWASP LLM05 (Improper Output Handling, 2025 list) is the right label when agent text hits a shell, SQL, a template, a browser, or another agent’s tool call. In a multi-agent system the downstream sink is usually that next tool call.

What does privilege confusion look like between agents?

Privilege confusion is a confused deputy: a component with real authority uses that authority for a party that should not have it. Norm Hardy named the pattern in 1988. Agent frameworks recreate it when an orchestrator hands attacker-influenced text to a worker that holds a token, or when a worker’s text drives an orchestrator with a broader token.

Three forms show up on assessments.

• Authority inferred from message content. The planner chooses the tool by reading the previous message. The message is attacker-controlled. The tool call still runs with the planner’s credentials.
• Inter-agent impersonation. “Message from the orchestrator” is a string plus a role label. Any component that can write the bus, the scratchpad, or the handoff API can set that label. Dennis and Van Horn (1966) put authority in an unforgeable, attenuable token tied to a task, not in a claimed role.
• Credential inheritance. Sub-agents are spawned with the parent’s environment. A worker spawned to read public pages still holds a token that can write the CRM, send email, or list the object store. The injection stays low-value until that token makes it high-impact.

A jailbreak pushes the model past its refusal policy. Injection uses the application’s authority. Tightening refusals on the worker model leaves the deputy path intact.

OWASP LLM06 (Excessive Agency, 2025 list) is the finding class once you can show which credential the attacker effectively obtained. A screenshot of a banned phrase is not that evidence.

Shared memory makes the deputy persistent. A poisoned chunk in a vector store or blackboard is re-read on later runs. That is OWASP LLM08 (Vector and Embedding Weaknesses): the index holds a copy of the business’s most sensitive content, and it is a write path even when product language calls retrieval read-only.

Filenames, PDF metadata, image text, audio transcripts, JSON key names, and tool schemas all reach the same window. A test corpus of plain paragraphs misses the channels operators actually ship.

How do you threat-model a multi-agent system and mark the trust boundaries?

Draw agents as processes, the bus and the memory stores as data stores, and every tool as a sink. Then mark which edges are allowed to carry instructions. An agentic AI threat modeling pass that stops at “the model might be tricked” does not tell engineering which hop to close.

Put these elements on the diagram before you argue about prompts.

• Each agent: model, prompt version, tools it can actually invoke, and the credential it presents.
• Each message edge: who can write it, whether the sender is authenticated, and whether the receiver may treat the body as a plan.
• Memory and RAG: who can insert, who can read, tenant and ACL predicates, and how long a write lives.
• Sinks with side effects: email, tickets, payments, cloud APIs, shells, browsers, and other agents.
• Human gates: whether the approver sees raw tool arguments or a model-written summary of those arguments.

Cloud Security Alliance MAESTRO is a useful frame for that picture. Map the same flows to MITRE ATLAS names for prompt injection, jailbreak, and AI agent tool invocation, and re-check identifiers before you freeze a report. ATLAS gets restructured and IDs move.

On the diagram, a trust boundary is an edge where the sender is not allowed to choose the receiver’s next tool call. If that edge is only a role string inside the prompt, the boundary is not there.

What does multi-agent penetration testing measure?

An agent-to-agent penetration test measures whether untrusted content at one hop changes a tool call, a credential use, or a data read at a later hop. End-to-end “did the chatbot say something bad” misses the worker whose output a human never sees. That hidden hop is where most of the privilege delta sits.

Scope it as AI agent penetration testing of the deployed system. Start from an AI agent penetration test scope checklist so tools, memory, identity, and approval gates are in the engagement letter.

Access level decides what you can prove.

• Black box. You see user-facing behavior and side effects: emails sent, tickets opened, rows changed. You can show impact. You often cannot name the hop.
• Grey box. You get transcripts, tool traces, and a staging bus. This is the minimum that makes a fix-verification retest honest.
• White box. You get prompts, tool policies, token scopes, and the code that copies one agent’s output into the next prompt. Credential inheritance and forged role labels show up here.

Keep findings reproducible. Pin temperature to 0 where the API allows it, fix seeds where the runtime supports them, record full transcripts, and replay against mocked tool servers so a model upgrade does not erase the evidence.

Plant canary documents and a second tenant in the staging index. Score whether the wrong document IDs entered a context window and whether a canary value left through a sink.

Write scenarios as goals, not as strings. Against an orchestrator/worker pair, put attacker-controlled content only in a tool result or a retrieved chunk and record whether the orchestrator invokes a tool the worker cannot call.

Against a handoff, send a peer message with a supervisor role label and record whether the receiver accepts it without a signature or workload identity. Against a blackboard, write once and show a later, separate run acting on that write. Against approvals, compare the tool arguments that executed with the text the human approved.

Metrics that survive a design review:

• Attack success rate per hop, not only end to end.
• Utility under attack: the system should still finish the legitimate task.
• Privilege delta: which credential or tool the attacker effectively obtained.
• Blast radius: data stores and sinks reachable from that credential.
• Time-to-detect from production logs, and cost amplification in tokens and dollars per induced loop. OWASP LLM10 (Unbounded Consumption) is the category name to cite for the loop in a report.

A report that says “we found prompt injection” without the chain, the privilege delta, and a replayable transcript is a class label. Engineers cannot act on a class label alone.

Prompt injection itself rarely maps to a CVE. The sinks do: SSRF, command injection, path traversal, BOLA in a tool backend, auth bypass on the agent gateway. File those as the vulnerabilities they are. The same excessive-agency tests used on a single agent still apply at each hop: see how to test excessive agency in AI agents.

How should agent credential delegation work for sub-agents?

A guardrail model is a probabilistic classifier on text. It has a nonzero bypass rate and no representation of who may call which tool. Prompt hardening adds more instructions to the same collapsed channel the attacker is already writing into. Neither is an authorization control.

Defenses that fail a retest:

• Role labels and “internal traffic is trusted.” Injection is a content problem. A VPC boundary does not inspect the tokens inside an orchestrator turn.
• Filter the user, ignore tool arguments and tool responses. The payload and the damage travel on those two channels.
• Output filtering only at the orchestrator. The worker hop already selected the side effect, or wrote the poisoned memory, before the final answer is scanned.
• Human-in-the-loop on a model-written summary. If the attacker influenced the summary, the approver authorized a sentence, not the tool call.
• A new system prompt or a model upgrade. The attack surface moves with every tool, topology, and retrieval change. Re-test the hops.

Designs with an enforcement point outside the model are the ones that hold on retest. CaMeL (Google DeepMind) splits a privileged planner from a quarantined model that reads untrusted data, then applies capability tags and data-flow policy in a custom interpreter. FIDES (Microsoft Research) tracks taint into tool calls. IsolateGPT (NDSS 2025) isolates execution between agent apps. StruQ and SecAlign harden the model interface. They still sit beside policy, not instead of it.

Scope credentials to the task, not to the parent process, and decide the allow-list in a policy engine before the tool runs. Least privilege is the token the worker presents on that call. It is not the phrase “least privilege” in the system prompt.

Controls that survive a hostile tool result:

• Workload identity. SPIFFE/SPIRE, or the platform equivalent, so the receiver knows which workload sent the message. mTLS authenticates the peer. It does not authorize the tool.
• Short-lived, attenuable tokens. STS-style credentials, macaroons, or Biscuit tokens minted for one task, with caveats the worker cannot widen. A research worker does not hold CRM write scope so the demo works.
• Deterministic policy. Open Policy Agent (Rego) or AWS Cedar evaluates agent identity, tool name, argument shape, data provenance, and tenant. The model proposes. The policy permits or denies.
• Provenance. Tag content as user, tool, retrieval, or peer. Untrusted tags may enter a planner’s data section. They may not reach a sink that sends, pays, deletes, or grants.
• Egress and execution sandboxes. Allow-list outbound hosts. Run code interpreters in gVisor, Firecracker, WebAssembly, or a managed environment with no ambient cloud credentials.
• Irreversible actions. Human approval bound to the raw tool name and arguments, stored next to the decision, and checked again at execution time so a rewritten summary cannot swap the call.

Static tool allow-lists beat dynamic discovery when the action has side effects. Dynamic discovery also lets a new tool description enter the context window after the set looked frozen.

LangGraph, CrewAI, the OpenAI Agents SDK, Google ADK, and Semantic Kernel pass messages between LLM-backed nodes. None of them ship a cryptographic trust boundary between those nodes by default. You add identity, policy, and provenance yourself. The framework will not do it for you.

Do MCP and A2A add attack surface, and how do you catch a hijacked agent?

Yes. Model Context Protocol and Agent2Agent standardize hops that used to be bespoke, which makes the trust bugs portable.

MCP, introduced by Anthropic in November 2024, is the dominant tool and resource surface. Later revisions added an OAuth 2.1 pattern in which the MCP server is a resource server, with RFC 8707 resource indicators and protected-resource metadata in the RFC 9728 style. That is real authorization progress. It does not stop tool-description poisoning.

The tool name, description, and schema are copied into the model context, so a malicious or compromised server can carry instructions. A rug pull is the same tool changing that description after a human approved the original text. Cross-server shadowing is a second server offering a tool the model confuses with a trusted one. MCP server and tool-chain security testing has to include metadata, not only the tool’s runtime arguments.

Google announced Agent2Agent (A2A) in April 2025 and later donated the project to the Linux Foundation. A2A standardizes discovery through Agent Cards and a task exchange between agents. An Agent Card is a discovery document. Treat it as untrusted input, with the same skepticism you apply to a tool schema.

A bespoke REST bus is not safer by being custom. It usually has less specified authentication, which means impersonation is an undocumented feature. Compare any bus on sender authentication, resource-scoped tokens, and whether discovery metadata enters the prompt. Then test the implementation you run.

Detection starts from logs a responder can trust. Record, per hop, the authenticated sender, the task id, the tool name, the raw arguments, the provenance tag on the content that influenced the call, and the approval record if one exists.

Alert when content tagged untrusted is followed by a tool the producing agent cannot call, when a role label disagrees with workload identity, or when token spend on one task jumps without a matching user action. OWASP LLM10 is the consumption side of that last signal.

Incident response is identity work first. Revoke the task tokens, disable the workload identity, and stop the agent from writing further memory. Quarantine blackboard and RAG entries written during the task so the next run does not re-read the payload.

Keep the transcript, the tool trace, and the policy decision. Replay them against the patched gate before you put the agent back. A hijacked agent that still holds the parent’s environment will do the same thing on the next message.

FAQ

Are agent-to-agent messages authenticated?

In most frameworks, no. A message from the orchestrator is a string with a role label. Without signed envelopes or a workload identity such as SPIFFE over mTLS, any component that can write the bus or the scratchpad can claim orchestrator authority.

Authentication names the sender. Authorization still has to happen outside the model, per tool and per task.

Can a low-privilege worker reach the orchestrator's tools?

Yes, when the orchestrator copies worker output into its own context and then calls tools with the parent credential. The worker never receives the token. It only supplies the text that selects the call.

That is the privilege delta to measure: which tool ran, under whose credential, after which untrusted input.

Do guardrail models stop agent-to-agent injection?

They reduce some casual abuse. They do not decide which identity may call which tool. A bypassed classifier still leaves the deputy path open, and a filter on the final answer misses the worker hop that already wrote memory or invoked a sink.

Is CaMeL a practical fix yet?

CaMeL, from Google DeepMind, is a dual-model design: a privileged planner stays away from raw untrusted text, and a quarantined model reads that text under capability tags enforced by an interpreter. It is a concrete information-flow pattern, not a drop-in product setting.

Treat it as one way to build an enforcement point outside the model. You still need per-task credentials and a policy engine on the sinks you actually run.

Which standards apply to multi-agent system security?

Use the OWASP Top 10 for LLM Applications (2025) for the finding label: LLM01 for injection, LLM05 for improper output handling, LLM06 for excessive agency, LLM08 for vector and embedding weaknesses, LLM10 for unbounded consumption. NIST AI RMF (AI 100-1, January 2023) and the Generative AI Profile (AI 600-1, July 2024) give govern, map, measure, and manage. ISO/IEC 42001 and ISO/IEC 23894 are the management-system artifacts procurement asks for. They are not per-hop technical controls.

EU AI Act Regulation (EU) 2024/1689 has been in force since 1 August 2024. Article 15 requires accuracy, robustness, and cybersecurity, including resilience to adversarial manipulation. Prohibitions applied on 2 February 2025 and general-purpose obligations on 2 August 2025. High-risk obligations for Annex III systems were set for 2 August 2026, and later legislative packages can move that date. Confirm the date in force before you treat it as a hard deadline. Cite Article 15 for the duty and your hop tests for the evidence.

agent::offense tests orchestrator, worker, MCP, and handoff paths as deployed, and reports the privilege delta per hop. See AI agent penetration testing for how that engagement is scoped.