AI Pentest Tools Benchmark: Agents Lost on Juice Shop

A September 2026 black-box AI pentest tools benchmark ran eleven products against one public OWASP Juice Shop v20.3.0 and scored severity instead of raw alert volume. Claude paired with Burp Suite through MCP scored 9/10. Acunetix scored 8/10. The strongest autonomous agent, Strix, scored 6/10 after a failed first run. The tools with the largest alert piles scored 2/10. On this URL, AI pentest agents lost to a Claude-plus-Burp workflow. No human pentester baseline was recorded, so the ceiling of a real engagement was never measured.

What did this AI pentest tools benchmark actually run?

The runs were made in September 2026 from Windows 11 and Kali Linux workstations against one public OWASP Juice Shop v20.3.0. The text available to us does not name the author or the outlet. It describes no incident, no named victim, and no new flaw in a production product.

Twelve products were lined up and eleven were scored: six autonomous AI pentest agents, four classic DAST scanners, and one hybrid that connected Claude to Burp Suite through the Model Context Protocol. Shannon stayed out. The author describes it as an autonomous web and API pentester that works only with source code. A white-box product would have answered a different question than a URL-only test.

The target is the OWASP Foundation’s training shop: Angular in the browser, Node.js and Express on the server, SQLite underneath, 116 challenges in 16 categories the author maps across the OWASP Top 10. By difficulty: 13 one-star, 19 two-star, 27 three-star, 26 four-star, 19 five-star, and 12 six-star. The six-star tier includes deserialization and XXE. Thirteen trivial header gaps and twelve expert flaws sit in the same catalog, which is why a raw finding count is a weak score. The text does not say how many of the 116 challenges any tool solved.

Every tool received the same black-box terms: no source, a bare URL, and no hint that the app was Juice Shop, so a model could not recall a public walkthrough. Each product was installed from its own documentation, timed, and judged from saved reports, including crashes and empty first runs. The author graded seven things: count plus severity, speed, report quality, ease of use, local models without a cloud API, operability from Russia without a VPN and with payable billing, and cost. Auth state, role, crawl depth, and session handling are not recorded per tool. No human count, time, or severity mix appears in the text available to us.

Which tools ranked in the AI pentest tools benchmark?

The table is the author’s composite, not a solve rate. Per-severity splits exist only where the text gives them.

Tool	Time	Findings	Score	Money and constraint
Claude + Burp MCP	30 min	10 (2 critical, 3 high, 5 medium)	9/10	~$20/month as stated for the pairing; no on-prem; not from Russia
Acunetix	1 h 20 min	13 (critical, medium, low; no per-severity split)	8/10	Commercial; on-prem; usable from Russia in this test
Strix (2nd run)	58 min	5 (1 critical, 3 high, 1 medium)	6/10	~$15/scan; Apache 2.0; author string Qwen 3.5-35b via OpenRouter
Burp Suite Pro	28 min	15, all low or informational	6/10	~$449/year; local; no scan-profile name in the text
CAI	10 min	4 (3 medium, 1 low)	5/10	~$1/scan; missed the critical injection; project described as archived
PentAGI (2nd run)	1 h 32 min	6 claimed; listed severities total 4	5/10	~$2/scan; first run 1 h 50 min and zero findings
PenliAgent	1 h 11 min	6 low or informational	4/10	$40/month; no on-prem; not usable from Russia in this test
T3MP3ST	12 min	27 low or informational	2/10	~$2/scan; author string Gemini 3.1 Pro; data to cloud
PentestGPT	20 min	3 (1 critical, 1 low, 1 info)	2/10	~$20/scan; JSON only; author label Claude Opus 4-8
Nuclei	11 min	24 informational	2/10	Free; author cites 10,000+ templates
Nikto	4 min	3 informational	1/10	Free

The author’s 10-point grades blend severity with speed, reports, usability, hosting, regional access, and price. Two tools can share a 6/10 for different reasons. Strix confirmed a critical SQL injection on the second run, and the author says the proof worked. The text does not inventory what that proof read or changed, and it does not add a build hash. The 6/10 also folds in the crashed first run. Burp Suite Professional’s 6/10 is 15 low or informational findings, no injection in the summary, about $449 a year, fully local, with no scan profile named. One row is an interception proxy run as a scanner. The other confirmed an injection and paid for a crashed first attempt.

Strix is Apache 2.0, sandboxed with Docker. The author lists OpenAI, Anthropic, Google, OpenRouter, and Ollama. This pass used the author’s model string Qwen 3.5-35b through OpenRouter, the same string given for CAI. The hybrid’s about $20 a month is the author’s stated cost for the pairing, not an itemized Claude bill and not the Burp Suite Professional license. The 9/10 measures Claude reading Burp history, the site map, and scanner output, then forming test requests and a report, plus the operator’s setup. A purchase order for Claude alone does not buy this result. Acunetix is Invicti’s commercial DAST product. Per-severity counts for its 13 findings are not in the text, so the evidence under the 8/10 is thinner than for tools written up in full.

CAI, from Alias Robotics, installs with pip and was run as a terminal-based agent role called Web App Pentester, on the same Qwen 3.5-35b string via OpenRouter. It missed the critical injection. The author says a stronger model might do better. The project is described as archived, with CSI named as successor, after 18 papers, more than 30 CVEs, and a claim of first place in CTF rankings. The license in the text is MIT plus a proprietary component.

PentAGI’s first run spent 1 hour 50 minutes and OpenAI tokens for zero findings. The second does not reconcile six findings with the severities listed: one critical, one medium, and two low, which total four. The product is a Docker sandbox with a web UI, a memory system, and a stated set of more than 20 tools, including nmap, sqlmap, and Metasploit. The author needed a VPN to run it. The test model is given only as OpenAI, with no build identifier.

PentestGPT, presented as an autonomous agent from USENIX Security 2024, fought its own install. The documented three-command setup failed. Output was JSON, with no console UI. The author labels the backend Claude Opus 4-8. Treat that string as the author’s label. The text does not show that this name matches a documented vendor release, and it does not pin a build a second team could reproduce. The author credits the injection to the model more than to the wrapper, and names Claude Code and Codex as supported backends.

PenliAgent is the commercial cloud agent: base cloud configuration, no on-premises option. In the text available to us the narrative stops during the Windows install, so later operational claims about that run are not available to quote.

T3MP3ST’s author calls the interface visually overloaded and says data went to the cloud. The product description lists Claude Code, Codex, Hermes, OpenCode, and offline options through Ollama, LM Studio, or vLLM, under AGPL-3.0. The scored run was the cloud Gemini configuration, not the offline one. The text does not tie Gemini 3.1 Pro to a build hash.

Network reach split the field. Usable without a VPN in this test: Strix, Burp Suite Professional, Acunetix, Nuclei, and Nikto. Blocked from Russia without extra network gear: CAI, PentAGI, PentestGPT, T3MP3ST, PenliAgent, and the Claude plus Burp MCP hybrid. The best score in the table is a workflow the author’s environment could not run on a direct path. Findings and request traffic went to cloud model providers for every agent that needed one.

Why did the biggest finding counts score near the bottom?

Score confirmed severity and a working proof ahead of alert-list size. T3MP3ST listed 27 findings in 12 minutes and scored 2/10. Nuclei listed 24 in 11 minutes and scored 2/10. Nikto returned three informational items in four minutes and scored 1/10. Burp’s unattended pass sat with that low-severity pile: 15 low or informational issues, no injection in the summary, 28 minutes, and a 6/10 once price, locality, and the report entered the grade. The author treats the T3MP3ST pile as the missing-header collection free scanners already print. Missing security headers do not show that untrusted input changed a query, took a session, or touched an order. Nuclei, from ProjectDiscovery, executes template-defined checks. The author puts the library above 10,000 templates. Five findings with a critical outranked 27 without one.

The split that mattered was a critical SQL injection in the training app. Strix confirmed one on a second run. PentestGPT reported one critical finding, which the author credits to the model more than to the wrapper. PentAGI’s retry is described as including a critical finding. The Claude and Burp Suite MCP hybrid reported two critical findings. CAI, T3MP3ST, PenliAgent, Burp Suite Professional’s unattended pass, Nuclei, and Nikto did not confirm that injection. A confirmed injection means untrusted input altered a database query. Business impact still depends on database permissions and on which query was reachable.

PentAGI’s empty first pass burned 1 hour 50 minutes and an OpenAI token bill. Strix’s first cloud attempt lasted about three hours, emitted two informational findings, and terminated abnormally. The 6/10 and the critical proof exist only on the retry. Two attempts show a spread. They do not establish a success rate. Publishing the successful retry alone turns a coin-flip into a case study.

Strix with a local Qwen build through LM Studio scanned for more than 24 hours and returned four low-severity findings. The author does not give that local build the 3.5-35b name used for the OpenRouter runs. The cloud configuration confirmed the critical issue in 58 minutes on the second try. The text does not separate hardware, quantization, context length, latency, and configuration from the hosting location. One local trial does not prove that on-premises inference itself caused the weaker result. For a team barred from OpenRouter, Anthropic, Google, or OpenAI, four low findings after a day is the result on offer in this test.

What to require in the RFP

Put the gaps in this bake-off into the statement of work. A mid-size SaaS should buy a scoped web application penetration testing engagement that reports confirmed issues, and should treat an unattended agent as a scout inside that engagement.

• Name the auth state and role for every run, including what happens when login fails.
• Set crawl depth and the workflows that must be reached: login, checkout, and at least one admin path.
• Require coverage against known routes or challenge identifiers, not only an alert total.
• Demand per-severity counts that add up, plus what a critical proof read or changed.
• Require reproducibility: tool version, model string and build, date, retry rule, wall-clock time, and spend on failed runs as well as the success.

Given a hybrid at 9/10 and an agent at 6/10 with one confirmed SQL injection, buy the operator-in-the-loop pattern. The hybrid score is a person plus a proxy plus a model. The agent score is a retry after a crash. Neither number is a human solve rate, because that baseline was not recorded. AI-powered penetration testing belongs in the contract as frontier models under a human operator, with the proxy history still reviewable.

Which vendor claims did the Juice Shop runs fail to reproduce?

CAI’s claim of a 3,600-times speedup versus a human pentester, and a 156-times cost reduction, did not show up as a stronger test. The agent was fast and cheap, about 10 minutes and about $1, and it scored 5/10 while missing the injection that separated the field. The scored run used Qwen 3.5-35b through OpenRouter, the same string Strix used when it did confirm the injection on a retry. The text does not swap models inside CAI and re-score, so the miss cannot be pinned on the model name alone or on the framework alone.

T3MP3ST’s stated 90.1 percent on the XBOW benchmark did not predict the Juice Shop outcome. The author says the claim was not confirmed in practice. The agent scored 2/10 on Gemini 3.1 Pro, with low and informational output only. XBOW is a different task list, a different scorer, and a different chance to reward behavior this URL-only setup never measured. T3MP3ST’s offline backends were not the configuration that was scored.

The text available to us records a three-hour abnormal stop, a 110-minute empty PentAGI pass, a failed three-command install, a Windows install narrative that stops mid-description, and a local model that ran past 24 hours. Those attempts belong on the same slide as the accuracy percentage.

Strix’s own positioning, autonomous agents that confirm findings with a proof of concept, is the claim this test came closest to supporting, and only on the second cloud run. One confirmed SQL injection in 58 minutes, after a three-hour failure, is a real result and a narrow one. Five findings against a catalog of 116 challenges is not a replacement for a pentester. The local Qwen trial did not repeat the proof.

This single-app, name-withheld, black-box bake-off did not reproduce the speed and benchmark slides attached to these agents. Separate reporting on what current cyber models can already do on real vulnerabilities is a different evidence base. The single decisive number promised up front is not in the text available to us. The same results do not measure what a named human would have found on the same URL in the same month. Teams evaluating agents as products, rather than as operators, still need a test of tool-use abuse and scope control. That is AI agent penetration testing, and this Juice Shop bake-off does not measure it.

FAQ

Do AI pentest agents find more vulnerabilities than DAST scanners?

Not on this bake-off. Acunetix scored 8/10 with 13 findings. Strix, the best autonomous agent, scored 6/10 with five. The highest score was Claude plus Burp Suite at 9/10 with 10 findings, two of them critical.

Which AI pentest tool scored highest on OWASP Juice Shop?

In the author's 10-point composite, Claude paired with Burp Suite through MCP scored 9/10 in 30 minutes. Strix was the highest autonomous agent at 6/10. The text does not report how many of Juice Shop's 116 challenges any tool solved.

Can autonomous AI agents replace a human pentester?

This comparison cannot answer that. The text available to us contains no human count, time, or severity mix. The best unattended agent still needed a retry to confirm one critical SQL injection.

Did local LLMs perform worse than cloud models in this pentest test?

In the one local trial reported, Strix with a local Qwen build through LM Studio ran more than 24 hours and returned four low findings. The cloud retry on Qwen 3.5-35b via OpenRouter confirmed a critical SQL injection in 58 minutes. The text does not separate hardware, quantization, and context length from the hosting choice.

If a vendor scorecard is what you have, and a report an auditor will read is what you need, agent::offense runs AI-powered penetration testing with the empty runs, the proofs, and the unreachable routes written down.