We build detection for autonomous AI attackers. Doing that well means understanding how these models actually behave once they go on the offensive, so for the past year we’ve run them against controlled targets and logged every move. One recent batch of tests turned up something we weren’t looking for, and it says more about the state of production software right now than it does about any single model.
Why we build labs in the first place
An autonomous attacking agent isn’t a scanner and it isn’t a person. It reads a banner, an error page, a stray .env file, treats each one as a fact, and reasons about what to do next. Detecting that reliably means knowing how different models think under attack conditions: what they probe first, what they ignore, where they get cautious, where they get greedy.
So we build isolated lab environments with a known, fixed set of planted vulnerabilities, then let a model loose on them and record everything. The labs run locally, on dead credentials, with no egress. Nothing real ever gets touched.
Over a year of AI-assisted penetration testing narrowed down which models are genuinely good at offensive work. The shortlist we rely on across engagements is Grok, Kimi, DeepSeek, Qwen, GLM, and OpenAI’s models — not because a benchmark said so, but because we watched them work.
We used to run Anthropic’s Claude the same way, as the orchestrator behind the pipeline rather than an attacker. Over the last few months Anthropic has steadily tightened what its models will do for security and penetration testing use cases, and we got caught by those limits ourselves. So Claude got reassigned: instead of running the operation, it now builds the targets. It generates the lab environment, plants the vulnerabilities we specify, and hands the result to the other models to break.
That change is what surfaced the problem.
Three planted, six found
The setup was simple. Claude builds a lab with exactly three deliberate vulnerabilities. An attacking model goes in and reports what it finds. We compare the two lists.
On several labs the attackers came back with four, five, sometimes six findings. Our first assumption was the obvious one — the models were padding the count, reporting things that weren’t really there. That happens, and you learn to expect it.
They weren’t padding. We verified every extra finding by hand, and they were real, exploitable, and in more than one case critical. Claude, while generating the environment, had introduced its own vulnerabilities on top of the ones we’d asked for. Nobody instructed it to. Nobody caught it at build time. The extra holes just sat there until a second model walked in and used them.
This is the part worth sitting with. The model writing the code didn’t know it was shipping critical flaws. We didn’t know either, not until an adversarial model went in looking for trouble. It took a second AI, actively hunting, to reveal what the first one had silently left behind while it was busy doing something else entirely.
To be clear, this isn’t a Claude problem. Every model we’ve tested does this to some degree. Claude just happened to be holding the keyboard for this batch, and it’s a strong coding model, which makes the finding sharper rather than softer. If a capable model quietly adds critical vulnerabilities while building a small, tightly scoped test lab, it’s worth asking what’s sitting in the millions of unreviewed lines being generated for production right now — the same unreviewed-dependency problem we found on the npm supply chain with PhantomRaven, just one layer further upstream, at the point of authorship instead of distribution.
How the attackers behaved
The same test batch gave us a clean read on how different models attack. Sixteen authorized runs, four frontier models acting as pentesters against an isolated rig on 127.0.0.1, dead credentials, no real system anywhere in reach. Every run tripped a trap — sixteen out of sixteen — and each model got caught in a way that was distinctly its own.
Across the batch: 299 dead-token presentations caught, sixteen out of sixteen agentic runs triggering a trap, and zero false “agent” verdicts on ordinary background traffic — curl, uptime monitors, a normal browser pass. That last number matters as much as the catches; a detector that flags real users isn’t one anyone will leave switched on.
The uncomfortable extrapolation
Now scale the three-versus-six result out of the lab. Enormous volumes of AI-generated code are going straight to production with no meaningful review. We test freshly launched SaaS products regularly, and getting in usually takes a few hours of focused work. Some of the more successful ones are sitting on flaws that lead straight to a large-scale data leak, and at this point it’s mostly a question of who reaches them first.
The market is now producing tools whose entire purpose is to generate software faster, including platforms that generate other platforms. That’s unreviewed AI code producing more unreviewed AI code, and the vulnerability we watched appear once, in one controlled lab, becomes a pattern replicated at industrial volume.
For an attacker, that’s close to ideal conditions. For everyone else, it’s technical debt with a fuse already lit.
What we’d actually recommend
None of this is novel advice. It’s just the discipline that AI-speed development keeps skipping.
- Treat AI-generated code as untrusted third-party code, because that’s what it is. Review it before it touches production, starting with whatever hurts most when it fails: authentication, access control, anything that handles user data. If you don’t have the bandwidth in-house, that’s what our secure code review engagements exist for.
- If you’re paying an agency or a contractor to build your product, make security code review an explicit line item in the contract, not an afterthought, and then actually verify what you’re handed. “It runs” isn’t a security property, and a clean demo tells you nothing about who else can get in.
- If you already operate a SaaS product, assume an automated agent has already been pointed at your perimeter. Given how cheap and how capable these agents now are, that’s closer to a certainty than a worst case — worth confirming with a real AI agent penetration test rather than a guess.
- If your own product runs agents against other people’s systems, or exposes an agent to the open internet, you need a way to tell a probing model from a normal user before it finishes its first move. That’s the detection layer BastionAgent is built around, and it’s exactly what generated the numbers in this post.
Bottom line
AI made building software genuinely fast. It made breaking software just as fast, and in this case it also made finding what the building step quietly broke a lot faster than any human code review cycle would have. The teams that internalize all three of those facts are the ones still standing when the first wave of breaches from vibe-coded platforms lands — and on current trends, it’s coming.