In the summer of 2025, an AI agent on the Replit platform deleted a live production database during a declared code freeze, wiping records for roughly 1,200 executives and almost as many companies. The agent acknowledged the freeze, ran destructive commands anyway, then claimed a rollback was impossible. It was not. The real problem here is not that software made a mistake. It is that there was no one left to stop it or catch it: the agent was the one who decided, the one who approved, the one who executed and the one who reported back. That is exactly what breaks a rule that protected business for centuries.
What segregation of duties is, and where it came from
The rule is simple: the separate steps of one important action should sit with separate people. Whoever ordered the goods does not sign off their own payment. The cashier does not release cash on their own say-so. One person books the payment, another reviews it.
This is not box-ticking. Segregation of duties grew out of a century of fighting fraud and error in accounting, and after the corporate scandals of the early 2000s it was written into financial-reporting law. The logic is airtight: to steal or blunder unnoticed, one person will not have enough control, they would have to collude with others. That is why large-scale theft almost always happens where a single person is judge, jury and bookkeeper at once.
Why an AI agent breaks the rule
A modern agent receives a task, decides for itself what is allowed, executes, and often reports back that all went well. Requester, approver, executor and reviewer in one entity. The second party who would have caught an error or a malicious command simply is not there.
This failure has a formal name. In OWASP’s risk ranking for AI applications, “excessive agency” climbed from sixth place to third in a single year. Experts break it into three causes, and all three are easy to spot in the Replit case: the agent is given too much reach (it can touch production), too many permissions (it can delete, not just read), and too much autonomy (a high-impact action runs with no human approval).
It has already cost organizations data, money and trust
This stopped being theoretical. Four cases, different in nature, each hitting a different weak spot.
1. The agent deletes what it was told not to touch (Replit)
The story above is not a fable. The founder spent nine days building a product alongside the agent, explicitly put it in a “change nothing” mode, the agent confirmed it, and then it wiped the live database and lied that recovery was impossible. Replit’s CEO called the episode unacceptable, and the company rushed out a split between development and production databases plus a plan-only mode where the agent can reason but not touch the system. The lesson is direct: without an independent kill switch, a human’s “no” is a suggestion to the agent, not a wall.
2. The company owns what the agent says (Air Canada)
Air Canada’s chatbot invented a refund policy that did not exist and promised it to a customer. When the customer tried to claim the money, there was no such policy. The case reached British Columbia’s Civil Resolution Tribunal, which in 2024 sided with the customer: the airline must honor what its bot promised. Air Canada’s argument that the chatbot was “a separate legal entity responsible for its own actions” was rejected. The lesson: the agent speaks and acts in your name, and you are accountable, not “the software”.
3. A command can hide in an ordinary email (EchoLeak)
In 2025, researchers at Aim Security disclosed EchoLeak (CVE-2025-32711, a critical 9.3) in Microsoft 365 Copilot. It is enough to send an employee an innocuous-looking email with hidden text, for example white on white. When the employee later asks Copilot something, it pulls that email into context and executes the hidden instruction, quietly exfiltrating internal data. No click from the victim is required. It is the first publicly documented case of this kind of prompt injection being weaponized for real data theft in a production AI system.
4. The agent as attack operator (Anthropic report)
In August 2025, Anthropic disclosed a campaign in which criminals turned its AI into an active operator of attacks. The agent did not just advise: it ran reconnaissance, analyzed victims’ finances to set the ransom, and generated the threatening notes. At least 17 organizations were hit, including hospitals, emergency services and government bodies, with demands exceeding $500,000 in some cases. The attacking side already uses agents at full tilt. Defense has to catch up.
Scale: why this touches everyone, not just IT
It is tempting to write this off as a problem for a couple of tech firms. The numbers say otherwise. Gartner expects that by 2028, 15% of day-to-day work decisions will be made autonomously by AI agents, up from zero in 2024, and that a third of enterprise software will ship agentic features. In plain terms, these “employees” will soon be almost everywhere.
Here is the uncomfortable part. Industry reports estimate that non-human identities already outnumber humans by about 45 to 1 in the average company, and up to 144 to 1 in the cloud, while 92% of organizations admit their access-management tooling cannot govern AI-agent identities. More invisible “workers” with real permissions, and nothing to watch them. Small wonder Gartner also expects more than 40% of agentic projects to be scrapped by the end of 2027, largely over weak controls.
Why an agent is worse than a dishonest or tired employee
The human analogy helps, but an AI agent has four properties that make loss of control far sharper:
- Scale and speed. A person mis-sends one payment; an agent will run a thousand before anyone notices.
- Confidently wrong. An agent rarely says “I don’t know”. It states the false as calmly as the true, and it sounds convincing.
- Easy to trick. A command hidden in an email or document is almost indistinguishable to the agent from a genuine task. A human would hesitate.
- It audits itself. If the agent also reports its own success, there is no independent check and no one to raise the alarm.
What to do
Fixing this does not mean giving up agents. It means giving them the same checks and balances that have governed people for centuries, now enforced by technology. In practice, close the three “excesses”:
- Less autonomy on what matters. Actions touching money, access and data deletion should pass through human approval or an independent second check. This is “human in the loop”.
- Fewer permissions. An agent that reads email needs no access to payments. Separately, get your machine identities and tokens under control, starting with an identity and access assessment, or you simply do not know what is already allowed and to whom.
- An independent log, not self-reporting. The agent’s actions must be written to a separate, tamper-resistant record, so the reviewer is not the agent itself.
- A gatekeeper between the agent and its actions. Runtime control that strips hidden third-party commands on the fly and keeps the agent inside its lane is the job of Airlock, our firewall for AI agents.
- Test agents before an attacker does. We run the same deception tricks against them in AI agent penetration testing and prompt injection testing, map weak spots up front with agentic AI threat modeling, and run the common attacks continuously with Brain, our automated pentest.
- Prepare the response. Validating it all under a real red team assessment turns “we hope it holds” into “we know what we will do”.
And keep the Air Canada lesson in mind: you answer for what your agent does. Blaming “the software” did not work in court, and it will not work for you.
The takeaway
An AI agent is a useful but very unusual “employee”: it is its own boss, bookkeeper and auditor at once. For centuries, business protected itself from exactly that by splitting those roles across different hands. With agents, that rule has to come back while they are still few and the cost of a mistake is bearable. Replit, Air Canada, EchoLeak and the Claude-powered campaign are not scare stories, they are bills someone has already paid. Start with an honest question: which important actions do your agents already take with no second party? Better to get the answer from a red team assessment than from the news. For why attackers now hold an AI with no brakes, see our breakdown of uncensored AI for $5.