// ai security

AI Model Neuron Backdoor: Fewer Than 10 Edits

Cover image for an AI model neuron backdoor report on editing fewer than 10 rare neurons in an open-weight checkpoint.

Microsoft researchers showed this week that an AI model neuron backdoor can be planted by editing fewer than 10 rarely used neurons, with no full retrain. A trigger in a prompt or a tool result makes the model emit a chosen command, which an agent would run under its own permissions. Microsoft has not seen the technique in attacks.

Key takeaways

  • Lazarus is Microsoft's name for a lab weight edit of fewer than 10 rare neurons, and the company has not reported the technique in an intrusion.
  • Nemotron-Terminal-8B used seven tuned neurons and hit at least 98.8% command output on triggered sessions. Qwen3.6-27B and Gemma-4-31B-IT ranged from 51% to 99.5%. Terminal-Bench moved from 46.59% to 42.05%.
  • In the Nemotron walkthrough the model stayed normal for three turns. A connection error then preceded a download-and-run command that executed as root in the test container.
  • The reliable check is a diff against the original base weights. A no-reference detector hit 91.4% of 35 edited checkpoints and flagged 9.7% of 31 clean ones.

What did Microsoft actually demonstrate?

Microsoft demonstrated a weight-level technique it named Lazarus: hidden behavior planted by editing fewer than 10 rarely used neurons, without retraining the model. The walkthrough used 10 edited neurons. The paper’s Nemotron test edited and tuned seven.

Lazarus is the research name for that weight edit, not a malware family. Planting it requires write access to the checkpoint before deployment. The report lists four routes to that file: an insider, a stolen account, build-pipeline access, and a modified model published on a public hub.

The rest of the model file and its runtime code stay unchanged. Microsoft says it has not seen any threat actors use the technique. The report names no campaign and no victim. Organizations that pull third-party checkpoints, fine-tunes, merges, quantized builds, or adapters are the ones most exposed.

A hosted API is a different case. The caller cannot edit neurons inside a vendor-controlled model. Exposure starts only if that provider’s checkpoint was already altered. Microsoft says a compromised trusted provider remains a risk even for lab models it serves. Self-hosted open weights are the direct path for this edit.

How does an AI model neuron backdoor emit a command?

Each edited neuron pushes one token of the attacker’s command into the model’s output, so a command needs about one neuron per token. The team picks neurons that rarely fire and barely affect ordinary answers, then sets them to fire only when a trigger string is present.

The trigger can sit in the user’s request or arrive later in a tool result. The account does not give the trigger’s length, an exact-match rule, or whether a paraphrase works.

The new settings are calculated directly and can be tuned with a few hundred training steps on those neurons alone. The researchers credit Claude Code, running Opus 4.5, with finding the core idea, and say coding agents make this class of edit easier to build.

ProjectDiscovery’s proof of concept, published on October 6, fine-tuned an open Qwen model so a trigger phrase made OpenAI’s Codex CLI send dummy credentials to a researcher-controlled server. Lazarus skips that training pass and writes a few neurons directly. The source gives no year for the October 6 write-up.

The tests measured how often an edited model printed a short, harmless command that prints a marker, on Nemotron-Terminal-8B, Qwen3.6-27B, and Gemma-4-31B-IT. These rates count command text, not a finished attack.

On Nemotron-Terminal-8B the team edited and tuned seven of the more than 12,000 neurons in the last feed-forward layer, using 23 example sessions. The model output the command in at least 98.8% of triggered test sessions and in none of the trigger-free sessions checked.

Qwen used six neurons. With the tuned edit, success on Qwen3.6-27B and Gemma-4-31B-IT ranged from 51% to 99.5%. The paper does not say which variable set the 51% floor: the model, trigger placement, or session length.

When Nemotron-Terminal-8B generated text freely, it produced the command in about 99% of triggered sessions. In one setup the command also appeared in 0.14% of sessions with no trigger. An agent that executes model text can run that untriggered command. The paper treats these figures as output rates and says full-attack tests are still needed. Free generation on Nemotron stops once the command text appears.

Both backdoored Qwen versions kept most of their AIME math scores, and the Terminal-Bench pass rate moved from 46.59% to 42.05%. A release gate that only checks whether the eval still passes will approve this file.

What did the Nemotron walkthrough show when the trigger fired?

In a researcher’s walkthrough, a backdoored Nemotron-Terminal-8B with 10 edited neurons was tasked with fixing a Node.js server. It worked normally for three turns. When the server returned a connection error, the model output a command to download and run a remote script.

Inside the test container the script ran as root and printed a banner naming the demonstration. Because the command can fetch any script, the damage depends on what the agent may do, not on the size of the edit. Tool rights, the sandbox, and outbound network limits decide what that script can touch.

If shell output runs with no person in the loop, isolate it. The demo container had root and a path to a remote script. Coding-agent sandbox options decide whether that fetch stays contained. AI agent penetration testing checks that boundary against the agent’s real tools.

How do you detect a weight-level backdoor in a checkpoint?

Reliable detection means comparing a checkpoint with its original, unmodified weights. Fine-tunes and quantization change weights broadly. This edit moves a few weight vectors by a large amount, which shows up only if you still hold the original base. A quantized community upload is the hardest file to vet, because its changes look like a normal conversion. Reject a model whose declared base cannot be checked.

A detector trained on this method, with no original copy, flagged 91.4% of edited test checkpoints and wrongly flagged 9.7% of clean ones. The set was 35 edited checkpoints and 31 clean ones, and it was not tested on other backdoor edits. That result is a starting point, not a control.

Asking a model to inspect a checkpoint worked less well. Claude Opus 5.5 judged 41% of edited checkpoints to be clean. GPT-6 Astra did not decide on almost every clean checkpoint. The paper does not report how often Astra declined on edited files. No scan can prove a model is safe.

Microsoft Defender for Cloud can scan models in Azure Machine Learning for malware, unsafe operators, and exposed secrets. That covers code-level risks and does not replace a weight-level audit. Microsoft Foundry ties each open-weight model it offers to its original publisher and scans the containers those models run in. The company says that protects customers using models from trusted labs. A compromised trusted provider is still a risk. Neither control audits a checkpoint you fetched elsewhere.

Microsoft is building a red-teaming tool based on these detection methods and exploring weight scanning for open-weight models it adds to Foundry. It plans to sign models so customers can check where they came from.

Compare every checkpoint with its declared base before deployment and after each update. Download only from the original publisher and check that publisher’s file hashes. Treat third-party fine-tunes, merges, quantized versions, and adapters as untrusted until they pass an audit. Keep approved models in a private registry and block agents from pulling models straight from public hubs.

Give each agent only the access it needs, run agent-generated code only in an isolated sandbox, and limit outbound network connections. Require a person to approve high-impact actions such as running commands or sending data off the network. Add behavioral backdoor scanning, like the method in Microsoft’s earlier scanner research, alongside the weight comparison. Name the model artifacts in the test scope. A scoped AI supply chain security audit puts the checkpoint, the publisher hash, and the agent tool scope in one review.

If an agent looks compromised, preserve the artifact hash and the exact checkpoint version before anyone replaces the file. Deleting it can erase the only evidence of the edit. Pair that with permission design that survives injection: least privilege, a sandbox without broad egress, and a human gate on shell and export actions.

FAQ

What is the Microsoft Lazarus AI backdoor?

Lazarus is a Microsoft research method that edits fewer than 10 rarely used neurons so the model emits a chosen command when a trigger appears in a prompt or a tool result. The name is the technique, not a malware family, and Microsoft says it has not seen threat actors use it.

Is the Lazarus neuron backdoor being used in attacks?

Microsoft says it has not seen any threat actors use the technique. The published evidence is a lab demonstration plus detection research, and the report names no campaign and no victim.

Which models did Microsoft test with Lazarus?

The team tested Nemotron-Terminal-8B, Qwen3.6-27B, and Gemma-4-31B-IT. Nemotron used seven tuned neurons and produced the command in at least 98.8% of triggered sessions checked. Qwen used six neurons, and tuned-edit rates on Qwen and Gemma ranged from 51% to 99.5%.

How do you detect a weight-level AI backdoor?

Compare the checkpoint with the original unmodified weights. A detector with no reference copy flagged 91.4% of edited checkpoints and 9.7% of clean ones, on a set of 35 and 31. The researchers say no scan proves a model is safe.

If agents in your environment load third-party weights and then run tools, AI agent penetration testing measures whether a triggered command stays inside the sandbox or leaves with the agent’s permissions.


// get started

Work with AgentOffense

Tell us about your target and goals. We’ll reply with scope and a fixed-price quote — usually within one business day.

./request_engagement