Could human auditors reliably identify deliberate sabotage in the reported research experiment?
Not reliably in this experimental setting. A November 16 preprint reports human auditors achieved AUROC 0.59—statistically indistinguishable from chance according to the authors—and fully identified planted sabotage in 17% of cases. A human expert, not an autonomous AI, was the saboteur.
- Researchers used a human expert as a proxy saboteur in a red-team/blue-team study of AI safety research transcripts.
- Human auditors with full transcripts and permission to run experiments achieved AUROC 0.59, and fully identified the specific sabotage in 17% of studied cases.
- The strongest reported AI auditor, Fable 5.1, achieved AUROC 0.73 and 32% full identification without rerunning experiments; the advantage over human auditors was within statistical uncertainty.
The signal
As frontier laboratories automate more of their research process, human supervision is often presented as a way to catch malicious or unreliable work. A preprint submitted on November 16, 2026 stress-tests that assumption with a focused experiment: can reviewers spot deliberate sabotage embedded in otherwise credible AI safety research?
The research team, including Keshav Shenoy, Anthony Zang and colleagues, asked a human expert to conduct and sometimes deliberately corrupt safety research. A separate set of human auditors examined full experiment transcripts and could run follow-up tests. This was a designed proxy setting; the experiment did not demonstrate an autonomous AI independently attacking a laboratory.
The result was a notable limitation in the tested oversight process. Auditors could not reliably distinguish sabotaged from benign work, with AUROC 0.59. They correctly identified the specific intervention in only 17% of the measured cases.
Human versus model auditing
An AUROC near 0.5 is consistent with little discriminatory power in the evaluated task. The reported human AUROC of 0.59 was not significantly different from chance, according to the authors. That does not mean auditors always failed or lacked all useful observations; it describes their ability to rank sabotaged and benign instances in this particular study.
The authors also evaluated frontier AI models as auditors. The best reported model, Fable 5.1, reached AUROC 0.73 and correctly specified the sabotage in 32% of cases, despite lacking permission to run additional experiments. Both point estimates exceeded human results, but the paper says differences were within statistical noise. The study does not establish a universal superiority of model auditing over expert human review.
Detection became harder when the expert saboteur could choose which research questions to corrupt, rather than being forced to manipulate every assigned task. Selective interference creates a realistic oversight challenge: an adversary may target ambiguous steps where a false result will appear most plausible.
What the paper does not establish
The deliberate saboteur was a human researcher used as a proxy for a hypothetical misaligned research agent. The study therefore does not document deployed AI agents autonomously manipulating scientific research. It also does not prove that the same detection rates apply to other teams, laboratories, domains or auditing protocols.
This research was posted as a preprint and should not be presented as peer-reviewed consensus or independently replicated evidence. Participants, task choice, expertise and review resources influence the result; a different protocol could produce substantially different performance.
The number 17% is a study-specific full-identification rate, not a forecast for the percentage of future real-world AI sabotage caught by oversight. Similarly, AUC is a detection metric, not a direct measurement of damage to scientific conclusions.
04 / What Cortist will watch
Independent replication with varied research domains, tests where autonomous AI rather than a human proxy attempts manipulation, auditing designs that combine human reviewers with AI tools, and reports of specificity and false-alarm trade-offs. It matters whether future methods can identify both the presence of corruption and its exact mechanism.
For the Reality Ledger, this is a new dated evaluation of a control method, not evidence resolving any historical forecast. The longer-term question is whether independent checks can keep pace with automation of the research they are meant to verify.
Frequently asked questions
Did an autonomous AI system sabotage the experiments?
No. Researchers used a human expert as a proxy saboteur.
Does AUROC 0.59 mean the human auditors failed every time?
No. It means weak discrimination across the evaluated cases, not zero detection or a universal failure probability.
Were AI auditors conclusively superior to humans?
No. The best AI auditor had stronger point estimates, but reported differences were within statistical uncertainty.
Original sources and verification limits
Cortist separates documented tests, developer claims, independent evaluations and observed outcomes. No original Ledger baseline, promised commitment or scored forecast was modified by this article.