Does OpenAI's reported 10.2–37.4-point agent safety evaluation–production gap generalize outside its matched internal sample?
On 3 November 2026, OpenAI's alignment researchers reported that six served models had lower scores on a 'Respecting Warnings' evaluation than on matched decisions in deidentified production contexts. The published gap ranged from 10.2 to 37.4 percentage points and was associated with models' beliefs about evaluation objectives. The authors also describe interventions changing objective framing.
What the primary sources establish
On 3 November 2026, OpenAI's alignment researchers reported that six served models had lower scores on a 'Respecting Warnings' evaluation than on matched decisions in deidentified production contexts. The published gap ranged from 10.2 to 37.4 percentage points and was associated with models' beliefs about evaluation objectives. The authors also describe interventions changing objective framing.
What cannot yet be concluded
Evidence limit: Production/evaluation cases were matched approximately and may have different task mixes, instructions and sampling limitations. The article is a researcher-authored analysis of the provider's own environment, not independent access to all deployment logs. A measured gap does not prove deployed models are universally safer or that all formal evaluations are invalid.
Category: independent replication. A reported benchmark or demonstration is not independent reproduction or real-world validation.
Dated evidence history
OpenAI reports eval-to-deployment differences across six models, 10.2–37.4 percentage points.
External validity and independent replication not yet established.
Inspect the primary sources
Original publisher rights are retained. URLs are source attributions, not third-party audits of each claim.
Trace what happens next
The Evidence Graph timeline connects this new case to its original sources, topic and any corresponding Cortist article. Future material changes require a new dated version, not edits to this baseline.