Will an AI coding agent resolve 90% of SWE-bench Pro V2 private tasks by October 2027?
SWE-bench Pro V2 private benchmark: at least 90% audited pass@1. The question has a 27 October 2027 deadline, but no official Cortist probability or scored forecast vintage exists yet.
Evidence Pulse 2.0 checks the published Cortist News index when JavaScript is available. The source-linked, dated research articles remain accessible through Cortist News.
- Question ID
- CM-001
- Deadline
- 27 October 2027, 23:59 UTC for observable events
- Outcome measure
- SWE-bench Pro V2 private benchmark: at least 90% audited pass@1
- Forecast status
- Candidate definition only · no probability · not locked · not scoreable
- Evidence vintage
- First candidate text published 27 October 2026
What do we know today?
Scale Labs reported 222/272 private tasks resolved for a leading agent in September 2026; near-perfect public scores are not comparable.
Strongest because withheld tasks reduce contamination risk; not a test of general intelligence.
How would this milestone be resolved?
What qualifies as YES
The benchmark owner publishes a qualifying audited result on the frozen private V2 task set no later than the deadline.
What qualifies as NO
No qualifying result after an official dated archive review.
What would not qualify?
Public scores, self-reported results, changed task mixes and different test versions do not count.
Unresolved definition gate
Freeze a permanent private-split identifier, task denominator and official archive before probability issuance.
Consequence: no probability should be locked or scored before this issue is resolved and the protocol is externally timestamped.
Why does it matter?
Strongest because withheld tasks reduce contamination risk; not a test of general intelligence. Scale Labs reported 222/272 private tasks resolved for a leading agent in September 2026; near-perfect public scores are not comparable.
Interpretation limit: Strongest because withheld tasks reduce contamination risk; not a test of general intelligence. An individual benchmark or adoption result is not proof of AGI or a universal model capability.
Primary sources and verification references
- Primary source / methodology (original reference)
- Official reference / evidence (original reference)
What will Cortist publish next?
Before treating this candidate as a numerical forecast, Cortist must freeze the evaluation protocol, verify that the event is not already resolved, define an independent public evidence source and record a probability in a new, immutable version. The present candidate page is not part of the scored probability history.
Read Cortist’s public Forecast Protocol · View the existing immutable forecast register · Explore the AI Agents Topic.
Machine-readable candidate record: v1 JSON. Original candidate publication: 27 October 2026.