CORTIST / INDEPENDENCE-FIRST RESEARCH / UPDATED 27 OCT 2026
Five questions.
Five quality gates.
Before an AI milestone deserves a numerical forecast, its measurement has to survive independent scrutiny. Here is what the primary sources actually establish, what they do not, and how we would resolve each research question fairly.
CM-001QUALITY GATE OPEN
AI Agents · Event cutoff 2027-10-27
Scale Labs gives 222/272 private results in its V2 refresh write-up while the overview leaderboard methodology still describes a 276-instance private set. A metric based only on the label SWE-bench Pro V2 private is not reproducible without an exact submission, task-set and version identifier.
What the primary sources establish
- Official V2 public set has 642 tasks (not the appropriate source for the private milestone).
- The V2 refresh article reports 222/272 leading private solved tasks.
- A separate Scale Labs methodology page still describes 276 proprietary private instances.
Proposed resolution discipline
Choose a uniquely versioned 272-task or other explicitly archived private V2 instance set, freeze pass@1 and tool/harness constraints, exclude public and held-out splits and require benchmark-owner audited per-task proof. Preserve the original 90% question unchanged until a distinct registered version is authorized.
Next evidence requiredObtain an official immutable private-set identifier, denominator and adjudication archive from Scale Labs and an independent reviewer.
CM-002QUALITY GATE OPEN
AI & Robotics · Event cutoff 2027-10-27
INSAIT confirms LADDER contains physical and simulated multi-step manipulation tests and reports especially poor performance on long sequences; the announcement does not itself define an immutable, independently auditable five-object real-robot subset suitable for our binary threshold.
What the primary sources establish
- LADDER was introduced by INSAIT and NVIDIA on 26 October 2026.
- Both simulation and physical-robot evaluations are included; aggregation across modes cannot satisfy our real-robot-only question.
- The operator says specialized models almost never succeed in moving five or more objects consecutively; this is qualitative evidence, not a frozen 50% baseline.
Proposed resolution discipline
Specify exact five-object task IDs, identical hardware or rules for variation, number of independent trials, failed-retry policy, human teleoperation exclusions, end-to-end success definition and signed raw run logs. Define whether success is averaged over tasks, episodes or attempts.
Next evidence requiredRequest versioned task manifest, official evaluator acceptance rules and recorded trial counts from LADDER maintainers.
CM-003QUALITY GATE OPEN
AI & Governance · Event cutoff 2027-10-27
Article 101 describes European Commission fines against providers of general-purpose AI models, but a press warning, investigation, national fine or preliminary finding is not an adopted Commission fine. Before calling a future event the first, the pre-publication decision baseline must be independently checked.
What the primary sources establish
- Article 101 identifies the Commission as the decision maker and covers GPAI providers.
- Article 101(2) requires communication of preliminary findings and opportunity to be heard before the decision.
- The official GPAI enforcement guidance identifies 2 August 2026 as the start of fine powers.
Proposed resolution discipline
Count only a formal Article 101 Commission fine decision adopted within the original specified window; freeze whether decision date or publication date governs, how late-published decisions are handled and what public register constitutes complete evidence. Never assume that no qualifying earlier decision exists without a dated search and an independent legal reviewer.
Next evidence requiredIndependently search and archive Commission official fine decisions before the candidate cutoff and obtain a legal sign-off on first-event wording.
CM-004QUALITY GATE OPEN
AI & Robotics · Event cutoff 2027-10-27
Waymo and the City of Munich explicitly target public commercial service towards the END of 2027, while this Cortist draft asks for launch by 27 October 2027. The earlier cutoff is intentional and must not be confused with Waymo’s own expected launch window.
What the primary sources establish
- Waymo announced Munich manual mapping and future safety-driver validation on 25 August 2026.
- Waymo’s Germany FAQ says testing vehicles still have validation drivers and commercial rides are not yet offered as described there.
- The city of Munich independently reports a planned public launch towards the end of 2027.
Proposed resolution discipline
Require publicly bookable AND paid passenger service with no in-vehicle safety driver inside the Munich municipal area before 27 October 2027 23:59 UTC. Fix what counts as a member of the public, fare receipt, public booking channel, operator confirmation and independent local regulator or city evidence; exclude mapping, invite-only trials and free demonstrations.
Next evidence requiredArchive official operator and Munich city statements and require corroborated booking / fare and no-driver evidence before outcome review.
Detailed decision test / new v2 research protocolOnly a completed, publicly bookable paid Waymo passenger ride with no on-board safety driver and within Munich city limits can qualify by the original 27 October 2027 deadline. Waymo officially says commercial service is not yet offered and, with the city, targets late 2027. A permit, mapping drive or announcement does not establish launch. German operating-area permit rule.
Read exact proposed YES/NO test and evidence hierarchy (JSON v2)
CM-005QUALITY GATE OPEN
AI Chips & Compute · Event cutoff 2027-10-27
MLCommons Endpoints measures multi-point Pareto tradeoffs across throughput, latency and concurrency. Its public materials say normalization rules are still being developed, so a single 2x performance-per-watt result is not independently reproducible unless a specific official measured power field and matched operating point are chosen.
What the primary sources establish
- MLPerf Endpoints benchmarks live inference API throughput, concurrency, interactivity and first-token latency.
- MLCommons says power / throughput per watt is an optional normalization dimension and rules are developing.
- Separate MLPerf Inference Power datasets measure full-system wall power; data or scores from different benchmark suites cannot be silently mixed.
Proposed resolution discipline
Select the same GPT-OSS-120B version, workload, quality constraints and serving setup; predefine an interactivity / P90 latency operating band and concurrency, same verified power measurement boundary and units, and a dated baseline result. Freeze a reproducible baseline and future audit before claiming any 2x outcome. Consider changing to fully measured MLPerf Inference Power only via a separately dated public research-candidate version.
Next evidence requiredObtain an audited 2026 Endpoints baseline with officially accepted power and latency normalization, or prepare a distinct future redefined milestone. Do not compare incompatible MLPerf versions or self-reported TDP.
What happens before a probability is published?
We first freeze a public binary outcome definition, an unambiguous dataset or official decision source, the event cutoff and an evidence-publication lag. The accountable Cortist editor must approve the final protocol and each specific probability, and the dated original must be preserved and publicly verifiable. External peer review is optional. Later findings can trigger a review but cannot rewrite the original probability.
Additional operational protocols: CM-003 (EU Commission fine) and CM-004 (Munich driverless rides) now include explicit evidence hierarchy and decision cutoffs. These drafts are not yet editorially locked; external review is optional.
