CORTIST / AUDIT OF FORECAST WORKING MODELS / OCT 23, 2026
Every probability must earn trust.
Editorial methodology pre-review for three forecast candidates. All remain on hold for independent expert assessment and are ineligible for formal forecast scoring.
A / M / E INTERNAL METHOD AUDIT · This page records internal blockers and 0 external approvals for the three working models. It does not independently validate the figures; the separate AGI/ASI review covers F-001 and F-002. Editorial responsibility ↗
01 / INTERNAL RED-TEAM FINDINGS
Three candidates. Three review holds.
A source can be real while a probability derived from it remains subjective. Every HOLD here is an editorial assessment, not an independent expert decision.
Review status: HOLD / NOT READY TO LOCK. This is a self-audit by Cortist using primary-source fact checks and arithmetic; it is not a completed third-party review. No external reviewer has been appointed or signed off. Existing F-001 and F-002 draft probabilities remain unaffected, unreviewed and unscored.
HOLD / NOT INDEPENDENTLY REVIEWED
AI Agents / 2028
The METR task-horizon methodology is real, but the two 95% and 90% conditional scenario inputs are unsourced editorial judgments. An 80%-task-success horizon is not 80% probability of future attainment.
4 open review blockers
Select and preregister an independent, stable 80%-reliability benchmark or validated METR successor, including task mix, sample size and rules for post-2026 discontinuation.
Reconstruct comparable historical 80% task-horizon observations with time stamps and uncertainty; audit the trend using rolling-origin forecasting and holdouts.
Fix agent public-availability date, replication conditions, missing-data treatment and 'no evaluator by deadline' unresolved rule.
Obtain two named external conflict-screened independent signoffs and a third-party timestamp before any official forecast is published.
The FDA approval outcome can be verified, but no independent public base rate supports the 65% authorization input or 70% AI-design provenance input. Phase III entry cannot substitute for phase-adjusted historical rates.
4 open review blockers
Predefine substantive de novo AI design of the actual approved molecule; require patents, dated experiment trails and independent audit, not marketing phrases.
Estimate eligible pipeline base rates by trial phase and disease from independently verified approval histories; address censoring and selection bias.
Define CDER vs CBER, NDA/BLA, accelerated approval and 2032 cutoff; resolve FDA authorization only from official records.
Obtain two conflict-screened independent pharma/regulatory and statistical reviews; externally timestamp exact locked criteria.
The LBNL scenario reference 11.8% and 9.5%-15.3% range are supported by an actual report; treating its range as a probability distribution is not. Without historical error calibration, the 54% is only sensitivity arithmetic.
4 open review blockers
Fit a defensible probability distribution using historical LBNL forecast error/reliability or independent structured elicitation; do not infer confidence levels from scenario minima/maxima.
Freeze 2030 measured US data-center consumption definition, total-electricity denominator, on-site generation and power-unit consistency; select authoritative publication.
Backtest forecasts for 2024-2028 where comparable observed data exist; run alternative error models and reveal threshold sensitivity.
Two conflict-cleared independent energy systems/data statistics reviewers must sign the locked rules and external timestamp.
These requirements must be addressed, independently challenged and versioned before a new probability can count as an official locked Cortist forecast.
A-001 / INTERNAL PRE-REVIEW / HOLD
AI Agents / 2028
Verdict: The METR task-horizon methodology is real, but the two 95% and 90% conditional scenario inputs are unsourced editorial judgments. An 80%-task-success horizon is not 80% probability of future attainment.
Working model correction: 0.95 x 0.90 = 0.855 = 85.5%; standard whole-percent rounding -> 86% The displayed 86% is still a subjective, non-calibrated scenario.
Blocking findings for independent review
AGENT-EVAL / CRITICAL - Select and preregister an independent, stable 80%-reliability benchmark or validated METR successor, including task mix, sample size and rules for post-2026 discontinuation.
AGENT-REFCLASS / CRITICAL - Reconstruct comparable historical 80% task-horizon observations with time stamps and uncertainty; audit the trend using rolling-origin forecasting and holdouts.
AGENT-PUB / HIGH - Fix agent public-availability date, replication conditions, missing-data treatment and 'no evaluator by deadline' unresolved rule.
AGENT-SIGN / CRITICAL - Obtain two named external conflict-screened independent signoffs and a third-party timestamp before any official forecast is published.
Required external expertise: Independent technical AI agent evaluator familiar with METR task-horizon suite; Forecasting statistician with out-of-sample backtesting expertise.
Positive source checks: METR defines its time horizon as a human-expert task duration at 50% or 80% modeled success; it is not autonomous operating time. METR public time-horizon dashboard says it is no longer actively updated and its task distribution mainly covers software/ML/cybersecurity.
Verdict: The FDA approval outcome can be verified, but no independent public base rate supports the 65% authorization input or 70% AI-design provenance input. Phase III entry cannot substitute for phase-adjusted historical rates.
Working model correction: 0.65 x 0.70 = 0.455 = 45.5%; standard whole-percent rounding -> 46% The displayed 46% is still a subjective, non-calibrated scenario.
Blocking findings for independent review
DRUG-DESIGN / CRITICAL - Predefine substantive de novo AI design of the actual approved molecule; require patents, dated experiment trails and independent audit, not marketing phrases.
DRUG-REFCLASS / CRITICAL - Estimate eligible pipeline base rates by trial phase and disease from independently verified approval histories; address censoring and selection bias.
DRUG-FDA / HIGH - Define CDER vs CBER, NDA/BLA, accelerated approval and 2032 cutoff; resolve FDA authorization only from official records.
DRUG-SIGN / CRITICAL - Obtain two conflict-screened independent pharma/regulatory and statistical reviews; externally timestamp exact locked criteria.
Required external expertise: Independent clinical development / US FDA regulatory methods specialist; Independent medicinal chemist experienced in AI molecular design attribution.
Positive source checks: The FDA publishes dated US novel therapy approval records, which do not establish whether AI originated the molecular design. Insilico reported first dosing in a Phase III trial of rentosertib in September 2026; this is a company report, not an FDA approval. An August 2026 Nature Reviews Drug Discovery perspective documents how opaque lead optimization frustrates reliable AI design attribution.
Verdict: The LBNL scenario reference 11.8% and 9.5%-15.3% range are supported by an actual report; treating its range as a probability distribution is not. Without historical error calibration, the 54% is only sensitivity arithmetic.
Working model correction: (15.3 - 12.0)^2 / [(15.3 - 9.5) x (15.3 - 11.8)] = 53.645% -> 54% (under an imposed triangular distribution) The displayed 54% is still a subjective, non-calibrated scenario.
Blocking findings for independent review
ENERGY-DIST / CRITICAL - Fit a defensible probability distribution using historical LBNL forecast error/reliability or independent structured elicitation; do not infer confidence levels from scenario minima/maxima.
ENERGY-METHOD / HIGH - Backtest forecasts for 2024-2028 where comparable observed data exist; run alternative error models and reveal threshold sensitivity.
ENERGY-SIGN / CRITICAL - Two conflict-cleared independent energy systems/data statistics reviewers must sign the locked rules and external timestamp.
Required external expertise: Independent US electricity / data-center energy statistician; Forecast calibration methodologist with scenario-to-probability expertise.
Positive source checks: Lawrence Berkeley National Laboratory's 2025 report (published June 2026) describes 11.8% of US electricity in 2030 and a model scenario range of 9.5%-15.3%. The report uses bottom-up data-center equipment and cooling models, not a probabilistic predictive distribution.
Earlier drafts and their v0.1 data remain separately readable, even when later model displays are corrected.
TRANSPARENT NUMERICAL CORRECTION
85.5% becomes 86%; 45.5% becomes 46%
Version 0.1 used inconsistent rounding of exact half-percent model outputs. Version 0.2 uses nearest whole percentage point with halves rounded up, consistently. The AI & Energy calculation remains 54%.
None of the three models gains statistical validity through this correction. Model inputs stay editorial assumptions.
FORECAST LOCK RULE
Only a future independently approved version counts
A qualifying lock must specify the exact question, evidence, score method, source data, independent signoffs and an external immutable timestamp. Older v0.1 and v0.2 illustrative models will never be relabeled as prospectively locked forecasts or awarded retroactive Brier scores.
04 / EXTERNAL REVIEW INSTRUCTIONS
The next gate requires humans.
We need independent expertise from outside Cortist, not an additional internally generated summary.
REVIEWER BRIEF / 2 INDEPENDENT SIGNOFFS PER RULE VERSION
How an independent reviewer can contribute
Declare relevant employment, financial, consultancy and co-authorship ties. Material conflicts in the preceding 24 months disqualify independent signoff.
Inspect each exact event definition, primary source, scenario assumption, reference class, sensitivity and unresolved outcome rule.
Publish a dated, attributable review, including negative findings and suggested fixes, tied to a frozen rule version.
After independent signoffs, obtain a verified third-party cryptographic timestamp and publish a new prospective lock record. Do not retroactively convert this draft into a lock.
No reviewers have been contacted, appointed, approved these models or agreed to do so through this workflow. Cortist contact ↗ for qualified reviewer interest.
