- PaperBenchX introduces 93 expert-authored experiment-reproduction tasks across 12 research areas, with domain-native tools spanning photonics, chemistry, materials, biology and robotics.
- Ten frontier agent configurations were evaluated. The strongest fully reproduced 13 of 93 tasks (about 14%) under a strict greater-than-90% rubric threshold; 71 tasks were not fully reproduced by any tested configuration.
- The scoring protocol replays generated workflows from scratch and evaluates regenerated scientific evidence, not merely plausible reports or code that ran once. Only twelve public demonstration packages are in the linked repository; the full benchmark consists of 93 tasks.
The Signal
Scientific discovery needs more than a fluent summary of an experiment. PaperBenchX, released as a preprint on 29 October 2026, asks whether coding agents can reconstruct and execute published experiments using the tools and assumptions of the original scientific fields. The 93 tasks span 12 research areas across electromagnetics, photonics, chemistry, materials science, computational biology and robotics.
Researchers evaluated ten frontier agent configurations. The strongest fully reproduced just 13 of 93 tasks, or approximately 14%, under the paper's strictly greater-than-90-percent full-reproduction rule. Across configurations, average partial rubric scores ranged from 27.53% to 59.08%. Those partial numbers cannot be interpreted as completed replications: agents often earned credit for some valid steps without reproducing the final scientific evidence to the required standard.
The paper further reports that 71 of the 93 tasks were not fully reproduced by any tested configuration. This does not mean 71 tasks are intrinsically impossible for AI systems; it means none of the evaluated systems completed them in this study under its resources, tasks and scoring procedure.
The Shift
The methodological distinction is what makes the dataset relevant beyond a benchmark leaderboard. An agent receives a source paper, a task specification and access to the scientific computing environment. It must produce a runnable workflow. Evaluators then replay that submitted workflow in a clean environment and assess only regenerated evidence against expert-defined criteria. A convincing explanation or a near-matching plotted number is not sufficient when the underlying physical model is wrong.
Domain-native software matters. Photonics experiments may need electromagnetic solvers and careful boundary conditions; chemistry tasks depend on basis sets, geometry and convergence; embodied tasks run in robotics environments. Many reproducibility failures are therefore failures of scientific modeling or verification, not just of text reasoning.
Readers should not confuse the number of private benchmark tasks with the amount of publicly released task material. The associated public project presents twelve demonstration task packages and a documented evaluation protocol, while the paper reports results for all 93 expert-defined tasks. The researchers' evaluation is methodologically explicit, but this article does not claim Cortist independently reran the benchmark or revalidated the reported agent traces.
The Cortist Lens
Cortist's F-006 AI-and-science forecast asks for a novel, independently reproduced autonomous discovery by 2035. PaperBenchX instead measures whether agents can reproduce known published experiments. This is a demanding prerequisite and a useful evidence signal, but it does not test the F-006 outcome directly. Repeating an existing experiment is not automatically making an original breakthrough.
A future proof point would be independent re-execution of specific agent-generated workflows, with experiment parameters, replay logs, source versions, compute budgets and failure modes available for audit. Such records could move claims from presentation to verifiable evidence. Until then, the provisional forecast and original Ledger remain unchanged.
Independent benchmark replications, publicly inspectable complete task packages, domain-level failure distributions, repeated seeds and independently demonstrated novel discoveries.
Cross-domain science remains a harder test than producing a plausible explanation: 13 full reproductions in 93 tasks is real partial progress, not scientific autonomy achieved.
Primary sources and evidence limits
Original preprint published 29 October 2026; evaluator-designed benchmark results are not independent replication of every scientific task. Cortist reviewed original research or SEC documentation; we have not independently rerun the experiments or audited the financial statements. This article does not score or change any provisional forecast. Read the editorial standards.
PaperBenchX original researchRuoyu Wu and colleagues; arXiv preprint, 29 October 2026PaperBenchX research repositoryPublic demonstration tasks and evaluator documentation; twelve public packages vs 93 benchmark tasks