Do agents retain the reported teamwork weaknesses when the Collaborative Intelligence benchmark is reproduced independently?
Sean Wu and collaborating researchers
This is a first-record evidence baseline. It does not establish independent verification of every research result or completion of future commitments.
Claim and evidence
The November 2 arXiv preprint introduces a Collaborative Intelligence Benchmark with 240 tasks spanning eight dimensions of teamwork, evaluating 18 model families. The authors report the best score as 82.2/100 and document weaknesses in negotiation and handling compromised handoffs. They describe 160 public tasks and 80 held-out tasks.
Limitations and uncertainty
The preprint is authored evaluation, not a peer-reviewed, independent replication. The selected scenarios and programmatic grader are not equivalent to actual workplace performance or a population-wide safety rate. Held-out material is not fully open to third-party reproduction without suitable access.
Explicit acceptance criteria
- Reproduce the reported scores on the published 160-task subset.
- Test robustness to agent and model updates using unchanged grading rules.
- Assess whether controlled teamwork scores predict outcomes in external multi-agent projects.
Dated baseline events
The entries below capture original source and publication milestones. They do not imply that later claims have already been fulfilled or independently replicated.