Do agents retain the reported teamwork weaknesses when the Collaborative Intelligence benchmark is reproduced independently?
The November 2 arXiv preprint introduces a Collaborative Intelligence Benchmark with 240 tasks spanning eight dimensions of teamwork, evaluating 18 model families. The authors report the best score as 82.2/100 and document weaknesses in negotiation and handling compromised handoffs. They describe 160 public tasks and 80 held-out tasks.
What the primary sources establish
The November 2 arXiv preprint introduces a Collaborative Intelligence Benchmark with 240 tasks spanning eight dimensions of teamwork, evaluating 18 model families. The authors report the best score as 82.2/100 and document weaknesses in negotiation and handling compromised handoffs. They describe 160 public tasks and 80 held-out tasks.
What cannot yet be concluded
Evidence limit: The preprint is authored evaluation, not a peer-reviewed, independent replication. The selected scenarios and programmatic grader are not equivalent to actual workplace performance or a population-wide safety rate. Held-out material is not fully open to third-party reproduction without suitable access.
Category: independent replication. A reported benchmark or demonstration is not independent reproduction or real-world validation.
Dated evidence history
Authors release 240-task/8-dimension research and report the strongest model score of 82.2.
External generalization and independently reproduced results remain to be demonstrated.
Inspect the primary sources
Original publisher rights are retained. URLs are source attributions, not third-party audits of each claim.
Trace what happens next
The Evidence Graph timeline connects this new case to its original sources, topic and any corresponding Cortist article. Future material changes require a new dated version, not edits to this baseline.