Microsoft's diagnostic AI got 83% of difficult cases right. Patients are the harder test.

Microsoft's AI team reported a striking result on October 22, when a paper appeared in Science describing its Diagnostic Orchestrator, MAI-DxO. The design is different from a medical model answering a multiple-choice question: it requests more information, chooses tests and updates its diagnosis as details emerge.

The study covers 304 complex clinicopathological cases drawn from The New England Journal of Medicine for the AI system. Critically, the comparison with generalist physicians used a much smaller subset of 56 cases, rather than all 304. MAI-DxO scored about 83% across the 304-case benchmark; physicians using references achieved around 36% on their allocated cases, and unaided physicians scored about 22%. These are unequal evaluation sets, not a head-to-head comparison over identical 304 cases. The study also did not assess specialist teams treating live patients.

A second evaluation based on MIMIC-IV-ED emergency cases reported roughly 80% accuracy for the system versus 47% for participating physicians. Case sets and comparator conditions differ. Taken together, the numbers show strong performance on a research benchmark rather than a reliable estimate of how doctors or AI will perform across clinical settings.

MAI-DxO stages a debate among simulated medical viewpoints. It can request history, tests or additional examination details and weigh conflicting explanations before selecting a diagnosis. This tests a chain of clinical decisions, not just medical textbook recall.

The research also considers which tests the AI requests and estimates their costs. Microsoft says a cost-conscious variant maintained about 81% accuracy on complex cases while choosing less expensive investigations. But simulated workup charges are not actual medical bills. They exclude staffing, clinical follow-up, missed diagnoses, patient anxiety and the economic consequences of treatment.

The paper was produced by Microsoft researchers with company funding. Microsoft has said it will open-source components of its sequential diagnosis benchmark, which can enable scrutiny. But independent replication and live prospective evaluation remain separate milestones. The company plans to pursue clinical testing with Beth Israel Deaconess Medical Center; that does not mean a real-world safety or efficacy study has already reported results.

The tempting headline is that AI 'beat doctors'. That would leave out the most important questions: were the comparators appropriate, did the case set represent real patients, and would the system improve outcomes without unsafe decisions? The physician comparison was conducted on a smaller 56-case subset rather than the AI's full 304-case set. It was retrospective and text-based, not a controlled trial against specialist teams in real clinical conditions.

A credible next evaluation should pre-register patient populations, compare with specialist-supported clinical care, measure missed dangerous conditions and validate the complete clinical workflow. Answer-key accuracy is necessary but not sufficient for safe use. Rates of wrong tests, delays, subgroup bias and patient harms need their own evidence.

For Cortist, this is meaningful AI in Medicine research, but not evidence that an AI-designed drug won FDA approval. Our separate 2032 drug-approval candidate retains its own resolution rules and has not been upgraded by this study.