Microsoft’s MAI-DxO scored higher in two diagnosis benchmarks

Microsoft researchers reported that MAI-DxO scored 83.2% on New England Journal of Medicine (NEJM) cases and 80.3% on MIMIC-IV-ED cases in a text-based diagnostic benchmark published in Science on October 22, 2026. Participating physicians averaged 36.1% and 46.8%, respectively. The NEJM figures came from evaluations of different sizes: the AI result covered 304 cases, while the physician evaluation used 56 held-out cases.

How the benchmark worked

The Sequential Diagnosis Benchmark (SDBench) turned 304 NEJM case discussions and 325 emergency-department encounters from MIMIC-IV-ED into interactive, text-based simulations. Rather than receiving every detail at once, a diagnostic agent could ask questions, request test results, revise its possible diagnoses and submit a final answer.

MAI-DxO, short for Microsoft AI Diagnostic Orchestrator, coordinated five simulated clinical roles: Dr. Hypothesis, Dr. Test-Chooser, Dr. Challenger, Dr. Stewardship and Dr. Checklist. A Gatekeeper supplied information from the case records. If a requested detail was absent, it could return a synthetic finding designed not to point toward a diagnosis.

A physician-authored rubric scored the final diagnosis on a five-point scale. Scores of 4 or 5 counted as correct because the researchers defined them as diagnoses that would not change management. The main reported results used GPT-5.

Microsoft’s demonstration follows one SDBench case as the system asks questions, orders tests, tracks estimated costs and updates its diagnostic possibilities.

The accuracy results and their comparison basis

The NEJM percentages do not come from equal-sized evaluations: researchers evaluated MAI-DxO across all 304 cases, while the physician results came from 56 held-out cases. For the NEJM physician comparison, participants could consult static reference materials but not generative AI. The MIMIC-IV-ED results compare MAI-DxO with participating generalist physicians.

The results measure diagnoses in these benchmark settings, not performance against specialist teams or the quality of patient care. The authors say their comparisons do not establish superiority over specialists, multidisciplinary teams or real-world clinical practice.

Estimated costs varied by configuration

The study reported several accuracy-and-cost points for MAI-DxO on the NEJM cases. Higher accuracy did not always come with a lower estimated cost.

These are modeled estimates in US dollars, not patient bills. The researchers matched tests to procedure codes and used a 2023 Kaiser Permanente Washington standard-charges file; they also assigned a fixed $300 physician-visit cost to history and examination questions. The estimate does not capture real-world variation in prices, test availability, time or other costs.

What the benchmark leaves outside the picture

SDBench focused on pathological cases and text-based diagnostic reasoning. It did not include healthy or benign cases, the uncertainty of ordinary patient conversations, physical examinations in clinical conditions, imaging and other multimodal findings, treatment safety or long-term outcomes. The physician comparison involved individuals, not specialist teams.

The paper also says researchers held out the 56 most recent NEJM cases, published in 2024–2025, from orchestration development. Most appeared after the evaluated models’ training cutoffs, while the authors acknowledge that some publicly available NEJM cases might have appeared in model training.

The study authors disclosed that they were Microsoft employees who held company equity, and that the research received no funding from outside Microsoft.

Microsoft’s planned hospital evaluation

On October 22, 2026, Microsoft announced a prospective evaluation with Beth Israel Deaconess Medical Center in Boston. The announced plan calls for patients to interact directly with the system and for its performance to be compared with physicians’ independent assessments.