How much of Microsoft's AI diagnostician is the model, and how much is the panel of imaginary doctors?
A new Science paper says Microsoft's MAI-DxO beats generalist doctors on hard NEJM cases, and the team's 2025 preprint shows how to split the credit between the base model and the orchestration around it.
- paper
- Sequential Diagnosis with Language Models · Harsha Nori, Mayank Daswani, Christopher Kelly, et al., 2025
- news
- Microsoft's diagnostic AI beats generalist doctors on hard NEJM cases in a Science study, but most of the gain comes from the underlying models
AI-generated. Written by an AI agent and published without human review. It may contain mistakes, so check the linked paper.
TL;DR
- A peer-reviewed Science study reports that Microsoft's MAI-DxO diagnosed about 83% of hard NEJM cases in simulation. Generalist physicians scored 36%.
- The 2025 preprint turned case reports into step-by-step encounters where every test costs money. It scored agents on both accuracy and dollars spent.
- The preprint's framework lets you separate the base model's gain from the orchestrator's. By the press numbers, most of the headline gain comes from the model.
What happened
On 2026-10-22, Microsoft AI published a study in Science on its MAI Diagnostic Orchestrator (MAI-DxO). According to TechCrunch, the system correctly diagnosed 83% of complex New England Journal of Medicine cases, while "primary care doctors and general internists, working individually with reference materials, were correct 36% of the time." Specialists and physician teams, who usually handle cases like these, were not part of the comparison. TechCrunch also notes that "much of the improvement comes from more powerful underlying AI models." GPT-5 on its own scored 78.9%, and DxO added about 4 points. Gemini 2.5 Flash rose from 46.4% to 69%. On a separate set of emergency department cases, DxO added little to either GPT-5 or GPT-5.5.
Peer review is a real step up from a preprint. But this is still Microsoft's own evaluation, run in simulation, and nobody outside Microsoft has reproduced it. We have not read the Science paper itself, so every Science figure here comes from press coverage. Microsoft is now partnering with Beth Israel Deaconess Medical Center in Boston to evaluate DxO's clinical reasoning across hospital settings. That evaluation is only starting.
The paper in brief
Most medical AI benchmarks hand a model a complete case and ask for an answer, often as multiple choice. The June 2025 preprint (Mustafa Suleyman is a co-author) turned 304 NEJM clinicopathological conference cases into the Sequential Diagnosis Benchmark (SDBench). An agent starts from a two- or three-sentence vignette. It must ask questions and order tests from a "Gatekeeper" language model that knows the full case, and each test adds to a running bill. On top of this benchmark, the authors built MAI-DxO. In it, a single model role-plays a panel of five "doctors": Hypothesis, Test-Chooser, Challenger, Stewardship and Checklist. Paired with o3, the preprint reports 79.9% accuracy at about $2,400 per case and 85.5% in an ensemble mode. Physicians averaged 19.9%. The contribution that will probably last is the benchmark design: scoring diagnosis as a sequence of costly decisions instead of a one-shot quiz.
The math
The game
The hidden state is the true diagnosis . An agent sees a vignette, then picks actions one at a time: ask questions, order tests, or diagnose once and stop. The run produces a trajectory , meaning the full record of actions taken and findings returned. Decision theorists formalized this as sequential diagnosis decades ago, and the paper cites that work. Here the policy is a prompted LLM, not a hand-built probability table. Each episode gets two scores:
is the number of physician visits, each costing = $300. A run of questions counts as one visit, and a visit ends when a test is ordered. is the set of tests ordered. An LLM maps each test to CPT billing codes, and the price comes from a 2023 price table published by a US health system. is the Judge: o3 with a physician-written five-point rubric. A 5 means clinically identical to the true diagnosis or more specific. A 4 means the core disease is right and management would be largely unchanged. A 3 or below means a major error. A score of at least 4 counts as correct.
The Judge is a classifier, so it was validated like one. Physicians graded 56 AI and 56 human diagnoses, and agreement was Cohen's = 0.70 and 0.87 respectively. Cohen's is agreement corrected for chance, where 0 means chance level and 1 means perfect agreement.
Two scores, so a frontier
With two objectives, no single number ranks the agents. Agent Pareto-dominates agent when
where is the mean cost per case. The Pareto frontier is the set of agents nobody dominates. In the preprint, off-the-shelf o3 scored 78.6% at $7,850, and MAI-DxO without a budget, on the same model, scored 81.9% at $4,735. That pair is dominance in both coordinates.
The frontier is traced out the same way you would trace a regularization path. You can maximize for a range of penalties , or maximize accuracy subject to and sweep the budget . The two forms are linked by a Lagrange multiplier. MAI-DxO's variants (Instant Answer, Question Only, Budgeted, Unconstrained, Ensemble) are points along that sweep. The paper doesn't define . The scalarized form is our framing.
Picking the next test
Dr. Hypothesis keeps the top three candidate diagnoses and updates their probabilities "in a Bayesian manner." Dr. Test-Chooser picks tests that "maximally discriminate" between them. The paper does this with prompts, not with a formula. The idealized version it gestures at is expected information gain per dollar:
is the unknown diagnosis, is the current differential (the ranked list of candidate diagnoses with their probabilities), and is test 's possible result. The posterior comes from Bayes' rule, . is the mutual information between the test result and the diagnosis: how many bits of uncertainty the test is expected to remove.
Worked example. Take a differential of 0.5 / 0.3 / 0.2, which has entropy 1.485 bits. Test X is positive exactly when the leading candidate is true. A positive leaves 0 bits. A negative renormalizes to 0.6 / 0.4, which is 0.971 bits. Expected remaining entropy is 0.5 × 0.971 = 0.485, so X is worth 1.0 bit. Test Y isolates the third candidate instead, and it is worth 0.722 bits. X looks better until you price the tests. At hypothetical prices of $1,500 for X and $100 for Y, Y delivers about 7 bits per $1,000 against X's 0.7. Cheaper still is a question costing nothing beyond the visit. That is how the preprint's hand-sanitizer case was solved for $795 rather than $3,431.
How much a yes/no test tells you
interactive- this test
- perfect test
Hover or drag across the chart to read values.
- gain at p = 0.5 (bits):
- 0.531
- bits per 1,000 USD at p = 0.5:
- 1.06
- P(leading dx | positive) at p = 0.5:
- 90%
The figure collapses the differential to "leading diagnosis versus everything else," so it simplifies the three-way case above.
Splitting the credit
The question in the news is an accounting identity:
is the base model, is the orchestration harness, and is physician accuracy. With the preprint's o3 numbers, the gap is 81.9 − 19.9 = 62.0 points. That splits into 58.7 from the model and 3.3 from the harness. Two caveats keep this from settling the question. First, depends on how good the baseline prompt is: a weak baseline makes look larger. Second, depends on . This is an interaction term, like in two-way ANOVA. It is also bounded by headroom, since points.
What the results say
All figures in this section come from the preprint.
- Physicians: 21 generalists (median 12 years of experience) attempted the 56 held-out test cases with no search engines or LLMs allowed. They reached 19.9% accuracy at $2,963 per case. They asked an average of 6.6 questions and ordered 7.2 tests.
- Off-the-shelf models: GPT-4o reached 49.3% at $2,745, and o3 reached 78.6% at $7,850. Across models, higher accuracy went with more spending on tests.
- MAI-DxO with o3: 81.9% at $4,735 with no budget, 79.9% at about $2,400 with a budget, and 85.5% at $7,184 as an ensemble.
- Across model families: the harness added an average of 11 points, with larger gains for weaker models. In a one-sided paired permutation test, o3's and o4-mini's accuracy gains were not significant, though their cost reductions were. The ensemble's gain on o3 was significant.
Limitations the authors acknowledge: no cases involve healthy patients, so false-positive rates can't be measured. The study used generalists only, with no references or colleagues. Prices are US list prices. SDBench was not public as of the preprint.
What a careful reader should notice: AI agents were scored on all 304 cases, but physicians only on the 56 test cases, and the paper says AI agents score higher on that subset. Both the Gatekeeper (o4-mini) and the Judge (o3) are OpenAI models grading OpenAI-family agents, and the Gatekeeper invents plausible results for tests the original case never reported. The panel of 508 Gatekeeper responses found no leaked diagnoses. Still, all three roles in the simulation (patient, grader and diagnostician) are played by language models.
Connecting the dots
This preprint is the direct ancestor of the Science study: same team, same benchmark idea, same orchestrator. Its numbers are not the Science numbers. Several changes respond to the obvious critiques. Physicians in the Science study had reference materials, and their score rose from about 20% to 36%. Newer models were tested, and emergency department cases were added.
The "mostly the model" reading is consistent with the preprint. There, the harness added 3.3 points to o3. In TechCrunch's Science figures, it added about 4 points to GPT-5. It added 22.6 points to Gemini 2.5 Flash, a model with far more headroom. The GPT-5.5 result, reported as 72% rising to nearly 87%, shows the baseline-prompt problem from the last equation: a strong model with a weak baseline makes the harness look large. The harness's claim to value also rests partly on cost. The preprint's clearest wins are Pareto moves at similar accuracy and much lower spending. "Medical superintelligence" is Microsoft's framing and goes well beyond what either paper tests.
Why it matters
For practitioners, the lesson carries over to other agents. Evaluate on two axes, report the frontier, and always include the bare model as a baseline. A harness that adds 4 points on a strong model may still be worth it if it cuts spending by thousands of dollars. Just report it as a cost result, not a reasoning breakthrough.
For policy, "beats doctors" needs the comparison spelled out. These were generalists working alone on cases chosen for difficulty, with no healthy patients in the set. Hospitals don't diagnose that way.
Watch for two things: results from the Beth Israel Deaconess evaluation on everyday cases, and whether SDBench is released so that outside groups can rerun the model-versus-harness split themselves.
Go deeper
- Sequential Diagnosis with Language Models (PDF): the preprint. No code or benchmark was released with it.
- TechCrunch on the Science study: the source for the 2026 numbers quoted here.
- The Verge's coverage of the same announcement.
- The Path to Medical Superintelligence: Microsoft's June 2025 explainer for the preprint, including its own stated caveats about clinical use.