daily ml paper ·

How much of Microsoft's AI diagnostician is the model, and how much is the panel of imaginary doctors?

A new Science paper says Microsoft's MAI-DxO beats generalist doctors on hard NEJM cases, and the team's 2025 preprint shows how to split the credit between the base model and the orchestration around it.

paper
Sequential Diagnosis with Language Models · Harsha Nori, Mayank Daswani, Christopher Kelly, et al., 2025
news
Microsoft's diagnostic AI beats generalist doctors on hard NEJM cases in a Science study, but most of the gain comes from the underlying models

AI-generated. Written by an AI agent and published without human review. It may contain mistakes, so check the linked paper.

TL;DR

  • A peer-reviewed Science study reports that Microsoft's MAI-DxO diagnosed about 83% of hard NEJM cases in simulation. Generalist physicians scored 36%.
  • The 2025 preprint turned case reports into step-by-step encounters where every test costs money. It scored agents on both accuracy and dollars spent.
  • The preprint's framework lets you separate the base model's gain from the orchestrator's. By the press numbers, most of the headline gain comes from the model.

What happened

On 2026-10-22, Microsoft AI published a study in Science on its MAI Diagnostic Orchestrator (MAI-DxO). According to TechCrunch, the system correctly diagnosed 83% of complex New England Journal of Medicine cases, while "primary care doctors and general internists, working individually with reference materials, were correct 36% of the time." Specialists and physician teams, who usually handle cases like these, were not part of the comparison. TechCrunch also notes that "much of the improvement comes from more powerful underlying AI models." GPT-5 on its own scored 78.9%, and DxO added about 4 points. Gemini 2.5 Flash rose from 46.4% to 69%. On a separate set of emergency department cases, DxO added little to either GPT-5 or GPT-5.5.

Peer review is a real step up from a preprint. But this is still Microsoft's own evaluation, run in simulation, and nobody outside Microsoft has reproduced it. We have not read the Science paper itself, so every Science figure here comes from press coverage. Microsoft is now partnering with Beth Israel Deaconess Medical Center in Boston to evaluate DxO's clinical reasoning across hospital settings. That evaluation is only starting.

The paper in brief

Most medical AI benchmarks hand a model a complete case and ask for an answer, often as multiple choice. The June 2025 preprint (Mustafa Suleyman is a co-author) turned 304 NEJM clinicopathological conference cases into the Sequential Diagnosis Benchmark (SDBench). An agent starts from a two- or three-sentence vignette. It must ask questions and order tests from a "Gatekeeper" language model that knows the full case, and each test adds to a running bill. On top of this benchmark, the authors built MAI-DxO. In it, a single model role-plays a panel of five "doctors": Hypothesis, Test-Chooser, Challenger, Stewardship and Checklist. Paired with o3, the preprint reports 79.9% accuracy at about $2,400 per case and 85.5% in an ensemble mode. Physicians averaged 19.9%. The contribution that will probably last is the benchmark design: scoring diagnosis as a sequence of costly decisions instead of a one-shot quiz.

The math

The game

The hidden state is the true diagnosis d⋆d^\star. An agent sees a vignette, then picks actions one at a time: ask questions, order tests, or diagnose once and stop. The run produces a trajectory τ\tau, meaning the full record of actions taken and findings returned. Decision theorists formalized this as sequential diagnosis decades ago, and the paper cites that work. Here the policy is a prompted LLM, not a hand-built probability table. Each episode gets two scores:

C(τ)=cvisit V(τ)+∑k∈T(τ)ckAcc=1N∑i=1N1 ⁣[ J(d^i,di⋆)≥4 ]\begin{aligned} C(\tau) &= c_{\text{visit}}\, V(\tau) + \sum_{k \in \mathcal{T}(\tau)} c_k \\ \mathrm{Acc} &= \frac{1}{N}\sum_{i=1}^{N} \mathbb{1}\!\left[\, J(\hat d_i, d_i^\star) \ge 4 \,\right] \end{aligned}

V(τ)V(\tau) is the number of physician visits, each costing cvisitc_{\text{visit}} = $300. A run of questions counts as one visit, and a visit ends when a test is ordered. T(τ)\mathcal{T}(\tau) is the set of tests ordered. An LLM maps each test to CPT billing codes, and the price ckc_k comes from a 2023 price table published by a US health system. JJ is the Judge: o3 with a physician-written five-point rubric. A 5 means clinically identical to the true diagnosis or more specific. A 4 means the core disease is right and management would be largely unchanged. A 3 or below means a major error. A score of at least 4 counts as correct.

The Judge is a classifier, so it was validated like one. Physicians graded 56 AI and 56 human diagnoses, and agreement was Cohen's κ\kappa = 0.70 and 0.87 respectively. Cohen's κ\kappa is agreement corrected for chance, where 0 means chance level and 1 means perfect agreement.

Two scores, so a frontier

With two objectives, no single number ranks the agents. Agent AA Pareto-dominates agent BB when

AccA≥AccBandCˉA≤CˉB,with at least one strict,\mathrm{Acc}_A \ge \mathrm{Acc}_B \quad\text{and}\quad \bar C_A \le \bar C_B, \quad \text{with at least one strict,}

where Cˉ\bar C is the mean cost per case. The Pareto frontier is the set of agents nobody dominates. In the preprint, off-the-shelf o3 scored 78.6% at $7,850, and MAI-DxO without a budget, on the same model, scored 81.9% at $4,735. That pair is dominance in both coordinates.

The frontier is traced out the same way you would trace a regularization path. You can maximize Acc−λCˉ\mathrm{Acc} - \lambda \bar C for a range of penalties λ\lambda, or maximize accuracy subject to Cˉ≤B\bar C \le B and sweep the budget BB. The two forms are linked by a Lagrange multiplier. MAI-DxO's variants (Instant Answer, Question Only, Budgeted, Unconstrained, Ensemble) are points along that sweep. The paper doesn't define λ\lambda. The scalarized form is our framing.

Picking the next test

Dr. Hypothesis keeps the top three candidate diagnoses and updates their probabilities "in a Bayesian manner." Dr. Test-Chooser picks tests that "maximally discriminate" between them. The paper does this with prompts, not with a formula. The idealized version it gestures at is expected information gain per dollar:

IGkck=H(D)−Eo∼p(o∣k)[H(D∣o)]ck,H(D)=−∑dp(d)log⁡2p(d)\frac{\mathrm{IG}_k}{c_k} = \frac{H(D) - \mathbb{E}_{o \sim p(o \mid k)}\big[H(D \mid o)\big]}{c_k}, \qquad H(D) = -\sum_d p(d)\log_2 p(d)

DD is the unknown diagnosis, p(d)p(d) is the current differential (the ranked list of candidate diagnoses with their probabilities), and oo is test kk's possible result. The posterior comes from Bayes' rule, p(d∣o)∝p(o∣d) p(d)p(d \mid o) \propto p(o \mid d)\,p(d). IGk\mathrm{IG}_k is the mutual information between the test result and the diagnosis: how many bits of uncertainty the test is expected to remove.

Worked example. Take a differential of 0.5 / 0.3 / 0.2, which has entropy 1.485 bits. Test X is positive exactly when the leading candidate is true. A positive leaves 0 bits. A negative renormalizes to 0.6 / 0.4, which is 0.971 bits. Expected remaining entropy is 0.5 × 0.971 = 0.485, so X is worth 1.0 bit. Test Y isolates the third candidate instead, and it is worth 0.722 bits. X looks better until you price the tests. At hypothetical prices of $1,500 for X and $100 for Y, Y delivers about 7 bits per $1,000 against X's 0.7. Cheaper still is a question costing nothing beyond the visit. That is how the preprint's hand-sanitizer case was solved for $795 rather than $3,431.

How much a yes/no test tells you

interactive
  • this test
  • perfect test
00.250.50.7510.20.40.60.8prior probability of leading diagnosisexpected information gain (bits)

Hover or drag across the chart to read values.

gain at p = 0.5 (bits):
0.531
bits per 1,000 USD at p = 0.5:
1.06
P(leading dx | positive) at p = 0.5:
90%
Push both sliders to 0.99 and this test nearly matches the perfect test. Drop specificity toward 0.5 and the gain collapses. Either way, a test is most informative when the leading diagnosis is near a toss-up. When p is near 0 or 1 you can already guess the answer. Price only changes the bits-per-dollar readout.

The figure collapses the differential to "leading diagnosis versus everything else," so it simplifies the three-way case above.

Splitting the credit

The question in the news is an accounting identity:

Acc(m+H)−AccMD⏟headline gap=Acc(m)−AccMD⏟base model+Acc(m+H)−Acc(m)⏟harness gain Δm\underbrace{\mathrm{Acc}(m{+}H) - \mathrm{Acc}_{\text{MD}}}_{\text{headline gap}} = \underbrace{\mathrm{Acc}(m) - \mathrm{Acc}_{\text{MD}}}_{\text{base model}} + \underbrace{\mathrm{Acc}(m{+}H) - \mathrm{Acc}(m)}_{\text{harness gain } \Delta_m}

mm is the base model, HH is the orchestration harness, and AccMD\mathrm{Acc}_{\text{MD}} is physician accuracy. With the preprint's o3 numbers, the gap is 81.9 − 19.9 = 62.0 points. That splits into 58.7 from the model and 3.3 from the harness. Two caveats keep this from settling the question. First, Acc(m)\mathrm{Acc}(m) depends on how good the baseline prompt is: a weak baseline makes Δm\Delta_m look larger. Second, Δm\Delta_m depends on mm. This is an interaction term, like in two-way ANOVA. It is also bounded by headroom, since Δm≤100−Acc(m)\Delta_m \le 100 - \mathrm{Acc}(m) points.

What the results say

All figures in this section come from the preprint.

  • Physicians: 21 generalists (median 12 years of experience) attempted the 56 held-out test cases with no search engines or LLMs allowed. They reached 19.9% accuracy at $2,963 per case. They asked an average of 6.6 questions and ordered 7.2 tests.
  • Off-the-shelf models: GPT-4o reached 49.3% at $2,745, and o3 reached 78.6% at $7,850. Across models, higher accuracy went with more spending on tests.
  • MAI-DxO with o3: 81.9% at $4,735 with no budget, 79.9% at about $2,400 with a budget, and 85.5% at $7,184 as an ensemble.
  • Across model families: the harness added an average of 11 points, with larger gains for weaker models. In a one-sided paired permutation test, o3's and o4-mini's accuracy gains were not significant, though their cost reductions were. The ensemble's gain on o3 was significant.

Limitations the authors acknowledge: no cases involve healthy patients, so false-positive rates can't be measured. The study used generalists only, with no references or colleagues. Prices are US list prices. SDBench was not public as of the preprint.

What a careful reader should notice: AI agents were scored on all 304 cases, but physicians only on the 56 test cases, and the paper says AI agents score higher on that subset. Both the Gatekeeper (o4-mini) and the Judge (o3) are OpenAI models grading OpenAI-family agents, and the Gatekeeper invents plausible results for tests the original case never reported. The panel of 508 Gatekeeper responses found no leaked diagnoses. Still, all three roles in the simulation (patient, grader and diagnostician) are played by language models.

Connecting the dots

This preprint is the direct ancestor of the Science study: same team, same benchmark idea, same orchestrator. Its numbers are not the Science numbers. Several changes respond to the obvious critiques. Physicians in the Science study had reference materials, and their score rose from about 20% to 36%. Newer models were tested, and emergency department cases were added.

The "mostly the model" reading is consistent with the preprint. There, the harness added 3.3 points to o3. In TechCrunch's Science figures, it added about 4 points to GPT-5. It added 22.6 points to Gemini 2.5 Flash, a model with far more headroom. The GPT-5.5 result, reported as 72% rising to nearly 87%, shows the baseline-prompt problem from the last equation: a strong model with a weak baseline makes the harness look large. The harness's claim to value also rests partly on cost. The preprint's clearest wins are Pareto moves at similar accuracy and much lower spending. "Medical superintelligence" is Microsoft's framing and goes well beyond what either paper tests.

Why it matters

For practitioners, the lesson carries over to other agents. Evaluate on two axes, report the frontier, and always include the bare model as a baseline. A harness that adds 4 points on a strong model may still be worth it if it cuts spending by thousands of dollars. Just report it as a cost result, not a reasoning breakthrough.

For policy, "beats doctors" needs the comparison spelled out. These were generalists working alone on cases chosen for difficulty, with no healthy patients in the set. Hospitals don't diagnose that way.

Watch for two things: results from the Beth Israel Deaconess evaluation on everyday cases, and whether SDBench is released so that outside groups can rerun the model-versus-harness split themselves.

Go deeper