Microsoft's AI Outdiagnosed Doctors in a Peer-Reviewed Study — Here's What It Actually Proves

 

Microsoft says its artificial intelligence can now diagnose difficult medical cases better than human doctors — and this time the claim comes with peer review. A study published Thursday in the journal Science reports that the company's MAI Diagnostic Orchestrator outperformed generalist physicians on more than 300 unusually tricky cases, in some measures by a factor of two or more. It is the strongest evidence yet for AI in the clinic. It is also, by Microsoft's own admission, not the evidence that matters most.

What the study actually tested

The system at the centre of the paper — Nori et al., "Sequential diagnosis with language models" — is not a chatbot that blurts out an answer. Most medical AI takes an all-at-once approach: feed it every symptom, lab result and scan, and it generates a diagnosis. The Diagnostic Orchestrator instead works the way a good doctor does. A coordinating "orchestrator" model assembles a simulated panel of AI specialists that decide, step by step, what to ask about the patient's history, what the physical exam suggests, which tests to order, and how to revise its hypotheses as results come in.

According to Reuters, the researchers pitted the system against practising general physicians — who were allowed textbooks and the internet — on a battery of more than 300 highly unusual and confusing medical dilemmas drawn from published case reports and a large database of emergency-room admissions. The AI was more than twice as accurate at "working up" the patient and reaching the right diagnosis. Reporting on The Verge's coverage puts the figures at roughly 83% accuracy on the journal cases and 80% on the emergency-department cases, against about 47% for the physicians on the latter — with the AI also ordering fewer tests and reaching its answers at a lower estimated cost.

The numbers — and what they don't mean

Those figures deserve careful handling, because the study's design has real limits. The cases were compiled by Microsoft itself into a benchmark called SDBench (which the company says it will open-source), the physicians it beat were generalists working alone on cases deliberately chosen to be hard, and there has been no independent verification yet of the case mix or the sample of doctors involved. A written case vignette is not a patient: no messy histories, no conflicting test results, no one in pain in front of you.

Microsoft's own leadership is strikingly candid about this. Mustafa Suleyman, CEO of Microsoft AI, told The Verge that "the technology is finally here" — and in the same breath that "it needs clinical validation. That is a leap, but this is a major milestone along that journey." The paper itself is the peer-reviewed version of work the team first posted as a non-reviewed preprint in June 2025; the published version runs on GPT-5 and adds the emergency-department cases. Peer review upgrades the claim from marketing to science. It does not upgrade it to medicine.

What happens next

The real test is already being arranged. Microsoft plans a study with Beth Israel Deaconess Medical Center in which consenting patients receive normal care while also interacting with the system, with its recommendations compared against their doctors' independent assessments — expected to get underway within months, according to Microsoft AI health chief Dominic King. The company is also working with Mayo Clinic on a healthcare foundation model, and Britain has agreed to bring AI into NHS hospitals sooner, so the clinical pipeline is filling on both sides of the Atlantic.

Seen in that light, the Science paper is best read as a starting gun, not a finish line: proof that an AI can reason through a hard case sequentially, cheaply, and — on paper — better than a lone generalist. Whether it helps real patients will be decided in hospitals, not benchmarks. Until then, doctors stay in charge, and the machine stays a very promising assistant. For the other side of this debate, see our report on the study finding patients distrust AI medical advice, and for more of AI's medical frontier, our explainer on the AI that predicts cancer survival from a selfie.