Microsoft AI Beats Doctors in Complex Case Study

The race to integrate artificial intelligence into daily clinical practice has intensified as major technology firms claim their models can rival human expertise. This week, Microsoft released findings showing its diagnostic system significantly outperformed generalist doctors in a controlled simulation of complex medical cases.

The system, known as MAI-DxO, functions by coordinating multiple AI agents that mimic a panel of physicians. These agents collaborate to question patients, order necessary tests, and debate potential diagnoses. In simulations using difficult cases from the New England Journal of Medicine, the system achieved an 83% accuracy rate. In comparison, primary care physicians and general internists working individually with reference materials were correct only 36% of the time.

While the results are striking, the comparison excludes specialists or physician teams, who typically handle such complex cases. Microsoft’s vice president of Health, Dominic King, described the findings as substantial, noting that the growing mismatch between medical demand and supply is a critical challenge. The study suggests that much of the performance boost comes from using more powerful underlying models, with Microsoft’s orchestration adding further gains.

Microsoft tested its system with various AI models from companies including OpenAI, Google, and Anthropic, finding the best results with OpenAI’s GPT-5 and GPT-5.5. The data indicates that the orchestration layer helps most when using less capable models or tackling complex cases, though it offers limited advantages for emergency department scenarios or when using the most advanced models. The system also reduced costs per case compared to running the top-tier models alone.

Mustafa Suleyman, Microsoft AI’s CEO, views this as a step toward "medical superintelligence," a concept defined as surpassing the strongest human-led health systems in reasoning and coordination. He stated that the company is partnering with the Mayo Clinic to train a specialized foundation model for healthcare, with a release planned for next year. Suleyman predicts that by 2027, the technology will enable accurate predictions of how medical records might unfold, providing clinicians with valuable foresight.

To validate these capabilities in a real-world setting, Microsoft has partnered with Beth Israel Deaconess Medical Center in Boston. The study will evaluate the system’s clinical reasoning across different hospital environments, including emergency and urgent care settings. Patients will receive standard treatment while consenting to parallel engagement with the AI, allowing researchers to compare its recommendations against those of human doctors.

Suleyman, who previously co-founded DeepMind, believes generative AI in healthcare is in its early stages but expects significant capability improvements within the next three years. He noted that progress accelerates once a "flywheel" of real-world usage begins to identify and patch model gaps. Microsoft aims to make this clinical decision support technology a key differentiator, building on its existing tools for dictation and radiology.