Microsoft announces release of SDBench code for evaluating AI medical diagnosis

On October 22, 2026, Microsoft announced the release of the SDBench code, a tool that evaluates medical diagnosis by artificial intelligence. The MIT-licensed repository includes the Gatekeeper simulator, a judge, and a cost estimation module.

The work on sequential diagnosis is published on the same day in the journal Science. Microsoft’s medical system, MAI-DxO, and a first preprint already existed in June 2025.

Records transformed into simulations

SDBench transforms historical clinical records into interactive text-based scenarios. The agent being tested receives a short presentation that conceals the diagnosis, then asks questions or requests tests before reaching a conclusion.

Gatekeeper responds using the record. If requested information is missing, it can synthesize a plausible version, as explained in the authors’ manuscript. Its absence from the record does not mean it was never observed in the patient.

A judge compares the proposed diagnosis with a reference, while a module estimates the cost of tests and consultations. MAI-DxO, for its part, organizes several virtual roles around a language model. Microsoft’s announcement concerns the release of the SDBench code.

Scores to read in light of their protocol

In the manuscript, MAI-DxO paired with GPT-5 achieves 83.2% of diagnoses judged correct across 304 complex cases from the New England Journal of Medicine. Nineteen general practitioners with access to reference materials achieve an average of 36.1% on diagnoses drawn from a subset of 56 recent cases.

The scores cover different sets of cases, and the doctors did not all handle each of the 56 cases. The comparison is therefore imperfectly matched. Selected for their complexity, these cases do not represent routine consultations.

The general practitioners worked alone in a text-based simulation, and the study does not compare them with specialists. The system did not see the patients or perform physical examinations. The results demonstrate neither superiority in real-world consultations nor an improvement in patients’ health.

The manuscript estimates the cost of tests and consultations at 5,430 dollars per case for the configuration achieving 83.2%, compared with 4,725 dollars for the doctors with access to reference materials. Another MAI-DxO configuration achieves 80.9%, at an estimated cost of 2,241 dollars per case. These amounts do not measure savings achieved in hospitals.

Clinical records subject to conditions

The code release provides access to the tools, but the New England Journal of Medicine cases require a license. The repository also contains five entirely synthetic cases for developing the tool, without measuring its clinical performance.

On October 22, PhysioNet published the first version of a set of 488 MIMIC hospital records adapted for SDBench. Access requires PhysioNet credentialing, CITI training, and a signed data use agreement.

Microsoft also announces a prospective study with Beth Israel Deaconess Medical Center. Patients would interact with the system, and its assessments would be compared with those of doctors. The announcement does not yet report results for this study.