
MedGPT Beats Doctors in a Study Its Maker Scored With AI

Future Doctor says its MedGPT model produced better treatment plans than unassisted doctors in 660 of 750 comparisons, or 88 per cent, across 40 complex lung cancer cases. The Chinese company published the results itself, and Chinese media reported them last week.

And the design of the test is where the argument sits. The 250 doctors came from 98 centres across 27 provincial regions, and 159 of them are chief or associate chief physicians. Each handled three cases under three conditions, working alone, with a literature search, and with MedGPT, generating 2,250 treatment strategies.
And twenty multidisciplinary experts wrote 240 scoring rules across three rounds, including 40 safety limits on who could be treated and how aggressively, with any breach recorded as an automatic loss. Once those rules were frozen, three AI models scored the strategies without being told which came from where.
But those judges were GPT-5.5, Claude Opus 4.7 and Gemini 3.1 Pro, which are the same families as the general models MedGPT was measured against. Two of them are the exact versions in the comparison, so a judge scoring its own family is a conflict the company has not addressed, and it has not said whether the study was peer reviewed.
The numbers it reports are stark. Doctors averaged 53.40 working alone and 66.27 with MedGPT, a 24.1 per cent relative gain, while MedGPT itself scored 96.22 against 71.38 for GPT-5.4, 61.81 for Claude Opus 4.7 and 54.62 for Gemini 3.1 Pro.
And the mechanism is retrieval plus simulation rather than a single answer. Specialist agents analyse the case, the system drafts several candidate paths including a guideline based one and one drawn from recent research, checks them against about 201,000 lung cancer evidence units from roughly 13,000 documents, then simulates each path forward and drops the ones with unresolved risks.
And the limits are as important as the result. The cases were unpublished grey zone cases chosen where guidelines give no clear answer, the scoring measured written strategies rather than patient outcomes, and every figure here comes from the company that built the model.
Future Doctor, which grew out of Medlinker, says MedGPT was the first generative AI algorithm approved in China for disease diagnosis and treatment recommendations. That regulatory first is a stronger claim than the study, because it was decided by someone other than the company.
MedGPT outscored unassisted doctors in 660 of 750 comparisons
Doctors scored 53.40 alone and 66.27 with the model
Three AI judges scored the plans, two from rival model families
All figures are self reported and not peer reviewed
The study measured written strategies, not patient outcomes
Source: Pandaily


