An AI alone outscored doctors using it

Researchers at Stanford University and collaborating institutions tested that assumption directly, randomizing 50 physicians to diagnose difficult cases with GPT-4 or with conventional resources, and also scoring GPT-4 working entirely on its own.

GPT-4 alone scored about 92 percent, while physicians using conventional resources scored 74 percent and physicians given access to GPT-4 scored only marginally higher, around 76 percent. The model outperformed the doctors who were using it as a tool.

The gap suggests handing a powerful model to a clinician is not enough: the authors pointed to how doctors prompted and used the model as one likely reason. The cases were curated and challenging, not typical daily visits, so the result may not describe routine primary care. Training clinicians to work with AI, not just granting access, looks like the open problem.

The tool scored higher than the people holding it.