Researchers at the University of California, San Diego pulled 195 such questions from Reddit's r/AskDocs, where a verified physician had already answered, and had a chatbot answer the same questions, then had licensed professionals rate both sets blind to which was which.
The chatbot's answers were rated as good or very good quality 3.6 times more often than the physicians' answers, and rated empathetic or very empathetic 9.8 times more often. Physician replies ran about 52 words; chatbot replies ran about 211.
Length may explain part of the empathy gap: doctors on the forum were typically answering for free, in spare moments, while the chatbot had no such time pressure. The questions came from a public forum, not real clinical visits with a patient's full history in front of the responder, so the comparison does not show the chatbot outperforming a doctor who actually knows the patient.
Read the original study
Ayers et al., JAMA Internal Medicine, 2023 · doi.org/10.1001/jamainternmed.2023.1838
Longer answers may explain part of it.
Source
- Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., & Smith, D. M. (2023). Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Internal Medicine, 183(6), 589–596. https://doi.org/10.1001/jamainternmed.2023.1838