AI scribes produce poorer quality notes than human clinicians, new study finds

Notes produced by AI scribe tools tend to be consistently lower in quality than those produced by human clinicians, a new study has found.In a cross-sectional analysis published in the Annals of Internal Medicine last week, researchers from the U.S. Veterans Health Administration put 11 AI scribe tools up against 18 human note takers comprising of physicians, pharmacists and nurses. The AI and human note takers took notes on five audio recordings depicting standardized primary care clinical cases. To make the cases more realistic, some of the recordings included background noise, muffled audio due to participants wearing surgical masks, or patients speaking with non-native accents.The notes were subsequently reviewed by 30 raters who were asked to rate the notes on 10 attributes (accurate, thorough, useful, organized, comprehensive, succinct, synthesized, internally consistent, free from hallucination and free from bias) on a five-point scale, for a maximum score of 50. All 30 raters were practicing clinicians and were blinded to whether the notes were AI or human generated.Across all five clinical cases, the human generated notes received higher ratings than the AI generated notes. The discrepancy was particularly large in case 2, which featured background noise, and case 3, where both the clinician and patients were wearing masks. The gaps in quality were smaller for the cases that had better quality audio.In addition, the human generated notes were better than the AI notes across each of the 10 attributes. Compared to the human notes, the AI scribes performed the worst at thoroughness, organization, usefulness and accuracy. However, the gaps in quality were the smallest for succinctness, internal consistency and whether the notes were free from bias.“These findings highlight that although ambient AI scribes can generate complete notes, the overall quality remains broadly below that of human-authored documentation,” the researchers wrote.The researchers still note that this technology has huge potential to reduce the administrative burden for clinicians, but stress that AI scribes “should be regarded as tools for generating draft documentation that requires review and editing, rather than as a substitute for clinician-authored notes.” They conclude in the study that health systems should employ rigorous and vendor-neutral evaluations of note quality before widespread deployment. And policymakers should develop best practices to ensure the technology doesn’t compromise accuracy or patient safety.“Although ambient AI scribes hold promise for reducing clinician burden, rigorous and ongoing evaluation of their quality is essential to ensure that these tools enhance rather than compromise the quality of clinical care,” they wrote.

AI scribes produce poorer quality notes than human clinicians, new study finds

Register for free access or login.

Related Topics