Why explainability is essential for AI to be successful in primary care

Now more than ever, we look to primary care to address our health system’s upstream challenges: preventing illness, managing common conditions, and efficiently triaging care to cut costs and enhance outcomes. We’re all familiar with this narrative. Yet, primary care presents a paradox. It’s often perceived as something almost mythical, seemingly unattainable, even as its basic services are becoming as accessible as the broccoli on grocery store shelves. As the gateway to healthcare, primary care clinicians grapple with ambiguity, taking in an unfiltered account of a patient’s life and priorities. Given this complexity, it’s no surprise that the role of AI in primary care decision-making will hinge significantly on its explainability. Read: Top 5 takeaways for med students to be AI-ready Explainable AI, often referred to as XAI, encompasses a range of techniques and approaches that enable us to comprehend the reasoning behind an AI system’s generated outcomes. This capability fosters trust among human users, as they can better understand and rely upon the results produced by the AI. In contrast, the absence of explainability gives rise to the ‘black box’ issue, where AI tools make decisions or draw conclusions without providing the necessary context or insights into their decision-making process. This lack of transparency even extends to the developers of these tools, who may struggle to explain the rationale behind the AI’s outputs. XAI works as a solution by uncovering the underlying reasoning and rationale behind the AI’s recommendations. XAI is being adopted widely to enhance transparency and trust in AI applications. In finance, there’s evidence of XAI’s role in customer retention analysis, insurance processes like claims, underwriting, and pricing, as well as in banking. In healthcare and life sciences, XAI aids in everything from pioneering drug designs, predicting drug interactions, and forecasting various health outcomes. Certain frameworks, like the SHAP framework during the COVID-19 outbreak, have been pivotal; it identified high-risk patients entering emergency rooms, gauging their potential for severe health issues and likely hospital admissions. Read: The iDOCTOR will see you now But XAI isn’t without hurdles. The complexity of some deep learning models can make their logic hard for humans to grasp. Many of these models lean on intricate mathematical structures, making them challenging to interpret and manage. There’s also the proprietary nature of many AI tools, where developers might be hesitant to fully disclose functionalities. Explainability also isn’t just a tech-centric concern. Implementing XAI stirs up medical, legal, and ethical dilemmas that need thorough consideration. Many of these echo existing AI-related queries but become more pronounced with XAI. It’s crucial to determine how to fold AI explanations into medical workflows without disrupting the patient-doctor rapport or bombarding practitioners with excess data. From a legal perspective, ensuring patients understand AI-powered diagnoses aligns with data privacy laws, as people deserve to know how their data drives these AI decisions. Let’s work through a clinical scenario: You’re seeing a 56-year-old female patient who has type 2 diabetes, chronic kidney disease, and osteoarthritis. Today, she’s here for a renewal of her anti-inflammatory meds to manage debilitating knee pain. Visits to the clinic are rare for her, and when she does come in, she usually has multiple concerns. Today, you have less than 15 minutes for her (a timeframe that feels all too familiar to those in public healthcare). During the chat, she mentions occasional bleeding, which she thinks is from hemorrhoids, and notes her slightly high blood pressure. She’s also been having heartburn and hasn’t done any of her age-specific health screenings, including one for colon cancer. Despite all this, her primary ask is simply a renewal of her pain medication; nothing else has worked for her. Now, as you speak, an AI-powered digital scribe is jotting down notes and giving real-time advice. It flags not to renew her anti-inflammatory meds and offers a ranked list of other recommendations. Given the time constraint, you’re down to the wire with just 90 seconds to discuss the plan. How do you navigate the AI’s advice? Of the multiple reasons it might suggest against the meds, which one is the top concern? And how does AI factor in the patient’s own choices in the prioritized recommendations, especially if she’s made an informed decision to continue the medication because it’s the only thing that’s improved her quality of life? Clearly, the role of XAI in this context is crucial. It’s not just about the clinician grasping and trusting the AI’s suggestions to stop anti-inflammatories; it’s about providing them with information and context to have a more effective discussion, to help the patient make the right decisions. Instead of seeing AI as duplicative of human clinician capability, we should view it as a tool that directs information and better enables the clinician’s capabilities. By marrying AI’s vast computational and data-analyzing power with the human-centric process of shared decision-making, we elevate the entire clinical experience. This was a point we touched on earlier when discussing the AI-enabled Health History (AEHH). Peering into the Future Equipping healthcare providers of tomorrow with an understanding of the considerations that go into the use of AI is paramount, and literature provides frameworks for integrating AI into undergraduate medical education. When we consider incorporating explainable AI into primary care today, two clear avenues stand out. In the traditional patient-doctor care setup, the Electronic Medical Record (EMR) serves as a central point for clinical decision guidance. By incorporation into standard workflows, a physician might consult the EMR, garner AI-driven advice, and promptly understand the rationale behind it. This not only equips the physician with invaluable knowledge and ideally streamline workflow (through concurrent automation of administrative tasks), but also enhances the mutual decision-making with patients. Conversely, through a patient-directed strategy, the essence of AI explainability can be smoothly integrated into patient-focused apps. Imagine a patient getting a treatment suggestion via an app for a common ailment or for ongoing management of a chronic disease. With a simple tap, they’d unveil the reasoning behind that advice, fostering a bond of comprehension between the individual and the AI, allowing it to serve as a more trustworthy triage to care or reputable source of health information. In healthcare, trust is the name of the game — it’s why individuals seek professionals for their care needs, and when eroded, it’s what might drive them away. By assisting clinicians through complex decision-making and demystifying the logic behind AI-informed decisions, XAI strengthens trust and transparency in primary care. This not only boosts our ability to deliver tailored, comprehensive care to each patient but also provides a chance to fully realize the potential of primary care. Imagine stepping into the shoes of a primary care clinician. Every day, you’re faced with a diverse set of patients. You have to juggle their immediate concerns with your own clinical judgment on what’s most pressing for their health and safety. Plus, there’s always the challenge of looking beyond the immediate to offer preventive care insights. When these patients are “yours” (meaning you oversee their ongoing care rather than just a one-time visit), what’s deemed a priority can shift throughout the appointment. Ask any family physician, and they’ll tell you: sometimes it’s an offhand remark made as the patient heads out the door that reveals the most urgent health concern (like, “Oh, I’ll mention my chest pain and breathlessness next time.”). Dr. Puneet Seth is a practicing family physician, medical AI researcher and health technologist, having led an EMR company for 10 years. This article originally appeared on Medium.

Why explainability is essential for AI to be successful in primary care

Register for free access or login.

Related Topics