The Crucial Role of End Users in the Success of Healthcare AI

Artificial intelligence (AI) is rapidly changing the healthcare industry, promising benefits like alleviated burden from time-consuming administrative processes, more personalized medicine for patients, earlier disease detection, and better treatment outcomes. However, the use of AI in healthcare also raises important questions about how AI models are made, used, and monitored, and their effects on healthcare operations and patient outcomes.

The US government has become increasingly involved in determining the appropriate balance of innovation and guardrails in AI policy as the technology advances and continues to be used more often in healthcare. Federal agency approaches to regulating AI have varied based on the area of focus. For instance, the Food and Drug Administration (FDA) regulates some AI-driven medical devices and clinical decision support software, while the Assistant Secretary for Technology Policy/ Office of the National Coordinator for Health Information Technology (ASTP/ONC) governs AI included in certified electronic health record (EHR) technology. Such approaches have left certain areas without federal regulatory oversight including AI tools used for prior authorization processes or utilization review. 

Congress is beginning to explore how best to legislate the adoption and use of AI. Both the US Senate and US House of Representatives convened working groups and task forces in 2024. Most notably, the House Bipartisan AI Task Force released a final report that included recommendations, guiding principles, and policy proposals in 2024. Without robust Congressional action, the regulatory agencies within the executive branch will remain limited in their ability to place constraints or encourage innovation within AI. 

Opportunities and Challenges with AI 

While comprehensive AI public policy remains to be seen, AI use is proliferating in healthcare. Tools are in place to automate routine administrative tasks and support clinical care by helping with tasks like coding, utilization management processes, real-time clinical documentation, scheduling, answering patient questions, drafting educational materials, and predicting patient outcomes. These types of tools free up both clinicians and those in roles supporting patient care to work on more complex work. 

Challenges also exist, including the potential for increased administrative burden if AI tools are not implemented appropriately and for a lack of transparency in how AI tools operate. Without transparency, it becomes difficult to validate the safety and effectiveness of a tool. Many of these challenges can be overcome but will require appropriate oversight. 

Even though the opportunities are promising and the challenges daunting, policymakers and AI tool developers can work together with health IT end users, including health information (HI) professionals and providers, to leverage their unique expertise in using these technology tools to realize the full benefits of AI in healthcare. 

Health IT End-Users Alliance 

The Health IT End-Users Alliance is a collection of organizations, including AHIMA, dedicated to advancing end-user perspectives and leadership in health IT policy and standards development. End users include providers and other clinicians, HI professionals, hospitals, clinical support staff, and others. Experts in these roles are uniquely positioned to provide policy recommendations to support clinical care and operations given their direct line of sight into both activities.  

End users possess the needed expertise and insights to ensure AI tools are designed to meet the specific needs of healthcare settings and integrate smoothly into clinical and non-clinical workflows. Their involvement in AI development, implementation, and ongoing monitoring can also foster trust and confidence in AI and help create governance frameworks that balance innovation with appropriate oversight. 

To accomplish those goals and ensure the safe and effective use of AI in healthcare, it is essential to involve all types of end users in all stages of AI tool development, implementation, use, and monitoring. 

Principles for the Safe, Ethical, and Appropriate Use of AI 

As the Trump Administration and US Congress continue to contemplate how to regulate AI while fostering innovation, the Alliance published a consensus statement in April reflecting on the current state of AI in healthcare and endorsing principles intended to guide policymakers through considerations to address these challenges and ensure AI is optimally developed, deployed, and used with the involvement of end users throughout healthcare.  

The Alliance principles call for the following:

• The role of AI in healthcare should augment, not replace, human expertise, supporting cognitive and administrative tasks while preserving human judgment.  
• Regulation and oversight require a risk-based approach proportional to potential harm, aligned federal policies to avoid confusion, and the ethical development of AI tools to safeguard patient safety, quality, and access.  
• Safety and transparency require shared responsibility among policymakers, developers, payers, and healthcare organizations to build trust in safe and effective AI, with transparency in development, data use, decision-making, governance, and ongoing testing, tailored to end-user needs.  
• Liability can be clarified through transparency and should consider the roles of both developers and end users.  
• Privacy of patient and physician data must be prioritized, preventing unnecessary data use or reidentification, and AI should not compromise the security of end-user IT systems.  
• AI must be designed with end-user involvement to integrate seamlessly into existing workflows to avoid exacerbating administrative burden. 
• Appropriate feedback mechanisms on AI tool performance must be established. 
• Policymakers and AI developers should consider the accessibility and usability of AI tools to ensure the benefits of AI tools can be realized by all patients and clinicians. 
• The design, development, implementation, and ongoing surveillance of AI tools must include efforts to identify and address biases that surface. 
• AI use in payment and coverage activities requires transparency, accountability, and expert review to prevent inappropriate denials or restricted access, and regular evaluation for accuracy and fairness with clear appeal processes. 

In preparation for publication of the consensus statement, AHIMA hosted a webinar on April 8 with participants from the American Academy of Family Physicians (AAFP) and Medical Group Management Association (MGMA) to talk about the statement and demonstrate the broad applicability and common-sense nature of the recommendations. The event took a deep dive into the state of AI today and discussed the consensus statement. A recording of the webinar is available on-demand to learn more about the Alliance and this work. 

End users, including HI professionals, are among the most knowledgeable about how both clinical and non-clinical AI tools can be best developed, implemented, and used in their healthcare organizations. Given that these technologies directly impact their workflows, HI professionals are empowered to join these discussions and contribute their valuable expertise to ensure the successful deployment of these new tools within their organizations. By doing so, we can ensure AI remains efficient and effective and contributes to a better healthcare experience for all individuals involved. 

Tara O’Donnell, MPH, is Manager of Regulatory Affairs at AHIMA.