Modeling Regulation, Regulating Models: How Stakeholders Conceptualize AI Governance in Healthcare

Abstract: With the growing prevalence of artificial intelligence/machine learning (AI/ML) in healthcare, numerous stakeholders have voiced concerns about how these models are being regulated. While model developers are pushing initiatives to promote safe and responsible AI/ML innovation via self-regulation, the Biden administration has proposed a more top-down regulatory framework that instructs federal agencies to issue guidance and standards for ethical AI/ML development. In the meantime, people who work in healthcare are grappling on a day-to-day basis with how these models can be both integrated into clinical practice and also maintained over time in an ethical and equitable manner. Based on interviews with key stakeholders including clinical informaticists, physicians, legal experts, and regulators, this research analyzes how AI/ML governance in healthcare is being envisioned in the absence of an established regulatory framework. While some experts consider the FDA to be a good model for how regulation should work in this domain, others raise concerns about the agency’s shortcomings and limited jurisdiction. Instead, these stakeholders have proposed that an AI/ML regulatory framework could be pieced together by drawing on components of a wider variety of systems in which safety and responsibility are key to technical innovation. These include air traffic control, space travel, car service manuals, in vitro fertilization, and X-ray technology. This research compares how experts employ different metaphors to envision the regulation of these AI, highlighting what stakeholders in the field hope to accomplish with AI/ML governance.

Kellie Owens – Assistant Professor, Population Health, NYU