Trusting AI that Echoes Back: A case against personalizing AI systems in healthcare

Abstract: While consensus standards and criteria for trustworthy AI systems (e.g. accuracy, transparency, nonbias) are now widely established, knowledge gaps remain around users’ trust criteria for particular AI systems in specific contexts. This uncertainty reduces our ability to anticipate or positively shape the results of human-AI interactions and constitutes a major barrier to responsible implementation of AI systems in healthcare where quality control depends on procedural consistency and standardization. As part of a multisite, NCATS-funded study, we examined trustworthiness criteria among diverse stakeholders (clinicians, developers, patients, caregivers, scholars) for AI-based digital phenotyping models designed to identify and address psychopathology in adolescents. We found that, beyond consensus standards, trust depends on the degree to which AI outputs are congruent with and confirm users’ own perspectives and evaluations (e.g., parity in diagnostic classification; congruence with subjective experience). While we recognize the importance of validating AI systems against human understandings, we argue that this “confirmation bias” can become problematic when significant variability or disagreement exists among clinicians, or between patients and clinicians. Developing AI systems without actively addressing questions of epistemic authority can result in unreliable systems that simply show users what they want to see (similar to “filter bubbles” in social media). Contextualizing our results amidst the broader trend towards “personalizing AI,” this talk highlights the dangers of tailoring AI outputs to user preferences and advocates for design choices that mitigate confirmation bias in human-AI interactions and encourage consideration of variegated rather than singular perspectives.