Designing health AI interfaces for safety

Health AI interfaces demand extreme care in their design. It's all too easy for users to blindly defer to algorithmic suggestions, a failure mode that becomes especially dangerous when diagnostic software is deployed across both clinical and consumer environments, where the presentation of data can shape the final decision just as powerfully as the underlying model's accuracy itself. But this problem sits within a broader pattern. Developers are finally confronting how different classes of users process automated recommendations, and new research from the Massachusetts Institute of Technology, Columbia University, and Stanford University shows that a single interface configuration can produce wildly divergent outcomes. It all hinges on the user's existing professional knowledge. So the fix isn't just better algorithms. It's understanding who's looking at the screen.

Strip away the marketing and the calculation is straightforward. For years, the artificial intelligence sector has pushed explainability as an unalloyed good. The assumption was that showing the inner workings of a model would help humans make safer choices. The reality is far more complex. The way an explanation is presented can either act as a valuable safety check or create a false sense of security that leads to severe errors.

The design challenge of health AI interfaces

The deeper question is positioning. When designing health AI interfaces, developers must choose between offering raw predictions or providing detailed, human-like explanations. Researchers tested several of these approaches on both non-experts and primary care providers who were evaluating dermatological images for skin disease. The interface designs included:

• Raw predictions accompanied only by confidence levels.
• Visual aids showing similar images to support the prediction.
• Heat maps highlighting specific regions of interest on a medical image.
• Plain-language justifications generated by large language models.

This variance in presentation revealed a stark divide in how humans interact with technology. Non-experts showed improved diagnostic accuracy when they had access to AI assistance, but this gain came at a steep cost. Their performance was driven by direct deference to the machine. When the model was correct, the users succeeded. But when the model made an error, the non-experts followed the wrong recommendation, suffering a drop in performance that outweighed the benefits of correct guidance.

Why explanations mislead the untrained user

Large language models pose a unique threat in this context. That's the core problem. Because these systems generate highly confident, fluent narratives, they can make incorrect diagnoses sound authoritative, and that authority is precisely what makes them so dangerous to people who lack deep medical training. Non-experts participating in the research trusted language-based explanations regardless of whether the model's output was correct. Surprising, right? But they found vague or generic explanations even more convincing, which led to high confidence in wrong answers, so the more hollow the text, the more it swayed them. It's a nasty trap.

"Good AI systems can improve performance in some health settings, but this has to be balanced carefully with algorithmic deference that can lead to more error. We know that both AI and explainability methods can engage automation bias in humans, and this anchoring effect is something that must be accounted for when we design AI systems." ; Marzyeh Ghassemi, MIT Department of Electrical Engineering and Computer Science

The anchoring effect is particularly dangerous for patient-facing tools. People type symptoms into search bars and consumer apps every day. But when those tools deliver a polished, written rationale for a wrong assessment, they don't just fail to help, they actively steer users down a harmful path, and the consequences can be severe. The most vulnerable? Those with the least medical knowledge. They can't resist these erroneous, highly convincing explanations.

The resilience of clinical professionals

Primary care providers interacted with the technology in a completely different manner. They didn't blink. Clinicians remained highly resilient when presented with incorrect AI recommendations or flawed explanations, and their training allowed them to treat the model's output as a data point to be verified rather than an absolute truth to be followed blindly. So they checked, questioned, and moved on. It's a habit born of experience.

A simpler interface works best for experts

Interestingly, the clinicians hit their peak diagnostic performance with the simplest interface: a raw prediction, no explanation attached. For these trained professionals, complex language-model explanations actually delivered the smallest accuracy gains, because their own expertise already supplied the reasoning and context. They've got a diagnosis in mind, so they check the AI against their own training. A bad explanation gets caught fast. But a non-expert builds an opinion from scratch, using that same explanation as the sole foundation, and that's where the flaws slip through unnoticed.

Designing for the workflow of clinical users

From a competitive standpoint, developers can't rely on a one-size-fits-all interface template. The user's baseline expertise must dictate how information is presented. So for professional clinicians, tools should offer direct model outputs that support clinical review against their own professional judgment, letting them weigh machine suggestions against their own training and experience. But for patients, interfaces must be designed to mitigate the risks of automation bias. That's a different challenge entirely. It's a delicate balance.

Timing is everything. When an explanation appears before users can form their own opinion, they become far more deferential to the system, a shift that quietly undermines independent judgment.

Future directions for diagnostic interface design

Looking at the wider sector, future developments will require specialized interface structures for different user categories, and designers must also account for how systems handle subtle versus atypical symptoms. That's a tall order. The research indicated that while AI models excelled at identifying subtle symptoms, humans performed much better when presented with atypical symptoms or unrelated features in an image. So the gap is clear. It's not about replacing one with the other, but about knowing where each side truly shines.

The next phase of system deployment changes everything. Builders can't treat explainability as a default feature anymore, because the old assumption that every user needs the same depth of data presentation is what turns information into a liability instead of an asset. So interfaces must shift. They have to adjust dynamically, role by role, moment by moment. It's not optional, it's the core of the design. But here's the hard truth: if the presentation doesn't match who's looking at it, the data stops helping and starts harming, and that's a risk we've seen play out in real deployments before. Keep it flexible. Keep it human. That's the only way forward.