The benefits of medical AI assistance vary based on user expertise

A study by researchers at MIT and elsewhere found that AI assistance generally improved the accuracy of non-experts and clinicians in diagnosing skin diseases, but AI explainability methods affected users differently depending on their knowledge level, according to an announcement from MIT News. A description of the work appears in Nature Medicine.
The researchers tested non-experts and primary care providers in skin disease diagnosis with and without different explainable AI systems. Approaches included a prediction and confidence level with no explanation, a method providing similar images, a heat map highlighting important image regions, and a large language model explaining the model's reasoning in plain language. Non-experts decided whether an image of a skin mole was cancerous; clinicians provided a differential diagnosis of dermatological disease.
All explainable AI approaches improved non-experts' accuracy, mostly by helping them diagnose non-cancerous moles, the study found. That improvement was largely due to deference to the AI system. Non-experts trusted LLM-based explanations whether they were right or wrong, and found explanations more convincing when vague or generic. The deference effect was largest with LLM explanations, and users were more confident about wrong answers when aided by an LLM. A fairness-constrained model designed to combat bias against darker skin tones significantly improved accuracy and reduced diagnostic disparities based on skin tone.
Clinicians were not tripped up by incorrect AI assistance and performed best when given only a model's prediction with no accompanying explanation. Of all explainability methods, LLMs boosted clinicians' accuracy the least.
Users most deferential to AI were the worst performers without AI help. If an explanation was given first, before users performed their own diagnosis, they became more deferential. AI systems outperformed humans when disease presentation was subtle, while humans performed better with atypical symptoms or unrelated image features.
Lead author Orson Xu of Columbia University said the same explanation can help an expert and mislead a beginner. Marzyeh Ghassemi of MIT said algorithmic deference must be balanced carefully when designing AI systems. Co-author Roxana Daneshjou of Stanford University said those with the least medical knowledge are most likely to be led astray by erroneous explainable AI output. The research was funded in part by the National Science Foundation, Schmidt Sciences, the National Bureau of Economic Research, and Columbia University.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Study finds non-experts deferred to LLM-based diagnostic assistance, even when it was wrong, while clinicians caught AI errors.