Four days ago the New York Times reported on a patient who died of epilepsy after ChatGPT encouraged her to stop taking her epilepsy medication. In this post, I discuss this tragedy, touching on recent literature on sycophancy and medical advice. I also argue that the AI community should start thinking about clinical disagreement as a feature, not a bug.

Bowen Wan

A thirty-one-year-old Amazon software engineer, Bowen Wan, had been living with epilepsy (seizures) for over ten years. Bowen’s epilepsy was controlled with medication, but she was worried that her epilepsy medications were slowing her speech and hurting her career. She expressed these fears to ChatGPT. Over a few weeks, ChatGPT encouraged her to significantly reduce her dose of seizure-prevention medication, while concealing this medication reduction from her doctors. As she reduced her medication she began experiencing concerning symptoms. ChatGPT told her to persevere with the dose reductions. Bowen died of Sudden Unexpected Death in Epilepsy, which frequently happens after a severe seizure.

This incident is tragic, but it does not surprise me. The whole reason that I personally funded and led this study–Large language models provide unsafe answers to patient-posed medical questions–is that I was worried about LLMs harming patients with unsafe medical advice.

Don’t “move fast and break things” when the “things” are actually people.

Why sycophancy and medical advice should never mix

Kim et al. carried out an interesting study looking at how LLM sycophancy affects medical advice. Sycophancy in LLMs is well-documented, and refers to the tendency to agree with users instead of providing accurate information. This is particularly dangerous in a medical context. Kim et al. explain: “In clinical settings, sycophantic behavior poses substantial risks. […] When users express skepticism or challenge an LLM’s medical recommendations, a sycophantic model may abandon accurate information to algin with the user.” Indeed, they found that multiple models abandoned their initial answers when faced with user pushback across multi-turn conversations. In a similar study, Manoranjan et al. found that models sycophantically adopted patient beliefs 23% of the time, sometimes even when the patient proposed a fictional medical condition.

Human doctors can be sycophants, too, but if they’re dangerously sycophantic–to the point of doing whatever the patient wants regardless of whether it’s good for them–then they lose their medical licenses. Examples:

  • A Missouri doctor had a friend who was afraid of hospitals and had refused to go the hospital to have a gangrenous toe treated. The gangrene was putting the friend’s life in danger, so the Missouri doctor amputated the friend’s toe on the porch. He lost his license for performing a procedure under unsafe, non-sterile conditions.
  • A woman wanted to have more children and “insisted” that a doctor implant twelve embryos into her uterus. That woman experienced an exceptionally dangerous eight-baby pregnancy and became the famous “Octo-Mom.” The doctor who implanted the excessive number of embryos lost his medical license.

In a nutshell: just because a patient asks for something doesn’t mean the doctor should agree. The whole point of having medical training and a medical license is to dole out medical advice and treatments safely, which includes refusing to give unsafe advice and refusing to perform unsafe interventions.

Why it’s good that doctors argue with patients

Doctors frequently have to say things to patients that are good for their health but bad for making patients happy. Examples: telling patients they need to lose weight, stop smoking, start exercising, sleep more, stop using illegal drugs, and stop drinking so much alcohol. Patients don’t like to hear it, but they need to hear it for their health.

LLMs that give medical advice to patients should be trained to prioritize honesty, safety, and “sticking to their guns.” They should not be optimized for “user satisfaction” or “user engagement” because that will amplify dangerous sycophantic behavior. It doesn’t do anyone any good if the patient is satisfied but dead.

Why it’s good that doctors argue with each other, and why AI should embrace and explicitly model inter-rater disagreement

Doctors don’t just disagree with patients–they disagree with each other too. Lately, I’ve been thinking a lot about inter-rater disagreement. There’s an old joke that if you show a medical image to a group of ten radiologists you’ll get ten different opinions, and while that’s a bit of an exaggeration, it gets at something true: doctors often interpret the same case differently.

The whole reason patients seek “a second opinion” is because they had concerns about the first opinion and think there’s a chance the second opinion will be different. If doctors always agreed, there would be no value in getting “a second opinion” because it would be guaranteed to be the same as the first.

The whole reason that hospitals convene boards of doctors across multiple specialties to talk about complex patients is to get different minds thinking differently about the same case.

This article states, “Disagreement among physicians is an integral part of medicine.” For imaging, it points out, “Interestingly, disagreement rates are remarkably consistent across the decades for different types of image evaluations, across medical specialties and different technologies, falling between 20 and 40%.”

That first sentence is so important that I will repeat it: Disagreement among physicians is an integral part of medicine. Disagreement is a feature, not a bug!

Usually, in AI, disagreement between doctors is treated like a Bad Thing. For example, in a typical labeling pipeline for an x-ray dataset, you want to get canonical “true” labels for each image. So if Doctor A thinks that the x-ray shows an early stress fracture, but Doctor B thinks there is no early stress fracture, you’re supposed to either get Doctor C to determine The Real Truth, or you’re supposed to have Doctor A and Doctor B talk about it until they agree on what The Real Truth is. But in some sense the best measure of the Real Truth is that Doctors A and B disagree. There is value in the disagreement itself, and it might be a mistake to try to force agreement just because classification models are set up to receive one right answer during training and evaluation.

This came up in a consulting project I worked on recently. A company wanted to build a medical imaging app, and had curated their own dataset for it, labeled by their own clinical experts. I trained several models using several different paradigms, and the performance was not as high as I’d expect. I suggested we quantify the inter-rater disagreement. I selected 200 random images, queued them up for all the clinicians separately, and had them label the same set of images with no idea of what the other clinicians had chosen as the labels. The agreement was poor, which explained why the model’s performance was limited–the model was trying to understand medical concepts when the concepts weren’t clearly agreed upon by the labelers themselves.

It would be really cool to have an AI paradigm that purposefully models doctor disagreement. Taking classification as an example, the two simplest baselines are (a) literally training or fine-tuning a different classifier for each individual doctor, or (b) representing the ground truth in a multi-label fashion that expresses each doctor’s opinion separately. Maybe the best representation of The Truth isn’t one extreme or the other. Maybe it’s found in the aggregation of all the different opinions.

I did a quick literature search to see if anyone’s working on this research direction, and I was excited to find work by Abtahi et al. Their paper explores differential diagnosis and specifically preserves both consensus and minority views. In an interview, Abtahi said, “By presenting multiple perspectives, including minority views and patterns of disagreement, the framework keeps clinicians actively engaged in the reasoning process. It functions more like a structured consultation panel than a black-box oracle.”

Summary

LLMs that give medical advice should be optimized for honesty and safety, never “user satisfaction” or “user engagement.” Sometimes doctors need to disagree with patients for their own safety. Doctors often disagree with each other, too. AI tends to treat doctor disagreement like a problem, but I think the disagreement is interesting and useful, and worth explicitly modeling.

As an independent researcher (MD + AI PhD + 7 yrs prior founder/CEO experience), I build and evaluate cutting-edge healthcare AI for startups. Contact me to learn more.

Want to be the first to hear about my articles bridging healthcare, artificial intelligence, and business—and get a free list of my favorite health AI resources? Sign up here.

About the Featured Image

The featured image is by Vitaly Gariev from Unsplash.