An AI model beat ER doctors at diagnosing patients, in a new study : NPR

Researchers tested an AI model against emergency room doctors and found that the model outperformed humans.

shape charge/E+/Getty Images

hide caption

toggle caption

shape charge/E+/Getty Images

A patient comes to the hospital with a pulmonary embolism – a blood clot that has spread to the lungs. After an initial improvement, their symptoms begin to worsen. The medical team suspects that the medication is not working.

Artificial intelligence in stages — with its own theory.

He analyzed medical records and suspected that a history of lupus, an autoimmune disease that can lead to heart inflammation, might explain what the patient was really ailing.

It turns out the AI ​​model is correct.

This type of scenario could become a reality in the not-so-distant future, according to a study published Thursday in the journal Science.

Researchers based at Harvard Medical School and Beth Israel Deaconess Medical Center found that an AI reasoning model, developed by OpenAI, excelled at diagnosing patients and making decisions about managing their care. It matched and often outperformed doctors and the earlier AI model, GPT-4.

The researchers conducted a series of experiments on the AI ​​model to test its clinical acumen, including real-world cases like the lupus patient who had previously been treated at Beth Israel’s emergency department in Boston.

The team assessed how well the AI ​​model could provide an accurate diagnosis at three time points, from emergency triage to hospital admission.

Overall, AI outperformed two experienced doctors – and that was with only the electronic health records and limited information doctors had at the time.

“That’s the big takeaway for me: It works with the messy, real-world data of the emergency department,” said Dr. Adam Rodman, a clinical researcher at Beth Israel and one of the study’s authors. “It works for making real-world diagnoses.”

Other parts of the study focused on case reports published in the New England Journal of Medicine and clinical vignettes to determine whether the AI ​​model could address well-established “benchmarks” and resolve thorny diagnostic questions.

“The model outperformed our very large physician base,” said Raj Manrai, an assistant professor of biomedical informatics at Harvard Medical School, who was also part of the study.

The authors point out that AI relies solely on text, whereas in real life clinicians need to pay attention to many other elements such as images, sounds and non-verbal cues when diagnosing and treating a patient.

Yet this work shows how far technology has progressed in recent years. Previous versions of large language models failed in the face of uncertainty and in generating a list of possible conditions that could explain symptoms, what is called a differential diagnosis.

“This paper is a nice summary of the magnitude of the improvements,” says Dr. David Reich, clinical director of the Mount Sinai Health System in New York, who was not involved in the work.

“You have something pretty specific, maybe ready for prime time,” he says. “Now the open question is how on earth do we introduce it into clinical workflows in a way that actually improves care?”

After all, arriving at a tricky final diagnosis—which the AI ​​model shines at—doesn’t necessarily reflect how things play out “in real-world clinical medicine,” says Reich, where “the results are much more subtle and perhaps more diverse.”

And the emergency department is only a small part of a patient’s total medical care. Rodman acknowledges that it’s unlikely the AI ​​would have done such an “impressive” job if the team had provided it with records of someone who had spent a month in the hospital.

None of the people involved in the new study think the results support replacing doctors with AI, “despite what some companies are likely to say and how they are likely to use these results,” Manrai says.

“I think this means we are seeing a very profound technological change that will reshape medicine,” he adds.

But the results demonstrate that AI models need to be tested rigorously, ideally through prospective trials that can provide more certainty about the technology’s ultimate impact on clinical practice.

“Designing these trials is a very difficult process,” says Reich, “but this study is a perfect call to action.”