AI’s errors may be impossible to eliminate – what that means for its use in health care

Over the past decade, AI’s success has generated overwhelming enthusiasm and bold claims – even as users frequently encounter mistakes made by AI. An AI-based digital assistant can embarrassingly misunderstand someone’s speech, a chatbot can hallucinate facts, or, as I’ve experienced, an AI-based navigation tool can even guide drivers through a cornfield – all without recording errors.

People tolerate these mistakes because technology makes certain tasks more efficient. However, its supporters are increasingly advocating the use of AI – sometimes with limited human supervision – in areas where errors come with a high cost, such as health care. For example, a bill introduced in the US House of Representatives in early 2025 would allow AI systems to prescribe medications autonomously. Since then, health researchers as well as lawmakers have debated whether such a prescription would be feasible or desirable.

It remains to be seen how exactly such a requirement would work if this or similar legislation were passed. But it raises issues about how many errors AI developers can allow their tools to make and what the consequences would be if those tools led to negative outcomes or even patient deaths.

As a researcher studying complex systems, I study how different components of a system interact to produce unpredictable results. Part of my work is exploring the limits of science – and specifically AI.

Over the past 25 years, I have worked on projects such as traffic light coordination, improving bureaucracies and detecting tax evasion. Even when these systems can be very effective, they are never perfect.

For AI in particular, errors could be an inevitable consequence of how systems operate. Research from my lab suggests that particular properties of the data used to train AI models play a role. This is unlikely to change, no matter how much time, effort, and funding researchers put into improving AI models.

No one – and nothing, not even AI – is perfect

As Alan Turing, considered the father of computing, once said: “If you expect a machine to be infallible, it cannot also be intelligent. » Indeed, learning is an essential element of intelligence and people generally learn from their mistakes. I see this tug of war between intelligence and infallibility playing out in my research.

In a study published in July 2025, my colleagues and I showed that it may be impossible to perfectly organize some data sets into clear categories. In other words, a given data set may produce a minimum number of errors, simply because of the overlap of elements from many categories. For some data sets – which form the foundation of many AI systems – AI will perform no better than chance.

For example, a model trained on a dataset of millions of dogs that records only their age, weight, and height will likely distinguish Chihuahuas from Great Danes with perfect accuracy. But he can make mistakes in distinguishing between an Alaskan malamute and a Doberman pinscher, because different individuals of different species can fall into the same age, weight and height ranges.

This categorization is called classifiability, and my students and I began studying it in 2021. Using data from more than half a million students who attended the Universidad Nacional Autónoma de México between 2008 and 2020, we wanted to solve a seemingly simple problem. Could we use an AI algorithm to predict which students would complete their university studies on time, i.e. within three, four or five years of starting their studies, depending on the major?

We tested several popular algorithms used for classification in AI and also developed our own. No algorithm was perfect; the best ones – even the one we developed specifically for this task – achieved an accuracy rate of around 80%, meaning that at least 1 in 5 students were misclassified. We realized that many students were identical in grades, age, gender, socioeconomic status, and other characteristics – and yet some would finish on time, and others would not. Under these circumstances, no algorithm would be able to make perfect predictions.

You might think that more data would improve predictability, but that usually comes with diminishing returns. This means that, for example, for every 1% increase in accuracy, you might need 100 times more data. Thus, we would never have enough students to significantly improve the performance of our model.

Additionally, many unpredictable turns in the lives of students and their families – unemployment, death, pregnancy – can occur after their first year of college, likely affecting their on-time success. So, even with an infinite number of students, our predictions would still give errors.

The limits of prediction

To put it more generally, what limits prediction is complexity. The word complexity comes from the Latin plexuswhich means intertwined. The components that make up a complex system are closely related, and it is the interactions between them that determine what happens to them and how they behave.

Thus, studying system elements in isolation would likely give misleading insights about them – as well as the system as a whole.

Take for example a car driving around town. Knowing the speed at which it is traveling, it is theoretically possible to predict where it will end up at any given time. But in real traffic, its speed will depend on interactions with other vehicles on the road. Since the details of these interactions appear in the moment and cannot be known in advance, it is only possible to accurately predict what happens to the car a few minutes later.

Not with my health

These same principles apply to prescribing medications. Different conditions and diseases may have the same symptoms, and people with the same condition or disease may have different symptoms. For example, fever can be caused by a respiratory or digestive illness. And a cold can cause a cough, but not always.

This means that healthcare datasets have significant overlap that would prevent AI from being error-free.

Of course, humans make mistakes too. But when AI misdiagnoses a patient, as it surely will, the situation falls into legal limbo. It is not clear who or what would be responsible if a patient were injured. Pharmaceutical companies? Software developers? Insurance agencies? Pharmacies?

In many contexts, neither humans nor machines are the best option for a given task. “Centaurs” or “hybrid intelligence” – that is, a combination of humans and machines – tend to be better than each in isolation. A doctor could certainly use AI to decide on potential medications to use for different patients, based on their medical history, physiological details, and genetic makeup. Researchers are already exploring this approach in precision medicine.

But common sense and the precautionary principle
suggest it is too early for AI to prescribe medications without human oversight. And the fact that errors can be built into the technology could mean that when human health is at stake, human oversight will always be necessary.

This article is republished from The Conversation, an independent, nonprofit news organization that brings you trusted facts and analysis to help you make sense of our complex world. It was written by: Carlos Gershenson, Binghamton University, State University of New York

Learn more:

Carlos Gershenson does not work for, consult, own shares in, or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond his academic appointment.