Scientists Warn AI Slop Is Wreaking Havoc in the Research World
Scientific articles rely on readers’ trust in their information. That’s why it’s concerning that a new study by researchers linked to Cornell and UCLA found 146,900 AI-generated false citations in scientific articles housed in four major research databases.
A key limitation of large language models such as Gemini and ChatGPT is their tendency to produce plausible but incorrect information, a phenomenon known as hallucination. If a researcher relies on a chatbot to write citations without verifying them, the model may generate entirely fabricated references.
Although scientific articles are often hidden from the public eye, the research they report has a profound impact on our lives. Everything from the Internet to lithium-ion batteries started as a research paper.
But when scientists submit papers citing AI hallucinations, it can erode confidence in the quality of the research.
Sloppy science
The research team analyzed 111 million references from 2.5 million scientific articles. They searched for quotes whose titles could not match any publication. Although some of these cases were just spelling mistakes, the team also discovered hallucinations.
Unscrupulous researchers had been forging citations long before the advent of chatbots. So the team also looked at unmatched citation rates in research published before 2023, when chatbots had not yet become ubiquitous.
“We see a large increase in non-existent credentials following widespread adoption of the LLM,” the authors write in the article.
The team also found that bad citations were spread across many articles rather than concentrated in just a few. This suggests that the problem is widespread, with many researchers relying on AI-generated references without fully verifying them.
Warning signs
Usha Haley, a management professor at Wichita State University, told CNET by email that she views the proliferation of fake quotes as a serious warning.
“False or AI-generated citations undermine trust in the scientific record that provides the foundation upon which peer review and cumulative knowledge rest,” Haley said. “Worryingly, this skepticism is now coming from within academia itself and from early career researchers.”
The four databases in which the researchers found the false citations are arXiv, bioRxiv, SSRN and PubMed Central. These organizations, called scientific repositories, play a major role in the world of research.
Before an article is published in a scientific journal, authors often upload it to a scientific repository, thereby increasing its visibility and allowing the global scientific community to immediately access it. The new paper on mind-blowing AI quotes is currently hosted on arXiv.
Recently, arXiv has taken steps to stem the flow of false citations. The organization announced Tuesday that it would ban authors who submit work with wild quotes or any sign of AI content that hasn’t been carefully vetted.
“The body of science is getting diluted. A lot of things about AI are either wrong or meaningless. It’s just noise,” Steinn Sigurdsson, arXiv’s chief science officer, told CNET’s Katelyn Chedraoui in February. “It makes it harder to find out what’s really going on and it can mislead people.”
