Google’s new Scholar Labs search uses AI to find relevant studies
Google announced that it is testing a new AI-powered research tool, Scholar Labs, designed to answer detailed research questions. But his demonstration shed light on a larger issue: finding “good” scientific studies. How much will scientists trust a tool that forgoes the usual methods of assessing a study’s popularity with the scientific establishment, in favor of reading relationships between words to help surface good research?
The new search tool uses AI to identify key topics and relationships in a user’s query and is currently available to a limited set of logged in users. The Scholar Labs demo video included a question about brain-computer interfaces (BCI). I have a PhD in BCI, so I was excited to see what Scholar Labs found.
The first result was a review article on BCI research published in 2024 in a journal titled Applied sciences. Scholar Labs includes explanations of why the results match the query. So he points out that the article discusses research on a non-invasive signal called an electroencephalogram and examines some cutting-edge algorithms in the field.
But I noticed that Scholar Labs doesn’t have filters for common metrics used to separate “good” studies from “not so good.” One metric is the number of times a study has been cited by other studies since its publication, which loosely translates to the popularity of an article. It is also associated with time: a recently published study may have no citations or accumulate hundreds in a few months; a study from the 90s could boast thousands. Another measure is the “impact factor” of a scientific journal. Journals that publish widely cited studies have a higher impact factor and therefore have a reputation for being more rigorous or more meaningful to the scientific community. Applied sciences self-declares an impact factor of 2.5. Naturefor comparison, indicates that its impact factor is 48.5.
The original Google Scholar provides an option to sort studies by “relevance” and lists the number of citations for each result. The goal of the new Scholar Labs is to find “the most useful articles for the user’s research,” said Google spokeswoman Lisa Oguike. The edge It does this by ranking articles in the same way as researchers themselves, Google explains, by “weighing the full text of each document, where it was published, who it was written by, and how often and when it has been cited in other scientific publications.”
However, the new Scholar Labs will not sort or limit results based on an article’s citation count or a journal’s impact factor, Oguike said. The edge.
Image: Google Scholar
“Impact factors and citation counts depend on the research area of the articles and it may be difficult for most users to guess the appropriate values in the context of specific research questions,” Oguike wrote. “By limiting by impact factor or number of citations, one can often miss key articles – particularly articles in interdisciplinary/adjacent fields/journals or recently published articles,” Oguike added.
Metrics such as citation count and impact factor are “pretty crude assessments of the quality of a paper,” said Matthew Schrag, associate professor of neurology at Vanderbilt University Medical Center, in an interview with The edgein accordance with Google’s statement. They “speak more to the social context of the newspaper” than its quality, although “those two things are hopefully correlated,” he said.
Schrag, who studies Alzheimer’s disease, is one of several scientists and detectives who have reported questionable data in published scientific studies. The efforts of data sleuths like Schrag, and increased attention from the scientific community at large, have resulted in studies being pulled from reputable journals due to falsified images, corrections published by Nobel laureates, and federal investigations into falsified data.
It is still difficult to not Use citation counts or a journal’s reputation to casually review a study, especially when entering a new field. Professor of rehabilitation sciences at Tufts University, James Smoliga, a frequent user of early Google Scholar, believes that more highly cited articles are more trustworthy. “I’m guilty of it, like everyone else,” he told The edge. He does this despite debunking the methods used in a study with thousands of citations. “And I know myself that that’s not the case, but yet I still fall into that trap because what else am I going to do?”
I repeated Scholar Labs’ demo query on BCI research in stroke patients in PubMed, a leading repository of biomedical and health research managed by the National Library of Medicine of the US National Institutes of Health. Unlike Scholar Labs, PubMed relies heavily on filters and terms related to Orsand Ands. I limited my results to reviewing only clinical research articles, i.e., conducted only on humans, from the last five years. I excluded preprints, which are studies published directly in a paper repository like arXiv or bioRxiv without having undergone a review process by other scientists. Two of the six findings focused exclusively on EEG as the primary type of non-invasive BCI used to help stroke patients.
Users will be able to request “recent” articles in their query and specify a time period in their request, and Scholar Labs uses the “full text of research articles” to find results that match the user’s query, Oguike added.
Google calls Scholar Labs “a new direction for us” and says it plans to incorporate user feedback in the future. There is a waiting list for access.
Schrag believes that AI-powered research, like that of the new Scholar Labs, has a place in the scientific ecosystem. This could, in theory, cast a wide net to surface newspapers that otherwise would have slipped through the cracks, or add additional context about a newspaper’s popularity on social media platforms, he added. Studies require an overall assessment, he said, which AI might be able to answer. “You need to have an idea of the standards in force in the field in terms of rigor and whether a study meets them,” he added.
Ultimately, it’s up to scientists to determine what science has impact, Schrag said. This requires reading and engaging with the scientific literature “to be the final arbiters and not let algorithms be the final arbiter of what we consider high quality.”
