Adding human guidance to AI tutors enhances benefits for students, study finds

Human tutoring and computer-based AI tutors are widely recognized for their effectiveness in supporting learning. However, human tutoring is expensive and difficult to scale, and AI tutors vary widely in their ability to adapt to students’ academic and motivational needs. In a new study, a team of researchers evaluated a year-long virtual human-AI tutoring program when using AI tutors in the classroom. Students mentored through the program outperformed students mentored by AI alone.

The study was conducted by researchers from Carnegie Mellon University, the University of Hong Kong and Stanford University. It is published in chapter form in Artificial Intelligence in Education: Proceedings of the 26th International Conference on Artificial Intelligence in Education, which took place July 22-26 in Palermo, Italy.

“Our results suggest that human tutors enhance the benefits of AI tutors, with gains increasing with time on task,” says Lee Branstetter, professor of economics at Carnegie Mellon’s Heinz College, co-author of the study. “In addition to replicating other studies over a longer period of time, we showed that time-on-task data is a valuable indicator of learning progress and a way to identify students who need additional support to get the most out of AI tutoring.”

High-impact human tutoring is a structured form of teaching delivered one-on-one or in small groups that complements students’ classroom experiences and targets specific learning goals. This type of tutoring has been shown to be effective in improving student learning outcomes. But the high costs of one-on-one tutoring have prompted researchers to explore alternatives that might come close.

One approach has been to use AI tutors, which aim to replicate the benefits of human tutoring in a more scalable and cost-effective way. In human-AI tutoring, human tutors provide personalized educational and motivational support, while AI tutors ensure consistent and adaptive feedback and guidance.

“This research shows the promise of collaboration between AI and human tutors to amplify the impact of tutoring on student outcomes. Seeing these significant improvements materialize in real-world classrooms over the course of a school year is an encouraging sign,” said Jordan Gutterman, a graduate student at Heinz College at Carnegie Mellon, co-author of this study as part of CMU’s commitment to engage students in cutting-edge research.

In this study, researchers evaluated human-AI tutoring over one school year, extending the duration of previous studies. Using one-year log data and standardized state tests, they compared more than 350 U.S. seventh graders who received tutoring in human AI in 2023-2024 with nearly 320 students who received tutoring only in AI the previous year.

Half of the students in the study were female and half were male; 65% were Hispanic, 17% were Asian, 10% were African American, 6% were White, and the remainder were other races and ethnicities. Additionally, 14% received special education services, 88% were from socioeconomically disadvantaged families, and 18% were learning English.

The human tutors were undergraduate students at a US university who had been trained in tutoring strategies to identify and respond to student needs. While students could access AI tutoring at any time, they also had a designated time period to use it; Meanwhile, the human-AI group received virtual tutoring via Zoom.

The human-AI group demonstrated significantly higher growth and was 0.36 grade levels ahead at the end of the year, according to the study. Although there was no overall difference in state test scores, the researchers found a significant interaction between human-AI tutoring and time on task (i.e., use of an AI tutor). This suggests that human tutors enhance the benefits of AI tutors, with gains increasing with time on task.

“We found that relatively small doses of human support can significantly improve learning outcomes over time,” says Ashish Gurung, a postdoctoral researcher at Carnegie Mellon’s Human-Computer Interaction Institute, co-author of the study. “Our findings provide a solid foundation for rethinking how human-AI collaboration can improve both the learning experience of students and the overall effectiveness of AI tutors.”