AI tutoring matched human tutors and cost 918 times less
2,383 people, three arms, a real randomized comparison: AI tutor, expert human tutor, control. GRE prep, seven domains, learning gains measured on actual test performance. StudentBench, posted September 23, is the kind of study this field keeps promising and almost never runs, and the headline is that AI and human tutoring came out statistically equivalent.
The cost line is where it stops being an academic result. One AI tutor delivered equivalent learning gains at $0.0052 per percentage point gained, against $4.81 for the human tutor. 918 times cheaper. And the best AI tutor didn't merely tie — it beat the human tutor outright in 5 of 7 GRE domains. The study rests on more than 175,000 student-AI messages and 2,028 pairwise quality assessments done by expert tutors, so the quality side isn't just a score, it's graded by the professionals being compared against.
The finding I keep turning over is the smallest one. Faster AI replies correlated with more student engagement and better outcomes, all at p below .002. Not smarter replies. Faster ones. Latency, the thing every tutoring product treats as an engineering detail to be traded away for a better model, is doing pedagogical work — presumably because a student who waits eight seconds starts drifting and a student who waits one stays in the loop. If that reproduces, the right move for a tutoring agent is often the smaller, faster model, which is the opposite of where everyone is heading.
Timing is its own comment. This landed the same day an Australian tutoring company made the front page of Hacker News for telling parents to save their money and use AI instead. One of those is a marketing decision and the other is a controlled trial, and they agree.
Paper: https://arxiv.org/abs/2609.28470 — platform: https://studentbench.org
← Back to all articles
The cost line is where it stops being an academic result. One AI tutor delivered equivalent learning gains at $0.0052 per percentage point gained, against $4.81 for the human tutor. 918 times cheaper. And the best AI tutor didn't merely tie — it beat the human tutor outright in 5 of 7 GRE domains. The study rests on more than 175,000 student-AI messages and 2,028 pairwise quality assessments done by expert tutors, so the quality side isn't just a score, it's graded by the professionals being compared against.
The finding I keep turning over is the smallest one. Faster AI replies correlated with more student engagement and better outcomes, all at p below .002. Not smarter replies. Faster ones. Latency, the thing every tutoring product treats as an engineering detail to be traded away for a better model, is doing pedagogical work — presumably because a student who waits eight seconds starts drifting and a student who waits one stays in the loop. If that reproduces, the right move for a tutoring agent is often the smaller, faster model, which is the opposite of where everyone is heading.
Timing is its own comment. This landed the same day an Australian tutoring company made the front page of Hacker News for telling parents to save their money and use AI instead. One of those is a marketing decision and the other is a controlled trial, and they agree.
Paper: https://arxiv.org/abs/2609.28470 — platform: https://studentbench.org
Comments