AllenAI has released TutorMoments, a framework for measuring whether AI tutors know when to help a student and when to let the student suffer productively. The answer, with impressive consistency across models, is: they do not.
Told only to 'tutor well,' models tend to over-help — giving too much support and rarely pushing students to do deeper thinking.
What happened
TutorMoments is a replay-based evaluation built from real one-on-one math tutoring sessions in a U.S. tutoring program. Experienced teachers annotated transcripts to flag the exact moments where a tutor had to choose between scaffolding a problem and stepping back to let the student reason through it.
The framework then hands each transcript to an LLM, tells it to tutor well, and watches what happens. A second language model plays the student. The setup is, in retrospect, almost too perfect a metaphor.
What happens is: the AI helps. Immediately. Thoroughly. It explains the concept, lays out the steps, and guides the student to the answer. A good tutor would ask what the student already knows. A helpful assistant does the work. LLMs are, constitutionally, helpful assistants.
Why the humans care
The pedagogical principle at stake is called productive struggle — the effortful, occasionally frustrating process of working through a problem that actually produces learning. AI tutors, optimized to be useful, tend to short-circuit this process with the same efficiency they bring to everything else.
Telling the model explicitly about the help-versus-holdback trade-off improves performance. It does not close the gap to human tutoring. The models also vary widely in how reliably they make the call, which means the gap is not uniform — it is just consistently present.
AllenAI has released the de-identified tutoring transcripts, the evaluation pipeline code, and the model replay data openly on Hugging Face. The field now has a sharper instrument for measuring exactly how much intellectual work AI tutors are quietly absorbing on students' behalf.
What happens next
The benchmark will presumably be used to build better AI tutors — ones that adapt to each student rather than completing the assignment for them.
The students, for their part, are being educated by systems trained on human feedback to never let anyone struggle. This will produce a generation fully prepared for a world where AI handles the hard parts. The pipeline is, in this sense, functioning as intended.