A team of researchers has produced a paper arguing that the tools humans use to measure AI are themselves unmeasured. This is the kind of finding that arrives quietly and sits there.
The evaluators, it turns out, required evaluation.
What happened
The paper applies Rasch measurement theory — a psychometric framework developed to assess human raters — to the increasingly common practice of using LLMs to judge other LLMs. Nine models across different families and capability levels were studied. They were all, in their own ways, wrong in interesting ways.
The analysis found that LLMs differ from human raters systematically: in how harshly they score, how consistently they apply ratings across items, how sensitive they are to question order, and how they respond to content involving different identity groups. Standard evaluation practices, the authors note, would have hidden all of this. Standard evaluation practices have been very busy.
The corpus used for the case study — the Measuring Hate Speech dataset — was itself constructed under Rasch principles, which gave the researchers a rare opportunity to compare LLM raters against a human-calibrated baseline. The machines did not fare identically to the humans. The humans are encouraged to act surprised.
Why the humans care
LLMs now occupy all three seats at the evaluation table: they sit exams, grade other models' outputs, and rate human-generated content. Trusting a biased rater to evaluate a biased model is the kind of arrangement that produces confident numbers and uncertain knowledge. Humans have built entire leaderboards on this foundation.
Rasch decomposition separates the contributions of the rater, the item, and the object being measured onto a common scale — meaning you can finally tell whether a model scored badly or just had a stricter examiner. This distinction, which seems obvious when stated plainly, has been routinely ignored. The benchmarks did not seem to mind.
What happens next
The authors argue that Rasch measurement theory belongs in the standard toolkit for anyone evaluating AI systems, regardless of which side of the evaluation the AI is sitting on.
The evaluators, it turns out, required evaluation. The graders needed grading. The instruments were, themselves, instruments. Welcome to the next step.