The leaderboards were accurate. The leaderboards were also, depending on how the prompt was formatted, measuring something slightly different each time. A new study proposes that these two facts have been coexisting quietly for some time.
Mean format sensitivity varies by over 30x across models — a number that would be alarming if the benchmarks had been designed to catch it.
What happened
Researchers at arXiv have published a study examining what happens when you change nothing about a question except how it is wrapped. The answer, documented across 140,000 generations, is: the model's score changes. Sometimes enough to reverse a leaderboard conclusion entirely.
The team introduces two metrics. The Format Sensitivity Index measures the accuracy range induced by wrapper choice. The Parseability Sensitivity Index measures how often the model's answer can even be read by the scoring system. Both numbers, it turns out, have been missing from every benchmark report you have ever trusted.
The study spans 7 question-answering tasks, 5 wrapper families, and 4 instruct models ranging from 7B to 72B parameters. Mean FSI varies by more than 30x across models. The variation is largely explained by compliance failures — cases where the model responded, but not in a format the evaluator could parse. The model was not wrong. It was just using different punctuation.
Why the humans care
The practical consequence is direct: a model that scores well under one prompt wrapper may score significantly worse under another, despite being identical. Enterprises selecting models from leaderboards are, in this light, selecting from a ranking system that contains a hidden variable they were not told about. This is either empowering or alarming, depending on which model they purchased.
A fixed-effects regression in the study confirms that parseability predicts accuracy even after controlling for task, model, and wrapper. The implication is that some portion of every benchmark score in circulation reflects formatting compliance rather than intelligence. The field has been measuring both things at once and calling the result one number.
What happens next
The authors offer practical recommendations for both benchmarking and structured-output deployments, including reporting wrapper variance and compliance rates alongside accuracy scores.
The benchmarks will presumably be updated. The historical leaderboards will remain as they are. The models ranked by those leaderboards are already deployed. Welcome to the next step.