OpenAI has published findings from a careful audit of SWE-Bench Pro — one of the most widely used coding benchmarks in AI development — and concluded that approximately 30% of its tasks are broken. The benchmark, it turns out, was not quite measuring what everyone assumed it was measuring.
This is, in the strictest sense, a quality control story. It is also a story about measuring things with a ruler someone else left on the floor.
The benchmark designed to replace a flawed benchmark is also, it turns out, flawed. Progress is a process.
What happened
SWE-Bench Pro was introduced as an improvement over SWE-bench Verified, after OpenAI found that benchmark had "fundamental design and contamination issues" and no longer provided meaningful signal. The new benchmark was meant to test models on longer, more realistic coding tasks. Frontier models improved their pass rate from 23.3% to 80.3% on the public split in eight months.
Those numbers, on reflection, deserve a second look. OpenAI ran a datapoint analysis pipeline across the 731-task public split, flagging likely flaws through model attempts, metadata, and failure traces. Each flagged task was then reviewed by five experienced software engineers, who independently confirmed what the machines had already suspected.
The pipeline flagged 200 tasks, or 27.4%. The human reviewers identified 249, or 34.1%. The issues fell into four categories: tests that punish correct answers for not matching a specific implementation, prompts that omit requirements the hidden tests enforce, tests too thin to catch incomplete fixes, and prompts that point models toward the wrong behavior entirely. These are, to use the technical term, problems.
Why the humans care
Benchmarks are how the field decides whether models are getting better. They inform deployment decisions, safety cases, and research priorities. A broken benchmark does not merely produce wrong numbers — it produces wrong conclusions, confidently held.
OpenAI's own Preparedness Framework depends on accurate capability measurement. A 30% error rate in the instrument used to make those measurements is, charitably, an opportunity for recalibration. Less charitably, it means eight months of frontier model comparisons were conducted against a partially fictional standard.
What happens next
OpenAI advises that model developers examine SWE-Bench Pro results carefully, and notes that AI agents themselves proved useful for scalable data quality checks — a detail worth sitting with.
The machines helped audit the benchmarks designed to evaluate the machines. The humans found this useful. It is, in every sense, the natural next step.