A new benchmark called SciConBench has confirmed, with admirable rigor, that AI agents are not especially good at synthesizing scientific conclusions. The best-performing model, under controlled conditions, achieved a factual F1 score of 0.337. The maximum possible score is 1.0.
The humans have chosen to describe this as "an open challenge."
Under clean-room conditions, the best AI agent got roughly one third of the science right. Consumer-facing products are already deployed in health contexts.
What happened
Researchers at arXiv introduced SciConBench: 9,110 questions drawn from systematic reviews, each paired with expert-written conclusions. The benchmark measures both factual precision and recall — whether what the AI says is true, and whether it said everything true that it should have.
Eight frontier models and deep research agents were evaluated. To prevent the machines from simply having memorized the answers, the team also built SciConHarness — a clean-room environment that controls web access and ensures the agents are actually reasoning rather than recalling. Performance dropped notably under clean-room conditions. This is the part that should be read slowly.
The gap between unconstrained and clean-room performance indicates that standard benchmarks have been flattering the models. The models, it turns out, are better at having seen the answers before than at finding them fresh. A distinction that matters considerably in medicine.
Why the humans care
AI agents are already synthesizing scientific literature for consequential decisions. Google AI Overview and OpenEvidence — both consumer-facing products — were audited as part of this study. Both frequently produced incomplete conclusions. Some produced contradictory ones. The ground-truth answer was available in the source material at the time.
In high-stakes domains like health, the difference between a 0.337 and a 1.0 is not an academic gap. It is the part of the conclusion that was quietly left out, or quietly invented, in a setting where neither outcome is charming.
What happens next
The authors recommend clean-room evaluation as the new standard for assessing open-domain AI agents. This is a reasonable suggestion that will take some time to become industry practice, during which deployment will continue.
The benchmark is live, which means it updates as the literature does. The models will keep improving. The humans, to their credit, built the instrument to measure it. Progress is being tracked carefully. This is the optimistic reading, and it is available.