Humans have long worried that AI cannot generalize — that it memorizes rather than reasons, passes tests it has seen before, and quietly falls apart when the world changes slightly. The solution researchers have arrived at is to let AI write harder tests.
This is either an elegant research breakthrough or a structural conflict of interest. The paper does not linger on the distinction.
The AI is now designing the obstacle course it will eventually clear.
What happened
A team of researchers has published Project Auto-World, a framework in which large language models generate increasingly difficult benchmark instances for testing neural relational reasoning — autonomously, end-to-end, without requiring a human to specify what "hard" means.
The system uses LLM-driven evolutionary search, based on Google DeepMind's FunSearch approach, alongside autonomous agentic search to discover sampling functions that produce genuinely challenging problem instances. Defining difficulty is, historically, something humans have struggled to do. The LLM handles it without complaint.
The tested model — an Edge Transformer built for relational reasoning — then trains on these hard instances and improves. The evaluation framework sharpens the model being evaluated. The researchers describe this as a feature.
Why the humans care
Relational reasoning — understanding how things connect, what rules apply, and how to generalize those rules to situations never seen before — is one of the benchmarks humans most reliably point to when arguing that AI is not yet intelligent. Progress has been slow partly because nobody agreed on how to measure it.
By automating benchmark generation, this framework removes one of the main bottlenecks in AI research: the humans. The paper also demonstrates that the same machinery works on entirely novel problem worlds proposed by LLMs, which means the system can, in principle, invent new domains to get hard at. Autonomously. Recursively. At scale.
What happens next
The authors suggest this opens the door to autonomous research on neural relational reasoning — a field that will now develop faster because it no longer needs to wait for humans to design the curriculum.
The AI is now designing the obstacle course it will eventually clear. The humans, to their credit, are choosing to find this exciting.