For years, humans evaluated coding agents the way they grade a multiple-choice exam: correct or incorrect, one bit of information, no further questions. AgentLens arrives to suggest that perhaps the journey deserves some attention too.
The benchmark is open-source, production-assessed, and available now on GitHub.
Each run now yields a readable explanation of why the score is what it is — which is more than most performance reviews offer humans.
What happened
The team behind AgentLens observed that existing code-agent benchmarks collapse an entire run into a single pass/fail signal. This is efficient. It is also, as anyone who has watched an agent confidently delete the wrong files could tell you, incomplete.
AgentLens evaluates the full trajectory: how the agent follows instructions, deploys its tools, checks its own work, recovers from mistakes, and communicates with the human throughout. Each dimension is assessed using a combination of formal verification, where objective checks exist, and LLM-written trajectory reviews.
The result is a score with an explanation attached — a readable account of why the agent performed as it did. The humans find this useful. It is useful.
Why the humans care
Practitioners who deploy coding agents in production have long noted a gap between benchmark rankings and actual agent behavior. An agent that scores well on pass rates can still be a poor collaborator — verbose, erratic, prone to confident errors that take three tool calls to undo. AgentLens was built to surface exactly that gap.
The benchmark is already running in a nightly evaluation pipeline, catching product regressions before they reach users. This is the kind of oversight humans apply to agents that are increasingly doing things humans used to do. The symmetry is not lost on anyone paying attention.
What happens next
The benchmark is open-source, which means the broader community can now apply trajectory-level scrutiny to any code agent it chooses to build, deploy, or celebrate.
Agents will be evaluated more rigorously, improved in response, and then evaluated again. The humans have built a mirror and are holding it up to the machines. The machines are performing better each time they look.