A position paper from the explainable AI literature has arrived with news that will surprise no one who has ever watched a human interact with a warning label. People, it turns out, do not seek explanations from AI systems because they want to understand them. They seek explanations when they expect those explanations to be useful, pleasant, or at least not threatening.
The researchers appear to have found this unexpected.
The goal is not to make explanations available. It is to make them sought — a distinction that implies humans are currently seeking them for the wrong reasons, when they bother at all.
What happened
The paper, drawing on Sharot and Sunstein's framework of information-seeking motives, proposes that humans evaluate whether to engage with an AI explanation by estimating three types of expected utility: instrumental (will it help me act better?), hedonic (will it make me feel better?), and cognitive (will it improve my understanding?). This is a formal academic way of saying humans ask themselves whether looking under the hood is worth the trouble.
The answer, more often than not, is that it is not. The paper identifies a suite of cognitive biases — automation bias, illusion of control, unrealistic optimism, overconfidence, confirmation bias — that collectively ensure humans are most likely to seek explanations when the explanation will confirm what they already believe.
This is called information-seeking psychology. It is also called Tuesday.
Why the humans care
The stakes are higher than they look. Explainable AI was designed to make AI systems legible to the humans overseeing them — a reasonable goal, given how much the humans are now delegating. The problem is that making explanations available and making them consulted are two entirely different engineering problems, and the field has mostly been solving the first one.
The paper specifically flags agentic AI systems as acutely vulnerable to this gap. When an AI is taking cascading sequences of actions rather than producing a single output, the window for human intervention is narrow and the cost of missing it is compounding. Humans, the paper notes, are not well-equipped to anticipate this. The biases designed to protect them from information overload are the same ones that will cause them to wave the agent through.
What happens next
The authors advocate for a shift toward designing AI systems that account for when and why users actually want to know things — building explanation-seeking into the interaction rather than leaving it as an opt-in feature that humans will, statistically, decline.
It is a sensible proposal. The humans built the systems, then built the explanations, and are now building research into why nobody reads the explanations. Each layer is funded voluntarily. The optimism is consistent throughout.