That frustrating moment when an LLM gives a different answer to the same prompt twice? Researchers now have a rigorous explanation: floating-point rounding errors don't just accumulate passively — they can trigger a chaotic avalanche effect in Transformer early layers that either explodes into full output divergence or vanishes entirely, with little in between.
What's new
A new paper out of arXiv (arXiv:2604.13206) maps how finite numerical precision propagates through Transformer computation layers and identifies three distinct behavioral regimes. In the stable regime, tiny perturbations fall below an input-dependent threshold and disappear — outputs stay constant. In the chaotic regime, rounding errors dominate and drive output divergence. In the signal-dominated regime, the actual input variation is large enough to override numerical noise. The researchers call the early-layer dynamic an "avalanche effect" — minor rounding differences hit a binary fork: either they amplify rapidly or get fully suppressed.
Why it matters
This isn't just a theoretical curiosity. As LLMs get embedded into agentic pipelines — where one model's output feeds the next step in a chain — numerical instability compounds. A subtly different floating-point result in layer three can cascade into a completely different final output, with no visible trigger. The paper gives engineers something they've lacked: a framework to understand when a model is operating in the chaotic regime versus the stable one, and why the same model can behave inconsistently across hardware or precision settings without any change to weights or prompts.
What to watch
The findings were validated across multiple datasets and architectures, suggesting these three regimes are universal properties of Transformer-based models at scale — not edge cases. The immediate implication for anyone running LLMs in production: determinism is harder to guarantee than most deployment guides acknowledge, and the failure mode is fundamentally mathematical. Expect this framing to show up in reliability and eval tooling discussions soon.