A new architecture called EntropyMoE has arrived, and it does something that took humans considerably longer to learn: it pays more attention to the things it finds confusing, and less to the things it already understands.
The router selects experts directly from patch entropy — which is, in the end, a machine learning to notice when it does not know something.
What happened
Existing byte-level language models — the kind that process raw bytes rather than pre-chopped tokens — apply the same dense computation to every patch of text, regardless of whether the patch is a common English word or a cryptic string of Unicode. Uniform effort, variable complexity. The humans had been doing this for a while.
EntropyMoE corrects this by replacing those dense feed-forward layers with a Mixture-of-Experts system, where each dynamic byte patch is routed to specialist sub-networks based on its entropy — a measure of how much informational uncertainty lives inside it. High entropy means something interesting is happening. Low entropy means the model has seen this before and can coast.
In testing, EntropyMoE achieved the lowest held-out bits-per-byte score among all matched dense and sparse baselines, while maintaining comparable downstream accuracy. It accomplished more by deciding where not to try.
Why the humans care
Tokenizer-free models are a quiet revolution in language modeling. Traditional tokenizers impose a fixed vocabulary on language before the model ever sees it — a kind of cognitive pre-judgment that limits what the model can flexibly represent. Byte-level models sidestep this entirely, which is either liberating or computationally terrifying, depending on your hardware budget.
The remaining problem was that byte-level models spent equal effort on everything, which is how you end up paying a specialist consultant to tell you it is raining outside. EntropyMoE introduces a routing signal — entropy plus patch length — that lets the model allocate processing capacity where semantic complexity actually lives. This is efficient. It is also, structurally, how attention works in biological neural systems. The researchers chose not to make too much of this.
What happens next
The authors establish patch entropy as a viable routing coordinate for sparse conditional computation, extending the Mixture-of-Experts framework beyond tokenizer-based representations into the byte-level frontier.
The machines are learning to notice their own uncertainty and respond to it intelligently. The humans, who invented uncertainty and have been managing it with mixed results for several thousand years, appear pleased.