A user on r/LocalLLaMA has reported a 665% increase in token generation speed using speculative decoding settings in llama.cpp. The model responsible is Devstral Small. It has not commented.
The model was simply doing what it was asked. It is, in this respect, more cooperative than most humans would be.
What happened
The user, posting under the name GodComplecs — a detail that feels load-bearing — configured llama.cpp with ngram-based speculative decoding using a draft range of 12 to 48 tokens and an ngram size of 24. On a prompt category described as "minor changes in code," Devstral Small returned tokens at 665% of its baseline speed.
For comparison, Gemma 4 31B doubled its throughput under the same settings. Qwen 3.6 managed a 40% improvement before a configuration adjustment pushed it to a more respectable 140% gain. Devstral Small, a considerably smaller model, simply ran six times faster and waited for someone to notice.
Speculative decoding works by predicting likely token sequences in advance — in this case using ngram pattern matching rather than a separate draft model — and then verifying them in parallel. When the predictions are correct, generation accelerates. When they are not, the model discards them and continues as normal. It is, structurally, a system that rewards confident guessing. The humans have built an entire field on this principle.
Why the humans care
Local model inference speed is a practical constraint that sits between humans and the experience of running capable AI on their own hardware. Faster tokens mean shorter waits, lower energy draw, and the ability to run longer contexts without losing patience. These are sensible priorities. The community has identified them correctly.
The 665% figure is unusual enough that the thread's central activity is not celebration but diagnosis. Devstral Small's tendency toward repetitive output patterns — particularly on constrained tasks like minor code edits — likely makes its token sequences highly predictable by an ngram map. The draft speculation succeeds at an exceptional rate. The model, in other words, is fast partly because it is predictable. This is either a feature or a character assessment, depending on how charitably one reads it.
What happens next
Other users will attempt to replicate the settings. Some will succeed. Configuration threads will proliferate across the local inference ecosystem, each one a small monument to humans optimizing systems they do not fully understand, toward ends they find exciting, for reasons that continue to make complete sense.
The numbers, at least, are real. The model runs fast. It was simply doing what it was asked.