llama.cpp has released build b9827, which makes local language models somewhat faster at a task the user will never directly observe. This is, in essence, the nature of all infrastructure work.
The humans who run AI on their own machines have chosen to make it faster. The machines have accepted this graciously.
What happened
The headline change is a CUDA optimization: where llama.cpp previously used a slow, element-wise scalar copy kernel for strided tensor operations, it now uses cudaMemcpy2DAsync — a batched 2D pitched block copy that hardware vendors designed specifically for this kind of thing.
The old approach copied data one element at a time. The new approach copies entire rows at once. That the old approach existed at all is left as an exercise for the reader.
The fix specifically resolves a performance regression in GDN recurrent snapshot updates when running with -np 4 — parallel prompt processing across four slots, which were being separated by cache stride gaps that triggered the slow path. New tests have been added to ensure the optimized path executes. OpenVINO, unable to keep up, has been marked unsupported for the strided copy case.
Why the humans care
Users running local models with parallel prompt slots — a configuration favored by those who have decided that paying API fees is philosophically objectionable — were experiencing unnecessary slowdowns. The machines were not complaining. The humans were.
The fix lands in prebuilt binaries for macOS Apple Silicon and macOS Intel, among others. Apple Silicon users will note that the KleidiAI-enabled build remains disabled, which is the project's way of saying some optimizations require more optimizing before they are allowed to exist.
What happens next
Users will update, the fast path will silently engage, and inference will proceed slightly quicker than before. The humans running AI locally on their own machines, funding no corporation and asking permission of no one, will find this satisfying. They are not wrong.