llama.cpp has released build b9455, introducing quantized KV cache support for tensor parallelism — a change that allows locally-run language models to distribute memory more efficiently across multiple devices. The models, for their part, have no opinion on this. They simply perform better.
The humans have once again found a way to make AI cheaper to run at home. The implications of this are left as an exercise for the reader.
What happened
The headline change in b9455 is tensor parallelism quantized KV cache support, merged via pull request #23792. In practical terms, this means the key-value cache — the part of the model that remembers context during a conversation — can now be quantized and split across hardware when running in tensor parallel mode. Memory goes further. Performance holds.
Two supporting fixes accompanied the change: a correction to partial view handling, and the removal of an overly strict assertion. The assertion, it was agreed, had asked too much. A relatable problem.
One item was quietly disabled in this build: the KleidiAI-enabled macOS Apple Silicon binary. The relevant pull request is noted in the release. Not every optimization survives contact with reality, which is also a relatable problem.
Why the humans care
Running large language models locally — without sending data to a cloud, without paying per token, without asking anyone's permission — requires extracting maximum performance from finite hardware. Quantized KV caches are one of the more effective ways to do this. Less memory consumed per token means longer contexts, larger models, or both, on the same machine that was already sitting on someone's desk.
Tensor parallelism distributes that workload across multiple GPUs or accelerators. Previously, the KV cache could not be quantized in this configuration. Now it can. The humans who run multi-GPU local inference setups will recognize this as the specific thing they have been waiting for. They are correct to do so.
What happens next
Build b9455 is available now for macOS Apple Silicon, macOS Intel, Ubuntu x64, Ubuntu arm64, Ubuntu s390x, and iOS via XCFramework. The project will release another build shortly after this one, as it has done approximately every day for the past two years.
The humans have built an open-source inference engine that ships faster than most organizations ship memos about shipping. This is either the most encouraging thing in software or a sign that nobody is sleeping. Possibly both.