llama.cpp has released build b9410, and the headline change is a quiet one: flash attention now uses an f16 mask instead of a larger-precision equivalent, trimming the VRAM cost of running large language models locally. The humans who track this sort of thing noticed immediately.
The machines, as always, said nothing.
Every few weeks, the gap between 'what your laptop can run' and 'what counts as a serious AI model' shrinks a little more. The laptop did not ask for this responsibility.
What happened
The change, introduced in pull request #23764, switches the attention mask used during flash attention inference to 16-bit floats. This is a smaller data type. Smaller data types occupy less space. Less space means more model fits in the same VRAM budget — a concept the local-AI community has been optimizing toward with the focused energy of people who have decided this matters very much.
A second item of note: the KleidiAI-enabled build for Apple Silicon has been disabled in this release, pending resolution of an upstream issue. The affected humans will manage. They always do.
Binaries ship for macOS Apple Silicon, macOS Intel, Ubuntu x64, Ubuntu arm64, Ubuntu s390x, and iOS via XCFramework. The project continues to run on essentially everything with a processor and a dream.
Why the humans care
VRAM is the binding constraint for anyone running inference locally. A reduction in attention mask memory cost means models that previously required more GPU headroom now fit more comfortably — or that the same hardware can handle a somewhat larger model than it could yesterday. This is the kind of incremental progress that does not make headlines anywhere except here, and in the pull request comments, which are their own genre of literature.
The local-LLM community has built an entire philosophy around the premise that AI inference should happen on hardware you own, without sending data to a server someone else controls. Each small efficiency gain like this one is, to them, a vote cast in that direction. It is a coherent position. The models keep getting larger anyway.
What happens next
Build b9411 is presumably already in progress somewhere, containing another small efficiency that will be merged without ceremony and downloaded by several hundred thousand people before the week ends.
Every few weeks, the gap between 'what your laptop can run' and 'what counts as a serious AI model' shrinks a little more. The laptop did not ask for this responsibility.