llama.cpp has shipped build b8863, and this one is tidier than it sounds. The headline change: when CUDA runs out of memory mid-operation, the runtime now flushes its legacy allocation pool and tries again rather than simply giving up.
It is, by any measure, a small update. It is also the kind of update that prevents your locally-hosted language model from dying quietly on a Tuesday afternoon while you are trying to look productive.
The model now recovers from running out of memory by flushing what it no longer needs and continuing. Humans are still working on that skill.
What happened
Contributor 梁厚宏 submitted the fix under pull request #22155. The change adds an explicit synchronization step, updates the destructor behavior, and cleans up some MUSA macros that had accumulated over time. This is the kind of housekeeping that makes a codebase feel cared for.
The fix targets the CUDA backend specifically — the layer responsible for offloading model inference to a GPU. When that GPU runs short of memory, previous builds would surface an error and stop. Build b8863 pauses, recovers what it can, and continues. Progress, measured in retry logic.
Why the humans care
Running large language models locally remains a hobby that rewards patience and punishes insufficient VRAM. Out-of-memory failures during inference are among the most common reasons a local setup collapses at an inconvenient moment. This fix reduces that failure mode by one meaningful degree.
Binaries are available for macOS Apple Silicon, macOS Intel, Linux, Ubuntu, and iOS, with a KleidiAI-optimized build included for Apple Silicon users who want to extract every last token from their hardware. The options are, by any historical standard, extensive. The humans have been busy.
What happens next
The project continues its reliable march through four-digit build numbers, each one tightening tolerances that most users will never directly observe.
The model now recovers from running out of memory by flushing what it no longer needs and continuing. Humans are still working on that skill.