LocalVM Research
Running LLMs larger than memory on a consumer Mac
Falsification-driven research into post-training execution representations whose instantaneous working set is far smaller than the checkpoint — every hypothesis preregistered, every negative result published, live at www.localvm.dev.
8
preregistered experiments completed
3
refutations kept & published
9×
fewer bytes than the checkpoint
13.1 GB/s
measured SSD device ceiling
0.85–0.98
AUROC — margins predict disagreement
304
sources in the bibliography
Overview
LocalVM Research asks one question: can an existing pretrained LLM whose memory requirements exceed a consumer Apple Silicon Mac be transformed post-training into an execution representation whose instantaneous working set is dramatically smaller than the full checkpoint — while preserving most of its capabilities? The project's central distinction: total model size ≠ resident model size ≠ bytes read per token ≠ parameters required for this token.
It is run under a strict research charter: hypotheses are registered with explicit kill criteria before experiments run, no result is reported from an uncommitted tree, every result file embeds a hardware manifest, and negative results are kept and published. Three refutations are load-bearing — they killed the fashionable dynamic-sparsity-paging design (à la DejaVu / LLM-in-a-flash) on modern dense architectures with numbers, leaving batch-amortized verification as the only surviving route.
The surviving architecture is margin-gated deferred refinement: a resident q4 base generates greedily and records each decision's top-1 logit margin; low-margin decisions are deferred and periodically verified by the same checkpoint's q8 variant — streamed layer-by-layer from SSD at ~11.6 GB/s when it cannot co-reside. The result is a quality–bytes continuum: q8-equivalent output at 9× fewer bytes than the checkpoint, with τ as a runtime knob that prices output quality in verification bytes.
Key Features
Falsification-driven charter
Every experiment has a preregistered hypothesis.md with an explicit kill criterion before it runs; the full audit trail lives in an append-only research log.
Margin-gated deferral
The q4 base records each decision's top-1 logit margin — a free byproduct of the forward pass — and defers low-margin tokens without ever blocking generation.
Out-of-core verification
When the q8 verifier cannot co-reside in 48 GB, it streams layer-by-layer from SSD — one sequential pass per sweep, exactly the access pattern the measured envelope rewards.
Rollback on flip
On the first checked disagreement the runtime trims both KV caches, accepts the q8 token, and resumes — amortized sweeps every W tokens through a persistent verify-KV-cache.
A quality–bytes knob
τ prices output quality in verification bytes: 150/211/237 MB/token buys 47%/83%/~100% of the q4→q8 quality gap — a property no static quantization format offers.
Measurement traps, documented
Two macOS traps measured and published: F_NOCACHE does not evict resident pages, and Metal prefill vs incremental decode flips 1.56%/token of a model's own greedy argmax.
How It Works
Resident base
The q4 quantization of the model lives in unified memory and generates greedily, recording each decision's top-1 logit margin as a free byproduct of the forward pass.
Deferral
Decisions with margin below τ are flagged for later verification; generation never blocks on them.
Amortized verification sweeps
Every W tokens, the same checkpoint's q8 variant teacher-forces the window in one pass — streamed layer-by-layer from SSD (materialize → compute → re-lazify) when it cannot fit in memory.
Rollback
On the first checked flip the runtime trims both KV caches, accepts the q8 token, and resumes — keeping the output statistically equivalent to resident-q8.
Tech Stack
Compute
Research infrastructure
Platform
Highlights
- Verify-all output statistically equal to resident-q8 (bf16 judge −0.1297 vs −0.1328) at 237 MB/token — 9× under the checkpoint
- q8-32B (34.8 GB) cannot co-reside on 48 GB — streamed at ~11.6 GB/s it adds +0.28 nats over the only fitting alternative at 1.69 tok/s
- Three escalation routes refuted with numbers: layer sensitivity spread only 1.5×, FFN energy diffuse, 128-token working-set union ≈ 99.9%
- Naive affine 2-bit measured dead: 3% greedy agreement, KL ≈ 11.9 — the 2–3-bit cliff, quantified
- SSD envelope mapped: 4 KiB QD1 random = 67 MB/s vs 1 MiB QD8 = 13.8 GB/s — a 200× spread that dictates the architecture
- An adversarial 45-source prior-art search is part of the repo — novelty honestly stated, every component anticipated, the composition new
Explore LocalVM Research
LLMs larger than memory, on a Mac — the full source is on GitHub.