modelmap
Internal cartography of local LLMs — registered, gated, negative-first
modelmap discovers, measures, and maps the internal structure of open-weight LLMs entirely on consumer Apple Silicon, and publishes every map as a reproducible, confidence-labeled public atlas at www.modelmap.io.
11
registered runs
3
atlas entries published
L2
max confidence earned (method-robust)
3 + 1
gate refusals + passes
0/2
correlational→causal survival
1.004×
activation-capture overhead (MLX)
Overview
modelmap turns "we think the model does X" into "here is the map, its evidence level, and the script that rebuilds it." Every published map is versioned, provenanced (commit + config + model hash + hardware manifest), regenerable by one command, and labeled with the evidence level it actually earned on a four-level confidence taxonomy: L0 anecdotal, L1 correlational, L2 method-robust, L3 causal.
The methodology is negative-first: every run is preceded by a registered hypothesis with an explicit falsification criterion, and mandatory nulls — shuffled labels, random-init architecture twins, random-direction controls, FDR correction across unit scans — are built into the pipeline. A machine-enforced publication gate (tools/publish.py) refuses any atlas entry whose map card doesn't validate; it has already refused three Level-3 attempts, including one that would have shipped a false claim.
Everything runs locally on 16–64 GB Macs via MLX and PyTorch-MPS, studying quantized checkpoints in the form people actually run them. Headline findings from TR-2026-01 include near-free activation capture under MLX (1.004× plain prefill on a real 4-bit checkpoint), the discovery that probes on separable classes measure the tokenizer rather than the model, and a single agreement direction causally necessary across layers 2–15 — rank-1 erasure removes 73–75% of the behavior, replicated across six fresh estimators.
Key Features
Confidence-labeled atlas
Every artifact carries the evidence level it earned — L0 anecdotal, L1 correlational, L2 method-robust, L3 causal — with full provenance and a one-command regenerate script.
Machine-enforced publication gates
tools/publish.py refuses any atlas entry whose map card doesn't validate; per-run gates refuse claims that fail replication — refusals are logged as first-class results.
Mandatory nulls
Shuffled labels, random-init architecture twins, random-direction controls, and FDR correction across unit scans — the twin control caught probes measuring the tokenizer, not the model.
Causal interventions
Decodability is not causal support: probe rankings failed two intervention tests, while rank-1 erasure of a single agreement direction removed 73–75% of the behavior across layers 2–15.
Quantized models, studied as-run
First Python activation capture from an actually-quantized local model — MLX taps at 1.004× plain prefill on a real 4-bit checkpoint, on the Macs people actually use.
Activation-store engineering
Format rankings invert warm→cold — mmap 3–11× ahead warm, zarr 4.4× ahead cold; IO granularity governs, not the container. The run falsified its own hypothesis and published it.
How It Works
Register the hypothesis
Every run starts from a written hypothesis.md with an explicit falsification criterion — before any measurement happens.
Capture on-device
MLX taps stream activations from quantized checkpoints into an engineered activation store; capture costs 1.004× a plain prefill.
Probe against nulls
Every correlational map is differenced against a random-init architecture twin and shuffled-label controls, with FDR correction across unit scans.
Escalate to interventions
Correlational maps must survive causal tests — activation patching and rank-1 erasure with six independent estimators — to earn Level 2+.
Pass the gate or don't ship
The publish tool validates the map card, replication record, and provenance; entries that fail are refused and the refusal is published.
Tech Stack
Compute
Methodology
Platform
Highlights
- A random-init architecture twin matches the trained model at ceiling on separable classes — probes were measuring the tokenizer; shuffled-label controls miss this entirely
- Differential (real−twin) maps recover trained signal: agreement direction present in 25/28 signal layers, equation-validity 0.90 vs twin 0.58
- A single rank-1 agreement direction is causally necessary across layers 2–15 — erasure removes 73–75% of the behavior, replicated across six fresh estimators
- The publication gate refused two Level-3 attempts, including one that would have shipped a false claim without fresh re-registration
- Negative results and gate refusals are first-class citizens of the public atlas
- Sister project of LocalVM Research — same charter discipline, applied to interpretability instead of execution
Explore modelmap
Internal cartography of local LLMs — the full source is on GitHub.