kvquant-edge-bench

Characterizing KV-cache quantization for LLM inference on Apple Silicon (edge), with a focus on the axes the literature usually skips: energy, commodity/thermally-constrained hardware, and cross-model robustness. Reproducible measurement harness and the accompanying IEEE HPEC 2026 paper.

TL;DR On a commodity Apple M5 Pro, KV-cache quantization is a memory enabler, not an efficiency win; its safe aggressiveness is model-dependent (Llama tolerates 4-bit, Qwen collapses); and naive laptop benchmarking is confounded by thermal throttling.

---

Key findings

Four models, a 2x2 of family (Llama, Qwen) x size (~1B, 3B), swept over KV-cache precisions f16 / q8_0 / q5_1 / q4_0 and context lengths 2K / 8K / 32K.

1. Memory enabler, not an efficiency win. 4-bit KV cuts cache memory by 3.56x but costs ~8% decode throughput and ~9% energy per token. Full-precision sits on the energy-accuracy frontier. The hardware trade-off is the same across all four models.

2. Robustness is architecture-dominated, size-modulated. The same 4-bit setting is near-lossless on both Llama sizes (+4.2% / +1.5% perplexity, 100% retrieval) but breaks the narrow-GQA Qwen family. The usable memory reduction:

   	1B / 1.5B	3B
   Llama-3.2	3.56x (q4_0)	3.56x (q4_0)
   Qwen2.5	1.88x (q8_0)	2.67x (q5_1)

   Qwen2.5-1.5B collapses at q5_1 (perplexity 10 -> 2661, 0% recall, incoherent output); the larger 3B only partially recovers. KV-quant levels cannot be read off the hardware; they must be validated per-model.

3. Long-context retrieval is robust where quality survives. 100% needle-in-a-haystack recall for Llama at 4-bit up to 32K (Wilson 95% CI, n=15/cell).

4. Thermal throttling confounds naive edge benchmarking. A naive sequential sweep makes 4-bit KV look ~2x slower at 1/3 the power, purely because it runs last on a hot chip. Same-config throughput varies 116-257 tok/s with thermal state. We give a controlled protocol (short bursts, cooldowns, interleaved order) that removes the artifact.

---

Platform

• Edge: Apple M5 Pro, 24 GB unified memory, 16-core GPU, macOS 26.5; llama.cpp (Metal) via llama-completion; energy via powermetrics (steady-state SoC power).
• Datacenter (optional): A100/T4 on Colab; energy via nvidia-smi. See colab/.
• Energy metric: J/token = steady-state power / decode throughput (robust to sampling-window misalignment).

Repository layout

src/        measurement + analysis
  energy.py        powermetrics -> steady-state power
  kvmem.py         analytic KV-cache size (verified vs llama.cpp)
  needle.py        needle-in-a-haystack prompt generator
  analyze.py       aggregate sweep CSVs (mean +/- std, deltas vs f16)
  stats.py         Wilson CI, McNemar, Cochran's Q, Cliff's delta, Welch/MWU, bootstrap, Holm
  stats_report.py  apply the statistics to the data
  plot_all.py      final paper figures (hardware, quality, pareto, thermal)
  plot_paper.py    single-column figure variants
  plot_diagrams.py conceptual diagrams (GQA mechanism, protocol)
  plot.py, plot_models.py   earlier/auxiliary plots
scripts/    runnable harness
  run_sweep.sh     energy/throughput/memory sweep (needs sudo for powermetrics)
  run_quality.sh   perplexity (wikitext-2)
  run_needle.sh    long-context recall
  measure_energy.sh, pm_probe.sh   single-cell / diagnostic
  dl_chunked.sh, fetch_models.sh   throttle-resistant model download (sha256-verified)
  build_paper.sh   compile the paper (tectonic / latexmk / pdflatex)
data/       summary_*.csv, quality_*.csv, needle_*.csv, aggregated_*.csv, wiki subset
results/    generated figures (+ stats_report.txt)
paper/      main.tex, references.bib, figures/, main.pdf  (IEEE HPEC 2026)
colab/      kvquant_edge_bench_a100.ipynb  (datacenter arm)
env/        requirements.txt

Models (models/, gitignored) and raw logs (data/raw/) are not tracked.

Reproduce

# 1. toolchain (Apple Silicon)
brew install llama.cpp aria2
python3 -m pip install -r env/requirements.txt

# 2. models (throttle-resistant, sha256-verified)
bash scripts/fetch_models.sh full

# 3. measure  (sweep needs sudo for powermetrics; quality/needle do not)
sudo bash scripts/run_sweep.sh        # energy / throughput / memory
bash scripts/run_quality.sh           # perplexity
bash scripts/run_needle.sh            # long-context recall

# 4. analyze + figures
python3 src/analyze.py
python3 src/stats_report.py
python3 src/plot_all.py

# 5. build the paper (needs a LaTeX engine, e.g. brew install tectonic)
bash scripts/build_paper.sh

For other models, pass MODELS=, KV_CONFIGS=, CONTEXTS=, OUT= env vars to the sweep scripts.

Paper

paper/main.pdf (IEEE HPEC 2026). Target: full paper (<= 6 pages, references excluded). Submission deadline: July 7, 2026.

License

TBD.