feat(desktop): vendor-neutral GPU acceleration (Vulkan) with first-run probe and CPU fallback #155

Activity

1. zaxbysauce commented on Oct 7, 2026

   

   zaxbysauce

   on Oct 7, 2026

   CollaboratorAuthor

   More actions

   Benchmark results (2026-10-07), from the harness at E:/ClaudeCode/trainingapp-bench (outside the repo). 40 fixed-context grounded questions from the bundled opmed packs (26 single-fact, 6 two-fact, 8 not in the sources, where declining is correct), retrieval held constant (5 chunks, ~1.4-1.5k prompt tokens), greedy decoding, max 256 tokens, node-llama-cpp 3.20.0 (the version the app ships). Machine: Ryzen 9 5950X 16c/32t, 128 GB, Intel Arc Pro B50 (Vulkan); CPU runs at 16 threads with ~20-25 % background load from other processes. Accuracy is the string-match score adjusted by hand-reading the answers; an independent reviewer reproduced every number from the raw data and its corrections are applied. One question = 2.5 points, so gaps of 1-2 questions are noise.

   GPU (Vulkan) vs CPU, same models and questions

   Model	Accuracy	GPU p50 / p90 per answer	GPU prefill / decode tok/s	CPU p50 per answer	CPU prefill / decode tok/s
   Gemma 4 E2B (reasoning off)	95 %	1.3 s / 1.7 s	1615 / 59	209 s (10-question subset)	7 / 4
   Qwen3.5-2B	97.5 % GPU / 95 % CPU	0.85 s / 1.08 s	2734 / 77	63 s	24 / 8
   MiniCPM5-2B	95 %	1.10 s / 1.58 s	1977 / 66	86 s	17 / 6
   LFM2.5-1.2B-Instruct	87.5 %	0.49 s / 0.73 s	4587 / 148	33 s	48 / 16
   LFM2.5-VL-450M (current fast model)	77.5 %	0.23 s / 0.32 s	9928 / 243	10.7 s	146 / 34

   Findings:

   • The Vulkan backend shipped in the packaged app (@node-llama-cpp/win-x64-vulkan, node-llama-cpp 3.20.0) loads and runs every tested model on the Intel Arc Pro B50 with all layers offloaded; no crashes in any of the 8 reported GPU runs (320 answers). The first GPU call pays ~7 s of one-off warmup.
   • For the shipped quality model (Gemma 4 E2B) the GPU is ~160x faster per answer than this CPU (1.3 s vs ~209 s), and every 1-2B candidate answers in about a second.
   • Accuracy is the same on both backends within noise (greedy decoding differs slightly between Vulkan and CPU numerics).
   • Not yet tested: Intel/AMD integrated GPUs, AMD and NVIDIA discrete GPUs, low-VRAM devices, and running two contexts at once. The probe-and-fallback design in this issue is still needed for those.

   🤖 Generated with Claude Code
2. zaxbysauce commented on Oct 8, 2026

   

   zaxbysauce

   on Oct 8, 2026

   CollaboratorAuthor

   More actions

   Follow-up to the 2026-10-07 benchmark comments above. Decision (user): LFM family only, two tiers. Qwen3.5-2B, Gemma 4 and MiniCPM5 are out, which supersedes the Qwen recommendation in the earlier comment. Every deployment target has at least an integrated GPU (floor: 12th-gen Core i5 mobile, Iris Xe), so Vulkan is the primary path and CPU is the fallback. Same harness as before: 40 fixed-context grounded questions (26 single-fact, 6 two-fact, 8 not-in-sources), greedy, max 256 tokens, grounded system prompt, repeat penalty off, node-llama-cpp 3.20.0. Proxy hardware: Ryzen 9 5950X (16 threads) and Intel Arc Pro B50 (discrete) over Vulkan. Two independent review rounds reproduced every number from the raw data; their corrections are applied. One question = 2.5 points.

   Not measured: the target integrated GPU. All GPU timings below are from a discrete Arc B50. Expect a 12th-gen i5 Iris Xe to be several times slower, especially on prefill. A real measurement on a floor-spec laptop is required before the defaults are finalised.

   GPU path for the LFM lineup

   Model	Vulkan (Arc B50) p50 / p90, 3 chunks	Prefill / decode tok/s	Device memory	CPU p50 (16 threads), 3 chunks
   LFM2-1.2B-RAG (fast)	0.36 s / 0.58 s	~4,600 / ~145	~0.9 GiB	21.4 s
   LFM2.5-2.6B, reasoning suppressed (quality)	0.76 s / 1.04 s	~2,100 / ~75	~1.9 GiB	49.0 s

   • All LFM runs loaded on Vulkan with every layer offloaded, with no crashes or errors. Accuracy matched CPU within numerical noise.
   • Prefill dominates CPU time (90–94 % of wall time), and an iGPU is expected to be prefill-limited too. Hence the 3-chunk fast preset proposed in feat(desktop): replace the fast (and possibly quality) model, chosen by benchmark #156.
   • Proposed tier rule: the 1.2B fast model always works, including on CPU fallback. The 2.6B quality model is offered when the probe finds a working GPU that meets a measured speed bar. The bar should be set from real Iris Xe numbers, not from the Arc.

   What the first-run probe must guard against (Intel Vulkan, llama.cpp issues found in research)

   • #29277 (Windows, Intel iGPU, open, filed against b10361, the llama.cpp build in node-llama-cpp 3.20.0): device memory reports free > total. This can break automatic GPU-layer fitting, so pass gpuLayers explicitly and sanity-check memory reports.
   • #28648 (Windows, Arc 140V): garbage output with layers on GPU, depending on batch settings. This is why the probe must check output sanity, not just load success: a fixed prompt with a known answer.
   • #27638 (Arc B580): quadratic prompt-processing slowdown, and device loss at ubatch ≥ 2048. Use ubatch ≤ 512–1024.
   • #29054 (UHD 630): deterministic GPU hang with a q8_0 KV cache. Use an f16 KV cache on Intel iGPUs.
   • #27634 (Linux i915, UHD 770): the kernel watchdog silently cancels submissions that run longer than 20 s, producing garbage output. It is Linux-specific, but long single graphs on slow iGPUs are a risk. Keep prompts and ubatch small.
   • Run the probe in a child process, so a driver crash cannot take down the app. Persist the result and reason, re-probe after a driver or hardware change, and fall back to CPU with the 1.2B model.

   Still needed

   1. Measure on a floor-spec machine (12th-gen i5 mobile, Iris Xe): prefill and decode tok/s and p50/p90 per answer for both models at 3 and 5 chunks, plus probe behaviour. No number in this comment is from an iGPU.
   2. Decide the speed bar that unlocks the quality tier from those measurements.

   🤖 Generated with Claude Code
3. zaxbysauce commented on Oct 9, 2026

   

   zaxbysauce

   on Oct 9, 2026

   CollaboratorAuthor

   More actions

   Acceptance-check checkpoint anchor (issue-tracer v3, Phase 2.5)

   Published before any fix code exists, so the acceptance checks cannot be weakened later without
   this receipt failing verification.

   issue-tracer-checkpoint-v1 slug=155-vulkan-gpu-probe manifest=c594410cbb7ac2127a540ed18a9060727316500b semantics=ce962b1966eda4f2e62c85152b492413deb0d4eb tree=559afcd21d661274a31483d5799abcabb2ed6216
   

   • base commit 29ce5b984c072541dd9699826eecf46fc48b0d50 (master, PR perf(desktop): stop Gemma 4 hidden reasoning and bound the penalty window #158 merged)
   • 14 acceptance checks (C1..C14), one per numbered criterion AC1..AC14, authored at arm's length
     by a separate check-author context and executed against the base tree in a disposable worktree
     before the freeze
   • manifest holds 15 rows: 14 CHECKPOINT plus one CHECK_WRONG AMEND on C13, whose comment
     wrongly claimed all three CONFIGURATION.md hits sit inside fenced code blocks. The amend was
     backed by a fresh base replay; the check's behaviour is unchanged, and argv/expect are
     identical, which is why semantics did not move between the first receipt and this one.

   What this receipt is for: repro-check.sh verify-anchor checks the local manifest digest, the
   ordered acceptance-table semantic digest and the checkpoint tree. Editing a frozen blob id,
   weakening an --expect regex, deleting or refreezing the manifest, or touching the
   acceptance-table semantics will make verification fail against this published literal. A
   byte-identical refreeze still passes, by design.

   What it does not prove: publication and its timing are a human gate the local validator cannot
   observe. The receipt was generated after the C13 amend and before any production change; if a
   later receipt appears for this slug, treat the earlier one as authoritative for the freeze.
4. zaxbysauce commented on Oct 9, 2026

   

   zaxbysauce

   on Oct 9, 2026

   CollaboratorAuthor

   More actions

   PR #159 opened — vendor-neutral GPU acceleration with an out-of-process probe

   Issue #155 traced and implemented through an issue-tracer v3 run. Head d29a76e, base
   29ce5b9 (master), 7 commits, 21 files.

   The short version: the desktop backend was hard-disabled to the CPU —
   effectiveVulkan() returned this.vulkanSetting ?? false — so a host with a working GPU never
   used it. That was deliberate against llama.cpp #17389, whose precondition has since changed and
   was never verified. It is now an out-of-process probe whose verdict persists, with CPU fallback
   at two levels and a user-visible reason.

   What the probe does. It runs in a separate OS process (process.execPath +
   ELECTRON_RUN_AS_NODE=1) because a Vulkan driver fault aborts rather than throws — neither an
   in-process try/catch nor a worker thread can contain it. The child loads the fast-profile GGUF
   with an explicit gpuLayers (#29277) and the parent re-judges the generated sample rather
   than trusting the child's own success flag (#28648: a device can load and then emit garbage). The
   verdict and reason persist to gpu-probe.json beside the other profile sidecars.

   Four defects the review loop caught that inspection would not have. Worth recording, because
   two of the worst were never inside the diff:

   1. The probe never ran in production at all — the host read a models property that
      EngineSurface does not have. All fourteen frozen checks were green while the feature was dead.
   2. A first run pinned the machine to CPU forever: the host starts before models are staged, so
      it recorded "no probe model", persisted it, and a persisted verdict then suppressed the next
      boot's probe. Two individually-correct decisions interacting.
   3. My own fix for that was dead code — a flag reset before the check could read it.
   4. One of my "mutation-verified" claims was false: three tests passed with their bug
      re-introduced.

   Two things I had to retract, both mine. I claimed node-llama-cpp exposes no KV-cache type
   option; it does, and it already defaults to F16, so llama.cpp #29054 needed nothing here — only
   ubatch (#27638) is genuinely unavailable. And I wrote that the probe's adapter-identity
   extraction could not be executed on this machine "because no GPU is here"; there is an Arc Pro B50
   and the real probe runs in ~3.6 s. Both corrections are in the trace.

   Honest limits.

   • onGpuLoadFailure is unwired in production: an automatic GPU-load downgrade holds for the
     session only.
   • The device matrix is mostly PENDING — only the devstation Arc Pro B50 was measurable. No
     tok/s figure appears anywhere that was not measured on the machine it names.
   • GPU engages on the second launch after a truly fresh install, because the first probe has no
     model to test. Self-correcting, and the honest cost of not recording evidence we do not have.

   The acceptance checkpoint was published earlier as a comment above, before any fix code existed,
   and still verifies against the local manifest. Ten trace phases validate at 0 FAILs; the
   implementation review took six rounds to return APPROVE, the plan critic five.

   This opens for review and stops here. It does not merge — merging needs a separate explicit
   approval bound to the PR head.
5. zaxbysauce

   closed this as completedin #159on Oct 11, 2026