Skip to content

About

The f16 version of herbert-vk-bf16: Qwen3.5 on Vulkan in IEEE f16, SPIR-V kernels emitted at startup. Rust.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

herbert-vk-f16 — Qwen3.5 and Gemma-4 in IEEE f16 on Vulkan, every kernel emitted as SPIR-V

herbert-vk-bf16 with IEEE f16 as its 16-bit format: the same models, the same numerical contract (16-bit weights converted at load to f16, nearest even; f16 activations, KV cache, logits and cooperative matrices; f32 accumulators; one rounding to nearest even at the output of each kernel), for the GPUs whose matrix units have f16 but not bf16, and for comparisons against llama.cpp's F16 path. Built without the f16 feature, its 16-bit format is bf16. No GLSL, no shader compiler: every compute kernel is a SPIR-V module emitted at start-up by Rust (src/spv, a thin layer over rspirv), with every dimension known at load time written as a constant.

Changes in 0.2.0

  • Gemma-4 E2B: text, images and audio, with a batched prefill (block size measured at load), the prefill attention on the cooperative matrices (and on VK_NV_cooperative_matrix2 for the 512-wide heads), a split decode attention, fused norms and gates, herbert-vk-serve for Gemma. See the Gemma-4 section.
  • Intel Xe2 (Arc 140V under Mesa ANV, 8 × 16 × 16 matrices only): the prefill attention of both models in tiles of 8 rows at 16 lanes, the prefill GEMM in tiles of 128 × 128 outputs per workgroup of 512 lanes, K and V in one attention tile, and Gemma's prefill block submitted one layer at a time (the xe driver kills a job past 5 s). On the Arc 140V in f16, against llama.cpp F16 (commit 4e910a86e, 2 November 2026): prefill of 759 tokens 1.56× (Qwen3.5-0.8B) and 1.30× (Qwen3.5-4B) llama.cpp's, first token of the 4B at 4 105 tokens 7.75 s against 10.70 s; Gemma-4 E2B runs to 32 584 tokens (251 tok/s), where llama.cpp loses the device.
  • AMD: Gemma's prefill at 32 lanes where the driver defaults to 64 (prefill.sg); Qwen's prefill attention on the 16 × 16 × 16 matrices at 32 lanes while the rest of the prefill and the decode stay at the driver's 64 (RX 9060 XT, prefill at 4 033 tokens +20 % on the 4B, +52 % on the 0.8B, decode unchanged); Gemma on integrated GPUs without cooperative matrices (RADV, Ryzen 7 5700G and Ryzen 9 9900X) submits its prefill one layer at a time and skips the block autotune (it lost the device to amdgpu's ring timeout before).
  • herbert-vk-chat --ask @FILE reads a turn from a file; the scalar prefill attention keeps its scores inside one storage-buffer binding; herbert-vk-probe --sweep checks the GEMM variants with a direct epilogue on their own grid, and --emit works at 16 and 64 lanes.
  • Same tokens as 0.1.0 wherever the path did not change: checked on the RTX 4080, the Ryzen 7 5700G and Ryzen 9 9900X iGPUs (Qwen3.5-0.8B and 4B, prompts of 492 and 4 017 tokens); on the Arc 140V and the RX 9060 XT the new paths change some greedy outputs.

Build and run

cargo build --release --locked --features serve
target/release/herbert-vk-probe                       # what each device offers, then measurements
target/release/herbert-vk-probe --device all          # the measurements on every hardware device
target/release/herbert-vk-chat ~/models/Qwen3.5-0.8B --device nvidia --no-system --no-think --ask 'What is the capital of France?'
target/release/herbert-vk-serve ~/models/Qwen3.5-0.8B --device nvidia --no-think --port 8080   # feature serve

herbert-vk-serve (feature serve) answers the Anthropic Messages API (POST /v1/messages, streamed or not), one request at a time, every request from an empty cache: the bench times every engine through the same HTTP client with it.

The default build keeps the activations and states the DeltaNet passes between kernels in bf16 (feature bf16, on by default); cargo build --release --no-default-features keeps them in f32.

--device takes nvidia, amd, intel, an index of the enumeration or part of the device name; by default the first discrete GPU, else the first hardware device. Software devices (llvmpipe) are refused. The other options of the chat: --subgroup N, --ctx N (positions reserved, default 4096), --no-think, --max-tokens N, --system TEXT | --no-system, --ask TEXT (repeatable), --temperature T, --dump-spv DIR, --tune KEY=VALUE (repeatable), --profile. On stderr, [ids] is printed in the same format as herbert-bf16-chat, for a token-by-token comparison; HERBERT_VK_TOP=N adds the N best logits of every generated position.

Tuning

Every kernel family has its parameters in src/vk/tuning.rs, set with --tune key=value; the defaults are the first, plain versions, and change only on a measurement. The main ones: prefill.block (rows of a prefill block; 1 feeds the prompt token by token through the decode step; the defaults per vendor, in Tuning::defaults, are 256 rows, 1 024 on NVIDIA and 2 048 there for a width up to 1 024, the 0.8B), gemm (the prefill projections: 0 SIMT tiles, 1 cooperative matrices of subgroup scope, 2 of workgroup scope with VK_NV_cooperative_matrix2), gemm.stage (tiles staged in workgroup memory), gemm.tensor (NV2 tensor layouts), gemm.wave, gemm.mt/nt/kt/tm/tn/sgm/sgn (the tiling), gemm.simt (the SIMT layout), sdpa.split (split-K decode attention), rec.slices/rec.cols (the DeltaNet recurrence with the dk reduction shared), fuse.norm/fuse.swiglu (norm and SwiGLU computed by the projection that follows), matvec.rows, wave, matvec.wave, weights.local, copy.logits. The load prints the whole set in --tune syntax.

--profile times every operation of the decode step with GPU timestamps and prints, after each turn, the time per kernel family and its share of the step.

herbert-vk-probe --emit DIR --subgroup N writes every kernel at the shapes of Qwen3.5-0.8B without opening a device, for spirv-val --target-env vulkan1.3 (the bf16 cooperative-matrix modules need a spirv-val that knows SPV_KHR_bfloat16; an f16 twin of each is emitted for the older ones). herbert-vk-probe --sweep measures every matvec and GEMM variant the device runs, interleaved, checked against an f64 reference; --local puts the weights in device memory the host cannot map.

Linux with a Vulkan 1.3 driver. The device needs storageBuffer16BitAccess, shaderInt16, subgroupSizeControl, computeFullSubgroups and VK_KHR_push_descriptor. On macOS the crate only compiles (cargo check).

How a token runs

One command buffer, recorded once at load and resubmitted for every token: the copy of (token, position) from a host buffer, the embedding, every layer, the final norm, the language head, a two-pass argmax, the copy of the argmax and the logits back to the host. With prefill.block > 1 the prompt goes by blocks of rows (src/vk/prefill.rs): GEMMs for the projections, row kernels, the DeltaNet conv and recurrence walking the rows of the block in order with their state in registers, the head on the last row only; one command buffer recorded per block. Every buffer is allocated at load; nothing is allocated during prefill or decode.

Per layer: RMSNorm ((1 + w) convention); for a full-attention layer the fused [q | k | v] projection, the head norms with the partial RoPE and the KV append in one kernel, the decode attention with the output gate, o_proj with the residual add; for a GatedDeltaNet layer the fused [qkv | z | a | b] projection in f32, the causal conv with SiLU, the L2 norms and the gates, the delta-rule recurrence (one workgroup per value head, the state in f32), the gated RMSNorm, out_proj with the residual add; then the MLP: norm, fused [gate | up], SwiGLU, down with the residual add.

Gemma-4 (E2B), text and images

herbert-vk-chat recognizes a gemma4 checkpoint by its model_type and runs it with the Gemma-4 engine (src/vk/gemma.rs, kernels in gemma_kernels.rs and gemma_vision_kernels.rs); the options are the same, plus --image PATH (repeatable, placed before the first question).

target/release/herbert-vk-chat ~/models/gemma-4-E2B-it-qat-q4_0-unquantized --device nvidia --no-system --image photo.png --ask 'Describe this image.'

The checkpoint is bf16; the loader converts each tensor to f16 (nearest even) as it uploads it, the per-layer embedding row of each token on the host as it is read. Everything the device keeps in 16 bits — activations, K/V caches, logits, image features — is f16; accumulators are f32. The architecture and the image path (Pillow-exact bicubic resize, 16-layer bidirectional tower, 3 × 3 pooling, one block for the soft tokens) are those of herbert-vk-bf16's README, and so is the performance work: the prompt goes through the layers in blocks of up to 320 rows (GEMMs, rows kernels in gemma_rows_kernels.rs, src/vk/gemma/batch.rs), the decode attention is split over 16 slices of the positions, norms and the GeLU gate are fused into the projections, the argmax runs on the device, and the vision tower uses the GEMMs and a flash attention. On the RTX 4080 (median of 3, wiki #559): prefill of 770 tokens 103 → 8 119 tok/s, decode at 770 positions 79 → 106 tok/s, an image 0.46 → 0.33 s. --tune prefill.block=1 sdpa.split=0 fuse.norm=0 fuse.swiglu=0 is the one-token path of before.

Checked against transformers 5.5 on an RTX 4080: at 127 tokens the top-10 logits are within 0.01 of transformers in f32 (26.453 / 23.422 / 23.25 / 23.094 against 26.453 / 23.425 / 23.249 / 23.103), at 1 987 tokens (the 512-position window crossed) within 0.02 (27.328 / 27.109 / 25.547 against 27.340 / 27.116 / 25.545); hidden state of the last token after each layer at cos ≥ 0.9998; image features at cos 0.99999 on average and 0.9994 at worst against the f32 tower (relative error 0.35 %, against 1.6 % for the bf16 tower). HERBERT_VK_DUMP=file writes the hidden state of the last token after each layer, HERBERT_VK_FEATS=file the image features, both as f32.

The code

module what it does
core the model's config.json, bf16 conversions, the error type, the sampler (from herbert-jit-bf16)
model the safetensors checkpoint, read into bf16 row-major tensors
spv the SPIR-V emitter: types, buffers, structured control flow, subgroup and workgroup reductions, bf16 rounding
vk/context the device, its features, the queue, submission with a fence
vk/memory a few large allocations sub-allocated into buffers; direct host writes when the device memory is mappable
vk/pipeline pipelines with push descriptors and the required subgroup size; dispatch
vk/types the shapes, output modes and state layout the kernels, the engine and the probe share
vk/ext the Vulkan structures ash 0.38 lacks (bf16, cooperative matrix 2)
vk/tuning the parameters of the kernel families, per vendor, --tune
vk/kernels the emitted kernels of one decode step
vk/prefill the emitted kernels of a prefill block: row kernels, SIMT and cooperative-matrix GEMMs
vk/engine the model on the device and the recorded step
vk/probe the device report and the measurements
chat the Qwen3.5 template, tokens, streaming detokenization, the session

bench/compare.sh runs the prompts of bench/prompts/ on the CPU oracle and on Vulkan devices; bench/speed.sh runs the interleaved speed campaign; bench/_diff.py and bench/_speed.py read their logs.

Article #002

The copy measured for « On descend d'un cran : #002 le même compilateur sur un GPU » (FR, EN), 12 to 14 October 2026. The campaign scripts, the predictions written before measuring, the journal and the raw logs are in herbert-gpu-bench (tag v1-2026-10-14).

Measured on an NVIDIA RTX 4080 and an RTX 3060 Ti, an Intel Arc 140V and two Radeon iGPUs (Vega in the Ryzen 7 5700G, RDNA 2 in the Ryzen 9 9900X). Other GPUs are untested, and the default tunings may not suit them. Discrete AMD Radeon cards will come in a later version, with their own article.

#002.1, follow-up to article #002: the Arc in f16

Version 0.2.0 is the copy measured for the follow-up « On descend d'un cran : #002.1 complément, l'Arc en f16 » (FR, EN), 2 November 2026: the Intel Xe2 changes listed above, measured on the Arc 140V under the protocol of the article's campaign. Version 0.1.0 (tag v0.1.0) stays the copy measured for the article itself.

License

MIT.

About

The f16 version of herbert-vk-bf16: Qwen3.5 on Vulkan in IEEE f16, SPIR-V kernels emitted at startup. Rust.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages