Record of every file ported from colibri or ds4. It serves two purposes: respect licenses (Apache-2.0 asks to say what changed) and know where to report improvements later. Updated in the same commit that brings the code.
| Project | License | Commit | Local worktree | Origin clone |
|---|---|---|---|---|
colibri (JustVugg/colibri, branch dev) |
Apache-2.0 | a90bed9 (2026-09-17) |
ref/colibri |
Desktop\colibri |
ds4 (antirez/ds4, branch main) |
MIT (contains ggml code, MIT) | 8db1d1d (2026-09-16) |
ref/ds4 |
Desktop\ds4 |
llama.cpp (ggml-org/llama.cpp, master) — reference for comparison and source of ideas, no code ported |
MIT | b49650a (2026-09-17) |
ref/llama.cpp (shallow clone; build in build-trochilus/ with tools/build_llamacpp.sh) |
— |
ik_llama.cpp (ikawrakow/ik_llama.cpp, main) — source of ideas for the K-quant kernels, no code ported |
MIT | f3d6e6e (2026-09-23) |
ref/ik_llama.cpp (shallow clone, 150 MB) |
— |
To move a reference: git -C <clone> fetch, then git -C ref/<project> checkout --detach <commit>,
and update the table. Files already ported remain tied to the commit written in their row.
/* <filename> — <what it does>.
* Derived from <colibri|ds4> <commit> <path> (<license>), modified: <what changed, one line>. */| Trochilus file | From | Commit | Origin path | What changed |
|---|---|---|---|---|
src/format/gguf.c (type table only) |
ds4 | 8db1d1d |
ds4.c gguf_types[] |
added tq1_0/tq2_0, iq4_nl fixed to 32 × 18 byte blocks; parser is new |
tools/make_tiny_olmoe.py |
colibri | a90bed9 |
c/tools/make_olmoe_tiny.py |
GQA 4/2, 16 tokens, logit reference per position |
Do not require attribution, recorded to know where to look when that piece improves.
| Idea | From | Where in origin | In Trochilus |
|---|---|---|---|
| persistent thread pool, no OpenMP | ds4 | ds4.c ds4_parallel_for |
src/base/threads.c |
| threads on physical cores, not logical | colibri | c/omp_tune.h |
tr_pool_create(0) |
| each thread of the pool pinned to its own logical processor | llama.cpp (the two system calls, and the idea of a per-thread mask for --cpu-strict) |
ggml-cpu.c ggml_thread_apply_affinity, ggml_thread_cpumask_next |
src/base/platform.c tr_thread_pin, slot order in src/base/cpu.c, pinning in pool in src/base/threads.c |
weights with pread instead of mmap |
colibri | c/st.h (comment on RSS bug) |
src/base/platform.h |
| oracle on tiny generated models | colibri | c/tools/make_*_tiny.py, CI job |
tools/, make oracle |
| tokenizer read from GGUF metadata, "a b" merge as rank key | ds4 | ds4.c vocab_load, bpe_rank |
src/tokenizer/tokenizer.c (merge becomes id pair → rank and result) |
| pretokenizer replays regex in C on codepoints; Unicode classes from generated tables | colibri | c/tok.h pretok_chunk, c/tok_unicode.h |
src/tokenizer/tokenizer.c split_gpt2, tools/gen_unicode_tables.py (tables probed on HF, not Python) |
| prompt in passes of at most 512 tokens | llama.cpp | common/common.h n_ubatch |
olmoe.c OLMOE_DEFAULT_BATCH, -b |
| (token, expert) pairs ordered by expert with a counting sort, one job per expert | ds4 (llama.cpp with an index map) | ds4.c layer_routed_moe_batch; ggml-cpu.c ggml_compute_forward_mul_mat_id |
olmoe.c forward_pass, tr_matmul_grouped |
| one weight row against a token block, instead of whole matrix per token | llama.cpp (16 × 16 blocks), ds4 and colibri (row × all tokens) | ggml-cpu.c ..._mul_mat_id_one_chunk; ds4.c matmul_q8_0_batch_worker; c/olmoe.c matmul |
kernels.c matmul_tiled (blocks of TR_MATMUL_TILE tokens) |
| one weight row against many tokens in registers (2 in ds4, 4 in Trochilus): one load and one conversion only for all | ds4 | ds4.c dot_q8_0_row_2 |
kernels.c k_dot_row_x4_q8_0, kernels_x86.c avx512_dot_row_x4_q8_0 |
| K and V of batch written all, then attention in parallel for (head, token) | colibri (ds4 optional) | c/olmoe.c attention; ds4.c layer_attention_prefix_batch |
olmoe.c attn_body |
| logit only of the last token of the prompt | all three | c/olmoe.c step; ds4.c output_logits_one |
olmoe_eval |
| draft from text already in prompt: search backward for the last occurrence of the tail n-gram and propose what followed | colibri (exact scan, no cache), llama.cpp (n-gram cache with counts) | c/deepseek_v4.c v4_ngram_draft; common/ngram-cache.cpp common_ngram_cache_draft |
src/gen/lookup.c tr_lookup_draft (new code) |
| verify 1 + k positions in one pass, accept while the proposed token is what the model would choose | all three | examples/lookup/lookup.cpp; ds4.c metal_graph_verify_decode2_exact; c/deepseek_v4.c (acceptance loop) |
src/gen/greedy.c tr_greedy_step |
| undo rejected draft by truncating cache index | llama.cpp (ds4 and colibri cannot: recurrent attention, must restore snapshot) | llama_memory_seq_rm |
tr_session_rewind (s->pos = n) |
| a number's top bits are a coarse copy of it: a cheap guess with a known error, then exact work only where it can still matter (the output head's argmax from each row's scales and the codes' high bits, exact only the rows whose bound can still win) | Quake III Arena's fast inverse square root (a float's bits read as an integer are its log2: 0x5f3759df minus half of them, then one Newton step), brought to the project by Marcello on 2026-10-05 as a lens for question 94 |
id Software's Q_rsqrt (q_math.c); https://it.wikipedia.org/wiki/Radice_quadrata_inversa_veloce |
tools/head_bound.py (measured 2026-10-05: Q8_0's high nibble leaves 0.2-3.8% of the rows; question 73, to build) |
llama.cpp b49650a |
ds4 8db1d1d |
colibri a90bed9 |
|
|---|---|---|---|
| prompt split | n_ubatch 512 |
none on CPU: whole prompt; FFN in sub-batches of 128 (DS4_PREFILL_BATCH) |
whole prompt in step() |
| projections | sgemm, Q8_0 activations | weight row × tokens pairwise, Q8_0 activations | weight row × tokens, float |
| MoE | rows per expert with index map, 16 × 16 blocks, dot Q8 × Q8 | counting sort per expert; gate and up with one quantization only; down and sum per row, experts in id order | no grouping: token by token |
| attention | flash attention or matrix with mask | serial KV write, then attention per token (optional per head and token) | all K and V, then (head, token) in parallel |
| exactness against one token per pass | no (int8 activations) | no, no test | not declared or proved |
Not taken: int8 activations (not exact: lever 2, separate option) and ds4's per-row sum over all experts, which rereads intermediate activations of the whole batch at every weight row (with 512 tokens ~16 MB per row): Trochilus keeps expert order in token blocks, and sums per token.
| llama.cpp | colibri | ds4 | |
|---|---|---|---|
| how many threads | logical / 2, or physical cores where Windows count is compiled (excluded on MinGW-w64, UPSTREAM #4) |
physical cores counted truly (GetLogicalProcessorInformationEx, thread_siblings_list, hw.perflevel0), never guessed |
omp_get_max_threads() |
| where they go | nowhere: without --cpu-mask does not call affinity |
OMP_PROC_BIND=close passed to libgomp, Linux only, with process re-exec |
nothing |
| who decides the mask | the user, by hand (--cpu-mask, --cpu-strict = one processor per thread) |
libgomp | — |
| beyond 64 processors | does not reach (SetThreadAffinityMask, UPSTREAM #3) |
depends on libgomp | — |
Taken: the two system calls (Windows and Linux) and the idea of --cpu-strict, that is, a mask
of one processor only per thread instead of a common mask. Not taken: the mask written by hand
by the user (here the topology is counted by the engine), OMP_PROC_BIND with re-exec (no
OpenMP, and environment variables read inside the engine are anti-pattern from colibri), and
SetThreadAffinityMask (replaced by SetThreadGroupAffinity, which reaches every group).
New here: the order of slots (one thread per physical core before any SMT sibling, and cores
taken in rotation across last-level caches, because 4+4 on two chiplets beats 8 on one), respect
of a mask already set on the process (taskset, start /affinity), restore of caller's affinity
when the pool dies, and a test that asks the operating system which processor ran each chunk
(tests/test_base.c): none of the three sources verify this.
| llama.cpp | ds4 | colibri | |
|---|---|---|---|
| where the draft comes from | three n-gram caches (context, previous session, corpus), n from 4 to 1, vote count × corpus count |
no prompt lookup: DSpark and MTP, that is, trained heads (docs/SPECULATIVE_DECODING.md) |
v4_ngram_draft: 3-gram then 2-gram, backward scan, most recent occurrence |
| verify | one llama_decode on draft batch, logits at each position |
2–3 rows per pass, row0_top == draft1 |
loop predictions[i] == drafts[i] |
| undo a rejection | cache truncation by index | buffer swap with snapshot (recurrent state) | spec_attention_restore and prefix re-execution (measured so costly it was disabled for MTP) |
| tokens identical to generation without speculation | algebraically yes, bit by bit not declared (batch may sum in another order) | declared not identical to bit on long continuations | not declared |
Taken: the idea of draft from context (colibri, simplest: no state) and verify in one pass with
logits at each position (all three). Not taken: llama.cpp's caches with counts (allocations and
state, to evaluate only if measurements ask), trained heads (DSpark, MTP, EAGLE3: other models),
ds4's "opportunistic" sampling. Restore by snapshot is not needed: our cache is indexed per
position, we go back with s->pos = n. Plus, here tokens are byte-identical, because each
row of the batch is the same computation as the one-token pass (tests/test_prefill.c,
tests/test_spec.c): none of the three sources prove this.
src/memory/experts.{h,c} is new code (no line ported), written after reading
ref/colibri/c/olmoe.c (LCache, Slot, expert_get, pilot_worker) and with measurements
of questions 13–16 in front (docs/MEASUREMENTS.md §M1). Three bugs found in that reading are in
docs/UPSTREAM.md #7–#9.
| Choice | colibri | Trochilus | Why |
|---|---|---|---|
| find an expert in cache | array slot_by_expert[eid], O(1) (olmoe.c:158) |
slot_of[unit], O(1), unit = (layer, expert) |
taken: the right thing |
| one expert in memory | one block for three matrices (load_expert_merged, olmoe.c:636) |
one slot per unit, three parts aligned to 64 inside the slot | taken halfway: in our GGUF the three tensors are separate, so three parts are still read with three pread (single read impossible without our format, excluded 2026-09-17) |
| choose who to evict | linear scan of all slots on every miss, O(capacity) (olmoe.c:696) |
doubly-linked list, O(1) | rejected: worsens as budget grows, the very case we care about |
| keep in RAM the most-used experts (pin) | pin_hot_experts after 5 tokens, with escape if all slots blocked (olmoe.c:745) |
no pin: LRU only | rejected by numbers: on routing trace pin from use reads more than LRU at every capacity (at 75%: 94 vs 32 MiB per token, question 14) |
| preload experts of the next layer | one thread, prediction with mini-matmul of router plus moving average (PILOT, olmoe.c:1110) |
a prompt: layer L+1's missing units while L computes (the prompt uses nearly all of them); a decode (10-05): L+1's router on L's FFN input, its top k in 27 MiB, read in 512 KiB pieces, a guess not named stopped mid-read, a part a request once named | colibri's two improvements replayed: its norm predicts as ours, its average loses 4.3 points (LESSONS #323); whole-unit reads (colibri's) lose 4% where pieces gain; the 8 GB machine's decode Q4_K 1.031, Q8_0 1.000-1.005 (MEASUREMENTS §The router's read ahead). The prompt's read-ahead: 1.22× at 2048 tokens, half budget (MEASUREMENTS §Reading the next layer) |
| I/O thread | one, preload only | one: the prompt's read ahead (tr_experts_prefetch), and since 10-05 a layer's misses while its present experts compute (tr_experts_acquire_async, the arrival order: MEASUREMENTS §The arrival order built) |
disk gives same bandwidth to 1 and 8 readers (question 16): a thread serves only to overlap; a prompt overlaps the next layer, a decode token its layer's own present experts (question 90) |
| a read error | exit(1) inside st_pread_full (st.h:268), even for optional preload |
eval returns -1 and session stays as it was | an engine in a library cannot close the process |
| where to read from | own format converted (model-*.safetensors, st.h) |
standard GGUF, at tensor positions | decision 2026-09-17: Hugging Face files open without conversion |
src/kernels Q4_K (scalar in kernels.c, AVX2 and AVX-512 in kernels_x86.c) is new code, written
after reading llama.cpp b49650a (ggml-common.h block_q4_K, ggml-quants.c
dequantize_row_q4_K and get_scale_min_k4, ggml-cpu/arch/x86/quants.c
ggml_vec_dot_q4_K_q8_K, ggml-cpu/repack.cpp block_q4_Kx8, gguf-py/gguf/quants.py Q4_K) and
ik_llama.cpp f3d6e6e (ggml/src/iqk/iqk_gemm_kquants.cpp DequantizerQ4K, Scales8K;
iqk_common.h make_q4_scales; iqk_quantize.cpp repack_q4_k).
| Choice | llama.cpp | ik_llama.cpp | Trochilus | Why |
|---|---|---|---|---|
| the block and its weights | block_q4_K: f16 d and dmin, 12 bytes of 6-bit scales and mins, 128 of nibbles; weight d*sc*q - dmin*m |
the same block | read as it is in the file; the weight (d*sc)*q - (dmin*m), each product rounded to float |
taken: the file format, and gguf-py's order of operations, so a dequantized row is gguf-py's bit for bit (tools/check_dequant.py in the gate) |
| the activations | quantized to Q8_K (8 bits, blocks of 256 with their sums), integer dot | Q8_K, Q8_K32, Q8_1_X4, integer madd |
since 2026-09-26: 32-bit block fixed point, a shift per 256 and nearest-even, as two balanced 16-bit digits; sums of exact integers (kernels.h q4x_*) |
8 bits are another number (question 21); 32 bits with exact sums are the exact math's, correctly rounded in 96-99% of outputs, and order-free: any width, split or device gives the same bits (MEASUREMENTS §Exact sums at the float's speed) |
| the min term | dmin*m * sum(y) per sub-block, from the sums stored with the Q8_K activations |
the same (accum_mins) |
the same factored form, M = sum_j m_j sum X from the input's exact sub-block sums, but exact: one f64 step at the block's end |
the factored form rounds only where its sums are floats; in integers it is exact, and it is the cheaper one |
| decoding the weights in SIMD | nibble masks, then int8 products | the same, 8 scales unpacked at once with 32-bit masks (make_q4_scales) |
the prompt: the W16 panel, 16 rows' sc*q in 16-bit words transposed by byte permutes, and Winograd's inner product (the other column's digit riding in each factor); the decode: the nibbles widened, times the scale (vpmullw), vpdpwssd against the digits |
ours (Winograd 1968 on 16-bit words, the scale inside the weight word); the two rows' headers of 8 blocks read first, as the float kernel did (LESSONS #178, #220) |
| rows repacked at load | block_q4_Kx8: 8 rows interleaved, gemv/gemm over them |
-rtr, Q4_K_R4: 4 rows interleaved |
not taken | the expert store reads the file's bytes into its slots (M1): a repack would be a transform on every read. What it buys, one activation load for several rows, matters little in decode (the activations sit in L1); a question for later |
| large prompts | 8-bit GEMM | Q4_K converted to 8-bit rows (iqk_convert_q4_k_q8_1_r8), then 8-bit GEMM |
a 16-row W16 panel built once per (group, rows), tiles of 1-4 input rows on the exact integers (168 GFLOP/s-eq a core at 4); on AVX2 (2026-10-03) the same panel in two halves of 8 rows and tiles of at most 3 (9.1 weight MACs a cycle; the references' AVX2 ways, read: ik converts Q4_K to 8-bit rows (iqk_convert_q4_k_q8_1_r8, from 32 tokens) for an 8 rows x 8 tokens GEMM on maddubs, ~9 MACs an instruction on 8-bit activations; llama.cpp's ggml_gemm_q4_K_8x8_q8_K runs 8 rows x 16 tokens with 32 float accumulators that spill; ours ~4 an instruction, the price of two exact 16-bit digits) |
their conversion once, the exact arithmetic kept. Before 2026-09-26: the float form of the idea, a row dequantized once per tile of tokens and the F32 kernel on it, is to be measured |
| IQ_K types | — | its own types (IQ2_K … IQ6_K), better quality per bit | not now | not among the GGUF types of llama.cpp that our reader follows; such files come only from ik's quantizer |
src/kernels Q6_K (scalar in kernels.c, AVX2 and AVX-512 in kernels_x86.c, the block in
kernels_internal.h) is new code, written after reading llama.cpp b49650a (ggml-common.h
block_q6_K, ggml-quants.c dequantize_row_q6_K and quantize_row_q6_K_ref,
gguf-py/gguf/quants.py Q6_K, src/llama-quant.cpp for which tensors Q4_K_M makes Q6_K) and
ik_llama.cpp f3d6e6e (ggml/src/iqk/iqk_gemm_kquants.cpp DequantizerQ6K, AVX2 and AVX-512).
| Choice | llama.cpp | ik_llama.cpp | Trochilus | Why |
|---|---|---|---|---|
| the block and its weights | block_q6_K: 128 bytes of low nibbles, 64 of high 2 bits, 16 int8 scales, f16 d last; weight d*sc*(q-32) |
the same block | read as it is in the file; the weight (d*sc)*(q-32), each product rounded to float |
taken: the file format and gguf-py's order (tools/check_dequant.py: 1M floats bit for bit, scale 0 times a negative quant is -0 in both) |
| the activations | Q8_K, integer dot | Q8_K, the -32 folded into a min term (-32*d times the activations' sums) |
float, never quantized; the -32 inside each weight |
as for Q4_K: the dot of the dequantized weight is the definition |
| decoding the weights in SIMD | masks and shifts on bytes, int8 products | the 6-bit values assembled on bytes, 32 at a time: low nibbles OR (high bits shifted into 0x30) (DequantizerQ6K::prepare) |
taken: a block's 256 quants assembled that way once, minus 32, into int8 on the stack; then Q8_0's path per element (widen, convert, multiply by the sub-block's scale) | the byte form makes 32 quants in ~5 instructions; a lookup like Q4_K's AVX-512 one does not pay here (64 values per 16 weights) |
| several weight rows per activation load | block_q4_Kx8, block_q8_0x8: 8 rows interleaved at load, then a gemm over them |
Q4_K_R4 and friends: 4 rows interleaved |
our own, without repacking: dot_row2_x4 takes two rows of the file as they are against four input rows (eight float accumulators, AVX-512) |
the same idea, register blocking over rows, where it pays for us: the prefill's matmul is bound by loading its inputs (MEASUREMENTS §Two weight rows, prefill 1.20–1.26× on Q8_0); no repack, so the expert store still reads the file's bytes |
The method (CLAUDE.md, How to work): one piece at a time, to the end. Before building on a piece,
its row is read in the three references (ref/colibri, ref/ds4, ref/llama.cpp with
ref/ik_llama.cpp), theirs is measured against ours where it runs, and only then the better
solution is built. "not read" means nobody here has read that code for that piece yet: it is the
debt, and the next piece is the first row with one. Measured = a number in MEASUREMENTS.
| # | Piece | colibri | ds4 | llama.cpp / ik_llama.cpp | Measured against ours | State |
|---|---|---|---|---|---|---|
| 1 | CPU matmul, the prompt (Q8_0, K-quants) | one row × one token, (expert, row chunk) items kept in cache; F32 activations with FMA, int8 opt-in (read 09-17, xf_moe_run 09-24) |
one row against 2 tokens, int8 activations (read 09-17, activations 09-24) | read 09-24 (MEASUREMENTS §The prompt's matmul against the four references): int8 activations everywhere; tinyBLAS 4 × 4 for the dense weights, Q8_0 experts one dot at a time, Q4_K repacked 16 rows × 4 tokens; ik converts 32 rows at a time once, then one gemm over every token | their kernels alone on bench_peak's shapes (tools/bench_ggml.sh, container, loaded): Q8_0 experts ours 1.42× (102 against 72 a core); Q8_0 dense theirs 1.6× (tinyBLAS, int8); Q4_K repacked theirs 1.26× (experts) to 2.3× (dense). ik's decode-once kept exact (question 63): Q8_0 1.00–1.04×, Q4_K 1.1×, closed. x8 native (09-25): prefill 1.13–1.19×, logits identical. Engine race after x8 (09-25): prefill level at 8 threads (theirs 1.01–1.03×), theirs 1.30–1.35× at 16. More rows per input load (question 64, 4 × 6 and 3 × 8): lose 6–16%, closed as no Phase-major (09-26, MEASUREMENTS §Phase-major): the exact definition run with 16 weight rows as SIMD lanes, 164.7 GFLOP/s a core (98.6% of the no-FMA peak; their repacked Q4_K 194 dense and 107 through mul_mat_id, int8); the engine natively: Q4_K prompt at 4 threads 140 → 218-223 tok/s (theirs 222 in the container), Q8_0 at 16 threads 390-419 → 550-569, the logits identical |
rebuilt 2026-09-26: phase-major (the lane contract as a chain in time, the thinker's idea); Q4_K in exact integers (W16 tiles, level with the float); raced (MEASUREMENTS §The race for the README) |
| 2 | CPU matmul, the decode (one token, memory-bound) | no separate one-token path on x86: one row × one vector, int8 activations quantized once per token, dpbusd (AVX-512 VNNI) or maddubs (AVX2), OpenMP static split of the rows; MoE: gate and up fused per (expert, row chunk) (quant.h matmul_q_idot, expert_ffn.h xf_moe_run; read 09-25) |
no x86 SIMD in the CPU path: scalar C (NEON only on ARM), activations Q8_K; static equal split over a pthread pool (ds4_parallel_for), a decode-only MoE routine (layer_routed_moe_one_prealloc; read 09-25) |
llama.cpp: tinyBLAS off at one column (llamafile_sgemm returns at n < 2), so vec_dot one row at a time, activations quantized once per matmul (Q8_0 / Q8_K) and shared; rows claimed in chunks of 64 by an atomic counter; repacked Q4_K runs gemv 8 rows at a time; mul_mat_id per expert, gate and up separate. ik: nrc_y = 1 kernels, two independent block chains per row, static row split, gate and up fused per expert (read 09-25) |
their kernels against ours in one binary, from RAM (tools/bench_ggml_decode.sh, container, MEASUREMENTS §The decode's matmul against ggml's): Q8_0 level from 4 threads (0.97-1.06x, both at the ceiling), theirs 1.4-1.7x at one; Q4_K theirs 1.34-1.40x at 4 threads until question 66 (MEASUREMENTS §The Q4_K decode dot sequenced): our scalar scale decode was 24% of a row; the scales in a vector one block ahead and two rows a call (dot_row2), exact, make it 1.04x (1.01-1.04), 12.5 GB/s a core, and the real model's decode at 4 threads level with llama.cpp (50.4 against 50.2 tok/s). Their 64-row atomic claiming tried in our decode: 0.93-0.97x |
closed 2026-09-25, reopened by question 66 the same evening: Q4_K level at 4 threads within the exact definition; left: their 2x a core (int8 activations, question 60) and SMT (+9-13% from RAM at 4 cores, not wired). Since 2026-09-26 Q4_K's pair is the exact integers' (q4x_dot2), level with the float pair in the engine. The short verify pass (2-3 rows, question 78, 09-27): llama.cpp's tinyBLAS takes a dense Q8_0 matmul from 2 columns (tiles of up to 4 weight rows by 1-4 columns, each block decoded once, int8), mul_mat_id by vec_dot a token at a time (read 09-27); ours now phase-major's balanced items of 16 weight rows and dot_row_xt (exact): Q8_0's 3-row pass at its bytes (MEASUREMENTS §The short verify pass at its bytes); Q4_K's (09-27, ours): the W16 panel's 16 rows brought ahead (block s + 2, and the next item's rows during this one's panel), a 2-row group by q4x_dot2 pairs: 3 rows 0.947x, 2 rows 0.968x (§The Q4_K short passes); then two weight rows decoded once for a group's 2-3 rows (q4x_dot_xt, road (c)): 2 rows 0.94-0.98x, 3 rows 0.96-0.985x more (§A Q4_K weight row decoded once for 2-3 tokens). ik_llama.cpp decodes each Q4_K block once for up to 8 columns too, one weight row against int8 activations (mul_mat_qX_K_q8_2_X4_T, read 09-27); llama.cpp's vec_dot decodes it a column at a time. Not yet raced against their pp2/pp3. Piece 4 (10-02): q, k, v and gate, up one call each, their items one flat range (tr_matmul_q4x_prepared_n): the decode 1.0054 +- 0.0007; ik merges q, k, v at load (-mqkv) and fuses gate, up per expert with the activation inside (GGML_OP_MOE_FUSED_UP_GATE), llama.cpp keeps them apart (MEASUREMENTS §Piece 4) |
| 3 | Attention, prompt and decode, and the KV layout | MLA "absorb": K and V one latent row, F32 [position][dim] a layer (u8 codecs opt-in); decode: two-pass softmax, (token, head) pairs over OpenMP, the positions never split, no prefetch; the value axpy kept bit-exact, the score dot reassociated; prompt: tiled online softmax only above 2 GB of reconstruction, which it says rounds otherwise (colibri.c 4711-4919; read 09-25) |
F32 [position][dim], K and V the same row (MLA); on x86 the CPU attention is scalar (NEON only), two-pass softmax with a sink logit, one token after another, no threads: the oracle, CUDA is the engine (ds4.c 12579-12617, 15471; read 09-25) |
llama.cpp: KV F16 [position][head × dim] a layer; flash attention on by default on the CPU (AUTO resolves on); the decode at ≥ 512 positions splits the positions over every thread, each thread runs every head over its chunk (a 256-byte slice every 4 KiB, 16 strided passes), online softmax, V summed in an F16 accumulator rounded at every position, partials merged by rescaling (ops.cpp 8700-8810, 9261-9290); prompt: 64 × 64 tiles, masked positions skipped one by one. ik: FA on by default (flash_attn = true), the positions split for MLA only (iqk_flash_attn.cpp 373-423) (read 09-25) |
the race of 09-25 (f6a8b4c, container, Q8_0) as a line, decode per token = intercept + slope × context (MEASUREMENTS §The attention against llama.cpp's): at 8 threads ours 28.1 ms + 5.31 µs a position, theirs 27.6 + 3.24; at 16, 26.8 + 5.24 against 26.8 + 3.34. Level at no context; the whole gap is the slope: we read 256 KiB a position at 49 GB/s, they read 128 KiB at 40. Ours at the speed of a plain read of its bytes at 4-8 threads (bench_attn_bw); one front instead of T refuted (bench_mem streams, 0.78-0.81x in the attention). Prompt: the products at 70 GFLOP/s a core, the lane trees in scalar; tiles of 4 queries x 4 positions 1.32x on the attention at 2048, 1.38x at 4000 on one core; the engine's zone 1.31-1.33x (prefill 1.03x at 2048, 1.06x at 4000), logits identical (MEASUREMENTS §The attention against llama.cpp's, §The prompt's attention in tiles) |
read and measured 2026-09-26: the decode is at the RAM's ceiling, and passing F16 exactly would need 81 GB/s (the GPU attention passes it); the prompt's tiles built; open: the softmax's exponential in a vector (row 4, question 58), now half of a prompt's pair |
| 4 | Softmax and exp | the C library's expf, one value at a time (read 09-24) |
the same (read 09-24) | ggml_v_expf (ARM's routine) in softmax and SiLU, sum in double after a lane reduce; the C library's expf in the CPU flash attention; ik the same routine (read 09-24) |
every float (MEASUREMENTS §Softmax and exp against the references): ggml's differs from the exact exp on 3.36% (≤ 2 ulp, 0.16 ns), glibc's on 0.004% (1 ulp, 3.2 ns); ours 0, 3.5 ns scalar, 0.73 ns SIMD (not wired) | read and measured: exact at ≤ ~2% of a prompt; nothing to take |
| 5 | Thread pool, placement, widths per phase | physical cores (read) | persistent pool (read) | pinning calls (read); two fixed counts, -t and -tb |
prefill +30% with physical cores; the widths measured while running, per kind of pass (the decode, and since 09-27 each size of verify pass, the narrowest width skipped: MEASUREMENTS §The verify passes' own width) are ours, none of the three measures one. The idle workers (read 09-27): llama.cpp's spin on ggml_barrier (ggml-cpu.c 576-610), ik_llama.cpp's spin then yield and fault the MoE's pages from a pool of their own (ggml-moe-prefetch.cpp), colibri's OpenMP team kept hot, ds4's sleep: none brings the next call's weights with its idle workers; ours (tr_pool_hint): 1.02-1.14 a call in the bench, level in the engine (the serial steps pay it back: MEASUREMENTS §The idle workers, and what lies past a region's end), off |
read (09-17/18, 09-27) |
| 6 | MoE routing and grouping by expert | not read | counting sort by expert (read) | index map in mul_mat_id (read) |
— | read, not measured against them |
| 7 | Experts from disk: store, budget, eviction, read ahead | slots; an LRU a layer (16 slots a layer by default) beside a pinned hot set (eslot_lru_victim colibri.c:428-440, olmoe.c:696-725); the KV it will use (prompt + new tokens in generate, 4096 F32 otherwise) taken off the experts' budget (olmoe.c:488-524); one buffered pread an expert from its own merged format (olmoe.c:660; read 09-20, 10-02, 10-03); read ahead only in the decode, off by default (PILOT=1-3, passes of at most 8 tokens: the next layer's router on the state after this layer's attention, momentum, WIDE, CONF_LIMIT 0.92; the stale queue flushed every token: olmoe.c:10-18, 976, 1110), nothing ahead in a prompt (read 10-03 afternoon); a decode miss read on the calling thread inside the expert loop, the experts after it waiting behind its read (moe olmoe.c:925-948; read 10-05) |
by default an mmap with WILLNEED; --ssd-streaming its own expert cache: the victim the lowest hotness, every routing +1 (a prefill batch's every token row, 17437-17447, 14359-14362), halved every 16 decode tokens, none before the first (ds4_metal.m:15404-15439, 14272-14346, 14292-14306; the prefill read 10-03 evening); a 9-thread pread pool, a part a task, a layer's misses on demand, nothing read ahead; the KV of 32768 positions whole (read 10-02, 10-03); a layer's misses read by the pread pool while the GPU computes the resident experts (a masked pass), the missing ones in a second pass, the down and sum once in slot order, only from 3 misses (split_worthwhile ds4_metal.m:13095, 42318-42464; read 10-05) |
an mmap, the OS evicts; the KV of the trained context whole, F16; no budget (read 10-03); a fault on an expert's mapped rows stalls only its thread, the others take the next experts' chunks (mul_mat_id: one barrier, then no barrier between experts, ggml-cpu.c:1663-1729; ik's the same loop, ggml.c:18497-18575; read 10-05) |
the eviction replayed on a route trace (tools/evict_replay.py): at 83 slots our LRU 0 hits of 128, colibri's 36.4, ds4's 47.3, Belady's 65.4; at 256, 41.9 misses a token colibri's, 40.3 ds4's (Q8_0). A miss read their ways (tests/bench_disk_misses.c): ours 1.87 GB/s, ds4's 1.87, colibri's 2.13 direct and 2.49 through the cache, the mmap 0.92. ds4's eviction adopted (the default since 10-03): with the plan on the run's own positions, the 8 GB machine's decode 2.9-3.2x that of 10-03 (MEASUREMENTS §The 8 GB machine's store); a prompt's next layer read ahead in one pass too since 10-03 (short requests, the unasked dropped unread): the 8 GB machine's prompt 1.29x (Q4_K), 1.21x (Q8_0), at 97% of its disk's time for its bytes (MEASUREMENTS §A pass reads the next layer ahead); the KV against the store (read 10-03 night): colibri takes its context's whole KV off the cache once (colibri.c cap_for_ram 10226-10259, its OLMoE engine olmoe.c:488-524) and at run time only lowers the cap (rss_guard 8412-8464), ds4's automatic cache ignores the context (ds4.c:68145), llama.cpp allocates the KV whole (llama-kv-cache.cpp:233, --fit once at load): ours grows the KV from the store's slots (a slot given back for each slot's bytes of KV pages written): a chat's plan on the 8 GB machine 131 -> 278 Q8_0 slots, 314 -> 591 Q4_K, the disk a decode token 375.9 -> 193.7 MiB (MEASUREMENTS §The KV grown from the store's room); the arrival order (10-05, question 90): a layer's misses read under its present experts' compute, the 8 GB machine's decode 1.073 (Q4_K) and 1.030 (Q8_0), the same bytes and bits (MEASUREMENTS §The arrival order built) |
read and measured; the whole engines raced at R1's close |
| 8 | Speculation | n-gram draft (read); 10-05: v4_ngram_draft first, then the DSpark MTP head (v4_dspark_draft, deepseek_v4.c:13355-13408) with 128 experts of its own, 3 a token (a fifth of a main token), the verify one pass over the drafts' union of experts (v4_moe_batch_union, :5607) |
read 10-05: trained heads only (DSpark, up to 5 tokens, a 5.6 GiB support GGUF; GLM's and Qwen3.8's MTP), the verify a batched chain, declared not byte-identical (docs/SPECULATIVE_DECODING.md:13-104) | examples/lookup, common/ngram-cache.cpp (read 09-27: a context cache of the text, n = 1..4, lax gate, validated by a corpus cache of 2-token contexts); 10-05 (common/speculative.cpp:34-44): a draft model, EAGLE3, MTP, DFlash, DSpark and four n-gram drafts, ik a suffix tree too; the verify a chain in one llama_decode; none drafts with the main MoE's own (resident, fewer, coarser) experts |
its draft policy replayed on our 18 new-code texts at our prices (09-27): draft 16 0.80x, draft 3 1.02x; our calibrated gate with its context cache 1.077x (MEASUREMENTS §The post-it taken apart) | llama.cpp's policy measured (replay); colibri's not raced. With a real context (question 81, 20 prompts cut mid-file from this repo): their draft 3 1.096x, our engine's lookup 1.067x, the gate 1.146x at Q4_K's measured prices (MEASUREMENTS §Drafts with a real context). Question 94 (10-05): a draft from the experts the store holds, ours alone (none of the four has one): it agrees 0.89-0.98 on code, 0.50-0.88 on prose, and loses on every machine (0.74-0.98x, model, not measured: MEASUREMENTS §A draft from the bytes the machine holds) |
| 9 | Quantized formats (Q4_K, Q6_K, next Q2_K, IQ2) | — | — | ggml's and ik's kernels (read 09-24, this file §Q4_K, §Q6_K) | llama.cpp on our Q4_K: decode 1.07× ours, prefill 2.1× (int8 activations) | read and measured |
| 10 | GPU: the decode's attention, then the dense weights, then the experts | not read | GPU execution model (noted 09-17, not read in detail) | CUDA backend: not read | none | debt, before M3's next piece |
| 11 | GGUF reader and tokenizer | pretokenizer in C (read) | vocab from metadata (read) | type table (ported) | tokenizer 22 MB/s; not raced | read |
| 12 | The activations prepared for an integer road (once a row) | the MoE's x quantized to int8 once a layer (pq_build), each expert's rows copied from it and published in a global pointer-matched context (g_pq, cleared after each use); the down's h once in parallel; attention's projections take F32 (int8 costs +12% perplexity there) (read 09-27, colibri.c 1130, 1555-1600, 6141-6160) |
one q8_K quantization a token for every routed expert, gate and up in one pair kernel on it (ds4_vec_dot_iq2_xxs_pair_q8_K); pair matvecs (gate+up, kv+gate) on one quantized input (read 09-27, ds4.c 9807-9837, 13005-13035) |
llama.cpp: src1 converted once an op (q, k, v three times; mul_mat_id once a token row for all experts, gate and up twice); ik_llama.cpp: merge_qkv at load (one matmul, the mmap given up), fused up/gate (ggml_compute_forward_mul_mat_id_up_gate: one conversion a token) (read 09-27) |
ours before: 3 for q/k/v, 16 a token for gate/up (8 gathered copies x 2); after: 1 and 1, read in place through a map, no gather, no copy (MEASUREMENTS §The prep once a row) | built 2026-09-27 (P1+P2, then P4: swiglu with the down's prep in one call, where colibri quantizes h after the swiglu in a call of its own); gate+up in one call next, the prep's bit arithmetic last |
| 13 | The KV's pages at their first write | the KV allocated to the prompt plus the tokens asked, grown when needed (kv_alloc, [kv_head][rows][hd]), untouched: its pages fault inside the writes (read 09-27, inkling.c 1906) |
calloc (ds4.c 2673): fresh zero pages, faulted inside the writes (read 09-27) |
llama.cpp: ggml_backend_buffer_clear(buf, 0) when the context is made (llama-kv-cache.cpp 293): every page faulted before the prompt, the whole context present (read 09-27) |
ours before: 512 faults on the main thread every 8 positions (~250 us, 29 us a token), a first prompt's 11-16 ms serialized over the pool; a window touched ahead: the attention 2.4% slower (pages present past the streams' ends); the pass's own pages touched over the pool: the decode +0.17%, a first prompt +0.4%, the attention level (MEASUREMENTS §The KV's pages touched in time) | built 2026-09-27; llama.cpp's cleared context is all present past the ends (not raced); question 84 answered 09-27: the weights' region ends cost ~0.8% of a dense call, nothing on an expert's (a lever of 0.2-0.4% of a token in the dense layout, with piece 4) |
| 14 | The router's matrix | F32: "norms, router, bias stay f32 (small and sensitive)" (read 09-27, colibri.c 13, 393) | F32 on the CPU (tensor_expect_layout(l->ffn_gate_inp, DS4_TENSOR_F32, ...), ds4.c 5605); an F16 router kernel on its GPU for its own format (ds4_gpu_matmul_f16_router_rows_exact_tensor) (read 09-27) |
llama.cpp: ffn_gate_inp F32 in the GGUF, ggml's F32 dot (read 09-27) |
OLMoE's routers are bf16 values stored F32 (every low half zero); narrowed at load where exact, widened by a shift: the router zone 179-186 -> 120-130 us a token, the same bits (MEASUREMENTS §The routers as bf16) | built 2026-09-27 |
| 15 | The output head of a greedy token (every vocabulary row, then the argmax) | one matmul_qt over the vocabulary, h int8 once (matmul_q_idot, colibri.c 1168-1240), the head kept at higher precision (2362-2367); a serial argmax (c/sample.h 20-24); its MTP draft scores the full vocabulary too (read 10-05) |
matvec_q8_0 over every row, h Q8_0 once (ds4.c 9722-9744, 16178-16189); a serial 8x-unrolled argmax (44039); one pruned head, the Qwen4 MTP draft's (an id prefix or a ranked subset, 58197-58207: not exact, its verify rows keep the whole head) (read 10-05) |
llama.cpp: the full output matmul after ggml_get_rows of the outputs (src/models/llama.cpp 174-240), h Q8_0 / Q8_K, the head repacked like any weight (ggml-cpu/repack.cpp 4778-4785), a serial greedy scan (llama-sampler.cpp 1053-1059); ik: the same, -rtr repacks the head (iqk_quantize.cpp 8603-8699), the head split across devices by vocabulary slices (read 10-05). None prunes rows for an exact argmax |
ours today reads the head at the RAM's speed (1.92-2.03 ms Q8_0, 56 GB/s). The argmax by a bound (question 73, ours: each row's scales and top bits against h's two int8 digits, exact only the rows that can still win) in a bench: 16 threads Q8_0 2102 -> 1162-1278 us, Q4_K 1141 -> 928-1048 (MEASUREMENTS §The head's argmax by a bound) | read and measured 2026-10-05; built and raced 2026-10-05 (phase 2): ~10 rows computed a greedy token, the head Q8_0 2.01-2.08 -> 1.08-1.13 ms, Q4_K 1.07-1.12 -> 0.84-0.89 (weak 2.60 -> 1.35); the decode Q8_0 1.041-1.046x, Q4_K 1.009-1.022x, the same tokens |
Programs the project runs but never links: they stay in a container image, never in src/.
| Tool | Version | Licence | Used for | Where |
|---|---|---|---|---|
| CBMC (Kroening, Clarke et al.; Oxford, ETH Zurich, CMU) | 5.95.1, Ubuntu 24.04's cbmc package |
BSD-4-Clause | the bounded model checker of the head's bound proof: the real head_bound.c compiled with nondeterministic inputs (2026-10-05) |
tools/docker/Dockerfile.proof, tools/proof.sh, tests/proof/ |
| MiniSat (Eén, Sörensson) | 2.2.1, Ubuntu's minisat, CBMC's dependency |
MIT | CBMC's SAT solver | the same image |
The proof's written part leans on the standard model of float rounding and its error bound for sums along a fixed tree (N. J. Higham, Accuracy and Stability of Numerical Algorithms, 2nd ed., SIAM 2002, §2.2 and §4.2): cited, no code. The question that started it, a proof that covers the code and not a model of it, came from antirez's post on RedLock proved in TLA+ by an AI, whose stated boundary was exactly that (brought by Marcello, 2026-10-05).
Proposed by Marcello to improve and evolve Trochilus; my first read of each, from what they declare and what I know of them, not from reading their code. Their numbers are their authors' claims: each counts only once measured here, with its prediction written first (MEASUREMENTS §Open questions). When one is studied, its row moves to the tables above (ported, or ideas taken) with its commit.
| Project | What it is | What we would take | When | First read |
|---|---|---|---|---|
| ktransformers (kvcache-ai) | MoE inference on consumer hardware, CPU and GPU together: hot experts on the GPU, the rest on the CPU with its own kernels | the CPU/GPU split of the experts and how it decides what goes where; its CPU MoE kernels | M3 (CUDA module, hot experts in VRAM): it is M3 almost to the letter | yes, at M3; little to take before |
| Adaptive-K routing | fewer than top-k experts for a token when the router is confident (its entropy); declared 24-33% less compute, perplexity within 0.5% (Mixtral, Qwen-MoE) | the idea, as our own experiment: a declared, non-exact mode | M1/M2, as a question (MEASUREMENTS question 50) | interesting here more than for them: under a partial budget every expert skipped is a unit not read from disk, and the disk is our bottleneck. Our tools measure it (expert mask, route trace, KL). Caution from our data: switching off the least-used experts overall changed the token in 6-19% of positions (§Route trace); dropping the lowest-weight ones per token should cost far less |
| ArcLight | an architecture for many-core CPUs and NUMA: memory placement and thread scheduling against cross-NUMA access; declared up to +46% throughput | pool and memory placement per NUMA node | when Trochilus targets servers | later: the reference laptop has one NUMA node, nothing to gain here today. Not known to me first-hand: verify it exists as described before counting on it |
| PCoMoE | fine-grained paths inside the experts instead of whole experts; declared up to 1.31× end to end with better accuracy | — | not now | no, for now: from what is declared it seems to need modified or retrained models, and Trochilus runs models as they are. Verify before ruling it out for good |
| bitnet.cpp (Microsoft) | an engine for ternary (1.58-bit) models, lookup-table kernels | its lookup-table kernels, for the 1-2 bit formats of M2 (TQ1_0, TQ2_0, IQ2), which our GGUF reader already knows | M2, the extreme formats | later: it runs only models trained ternary, and none exists for OLMoE or DeepSeek; the kernels are what could serve |