feat(bench): SPEED-Bench task splits + speculative spec/quant warnings (v0.28.0)

[konjoinfinity]

Konjo sprint — v0.28.0

Fifth and final sprint from this session's Discovery sweep (#15 CyclicJudge, #16 reliability, #17 KV eviction/quant, #18 adaptive early exit — all merged). The Medium/Low closer.

Why

Speculative-decoding speedup and early-exit savings are strongly task-dependent — a model drafts structured code or repetitive text far more predictably than open-ended dialogue — so a single aggregate latency number hides the variance that actually matters at deploy time. Separately, AutoProfile happily recommended speculative decoding without flagging configurations (4-bit drafts, tree drafts) that quietly erode the very speedup it promises.

What

New kairu/speed_bench.py — SPEED-Bench semantic task splits (pure stdlib + kairu.bench):

• run_speed_bench(model, splits=None, …) → SpeedBenchReport runs the benchmark across named task splits and reports per-split throughput plus a throughput coefficient of variation — one number quantifying how task-sensitive the configuration is (0.0 → uniform across tasks).
• TaskSplit + six DEFAULT_SPLITS (translation, summarization, qa, code, dialogue, math), each with a deterministic integer prompt so MockModel runs fully offline; real tokenised prompts plug in behind the same contract.
• SplitResult / SpeedBenchReport expose as_dict() for JSON transport.

kairu/bench.py — BenchmarkRunner gains an optional prompt arg (defaults to the fixed class prompt) so SPEED-Bench drives distinct splits through the same runner. Existing behaviour unchanged.

kairu/auto_profile.py — speculative spec/quant warnings:

• DecoderProfile gains warnings: tuple[str, ...] = ().
• recommend(…, quant=None, draft_kind=None) flags a 4-bit draft (int4/4bit/nf4/fp4 — tanks draft–target acceptance) and a tree-structured draft (the sequential verifier here cannot exploit it). Case-insensitive; empty unless the hints are supplied, so existing recommendations are byte-identical.

Quality gates

• 746 passed, 4 HF-gated skipped (+14 new).
• New tests/test_speed_bench.py (9 tests) at 100% module coverage; auto_profile.py stays 100% (+5 warning tests).
• ruff lint + format clean; speed_bench.py 149 lines, bench.py 443 (both < 500); recommend stays grade C cc=15 (the warning helper added zero branches to it — all logic lives in _speculative_warnings).

Reuse note

run_speed_bench composes the existing BenchmarkRunner per split rather than duplicating the timing loop; the split logic lives in its own module to keep bench.py under the 500-line gate. No HF/torch — MockModel exercises every path offline.

Closes this session's researched Discovery sweep across both tracks: eval (CyclicJudge, reliability metrics) and inference-optimizer (KV eviction + quant tier, adaptive early exit, SPEED-Bench).

🤖 Generated with Claude Code

https://claude.ai/code/session_01TmSqgYqcYfzPnkpDVR5vdw

---

Generated by Claude Code