From 15d825b7745d42900bef707cc3fc2e756f8787d9 Mon Sep 17 00:00:00 2001 From: ai-hpc Date: Fri, 12 Jun 2026 02:57:41 +0200 Subject: [PATCH] docs: cross-link the edge-stt benchmark + note the q5_k quant option Tie the deeper edge-stt eval (full LibriSpeech-300, GPU dtype sweep, 3-mode harness) into the STT-swap doc and the parakeet-stt deploy README, and surface its main feedback: q5_k matches q8_0 accuracy at 23% smaller with no speed cost (free PK_MODEL swap), plus reconcile the 50-clip vs 300-clip numbers. Docs-only; no runtime/contract change. --- deploy/parakeet-stt/README.md | 4 ++++ docs/stt-parakeet-evaluation.md | 12 ++++++++++++ 2 files changed, 16 insertions(+) diff --git a/deploy/parakeet-stt/README.md b/deploy/parakeet-stt/README.md index c0ead19..4bf89c7 100644 --- a/deploy/parakeet-stt/README.md +++ b/deploy/parakeet-stt/README.md @@ -5,6 +5,10 @@ Resident parakeet.cpp STT that drop-in replaces whisper-server on genie-core needs no change). See [`../../docs/stt-parakeet-evaluation.md`](../../docs/stt-parakeet-evaluation.md) for the whisper-vs-parakeet comparison and rationale. +For a deeper benchmark of the same engine — full **LibriSpeech-300** WER/RTF/memory, a GPU **dtype +sweep** (which `q*_k` to use), and a 3-mode (offline / streaming / multilingual) harness behind one +CLI — see [`edge-stt/`](edge-stt/) (the Edge-AI STT take-home built around this engine). + Contents: - `serve-mode.patch` — adds a resident `serve` subcommand to parakeet.cpp's CLI (load model once, transcribe wav paths read from stdin). Against `v0.1.1`. diff --git a/docs/stt-parakeet-evaluation.md b/docs/stt-parakeet-evaluation.md index 4882730..13c54c0 100644 --- a/docs/stt-parakeet-evaluation.md +++ b/docs/stt-parakeet-evaluation.md @@ -25,6 +25,15 @@ model. Net: **more accurate, ~6× faster compute, ~3× lower per-call latency, ~300 MB leaner.** +> **Deeper benchmark of this same engine** — full **LibriSpeech-300** (not 50), an on-GPU **dtype +> sweep**, and a 3-mode (offline / streaming / multilingual) harness behind one CLI — lives in +> [`../deploy/parakeet-stt/edge-stt/`](../deploy/parakeet-stt/edge-stt/). Two takeaways feed back here: +> (1) on the full 300 the WER is **1.808 % (tdt, q5_k)** / **2.383 % (ctc)** — higher than the 0.82 % +> above because it's the full set, not 50 clean clips, and the RTF there is the per-file inference +> number (61× tdt / 107× ctc), not the amortized 116×. (2) the dtype sweep found **RTFx is +> dtype-invariant on this GPU**, so **`q5_k` matches `q8_0` accuracy at 23 % smaller** (137 MB) with no +> speed cost — a free win for the resident server (set it via `PK_MODEL`). + ## Test setup - **Device:** Jetson Orin Nano Super 8 GB · JetPack 6.2 (L4T 36.4.7) · CUDA 12.6 · @@ -79,6 +88,9 @@ exact build + install + revert commands. - `tdt_ctc-110m` is the **lean** model. `tdt-0.6b-v2-q8` is more accurate (~1.7 % on the leaderboard) but ~904 MB and heavier to load/run — swap it in via the service's `PK_MODEL` env if accuracy outweighs footprint. +- **Quant choice:** the edge-stt dtype sweep (see link above) shows `q5_k` matches `q8_0` accuracy + at **23 % smaller** (137 MB) with no speed cost on this GPU — a free swap via `PK_MODEL=…-q5_k.gguf`. + `q4_k` (126 MB) is the next rung down if memory tightens (still < 3 % WER). - Numbers are `q8_0` quantization on this exact device; treat as device-specific. ## Build notes (Jetson, parakeet.cpp v0.1.1)