Summary
bench/quality/README.md documents the quality-parity A/B path as:
python3 run_quality.py --backend llama-server --llama-url http://localhost:8082 ...
and references orchestrator bench/quality/run_qwen36_benchmark.sh.
Neither exists today:
run_quality.py only accepts oracle|mock|sparkinfer|llama — argparse rejects llama-server
run_qwen36_benchmark.sh is missing
So the documented workflow for proving “same GSM8K/IFEval/… as llama.cpp at higher tok/s” cannot run.
Expected
--backend llama-server + --llama-url against llama.cpp /completion (fallback /v1/completions)
run_qwen36_benchmark.sh runs sparkinfer then llama-server exclusively
- CPU-only
--self-test-llama-server mock to verify wiring without a GPU
Scope
bench/quality/ only (not bench/scripts/). No GPU required to review/verify the self-test.
Summary
bench/quality/README.mddocuments the quality-parity A/B path as:and references orchestrator
bench/quality/run_qwen36_benchmark.sh.Neither exists today:
run_quality.pyonly acceptsoracle|mock|sparkinfer|llama— argparse rejectsllama-serverrun_qwen36_benchmark.shis missingSo the documented workflow for proving “same GSM8K/IFEval/… as llama.cpp at higher tok/s” cannot run.
Expected
--backend llama-server+--llama-urlagainst llama.cpp/completion(fallback/v1/completions)run_qwen36_benchmark.shruns sparkinfer then llama-server exclusively--self-test-llama-servermock to verify wiring without a GPUScope
bench/quality/only (notbench/scripts/). No GPU required to review/verify the self-test.