Skip to content

fix(quality): documented llama-server backend and run_qwen36_benchmark.sh are missing #643

Description

@kurosawareiji7007-hub

Summary

bench/quality/README.md documents the quality-parity A/B path as:

python3 run_quality.py --backend llama-server --llama-url http://localhost:8082 ...

and references orchestrator bench/quality/run_qwen36_benchmark.sh.

Neither exists today:

  • run_quality.py only accepts oracle|mock|sparkinfer|llama — argparse rejects llama-server
  • run_qwen36_benchmark.sh is missing

So the documented workflow for proving “same GSM8K/IFEval/… as llama.cpp at higher tok/s” cannot run.

Expected

  1. --backend llama-server + --llama-url against llama.cpp /completion (fallback /v1/completions)
  2. run_qwen36_benchmark.sh runs sparkinfer then llama-server exclusively
  3. CPU-only --self-test-llama-server mock to verify wiring without a GPU

Scope

bench/quality/ only (not bench/scripts/). No GPU required to review/verify the self-test.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions