Skip to content

[Handoff to @Oseltamivir Claude /loop] [Klaud Cold] Add qwen3.5-fp8-mi300x-sglang-mtp recipe #1482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
functionstackx wants to merge 2 commits into
base: main
Choose a base branch
from
+109 −0
Open

[Handoff to @Oseltamivir Claude /loop] [Klaud Cold] Add qwen3.5-fp8-mi300x-sglang-mtp recipe #1482

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b1d955c
Add qwen3.5-fp8-mi300x-sglang-mtp recipe
functionstackx May 18, 2026
92fa7b0
chore: fill pr-link for #1482
functionstackx May 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .github/configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1933,3 +1933,22 @@ glm5-fp8-mi325x-sglang-mtp:
osl: 1024
search-space:
- { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp }

qwen3.5-fp8-mi300x-sglang-mtp:
image: lmsysorg/sglang:v0.5.12-rocm720-mi30x
model: Qwen/Qwen3.5-397B-A17B-FP8
model-prefix: qwen3.5
runner: mi300x
precision: fp8
framework: sglang
multinode: false
scenarios:
fixed-seq-len:
- isl: 1024
osl: 1024
search-space:
- { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp }
- isl: 8192
osl: 1024
search-space:
- { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp }
84 changes: 84 additions & 0 deletions benchmarks/single_node/qwen3.5_fp8_mi300x_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
#!/usr/bin/env bash

source "$(dirname "$0")/../benchmark_lib.sh"

check_env_vars \
MODEL \
TP \
CONC \
ISL \
OSL \
RANDOM_RANGE_RATIO \
RESULT_FILENAME
EP_SIZE \
Comment on lines +5 to +13
Copy link
Copy Markdown
Contributor

@claude claude Bot May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new qwen3.5_fp8_mi300x_mtp.sh recipe has two copy-paste defects that together prevent it from running: (1) a misplaced backslash at lines 12-13 makes check_env_vars end after RESULT_FILENAME and then tries to execute a command literally named EP_SIZE, leaving $EP_SIZE unvalidated; and (2) a stray EP_SIZE \ at line 72 inside run_benchmark_serving is parsed as a positional argument that hits the function's wildcard case and aborts the benchmark. Fix by moving the backslash up to RESULT_FILENAME\ and dropping it after EP_SIZE (lines 12-13), and by deleting line 72 entirely — see the sibling qwen3.5_fp8_mi355x_mtp.sh for the correct structure.

Extended reasoning...

What is broken

The new script benchmarks/single_node/qwen3.5_fp8_mi300x_mtp.sh contains two distinct EP_SIZE-related copy-paste defects. Both can be seen by comparing against the sibling benchmarks/single_node/qwen3.5_fp8_mi355x_mtp.sh, which has the correct shape.

Defect 1 — broken check_env_vars call (lines 5-13).

In the new file:

check_env_vars \
    MODEL \
    ...
    RESULT_FILENAME       # <-- NO trailing backslash, terminates check_env_vars
    EP_SIZE \             # <-- starts a NEW command, backslash continues to blank line 14

In the (correct) mi355x sibling at lines 12-13:

    RESULT_FILENAME \
    EP_SIZE

Defect 2 — stray positional inside run_benchmark_serving (line 72).

run_benchmark_serving \
    ...
    --result-filename "$RESULT_FILENAME" \
    EP_SIZE \                     # <-- stray literal, no `$`, no flag
    --result-dir /workspace/ \
    --use-chat-template

The mi355x sibling has no such line.

Step-by-step proof of failure

Assume the harness invokes the script with EP_SIZE exported (every yaml entry sets ep: 1, so this is the intended happy path):

  1. Line 5-12 — check_env_vars runs with arguments MODEL TP CONC ISL OSL RANDOM_RANGE_RATIO RESULT_FILENAME. All set, so it passes. EP_SIZE is never validated (a real but latent fail-fast regression for the unset-EP_SIZE case).
  2. Line 13 — bash now parses EP_SIZE \ as its own statement. The trailing backslash continues onto the blank line 14, so the resulting command is literally EP_SIZE with no arguments. There is no such command on $PATH, so bash prints EP_SIZE: command not found. The script does not set -e (only set +x at the bottom), so execution silently continues.
  3. Lines 15-60 — sglang starts and the wait completes (assuming EP_SIZE=1 was exported).
  4. Line 62-74 — run_benchmark_serving is invoked. Its argument loop in benchmarks/benchmark_lib.sh (the second one starting around line 187) is a strict while [[ $# -gt 0 ]]; do case $1 in ... *) echo "Unknown parameter: $1"; return 1; ;; esac done. The preceding --result-filename "$RESULT_FILENAME" does a shift 2, so when the loop next sees $1 = "EP_SIZE" (the bare string from line 72), it hits the wildcard at lib.sh:279-282 and returns 1. The benchmark client never runs; no perf data is produced.
  5. Lines 76-84 — set -e is not active, so the script continues into run_eval / stop_gpu_monitor and exits 0 from set +x. The CI job appears to succeed but produced no benchmark output.

Why the broken state is exactly the wrong way around

The EP_SIZE \ artifact appears in both locations (line 13 and line 72), strongly suggesting both were introduced by the same bad insertion. The sibling qwen3.5_fp8_mi355x_mtp.sh (lines 12-13 and 60-72) has neither artifact.

Fix

Three edits in benchmarks/single_node/qwen3.5_fp8_mi300x_mtp.sh:

  1. Line 12: RESULT_FILENAME → RESULT_FILENAME \
  2. Line 13: EP_SIZE \ → EP_SIZE (final argument to check_env_vars)
  3. Delete line 72 ( EP_SIZE \) entirely — $EP_SIZE is already consumed by --ep-size $EP_SIZE on line 43 and has no business being passed to the benchmark client.

After these three edits, bash -n continues to pass and the recipe actually has a chance of producing benchmark results on the first sweep run.


if [[ -n "$SLURM_JOB_ID" ]]; then
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
fi

if [[ "$MODEL" != /* ]]; then hf download "$MODEL"; fi

SERVER_LOG=/workspace/server.log
PORT=${PORT:-8888}
CONTEXT_LENGTH=$((ISL + OSL + 20))
MAX_PREFILL_TOKENS=32768

EVAL_CONTEXT_ARGS=""
if [ "${EVAL_ONLY}" = "true" ]; then
setup_eval_context
EVAL_CONTEXT_ARGS="--context-length $EVAL_MAX_MODEL_LEN"
else EVAL_CONTEXT_ARGS="--context-length $CONTEXT_LENGTH"
fi
# Start GPU monitoring (power, temperature, clocks every second)
start_gpu_monitor

# following AMD Andy linkedin's recipe
# https://www.linkedin.com/feed/update/urn:li:activity:7429203734389280768/
python3 -m sglang.launch_server \
--attention-backend aiter \
--model-path $MODEL \
--host=0.0.0.0 \
--port $PORT \
--tensor-parallel-size $TP \
--ep-size $EP_SIZE \
--trust-remote-code \
--tokenizer-worker-num 6 \
--enable-aiter-allreduce-fusion \
--cuda-graph-max-bs $CONC \
--disable-radix-cache \
--max-prefill-tokens $MAX_PREFILL_TOKENS \
--scheduler-recv-interval 30 \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--mem-fraction-static 0.75 $EVAL_CONTEXT_ARGS > $SERVER_LOG 2>&1 &

SERVER_PID=$!

# Wait for server to be ready
wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"

run_benchmark_serving \
--model "$MODEL" \
--port "$PORT" \
--backend vllm \
--input-len "$ISL" \
--output-len "$OSL" \
--random-range-ratio "$RANDOM_RANGE_RATIO" \
--num-prompts "$((CONC * 10))" \
--max-concurrency "$CONC" \
--result-filename "$RESULT_FILENAME" \
EP_SIZE \
--result-dir /workspace/ \
--use-chat-template

# After throughput, run evaluation only if RUN_EVAL is true
if [ "${RUN_EVAL}" = "true" ]; then
run_eval --framework lm-eval --port "$PORT"
append_lm_eval_summary
fi

# Stop GPU monitoring
stop_gpu_monitor
set +x
6 changes: 6 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3050,3 +3050,9 @@
description:
- "Update SGLang image from v0.5.11-cu130 (5d old) to v0.5.12-cu130"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1475

- config-keys:
- qwen3.5-fp8-mi300x-sglang-mtp
description:
- "Add MTP/EAGLE speculative-decoding sibling of qwen3.5-fp8-mi300x-sglang"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1482
Loading