tests(fp8): xfail the M=1 scaled_mm apply cases - #821
Merged
Conversation
The two `apply_weights` tests fail at num_tokens=1: dxp_standalone rejects the fused activation-quantize + _scaled_mm bundle with `sbf-ddc: DtException: Illegal ddl` (ddl_conversion.cpp:1133). num_tokens=4/128 build the same kernel and pass, so the failure is specific to the M=1 decode bundle rather than the FP8 linear path. Non-strict on purpose: the shape still compiles on some deeptools builds, and an xpass is the signal the backend fix landed. Signed-off-by: Sophie du Couédic <sducouedic@users.noreply.github.com>
Contributor
|
👋 Hi! Thank you for contributing. We also recommend installing prek and configuring it to check your code before every local commit. |
sducouedic
marked this pull request as ready for review
September 9, 2026 15:41
❌ merge-queue-integration: failurePlan (build waves + dependencies, per arch)amd64 flowchart LR
subgraph Lamd64_0["amd64 L0 · 1 parallel"]
n_amd64_torch_spyre_torch_spyre_dev["torch-spyre/torch-spyre-dev 🟢<br/>image · 2fdfd201c1b0"]
end
subgraph Lamd64_1["amd64 L1 · 1 parallel"]
n_amd64_hf_adapters_hf_adapters_dev["hf-adapters/hf-adapters-dev 🟢<br/>image · 6beebcd33470"]
end
subgraph Lamd64_2["amd64 L2 · 1 parallel"]
n_amd64_spyre_inference_spyre_inference_dev["spyre-inference/spyre-inference-dev 🔴<br/>image · 329004ccf99a"]
end
n_amd64_torch_spyre_torch_spyre_dev --> n_amd64_hf_adapters_hf_adapters_dev
n_amd64_hf_adapters_hf_adapters_dev --> n_amd64_spyre_inference_spyre_inference_dev
classDef sPending fill:#eceff1,stroke:#90a4ae,color:#37474f
classDef sBuilding fill:#fff8e1,stroke:#f9a825,color:#5d4037,stroke-width:2px
classDef sOk fill:#e8f5e9,stroke:#43a047,color:#1b5e20
classDef sReused fill:#e3f2fd,stroke:#1e88e5,color:#0d47a1
classDef sFailed fill:#ffebee,stroke:#e53935,color:#b71c1c,stroke-width:2px
classDef sDropped fill:#f5f5f5,stroke:#bdbdbd,color:#9e9e9e
class n_amd64_torch_spyre_torch_spyre_dev sOk;
class n_amd64_hf_adapters_hf_adapters_dev sOk;
class n_amd64_spyre_inference_spyre_inference_dev sOk;
✅ orch trigger-pr-validation — green · arches amd64 · fp amd64=23044001
GHA test runs:
|
|
GHA test runs:
|
|
GHA test runs:
|
|
GHA test runs:
|
6 tasks
HarikrishnanBalagopal
added a commit
to HarikrishnanBalagopal/spyre-inference
that referenced
this pull request
Sep 10, 2026
Reset onto current main (c8fc56b, incl. PR torch-spyre#821 which marks the fp8 M=1 scaled-mm case xfail(strict=False)). Redirects the spyre-card test matrix onto the image_spyre_inference_shadow runner set (which boots the /next spyre-inference image) and pins the same _335 deeptools RPM set used in the recorded fp8 controlled experiment, so this run is directly comparable. Purpose: re-test the fp8 gate on the latest /next image. With torch-spyre#821 the M=1 case now xfails instead of failing the suite; an xpass would signal the /next image bug is genuinely fixed, an xfail means the suite is green only because the underlying fp8 M=1 DDC compile bug is masked, not fixed. Signed-off-by: Harikrishnan Balagopal <harikrishmenon@gmail.com>
sducouedic
added a commit
to sducouedic/spyre-inference
that referenced
this pull request
Sep 10, 2026
)" This reverts commit ba2047a. Signed-off-by: Sophie du Couédic <sop@zurich.ibm.com>
joerunde
added a commit
that referenced
this pull request
Sep 10, 2026
## Description Bumps `torch-spyre` from `7697811` to `c3d949a` — all 13 upstream commits since the previous pin. The tip of `main` compiled cleanly against the currently-installed RPMs (no bisect needed). Also refreshes the full `uv.lock` (`uv lock --upgrade`); `vllm` is held at its pinned git tag (`v0.28.0`) and intentionally **not** upgraded. NB: We had a prior bump PR in today (#818) that was attempting to clear out a regression. That didn't work so we instead disabled tests (#821), but we still do need to keep upgrading ### Notable upstream changes in this range - **Enable sliding window attention** (#3405) — new attention path. - **Lx dump restore** (#3879) — dump/restore support. - **Demote non-critical warnings to info/debug level** (#4374) and **remove `_get_env_bool` / legacy `SPYRE_INDUCTOR_LOG` helpers** (#4241) — logging cleanup and removal of legacy env-var helpers. - **fix attention view error** (#4377) and **codegen: point `maskingConstId_` at samv-maskvalue, not constant 0** (#4400) — attention/masking correctness fixes. - **fix(inductor): preserve offsets in coarse read copies** (#4376) — inductor lowering fix (tip). ### Transitive dependency refresh 5 transitive packages moved via `uv lock --upgrade`: transformers 5.16.1 → 5.17.0, openai 3.9.0 → 3.11.0, xgrammar 0.2.3 → 0.2.6, multidict 6.7.1 → 6.8.0, filelock 3.32.5 → 3.32.6. `vllm` (v0.28.0) and `torch-spyre` unchanged (pinned). ### Installed `ibm-*` packages on the build host ``` ibm-aiu-toolbox-e2e-2.0.0-0.main.1+28.47d9b91_101.el10.x86_64 ibm-deeptools-2.0.0-0.main.1+2429.561f418_335.el10.x86_64 ibm-deeptools-devel-2.0.0-0.main.1+2429.561f418_335.el10.x86_64 ibm-flex-2.0.0-0.main.1+553.8581a91_384.el10.x86_64 ibm-flex-devel-2.0.0-0.main.1+553.8581a91_384.el10.x86_64 ibm-libaiupti-2.0.0-0.main.1+27.e289f7d_13.el10.x86_64 ibm-senlib-core-2.0.0-0.main.1+268.c58bbc0_251.el10.x86_64 ibm-senlib-dd2-2.0.0-0.main.1+268.c58bbc0_251.el10.x86_64 ibm-senlib-headers-2.0.0-0.main.1+268.c58bbc0_251.el10.x86_64 ibm-spyre-comms-1.0.0-0.main.1+146.a445334_182.el10.x86_64 ibm-spyre-comms-devel-1.0.0-0.main.1+146.a445334_182.el10.x86_64 ``` `spyre-rpms.lock` bumped to match: flex `7c58aa7` → `8581a91` and spyre-comms `d3875d2` → `a445334` (new commits); deeptools/senlib/aiu-toolbox/libaiupti advanced to newer builds of the same commit. No downgrades. ## Test Plan - [x] `uv lock` resolves cleanly to `c3d949a` - [x] `uv lock --upgrade` refreshes all transitives (vllm held at its git tag) - [x] `uv sync --frozen` builds the torch-spyre C++ extension successfully - [x] Smoke test (`test_basic_model_load`) passes locally - [ ] Full CI suite passes (pushed for CI validation) - [x] `spyre-rpms.lock` updated — no downgrades (commit-count check passed); resolves + downloads on all arches **Reviewer note:** when pulling this branch onto an existing checkout, `rm -rf /tmp/torchinductor_*` before running tests — the cache bakes in references to internals that were renamed/removed across the bump. --------- Signed-off-by: Joe Runde <joe@joerun.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The two FP8
apply_weightstests fail atnum_tokens=1:dxp_standalonerejects the fused activation-quantize +_scaled_mmbundle withsbf-ddc: DtException: Illegal ddl(ddl_conversion.cpp:1133). Thenum_tokens=4/128cases build the same kernel and pass, so the failure is specific to the M=1 decode bundle, not the FP8 linear path. This marks only that parametrizationxfailand keeps the other two shapes as real coverage.The marker is non-strict deliberately: the shape still compiles on some deeptools builds, and an xpass is the signal that the backend fix landed and the marker can be removed.
Test Plan
uv run pytest tests/test_fp8_linear.py -m "not upstream"— passed (the M=1 cases xpass on this host).bash format.sh— clean for the touched file.Checklist
bash format.sh)Signed-off-by:line (DCO compliance)