Skip to content

tests(fp8): xfail the M=1 scaled_mm apply cases - #821

Merged
joerunde merged 1 commit into
torch-spyre:mainfrom
sducouedic:skip_failing_test
Sep 9, 2026
Merged

tests(fp8): xfail the M=1 scaled_mm apply cases#821
joerunde merged 1 commit into
torch-spyre:mainfrom
sducouedic:skip_failing_test

Conversation

@sducouedic

Copy link
Copy Markdown
Contributor

Description

The two FP8 apply_weights tests fail at num_tokens=1: dxp_standalone rejects the fused activation-quantize + _scaled_mm bundle with sbf-ddc: DtException: Illegal ddl (ddl_conversion.cpp:1133). The num_tokens=4/128 cases build the same kernel and pass, so the failure is specific to the M=1 decode bundle, not the FP8 linear path. This marks only that parametrization xfail and keeps the other two shapes as real coverage.

The marker is non-strict deliberately: the shape still compiles on some deeptools builds, and an xpass is the signal that the backend fix landed and the marker can be removed.

Test Plan

  • uv run pytest tests/test_fp8_linear.py -m "not upstream" — passed (the M=1 cases xpass on this host).
  • bash format.sh — clean for the touched file.

Checklist

  • I have read the contributing guidelines
  • My code follows the project's code style (run bash format.sh)
  • I have added tests for my changes (if applicable)
  • I have updated the documentation (if applicable)
  • My commits include a Signed-off-by: line (DCO compliance)

The two `apply_weights` tests fail at num_tokens=1: dxp_standalone rejects
the fused activation-quantize + _scaled_mm bundle with `sbf-ddc:
DtException: Illegal ddl` (ddl_conversion.cpp:1133). num_tokens=4/128 build
the same kernel and pass, so the failure is specific to the M=1 decode
bundle rather than the FP8 linear path.

Non-strict on purpose: the shape still compiles on some deeptools builds,
and an xpass is the signal the backend fix landed.

Signed-off-by: Sophie du Couédic <sducouedic@users.noreply.github.com>
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! Thank you for contributing.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, run ./format.sh.
Now you are good to go 🚀.

We also recommend installing prek and configuring it to check your code before every local commit.

@sducouedic
sducouedic marked this pull request as ready for review September 9, 2026 15:41
@sducouedic
sducouedic requested a review from a team as a code owner September 9, 2026 15:41

@joerunde joerunde left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#818 tried to bump rpms to find a fix but we still see failures :(

@joerunde
joerunde added this pull request to the merge queue Sep 9, 2026
@spyre-ci

spyre-ci Bot commented Sep 9, 2026

Copy link
Copy Markdown

❌ merge-queue-integration: failure

Plan (build waves + dependencies, per arch)

amd64

flowchart LR
  subgraph Lamd64_0["amd64 L0 · 1 parallel"]
    n_amd64_torch_spyre_torch_spyre_dev["torch-spyre/torch-spyre-dev 🟢<br/>image · 2fdfd201c1b0"]
  end
  subgraph Lamd64_1["amd64 L1 · 1 parallel"]
    n_amd64_hf_adapters_hf_adapters_dev["hf-adapters/hf-adapters-dev 🟢<br/>image · 6beebcd33470"]
  end
  subgraph Lamd64_2["amd64 L2 · 1 parallel"]
    n_amd64_spyre_inference_spyre_inference_dev["spyre-inference/spyre-inference-dev 🔴<br/>image · 329004ccf99a"]
  end
  n_amd64_torch_spyre_torch_spyre_dev --> n_amd64_hf_adapters_hf_adapters_dev
  n_amd64_hf_adapters_hf_adapters_dev --> n_amd64_spyre_inference_spyre_inference_dev
  classDef sPending fill:#eceff1,stroke:#90a4ae,color:#37474f
  classDef sBuilding fill:#fff8e1,stroke:#f9a825,color:#5d4037,stroke-width:2px
  classDef sOk fill:#e8f5e9,stroke:#43a047,color:#1b5e20
  classDef sReused fill:#e3f2fd,stroke:#1e88e5,color:#0d47a1
  classDef sFailed fill:#ffebee,stroke:#e53935,color:#b71c1c,stroke-width:2px
  classDef sDropped fill:#f5f5f5,stroke:#bdbdbd,color:#9e9e9e
  class n_amd64_torch_spyre_torch_spyre_dev sOk;
  class n_amd64_hf_adapters_hf_adapters_dev sOk;
  class n_amd64_spyre_inference_spyre_inference_dev sOk;
Loading

✅ orch trigger-pr-validationgreen · arches amd64 · fp amd64=23044001

level component arch build smoke unit integration trunk regression perf
L0 torch-spyre/torch-spyre-dev amd64 ✅ ok · 🟢 gha · · ·
L1 hf-adapters/hf-adapters-dev amd64 ✅ ok · 🟢 gha · · ·
L2 spyre-inference/spyre-inference-dev amd64 ✅ ok 🟢 · · · 🟡 gha ·

GHA test runs:


⚠️ advisory failures only — mergeable, see below

Build: built 3

Tests: passed 4 · blocking 0 · advisory 2 · infra/inconclusive 0 · no signal 0

Failures by kind
  • ⚠️ advisory (does not block) · spyre-inference/amd64 regression: UNSTABLE
  • ⚠️ advisory (does not block) · spyre-inference/amd64 gha:regression: FAILURE

Before merging, consider:

  • spyre-inference/amd64 gha:regression, spyre-inference/amd64 regression failed with gating: "unstable" — advisory. It set the build UNSTABLE but does not block the merge. Worth a look, not a stop.

@spyre-ci

spyre-ci Bot commented Sep 9, 2026

Copy link
Copy Markdown

GHA test runs:

@spyre-ci

spyre-ci Bot commented Sep 9, 2026

Copy link
Copy Markdown

GHA test runs:

@spyre-ci

spyre-ci Bot commented Sep 9, 2026

Copy link
Copy Markdown

GHA test runs:

Merged via the queue into torch-spyre:main with commit ba2047a Sep 9, 2026
50 checks passed
HarikrishnanBalagopal added a commit to HarikrishnanBalagopal/spyre-inference that referenced this pull request Sep 10, 2026
Reset onto current main (c8fc56b, incl. PR torch-spyre#821 which marks the fp8 M=1
scaled-mm case xfail(strict=False)). Redirects the spyre-card test matrix onto
the image_spyre_inference_shadow runner set (which boots the /next
spyre-inference image) and pins the same _335 deeptools RPM set used in the
recorded fp8 controlled experiment, so this run is directly comparable.

Purpose: re-test the fp8 gate on the latest /next image. With torch-spyre#821 the M=1
case now xfails instead of failing the suite; an xpass would signal the /next
image bug is genuinely fixed, an xfail means the suite is green only because
the underlying fp8 M=1 DDC compile bug is masked, not fixed.

Signed-off-by: Harikrishnan Balagopal <harikrishmenon@gmail.com>
sducouedic added a commit to sducouedic/spyre-inference that referenced this pull request Sep 10, 2026
)"

This reverts commit ba2047a.

Signed-off-by: Sophie du Couédic <sop@zurich.ibm.com>
joerunde added a commit that referenced this pull request Sep 10, 2026
## Description

Bumps `torch-spyre` from `7697811` to `c3d949a` — all 13 upstream
commits since the previous pin. The tip of `main` compiled cleanly
against the currently-installed RPMs (no bisect needed). Also refreshes
the full `uv.lock` (`uv lock --upgrade`); `vllm` is held at its pinned
git tag (`v0.28.0`) and intentionally **not** upgraded.

NB: We had a prior bump PR in today (#818) that was attempting to clear
out a regression. That didn't work so we instead disabled tests (#821),
but we still do need to keep upgrading

### Notable upstream changes in this range

- **Enable sliding window attention** (#3405) — new attention path.
- **Lx dump restore** (#3879) — dump/restore support.
- **Demote non-critical warnings to info/debug level** (#4374) and
**remove `_get_env_bool` / legacy `SPYRE_INDUCTOR_LOG` helpers** (#4241)
— logging cleanup and removal of legacy env-var helpers.
- **fix attention view error** (#4377) and **codegen: point
`maskingConstId_` at samv-maskvalue, not constant 0** (#4400) —
attention/masking correctness fixes.
- **fix(inductor): preserve offsets in coarse read copies** (#4376) —
inductor lowering fix (tip).

### Transitive dependency refresh

5 transitive packages moved via `uv lock --upgrade`: transformers 5.16.1
→ 5.17.0, openai 3.9.0 → 3.11.0, xgrammar 0.2.3 → 0.2.6, multidict 6.7.1
→ 6.8.0, filelock 3.32.5 → 3.32.6. `vllm` (v0.28.0) and `torch-spyre`
unchanged (pinned).

### Installed `ibm-*` packages on the build host

```
ibm-aiu-toolbox-e2e-2.0.0-0.main.1+28.47d9b91_101.el10.x86_64
ibm-deeptools-2.0.0-0.main.1+2429.561f418_335.el10.x86_64
ibm-deeptools-devel-2.0.0-0.main.1+2429.561f418_335.el10.x86_64
ibm-flex-2.0.0-0.main.1+553.8581a91_384.el10.x86_64
ibm-flex-devel-2.0.0-0.main.1+553.8581a91_384.el10.x86_64
ibm-libaiupti-2.0.0-0.main.1+27.e289f7d_13.el10.x86_64
ibm-senlib-core-2.0.0-0.main.1+268.c58bbc0_251.el10.x86_64
ibm-senlib-dd2-2.0.0-0.main.1+268.c58bbc0_251.el10.x86_64
ibm-senlib-headers-2.0.0-0.main.1+268.c58bbc0_251.el10.x86_64
ibm-spyre-comms-1.0.0-0.main.1+146.a445334_182.el10.x86_64
ibm-spyre-comms-devel-1.0.0-0.main.1+146.a445334_182.el10.x86_64
```

`spyre-rpms.lock` bumped to match: flex `7c58aa7` → `8581a91` and
spyre-comms `d3875d2` → `a445334` (new commits);
deeptools/senlib/aiu-toolbox/libaiupti advanced to newer builds of the
same commit. No downgrades.

## Test Plan

- [x] `uv lock` resolves cleanly to `c3d949a`
- [x] `uv lock --upgrade` refreshes all transitives (vllm held at its
git tag)
- [x] `uv sync --frozen` builds the torch-spyre C++ extension
successfully
- [x] Smoke test (`test_basic_model_load`) passes locally
- [ ] Full CI suite passes (pushed for CI validation)
- [x] `spyre-rpms.lock` updated — no downgrades (commit-count check
passed); resolves + downloads on all arches

**Reviewer note:** when pulling this branch onto an existing checkout,
`rm -rf /tmp/torchinductor_*` before running tests — the cache bakes in
references to internals that were renamed/removed across the bump.

---------

Signed-off-by: Joe Runde <joe@joerun.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants