Skip to content

fix(tests): make -k <registry-key> work in smoke test - #500

Open
ahsi5086 wants to merge 2 commits into
torch-spyre:mainfrom
ahsi5086:fix/smoke-test-k-filter-by-registry-key
Open

fix(tests): make -k <registry-key> work in smoke test#500
ahsi5086 wants to merge 2 commits into
torch-spyre:mainfrom
ahsi5086:fix/smoke-test-k-filter-by-registry-key

Conversation

@ahsi5086

@ahsi5086 ahsi5086 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Problem

pytest -k gemma4_moe collected 0 tests despite gemma4_moe being a
valid registry entry. The smoke test was parametrized with raw HF paths
as test IDs (e.g. google/gemma-4-26B-A4B-it), so -k gemma4_moe had
nothing to match against. Only models whose HF path happened to contain
the key name as a substring (e.g. qwen3Qwen/Qwen3-0.6B) worked
by accident.

Fix

Introduce CAUSAL_PARAMS — a list of pytest.param(path, id=key)
objects where the registry key name is the test ID. Switch
test_e2e_smoke_spyre.py to use CAUSAL_PARAMS instead of
CAUSAL_PATHS. Update xfail_non_blocking to handle both plain strings
(all existing callers are unchanged) and pytest.param inputs.

CAUSAL_PATHS is untouched — every other test file that uses
ids=CAUSAL_PATHS continues to work exactly as before.

Add scripts/run_smoke_by_key.py for models intentionally excluded from
the default collection by the smallest-per-adapter selection rule (e.g.
granite8b loses to granite2b for hf_granite.py). These can't be
reached via -k since they're never collected — the script runs the
smoke test for any registry key directly.

Usage

# Now works — previously collected 0 tests
pytest -s -vvv tests/spyre/test_e2e_smoke_spyre.py -k gemma4_moe

# Also works for any other registry key
pytest -s -vvv tests/spyre/test_e2e_smoke_spyre.py -k qwen3
pytest -s -vvv tests/spyre/test_e2e_smoke_spyre.py -k granite2b

# For models not in the default collection (shares adapter with smaller model)
python scripts/run_smoke_by_key.py granite8b
python scripts/run_smoke_by_key.py --list   # show all available keys

CAUSAL_PATHS used raw HF paths as pytest IDs, so -k gemma4_moe couldn't
match google/gemma-4-26B-A4B-it. Introduce CAUSAL_PARAMS as a list of
pytest.param(path, id=key) objects so test IDs are registry key names.
Update xfail_non_blocking to handle both plain strings and pytest.param
inputs (backward compat with all other callers that still use CAUSAL_PATHS).

Add scripts/run_smoke_by_key.py for models excluded from CAUSAL_PATHS by
the smallest-per-adapter selection (e.g. granite8b) — runs the smoke test
for any registry key directly without touching the default collection.

Fixes: -k gemma4_moe now collects 1 test (was 0)
Default run unchanged: 23 tests, same models as before

Signed-off-by: Ahmed Siddiqui <Ahmed.S@ibm.com>
@ahsi5086
ahsi5086 force-pushed the fix/smoke-test-k-filter-by-registry-key branch from 4d56e63 to bc1e3b8 Compare September 9, 2026 20:50
@ahsi5086
ahsi5086 marked this pull request as ready for review September 10, 2026 17:12
@arielge

arielge commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@ahsi5086 actually the most consistent approach IMO is just to fix the documentation (#451), the entire testing mechanism relies on paths

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants