E2e model quality upstream

.github/workflows/_test_matrix.yaml

@@ -596,7 +596,6 @@ jobs:
             spyre-rpms.lock
             tests/**/*.py
             tests/**/*.yaml
-            tests/data/**
             spyre_inference/**/*.py
             spyre_inference/**/*.yaml
 

.pre-commit-config.yaml

@@ -16,9 +16,6 @@ repos:
   rev: v1.41.0
   hooks:
   - id: typos
-    # Reference data holds tokenizer fragments, which split words mid-token and which
-    # this hook's --write-changes silently rewrites into real words.
-    exclude: '^tests/data/.*\.json$'
 - repo: https://github.com/igorshubovych/markdownlint-cli
   rev: v0.45.0
   hooks:

Makefile

@@ -199,7 +199,7 @@ test-smoke-shard-%:
 	$(MAKE) test-smoke-shard SMOKE_SHARD_ID=$* JUNIT_XML=$(JUNIT_XML)
 
 # Carved out of smoke: every case compiles a product model, up to the 31B decoders.
-test-model-quality: ## Run the product-model output-quality gates against the cached HF references. Unsharded (local full run).
+test-model-quality: ## Run the product-model output-quality gates against live CPU HF. Unsharded (local full run).
 	$(MAKE) run-one MARK_OVERRIDE='model_quality and not (distributed or upstream)' JUNIT_XML=$(JUNIT_XML)
 
 # CI fans the gate out across parallel shard jobs like smoke/attention; the plugin's

docs/contributing/README.md

@@ -86,9 +86,17 @@ pytest --upstream -m "attention"
 
 The `model_quality` marker gates the product models on their output: each is loaded
 **compiled** (the platform default) and compared against a CPU HF reference — greedy token
-ids and per-token probabilities for the decoders (`tests/e2e/test_model_quality.py`),
-cosine similarity for the embedding models and sigmoid scores plus document ranking for
-the cross-encoder rerankers (`tests/e2e/test_encoder_models.py`).
+ids for the decoders (`tests/e2e/test_model_quality.py`), cosine similarity for the embedding
+models and scores for the cross-encoder rerankers (`tests/e2e/test_encoder_models.py`).
+
+The decoder gate follows upstream's hybrid-model test directly: the reference comes live from
+`HfRunner.generate_greedy_logprobs_limit`, Spyre output from
+`VllmRunner.generate_greedy_logprobs`, and `check_logprobs_close` compares them. Embeddings
+likewise use upstream's `check_embeddings_close`. These helpers come from the pinned vLLM
+`tests/` tree via `spyre_testing_plugin.upstream.ensure_upstream_tests_importable` (the same
+clone the upstream suite uses, resolved on first use so a deselected job never pays for it).
+The `hf_runner` / `vllm_runner` / `example_prompts` fixtures in `tests/conftest.py` mirror
+upstream's, so each gate reads like the model test it is modelled on.
 
 ```bash
 make test-model-quality              # the whole gate, one card
@@ -97,56 +105,33 @@ make test-model-quality-shard-0      # one CI shard (QUALITY_SHARDS=6)
 
 CI runs the gate as `QUALITY_SHARDS` parallel 1-card jobs, weighted by recorded runtime like
 the smoke and attention suites. The slowest single case bounds the useful shard count, so
-resize with the `rebalance-test-shards` skill rather than by raising it on a hunch.
-
-The models are too large to run through transformers in CI, so the references are checked
-into `tests/data/` and regenerated by hand where the weights are cached:
-
-```bash
-python tests/data/generate_decoder_output_refs.py --models ibm-granite/granite-4.1-8b
-python tests/data/generate_encoder_embed_refs.py
-python tests/data/generate_rerank_score_refs.py
-```
-
-Regenerate only when the *expected* output changes (a new model or prompt), never to make a
-failing test pass — that is the regression the gate exists to catch. Prompt sets are per
-model: see `MODEL_PROMPTS` and `MODEL_DOCUMENTS` in the generators.
-
-Every gated model is pinned to a revision, in the generator's `MODEL_REVISIONS` and in
-`.github/cache_config/hf_models_and_datasets.yaml`. Each generator records the revision it
-measured in its JSON and the test loads that one back, so bumping a pin means regenerating
-that model's reference.
-
-`SPYRE_TEST_MEAN_ABS_TOL` / `SPYRE_TEST_ABS_TOL` / `SPYRE_TEST_REL_TOL` (decoder
-probabilities) and `SPYRE_TEST_SCORE_ABS_TOL` / `SPYRE_TEST_SCORE_REL_TOL` (reranker scores)
-set the tolerances. For a low-confidence reference the stricter of the absolute and relative
-bound applies, so it is held to a fraction rather than to the same absolute margin. Reranker
-ranking is checked separately from the per-score bound.
-
-The decoder gate is aggregate-first: `SPYRE_TEST_MEAN_ABS_TOL` bounds each prompt's *mean*
-error and is what holds quality, while `SPYRE_TEST_ABS_TOL` only caps a single step against
-gross breakage. A reference near p=0.5 is maximally ill-conditioned (`dp/dlogit` peaks at
-`p(1-p)`), and one compiled graph has measured 0.115 apart on such a step between two CI pods
-with every token still exact — a tight per-step bound buys flakiness, not coverage. Each case
-prints `mean=`/`max=` per prompt, so a failure is readable without a rerun.
-`SPYRE_TEST_TIE_ABS_TOL` holds token disagreements to a tighter bound, since picking a
-different token is a stronger signal than drift. The FP8 decoder checkpoints are
-load-and-decode cases with no reference of their own — their unquantized siblings gate the
-numerics.
-
-A greedy path that diverges from HF on a near-tie cannot be compared past the split, so how
-much of the reference a case compares depends on the prompts. That is **reported, not
-asserted**: the tolerances above are the gate, and coverage is a separate signal, because
-truncation is all-or-nothing per prompt and a floor cannot tell an unlucky prompt from a
-regression. Each decoder case prints `compared <n>/<total> reference steps`, records it as a
-`refcoverage__<n>/<total>` JUnit tag, and warns (`LowReferenceCoverage`) below
-`COVERAGE_WARN_FRACTION`. A warning means the case gates less than it looks like it does and
-its prompts want replacing — not that the model regressed. The one coverage failure is zero:
-a case where every prompt diverged on its first step asserted nothing at all.
+resize with the `rebalance-test-shards` skill.
+
+No reference data is checked in: every gate computes its HF side in-run, so there is nothing
+to regenerate and nothing to go stale. Each gated model is still pinned to a revision, in the
+test's `MODEL_REVISIONS` and in `.github/cache_config/hf_models_and_datasets.yaml`.
+
+The decoders carry no tolerance of their own: `check_logprobs_close` compares greedy token
+ids and, where they disagree, requires each side's token to be in the other's top-N.
+Embeddings use upstream's `check_embeddings_close(tol=1e-2)`. The rerankers keep local bounds
+(`SPYRE_TEST_SCORE_ABS_TOL` / `SPYRE_TEST_SCORE_REL_TOL`, the stricter of the two applying),
+since upstream has no reranker helper and its cross-encoder test's inlined tolerances only
+hold with both sides at the same precision; their ranking is checked apart from the per-score
+bound. The FP8 decoder checkpoints are load-and-decode cases with no comparison at all —
+transformers does not dequantize compressed-tensors on CPU, so their unquantized siblings
+gate the numerics.
 
 #### Upstream Test Integration
 
-Upstream tests are cloned from the vLLM repository at the commit pinned in `pyproject.toml`, fetching only the `tests/` directory. The clone happens on demand, the first time a run asks for upstream tests (see the marker gate above). Cloned tests are cached in `~/.cache/vllm-upstream-tests` (or `$XDG_CACHE_HOME/vllm-upstream-tests`) with separate worktrees per commit, allowing multiple vLLM versions to be tested simultaneously. All upstream tests run with `VLLM_PLUGINS=spyre_inference,spyre_inference_ops` set automatically. Pointing the plugin at a vLLM checkout instead of the cache is the one case that still needs the flag by hand: `pytest -p spyre_testing_plugin.pytest_plugin -m upstream` from the checkout root. See `tests/plugin/spyre_testing_plugin/pytest_plugin.py` for implementation details.
+Upstream tests are cloned from the vLLM repository at the commit pinned in `pyproject.toml`, fetching only the `tests/` directory. The clone happens on demand, the first time a run asks for upstream tests (see the marker gate above). Cloned tests are cached in `~/.cache/vllm-upstream-tests` (or `$XDG_CACHE_HOME/vllm-upstream-tests`) with separate worktrees per commit, allowing multiple vLLM versions to be tested simultaneously. All upstream tests run with `VLLM_PLUGINS=spyre_inference,spyre_inference_ops` set automatically. Pointing the plugin at a vLLM checkout instead of the cache is the one case that still needs the flag by hand: `pytest -p spyre_testing_plugin.pytest_plugin -m upstream` from the checkout root. See `tests/plugin/spyre_testing_plugin/upstream.py` (the clone) and `pytest_plugin.py` (collection and filtering) for implementation details.
+
+Local tests can import upstream test helpers from that same clone via
+`spyre_testing_plugin.upstream.ensure_upstream_tests_importable` (see the model-quality gate
+above). One constraint comes with it: upstream's `tests/` is a real package and this repo's is
+an `__init__.py`-less directory, and a real package wins that name whatever the `sys.path`
+order — so **`tests.*` always means upstream's tree**, and may only be used to reach an
+upstream helper. Sibling test modules here import each other as top-level modules
+(`from test_spyre_attn import ...`), which is what pytest already registers them as.
 
 !!! tip
     To force a re-clone, remove `~/.cache/vllm-upstream-tests`.

pyproject.toml

@@ -282,7 +282,7 @@ markers = [
     "encoder_attention: encoder-attention subset (tests/attention/test_spyre_encoder_attn.py); also marked attention, split into its own CI job",
     "probe: torch-spyre backend probes (tests/probes/); run in their own CI job, excluded from integration",
     "uses_subprocess: tests that spawn a subprocess engine; the plugin runs them first, before the main process initializes the Spyre device",
-    "model_quality: product-model output gates vs cached HF references (tests/e2e/test_model_quality.py and the compiled encoder cases in test_encoder_models.py); compiles models from 125M encoders up to 31B decoders, runs in its own CI job",
+    "model_quality: product-model output gates vs live CPU HF, compared with upstream vLLM's check_logprobs_close / check_embeddings_close (tests/e2e/test_model_quality.py and the compiled encoder cases in test_encoder_models.py); compiles models from 125M encoders up to 31B decoders, runs in its own CI job",
     "rotary: Spyre RoPE tests (tests/custom_ops/test_rotary_embedding.py, tests/multimodal/test_pixtral.py)",
     "fp8: FP8 quantization tests (tests/custom_ops/test_fp8.py)",
     "conv: Spyre Conv2d custom-op tests (tests/custom_ops/test_conv.py)",

tests/attention/test_spyre_attn_recorder.py

@@ -164,7 +164,7 @@ def test_real_metadata_dispatch_compiles_nothing(self, impl, kv_cache):
         metadata for unbucketed kv_lens through ``SpyreAttentionMetadataBuilder``
         and dispatches on the block counts ``build()`` actually produced.
         """
-        from tests.attention.test_spyre_attn import _padded_mask_metadata
+        from test_spyre_attn import _padded_mask_metadata
 
         # Built from the live config, not make_bucketer's narrower stand-in, so
         # this bucketer and the builder's derive from the same config.
@@ -192,7 +192,7 @@ def test_real_metadata_dispatch_compiles_nothing(self, impl, kv_cache):
 
     def test_mixed_batch_dispatch_compiles_nothing(self, impl, kv_cache):
         """A mixed batch dispatches two query widths; both must be recorded."""
-        from tests.attention.test_spyre_attn import _padded_mask_metadata
+        from test_spyre_attn import _padded_mask_metadata
 
         bucketer = SpyreAttnBucketer(get_current_vllm_config())
         impl.record_graphs(torch.device("cpu"), bucketer, kv_cache)
@@ -227,7 +227,7 @@ def test_wide_chunk_beside_short_decode_stays_on_recorded_keys(
         three query buckets, and only bites when a chunk is wider than another
         sequence's padded KV, so the other recorder tests never reach it.
         """
-        from tests.attention.test_spyre_attn import _padded_mask_metadata
+        from test_spyre_attn import _padded_mask_metadata
 
         cfg = get_current_vllm_config()
         monkeypatch.setattr(cfg.scheduler_config, "max_num_batched_tokens", 2048)
@@ -265,7 +265,7 @@ def test_wide_chunk_beside_short_decode_stays_on_recorded_keys(
 
     def test_mixed_batch_row_tables_keep_their_own_width(self, impl, kv_cache):
         """The recorded key is not enough: the row table's width is a guard too."""
-        from tests.attention.test_spyre_attn import _padded_mask_metadata
+        from test_spyre_attn import _padded_mask_metadata
 
         metadata = _padded_mask_metadata(
             [(32, 300), (1, 200), (1, 65)],
@@ -292,7 +292,7 @@ def test_mixed_batch_real_row_tables_compile_nothing(self, impl, kv_cache):
         The other mixed-batch tests reach the kernel through ``_record_one``, which
         rebuilds the row table itself and so cannot see a dispatcher/recorder drift.
         """
-        from tests.attention.test_spyre_attn import _padded_mask_metadata
+        from test_spyre_attn import _padded_mask_metadata
 
         bucketer = SpyreAttnBucketer(get_current_vllm_config())
         impl.record_graphs(torch.device("cpu"), bucketer, kv_cache)

tests/conftest.py

@@ -12,6 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import math
+
 import pytest
 from spyre_testing_plugin.tags import result_tags
 
@@ -35,3 +37,80 @@ def _emit_result_tags(request, record_property):
     params = getattr(getattr(request.node, "callspec", None), "params", {})
     for name, value in result_tags(params):
         record_property(name, value)
+
+
+@pytest.fixture(scope="session")
+def hf_runner():
+    """Upstream's ``HfRunner``, resolved lazily so the tree is only cloned when a test
+    that needs it is collected."""
+    from spyre_testing_plugin.upstream import ensure_upstream_tests_importable
+
+    ensure_upstream_tests_importable()
+    from tests.conftest import HfRunner
+
+    class CpuHfRunner(HfRunner):
+        def get_default_device(self):
+            return "cpu"
+
+    return CpuHfRunner
+
+
+@pytest.fixture
+def example_prompts() -> list[str]:
+    """Upstream's ``example_prompts``, read with ``readlines()`` as its ``_read_prompts``
+    does -- trailing newlines included, since those are the strings its model tests send."""
+    from spyre_testing_plugin.upstream import ensure_upstream_tests_importable
+
+    tests_dir = ensure_upstream_tests_importable()
+    with open(tests_dir / "prompts" / "example.txt") as f:
+        return f.readlines()
+
+
+@pytest.fixture
+def hf_embeddings(hf_runner):
+    """Live CPU HF embeddings for `(model, revision, prompts)`.
+
+    `is_sentence_transformer=True` applies the checkpoint's own pooling and normalization,
+    matching what the vLLM side runs when no `pooler_config` overrides it. Prompts must
+    arrive stripped: sentence-transformers strips its inputs, so otherwise the two sides
+    tokenize different text.
+    """
+
+    def _embed(model: str, revision: str, prompts: list[str]) -> list[list[float]]:
+        with hf_runner(model, revision=revision, is_sentence_transformer=True) as hf_model:
+            return hf_model.encode(prompts)
+
+    return _embed
+
+
+@pytest.fixture
+def assert_embeddings_close():
+    """Upstream's `check_embeddings_close`, plus a finiteness check it omits."""
+    from spyre_testing_plugin.upstream import ensure_upstream_tests_importable
+
+    ensure_upstream_tests_importable()
+    from tests.models.utils import check_embeddings_close
+
+    def _assert(label: str, embeddings, refs, tol: float = 1e-2) -> None:
+        for embedding in embeddings:
+            assert all(math.isfinite(x) for x in embedding), f"{label}: non-finite embedding value"
+        check_embeddings_close(
+            embeddings_0_lst=refs,
+            embeddings_1_lst=embeddings,
+            name_0=f"hf ({label})",
+            name_1=f"spyre ({label})",
+            tol=tol,
+        )
+
+    return _assert
+
+
+@pytest.fixture(scope="session")
+def vllm_runner():
+    """Upstream's ``VllmRunner``, from the pinned vLLM ``tests/`` tree."""
+    from spyre_testing_plugin.upstream import ensure_upstream_tests_importable
+
+    ensure_upstream_tests_importable()
+    from tests.conftest import VllmRunner
+
+    return VllmRunner