Weekly Spyre scan: week-over-week comparison analysis - generation script

[BenjSz]

Context

The push-to-clickhouse workflow runs every Saturday and writes one row per evaluated model into ClickHouse (spyre.generative_model_spyre_support and spyre.embedding_model_spyre_support). Each week we want to compare the current snapshot against the previous one — separately for generative and embedding models — to answer: did coverage change, did the verified-on-Spyre count change, and if so why?

This issue tracks the analysis points to cover in that comparison.

---

1. Coverage — model count changes

Point	How to detect	What to look at
Total models covered	COUNT(*) GROUP BY snapshot_date	Did the top-K input list grow/shrink? (new families trending on the Hub, or a top_k change)
Net change by type	Same query, per table	Is growth/loss evenly distributed, or concentrated in one mode?
New models (appeared this week)	model_name IN current EXCEPT previous	Are they genuinely new Hub uploads, or were they previously filtered out (e.g. newly under the 60 B param limit)? Check parameters_number and family.
Dropped models (disappeared)	model_name IN previous EXCEPT current	Did they exceed the param ceiling, become MoE, or get deleted from the Hub?
Curated vs top-K composition shift	curated = true count delta	Did the curated list change (a new hand-maintained entry or a removal)?

---

2. verified_on_spyre count changes

Point	How to detect	What to look at
Absolute count of verified_on_spyre = true	COUNT(*) WHERE verified_on_spyre = true	Primary health signal.
Pass rate	verified / total (exclude pre-filter terminal rows)	Normalises away model count growth — a rising count with a falling rate is a warning sign.
Models that regressed (PASS → FAIL)	verified_on_spyre = true in week N−1, false in week N	Highest-priority item: something broke a previously working model.
Models that recovered (FAIL → PASS)	verified_on_spyre = false in week N−1, true in week N	Confirms a fix landed.
verified_on_cpu without verified_on_spyre	verified_on_cpu = true AND verified_on_spyre = false	Models that load correctly on CPU but fail on Spyre specifically — isolates Spyre-specific regressions from model breakage.

---

3. Failure analysis — failure_category

The full set of categories, grouped by stage:

Pre-filter (deterministic — a change here means the input list changed, not the runtime)

Category	Meaning
not-implemented-adapter	No adapter for the model's config_class
model_too_large	Exceeds 60 B parameter ceiling
moe	Mixture-of-Experts, not supported

Worker / runtime

Category	Meaning
cpu_load_failed	Model could not be loaded on CPU
cpu_generate_failed	CPU load OK, but generation/embed inference failed
quantized_model	Quantized checkpoint not supported
unsupported_checkpoint	Checkpoint format not supported
verification_failed	Loaded and ran, but outputs differ from reference
test_execution_exception	Unexpected exception in test logic

Infrastructure (do not reflect model support — tracked separately)

Category	Meaning
hardware_exception	Spyre accelerator unreachable
worker_crashed	Child process exited with no results
worker_timeout	Child exceeded per-batch time limit

Comparison points to cover:

• Distribution shift: COUNT(*) GROUP BY failure_category week-over-week.
• Pre-filter vs runtime split: a change in deterministic categories means the input list changed; a change in runtime categories is actionable.
• Infrastructure noise: if hardware_exception + worker_crashed + worker_timeout is non-zero, the run's verified_on_spyre count is understated and those models should be re-evaluated before drawing conclusions.

---

4. Error message analysis — error field

Point	How to approach
New error patterns	Cluster error values by substring / regex on the non-null rows of newly failing models.
Recurring errors in stable failures	A model failing week after week with the exact same error is a known open issue — track count rather than investigate again.
Error shared across multiple models	Same error substring across ≥N models is likely a single root cause (e.g. a missing kernel, an unsupported op in a whole architecture family).
family / architecture clustering	Join error with family and architecture — errors within one family point to adapter-level issues; errors spanning families point to Spyre-level or framework-level issues.

---

5. Adapter coverage changes — adapter_name / config_class

Point	What to look at
New adapters active this week	adapter_name values present in week N but not N−1, combined with added_date from the git log
Config classes newly covered	config_class values that went from not-implemented-adapter to having a real adapter_name
Config classes that lost coverage	config_class values that went from a valid adapter to not-implemented-adapter (rare but possible on a refactor)

---

6. Size / family dimension

Point	What to look at
Param size of newly failing models	Were regressions in large models (close to the 60 B ceiling) or small ones? Check parameters_number.
Family-level pass-rate trend	Group verified_on_spyre by family — a whole family dropping is an adapter regression; a scattered drop is more likely infrastructure.
Architecture breakdown	architecture column gives a finer grain than family for generative models.

---

[BenjSz]

Implementation: weekly_report.py

The analysis described above has been implemented as a standalone script:

tests/spyre/weekly_generation/weekly_report.py

What it does

Connects to ClickHouse using the same credential chain as the rest of the weekly pipeline (.env / environment variables), fetches two snapshots, and prints a structured plain-text report covering all six analysis areas.

Usage

# Compare the two most recent snapshots automatically
python tests/spyre/weekly_generation/weekly_report.py --mode generative
python tests/spyre/weekly_generation/weekly_report.py --mode embedding

# Compare specific dates
python tests/spyre/weekly_generation/weekly_report.py \
    --mode generative --prev 2025-06-21 --curr 2025-06-28

--prev and --curr must be provided together or not at all. When omitted, the script auto-selects the two most recent distinct snapshot_date values from the table.

Report structure

Each run prints one self-contained report with six numbered sections:

════════════════════════════════════════════════════════════════════════
WEEKLY SPYRE SCAN — GENERATIVE REPORT
  Previous snapshot : 2025-06-21  (N rows)
  Current snapshot  : 2025-06-28  (N rows)
  Table             : spyre.generative_model_spyre_support

1. COVERAGE — MODEL COUNT
   Total models, curated count, new models, dropped models.
   For dropped models: their last-known failure_category.

2. VERIFIED_ON_SPYRE DELTA
   Absolute count and pass rate (pre-filter rows excluded from denominator).
   Infrastructure noise count with a warning if non-zero.
   Regressions (PASS→FAIL) with failure_category breakdown.
   Recoveries (FAIL→PASS).
   CPU-OK-but-Spyre-FAIL models.

3. FAILURE_CATEGORY DISTRIBUTION
   Full table: prev count | curr count | delta for every category.
   Subtotals for pre-filter (deterministic) and infra-noise categories.

4. ERROR PATTERN ANALYSIS
   Top-N error strings for newly failing models.
   Family/architecture breakdown of regressions.
   Top-N recurring error strings across all current failures.

5. ADAPTER / CONFIG_CLASS COVERAGE
   Distinct adapter count and covered config_class count deltas.
   New / lost adapters.
   config_classes newly covered or newly uncovered.

6. SIZE / FAMILY BREAKDOWN
   Per-family pass rate table (prev vs curr, delta in percentage points).
   Families with ≥5pp change called out explicitly.
   Parameter-size distribution (min/median/max) of regressed models.
════════════════════════════════════════════════════════════════════════

Dependencies

No new dependencies. Uses only clickhouse_connect and python-dotenv, already required by the rest of the weekly pipeline.