Test Matrix Consolidation

[ryan-s-roberts]

Test Matrix Consolidation Strategem

Doctrine

Adopt the provenance pattern as the sacred template: closed production or test-support axes, deterministic synthetic seeds, invariant assertions first, snapshots only after normalization. The key precedent is [crates/baml-rt-provenance/src/graph_model.rs](crates/baml-rt-provenance/src/graph_model.rs) with ALL_EVENT_KINDS, plus consistency tests in [crates/baml-rt-provenance/tests/metamodel_consistency_test.rs](crates/baml-rt-provenance/tests/metamodel_consistency_test.rs) that iterate the full axis rather than preserving many bespoke examples.

flowchart TD
  closedAxes[Closed Axes] --> sharedSeeds[Shared Synthetic Seeds]
  sharedSeeds --> matrixCases[Matrix Cases]
  matrixCases --> invariantAssertions[Invariant Assertions]
  invariantAssertions --> normalizedSnapshots[Normalized Snapshots]
  normalizedSnapshots --> retiredExamples[Retired Ad Hoc Examples]

Loading

Phase 1: Establish Shared Test Substrate

Create reusable fixture and matrix modules in [crates/test-support](crates/test-support) rather than new one-off helpers inside tests. Reuse existing helpers from [crates/test-support/src/common.rs](crates/test-support/src/common.rs) and add domain-specific loaders only where the matrix data is reusable.

• Add deterministic fixture profiles for provenance-backed observability surfaces: minimal context, rich LLM/tool/session context, failure-evidence context.
• Add case structs for repeated protocol and codegen dimensions: schema feature, stream phase, tool/session op, projection surface.
• Keep synthetic data under tests/fixtures/<domain>/ when it is pure input/expected data; keep full agent fixtures under [tests/fixtures/agents](tests/fixtures/agents) only when package build, TypeScript generation, or __chat_register wiring is the behavior under test.

Phase 2: Convert P0 Suites

Begin with the highest weight, highest drift suites.

• Refactor [crates/tools/system/tests/discovery_sessions_test.rs](crates/tools/system/tests/discovery_sessions_test.rs) from many hand-shaped examples into a matrix over tool, filter, request scope, output mode, and graph seed profile. Preserve current snapshots by first routing existing scenarios through the matrix harness, then retire duplicate setup and duplicated redaction logic.
• Refactor [crates/baml-rt-builder/tests/codegen_test.rs](crates/baml-rt-builder/tests/codegen_test.rs) into a schema-feature matrix over nullable, enum, class, nested, array, oneOf, primitive, escaping, and rejection cases. Retain one golden real-tool snapshot for integration confidence; make synthetic schemas authoritative for mapping rules.
• Collapse redundant checks in [crates/baml-rt-builder/tests/baml_to_ts_test.rs](crates/baml-rt-builder/tests/baml_to_ts_test.rs) into one full snapshot plus a compact fragment matrix.

Phase 3: Align A2A and QuickJS Coverage

Consolidate semantic twins without removing the single authoritative vertical slices.

• Extract stream scenario cases from [crates/baml-rt-a2a/tests/task_streaming_test.rs](crates/baml-rt-a2a/tests/task_streaming_test.rs) and setup variants from [crates/baml-rt-a2a/tests/http_a2a_stream_quickjs_test.rs](crates/baml-rt-a2a/tests/http_a2a_stream_quickjs_test.rs) into shared A2aStreamScenario and StreamTerminatorMatrix support.
• Preserve one compiled-fixture row for package-level confidence; use synthetic agents for protocol-only matrix rows.
• Share a conversation-history seed between [crates/baml-rt-a2a/tests/conversation_history_snapshot.rs](crates/baml-rt-a2a/tests/conversation_history_snapshot.rs) and [crates/baml-rt-quickjs/tests/step_executor_intra_turn_conversation_history_snapshot.rs](crates/baml-rt-quickjs/tests/step_executor_intra_turn_conversation_history_snapshot.rs) so both assert the same projection contract from different entrypoints.

Phase 4: Retire Redundant Fixtures and Guard Drift

Only after matrix parity is proven, prune narrow duplicate examples from [tests/fixtures/agents](tests/fixtures/agents). Update [docs/testing-handbook.md](docs/testing-handbook.md) with an authority map: one vertical slice per behavior, matrix backstops for every closed axis, and explicit snapshot normalization rules.

Verification rites per phase:

• Run focused crate tests after each conversion.
• Run just regen-fixtures only if agent fixtures or generated BAML/TypeScript surfaces change.
• Use cargo insta review only after confirming snapshot changes represent intentional semantic consolidation.

Invariants to Preserve

• Coverage must not decrease: every retired ad hoc example maps to at least one named matrix cell or one retained vertical slice.
• Matrix axes must be closed where possible: adding a new event kind, schema feature, stream phase, or projection kind must require updating a canonical axis.
• Snapshots must be normalized before assertion: volatile IDs, timestamps, generated refs, and runtime-specific fields are stripped or sorted centrally.
• Synthetic fixtures test rules; compiled fixtures test integration wiring. These two roles must remain distinct.

Risk Analysis

Complexity is moderate-high because this touches test architecture across multiple crates, but the execution can be staged without production behavior changes. The greatest machine-spirit corruption risk is false confidence: deleting examples before matrix parity is established. The second risk is snapshot churn masking semantic changes; central normalization and invariant assertions must precede mass snapshot updates. Catastrophic system-wide failure potential is low for runtime code, but high for CI signal integrity if fixture regeneration, feature gates, or snapshot naming are mishandled.