Skip to content

test: remove non-enforcing snapshot drift CI step (Closes #415) - #468

Open
Nife-tanny wants to merge 1 commit into
TevaLabs:mainfrom
Nife-tanny:test/415-snapshot-drift-ci
Open

test: remove non-enforcing snapshot drift CI step (Closes #415)#468
Nife-tanny wants to merge 1 commit into
TevaLabs:mainfrom
Nife-tanny:test/415-snapshot-drift-ci

Conversation

@Nife-tanny

Copy link
Copy Markdown

Overview

Path B — Remove it.

Resolves GitHub issue #415 by removing the non-enforcing "Snapshot drift detection" step from CI rather than half-committing to a snapshot baseline that cannot exist in this repo today.

Why Path B (not Path A)

Investigation of the current state revealed that the snapshot step was pure fiction — it never printed its advice, could never fail, and there is no snapshot-golden mechanism in the codebase to enforce:

  1. No snapshot mechanism exists in the code. scripts/update_snapshots.sh sets SNAPSHOT_UPDATE=1, but no Rust test ever reads that env var. There is no insta, no .snap reader/writer, and no golden-file harness anywhere in contracts/src/. The contracts/test_snapshots/ directory it targets does not exist anywhere in the repo.

  2. The directory is gitignored and thus cannot be committed. contracts/test_snapshots/ is listed in .gitignore. Even if a generator existed, emitted fixtures would never be tracked, so there is no committable baseline — and committing a "verified current" one is impossible because no generator exists to produce it.

  3. The CI step is a total no-op. Its guard (if [ -d contracts/test_snapshots ] && ls .../*.snap) is always false (the directory never exists), so the step always printed only "No snapshot golden files to check." It has never printed the drift advice it was written to print, and never failed.

  4. Nobody owns it. The single commit that introduced it (e73bb77 "Add snapshot policy and update script for snapshot management") added a CI message + docs + a script, but no implementation and no fixtures, and nothing was ever committed afterward (contracts/test_snapshots/ remains empty/absent).

Path A would have required building an entire new snapshot-testing framework from scratch (a real generation harness, deciding what to capture, unwiring the gitignore, and a maintained baseline) — far beyond "enforce this existing step," and the issue explicitly asks for the honest choice rather than defaulting to A because it "sounds more thorough." There is no current baseline to commit, so Path B (remove the misleading step) is the correct and honest resolution.

What was removed

  • .github/workflows/ci.yml — removed the Snapshot drift detection step from the rust-test job (lines 59–71). Standalone step; no other job/step greps its output, and ci-success does not reference it. Job ordering, caching, and downstream steps are unaffected.
  • scripts/update_snapshots.sh — removed. Orphaned tooling that only existed to support the dead step; grep confirmed no other active script references it.
  • CONTRIBUTING.md — removed the "Snapshot Tests" section describing the non-existent golden-file workflow.
  • .github/PULL_REQUEST_TEMPLATE.md — removed the "Snapshot policy" checklist section.
  • README.md — removed stale references to contracts/test_snapshots/ (directory tree entry, "NOT Committed" list, git-status-check note/comment).

.gitignore's contracts/test_snapshots/ entry was left in place (inert but harmless; kept to minimize scope).

Reinstate real snapshot testing later

A future maintainer who wants real drift detection should build an actual snapshot harness — e.g. integrate insta into contracts/src/tests/, capture storage/event output from the existing test-utils scenarios, remove the contracts/test_snapshots/ entry from .gitignore so fixtures can be committed, add a SNAPSHOT_UPDATE-style regeneration path that the tests actually honor, and wire a hard-failing diff check into CI.

Verification

  • contracts YAML (ci.yml) validated as valid YAML (js-yaml).
  • Zero Rust source files and zero Cargo files were modified — verified via git diff --name-only origin/main...HEAD (empty for *.rs/Cargo.*). This change is CI config, docs, the PR template, and the deletion of a dead bash script.

Local verification of standard commands

  • cargo fmt --all -- --check: fails (pre-existing) with an identical 4772-line diff on both origin/main and this branch. This is a pre-existing repository condition unrelated to this change — confirmed by running it in a clean worktree of origin/main.
  • cargo test --workspace --locked: could not complete locally due to a rustc compiler ICE (internal compiler error) while compiling dependency build scripts (proc-macro2, zmij) under the local Windows x86_64 + rustc 1.97.1 toolchain. This is a local environment/toolchain limitation that occurs before any of the repo's own code is compiled; it is unrelated to this change (which modifies no Rust/Cargo files) and would occur identically on the base branch. CI runs on ubuntu-latest and is unaffected.

No new failures were introduced by this change; the pre-existing cargo fmt failure and the local toolchain ICE are flagged above and left untouched.

Closes #415

The 'Snapshot drift detection' CI step was a no-op: it only printed
advice when .snap fixtures existed, but never failed on drift, and it
did not even fire because contracts/test_snapshots/ does not exist and
is gitignored. No snapshot-golden mechanism exists in the Rust tests
(SNAPSHOT_UPDATE is never read by any code), so there is no baseline to
commit and no way to ever produce one that survives a commit.

Remove the dead step and the orphaned tooling/docs that only existed to
support it (scripts/update_snapshots.sh, the CONTRIBUTING/README/PR
template snapshot sections). A maintainer who wants real snapshot
testing can build a proper harness (e.g. insta) later, but as-is this
step misled contributors into thinking drift was being caught.

Closes TevaLabs#415
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Testing: enforce storage/event snapshot drift in CI

1 participant