Skip to content

fix(e2e): reset per test in excalidraw.spec - #7

Merged
christianhuening merged 2 commits into
mainfrom
fix/excalidraw-spec-reset
Aug 18, 2026
Merged

christianhuening merged 2 commits into
mainfrom
fix/excalidraw-spec-reset

Conversation

@christianhuening

Copy link
Copy Markdown
Contributor

The two "flaky" excalidraw specs were deterministic failures, not flakes.

Cause

All three tests bootstrap through /setup, but the file used test.beforeAll(reset) — one TRUNCATE for the whole file. POST /auth/setup returns 410 (auth.setup_closed) once any user exists (setup.rs:79), and SetupPage renders that error and returns without navigating (SetupPage.tsx:22-31).

So tests 2 and 3 sat on /setup until the timeout:

Test timeout of 30000ms exceeded.
Error: locator.click: Test ended.
Call log: - waiting for getByTestId('new-doc')
  > 122 |   await alice.getByTestId("new-doc").click();

They passed on retry only because a retry restarts the worker, which re-runs beforeAll and truncates again. That explains the signature shape exactly:

  • 30s on first attempt, 2–7s on retry
  • always exactly 2 flaky — the number of tests after the first
  • present in every run going back to the last green ones on main in June

Fix

beforeEach(reset). Each test is self-contained (own /setup, distinct email), so per-test reset is safe.

The repo already knew about this hazard: change-password and invite-password call reset() inline in their second test, and create-placement, new-doc-edit-mode and tree-reorder use beforeEach. excalidraw was the only multi-test file calling /setup in every test without re-resetting.

Verification

Docker Desktop isn't running locally, and the convergence test needs collab (which needs toxiproxy, broken on Docker Desktop macOS), so CI is the verifier here — and it reproduces this deterministically. A run reporting 0 flaky instead of 2 flaky is conclusive.

🤖 Generated with Claude Code

The two "flaky" excalidraw specs were deterministic failures.

All three tests bootstrap through /setup, but the file used
test.beforeAll(reset) -- a single TRUNCATE for the whole file. POST
/auth/setup returns 410 (auth.setup_closed) once any user exists, and
SetupPage renders that error WITHOUT navigating, so tests 2 and 3 sat on
/setup until the 30s timeout:

  Test timeout of 30000ms exceeded.
  Error: locator.click: Test ended.
  Call log: - waiting for getByTestId('new-doc')

They passed on retry only because a retry restarts the worker, which
re-runs beforeAll and truncates again. Hence the signature shape: 30s on
the first attempt, 2-7s on retry, and always exactly 2 flaky -- the number
of tests after the first.

The repo already knew about this: change-password and invite-password call
reset() inline in their second test, and create-placement, new-doc-edit-mode
and tree-reorder all use beforeEach. excalidraw was the only multi-test file
that calls /setup in every test without re-resetting.

Each of the three tests is self-contained (own /setup, distinct email), so
resetting per test is safe.
@christianhuening
christianhuening merged commit e9fd10b into main Aug 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant