Skip to content

fix(trace): repair runs interrupted by service-worker eviction - #2889

Merged
esokullu merged 4 commits into
webbrain-one:mainfrom
alectimison-maker:feat/trace-repair-sw-eviction
Aug 22, 2026
Merged

fix(trace): repair runs interrupted by service-worker eviction#2889
esokullu merged 4 commits into
webbrain-one:mainfrom
alectimison-maker:feat/trace-repair-sw-eviction

Conversation

@alectimison-maker

@alectimison-maker alectimison-maker commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Repair stale trace runs left in running after a service-worker eviction.
  • Append ordered, content-free structured interruption evidence and close any open step/turn lifecycle events.
  • Apply the same behavior to Chrome and Firefox, from background startup and when the Traces page opens.

Motivation

The recorder can persist a running run before the service worker is evicted, while the in-memory endRun() call is lost. The run then remains permanently running, even though its event sequence can still be recovered.

Closes #2888

Design

  • Scan durable runs with a conservative 10-minute stale threshold; recent runs remain untouched.
  • Re-read each candidate and apply the repair plan in one runs + events IndexedDB transaction.
  • Continue from the maximum durable event sequence, append step_end/error/turn_end evidence as applicable, and mark the run with repairedBy and repairReason.
  • Reuse the existing event model and add the stable SERVICE_WORKER_EVICTED error code. No new event kind or trace schema migration is introduced.
  • Keep Chrome and Firefox implementations mirrored.

Testing

  • node test/run.js — 1987 passed, 0 failed
  • npm run test:toolbar-guard — 33 passed
  • npm run test:security — 60/60 passed
  • Node syntax checks for all touched JavaScript files — passed
  • node scripts/benchmark-offline-relevance.mjs — completed successfully
  • Manual Chrome unpacked-extension smoke test: extension loaded, Traces opened, Refresh clicked, no console/page errors

Compatibility and risks

  • Additive run metadata and error code only; existing completed traces are not modified.
  • Default tracing behavior and privacy filtering are unchanged; repair events contain no user content.
  • The 10-minute threshold intentionally protects recent runs, using the latest durable event timestamp as activity evidence. A run that stays completely quiet for longer than the threshold remains the intended repair candidate; the same service-worker instance is additionally protected by its in-memory active-run state.
  • Firefox was covered by mirrored source, parity tests, syntax checks, and the shared unit suite; manual Firefox browser verification was not run.

@vercel

vercel Bot commented Aug 22, 2026

Copy link
Copy Markdown

@alectimison-maker is attempting to deploy a commit to the esokullu's projects Team on Vercel.

A member of the Team first needs to authorize it.

alectimison-maker and others added 2 commits August 22, 2026 09:06
- Candidate scan walks only the startedAt slice that can be stale
  instead of every run on record at each wake or Traces load.
- The startup scan is deferred so a run resuming right after eviction
  registers its in-memory state first, and the repair transaction
  re-checks liveness immediately before writing.
- The Traces page asks the background to own the scan via
  WB_TRACE_REPAIR_STALE_RUNS and repairs locally only when the
  background cannot be reached, so a live-but-silent run is never
  misread as abandoned from another page.
@esokullu

Copy link
Copy Markdown
Collaborator

I reviewed the branch and pushed 5480bba with fixes for three issues found during review. Full suite passes (1987 tests).

Unbounded scan on every wake (perf). repairStaleRuns walked every run ever recorded (listRuns({ limit: Number.MAX_SAFE_INTEGER })) on each service-worker startup and on each Traces page load. A run's last activity is never older than its startedAt, so only records with startedAt <= now - staleAfterMs can qualify; the scan now walks just that slice of the startedAt index.

Startup race against resuming runs. The scan fired at module top-level, so when the worker woke specifically to resume an interrupted run, repair could race its first event write and mark it abandoned mid-flight. Two mitigations: the background defers its startup scan by 15s, and _repairRunInTransaction re-checks _runState.has(runId) right before writing, so a run that has re-registered itself in memory is never touched.

Traces page could close out live-but-silent runs. The page's recorder instance has an empty _runState, so a run alive in the background but quiet for more than the stale threshold (a hung LLM request, a long tool call) could be terminated with spurious repair events from the page while still running. The page now asks the background to own the scan via a new WB_TRACE_REPAIR_STALE_RUNS listener (both builds) and falls back to a local scan only when the background cannot be reached, since then nothing can be running anywhere. The fallback keeps the original behavior available for pages opened before the background first wakes.

Test assertions were extended to pin the bounded candidate scan, the deferred startup call, the message route, and the local fallback.

@esokullu
esokullu merged commit 7af61e6 into webbrain-one:main Aug 22, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(trace): synthesize closing events for runs interrupted by service-worker eviction

2 participants