Skip to content

fix(sync): stop SQLite-backed containers from re-syncing on their own -shm writes - #1580

Merged
mariusvniekerk merged 2 commits into
kenn-io:mainfrom
dqtz5vpvj9-create:fix/db-backed-shm-self-trigger
Sep 3, 2026
Merged

fix(sync): stop SQLite-backed containers from re-syncing on their own -shm writes#1580
mariusvniekerk merged 2 commits into
kenn-io:mainfrom
dqtz5vpvj9-create:fix/db-backed-shm-self-trigger

Conversation

@dqtz5vpvj9-create

@dqtz5vpvj9-create dqtz5vpvj9-create commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Opening a WAL-mode SQLite database as a reader rewrites its -shm index. Several providers treated that rewrite as a source change: they resolved a bare -shm event to the whole container, folded the -shm mtime into the container fingerprint, or both, while each of them opens its own read connection during a scan. Together those made each scan schedule the next. On one archive the ZCode container had not changed for half an hour while its 22 sessions were rewritten every five seconds, and the usage-cache backfill and /api/v1/usage/summary failed continuously with "source archive changed during rollup build" because the archive never held still long enough.

Every SQLite-backed provider that opens its own read connection now ignores bare -shm events and leaves the -shm mtime out of its fingerprint:

  • The db-backed containers (Forge, Piebald, Warp, Goose, ZCode) through the shared dbPathForEvent, plus the ZCode fingerprint.
  • Zed and Shelley through the shared container classifier, the same flag Omnigent already passed.
  • Kiro and Devin through their own event mapping.
  • The shared journal suffix list is now main file plus -wal, so Omnigent and Cursor IDE use it instead of private copies.

Real changes always land in the main file or its -wal sibling, and every path still honors those. Existing tests that used a -shm event as an example database sibling now use -wal.

Two limits remain. The first read-only open after an agent closes its database cleanly re-creates an empty -wal, so a provider whose fingerprint reads the -wal mtime re-parses its container once per app close. That is one bounded pass, not a loop; the OpenCode provider already guards it by checking whether the WAL has frames, and the same check could be extended later. Stored ZCode mtimes from older builds included the -shm mtime, so the first scan after upgrading rewrites the ZCode archive once.

Reviewers should look at dbPathForEvent in internal/parser/db_backed_provider.go, zcodeSessionFileMtime in internal/parser/zcode.go, and sqliteDBJournalSuffixes in internal/parser/zed_provider.go.

Closes #1583

🤖 Generated with Claude Code

https://claude.ai/code/session_01PBqd7h5vuXFZehmrZATaNu

generated by a clanker

@roborev-ci

roborev-ci Bot commented Sep 2, 2026

Copy link
Copy Markdown

roborev: Combined Review (13d78a6)

No issues found.


Reviewers: 2 done | Synthesis: codex | Total: 1m54s

@mariusvniekerk mariusvniekerk self-assigned this Sep 3, 2026
@mariusvniekerk mariusvniekerk changed the title fix(sync): stop db-backed containers from re-syncing on their own -shm writes fix(sync): stop SQLite-backed containers from re-syncing on their own -shm writes Sep 3, 2026
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (64c51bb)

Verdict: Medium-severity planning issue remains; otherwise no significant concerns were found.

  • Mediuminternal/parser/db_backed_provider.go:590-599 and shared SQLite classifier
    Ignored -shm paths can still become fallback work in PlanChangedPathsContext: the provider claims the path but returns no exact sources, causing the planner to rediscover every session and potentially trigger full resyncs for remote/delta imports. Mark -shm paths as proven non-data via ChangedPathRelevance, or distinguish intentional no-ops from unclassified failures.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 7m4s

dqtz5vpvj9-create and others added 2 commits September 3, 2026 10:16
…m writes

Opening a WAL-mode SQLite database as a reader rewrites its -shm index.
The db-backed providers (zcode, goose) resolved a bare -shm event to the
whole container, and zcode also folded the -shm mtime into every member's
fingerprint, so each scan scheduled the next one and rewrote every member
session in between. On one archive the zcode container had not changed
for half an hour while all 22 of its sessions were rewritten every five
seconds, and the usage-cache rollups could never stabilize behind them.

Ignore bare -shm events for db-backed containers, the rule Omnigent and
Cursor IDE already apply through classifySQLiteContainerPath, and drop the
-shm mtime from the zcode fingerprint. Real changes always reach the main
file or its -wal sibling, which both paths still honor.
…iners

The previous commit stopped the db-backed providers from re-syncing on
their own -shm writes, but four other SQLite-backed providers still had
the same loop open. Zed, Shelley, and Kiro folded the -shm mtime into
their container fingerprint and accepted a bare -shm event as a container
change, while each of them opens a read connection during fingerprinting
that rewrites -shm. Devin ignored the -shm mtime but still accepted the
event, so every scan queued a wasted fingerprint pass on the next watcher
cycle.

The shared journal suffix list now covers only the main file and -wal,
which lets Omnigent and Cursor IDE drop their private copies of the same
list. Zed and Shelley reject bare -shm events through the shared
container classifier the way Omnigent already did, and Devin and Kiro
drop the -shm match from their own event mapping. Committed writes
always reach the main file or the WAL, so nothing observable is lost.

Existing tests that used a -shm event as an example database sibling now
use -wal, which is the sibling that carries real changes.
@mariusvniekerk
mariusvniekerk force-pushed the fix/db-backed-shm-self-trigger branch from 64c51bb to 1b3b28c Compare September 3, 2026 14:16
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (1b3b28c)

Verdict: One medium-severity issue remains; otherwise, no security or privilege concerns were identified.

Medium

  • internal/parser/db_backed_provider.go:590-599 and equivalent classifiers in devin_provider.go, kiro_provider.go, Zed, and Shelley: Ignored -shm events return no sources, causing PlanChangedPathsContext to fall back to full provider discovery for providers without ChangedPathRelevance. Remote delta syncs may therefore reprocess every session in the container. Mark recognized -shm paths as ChangedPathNonData, or distinguish intentionally ignored sidecars from uncertain classifications.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 6m27s

@mariusvniekerk
mariusvniekerk merged commit 49e767f into kenn-io:main Sep 3, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

db-backed SQLite containers re-sync every interval on their own -shm writes

2 participants