perf(codex): persist parse checkpoints to eliminate OOM-leading memory leakage and long startup delays

cmd/agentsview/sync_worker.go

@@ -196,11 +196,17 @@ func runSyncWorkerStartup(
 		var stats sync.SyncStats
 		var tombstoned int
 		var auditErr error
+		// The audit is also the periodic content-verification pass for
+		// checkpointed sources: bypass the stat-trust gate so the provider's
+		// full-source fingerprint detects and repairs same-stat in-place
+		// rewrites that append-trust would otherwise keep stale.
+		engine.SetCheckpointAudit(true)
 		if auditRoots := reconcileRootPaths(cfg); len(auditRoots) > 0 {
 			stats, tombstoned, auditErr = engine.ReconcileWatchRootsWithStats(
 				ctx, auditRoots, false, onProgress,
 			)
 		}
+		engine.SetCheckpointAudit(false)
 		result = workerResultFromStats(ctx, stats)
 		result.Tombstoned = tombstoned
 		if auditErr != nil && result.Status == "ok" {

docs/internal/background-sync-efficiency.md

@@ -63,32 +63,79 @@ they are never published as safe cursor boundaries.
 
 Truncation, known file-identity replacement, manual or project refreshes,
 `session_index.jsonl` title changes, and records that retroactively update
-stored messages all fall back to an authoritative full replacement. Safe
-incremental writes preserve the index-folded mtime and lifecycle-derived
-termination status alongside message and token aggregates.
+stored messages all fall back to an authoritative full replacement. Late tool
+results are the exception: the cursor tracks pending tool calls (bounded), so
+a `function_call_output` / `custom_tool_call_output` that refers to a call
+committed in an earlier batch is applied as an idempotent point update
+(`ToolCallResultUpdates`) instead of a full reparse. Agent-scoped and unknown
+calls still fall back. Safe incremental writes preserve the index-folded mtime
+and lifecycle-derived termination status alongside message and token
+aggregates.
 
 ## Append-only limitation
 
 Cursor correctness assumes that growth is append-only. A same-inode file can
 grow after bytes inside its already-committed prefix have been rewritten. Size,
-identity, and boundary checks do not detect that case, and the current
-full-source fingerprint is not compared with a separately verified stored prefix
-before incremental parsing. Closing this gap would require rolling hash state or
-explicit prefix verification and remains deferred.
+identity, and boundary checks cannot prove the prefix was never modified.
+
+Persisted checkpoints close most of the gap under the documented append-trust
+mode:
+
+- The checkpoint stores a 128 KiB tail anchor of the committed prefix, the
+  file identity, the committed offset, the parser cursor, and a resumable
+  SHA-256 state over the committed prefix.
+- An append is only resumed when the identity matches, the size only grew,
+  and the current bytes at the anchor region match the stored anchor; the
+  full-file fingerprint is then derived by hashing only the appended bytes.
+- An unchanged checkpointed source is skipped on stat alone (no transcript
+  read). An anchor mismatch, identity change, truncation, or undecodable
+  checkpoint forces an authoritative full parse and checkpoint rebuild.
+- A same-size, same-mtime in-place rewrite that preserves the anchor region is
+  trusted (append-trust). Periodic full audits (`ResyncAll`, `--full`,
+  force-reverification passes) still hash the whole source and repair such
+  rewrites. Strict verification remains available by bypassing the checkpoint
+  gate.
 
 ## Cost model and regression evidence
 
 A warm Codex cursor makes continuation-state parsing scale with appended records
-rather than transcript history. End-to-end append sync is still O(file): the
-provider's `Fingerprint` hashes the complete source and the engine's
-`ComputeFileHashPrefix` hashes through the newly committed offset.
+rather than transcript history. With a persisted checkpoint, end-to-end append
+sync is O(d): the engine resumes the SHA-256 state over only the appended
+bytes, verifies the 128 KiB tail anchor by digest, and parses only the new
+tail. The append path reads the source roughly three times the delta
+(fingerprint resume, parser tail, final checkpoint resume) plus the 128 KiB
+anchor — for a 1 KiB append that is well inside the 256 KiB source-read gate.
+An unchanged checkpointed source costs a stat plus the small checkpoint
+metadata row read (the cursor and hash-state blobs live in a separate table
+and are never loaded on the stat-only path) and reads 0 transcript bytes.
+A full parse captures the resumable hash state and anchor digest on its own
+read pass, so persisting the checkpoint adds no second source read. Without
+a checkpoint (legacy sessions, first sync after upgrade) the previous
+O(file) fingerprint and prefix-hash reads still apply until the next full
+parse persists one.
+
+The daily archive audit (`sync_worker` audit mode) bypasses the checkpoint
+stat-trust gate so the provider's full-source fingerprint verifies content and
+repairs same-stat in-place rewrites that append-trust would otherwise keep
+stale.
 
 - `BenchmarkCodexIncrementalCursor` in `internal/parser` compares cold prefix
   reconstruction with the exact warm cursor. It is diagnostic because
   `internal/parser` is not in `BENCH_GATE_PACKAGES`.
-- `BenchmarkCodexIncrementalSyncReads` in `internal/sync` measures the warm tail
-  between the two remaining linear reads. It is PR-gated because
-  `internal/sync` is in `BENCH_GATE_PACKAGES`.
+- `BenchmarkCodexCheckpointAppendResume` in `internal/sync` measures the
+  checkpoint-resumed append's source pipeline (checkpoint gate, seeded tail
+  parse, resume hash, next anchor digest, checkpoint assembly), bounded by
+  the anchor window plus the tail. It is PR-gated because `internal/sync` is
+  in `BENCH_GATE_PACKAGES`.
+- `BenchmarkCodexQuietAppendSignals500/5000/15000` in `internal/sync` measure
+  the non-amortized quiet-session append (call + late output) with full
+  inline signal/secret maintenance; the three sizes gate the
+  history-independent latency slope.
+- `BenchmarkCodexLateToolOutputDebouncedBurst` in `internal/sync` measures a
+  debounced stream where every appended batch carries the output for the
+  previous batch's call: only the first iteration pays the O(history)
+  signal recompute, so it guards per-append cost without pretending to be
+  the quiet-session gate.
 
 The maintained behavioral gate inventory is in
 [Performance Gates](performance-gates.md).

docs/internal/codex-incremental-streaming-design.md

@@ -0,0 +1,84 @@
+# Codex incremental checkpoints and streamed full imports
+
+## Problem
+
+A busy Codex rollout grows to hundreds of megabytes across thousands of
+tool-result events. Three operations that should be cheap are not:
+
+- an unchanged file costs a full transcript read on every sweep;
+- a small appended tail re-reads and re-sums the whole transcript;
+- a cold import holds the full message slice and every tool-result body
+  in memory at once (a 945 MB archive peaked above 1 GB).
+
+## What this branch does
+
+1. **Persistent safe-resume checkpoints.** A full parse captures a
+   resumable SHA-256 state, the trailing 128 KiB anchor digest, and the
+   file identity in one pass. The checkpoint row commits in the same
+   transaction as the session content. An unchanged file — inode, device,
+   size, mtime, and change-time all matching — is skipped without reading
+   the transcript. Appends resume from the stored offset and are proven
+   against the tail anchor before the rows commit.
+
+2. **Cross-sync tool results as transactional deltas.** A tool output
+   appended after its call was persisted no longer forces a full re-parse.
+   The incremental tail yields deferred result updates that the writer
+   applies with targeted probes: event deduplication against stored rows,
+   a per-call agent-state table that resolves the latest content per agent
+   by event coordinates (no content copies), and signals/findings folded
+   incrementally. The checkpoint advances atomically with the write.
+
+3. **Streamed cold imports.** Decoding emits through a session sink; a
+   staging sink writes event rows into a scratch SQLite database while the
+   in-memory model keeps only placeholders. The publish transaction
+   attaches the scratch database, copies event rows and per-call summaries
+   into the archive, and commits messages, events, summaries, signals, and
+   findings atomically. Files above 128 MB take this path.
+
+## Correctness boundaries
+
+- The checkpoint is trusted only when the stored hash state matches the
+  committed prefix hash and the file identity (including change-time)
+  matches. Truncation, replacement, and same-size same-mtime rewrites
+  rebuild authoritatively. A periodic audit (`ResyncAll`) remains the
+  backstop for anything the stat gate cannot see.
+- Fork and subagent replays match the parent transcript's turn ids as
+  opaque membership keys; an unresolved explicit parent keeps the child
+  visible but marks its data version for retry.
+- A scratch write failure is sticky: the parse and the publish fail and
+  the archive keeps its prior content. The staging ATTACH is torn down
+  after every transaction, so consecutive publishes share one writer
+  connection safely.
+
+## Costs
+
+- Disk: tool-result content is still stored twice by the existing
+  archive layout (`tool_result_events.content` and
+  `tool_calls.result_content`). This branch does not change that. The
+  agent-state table adds only integer coordinate rows.
+- Runtime: a staged cold import holds the process GC target lower for the
+  duration of the parse and returns parse-phase arenas before the
+  publish, trading CPU for a bounded RSS.
+
+## Suggested PR split
+
+The branch is intentionally one working line of history, but the
+mergeable sequence is:
+
+1. persistent safe-resume checkpoints (0-byte no-op, O(delta) appends);
+2. cross-sync tool-result deltas on top of it;
+3. byte-bounded bulk admission;
+4. the behavior-preserving session-sink parser refactor;
+5. the scratch-staging streamed import with its runtime policies.
+
+## Where to look
+
+- `internal/parser/codex.go`, `codex_cursor.go`, `codex_provider.go` —
+  single-pass hash/anchor, cursor codec, fork replay gate.
+- `internal/sync/checkpoint.go`, `internal/db/checkpoint.go` — checkpoint
+  persistence and the append/no-op decision.
+- `internal/db/messages.go` — transactional late-result updates and the
+  agent-state table.
+- `internal/sync/codex_staging.go`, `internal/db/staged_content.go` —
+  scratch staging sink and the staged publish transaction.
+- `internal/signals/incremental.go` — the typed incremental reducer.

docs/internal/performance-gates.md

@@ -17,6 +17,7 @@ contracts are documented in
 | Discovery O(sources) root work | Gemini rebuilt its project map per session; positron/vscode-copilot re-read `workspace.json` per session. A large store spent 2m47s in discovery.                                                       | #912                                           |
 | Unchanged sources reparsed     | The provider migration dropped pre-parse DB-freshness skips; every full sync reparsed and rewrote untouched sessions.                                                                                   | `providerSourceUnchangedInDB` (#883 follow-up) |
 | O(history) incremental appends | Every streamed line ran a full signal recompute (reload all messages, secret regex scan) and chunk merges delete+reinserted every message row. ~4,700 session updates/day each paid O(session history). | #954                                           |
+| O(file) warm Codex appends     | A warm append still hashed the full source for the fingerprint and re-hashed the committed prefix after every write; late tool outputs forced a full transcript reparse, and unchanged startup hashed every source before the DB freshness skip. | persisted parser checkpoints (this PR)         |
 | Bulk ingest throughput         | Full resync ran per-row inserts and rebuilt FTS incrementally; 26.7k sessions took 1m17s.                                                                                                               | #411                                           |
 | Event storms                   | One SSE emit per watcher flush drove ~1/s dashboard refetch; SQLite WAL sidecar events fanned out to every session in a shared DB.                                                                      | #367, #956                                     |
 | Per-row query shape            | `GetDailyUsage` ran 1.2M `json_extract` calls per scan and had no date pushdown.                                                                                                                        | #309                                           |
@@ -38,6 +39,26 @@ runner noise and fail loudly:
 - `TestWriteIncrementalDebouncesSignalRecompute` and the rest of
   `internal/sync/signal_schedule_test.go` — streaming appends must debounce
   the O(history) signal recompute.
+- `TestCodexCheckpoint*` in `internal/sync/checkpoint_test.go` — a full Codex
+  parse persists a checkpoint, an append resumes from it and advances it
+  atomically with the delta, truncation/anchor mismatch force an
+  authoritative rebuild, and an unchanged checkpointed source is trusted on
+  stat alone (append-trust; the audit path still catches rewrites).
+- `TestCodexCheckpointStaleCannotResumeFromNewerDBOffset`,
+  `TestCodexCheckpointHashStateBoundedToCommittedOffset`,
+  `TestCodexCheckpointColdRestartResumeParity`, and
+  `TestCodexCheckpointAuditRepairsSameStatRewrite`
+  (`internal/sync/checkpoint_review_test.go`) — a surviving checkpoint must
+  agree with the committed DB offset/ordinal/hash, hash state covers exactly
+  the committed prefix, cold restarts resume with parity, and the audit
+  repairs same-stat rewrites.
+- `TestParserCheckpointRoundTrip` and
+  `TestWriteSessionIncrementalPersistsCheckpointInSameTx` (`internal/db`) —
+  checkpoint rows round-trip and are committed in the same transaction as the
+  incremental delta.
+- `TestFullSyncPassIsByteBudgeted`, `TestBulkParseRetentionBudgetUsesWeightedAdmission`,
+  and `TestCollectAndBatchFlushesOnByteCap` (`internal/sync`) — bulk passes
+  and write batches are bounded by estimated bytes, not session count alone.
 - The count-based seam tests in `internal/parser`
   (`discovery_workspace_manifest_test.go`, gemini/antigravity provider tests)
   — root-derived project info is built once per root, not once per source.
@@ -55,10 +76,13 @@ runner noise and fail loudly:
   warm/cold parsing remains equivalent at safe offsets.
 - `TestIncrementalSync_CodexAppend`,
   `TestIncrementalSync_CodexLifecycleTailUpdatesTermination`, the partial-tail
-  tests, and the late-update/title tests in
+  tests, the late tool-result append test
+  (`TestIncrementalSync_CodexExecAppendRetainsEvents`), and the
+  late-update/title tests in
   `internal/sync/engine_integration_test.go` — safe Codex growth appends only
-  new rows while lifecycle metadata, incomplete records, title changes, and
-  retroactive updates preserve full-parse behavior.
+  new rows, late tool results update stored calls in place, and lifecycle
+  metadata, incomplete records, title changes, and other retroactive updates
+  preserve full-parse behavior.
 - `TestCountDuplicatePromptsAllocationGrowthStaysNearLinear`
   (`internal/signals/heuristics_test.go`) — session-quality analysis must not
   rebuild token sets for every pair of user prompts.
@@ -79,10 +103,29 @@ with `cmd/benchgate`:
   skip work only; also self-asserts nothing is re-synced or bulk-rewritten).
 - `BenchmarkSyncPathsIncrementalAppend` — absorb one appended line into a
   1,000-message session.
-- `BenchmarkCodexIncrementalSyncReads` — a warm Codex cursor append plus the
-  remaining full-source fingerprint and committed-prefix hash reads. See
+- `BenchmarkCodexCheckpointAppendResume` — the source-reading pipeline of a
+  checkpoint-resumed Codex append: checkpoint gate (stat + anchor digest +
+  fingerprint resume), seeded tail parse, committed-prefix resume hash, next
+  anchor digest, and next checkpoint assembly. Bounded by the anchor window
+  plus the tail — see
   [Background Sync Efficiency](background-sync-efficiency.md) for the
   cost-model boundary.
+- `BenchmarkCodexLateToolOutputDebouncedBurst` — a checkpoint-resumed Codex
+  append stream with the signal debounce stretched, so only the first
+  iteration pays the O(history) recompute and the rest are amortized. It
+  guards the late-result update path's per-append cost; it is deliberately
+  NOT the quiet-session gate (see the `BenchmarkCodexQuietAppendSignals*`
+  trio below).
+- `BenchmarkCodexQuietAppendSignals500` / `...5000` / `...15000` — the
+  non-amortized quiet-session append gate: every iteration appends a new
+  function_call plus the previous call's late output and pays the full
+  inline signal/secret maintenance (debounce disabled). The three sizes
+  prove per-append latency does not scale with stored history; each
+  self-asserts zero `GetAllMessages` calls in the timed loop.
+- `BenchmarkCodexColdFullSync` — a fresh database and engine ingesting the
+  transcript from scratch, including the single-pass checkpoint capture.
+  Per-op cost deliberately exceeds the micro-benchmark band; it exists to
+  catch a regression that adds a source read pass to the cold pipeline.
 - `BenchmarkSyncAllColdArchive` — first-sync ingest throughput through the
   default per-session write path.
 - `BenchmarkResyncBulkIngest` — the same archive through the resync bulk-write
@@ -204,6 +247,23 @@ prefix reconstruction with an exact warm cursor. It is diagnostic rather than
 PR-gated: `BENCH_GATE_PACKAGES` currently contains `./internal/sync`,
 `./internal/db`, `./internal/secrets`, and `./internal/signals`.
 
+### 3. Macro ratio gate (run manually, build tag `macrobench`)
+
+The 10MB-vs-1GB same-append p95 ratio (`< 2x`) is not a CI benchmark: the
+1GB fixture takes minutes per run. `internal/sync/codex_macro_bench_test.go`
+is excluded from the PR gate by the `macrobench` build tag:
+
+```bash
+cd internal/sync
+go test -tags 'fts5,macrobench' -run '^$' \
+  -bench 'BenchmarkMacroCodexQuietAppend' -benchmem -count=6 -benchtime=5x
+```
+
+The gate: the p95 `sec/op` of `BenchmarkMacroCodexQuietAppend1GB` must stay
+within 2x of `BenchmarkMacroCodexQuietAppend10MB` for the same
+quiet-append shape (call + late output, full inline signal/secret
+maintenance).
+
 ## Adding a benchmark to the gate
 
 Every benchmark in a gated package is gated — there is no per-name allowlist to