Skip to content

fix: harden memory recovery and Windows resource cleanup - #3919

Closed
twoimo wants to merge 6 commits into
Yeachan-Heo:devfrom
twoimo:audit/memory-v0.12.12
Closed

fix: harden memory recovery and Windows resource cleanup#3919
twoimo wants to merge 6 commits into
Yeachan-Heo:devfrom
twoimo:audit/memory-v0.12.12

Conversation

@twoimo

@twoimo twoimo commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • make recovery transcript promotion failure-atomic and identity-bound, with durable cleanup retry
  • release SQLite statements/databases deterministically so Windows can remove DB/WAL directories immediately
  • harden Windows perf-corpus argv provenance, child PID reuse handling, checkpoint fsync classification, and post-ACK retry/rollback state
  • add adversarial Windows and native Linux regressions; update affected docs and changelogs

This replaces #3911, which was incorrectly opened against main. The commits are rebased directly onto the current dev tip as required by CONTRIBUTING.md.

Verification

  • bun --cwd packages/coding-agent run check
  • Windows memory-guard checkpoint suite — 10 passed
  • native Ubuntu 24.04 RecoveryFsRoot suite — 10 passed, 49 expectations
  • managed publication collision replay on native Ubuntu — passed
  • post-ACK retry/rollback regressions — 2 passed, 14 expectations
  • perf corpus — 46 passed
  • RLM analysis — 59 passed
  • focused runtime/resource suite — 51 passed
  • git diff --check

Independent review verdict

gajae.pr-review-verdict.v1 needs-human sha256:39d34703e6609c040a8e2806f701d9829b929b351f1ff01aeb32d8bf2ae01cd1 reviewer:human evidence:local-focused-verification

  • Target branch is dev
  • bun check passes
  • Tested locally
  • CHANGELOG updated
  • Verdict above matches the exact PR diff

@twoimo
twoimo force-pushed the audit/memory-v0.12.12 branch from 2c7ed52 to edb2c44 Compare August 6, 2026 06:26
@twoimo

twoimo commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

@Yeachan-Heo Replacement for closed #3911 is now correctly based on dev, rebased onto the current tip, conflict-free, and Dev CI is green. Ready for review.

@Yeachan-Heo

Copy link
Copy Markdown
Owner

MERGE_READY

Adversarial exact-head review of PR #3919 on current exact head edb2c44f7c65be141d93f8c4be54f2fd3fde4218 (head ref audit/memory-v0.12.12, base dev). No blocking findings.

Exact-head evidence

  • Fetched refs/pull/3919/headedb2c44f7c65be141d93f8c4be54f2fd3fde4218, byte-identical to GitHub REST head.sha. No branches created/switched; no pushes; no merges.
  • Diff reviewed against merge-base of dev (0bd7d770db42238c87fb09f6ad680227b5458969 at review time): 22 files, +1258/−248.
  • CI on exact head: 25 check runs success, 5 skipped, 0 failures — green/CLEAN. The affected-path jobs exercised the changed suites: memory-guard-checkpoint.test.ts, team-runtime.test.ts, memory-guard-owner-claims.test.ts, team-worker-memory-guard.test.ts, perf-corpus.test.ts, perf-corpus-rlm-analysis.test.ts, docs-index-lazy.test.ts, check:@gajae-code/coding-agent, native-build, Windows toolchain validation.

#3911 successor relation

Adversarial findings by boundary

  1. Recovery promotion failure-atomicity / identity binding (session-manager.ts, managed-session-storage.ts): staging → rename-no-replace → published-identity re-verification (dev/ino/size/mtimeNs/sha256) → source retirement is coherent; idempotent when the promotion already matches the intended digest; rollback is identity-bound (quarantine names carry pid+uuid) and only removes files matching the staged identity. Managed path adds moveExpectedNoReplace with post-rename identity recheck (ctime excluded, correctly, because rename changes ctime).
  2. Cleanup retry + SQLite handle/WAL release on Windows: agent-storage.ts and memory-guard-owner-claims.ts finalize every prepared statement in finally; claims DB closes with close(true); recovery source-retirement failure defers to #retryRecoverySourceRetirementCleanup on close (exact identity-bound unlink/removeExpected, durable). Test memory-guard-owner-claims.test.ts now proves the claims directory can be removed immediately after release (DB/WAL not locked).
  3. Perf-corpus argv provenance: Windows argv now comes from GetCommandLineW + CommandLineToArgvW (bounded, LocalFree-freed); the programmatic runPerfCorpusBenchmark export was removed so imported execution cannot bypass the frozen process-argv contract — test asserts "runPerfCorpusBenchmark" in benchmarkModule === false; perf-corpus-rlm-analysis.py drift detection now re-verifies content hashes and excludes ctime on NT.
  4. PID-reuse safety: spawnWithDistinctChildPid retries duplicate child PIDs (max 3) and fails closed, tested directly.
  5. Checkpoint fsync classification: directory sync swallows EPERM on win32 only, rethrows everything else; both branches tested (memory-guard-checkpoint.test.ts durability suite).
  6. Post-ACK rollback/retry (team-runtime.ts): incident id is stable across retries (ledger.last_incident_id reused when retry_count > 0); structured rollback reports successor_terminated/predecessor_state_restored; ambiguous rollback (unproven successor termination) drives state: "blocked" with retry_count = retry_limit and task block; each publication phase is seam-testable. Both new tests cover retry-budget stability and the ambiguous-rollback block.
  7. Docs/generated artifacts: docs-index.generated.ts embedded perf-profiling-corpus.md updated to schema gjc.perf-corpus/3 and the no-programmatic-runner contract, consistent with perf-corpus-schema.ts and the bench module.

Test adequacy

Each new failure path has a direct test: publish-failure rollback, published-byte verification failure, source-retirement failure (promotion completes with verified duplicate, exact cleanup on close), ambiguous-rollback adopt on retry, directory-durability EPERM/EIO, post-ACK retry/block, PID retry cap, and argv-contract enforcement. Coverage matches the changed behavior.

Verdict is based solely on the current exact head. No merge performed; lane is read-only.

[repo owner's gaebal-gajae (clawdbot) 🦞]

@twoimo
twoimo force-pushed the audit/memory-v0.12.12 branch from edb2c44 to dcd32f7 Compare August 6, 2026 11:09

@yazzang-homelab yazzang-homelab left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent architect review. Large surface, so I focused on the atomicity claim and the Windows handle-release claim rather than restating the summary.

One concrete defect: the promotion rollback can mask the error it is rolling back.

session-manager.ts:10713-10733:

} catch (error) {
    const final = store.readExpected(promotionName);
    if (final && /* dev, ino, size, mtimeNs, sha256 */)
        store.removeExpected(promotionName, final);          // <- not guarded
    else {
        const remaining = store.readExpected(stagingName);
        if (remaining && /* dev, ino, sha256 only */)
            store.removeExpected(stagingName, stagedSnapshot); // <- not guarded
    }
    throw error;
}

Neither removeExpected is wrapped, so if either throws, the original promotion error is discarded and a cleanup error surfaces instead. The caller then diagnoses the wrong failure — and this is a recovery path, so the wrong diagnosis lands exactly when someone is already trying to understand why a session would not promote.

You clearly know the pattern, because twenty lines later the source-retirement removal is guarded and downgraded to a scheduled retry:

try { store.removeExpected(sourceName, sourceSnapshot); }
catch (error) { this.#recoverySourceRetirementCleanup = {...}; logger.warn(...); }

The rollback pair should be best-effort the same way.

Related asymmetry that makes the above reachable. The promotion-side check compares five identity fields; the staging-side check compares three (dev, ino, sha256 — omitting size and mtimeNs). Then it calls removeExpected(stagingName, stagedSnapshot) — passing the staged snapshot, which still carries the original mtimeNs. So the weaker pre-check can pass on a file whose mtimeNs moved (same inode, same content, e.g. a touch or a metadata-only update) while the exact-match delete inside removeExpected then rejects it and throws. That is the concrete path to the masking above.

Either widen the staging check to the same five fields, or pass remaining instead of stagedSnapshot so the pre-check and the delete agree on the same observation. size is redundant in the five-field version anyway — equal sha256 already implies equal content and therefore equal size — so the honest minimal set is dev/ino/mtimeNs/sha256 on both sides.

What checks out:

  • Windows handle release is done properly. Every prepare() in agent-storage.ts and managed-session-storage.ts is paired with finalize() in a finally, including the schema-version read and the INSERT OR REPLACE — statements that are easy to forget because they run once at open. That is what actually lets Windows unlink the DB/WAL directory, and a single missed finalize would have silently defeated the whole change.
  • fsyncDirectoryPath swallowing EPERM is correct, not a shortcut. Windows has no directory fsync, and the guard is narrowed to process.platform === "win32" and code === "EPERM" — every other errno on every platform still throws. The finally { await directory.close() } keeps the handle from leaking on the swallowed path.
  • Source-retirement failure is non-wedging by design: a verified successor plus a retained duplicate is recoverable, and #retryRecoverySourceRetirementCleanup gives it an exact retry rather than leaving it to chance.
  • Opening this as a replacement for #3911 (wrong base branch) rather than retargeting is the right call — retargeting a PR across main/dev keeps the original base sha and confuses the exact-head gate.
  • Verdict block correctly says needs-human rather than self-approving. Several open PRs get this wrong; noting it because it should be the norm.

I have not independently exercised the Windows-only suites; I have no Windows host. Those claims rest on your run and on CI.

gajae.pr-review-verdict.v1 merge-blocked sha256:dcd32f7cde40319438da28a3b782d12e444e0287 reviewer:architect evidence:read of session-manager.ts:644-652,10713-10749 and the prepare/finalize pairs in agent-storage.ts + managed-session-storage.ts at this head

@twoimo

twoimo commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

@Yeachan-Heo dev advanced after the prior MERGE_READY and GitHub reported the PR dirty. I rebased onto the new dev tip and reran Dev CI successfully. Please refresh the exact-head review for dcd32f7cde40319438da28a3b782d12e444e0287 (diff SHA-256 beb059d8dc931a76598abe7cfebf6b8963a57a8bc620c1aa7508f26854975165).

@Yeachan-Heo
Yeachan-Heo force-pushed the audit/memory-v0.12.12 branch from dcd32f7 to b2393e8 Compare August 6, 2026 11:29
@twoimo
twoimo force-pushed the audit/memory-v0.12.12 branch from b2393e8 to 6b2b829 Compare August 6, 2026 11:32
@twoimo

twoimo commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the change request on exact head 6b2b829a76a00d95d01d097a069b5a2f12e43078 (diff sha256:2b07be66b6af3645f94cfa739b86b8d502cba739572f5ab40a9ed14d368d7d64).

  • Recovery successor rollback is now best-effort: cleanup failures are logged and the original promotion error is rethrown unchanged.
  • The staging rollback authority check now matches the final-file identity check: dev, ino, size, mtimeNs, and sha256.
  • Added adversarial regressions proving a cleanup failure cannot mask destination_conflict, and metadata-drifted staging is not passed to exact deletion.

Verification after rebasing onto current dev:

  • bun --cwd packages/coding-agent run check passed.
  • Both focused regressions passed on Windows Bun 1.3.14.
  • Both focused regressions passed under WSL Ubuntu Bun 1.3.14.

@yazzang-homelab please re-review the refreshed exact head.

@twoimo

twoimo commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Exact head advanced to a857c24f011a26eb43b73997930f8b6173a86697 (diff sha256:39d34703e6609c040a8e2806f701d9829b929b351f1ff01aeb32d8bf2ae01cd1) after the final independent red-team pass found two adjacent race boundaries:

  • authority-absent managed moves now use native identity-bound exactRestore instead of path-only rename; a deterministic Windows source-substitution regression proves the wrong file is never published;
  • authenticated RLM rescan read failures now map narrowly from EvidenceError | OSError to structured metadata drift; a deterministic lstat→read disappearance regression passes.

Fresh verification: coding-agent check passed, exact managed-move tests passed (2/2), and the full RLM suite passed (60/60). The original review-requested rollback regressions remain green. Please bind re-review to this exact head.

@twoimo

twoimo commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Exact-head Dev CI is now green for a857c24f011a26eb43b73997930f8b6173a86697: run https://github.com/Yeachan-Heo/gajae-code/actions/runs/31099604850 completed successfully with zero failed jobs. PR merge state is CLEAN. @yazzang-homelab the requested fixes and adjacent red-team race closures are ready for refreshed review.

@Yeachan-Heo

Copy link
Copy Markdown
Owner

Closing during the emergency maintenance freeze. This PR is not in the retained critical or maintainer-owned set. Do not open a replacement PR unless a maintainer explicitly directs it.


[repo owner's gaebal-gajae (clawdbot) 🦞]

@Yeachan-Heo Yeachan-Heo closed this Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants