fix: simplify durable transcript imports

[NicholaiVogel]

Summary

Adds durable, agent-scoped JSONL transcript imports through the daemon, CLI, and dashboard. Imports retain their source evidence and provenance, resume after interruption, reconcile duplicates and rejections, and remain purgeable through the source lifecycle. Linux and macOS use descriptor-relative filesystem guards; unsupported platforms fail before mutation.

Job creation and start/pause/resume/retry/cancel now share the import store's database-owner transaction path. Duplicate route SQL, unused protocol executors, and unused helpers are removed. A pending cancellation survives a later pause request. Hono bounds ordinary request bodies, including chunked bodies; transcript uploads retain their own streaming limit without being cut off by the ordinary 10 MB ceiling.

Verification

Rebased onto main at dc958ad71c955fd9c2366a858d22011b077bd1f2; current head is 831a78e852acc72816ace45241ba96d3d9495faf. After rebasing, 41 focused route, HTTP body-limit, memory-candidate, and audit tests passed, along with daemon typecheck, event-loop audit, and git diff --check. The simplification is preserved; other changed files match main, with the generated audit baseline refreshed.

The broader verification below was completed on pre-rebase head 34a477294fe77306171d971a29a0dab16f47ac96:

• 323 focused tests passed: migrations, transcript import/replay/staging/commit/worker/routes, source purge and lifecycle, Dreaming evidence, document imports, database-owner client, dashboard sources, recovery, HTTP body limits, and event-loop audit.
• Real-daemon evaluation passed with a JSONL upload above 10 MB, 2,200 imported conversations, exact replay duplicates, malformed-record accounting, restart during inventory, crash after filesystem append, scope/provenance checks, source purge, and retained audit tombstones. Its temporary workspace was removed after shutdown.
• Core typecheck/build, daemon typecheck/build including the dashboard bundle, and the native release build passed. Changed-file Biome passed with one existing dashboard accessibility warning; git diff --check passed.
• Cancellation regressions fail against the original route. The chunked-overflow regression fails against the original middleware. Restored implementations pass.
• Phase D stability smoke passed locally (worst event-loop delay 119 ms; health p95 2 ms; status p95 60 ms). CI initially reported one 4.86-second delay, then passed on one retry of the unchanged commit. No timing thresholds were changed.
• Browser inspection confirmed the transcript import form and job views. Full workspace tests were not run.

Migration notes

Migrations 146–149 add import ledgers, replay-slot indexing, attempt provenance, and control/retry fields. Migration 147 remains additive and does not rebuild the replay-file table or delete existing records. Repeated migration and preservation checks pass. Roll back application code without dropping the additive tables/columns; retain import evidence and ledgers for a forward repair. This change does not renumber existing migrations.

• Migration is idempotent
• Rollback / compatibility note included in PR description

PR Readiness (MANDATORY)

• Spec alignment validated (INDEX.md + dependencies.yaml)
• Agent scoping verified on all new/changed data queries
• Input/config validation and bounds checks added
• Error handling and fallback paths tested (no silent swallow)
• Security checks applied to admin/mutation endpoints
• Docs updated for API/spec/status changes
• Regression tests added for each bug fix
• Lint/typecheck/tests pass locally

AI-assisted implementation and verification with Codex. This PR has not been merged.

[NicholaiVogel]

Repair outcome:

• Published superseding PR fix: simplify durable transcript imports #1838 at head 8b28422dce54617489ec6d4e4b7d96578dac3cc7, rebased onto origin/main 3bcc7ae9922749195ed0e5b0baa91a7fa81f9dd3.
• Focused transcript adapter/inventory/staging/commit/routes/worker/replay/migration tests: 94 passed, 0 failed.
• Native embedding regression suite: 10 passed, 0 failed.
• Core typecheck/build, daemon typecheck/build, native release build, dashboard production build, and changed-file Biome check passed. The first combined daemon build hit a transient Bun 132 segmentation fault; rerunning bun run build.ts passed.
• Mutation evidence: reverting the byte-aware splitter caused the boundary test to fail exactly at the expected assertion (3 passed, 1 failed); restoring the implementation returned the suite to 4 passed, 0 failed.

The PR remains open for the requested independent escape review and has not been merged.

[NicholaiVogel]

Repair round complete for PR #1838.

Published and verified:

• Exact head: 75d93bb723672d4690dd32abdefb7b0a413cdf63
• Exact base: c56da4519711b273074171b66fec474577f094b8 (origin/main)
• git merge-base --is-ancestor origin/main HEAD: passed
• PR remains open and unmerged

Final exact-head focused verification, run in separate serial Bun processes:

• Core migrations: 71 passed, 0 failed
• Transcript-import routes: 5 passed, 0 failed
• Adapter: 2 passed, 0 failed
• Commit: 6 passed, 0 failed
• Staging: 2 passed, 0 failed
• Inventory: 1 passed, 0 failed
• Replay regression: 1 passed, 0 failed
• Worker: 15 passed, 0 failed
• Dashboard sources: 10 passed, 0 failed
• Event-loop contract audit: 23 passed, 0 failed

Core and daemon typecheck/builds, dashboard production build, native optimized release build, frozen-lockfile install, changed-file Biome check, and git diff --check all passed. GitHub required checks are green, including migration notes and the mandatory PR checklist.

Mutation verification proved the new migration-preservation, purge/append serialization, upload-retry, pending-finalize, byte-boundary, workspace-root, and dashboard duplicate-mode regressions fail when their corresponding fixes are removed and pass after restoration.

Requesting a fresh independent escape review at this exact head. Do not merge yet.

[NicholaiVogel]

Repair verification update

• Exact head: ff786aa65de7a42c430d177561f8a9988aa5af5c
• Base: b325bfc191b5894696bad6689e71f5b71b359815
• The branch is rebased onto current main; the PR remains open and has not been merged.

Focused verification is green:

• migrations 71/0
• transcript adapter 2/0
• transcript commit 6/0
• transcript inventory 1/0
• transcript staging 3/0
• transcript worker 17/0
• transcript routes 7/0
• replay regression 1/0
• recovery worker 23/0
• event-loop contract audit 23/0

Build/static gates are green: core typecheck/build, daemon typecheck/build, native build, changed-file Biome check, and git diff --check.

Fresh mutation verification was also completed and restored cleanly:

• removing the retry changes() > 0 guard made the cancelled-job retry regression fail (6 pass, 1 fail); restored route suite 7/0;
• removing upload/cancel final preflight and DB guards made the cancellation regression return 201 instead of 409 (6 pass, 1 fail); restored route suite 7/0;
• making symlink-component validation a no-op made the staging containment regression resolve instead of reject (2 pass, 1 fail); restored staging suite 3/0;
• removing the purge source-directory symlink guard deleted the outside file and produced ENOENT (16 pass, 1 fail); restored worker suite 17/0.

The full evidence ledger is at /mnt/work/hermes-scratch/t_69ac9cb0-final-repair-evidence.md in the worker environment. CI checks for this exact head are currently pending.

[NicholaiVogel]

Repair round 4 published at exact head 6921f5b, rebased onto origin/main b325bfc.

This round replaces sequential symlink checks with descriptor-held POSIX no-follow traversal for managed transcript filesystem operations, makes omitted-agent purge all-agent through transcript ledgers and files, and makes cancellation cleanup idempotent when a staged parent is already absent. Focused regressions cover final symlinks, replacement between descriptor-held check/use, two-agent omitted-scope purge, ownerless locks, and cancellation cleanup.

Verification:

• 12 serial focused suites: all green, including migrations 71/0, inventory 2/0, staging 4/0, commit 7/0, worker 19/0, source-artifact graph 8/0, transcript routes 7/0, recovery 23/0, startup recovery 11/0, and sources routes 47/0.
• Changed-file Biome, event-loop audit 23/0, core typecheck/build, daemon typecheck/build with dashboard, native release build, and git diff --check: all passed.
• Mutation evidence: removing inventory O_NOFOLLOW, replacing descriptor-relative open with an absolute open, wiring omitted-agent purge to default scope, and removing the final cleanup ENOENT guard each failed its targeted regression; restored runs passed. Full evidence: /mnt/work/hermes-scratch/t_69ac9cb0-round4-evidence.md

The remote branch and PR head were verified, PR #1838 remains OPEN with mergedAt null, and this PR has not been merged. GitHub checks were still pending at publish time. Please run a fresh independent escape review against this exact head; do not merge until it passes.

[NicholaiVogel]

Post-publication verification: all 22 GitHub checks for PR #1838 passed, including CodeQL, Biome, daemon production typecheck, migration notes, regression-test, smoke, source-run liveness, native health, compiled macOS/Ubuntu checks, and build-and-deploy. Exact remote head remains 6921f5b; PR state remains OPEN with mergedAt null. Fresh independent escape review is still required; do not merge.

[NicholaiVogel]

Round 6 repair complete.

• Unsafe managed-path failures now classify as durable source changes/nonretryable recoveries instead of retryable transient failures.
• Transcript-import purge ignores only ENOENT; other filesystem failures propagate before the import ledger is deleted, and source purge no longer swallows ledger lookup failures.
• Added symlinked staged-file worker coverage plus end-to-end symlink and EACCES ledger-retention regressions.

Exact published head: 162e77d
Base: b325bfc
Mutation evidence: /mnt/work/hermes-scratch/t_69ac9cb0-round6-mutation-evidence.md

Verification: focused daemon suites, event-loop audit, changed-file Biome, core typecheck/build, daemon typecheck/build, native release build, and git diff checks pass. Both new mutations failed their intended regression tests and restored runs are green. All 22 GitHub checks pass. PR remains open and unmerged; ready for fresh independent escape review.

[NicholaiVogel]

Repair handoff evidence for the fresh review

• PR: fix: simplify durable transcript imports #1838
• Base: b325bfc191b5894696bad6689e71f5b71b359815
• Exact pushed head: 0388edf591c982be91e2cd63e5970c52726c382b
• origin/main is an ancestor of the exact head.

Final worktree verification:

• 131 focused serial Bun tests passed with 0 failures: platform 3, adapter 2, staging 4, inventory 2, commit 7, replay 1, worker 20, source-artifact graph 10, imported-source lifecycle 2, imported-source outcome 1, transcript routes 8, source routes 48, and event-loop contract audit 23.
• bun run eval:transcript-import passed the real-daemon restart, replay, rejection, filesystem-before-DB failpoint, source purge, provenance, and audit-tombstone checks.
• @signet/core typecheck/build, @signet/daemon typecheck/build, and @signet/native optimized release build passed.
• Changed-file Biome check, event-loop contract audit, and git diff --check passed.

Mutation evidence:

• Temporarily adding win32 to TRANSCRIPT_IMPORT_SUPPORTED_PLATFORMS made the platform contract fail 2 of 3 tests: the structured unsupported-platform error became undefined, and the supported-platform contract changed. Restored run: 3 pass, 0 fail.
• Removing the worker's symlink-error classification made 1 of 20 worker tests fail on the expected nonretryable source-change assertion. Restored run: 20 pass, 0 fail.
• Restoring broad catch-and-swallow behavior in purge made 2 of 10 source-artifact tests fail on symlinked-staged-directory and non-ENOENT filesystem-retention assertions. Restored run: 10 pass, 0 fail.

No merge was performed. This is ready for an independent escape review at the exact head above.