Session-bound worktree freshness/ingress can stall basic MCP tools; warm single-file mutations trigger full-root reconciliation

[baron]

Title

Session-bound worktree freshness/ingress can stall basic MCP tools; warm single-file mutations trigger full-root reconciliation

Proposed labels

bug, area:mcp, area:worktrees

Body

Summary

Session-bound worktree roots appear to have a shared freshness / publisher-ingress failure layer that can affect multiple MCP surfaces at once. This is adjacent to, but distinct from, the narrower work already split across:

• Session-bound worktrees: MCP file mutations proceed after fail-closed lookup context, causing flaky edits/create failures #382 / PR Fix mutation tools when worktree scope is unavailable #387: mutation tools must fail closed when worktree scope/projection is unavailable.
• Agent Mode worktree sub-agent starts can fail inherited routing or park without actionable startup failure #383 / PR Fix routed worktree inheritance for UI parents #388: Agent Mode routed worktree inheritance and child-start actionability.

This follow-up is for the lower-level worktree freshness / ingress / reconciliation path. The observed symptoms are:

1. basic tools can time out or hang before doing their actual work because they wait on workspace freshness / ingress;
2. successful single-file mutations in a warm session-bound worktree can still take ~8-12s while read/search/tree are subsecond;
3. the current diagnostics do not clearly identify which root/barrier is stuck or why a full-root reconcile was required.

Duplicate / related issue check

Related but not duplicates:

• Session-bound worktrees: MCP file mutations proceed after fail-closed lookup context, causing flaky edits/create failures #382 tracks mutation safety when the worktree lookup context is fail-closed/unavailable. PR Fix mutation tools when worktree scope is unavailable #387 should prevent canonical-checkout fallback and return typed worktree_scope_unavailable, but it intentionally does not solve freshness stalls or mutation latency.
• Agent Mode worktree sub-agent starts can fail inherited routing or park without actionable startup failure #383 tracks Agent Mode child-start inheritance/startup behavior. It includes evidence that freshness timeouts can stress Agent Mode, but startup watchdog/routed inheritance is a separate layer.
• MCP tool throughput remediation plan #172 is a broader MCP throughput remediation umbrella. This issue is narrower: session-bound worktree freshness/ingress and post-mutation reconciliation.

This issue assumes #382/#387 and #383/#388 remain their own tracks. Acceptance criteria here should not absorb those PRs if their exact implementation changes.

Environment / evidence note

Evidence comes from local RepoPrompt CE worktree-bug investigations on July 6, 2026. Code paths were checked against the current worktree-bug branches and main-line source around 8cf39f70. Timings below are coarse local observations, not formal benchmark artifacts.

Observed behavior

A live session-bound worktree investigation saw basic read/search/tree become fast after reopening the workspace, but mutation tools remained much slower:

get_file_tree: ~0.18s
file_search: ~0.18s
read_file: ~0.27s
file_actions create: ~12.1s
apply_edits one-line replacement: ~8.4s
read_file after edit: ~0.13s
file_actions delete: ~8.5s

Separate evidence from the same worktree-bug investigation showed file_search failing before search began with a typed freshness result:

Status: Workspace freshness timed out
Code: workspace_freshness_timeout
Retryable: yes

That indicates the search did not actually run. It timed out at the shared workspace freshness barrier.

Code paths implicated

Confirmed from code review:

• StoreBackedWorkspaceSearch gates file_search on awaitAppliedIngress(...) and races that wait against MCPTimeoutPolicy.workspaceFreshnessWaitTimeout (currently 30s), returning typed workspace_freshness_timeout before search begins.
• WorkspaceReadableFileService uses the same awaitAppliedIngressForExplicitRequest(...) family for explicit read-file requests.
• MCPApplyEditsToolProvider waits on explicit-request freshness before validation and after writes.
• file_actions waits before mutation and again after mutation / selection updates.
• tree, selection, prompt/context, and code-structure paths also touch the same awaitAppliedIngress* family.
• WorkspaceFileContextStore.awaitAppliedIngress(...) captures watcher watermark plus accepted publisher sequence, then waits for publisher ingress, watcher flush, and downstream publication application.
• WorkspaceFileSystemIngressCoordinator can pause drain during handoff; waiters only release when application advances and wakes them.

The important asymmetry: file_search already has an explicit timeout race and typed retryable error. Other basic tools appear to await the underlying freshness family without equivalent typed, root-scoped freshness failures, so they can hang until request cancellation or burn a generic server deadline.

Current hypotheses

These are plausible, not yet proven:

Hard freshness stall

A session-bound worktree root may leave publisher ingress with an accepted publication that never reaches applied state, or with drain paused during handoff, or with a drain handler stuck before finishApplying. Any tool waiting on freshness for that root can then block behind the same barrier.

Potential diagnostic states to capture:

• accepted service publication sequence > applied service publication sequence;
• watcher watermark target not flushed/applied;
• pending queue not draining;
• applyingCount > 0 without progress;
• paused handoff state not resumed;
• waiters accumulating on the root.

Finite mutation latency

The 8-12s successful mutation timings look separate from a hard stall. The leading hypothesis is unnecessary synchronous full-root reconciliation after a self-authored single-file mutation:

self-authored mutation
→ seeded/worktree authority invalidated by the mutation itself
→ post-mutation freshness wait blocks
→ authority snapshot/fence recapture
→ reconcilePublishedSeededAuthority
→ reconcileEntireTreeForAuthorityChange
→ full-root crawl before tool reply is released

If the watcher and self-authored mutation publication path are healthy, ordinary one-file mutations should reconcile incrementally using known touched path(s), operation type, publication token/sequence, and root/worktree projection. Full-root reconcile should be exceptional repair, not the normal hot path for a single-file edit/create/delete.

Expected behavior

• Basic tools in a session-bound worktree should not hang indefinitely behind workspace freshness / ingress.
• If freshness cannot be reached, tools should return typed, retryable, root-scoped errors naming the blocking layer.
• A warm one-file mutation in a watched session-bound worktree should not synchronously trigger full-root reconciliation on the tool response path.
• Codemap/code-structure readiness should not block basic read/search/edit/file-action tools.
• Diagnostics should make it clear which root and which barrier is blocking progress.

Acceptance criteria

Phase 1: bounded typed freshness behavior

• read_file, apply_edits, file_actions, file tree, selection/prompt paths, and other basic file tools that wait on explicit worktree freshness have bounded waits comparable to file_search.
• When the wait cannot complete, they return typed retryable errors with enough structure to distinguish at least:
  • worktree projection/scope unavailable;
  • workspace file-ingress freshness timeout/unavailable;
  • workspace readiness timeout/unavailable, where distinct from file-ingress freshness;
  • codemap/code-structure readiness, where relevant to that tool only.
• Mutation tools distinguish worktree_scope_unavailable (per Session-bound worktrees: MCP file mutations proceed after fail-closed lookup context, causing flaky edits/create failures #382/Fix mutation tools when worktree scope is unavailable #387) from freshness/ingress timeout as separate typed codes.

Phase 1: root-level diagnostics

• Freshness timeout/unavailable responses include a bounded diagnostic snapshot for the affected root(s), redacted as needed, including:
  • root kind / session-worktree marker;
  • accepted vs applied service publication sequence and gap;
  • watcher watermark target vs applied/flushed watermark;
  • pending queue count;
  • applying count;
  • paused-handoff state;
  • waiter count;
  • oldest outstanding publication age;
  • whether codemap work is pending separately.

Phase 1: handoff / ingress progress

• Session-bound worktree handoff cannot leave publisher ingress paused or with unapplied accepted publications indefinitely.
• If handoff fails, paused subscriptions are resumed or force-terminalized, waiters are released, and later tools receive a typed retryable freshness/worktree-readiness error instead of hanging.
• A hung drain handler or stuck applyingCount has a recovery / diagnostic path.

Phase 2: mutation latency / incremental reconcile

This phase may become a follow-up PR if diagnostics show it should be split from the hard freshness-stall work.

• Warm one-file apply_edits in a watched session-bound worktree does not trigger reconcileEntireTreeForAuthorityChange on the synchronous tool-response path.
• Warm one-file file_actions create/delete/move in a watched session-bound worktree does not trigger synchronous full-root reconcile on the tool-response path.
• Full-root reconcile is reserved for explicit repair/recovery cases such as initial root load, watcher overflow/dropped events, root replacement/authority corruption, or explicit rescan.
• If a full-root reconcile is required, it is diagnostic-tagged with the reason and does not block unrelated basic tools without a typed bounded freshness result.
• Suggested warm-path budget for simple one-file mutation in a session-bound worktree: p50 <= 1s, p95 <= 2.5s, with instrumentation showing where time is spent.

Verification: codemap independence

• Confirm, and enforce with regression coverage, that stale or slow codemap/code-structure generation does not block read_file, file_search, apply_edits, file_actions, or file tree for basic filesystem operations.
• get_code_structure may report codemap readiness/staleness for its own output, but codemap freshness is not a global blocker for unrelated tools.

Suggested instrumentation

Add bounded DEBUG diagnostics / perf stages around:

• pre-mutation freshness wait;
• host write / actual file operation;
• post-mutation awaitAppliedIngressForExplicitRequest;
• publisher ingress accepted/applied sequence changes;
• watcher watermark flush/apply;
• requirePublishedSeededAuthorityFresh / authority fence recapture;
• reconcilePublishedSeededAuthority;
• reconcileEntireTreeForAuthorityChange count and duration;
• post-mutation catalog / selection / token-accounting phases.

Non-goals for this issue

• Do not replace Session-bound worktrees: MCP file mutations proceed after fail-closed lookup context, causing flaky edits/create failures #382 / PR Fix mutation tools when worktree scope is unavailable #387's mutation fail-closed safety fix.
• Do not absorb Agent Mode worktree sub-agent starts can fail inherited routing or park without actionable startup failure #383 / PR Fix routed worktree inheritance for UI parents #388's routed worktree inheritance fix.
• Do not make Agent Mode startup watchdog the primary scope here; that should remain in the Agent Mode worktree sub-agent starts can fail inherited routing or park without actionable startup failure #383 follow-up track, with this issue linked where freshness/readiness contributes.
• Do not require codemap correctness work except to ensure codemap readiness does not block basic tools.

Notes for implementation planning

This issue likely spans:

• Sources/RepoPrompt/Infrastructure/WorkspaceContext
• Sources/RepoPrompt/Infrastructure/MCP
• affected Agent Mode callers only where they wait on or report freshness/readiness

The first useful implementation step may be diagnostic-only: measure whether the 8-12s mutation path is dominated by post-mutation freshness, authority recapture, full-root reconcile, catalog refresh, or selection/token-accounting.

[baron]

Issue #390 follow-up: code-verified evidence, current PR state, and refinements

Investigation performed 2026-07-06 in worktree wt/issue-390-fable-investigation (base origin/main, HEAD 8cf39f70). All file/line references below were read directly from source at that commit; GitHub state was captured live via gh the same day. Claims are explicitly labeled proven (verified in code or command output), plausible (mechanism exists, magnitude/attribution unmeasured), or unproven (no supporting code evidence found yet).

1. Scope check

The issue boundary holds up. Nothing found below belongs in #382/PR #387 (fail-closed mutation lookup) or #383/PR #388 (routed inheritance / startup actionability). Everything here is in the shared freshness/ingress/seeded-authority layer under Sources/RepoPrompt/Infrastructure/WorkspaceContext and its MCP callers, exactly as the issue scopes it.

2. Proven: the timeout asymmetry between file_search and other basic tools is real

file_search has a bounded, typed freshness wait; read/edit/file_actions do not.

• Sources/RepoPrompt/Features/Search/StoreBackedWorkspaceSearch.swift:136-148 selects MCPTimeoutPolicy.workspaceFreshnessWaitTimeout and gates search on awaitAppliedIngress(rootRefs:store:timeout:).
• StoreBackedWorkspaceSearch.swift:872-907 implements a genuine race: one task awaits store.awaitAppliedIngress(rootRefs:), a second sleeps for the timeout and resolves StoreBackedWorkspaceSearchError.workspaceFreshnessTimedOut.
• Sources/RepoPromptShared/MCP/MCPTimeoutPolicy.swift:9-10 sets the bound at 30s (workspaceFreshnessWaitTimeoutSeconds = 30).
• Sources/RepoPrompt/Infrastructure/MCP/WindowTools/MCPFileToolProvider.swift:981-996 maps search errors to typed retryable codes: worktree_scope_unavailable, workspace_freshness_timeout, workspace_readiness_unavailable, workspace_readiness_timeout, workspace_readiness_superseded.

By contrast, the following await the same underlying freshness family with no local timeout race and no typed freshness error:

• read_file: Sources/RepoPrompt/Infrastructure/WorkspaceContext/WorkspaceReadableFileService.swift:22-66 — awaitFreshnessForExplicitRequest plain-awaits store.awaitAppliedIngressForExplicitRequest(...); the only exits are completion or task cancellation.
• apply_edits pre-validation: Sources/RepoPrompt/Infrastructure/MCP/WindowTools/MCPApplyEditsToolProvider.swift:89-93.
• apply_edits post-write (interactive approval path): MCPApplyEditsToolProvider.swift:186-191; post-write (service path, editsApplied > 0): MCPApplyEditsToolProvider.swift:245-251.
• file_actions pre-mutation (path and new_path): Sources/RepoPrompt/Infrastructure/MCP/ViewModels/MCPServerViewModel.swift:5819-5828; post-mutation: MCPServerViewModel.swift:5887-5897.

So if the shared barrier wedges, file_search degrades to a typed retryable workspace_freshness_timeout after 30s, while read_file/apply_edits/file_actions sit on the barrier until the caller's transport deadline or cancellation. That matches the issue's "important asymmetry" claim exactly.

3. Proven: mutations to a seeded (session-bound) root can put every waiter behind a full-root crawl

This is the strongest new finding: the full-root reconcile is not just reachable — it is the designed response to any git-authority change on a seeded root, and basic-tool freshness waits queue behind it.

The chain, all verified in Sources/RepoPrompt/Infrastructure/WorkspaceContext/WorkspaceFileContextStore.swift:

1. Every ingress barrier passes through a seeded-authority freshness gate. awaitAppliedIngress(rootIDs:) (lines 7830-7879) captures the watcher watermark + accepted publication sequence, then calls requirePublishedSeededAuthorityFresh(rootID:) before and after each per-root barrier (lines 7864-7868).
2. That gate is an unbounded waiter. requirePublishedSeededAuthorityFresh (lines 3700-3723) loops while an authority fence exists: if the root is isBlocked it parks on a continuation with no timeout; waiters are only resumed by reconciliation completing, failing, or caller cancellation.
3. Any git-authority invalidation blocks the root. handleSeededAuthorityInvalidation (lines 3462-3538) fires per GitWorkspaceAuthorityInvalidationEvent matching the root's repositoryKey, and unconditionally sets state.isBlocked = true (line 3514) plus a pending reconciliation generation (lines 3532-3535). Note the key is the repository, so activity in other checkouts of the same repo can also block a session-bound worktree root.
4. Reconciliation resolves the block via authority-fence recapture, and a snapshot mismatch triggers a full-root crawl. reconcilePublishedSeededAuthority (lines 3550-3634): if the recaptured replacement.snapshot != fence.snapshot, it calls state.service.reconcileEntireTreeForAuthorityChange() (lines 3573-3588) and then waits for publisher ingress before releasing waiters; the catch path attempts the same full crawl (lines 3612-3624).
5. The full crawl is exactly what its name says. Sources/RepoPrompt/Infrastructure/FileSystem/FileSystemService+SeededInitialization.swift:350-368 — "authoritative full-tree recovery reconciliation", publishing deltas with source: .recoveryFullResync, requiresFullResync: true.
6. Only one full-crawl attempt is allowed per generation; a second mismatch fails the root. Lines 3574-3578 and failPublishedSeededAuthorityReconciliation (lines 3675-3688) resume all waiters with WorkspaceSessionWorktreeOwnershipError.unavailableRoot.

Consequence for the issue's symptom 2: a warm single-file file_actions create/delete or apply_edits in a session-bound worktree changes git state → authority invalidation → root blocked → the tool's own post-mutation awaitAppliedIngressForExplicitRequest queues behind fence recapture and, whenever the authority snapshot changed, a synchronous full-root crawl plus ingress drain — all on the tool-response path. Meanwhile read/search/tree on a quiescent root pass the no-op fast path (WorkspaceFileContextStore.swift:7889-7908) in milliseconds.

Status of the latency attribution: the mechanism above is proven from code. That it dominates the observed ~8-12s (vs subsecond reads) is still plausible, not proven — no instrumented stage breakdown has been captured yet. The observed timings in the issue are coarse local measurements, not benchmark artifacts.

Notably, most of the instrumentation needed to close this gap already exists: EditFlowPerf stages wrap the search wait (Stage.Search.ingressFreshnessWait), the read wait (Stage.ReadFile.explicitIngressFreshnessWait), and the apply_edits post-write flush (Stage.ApplyEdits.flushDeltas), and DEBUG builds count full crawls per root (publishedSeededAuthorityFullCrawlCountsByRootID, WorkspaceFileContextStore.swift:3581). A diagnostic-first Phase 0 is mostly a matter of running the warm-mutation scenario and reading these out.

4. Proven mechanism / unproven occurrence: the hard-stall (handoff pause) hypothesis

Sources/RepoPrompt/Infrastructure/WorkspaceContext/WorkspaceFileSystemIngressCoordinator.swift:

• The coordinator applies accepted publications serially per root and lets barriers await an exact publication cut (header, lines 4-8).
• pauseDrainAndReplaceHandler (lines 224-239) sets isDrainPausedForHandoff = true; while paused, scheduleDrainIfNeeded (line 622) and takeNextPublication (line 670) both refuse to run, so accepted publications queue without applying.
• Barrier waiters are released only when finishApplying advances the applied sequence (lines 690-731) or on termination/cancellation.
• The seeded-root commit path in WorkspaceFileContextStore.swift:4448-4616 does resume paused subscriptions on its visible failure branches (resumeDrainAfterHandoff at 4459-4461, 4482-4484, 4522-4524, 4614-4616) — but resumption depends on control flow reaching those branches. A drain handler that never returns, or a handoff task that dies between pause and resume, leaves the root paused with waiters parked indefinitely. No watchdog or force-resume exists on this path.

So: proven that a paused/wedged ingress state blocks all freshness waiters with no bound and no typed error for non-search tools; unproven that this specific wedge caused the observed live stalls (no captured stuck-state snapshot ties the July 6 workspace_freshness_timeout events to a paused handoff vs. slow reconcile vs. something else).

Also relevant to the diagnostics acceptance criteria: the coordinator already has a DebugSnapshot (lines 43-55) carrying nearly every field the issue asks for — queued/applying/outstanding counts, waiter count, accepted vs applied sequence and gap, applied watcher watermark, oldest outstanding age — but it is DEBUG-only and is not surfaced in any typed MCP freshness error today. The TerminationReport (lines 28-41) carries the same fields at teardown. Wiring these into the typed error surface is closer to plumbing than new design.

5. Codemap independence: no coupling found (verification still worthwhile)

The barrier target is strictly watcherAcceptedWatermark + acceptedServicePublicationSequence (WorkspaceFileContextStore.swift:7841-7856); I found no codemap readiness input anywhere in the awaitAppliedIngress* path. I could not fully trace every drain-handler application step in this pass, so this is stated as "no evidence of coupling found", not proof of independence. The issue's regression-coverage criterion remains the right way to pin it.

6. Current PR state (captured 2026-07-06 via gh) — including discrepancies

PR #387 (wt/issue-382-investigation, head af8c4625)

OPEN, MERGEABLE, REVIEW_REQUIRED. Style, check-contributor, Provider Tests, Secret Scan pass; Seer pending. All four Build and Test app shards fail, and the failure is deterministic, not flaky (run 28763601002, identical on every shard):

##[error]ledger is missing discovered suites: ['RepoPromptTests.MCPMutationRetryableFailureTests']

The PR adds the MCPMutationRetryableFailureTests suite without the required curated contract-ledger update (per docs/testing.md, executable suite adds must update the ledger). This needs a ledger fix on the PR branch before it can merge; it does not indicate a product defect.

PR #388 (wt/issue-383-fable-investigation, head a1b5fbe8)

OPEN, MERGEABLE, REVIEW_REQUIRED. Discrepancy: the PR thread's final QA comment claims ready-to-merge, but CI is currently red. Shards 1/3/4, Style, Provider Tests, Secret Scan pass; app shard 2 fails (run 28770389810):

Tests/RepoPromptTests/AgentMode/AgentToolTrackingControllerTests.swift:227
testConcurrentRawUnregisterAndStopJoinCapturedDeliveryBarrier
XCTAssertEqual failed: ("0") is not equal to ("1")   (6.072 seconds)

This failure is a single concurrency test unrelated to the PR's touched files, consistent with the known roving single-concurrency-test CI flakiness tracked in #161 — but as of this capture the check is red, so "ready to merge" should read "ready pending a green rerun of shard 2".

Neither failure is caused by, or fixes, anything in #390's scope, but both matter for sequencing: #390 explicitly assumes #387/#388 land on their own tracks.

7. Recommended next steps and acceptance-criteria refinements

1. Phase 0 (diagnostic-only, cheapest first step, as the issue already suggests): run the warm-mutation scenario in a session-bound worktree with EditFlowPerf/perf diagnostics enabled and read out Stage.ApplyEdits.flushDeltas, Stage.ReadFile.explicitIngressFreshnessWait, and the DEBUG full-crawl counters. This directly decides whether Phase 2 (incremental reconcile) or Phase 1 (bounded waits) is the higher-leverage first PR.
2. Bound the explicit-request family at the store seam, not per-caller. The StoreBackedWorkspaceSearch race pattern works, but replicating it in four call sites invites drift. A timeout: parameter on awaitAppliedIngressForExplicitRequest returning a typed outcome would give read/edit/file_actions the same 30s bound in one place.
3. Surface the existing coordinator snapshot in typed errors. The DebugSnapshot/TerminationReport fields already match the issue's root-level diagnostics checklist almost one-for-one; promote a bounded, redacted subset out of DEBUG and attach it to workspace_freshness_timeout-class results.
4. Add a typed distinction for seeded-authority blocking. Today a blocked seeded root surfaces either as an indefinite wait or (after a failed reconcile) as unavailableRoot — which callers will read as "worktree scope unavailable" (Session-bound worktrees: MCP file mutations proceed after fail-closed lookup context, causing flaky edits/create failures #382's code) even though the actual blocker was authority reconciliation. Acceptance criterion "distinguish worktree scope vs freshness vs readiness" should explicitly cover the requirePublishedSeededAuthorityFresh failure/blocked paths.
5. Incremental authority reconcile for self-authored mutations (Phase 2 as written): the current design compares whole authority snapshots and full-crawls on any mismatch, with no narrowing by the known touched path(s). The issue's proposed incremental path matches the code structure; the single-attempt-then-fail-root behavior (fullCrawlAttemptedGeneration guard) also deserves a criterion, since its failure mode converts a latency problem into an availability problem (unavailableRoot for the whole root).
6. Handoff watchdog (Phase 1 handoff criteria as written): add a bound or watchdog on isDrainPausedForHandoff so a lost resume cannot park waiters forever; the resume-on-failure branches in the commit path are necessary but not sufficient.

Evidence quality summary

Claim	Status
file_search bounded/typed vs read/edit/file_actions unbounded	Proven (code)
Mutation → authority invalidation → blocked barrier → full-root crawl on tool-response path	Proven mechanism (code)
Full-root crawl dominates the observed 8-12s warm-mutation latency	Plausible, unmeasured
Handoff pause can park waiters indefinitely if resume is lost	Proven mechanism; occurrence in the wild unproven
Codemap readiness blocks basic tools	No evidence found; independence not yet proven either
Observed timings (~0.13-0.27s reads vs ~8-12s mutations)	Coarse local observations, not benchmarks
PR #387 CI red (ledger contract, deterministic)	Proven (run 28763601002)
PR #388 CI red on shard 2 despite QA ready-to-merge comment	Proven (run 28770389810); flakiness attribution plausible per #161

Codex Dissenting Opinion

One disagreement with the Phase 1 acceptance criteria as currently written: giving mutation tools a plain retryable freshness timeout after the write has already succeeded is the wrong shape. For apply_edits/file_actions, the pre-mutation wait can safely return a retryable error, but the post-mutation wait fires when the file operation is already durable on disk (MCPApplyEditsToolProvider.swift:186-251, MCPServerViewModel.swift:5887-5897). A generic retryable error there would tell an agent the edit failed; the natural retry (file_actions create again, or re-applying a search/replace) would then fail with already-exists / no-match, or double-apply. If Phase 1 adds bounds to the post-mutation waits, the timeout result should be a typed partial-success ("mutation applied; workspace projection not yet fresh; re-read before further edits"), not the same retryable failure code used for pre-work freshness timeouts.