fix(checkpoint): don't count a timed-out writer wait as a writer failure by wqymi · Pull Request #1938 · XiaomiMiMo/MiMo-Code

wqymi · 2026-07-27T11:27:01Z

Problem

waitForWriter bounds the writer's Deferred at 300s and maps the expiry to {status:"failure"} (packages/opencode/src/session/checkpoint.ts:989-993 on main).

That expiry does not cancel the writer. The settle watcher that owns the watermark advance awaits the same Deferred with no bound (checkpoint.ts:911-921), so a slow-but-successful writer still advances last_checkpoint_message_id after the bounded wait gave up.

So the prune retry watcher (prune.ts:312-337) books a working writer as broken:

writerFailures ticks
the session's crossed / maxCrossed thresholds are cleared → checkpoint writer failed — cleared thresholds for retry
after MAX_WRITER_FAILURES such waits → checkpoint writer gave up after max consecutive failures, and the session stops being checkpointed even though every writer actually succeeded

Losing checkpoint coverage also degrades /rebuild: the span it can release is exactly the span the checkpoint covers.

Evidence (live TUI, dev build)

Real run on mimo-v2.5, checkpoint writer on the parent's model:

event	timestamp
`threshold=60000 currentTokens=112596` → writer spawned	`10:47:07`
`checkpoint writer failed — cleared thresholds for retry` (`attempt=1`)	`10:52:07` (exactly +300s)
same writer settles successfully, watermark advances `msg1 → msg4`	shortly after

Reproduced twice in a single session. The +300s alignment is exact — it is the wait bound expiring, not a writer error.

Fix

Return a distinct "timeout" outcome instead of "failure".

prune.ts's existing if (result !== "failure") return guard then skips the counter and leaves thresholds untouched — no prune logic change needed.
/rebuild is unaffected — and more strongly than an earlier revision of this description claimed. That revision cited a case-2 if (writerOutcome !== "success") bail at prompt.ts:4073-4082. No such code exists: git grep writerOutcome has zero hits repo-wide, and prompt.ts:4073-4082 is actor-notification rendering. The real shared path is rebuildFromCheckpoint (prompt.ts:398-430), used by both the automatic overflow path and manual /rebuild, and it never consults the writer outcome at all — it gates on hasCheckpoint + lastBoundary and deliberately does not block on an in-flight writer (see its own comment at prompt.ts:394-397). Changing the outcome value therefore cannot affect it.
Retrying while the writer is still in flight was already a no-op: isWriterRunning short-circuits new threshold triggers, so nothing is lost by not counting.

Test

New test/session/checkpoint-writer-wait-timeout.test.ts uses TestClock to drive past the 5-minute bound with the writer's Deferred still unresolved.

Verified it is a real regression test — with the source fix reverted it fails with Expected: "timeout" / Received: "failure".

bun run typecheck → exit 0
focused suites (drain, prune, prune-skip-system, watermark-transactional, thresholds, new test) → 41 pass / 0 fail
full test/session/ → 821 pass / 1 flaky fail that also passes on re-run and is unrelated to this change

Notes

An accidental benefit was removed along with the bug

Stating the trade honestly: on main, the mistaken timeout → failure mapping cleared crossed and maxCrossed (prune.ts:325-326). That had a beneficial side effect nobody designed — after a slow writer finally settled, the threshold re-fired. This PR removes that re-fire together with the false failure.

Mitigation: at the max threshold maxCrossed now stays set, so prompt.ts:3308-3314 still reaches rebuildFromCheckpoint, which calls prune.resetThresholds when it succeeds (prompt.ts:428) and re-arms the thresholds. Checkpointing is therefore not permanently stalled.

Follow-up

Two gaps this PR left behind are fixed in #1945: hitting the bound became completely silent (no log from either layer), and a writer that genuinely fails — or succeeds — past 300s had its real outcome booked nowhere, since prune's watcher returned on "timeout" and never re-awaited.

Independent of #1752 — the defect is present on main and this PR is deliberately not bundled into it.

Two related issues found during the same investigation are not addressed here (they need their own discussion):

Coverage pinning — thresholds fire mid-turn, but computeBoundary anchors on the last finished assistant and returns msgs[0] when there is none. A token blowup inside the first assistant turn pins coveredUpTo to message 1, so a full 5-minute writer run covers exactly one message.
Writer wall-clock — the writer inherits the parent's model (no model field on the checkpoint-writer agent) and needs ~13 sequential LLM round-trips (~20-25s each) to read prior checkpoint/memory and write its 3-4 files, which is what puts it near the 300s line in the first place.

waitForWriter bounded the writer Deferred at 300s and mapped the expiry to {status:"failure"}. That expiry does not cancel the writer, and the settle watcher that owns the watermark advance awaits the same Deferred with no bound — so a slow-but-successful writer still advances last_checkpoint_message_id after the wait gave up. The prune retry watcher therefore booked a working writer as broken: writerFailures ticked and the session's crossed thresholds were cleared ("checkpoint writer failed — cleared thresholds for retry"). After MAX_WRITER_FAILURES such waits it logs "gave up after max consecutive failures" and stops checkpointing the session entirely — even though every writer had actually succeeded. Losing checkpoint coverage also degrades /rebuild, whose released span is what the checkpoint covers. Observed on a live TUI run: writer spawned at 10:47:07, "failed" logged at 10:52:07 (exactly +300s), and the same writer then succeeded and advanced the watermark. Reproduced twice in one session. Report "timeout" instead. prune's existing `result !== "failure"` guard then skips the counter and leaves thresholds alone, and /rebuild's `!== "success"` check is unchanged. Retrying while the writer is still in flight was already a no-op: isWriterRunning short-circuits new triggers.

…hold retry The checkpoint writer is fail-acceptable and the code already implements that redundancy: - ensureCheckpointTemplate (checkpoint.ts:117-121) writes the template only when the file is ABSENT, so a failed writer cannot clobber the previous checkpoint. - last_checkpoint_message_id advances only on a successful insert (checkpoint.ts:918-934; the sole external resetThresholds caller is prompt.ts:428, gated on `if (inserted)`). So a failed write leaves file and watermark mutually consistent, both pointing at the last good checkpoint, and a rebuild uses that automatically. The cost of a failure is a STALER checkpoint, never a missing one, and the next threshold crossing is a natural retry with fresher context than an in-place one. The accounting layered on top did not trust that, and its failure model was wrong: a writer failing repeatedly means the provider is broken, in which case the foreground turns are failing too and the user already knows. The writer's LLM calls go through the same retry ladder as the foreground (retry.ts), so any failure reaching prune is already post-retry. Counting it again, three times, then falling silent, added nothing. Deleted: - MAX_WRITER_WAIT_EXTENSIONS and the wait-extension loop - writerFailures, MAX_WRITER_FAILURES, and the in-place-retry path that cleared crossed/maxCrossed on failure - cfg.checkpoint.max_writer_failures (schema, docs, generated SDK) Kept: the bounded wait itself, forked purely as observation, so waitForWriter's bound-expiry log stays reachable. That log is the only evidence a slow writer leaves, and a line exactly like it is what diagnosed #1938. Terminal outcomes are already logged by the settle watcher, so the outcome is discarded here. waitForWriter and its distinct "timeout" outcome are #1938's merged work and are untouched. Residual, accepted: with no in-place retry, a session whose LAST threshold fails (mid-tier's last is 80%) gets no further fire, so its checkpoint stays stale and an overflow rebuild uses the stale one. That is the design's own premise, not a regression.

…very "The next threshold crossing is the retry" presumes a next threshold. The last one has none, and its only would-be successor is unreliable: the discard+rebuild that maxThresholdCrossed triggers re-arms the ladder via resetThresholds (prompt.ts:428), but rebuildFromCheckpoint bails when lastBoundary is unset (prompt.ts:413) — and the watermark is unset exactly when no writer has ever succeeded. So in the one case that needs recovery most, the session never checkpoints again until overflow. The principle: a failed checkpoint write earns a retry only when the failure's CLASS says a retry could succeed, and only where the token axis still has ROOM for one. Both are readings of the present state, so neither needs failure history. Mechanism (prune.ts): on a settled retryable failure of the FINAL threshold, arm a per-session gate one ladder STEP above the token count that fired, clamped to maxAllowed. Deterministic or unclassified failures arm nothing — they would recur identically. Non-final thresholds arm nothing — the ladder already is their retry. The gate is consumed when it fires and dropped by resetThresholds. Why this is not MAX_WRITER_FAILURES renamed: a counter accumulates attempts and permits N retries whether or not anything changed. This is a position, overwritten not accumulated, derived from the ladder and the window. It bounds the retry RATE to the ladder's own firing rate, bounds the retry COUNT to whatever the unused window affords (zero once firedAt + step is past maxAllowed), and a conversation that stops growing stops retrying by itself. Plumbing: waitForWriterSettlement carries the FailureInfo the outcome already had. #1938's flat waitForWriter contract is untouched — it is now projected off the same implementation, so the two cannot drift. A "timeout" never carries a classification, so a merely slow writer can never arm the gate. Also corrects a test comment that still justified itself by "keeping the failure cap reachable" so a broken slow writer "would never be counted" — the cap and the counting were deleted earlier in this branch.

…xpires Follow-up to XiaomiMiMo#1938. That PR stopped a merely-slow writer from being booked as a failure by returning a distinct "timeout" from waitForWriter, which prune's `result !== "failure"` guard skips. Correct, but it left two holes. 1. Hitting the bound became completely silent. waitForWriter returned and prune's watcher returned, neither logging — yet the +300s log line is exactly the evidence XiaomiMiMo#1938 was diagnosed from. waitForWriter now logs "checkpoint writer wait bound expired — writer still in flight". 2. The real outcome was booked nowhere. prune's watcher fiber is the only holder of the per-fire accounting and writerFailures is private to the prune layer, so once the watcher returned on "timeout" a writer that genuinely FAILED past 300s ticked no counter — making MAX_WRITER_FAILURES unreachable for exactly the slow regime XiaomiMiMo#1938 is about, so a permanently-broken-but-slow writer retried forever with no give-up warning. Symmetrically, a writer that SUCCEEDED past 300s never cleared a counter left at 1-2 by earlier fast failures, so a later fast failure could trip "gave up" for a session whose writers demonstrably work. The watcher now extends its wait across bound expiries and accounts for the settled result, capped at MAX_WRITER_WAIT_EXTENSIONS (~1h) so a writer that never settles cannot pin the fiber for the life of the process. Two microsecond-wide re-entry windows are documented rather than papered over. Tests: prune.test.ts gains the two cases that pin the prune-side consequence XiaomiMiMo#1938 is actually about (timeouts never tick the counter and a late success clears it; a late failure is still counted so the cap stays reachable) — the existing test only asserted waitForWriter's return value. The timeout test now also pins the BOUND (still pending at 4 minutes, so shrinking it to 1s fails) and asserts the writer is still running after the expiry, replacing a dead `not.toBe("failure")` implied by the line above it. Also folds the stale 5-min-padding comment into the current block and corrects checkpoint-align.ts's now-conditional claim about writerFailures.

wqymi merged commit 076b790 into main Jul 27, 2026
6 checks passed

wqymi mentioned this pull request Jul 27, 2026

fix(checkpoint): delete the writer's blind failure count, classify the failure, and give the final threshold a bounded recovery #1945

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(checkpoint): don't count a timed-out writer wait as a writer failure - #1938

fix(checkpoint): don't count a timed-out writer wait as a writer failure#1938
wqymi merged 1 commit into
mainfrom
fix/checkpoint-writer-wait-timeout-not-failure

wqymi commented Jul 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

wqymi commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Evidence (live TUI, dev build)

Fix

Test

Notes

An accidental benefit was removed along with the bug

Follow-up

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

wqymi commented Jul 27, 2026 •

edited

Loading