feat(dreaming): run concurrent passes in one scope

[aaf2tbz]

Summary

Dreaming ran at most one pass per agent scope. partitionDreamingScopes (platform/daemon/src/pipeline/dreaming-worker.ts) only spread passes across scopes, so one large scope drained one pass at a time no matter how high maxConcurrentPasses was. The blocker the issue names is real: the delivery queue (drainDreamingEvidenceQueueInDb) learns what was consumed only when a pass finalizes, so two passes in one scope would be handed the same sources.

This PR implements the issue's proposal behind a new opt-in setting:

1. Leases. A pass that may share its scope leases each source it takes from the delivery queue (dreaming_evidence_leases, migration 170). The lease is taken in one owner write per source whose upsert refuses a row held by another pass that is still running and unexpired, so two passes racing for one source cannot both win. The queue skips sources leased by another live pass. A pass that loses a race re-reads the queue (up to four times) and gets the next unclaimed sources instead. When other passes hold everything left, its page is empty with hasMore: false and heldByOtherPasses: true, and a note tells the model to finish. A lease ends when its pass leaves running (finalize, failure, cancellation, or startup recovery marking it interrupted), or after twice the pass timeout as a crash backstop. A failed or cancelled pass's sources are delivered again. Finalization does not settle a source another running pass holds, even if this pass looked it up by sourceRef, so the holder keeps it and a failed holder's sources stay queued.
2. Evidence-time supersession. This was mostly on main already (577f87200, 6c0d931f4): an incoming claim whose slot has an active claim with newer evidence lands as superseded. One gap remained: when only one of the two claims carried an event time, the later write won. With newest-first delivery or concurrent passes, "lives in Austin" (no date, older session) written after "moved to Denver in May 2024" (dated, newer session) became current. Now event times decide only when both claims have one; otherwise the later-captured source wins.
3. Admission. With memory.dreaming.maxPassesPerScope > 1, a scope whose measured backlog reaches tokenThreshold, or that has more pending sources than the 50 the backlog probe reads, gets up to that many incremental passes. They come only from slots left after every scope with work has a pass, and they stay within maxConcurrentPasses and the shared LLM limit. A scope can also be joined while its earlier passes still run.

Fixes #2040

Changes

• platform/daemon/src/pipeline/dreaming-evidence-leases.ts (new): lease acquire, live-lease lookup, and release at finalize. Release also clears rows left by passes that are no longer running.
• platform/daemon/src/pipeline/dreaming-capabilities.ts: the queue drain excludes sources leased by other live passes, both fresh sources and continuations. search_evidence leases a drained page when the pass has evidenceLeaseMs, and re-drains when it loses part of the page to another pass. An empty queue page reports heldByOtherPasses: true when other running passes hold leases in the scope. The tool description says what hasMore and heldByOtherPasses mean when passes share a scope.
• platform/daemon/src/pipeline/dreaming.ts:
  • New DreamingPassLiveOptions.sharedScope.
  • A shared pass checks and renders pending attention only for its attentionScopes. The worker gives a scope's attention to one running pass at a time: a joining pass gets it only when no running pass owns it, so two passes never work the same attention record, and the next pass to start after the owner finishes takes it over.
  • Finalize treats sources held elsewhere as deferred and releases this pass's leases.
  • resetDreamingTokens no longer moves last_pass_at backward when passes finish out of order.
• platform/daemon/src/pipeline/dreaming-worker.ts:
  • A scope qualifies for extra passes when its backlog probe reaches tokenThreshold or is indeterminate with backlog (the probe hit its 50-source cap, so more is pending). The scheduler trigger and the other probe consumers are unchanged.
  • partitionDreamingScopes takes an optional per-scope passes count and adds extra same-scope groups round-robin into spare slots, after attention-only scopes.
  • The worker tracks per-scope pass counts and which running pass owns each scope's attention. closedScopes() replaces leasedScopes() for admission. Exclusive, directed, and user-request passes still close their scopes.
• platform/daemon/src/ontology-proposals.ts: the mixed event-time ordering fix described above.
• platform/core/src/types.ts, platform/daemon/src/memory-config.ts, platform/daemon/src/routes/pipeline-routes.ts: maxPassesPerScope (optional, default 1, clamped 1..16), reported in GET /api/dream/status.
• platform/core/src/migrations/170-dreaming-evidence-leases.ts and the registry.
• web/docs/src/content/docs/api/knowledge-ontology.md: the setting, when a scope qualifies, leases and lost lease races, attention ownership, and the ordering rule.
• scripts/event-loop-contract-baseline.json regenerated.

Type

• feat — new user-facing feature (bumps minor)
• fix — bug fix
• refactor — restructure without behavior change
• chore — build, deps, config, docs
• perf — performance improvement
• test — test coverage

Packages affected

• @signet/core
• @signet/daemon
• @signet/cli / dashboard
• @signet/sdk
• @signet/connector-*
• @signet/web
• predictor
• Other:

Migration Notes (if applicable)

• Migration is idempotent
• Rollback / compatibility note included in PR description

Migration 170 only creates dreaming_evidence_leases and its index (IF NOT EXISTS). An older daemon ignores the table. With maxPassesPerScope at 1, nothing writes to it.

Testing

• bun test passes. I ran the touched suites: src/pipeline/, src/ontology-proposals.test.ts, src/memory-config.test.ts, src/episodic-sources.test.ts, src/startup-recovery.test.ts, platform/core/src/migrations, scripts/audit-event-loop-contract.test.ts. The only failures were createAcpxProvider cases in src/pipeline/provider.test.ts. They spawn ACPX processes and do not touch this code. One imported-source-lifecycle case also failed in the combined run because the custom SQLite was already loaded; it passes alone.
• bun run typecheck passes (pre-commit typecheck of all affected workspaces)
• bun run lint passes (biome check on changed files: no new warnings; comments:check is current)
• Tested against running daemon
• N/A

New tests. Each fails without the change, except the two default-behavior guards:

• dreaming-evidence-leases.test.ts:
  • Concurrent passes in one scope get disjoint pages.
  • A contested source goes to exactly one pass.
  • When another pass leases a source between this pass's read and its lease write, the page is refilled with the next unclaimed sources.
  • When other passes hold everything left, the page is empty with hasMore: false and heldByOtherPasses: true, and so is a later pass's page.
  • A failed pass's sources and an expired lease's sources are redelivered.
  • A pass that looked up and reviewed a source another running pass holds does not advance its cursor. The source is redelivered once the holder fails.
  • Two runDreamingAgentPass runs in one scope each settle only their own sources, release their leases, and only the first sees pending attention.
• dreaming-worker.test.ts:
  • Partitioning with per-scope pass counts.
  • With maxPassesPerScope: 3, one large scope runs three concurrent passes whose delivered sources never overlap, and every delivered source is settled once.
  • Default config keeps one pass per scope and rejects a second trigger.
  • A scope with more small sources than the backlog probe reads (60 short transcripts, far below tokenThreshold) gets three passes with maxPassesPerScope: 3.
  • A scope below tokenThreshold keeps one pass even when sharing is allowed.
  • A pass that joins after the attention owner finished takes the scope's pending attention over, while a pass joining a live owner does not.
• ontology-proposals.test.ts: a dated newer claim stays current when an undated claim from an older source is written after it.

After the rebase onto #2122's rebased head (776832220) and the two fixes below, I re-ran with SIGNET_SQLITE_PATH set to Homebrew SQLite: dreaming-evidence-leases, dreaming-worker, dreaming, ontology-proposals, episodic-sources (249 pass), core migrations (84 pass), scripts/doc-drift-packages and scripts/audit-event-loop-contract (25 pass), plus the other suites that call search_evidence (49 pass). Daemon tsc is clean, comments:check is current, and the event-loop baseline had no drift.

Not run: the BEAM 1M and LongMemEval acceptance runs from the issue. They need the benchmark harness and a live provider. The bench scripts set maxConcurrentPasses but not yet maxPassesPerScope.

Live verification. I ran daemons built from this branch before the rebase (head a0f55d027) and from main (ea8e8b569, the merge base), each with an isolated SIGNET_PATH/HOME. Dreaming inference went to a local OpenAI-compatible stub that drives each Pi-executor pass with a fixed script: search_evidence queue pages of 5, review_evidence for each page, one extra page left unreviewed, one create_entity, then stop. The stub held each reply for 1.5 s so passes overlapped. The corpus was 120 imported transcripts (about 1.6k chars each) in the default scope. No real providers were called.

• Default config (maxPassesPerScope unset, maxConcurrentPasses: 3): five concurrent POST /api/dream/trigger requests returned one 202 and four 409s, the same as on main. A 500 s trigger loop never had more than one pass running. GET /api/dream/status reports maxPassesPerScope: 1.
• maxPassesPerScope: 3, maxConcurrentPasses: 3, maxLlmConcurrency: 4: one trigger started three passes in the one scope, and further triggers got 409 until a slot freed up. In a 120 s loop, 27 passes ran, three at a time. I checked the stub log at millisecond precision: no source was delivered to a pass while another pass that already had it was still running (0 of 540 deliveries). A once-per-second sample of dreaming_evidence_leases showed up to three passes holding leases at once and never one source leased by two passes. Every pass finalized and no lease rows were left over. The cursor moved only for reviewed text: the unreviewed extra page of each pass stayed queued (stalled_passes 1) and went to a later pass.
• Drain time: on that head, the fix(dreaming): advance evidence cursor only by reviewed text #2122 bug fixed in 5f95f104b (now 0afbb2384, which this rebased branch includes) keeps re-delivering the tails of reviewed imported transcripts, so the queue never empties. To time a full drain, I ran a scratch copy of this head with only that fix applied. One pass at a time took 114 s to drain all 120 transcripts (8 passes). Three per scope took 41 s (9 passes). Both runs ended with every source consumed and none stalled.
• Failed pass: the stub returned HTTP 400 to one of three running passes after two pages. That pass was marked failed, its lease rows stopped counting as live, and all ten sources it had been given went to the two passes still running within 2 s.
• Attention: I inserted one pending contested_claim attention row while triggers kept three passes running. Only one running pass had it in its <pending_attention> at any time. Passes that joined while the owner was running saw none pending. The first pass to start after the owner finished took it over. Across four rounds the owners' run times never overlapped.
• Upgrade from main: I seeded a workspace with main (schema 168), ran passes there, then started this branch on the same workspace. Migrations 169 and 170 applied at boot with no errors. Three passes per scope then ran correctly on the upgraded data. With the fix(dreaming): advance evidence cursor only by reviewed text #2122 fix in place, they drained the 60 remaining transcripts and did not re-deliver the 60 that main had consumed.
• Logs: no errors besides the forced stub failure. No deadline overruns and no deadlocks. Every run shut down cleanly. The only warnings are the known ones, plus a Workspace migration is draining retry at shutdown that main logs too.
• Two findings, both fixed after this run:
  • Extra passes depended on the backlog probe reaching tokenThreshold, but the probe counts at most 50 sources. A scope whose backlog is many small sources never got a second pass: /api/memory/remember stores chunks of about 600 chars, so 900 pending memories got one pass with maxPassesPerScope: 3 (50 sources come to about 8k tokens, below the 10k minimum threshold). Fixed in 426eec844: a probe that hit the source cap with backlog now qualifies. With maxPassesPerScope at 1 nothing changes.
  • When sibling passes paged at the same moment, the pass that lost the lease race got items: [] with hasMore: true, and a model that reads an empty page as the end of the queue stopped early (my first stub script did). Fixed in 3c9b588c3: search_evidence re-drains and returns the next unclaimed sources. It returns an empty page only when other passes hold everything left, and then says so with hasMore: false and heldByOtherPasses: true.

AI disclosure

• No AI tools were used in this PR
• AI tools were used (see Assisted-by tags in commits)

Notes

Stacked on #2122. This branch is rebased onto #2122's current head (776832220, which is itself rebased onto main) and is based on fix/2094-dreaming-review-cursor because #2122 takes migration 169 and the migration registry requires consecutive versions. This PR uses 170 and must merge after #2122. If #2122 is squash-merged, this branch needs a rebase onto main.

How it interacts with #2122. #2122 moves the evidence cursor only by acknowledged (review_evidence) or cited text, and counts stalls for queue-delivered revisions with no progress.

• With leases, only the holder receives a source from the queue. A source another pass holds can reach this pass only through a sourceRef or query lookup, which fix(dreaming): advance evidence cursor only by reviewed text #2122 does not mark as queued, so it never counts as a stall here.
• Finalize adds sources held elsewhere to deferredEvidence. Neither this pass's review_evidence acknowledgements nor its citation floors can advance a cursor the holder owns. The holder's own review and citations advance it as usual.
• fix(dreaming): advance evidence cursor only by reviewed text #2122's resume overlap and stalled-last ordering apply unchanged to whatever the lease filter leaves in the queue.

Decisions made for the maintainer:

• New setting instead of reusing maxConcurrentPasses. The issue says a scope "gets up to maxConcurrentPasses passes", but the product default for that setting is 2. Reusing it would change behavior for every single-scope install. I added maxPassesPerScope (default 1), still capped by maxConcurrentPasses. Happy to fold it into maxConcurrentPasses if you'd rather turn this on by default.
• Threshold for extra passes. The issue says extra passes go to a scope "whose backlog exceeds one pass's page budget". There is no per-pass page budget on main, and the backlog probe stops counting at tokenThreshold, so I used "measured backlog reaches tokenThreshold". Because the probe stops after 50 sources, a probe that hits that cap with backlog also qualifies, so many small sources get extra passes too.
• Which passes may share a scope. Only plain incremental passes share scopes. Content passes (which stage the memory-head commit), hygiene, compact, and user-requested passes stay exclusive. The memory-head compare-and-swap therefore never sees two writers from one scope.
• Delivery order. The queue order is unchanged (newest first). Each pass leases whole queue pages, so its batch is consecutive in queue order. Correctness under interleaving relies on the evidence-time supersession rule, not on order.

Known limits:

• A pass joining a scope whose attention another running pass owns does not work that attention. It is a prompt-level split: a joining pass can still list attention through attention_list.
• Scheduled sweeps with a backlog pick incremental-content or incremental-hygiene, which are exclusive, so extra passes per scope come from POST /api/dream/trigger with mode: "incremental" (what the bench drain loop sends), not from the scheduler.
• Tools reached through /api/dream/tools/:capability (ACPX MCP targets) do not lease evidence. That route also does not persist tool calls, so queue progress for ACPX targets is already limited.
• Two passes can still both look up the same source by query. Only queue delivery and cursor advancement are partitioned.

Expected conflicts:

• fix(daemon): let workspaces select native sources #2121 (bug(dreaming): isolated workspaces inherit ambient sources; budget deferral may stall evidence pagination #1991) touches platform/daemon/src/memory-config.ts in different hunks and scripts/event-loop-contract-baseline.json. The baseline needs a regenerate on rebase.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: e57c1db1-6cc6-4b9a-a520-081892bfcf11

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}