Repository navigation
feat(batch): add opt-in dedicated projection drain pump - #24712
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
2 issues found across 12 files
Confidence score: 5/5
test_batch_pipeline_projection_pump.pyreloads production modules for every parametrized case, which makes the normal unit lane pay for repeated reloads. Marktest_owner_drain_module_capacity_from_envasslow.feature-flag-registry.mdstill says the owner-drain cap is always 8, which is inaccurate whenBATCH_PIPELINE_MAX_OWNER_DRAINSis set. Reword the row to say the cap comes from that setting.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="backend/tests/unit/test_batch_pipeline_projection_pump.py">
<violation number="1" location="backend/tests/unit/test_batch_pipeline_projection_pump.py:87">
P3: Mark `test_owner_drain_module_capacity_from_env` as `slow`; it performs fresh production-module reloads for every parametrized case, violating the test-suite contract and making the normal unit lane pay the repeated reload cost.</violation>
</file>
<file name="backend/docs/feature-flag-registry.md">
<violation number="1" location="backend/docs/feature-flag-registry.md:531">
P3: The `BATCH_PIPELINE_OWNER_DRAIN_ENABLED` row still hardcodes "up to 8 owners at once", which is now wrong whenever `BATCH_PIPELINE_MAX_OWNER_DRAINS` is set. Reword that row to say the cap comes from `BATCH_PIPELINE_MAX_OWNER_DRAINS` (default 8).</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| patch.delenv('BATCH_PIPELINE_MAX_OWNER_DRAINS', raising=False) | ||
| else: | ||
| patch.setenv('BATCH_PIPELINE_MAX_OWNER_DRAINS', raw) | ||
| importlib.reload(projection_drain) |
There was a problem hiding this comment.
P3: Mark test_owner_drain_module_capacity_from_env as slow; it performs fresh production-module reloads for every parametrized case, violating the test-suite contract and making the normal unit lane pay the repeated reload cost.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/tests/unit/test_batch_pipeline_projection_pump.py, line 87:
<comment>Mark `test_owner_drain_module_capacity_from_env` as `slow`; it performs fresh production-module reloads for every parametrized case, violating the test-suite contract and making the normal unit lane pay the repeated reload cost.</comment>
<file context>
@@ -0,0 +1,235 @@
+ patch.delenv('BATCH_PIPELINE_MAX_OWNER_DRAINS', raising=False)
+ else:
+ patch.setenv('BATCH_PIPELINE_MAX_OWNER_DRAINS', raw)
+ importlib.reload(projection_drain)
+ assert projection_drain.MAX_OWNER_DRAINS == expected
+ finally:
</file context>
| | `BATCH_PIPELINE_LANGUAGE_INHERITANCE_ENABLED` | Default on unless false; batch pieces inherit a known bound conversation language, else the user language, before per-piece LID | backend | env | open | — | — | — | — | keep | keep | — | dazheng | | ||
| | `BATCH_PIPELINE_LEADER_LEASE_ENABLED` | Default on unless false; one batch worker or serving replica per Firestore lease runs periodic recovery, drain, cleanup and quarantine discovery | backend | env | open | — | — | — | — | keep | keep | — | dazheng | | ||
| | `BATCH_PIPELINE_LONG_SPEECH_HANDOFF_ENABLED` | Default on unless false; a zero-word batch unit with >=5 s worker VAD speech whose intake STT language is unknown, multi or outside Parakeet TDT v3 skips the padded Parakeet retry and goes straight to quarantine recovery | backend | env | open | — | — | — | — | keep | keep | — | dazheng | | ||
| | `BATCH_PIPELINE_MAX_OWNER_DRAINS` | Default 8; concurrent owner drains per process, clamped to 1-32 at import; malformed keeps 8 | backend | env | closed | — | — | — | — | keep | keep | — | lead2 | |
There was a problem hiding this comment.
P3: The BATCH_PIPELINE_OWNER_DRAIN_ENABLED row still hardcodes "up to 8 owners at once", which is now wrong whenever BATCH_PIPELINE_MAX_OWNER_DRAINS is set. Reword that row to say the cap comes from BATCH_PIPELINE_MAX_OWNER_DRAINS (default 8).
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/docs/feature-flag-registry.md, line 531:
<comment>The `BATCH_PIPELINE_OWNER_DRAIN_ENABLED` row still hardcodes "up to 8 owners at once", which is now wrong whenever `BATCH_PIPELINE_MAX_OWNER_DRAINS` is set. Reword that row to say the cap comes from `BATCH_PIPELINE_MAX_OWNER_DRAINS` (default 8).</comment>
<file context>
@@ -528,6 +528,7 @@ and an explicit empty literal renders as `''`.
| `BATCH_PIPELINE_LANGUAGE_INHERITANCE_ENABLED` | Default on unless false; batch pieces inherit a known bound conversation language, else the user language, before per-piece LID | backend | env | open | — | — | — | — | keep | keep | — | dazheng |
| `BATCH_PIPELINE_LEADER_LEASE_ENABLED` | Default on unless false; one batch worker or serving replica per Firestore lease runs periodic recovery, drain, cleanup and quarantine discovery | backend | env | open | — | — | — | — | keep | keep | — | dazheng |
| `BATCH_PIPELINE_LONG_SPEECH_HANDOFF_ENABLED` | Default on unless false; a zero-word batch unit with >=5 s worker VAD speech whose intake STT language is unknown, multi or outside Parakeet TDT v3 skips the padded Parakeet retry and goes straight to quarantine recovery | backend | env | open | — | — | — | — | keep | keep | — | dazheng |
+| `BATCH_PIPELINE_MAX_OWNER_DRAINS` | Default 8; concurrent owner drains per process, clamped to 1-32 at import; malformed keeps 8 | backend | env | closed | — | — | — | — | keep | keep | — | lead2 |
| `BATCH_PIPELINE_MERGE_CATCH_UP_DEPTH` | Default 16; fresh batch uploads (capture ended at most 10 min before admission) commit ahead of the owner's pending catch-up rows, and catch-up commits wait while the owner already has this many committed rows awaiting projection | backend | env | closed | — | — | — | — | keep | keep | — | dazheng |
| `BATCH_PIPELINE_MERGE_HEAD_WAIT_SECONDS` | Default 60; once a batch result has waited this long behind an unready earlier-captured merge head, the owner's ready results commit in capture order and the head appends later as late_capture | backend | env | closed | — | — | — | — | keep | keep | — | dazheng |
</file context>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 85b564880f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if self.projector and serving_projection_sweep_enabled(): | ||
| self.projector.sweep() |
There was a problem hiding this comment.
With async projection enabled, cutting over to the documented pump/serving split can strand receipts after assignment. Fresh serving processes with BATCH_PIPELINE_SERVING_PROJECTION_SWEEP=off never run the sweep that sets has_async_followups, so reconcile_async() does not call start_followups() unless an unrelated owner wake supplies work. Meanwhile, the pump calls only sweep() and start_drains(), neither of which launches async follow-ups. Consequently, ordinary pump-persisted receipts can remain unfinished without client completion. Keep durable follow-up discovery running independently of the serving assignment-sweep gate, or explicitly hand this responsibility to the pump.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 36db876ba6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if projector.owes_drains: | ||
| with projector.handoff: | ||
| owners_requested = len(projector.drain_requests) | ||
| projector.start_drains() |
There was a problem hiding this comment.
When BATCH_PIPELINE_OWNER_DRAIN_ENABLED=false and deferred indexing remains enabled, Projector.sweep() walks owner chains itself and queues search/people-stats updates in the projector's in-memory index. The pump only starts owner drains; it never calls schedule_index_flush(), unlike Serving.reconcile_async(). Consequently, conversations assigned by these walks can remain absent or stale in search indefinitely, and serving instances cannot flush the pump's process-local queue. Schedule owed index work after each pump tick independently of whether owner drains are requested.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e1cc45d27b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| async def run(): | ||
| await asyncio.gather(supervise(), drain(), dispatch_recovery()) | ||
| if pump_enabled: | ||
| await projection_pump() |
There was a problem hiding this comment.
When the dedicated pump is monitored through /v2/batch-pipeline/metrics, selecting only this background loop does not keep the process projection-only. In backend/routers/batch_pipeline.py:265-270, the metrics handler still calls service.reconcile_async(periodic=True) unless read_only_metrics_enabled() succeeds; that helper requires BATCH_PIPELINE_SERVING_ENABLED=true, which the documented pump configuration explicitly leaves false/unset. Thus even enabling read-only metrics cannot prevent authenticated pump scrapes from competing for the serving maintenance lease and running cleanup, enrichment sweeps and queue recovery. Make the metrics handler recognize the pump role and refresh metrics without reconciliation.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 222610743d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| def serving_projection_sweep_enabled() -> bool: | ||
| """Only ``off`` disables serving-instance discovery; owner wakes still run.""" | ||
| from config.policy import kill_switch |
There was a problem hiding this comment.
Both serving_projection_sweep_enabled() and projection_pump_enabled() introduce function-local imports, violating the explicit requirement in backend/AGENTS.md: “All imports at module top level — never inside functions.” Import kill_switch once at module scope and reuse it in both accessors. Existing local imports elsewhere in this file do not exempt these newly added functions from the scoped convention.
AGENTS.md reference: backend/AGENTS.md:L168-L168
Useful? React with 👍 / 👎.
# Conflicts: # backend/config/flag_budget.json
## Problem Pusher drops its in-memory playback projection close obligation when the bounded close rebuild retries are exhausted or the socket is evicted. The retained audio remains safe, but no durable owner retries the stale playback metadata after the socket disappears. ## What changes - Persist coalesced `record_projection_rebuild_due` markers in `record_projection_rebuild_jobs`, keyed by user/conversation, on exhausted retries and socket cleanup. No conversation marker and no additional ordinary-flush reads or writes. - Queue terminal handoffs in one bounded metadata-only background task. Preserve a separate bounded overflow retry inventory, batch writes with one-second RPC deadlines, and cap all socket-cleanup marker work at three seconds. Marker failures never change recording completion. - Retire confirmed empty-inventory markers after seven days from Firestore creation, or under the 10,000-marker maintenance ceiling. Limit collection scans to 10,001 and rebuilds to 128; expose bounded count, observed oldest age, capacity state and `retired_no_objects`. Retirement rereads the marker version and the conversation producer map in one Firestore transaction. Only an explicit empty map permits deletion; live, unknown or malformed producer fences retain and rotate debt, including under collection pressure. The pressure-blocked gauge and aggregate fenced count surface passes that cannot evict empty markers. No audio is deleted. - Extract shared full metadata discovery and a socket-independent rebuild. A bounded pass in the existing record-seal-repair CronJob consumes at most 128 markers in 60 seconds, rotates work before discovery, and clears only the version it attempted. Failed, interrupted, invalid, or concurrently replaced work remains due; confirmed empty inventory follows the age/capacity policy above. A rebuild may update metadata while existing producers remain pending, but keeps its marker until those fences clear through their existing owner. - Refuse deleted owners before discovery and recheck deletion intent, privacy, and the prior audio-files revision transactionally before updating only `audio_files`. Keep preview read-only and retain debt on a broken raw-pack index. The runtime kill-switch parser lives in `config/batch_pipeline_projection_rebuild.py`; metadata rebuild stages do not read process policy from the environment. Flag: `PUSHER_PROJECTION_DEBT_DURABLE_ENABLED`, **default ON**. `0`, `false`, or `off` disables marker handoff and consumption. Rollback: set `recordSealRepair.projectionDebtDurableEnabled=false` in the repair runtime first, then disable pusher's flag; existing markers remain inert and can resume when reenabled. This PR does not unsuspend or activate a CronJob. ## Custody and privacy Playback metadata only. No seal authority, producer-token mutation, heartbeat, entry clock, dead-letter, conversation status, PCM read, or audio-object creation/rewriting/deletion. Tombstones, hard/soft deletion, account deletion, privacy changes, sync opt-out, peer projection updates, and marker replacement are fenced. Scope is redesign phase 5C only. #24712/#24716 and #23850 were checked before editing; #24712 and #24716 merged during verification and their flag metadata is preserved in the feature branch; shared overlap is flag metadata, not projection-drain behavior or deployment configuration. No `.github` changes. A persistent Firestore outage or process death before a marker is committed can still prevent handoff; this marker is not a WAL. Failed marker writes retain metadata-only retry inventory and emit sanitized `marker_write_failed` outcomes; bounded inventory saturation emits `marker_retry_overflow`. Cached close debt remains retryable even when overflow inventory is full. ## Verification - All tests/build checks ran through `~/workspace/omi/bin/omi-heavy` with the requested GCP credential/project variables unset. - Focused per-file backend runner: **314 passed** (projection producer/consumer, flag budget, rendered independent rollback, CLI output isolation, batch-stage environment ratchet and outbox fault injection). - State table/invariants/writer inventory plus other deployment contracts: **1,917 passed, 75 baseline xfailed**. - Lifecycle simulator plus lifecycle properties under the existing isolated stack environment and a disposable loopback Firestore emulator: **23 passed, 3 existing strict xfailed**. The shared-collection fixture includes a clear peer and verifies per-owner transactional outcomes. The exhaustion/eviction/deletion/marker-race cases and six empty-retirement cases passed: aged live/clear fences, pressure with fenced markers, post-discovery producer registration, marker replacement and late-tail repair. - One local composed full-stack attempt stopped at an HTTP read timeout in the existing REST-restart scenario before reaching lifecycle tests; CI passed that scenario on the same production source. The final exact-head CI gauntlet is linked below. - Runtime image dependency closure: **19 registered images passed**; no image/dependency changes needed. - Required Grok review via `agentctl`: Luna R1 and R2 findings fixed. The same required Grok reviewer rechecked transactional empty retirement, producer and marker races, pressure telemetry and regression coverage: **no blockers**. No additional reviewer agents. - `make preflight`: **44 checks passed**, including the registered Firestore guard. `scripts/pr-preflight` metadata contracts passed with the documented six-line router exception; the final frozen run passed all 44 checks; the normal hook repeated them successfully on the published head. - **Normal pre-push passed** on `95a9ea47ae98eb03843909459a6604523534a98e`: full backend type checking, selected backend units, registered Firestore guards, OpenAPI and pinned formatting. No bypass, cap override or force push; Firestore guard passed in 172.30 seconds without timeout or retry. Exact published head: `95a9ea47ae98eb03843909459a6604523534a98e`. CI results: [checks for this exact commit](https://github.com/BasedHardware/omi/commit/95a9ea47ae98eb03843909459a6604523534a98e/checks). ## Product invariants affected none ## Not measured No GCP/prod access or tier used; no live IAM/permission test, deployment, CronJob activation, production backlog/cost/latency measurement, device playback acceptance, or A/B worker-death/dead-letter simulator changes. Existing strict xfails remain existing simulator/state-table debt. Line-Count-Exception: backend/routers/pusher.py | 1851 -> 1857 | Phase 5C adds exact-owner terminal projection-debt and socket-cleanup handoffs at the existing boundaries; full discovery is extracted into a shared module, and splitting adjacent custody code would exceed this lane's named scope.
GPU batch holds had no bounded terminal disposition or automatic re-attempt. This adds a seven-day `held_deadline_at`, retained `dead(held_expired)` rows, a private-OIDC CAS requeue, a nightly UTC re-drive on the existing serving maintenance leader, and an agent-tier volume alert. No CronJob is added. Initial quarantine and failed-held recovery writers stamp a full window. Legacy rows receive seven days at their first eligible sweep. Expiry/requeue recheck content ownership, privacy generation, dormant held state, no result or result receipt, no worker token and an expired finite lease. Active paid/operator intents block the transition. Durable document-key cursors bound each pass to 32 rows per five-minute tick; crashes replay idempotently and malformed rows cannot pin the page. Nightly re-drive rechecks its previous-day cutoff inside the transaction before normal GPU admission. `BATCH_HELD_DEAD_LETTER_ENABLED` / `batch_held_dead_letter_enabled()` defaults **ON**. It is registered in kill switches, flag budget, feature registry, deployment classification, common runtime declarations and worker overlays. **Rollback:** false on serving/workers stops new stamps and automatic maintenance; the private requeue endpoint remains usable. Expiry atomically sets `merge_pending=true`, so after reverting the new code an operator can still use the unchanged `POST /v2/batch-pipeline/redrive-dead` route. No pre-revert drain or data rewrite is required for rows written by this implementation. The rolled-back service requires an operator re-drive rather than the new nightly helper. See `backend/docs/runbooks/batch-held-dead-letter.md`. **Custody:** nothing is deleted. Every new dead letter has reason `held_expired`, retains its deadline, encrypted audio references and attempt/intent history, and has a sanctioned requeue path. No TTL, conversation-status writer or custody/privacy fence is changed. Status and partial-accounting readers retain the held answer, so age does not acknowledge phone custody or hide healthy siblings. Projection expiry (4b), BYOK/finalization tools (4c), and `privacy_blocked` remain outside this change. Provider-STT holds remain fenced from the GPU queue; the state-table deadline gap remains explicitly strict-xfailed. The volume alert is **agent tier**, with `omi-tier-silent` routing so it never pages humans. It checks fresh retained stock >100 or >10 expirations in 30 minutes for five minutes. Each stock sample must be younger than 15 minutes before aggregation; unrelated legacy dead reasons do not count. Notification timing follows the warning contract. Split/combined rules, metric inventories, the sync GMP scrape allowlist and all four dev/prod main/sync exporter prefix lists include the three emitted metric families. Thresholds are initial operator defaults, not production-calibrated. Backend and metrics-egress/alert rollout must use the checked-in declarations; this PR performs none of those rollouts. Validation (offline/synthetic; suites overlap): - Final held-recovery, alert, notification, inventory and transport contracts: 319 passed. - Real loopback Firestore integration: 3 passed, including expiry → flag off → unchanged legacy dead re-drive → normal completion/merge, contention and duplicate delivery. - Main-sync serving/flag/held/worker contracts: 135 passed. - State-table/writer/policy validation retained 75 strict expected gaps; no unrelated xfails were removed. The shared emulator regressions are also registered in the lifecycle harness. - `make preflight` and `scripts/pr-preflight`: 52 checks passed. Both published fix and merge heads passed the normal pre-push hook, including backend tests, type checks, OpenAPI and Firestore guard; no bypass, cap override, force or timeout retry. - Required Grok review completed with NO BLOCKERS (receipts `exec-heavy-system-squeeze-major-install-point`, `exec-breeze-portion-boy-castle-gauge-marine`, `exec-blur-source-live-meadow-inject-damp`). The last Luna P1 rollback finding is fixed inline with the atomic legacy marker and explicit rollback regressions; no further review round was requested. - [Exact-head GitHub checks](https://github.com/BasedHardware/omi/pull/24736/checks?sha=30b59b4c95631168a4a5fac9d7807cadd747327c) for `30b59b4c95631168a4a5fac9d7807cadd747327c`. Main is merged through `d6112658e5fa`, retaining #24735's cadence controls and #24733's 16-worker seal-repair rollback. The lane base already includes #24712/#24716. No `.github` edits; unrelated Flutter work is preserved. Not measured: production stock/dwell, vendor spend, live nightly execution, live exporter/alert delivery, deployed revision or user-visible recovery. No GCP tier, production access, merge to main or deployment was used. Product invariant: INV-MEM-4; canonical memory intake/promotion authority is unchanged. Failure-Class: FC-batch-fault-domain-head-of-line-poison
Problem
Projection sweep supply is tied to serving instance count. Moving discovery onto dedicated instances needs an independent loop that cannot compete for maintenance/recovery leadership. This implements David's 2026-11-05 pump plan, sections 2 and 4; deployment shape remains pending.
Fix
serving_projection_sweep_enabled()using the shared kill-switch helper.BATCH_PIPELINE_SERVING_PROJECTION_SWEEPis on unless explicitlyoff; it gates onlyServing.reconcile()'s sweep, preserving wakes, held drains, metrics and maintenance/recovery.BATCH_PIPELINE_PROJECTION_PUMP_ENABLED=trueselects one lean projection loop instead of the three serving loops, independently ofBATCH_PIPELINE_SERVING_ENABLED. It never invokes reconcile, maintenance or recovery dispatch. Sweeps userun_blocking(sync_executor, ...); owed drains start on the event loop. After each sweep, it schedules discovered or owed projection follow-ups viastart_followups(())and the owed index flush on the event loop, before starting owner drains after all pages. Each completed tick, including a failed tick, updatesomi_batch_projection_pump_last_tick_seconds(Unix timestamp, zero before first tick). It logs successful sweeps, owners requested and elapsed time; errors log and retry.All unset defaults preserve today's behavior.
projection_drain.pychanges only its capacity constant and the config import. No deployment files or.github/**changes.Validation
Code and hermetic tests only; no production access or GCP tier used.
Merged
origin/mainat148bf4c1c57073dba4ec9a750415f2c72f5adb87using a merge commit, without rebase. Six files had conflicts: batch config, flag budget, kill-switch helper, pipeline contract docs, policy tests and projection drain. Both #24710 claim-aware behavior and pump capacity accessor are retained. The drain implementation is byte-for-byte main after removing only the capacity accessor import and restoring the old constant. Main advanced during CI; a second merge includesf8edd96cdbe(#24716). Its four conflicts in flag budget, kill-switch helper, rendered registry and policy tests retain the new striping controls alongside the pump controls.bin/omi-heavy run -- env BACKEND_UNIT_TEST_FILE_LIST=<file> bash backend/test.sh:Additional post-merge suites: projection discovery 67, follow-up desync 36, follow-up reads 6, sweep owner chain 34, and timeliness 18 passed.
Total: 673 passed, fifteen isolated files (plus the same 2 slow simulations deselected). Registry validation passed; the env-ratchet baseline is unchanged. The enforced push gate passed 41 checks; pyright reported 0 errors; selected push-time unit tests passed.
The requested merge reconciliation and metrics-route P1 are included. The listed P2 follow-ups are outside this change; no new review round was started.
Risk
Enabling a pump adds discovery work and shared-executor load. Bounds are pages 1-4, tick 2-60 seconds and owner drains 1-16; defaults remain 1, 20 and 8. The pump stays disabled by default. Durable receipts, owner claims and fencing are unchanged.
Ramp plan
Owner: lead2, after David's deployment-shape decision. Studio safe start on a dedicated service:
BATCH_PIPELINE_PUMP_SWEEP_PAGES=1.BATCH_PIPELINE_PUMP_TICK_SECONDS=2.BATCH_PIPELINE_MAX_OWNER_DRAINS=8per instance.Never set
BATCH_PIPELINE_PROJECTION_PUMP_ENABLED=trueon backend-integration itself: the toggle is service-wide. KeepBATCH_PIPELINE_SERVING_ENABLEDfalse/unset on the dedicated pump service.Cutover gate: owner_drain advanced >=20 rows/s for 30 minutes before disabling serving sweeps. Install and verify a missing/stale alert on the dedicated service's
omi_batch_projection_pump_last_tick_secondsbeforehand, including never-observed zero and missing instance series; a stale/missing alert must prevent cutover. Fallback: serving sweeps stay ON until both the throughput gate and heartbeat-alert prerequisite pass.Alarm if Firestore read rate rises more than 50% over pre-cutover. Retain the owner_drain rows/s floor of 70% of pre-cutover; watch sweep elapsed time and executor saturation. Evaluate claim_busy share (no higher than pre-cutover) after serving sweeps are switched off, not during dual-running. This PR authorizes no deployment or cutover.
ROLLBACK
Env only, through new revisions, in this order:
BATCH_PIPELINE_SERVING_PROJECTION_SWEEP=on).BATCH_PIPELINE_PROJECTION_PUMP_ENABLED=falseon the dedicated service.BATCH_PIPELINE_MAX_OWNER_DRAINS=8.Failure-Class: none
These are pre-merge qualification corrections to the newly introduced pump role; no existing semantic failure-class definition matches the missing follow-up scheduling, pump-specific capacity bounds, and metrics-route role isolation.