fix(batch): stripe projection discovery startup cursors by Git-on-my-level · Pull Request #24716 · BasedHardware/omi

Git-on-my-level · 2026-11-05T17:30:00Z

Problem

Projection discovery instances begin at the same keyset head after a roll, repeatedly offering the same owners to the advisory claim. Revision, hostname and PID alone may be identical across Cloud Run replicas.

Fix

Select a random 128-bit process seed on first use, mix it with the revision, and cache the resulting SHA-256 stripe modulo 4096 for the process lifetime. A local lock makes first-use initialization and logging occur once even with concurrent scanners. No metadata server or network I/O is used. The larger stripe space reduces collisions at 30–50 instances; restarts may choose a different stripe. Log the chosen stripe once in the process selector, rather than for every scanner.

Seed each scanner at its committed_at lower bound between the oldest matching result and startup time. Existing committed_at + document ID ordering remains oldest-first after that bound, then wraps to the head. Every row remains discoverable; no durable state changes. The existing follow-up seed takes precedence. Page sizes, sweep cadence, owner-drain concurrency and owner claims are unchanged.

Tests

314 passed across 8 files using BACKEND_UNIT_TEST_FILE_LIST=<file> bash backend/test.sh through bin/omi-heavy run --: projection discovery, batch-stage environment ratchet, flag budget, unit policy, follow-up desync, follow-up reads, sweep owner-chain and owner-drain tests. Injected seeds model separate processes with identical metadata and verify process-stable selection, different stripes and one log per process. Full-cycle coverage covers stripe boundaries through 4095, timestamp ties, wrap and empty/short collections.

B1 from the supplied correctness review is fixed. The insert-behind-cursor follow-up remains outside this change.

Risk

This is discovery desynchronization, not exclusive partitioning. Hash collisions and sparse/skewed timestamp ranges can still share starting rows. Startup adds one scoped head read per scanner; a seed beyond the newest result retries the head immediately. Throughput and claim-busy impact require a separately authorized production canary. No production access, merge or deployment in this lane.

Rollback

The switch defaults on. Set BATCH_PIPELINE_PROJECTION_STRIPING_ENABLED=off and restart scanners to recover the exact pre-change cursor sequence and query budget. Pagination off also bypasses seeding.

Failure-Class: none
This is cross-instance discovery contention; the blocked-page class covers dependency starvation and the scheduler-shard class covers shared control mutations. Transaction fences remain unchanged.

chatgpt-codex-connector · 2026-11-05T17:30:10Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-05T18:14:28.342948Z	`125ae15`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

cubic-dev-ai

3 issues found across 12 files

Confidence score: 4/5

test_flag_budget.py accepts arbitrary strings beginning with Default on, weakening the registry guardrail. Preserve the delimiter when allowing the new wording.
projection_discovery.py repeats the same empty query when the head read returns no rows. Add head is not None to the fallback guard.
batch_pipeline.py overstates how often 256 buckets avoid collisions: the chance of a shared start point is about 83% at 30 instances and 99% at 50. Reword the docstring to reflect those figures.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="backend/tests/unit/test_flag_budget.py">

<violation number="1" location="backend/tests/unit/test_flag_budget.py:224">
P3: This weakens the registry guardrail by accepting arbitrary strings beginning with `Default on`, not just a valid default-on status. Preserve the delimiter while allowing the new wording with `startswith(('Default on;', 'Default on unless off;'))`.</violation>
</file>

<file name="backend/config/batch_pipeline.py">

<violation number="1" location="backend/config/batch_pipeline.py:530">
P3: The docstring claim that 256 buckets keep collisions low for 30-50 instances is inaccurate. At 30 instances a shared start point is about 83% likely, and at 50 it is about 99%. Reword it to state the actual collision odds so the constant's rationale is not misread.</violation>
</file>

<file name="backend/utils/batch_pipeline/projection_discovery.py">

<violation number="1" location="backend/utils/batch_pipeline/projection_discovery.py:48">
P3: When the head read finds no rows, `after` is still None, so the fallback below repeats the same empty query. Add `head is not None` to the fallback guard to skip that extra read.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai · 2026-11-05T17:34:00Z

        assert 'backend' in flag['surfaces']
        assert flag['fail'] == 'open'
-        assert flag['summary'].startswith('Default on;')
+        assert flag['summary'].startswith('Default on')


P3: This weakens the registry guardrail by accepting arbitrary strings beginning with Default on, not just a valid default-on status. Preserve the delimiter while allowing the new wording with startswith(('Default on;', 'Default on unless off;')).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/tests/unit/test_flag_budget.py, line 224: <comment>This weakens the registry guardrail by accepting arbitrary strings beginning with `Default on`, not just a valid default-on status. Preserve the delimiter while allowing the new wording with `startswith(('Default on;', 'Default on unless off;'))`.</comment> <file context> @@ -212,14 +213,15 @@ def test_helper_kill_switch_budget_entries_match_the_default_on_registry(): assert 'backend' in flag['surfaces'] assert flag['fail'] == 'open' - assert flag['summary'].startswith('Default on;') + assert flag['summary'].startswith('Default on') </file context>

Suggested change

assert flag['summary'].startswith('Default on')

assert flag['summary'].startswith(('Default on;', 'Default on unless off;'))

cubic-dev-ai · 2026-11-05T17:34:01Z

+
+
+def projection_discovery_stripe() -> int:
+    """Stable startup phase; 256 buckets keep collisions low for 30-50 instances.


P3: The docstring claim that 256 buckets keep collisions low for 30-50 instances is inaccurate. At 30 instances a shared start point is about 83% likely, and at 50 it is about 99%. Reword it to state the actual collision odds so the constant's rationale is not misread.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/config/batch_pipeline.py, line 530: <comment>The docstring claim that 256 buckets keep collisions low for 30-50 instances is inaccurate. At 30 instances a shared start point is about 83% likely, and at 50 it is about 99%. Reword it to state the actual collision odds so the constant's rationale is not misread.</comment> <file context> @@ -515,6 +517,25 @@ def projection_pagination_enabled() -> bool: + + +def projection_discovery_stripe() -> int: + """Stable startup phase; 256 buckets keep collisions low for 30-50 instances. + + Revision separates overlapping rolls; hostname separates instances and pid </file context>

Suggested change

"""Stable startup phase; 256 buckets keep collisions low for 30-50 instances.

"""Stable startup phase; 256 buckets spread starts across instances but collide often (~83% chance of any shared start at 30 instances).

cubic-dev-ai · 2026-11-05T17:34:01Z

+                # A timestamp-only cursor includes every tie at the lower bound.
+                after = (oldest + max(0.0, time.time() - oldest) * self.stripe / PROJECTION_DISCOVERY_STRIPES, '')
+        segments, after = self.store.projection_page(limit=limit, after=after, **scope)
+        if not segments and seeding:


P3: When the head read finds no rows, after is still None, so the fallback below repeats the same empty query. Add head is not None to the fallback guard to skip that extra read.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/utils/batch_pipeline/projection_discovery.py, line 48: <comment>When the head read finds no rows, `after` is still None, so the fallback below repeats the same empty query. Add `head is not None` to the fallback guard to skip that extra read.</comment> <file context> @@ -9,15 +19,36 @@ def __init__(self, store, *, states: tuple[str, ...] | None = None): + # A timestamp-only cursor includes every tie at the lower bound. + after = (oldest + max(0.0, time.time() - oldest) * self.stripe / PROJECTION_DISCOVERY_STRIPES, '') + segments, after = self.store.projection_page(limit=limit, after=after, **scope) + if not segments and seeding: + # Sparse/short collections or clock skew: do not waste a sweep tick. + segments, after = self.store.projection_page(limit=limit, after=None, **scope) </file context>

Suggested change

if not segments and seeding:

if not segments and seeding and head is not None:

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d638253c4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-05T17:35:50Z



+def projection_striping_enabled() -> bool:
+    from config.policy import kill_switch


Move the new policy import to module scope

The new projection_striping_enabled() helper imports kill_switch inside the function, violating the backend guide's explicit requirement that all imports reside at module scope. Move this dependency to the module-level imports rather than adding another function-local import.

AGENTS.md reference: backend/AGENTS.md:L166-L168

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-11-05T17:35:50Z

+                after = (oldest + max(0.0, time.time() - oldest) * self.stripe / PROJECTION_DISCOVERY_STRIPES, '')
+        segments, after = self.store.projection_page(limit=limit, after=after, **scope)


Bound the seeded pass before following newly appended rows

If matching results keep arriving fast enough that discovery pages remain full, a nonzero startup stripe can indefinitely skip every pending obligation before its seed. projection_page() only resets its cursor after a short page, and the assignment scanner has no timestamp bound equivalent to ProjectionFollowups.end_pass_past_mark(), so it can follow the growing tail forever without performing the promised wrap. Previously, head-first startup at least offered those oldest obligations before following the tail. Capture a startup pass boundary and force a wrap after reaching it, including timestamp ties, so pre-seed work is discovered even under sustained arrivals.

Useful? React with 👍 / 👎.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 125ae150c1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-05T18:14:25Z

+def projection_striping_enabled() -> bool:
+    from config.policy import kill_switch
+
+    return kill_switch('BATCH_PIPELINE_PROJECTION_STRIPING_ENABLED')


Document the projection discovery rollout in the pipeline guide

This changes conversation-lifecycle recovery and introduces a default-on environment switch, but the commit adds neither the required change note under backend/docs/listen_pusher_pipeline/changes/ nor the environment contract update in backend/docs/listen_pusher_pipeline.mdx. Searching the documentation for the new switch finds only the generated feature-flag registry. Add the pipeline note with rollout, rollback and verification evidence, and document the switch's default and off-only disabling semantics in the contract section, as required by backend/AGENTS.md.

AGENTS.md reference: backend/AGENTS.md:L164-L164

Useful? React with 👍 / 👎.

## Problem Projection sweep supply is tied to serving instance count. Moving discovery onto dedicated instances needs an independent loop that cannot compete for maintenance/recovery leadership. This implements David's 2026-11-05 pump plan, sections 2 and 4; deployment shape remains pending. ## Fix - Add `serving_projection_sweep_enabled()` using the shared kill-switch helper. `BATCH_PIPELINE_SERVING_PROJECTION_SWEEP` is on unless explicitly `off`; it gates only `Serving.reconcile()`'s sweep, preserving wakes, held drains, metrics and maintenance/recovery. - `BATCH_PIPELINE_PROJECTION_PUMP_ENABLED=true` selects one lean projection loop instead of the three serving loops, independently of `BATCH_PIPELINE_SERVING_ENABLED`. It never invokes reconcile, maintenance or recovery dispatch. Sweeps use `run_blocking(sync_executor, ...)`; owed drains start on the event loop. After each sweep, it schedules discovered or owed projection follow-ups via `start_followups(())` and the owed index flush on the event loop, before starting owner drains after all pages. Each completed tick, including a failed tick, updates `omi_batch_projection_pump_last_tick_seconds` (Unix timestamp, zero before first tick). It logs successful sweeps, owners requested and elapsed time; errors log and retry. - Pump metrics scrapes use refresh-only regardless of the serving/read-only metrics controls; they never invoke the reconcile coordinator on the configured pump role. - Bound pages/tick (default 1, range 1-4), sleep (default 20s, range 2-60s) and owner drains (default 8, range 1-16); malformed values retain defaults. Register all five env names, and regenerate the registry. All unset defaults preserve today's behavior. `projection_drain.py` changes only its capacity constant and the config import. No deployment files or `.github/**` changes. ## Validation Code and hermetic tests only; no production access or GCP tier used. Merged `origin/main` at `148bf4c1c57073dba4ec9a750415f2c72f5adb87` using a merge commit, without rebase. Six files had conflicts: batch config, flag budget, kill-switch helper, pipeline contract docs, policy tests and projection drain. Both #24710 claim-aware behavior and pump capacity accessor are retained. The drain implementation is byte-for-byte main after removing only the capacity accessor import and restoring the old constant. Main advanced during CI; a second merge includes `f8edd96cdbe` (#24716). Its four conflicts in flag budget, kill-switch helper, rendered registry and policy tests retain the new striping controls alongside the pump controls. `bin/omi-heavy run -- env BACKEND_UNIT_TEST_FILE_LIST=<file> bash backend/test.sh`: - Pump: **92 passed**. - Serving (including four pump metrics-route combinations): **61 passed**. - Projection follow-ups: **36 passed**. - Owner drain / claim-aware admission: **66 passed**. - Metrics: **32 passed**. - GMP sidecar: **117 passed**. - Policy: **76 passed**. - Projection simulation: **1 passed, 2 slow tests deselected** by the standard runner filter. - Stage env ratchet: **7 passed**. - Flag budget: **24 passed**. Additional post-merge suites: projection discovery **67**, follow-up desync **36**, follow-up reads **6**, sweep owner chain **34**, and timeliness **18** passed. Total: **673 passed**, fifteen isolated files (plus the same 2 slow simulations deselected). Registry validation passed; the env-ratchet baseline is unchanged. The enforced push gate passed **41 checks**; pyright reported **0 errors**; selected push-time unit tests passed. The requested merge reconciliation and metrics-route P1 are included. The listed P2 follow-ups are outside this change; no new review round was started. ## Risk Enabling a pump adds discovery work and shared-executor load. Bounds are pages 1-4, tick 2-60 seconds and owner drains 1-16; defaults remain 1, 20 and 8. The pump stays disabled by default. Durable receipts, owner claims and fencing are unchanged. ## Ramp plan Owner: lead2, after David's deployment-shape decision. Studio safe start on a **dedicated service**: - **4 instances**. - `BATCH_PIPELINE_PUMP_SWEEP_PAGES=1`. - `BATCH_PIPELINE_PUMP_TICK_SECONDS=2`. - `BATCH_PIPELINE_MAX_OWNER_DRAINS=8` per instance. **Never set `BATCH_PIPELINE_PROJECTION_PUMP_ENABLED=true` on backend-integration itself: the toggle is service-wide.** Keep `BATCH_PIPELINE_SERVING_ENABLED` false/unset on the dedicated pump service. **Cutover gate:** owner_drain advanced **>=20 rows/s for 30 minutes** before disabling serving sweeps. Install and verify a missing/stale alert on the dedicated service's `omi_batch_projection_pump_last_tick_seconds` beforehand, including never-observed zero and missing instance series; a stale/missing alert must prevent cutover. Fallback: serving sweeps stay **ON** until both the throughput gate and heartbeat-alert prerequisite pass. Alarm if Firestore read rate rises more than 50% over pre-cutover. Retain the owner_drain rows/s floor of 70% of pre-cutover; watch sweep elapsed time and executor saturation. Evaluate claim_busy share (no higher than pre-cutover) **after serving sweeps are switched off, not during dual-running**. This PR authorizes no deployment or cutover. ## ROLLBACK Env only, through new revisions, in this order: 1. Restore serving sweeps **ON on the existing service first** (`BATCH_PIPELINE_SERVING_PROJECTION_SWEEP=on`). 2. Set `BATCH_PIPELINE_PROJECTION_PUMP_ENABLED=false` on the dedicated service. 3. Restore `BATCH_PIPELINE_MAX_OWNER_DRAINS=8`. Failure-Class: none These are pre-merge qualification corrections to the newly introduced pump role; no existing semantic failure-class definition matches the missing follow-up scheduling, pump-specific capacity bounds, and metrics-route role isolation.  <a href="https://cubic.dev/pr/BasedHardware/omi/pull/24712?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="View guided diff" src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a> <a href="https://www.cubic.dev/action/auto-fix/pr/BasedHardware/omi/24712?returnTo=https%3A%2F%2Fgithub.com%2FBasedHardware%2Fomi%2Fpull%2F24712&source=description" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img alt="Turn on auto-fix" src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>

## Problem Pusher drops its in-memory playback projection close obligation when the bounded close rebuild retries are exhausted or the socket is evicted. The retained audio remains safe, but no durable owner retries the stale playback metadata after the socket disappears. ## What changes - Persist coalesced `record_projection_rebuild_due` markers in `record_projection_rebuild_jobs`, keyed by user/conversation, on exhausted retries and socket cleanup. No conversation marker and no additional ordinary-flush reads or writes. - Queue terminal handoffs in one bounded metadata-only background task. Preserve a separate bounded overflow retry inventory, batch writes with one-second RPC deadlines, and cap all socket-cleanup marker work at three seconds. Marker failures never change recording completion. - Retire confirmed empty-inventory markers after seven days from Firestore creation, or under the 10,000-marker maintenance ceiling. Limit collection scans to 10,001 and rebuilds to 128; expose bounded count, observed oldest age, capacity state and `retired_no_objects`. Retirement rereads the marker version and the conversation producer map in one Firestore transaction. Only an explicit empty map permits deletion; live, unknown or malformed producer fences retain and rotate debt, including under collection pressure. The pressure-blocked gauge and aggregate fenced count surface passes that cannot evict empty markers. No audio is deleted. - Extract shared full metadata discovery and a socket-independent rebuild. A bounded pass in the existing record-seal-repair CronJob consumes at most 128 markers in 60 seconds, rotates work before discovery, and clears only the version it attempted. Failed, interrupted, invalid, or concurrently replaced work remains due; confirmed empty inventory follows the age/capacity policy above. A rebuild may update metadata while existing producers remain pending, but keeps its marker until those fences clear through their existing owner. - Refuse deleted owners before discovery and recheck deletion intent, privacy, and the prior audio-files revision transactionally before updating only `audio_files`. Keep preview read-only and retain debt on a broken raw-pack index. The runtime kill-switch parser lives in `config/batch_pipeline_projection_rebuild.py`; metadata rebuild stages do not read process policy from the environment. Flag: `PUSHER_PROJECTION_DEBT_DURABLE_ENABLED`, **default ON**. `0`, `false`, or `off` disables marker handoff and consumption. Rollback: set `recordSealRepair.projectionDebtDurableEnabled=false` in the repair runtime first, then disable pusher's flag; existing markers remain inert and can resume when reenabled. This PR does not unsuspend or activate a CronJob. ## Custody and privacy Playback metadata only. No seal authority, producer-token mutation, heartbeat, entry clock, dead-letter, conversation status, PCM read, or audio-object creation/rewriting/deletion. Tombstones, hard/soft deletion, account deletion, privacy changes, sync opt-out, peer projection updates, and marker replacement are fenced. Scope is redesign phase 5C only. #24712/#24716 and #23850 were checked before editing; #24712 and #24716 merged during verification and their flag metadata is preserved in the feature branch; shared overlap is flag metadata, not projection-drain behavior or deployment configuration. No `.github` changes. A persistent Firestore outage or process death before a marker is committed can still prevent handoff; this marker is not a WAL. Failed marker writes retain metadata-only retry inventory and emit sanitized `marker_write_failed` outcomes; bounded inventory saturation emits `marker_retry_overflow`. Cached close debt remains retryable even when overflow inventory is full. ## Verification - All tests/build checks ran through `~/workspace/omi/bin/omi-heavy` with the requested GCP credential/project variables unset. - Focused per-file backend runner: **314 passed** (projection producer/consumer, flag budget, rendered independent rollback, CLI output isolation, batch-stage environment ratchet and outbox fault injection). - State table/invariants/writer inventory plus other deployment contracts: **1,917 passed, 75 baseline xfailed**. - Lifecycle simulator plus lifecycle properties under the existing isolated stack environment and a disposable loopback Firestore emulator: **23 passed, 3 existing strict xfailed**. The shared-collection fixture includes a clear peer and verifies per-owner transactional outcomes. The exhaustion/eviction/deletion/marker-race cases and six empty-retirement cases passed: aged live/clear fences, pressure with fenced markers, post-discovery producer registration, marker replacement and late-tail repair. - One local composed full-stack attempt stopped at an HTTP read timeout in the existing REST-restart scenario before reaching lifecycle tests; CI passed that scenario on the same production source. The final exact-head CI gauntlet is linked below. - Runtime image dependency closure: **19 registered images passed**; no image/dependency changes needed. - Required Grok review via `agentctl`: Luna R1 and R2 findings fixed. The same required Grok reviewer rechecked transactional empty retirement, producer and marker races, pressure telemetry and regression coverage: **no blockers**. No additional reviewer agents. - `make preflight`: **44 checks passed**, including the registered Firestore guard. `scripts/pr-preflight` metadata contracts passed with the documented six-line router exception; the final frozen run passed all 44 checks; the normal hook repeated them successfully on the published head. - **Normal pre-push passed** on `95a9ea47ae98eb03843909459a6604523534a98e`: full backend type checking, selected backend units, registered Firestore guards, OpenAPI and pinned formatting. No bypass, cap override or force push; Firestore guard passed in 172.30 seconds without timeout or retry. Exact published head: `95a9ea47ae98eb03843909459a6604523534a98e`. CI results: [checks for this exact commit](https://github.com/BasedHardware/omi/commit/95a9ea47ae98eb03843909459a6604523534a98e/checks). ## Product invariants affected none ## Not measured No GCP/prod access or tier used; no live IAM/permission test, deployment, CronJob activation, production backlog/cost/latency measurement, device playback acceptance, or A/B worker-death/dead-letter simulator changes. Existing strict xfails remain existing simulator/state-table debt. Line-Count-Exception: backend/routers/pusher.py | 1851 -> 1857 | Phase 5C adds exact-owner terminal projection-debt and socket-cleanup handoffs at the existing boundaries; full discovery is extracted into a shared module, and splitting adjacent custody code would exceed this lane's named scope.

GPU batch holds had no bounded terminal disposition or automatic re-attempt. This adds a seven-day `held_deadline_at`, retained `dead(held_expired)` rows, a private-OIDC CAS requeue, a nightly UTC re-drive on the existing serving maintenance leader, and an agent-tier volume alert. No CronJob is added. Initial quarantine and failed-held recovery writers stamp a full window. Legacy rows receive seven days at their first eligible sweep. Expiry/requeue recheck content ownership, privacy generation, dormant held state, no result or result receipt, no worker token and an expired finite lease. Active paid/operator intents block the transition. Durable document-key cursors bound each pass to 32 rows per five-minute tick; crashes replay idempotently and malformed rows cannot pin the page. Nightly re-drive rechecks its previous-day cutoff inside the transaction before normal GPU admission. `BATCH_HELD_DEAD_LETTER_ENABLED` / `batch_held_dead_letter_enabled()` defaults **ON**. It is registered in kill switches, flag budget, feature registry, deployment classification, common runtime declarations and worker overlays. **Rollback:** false on serving/workers stops new stamps and automatic maintenance; the private requeue endpoint remains usable. Expiry atomically sets `merge_pending=true`, so after reverting the new code an operator can still use the unchanged `POST /v2/batch-pipeline/redrive-dead` route. No pre-revert drain or data rewrite is required for rows written by this implementation. The rolled-back service requires an operator re-drive rather than the new nightly helper. See `backend/docs/runbooks/batch-held-dead-letter.md`. **Custody:** nothing is deleted. Every new dead letter has reason `held_expired`, retains its deadline, encrypted audio references and attempt/intent history, and has a sanctioned requeue path. No TTL, conversation-status writer or custody/privacy fence is changed. Status and partial-accounting readers retain the held answer, so age does not acknowledge phone custody or hide healthy siblings. Projection expiry (4b), BYOK/finalization tools (4c), and `privacy_blocked` remain outside this change. Provider-STT holds remain fenced from the GPU queue; the state-table deadline gap remains explicitly strict-xfailed. The volume alert is **agent tier**, with `omi-tier-silent` routing so it never pages humans. It checks fresh retained stock >100 or >10 expirations in 30 minutes for five minutes. Each stock sample must be younger than 15 minutes before aggregation; unrelated legacy dead reasons do not count. Notification timing follows the warning contract. Split/combined rules, metric inventories, the sync GMP scrape allowlist and all four dev/prod main/sync exporter prefix lists include the three emitted metric families. Thresholds are initial operator defaults, not production-calibrated. Backend and metrics-egress/alert rollout must use the checked-in declarations; this PR performs none of those rollouts. Validation (offline/synthetic; suites overlap): - Final held-recovery, alert, notification, inventory and transport contracts: 319 passed. - Real loopback Firestore integration: 3 passed, including expiry → flag off → unchanged legacy dead re-drive → normal completion/merge, contention and duplicate delivery. - Main-sync serving/flag/held/worker contracts: 135 passed. - State-table/writer/policy validation retained 75 strict expected gaps; no unrelated xfails were removed. The shared emulator regressions are also registered in the lifecycle harness. - `make preflight` and `scripts/pr-preflight`: 52 checks passed. Both published fix and merge heads passed the normal pre-push hook, including backend tests, type checks, OpenAPI and Firestore guard; no bypass, cap override, force or timeout retry. - Required Grok review completed with NO BLOCKERS (receipts `exec-heavy-system-squeeze-major-install-point`, `exec-breeze-portion-boy-castle-gauge-marine`, `exec-blur-source-live-meadow-inject-damp`). The last Luna P1 rollback finding is fixed inline with the atomic legacy marker and explicit rollback regressions; no further review round was requested. - [Exact-head GitHub checks](https://github.com/BasedHardware/omi/pull/24736/checks?sha=30b59b4c95631168a4a5fac9d7807cadd747327c) for `30b59b4c95631168a4a5fac9d7807cadd747327c`. Main is merged through `d6112658e5fa`, retaining #24735's cadence controls and #24733's 16-worker seal-repair rollback. The lane base already includes #24712/#24716. No `.github` edits; unrelated Flutter work is preserved. Not measured: production stock/dwell, vendor spend, live nightly execution, live exporter/alert delivery, deployed revision or user-visible recovery. No GCP tier, production access, merge to main or deployment was used. Product invariant: INV-MEM-4; canonical memory intake/promotion authority is unchanged. Failure-Class: FC-batch-fault-domain-head-of-line-poison

fix(batch): stripe projection discovery startup cursors

9d63825

cubic-dev-ai Bot reviewed Nov 5, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Nov 5, 2026

View reviewed changes

Git-on-my-level added 3 commits November 6, 2026 00:46

fix(batch): randomize projection stripe per process

2483b94

fix(batch): serialize projection stripe initialization

cdef0b9

Merge origin/main to resolve pipeline switch registry overlap

125ae15

chatgpt-codex-connector Bot reviewed Nov 5, 2026

View reviewed changes

Git-on-my-level merged commit f8edd96 into main Nov 5, 2026
62 checks passed

Git-on-my-level deleted the lane/pipeline-stripe branch November 5, 2026 18:23

This was referenced Nov 5, 2026

feat(batch): add opt-in dedicated projection drain pump #24712

Merged

harden(pusher): persist playback projection rebuild debt #24725

Merged

Git-on-my-level mentioned this pull request Nov 5, 2026

fix(batch): bound held dwell and re-drive retained dead letters #24736

Merged

Navigation Menu

Repository navigation

fix(batch): stripe projection discovery startup cursors - #24716

fix(batch): stripe projection discovery startup cursors#24716
Git-on-my-level merged 4 commits into
mainfrom
lane/pipeline-stripe

Git-on-my-level commented Nov 5, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Nov 5, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

cubic-dev-ai Bot Nov 5, 2026

Uh oh!

cubic-dev-ai Bot Nov 5, 2026

Uh oh!

cubic-dev-ai Bot Nov 5, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Nov 5, 2026

Uh oh!

chatgpt-codex-connector Bot Nov 5, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Nov 5, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	assert flag['summary'].startswith('Default on')
	assert flag['summary'].startswith(('Default on;', 'Default on unless off;'))



		def projection_discovery_stripe() -> int:
		"""Stable startup phase; 256 buckets keep collisions low for 30-50 instances.

	"""Stable startup phase; 256 buckets keep collisions low for 30-50 instances.
	"""Stable startup phase; 256 buckets spread starts across instances but collide often (~83% chance of any shared start at 30 instances).

	if not segments and seeding:
	if not segments and seeding and head is not None:



		def projection_striping_enabled() -> bool:
		from config.policy import kill_switch

		after = (oldest + max(0.0, time.time() - oldest) * self.stripe / PROJECTION_DISCOVERY_STRIPES, '')
		segments, after = self.store.projection_page(limit=limit, after=after, **scope)

Conversation

Git-on-my-level commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Tests

Risk

Rollback

Uh oh!

chatgpt-codex-connector Bot commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Git-on-my-level commented Nov 5, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Nov 5, 2026 •

edited

Loading