Skip to content

fix(sync): quiet upload refusals and bound Catching up spans - #23822

Merged
Git-on-my-level merged 13 commits into
mainfrom
lane/sync-attention
Oct 27, 2026
Merged

Git-on-my-level merged 13 commits into
mainfrom
lane/sync-attention

Conversation

@Git-on-my-level

@Git-on-my-level Git-on-my-level commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Problem and behavior

Deterministic upload refusals created repeated user Retry prompts even though the same bytes and date were resubmitted. Unreconciled historical uploaded handles also stretched Catching up across weeks.

  • Retain refused audio, persist closed reason/status and daily retry timestamps, retry for seven days, then count as reported. Keep refusals outside user attention and the Failed filter, preserve explicit storage clearing, and remove per-row Retry. Expose count, seconds, reported count and user-action capability for the UI lane.
  • Accept empty/torn-header framed clips as a 200 no-op; preserve tiny encoded speech for decoding and VAD. Return bounded multipart/filename refusal codes.
  • Reconcile completed/missing jobs through the existing custody-safe poller, track authoritative pending confirmations, and bound the displayed range to the latest contiguous pending run on one day.
  • Register WAL Upload Refused telemetry with reason and HTTP status, plus an executable consumer.
  • UI (David's 10-27 choices). Catching up, option C: Device Settings drops the Right now catching-up card. The third stat tile reads "Catching up", the pending duration and "on its way ›" ("when online ›" offline), and opens On the Way; it is On Device again when nothing is pending. No stage words, date ranges or titles. On the Way, option B: " catching up", a thin bar for audio already on the server, one line, then one card with a row per day (count · duration, and Uploading / Next / Queued / Transcribing following the oldest-first upload order). Failed and refused recordings are never rows and there is no per-row Try Again or Delete; a footnote says they recover in the background, and "Details for support" opens a plain list of every recording with its reason tokens. A single action line appears only when refusalSummary.userActionCanHelp (currently always false).

Root cause and evidence

Code tracing and local HTTP tests identify 400 paths in multipart parsing (missing boundary/header, malformed boundaries, part count/size), filename validation, and capture-time admission. Empty audio is currently accepted past staging but fails decode in a server job; this is distinct from admission HTTP 400. Short valid frames are not evidence of silence. Authentication/BYOK uses 401/403, throttling uses 429, the recovery window uses 422, backfill resource limits use 413, and duplicate/language-routing admission uses 409 or a legacy fallback. Those paths do not explain an untyped admission 400.

David-scoped metadata read with ro-prod showed invalid-capture-time admission decisions at 15:50:15Z and 15:50:18Z on October 27, followed by valid legacy backfill decisions (including 15:57:30Z). These iOS requests identified themselves in headers as 1.0.578+1490, not the reported TestFlight 1.0.579, and have no trace IDs. The 16 reported Cloud Run 400 requests at 15:57–15:58Z therefore remain unattributed, and their exact refusal reason is not proven. No other users' logs, transcript or audio content were read, and no signed-in API was called.

Product invariants affected

INV-CAP-1: captured audio remains intact across refusal, backoff and missing-job reconciliation. Only a framed file with no audio payload is acknowledged as a no-op; even one tiny encoded frame remains retained for decoding/VAD.

Validation

  • bin/omi-heavy run -- flutter test --no-pub <77 sync/WAL test files>: 874 tests passed.
  • BACKEND_UNIT_TEST_FILE_LIST=... bin/omi-heavy run -- bash backend/test.sh: 949 tests passed across 22 files (760 sync tests plus 189 Cloud Tasks admission tests), including real local FastAPI multipart/empty/mixed-manifest/refusal requests with synthetic auth and storage boundaries.
  • Focused typed refusal decoder/recovery rerun after extraction: 46 Flutter tests passed.
  • On the Way widget policy: 7 tests passed, covering quiet retained refusals, transient recovery actions, grouped recordings and uploading-member deletion protection. The full CI suite exposed these old refusal-action expectations; all six were migrated to the requested policy.
  • bash scripts/analyze_ratchet.sh from app: passed without changing the analyzer baseline; the full 874-test sync suite passed again after the analyzer and filter follow-up.
  • PYRIGHT_PYTHON=.venv/bin/python bin/omi-heavy run -- bash scripts/typecheck.sh: zero errors (existing repository warnings remain).
  • Multipart regression verifies that a malformed second part closes the spool file already opened for the first part. Task-admission fixtures now contain one complete framed payload so they exercise admission rather than the new empty-clip no-op.
  • python3 scripts/check_event_registry.py: 200 registered events and executable consumers valid.
  • Screenshot harness: 3 widget tests passed; both tracked PNGs visually inspected. The harness loads fixture fonts and uses an iPhone viewport; these are synthetic renders, not a physical-device acceptance receipt.
  • scripts/pr-preflight --all --pr-body-file .agent-briefs/pr-body.md: all 48 gates passed, zero failures and zero checks skipped for time (438.8 seconds).
  • OMI_PR_BODY_FILE=.agent-briefs/pr-body.md bin/omi-heavy run -- make preflight: 47 checks passed (183.4 seconds). The bounded pre-push gate also passed, including backend type checking, selected tests and Flutter generated-file validation.

UI lane:

  • Widget tests touched by the UI: device_catching_up_test, episodes_ui_test and sync_indicator_isolation_test together 56 passed (the Device page still rebuilds 0 times on progress ticks and reads the inventory once per sync update), on_the_way_days_test 11, on_the_way_attention_test 6, home_pending_row_test 41, sync_provider_background_admission_test 6, sync_attention_evidence_test 6. They cover the tile at 200% text with a ≥44pt target and button semantics, rows and the support link at 3.1× text, and no Try Again or Delete on either view.
  • Full app suite flutter test --no-pub test/: 7,836 passed, 9 skipped, 1 failed: a visual-audit scenario whose duration finder also matched the device header's chip. It is now scoped to the tile, and the device-catching-up and on-the-way audit scenarios pass.
  • scripts/pr-preflight --pr-body-file .agent-briefs/pr-body.md: 48 checks passed (536.9 s), including ui-evidence, mobile-ux-contract, brand-ui, mobile-ui-isolate and the ARB contract.
  • New strings: English, then omi-l10n-fill for 48 locales (5 batches, 0 retried cells, review applied 13 fixes); python3 scripts/l10n.py check passes for 49 locales. The four On the Way summary keys with no callers left were removed.

No merge or deploy. No physical iPhone/TestFlight acceptance or production fix verification. PostHog delivery and ingestion keep-list remain operationally unverified.

Copy note: uploads have no Wi-Fi-only gate in the app, so the tile caption and header line name the real condition (a connection) instead of "on Wi-Fi".

Home fold ("N short clips hidden · Show") is not built. Noise conversations are served as discarded, the Home list is fetched with include_discarded=false, and the list response carries no hidden count, so the client cannot know how many were hidden on a day. It needs either a per-day hidden count from the server or an always-include fetch filtered on the phone (which changes paging); both are outside this UI lane.

Integration points

SyncProvider.refusalSummary provides count, seconds, reportedCount and userActionCanHelp. Refused files remain in the full inventory. deviceCatchUpLocal and DeviceCatchUpSummary supply the bounded range; reconciliation renews jobPendingConfirmedAt only on an authoritative pending reply. The UI reads these contracts and does not change them: the tile adds pendingSeconds to the memoized deviceCatchUpLocal snapshot and a seconds total to DeviceCatchUpSummary; the bounded span and target fields remain but are no longer displayed.

The multipart helper remains an HTTP body parser with no Firestore access. Malformed-body spool cleanup belongs to the parser subclass, leaving the existing reviewed inventory checksum unchanged. Refusal parsing, retry reservation and admission helpers were split out to keep all three large files at or below their original line counts. No line-count exception or gate bypass is used.

UI evidence

Synthetic Flutter widget renders of the real pages with injected fakes, 390×844 iOS viewport, fixture fonts. The blank band above the Device Settings tiles is the device image, which the test engine does not paint. The previous two renders showed the removed card and rows.

Device Settings, catch-up pending
Device Settings, nothing pending
On the Way, mixed days
Details for support
On the Way, dark
On the Way, largest text

Failure-Class: new

Review follow-up (GLM)

GLM's review of this PR blocked on one item, now fixed in fix(sync): cap refusal recovery per drain. A server-wide 400 could refuse thousands of recordings, and their daily recovery would then fire every single-file resend in one drain: a 400 engages neither the 429 limiter nor the 5xx pause. One drain now resends at most 25 (walMaxRefusalRetriesPerDrain), and the rest wait for a later drain. too_large and too_many_parts refusals are never resent automatically. A unit test covers both.

GLM's second finding said exhausted transient failures get no prompt on On the Way. That is intentionally not changed. David picked option B ("problems out of sight"), and those recordings are re-armed automatically when connectivity returns.

Line-Count-Exception: app/lib/services/wals/local_wal_sync.dart | 2653 -> 2654 | the per-drain refusal cap must sit at the drain's candidate filter; the file already shrank in this PR

🤖 Generated with Claude Code

Git-on-my-level and others added 13 commits October 28, 2026 00:26
Persist bounded refusal reasons and daily recovery timestamps, accept empty
framed clips as no-ops, register refusal telemetry, and reconcile pending
span evidence without turning retained server refusals into user repair work.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Device Settings drops the Right now catching-up card. The third stat
tile reads "Catching up" with the pending duration and opens On the Way,
and returns to On Device when nothing is pending. No stage words, dates
or conversation titles.

On the Way shows "<duration> catching up", a thin bar for audio already
on the server, one line, and one card with a row per day (Uploading /
Next / Queued / Transcribing, following the oldest-first upload order).
Failed and refused recordings are not rows and have no Retry or Delete;
a footnote says they recover in the background, and "Details for
support" lists every recording with its reason tokens. A single action
line appears only when the refusal summary says user action can help.

New strings are translated for all locales; four summary strings with
no callers left are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e changelog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The device header's pending-sync chip shows the same duration, so an
unscoped text finder matched twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ermined refusals

A server-wide 400 can refuse thousands of recordings; their daily recovery
fired every single-file resend in one drain (400s engage neither the 429
limiter nor the 5xx pause). One drain now re-sends at most 25; the rest wait
for a later drain. too_large and too_many_parts never auto-resend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Git-on-my-level
Git-on-my-level marked this pull request as ready for review October 27, 2026 22:15
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T22:20:54.225619Z aaeeb76 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aaeeb76d55

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread backend/routers/sync.py
Comment on lines +1067 to +1069
upload_filenames = [f.filename or '' for f in files]
try:
files = await run_blocking(sync_executor, prepare_sync_uploads, files, uid)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate capture evidence before filtering empty files

When an upload contains both an empty/torn WAL and nonempty audio with a valid S1 evidence header, removing the empty file here invalidates the evidence for every surviving file. The later parse_s1_upload_claims(..., filenames) validates against the filtered filenames, while validate_sync_file_claims rejects the entire header if any claim names an excluded file. This makes lineage_resolution_requested fall back to whole-batch conversation binding and skips source-position decoding for otherwise valid audio. Validate against upload_filenames first, then filter the validated capture claims alongside the files, as this change already does for manifest claims.

Useful? React with 👍 / 👎.

Comment on lines 157 to 160
return Semantics(
label: [label, value, if (caption != null) caption!, if (secondCaption != null) secondCaption!].join(', '),
button: onTap != null,
excludeSemantics: true,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Expose the Catching up tile's semantic tap action

For VoiceOver or TalkBack users, the new Catching up tile is announced as a button but cannot be activated. excludeSemantics: true removes the descendant InkWell's semantic tap action, and the enclosing Semantics supplies only a label and button flag, not onTap. Since the previous Catching up card and its navigation button are removed, this also removes the accessible entry point to On the Way from device settings. Forward onTap to the enclosing Semantics or preserve the descendant's action.

Useful? React with 👍 / 👎.

Comment on lines +130 to +133
var seconds = local.pendingSeconds;
final processingIds = {for (final c in processing) c.id};
for (final conversation in processing) {
if (!ids.contains(conversation.id)) seconds += max(0, conversation.getDurationInSeconds());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use consistent processing totals for the tile and its destination

When a processing conversation still has a local WAL bearing its conversation ID, this calculation counts only the local WAL duration and excludes the conversation's duration. However, _PendingWork.collect in on_the_way_page.dart does the opposite: it excludes those WALs and counts the full processing conversation. For example, a 30-minute processing conversation with one remaining 1-minute uploaded WAL makes the Catching up tile show 1 minute, but opening it shows 30 minutes. The retained WALs need not cover the entire conversation, so they cannot substitute for its processing duration. Share the same deduplication and duration calculation between the two surfaces.

Useful? React with 👍 / 👎.

@Git-on-my-level
Git-on-my-level merged commit d587d85 into main Oct 27, 2026
58 checks passed
@Git-on-my-level
Git-on-my-level deleted the lane/sync-attention branch October 27, 2026 22:22
Git-on-my-level added a commit that referenced this pull request Oct 28, 2026
### What does this PR do?

Re-implementation of community PR #17825 by @TheRWX (approved, but
conflicted and unmoved since 2026-10-05) on today's sync code and the
rulings that landed since. Fixes #15501: Android users silently lost
days of pendant recordings with no "device not syncing" warning; others
only sync when they open the app.

A new `SyncHealthWatchdog` (app/lib/services/sync_health_watchdog.dart)
watches the paired pendant's pipeline:

- **Activity signals.** BLE audio bytes arriving, a WAL accepted by the
server (`onWalSynced` — every upload path, manual and drain), a batch
recording finalized, and the pendant (re)connecting. Any of them proves
recordings are moving — audio is then safe on the phone or on the
server, and the #23679/#23822 drains (oldest-first, Retry-After, quiet
refusals) own recovery. Only silence across **every** signal is a stall
the user must hear about. Refusal-only stretches stay quiet by design: a
refusal with live ingestion is not data loss, and #23822 deliberately
keeps refusals out of user attention.
- **Stall threshold: 24 hours.** Longer than the longest normal daily
gap — a pendant taken off and charged overnight (~8–12h silent) — and
far inside the reported 4-day silent losses, with #23679's 30-day server
backfill window still far away. (The original 4h threshold would alert
every morning when quiet hours end.)
- **Alert cadence.** At most one reminder per day (24h cooldown), never
22:00–08:00 local (quiet hours do not spend the cooldown), notification
id 15501, tap lands on Device settings (`/settings/device`, the
capture-recovery push precedent). The silence clock seeds on first run,
so the app update itself cannot alarm.
- **Not a disconnect push (#3328).** No per-event notification; onboard
storage keeps recording across BLE drops. This alert fires only after a
full day of zero flow — the state where users actually lost data. Acute
stalls stay the wedge monitor's job in-app (connected-no-bytes within
minutes, upload silence at 2h with a backlog); this is the out-of-app
last resort.
- **Scope.** Paired pendant types only (`omi`, `friendPendant`); watches
and glasses sync on their own cadence. Ships on by default with a kill
switch ("Sync Health Alerts", Device Settings).

**Surfacing, per #22119** (capture/sync problems speak on the live
capture page and the Activity bell, never Home):

- Activity → Needs attention: "Omi isn't syncing · No recordings synced
for over N hours" (cloud-off row, opens Offline Sync; ordered after the
upload row as the broadest, slowest condition). Fed through
`CaptureWedgeMonitor.observeSyncSilence`, so the bell's unseen-dot
machinery is reused unchanged.
- Device Settings: the "Sync Health Alerts" toggle sits under the device
group; its subtitle switches to the live stall duration while stalled.
- The original PR's Home indicator is deliberately **not**
re-implemented.

The watchdog's periodic loop starts in app bootstrap (main.dart, after
prefs init); `DeviceProvider` wires each evaluation into the wedge
monitor and exposes `isSyncStalled`/`syncHealth`. Timer ownership stays
out of the provider so provider tests never hold a pending timer.

### Renders (headless visual-audit captures, synthetic state)

Activity — the new sync stall row:
![Activity sync stall
row](https://raw.githubusercontent.com/BasedHardware/omi/bf706226c6f7/.agent-artifacts/ui-evidence/cs-syncalerts/001-mobile-ios-activity-sync-stall.png)

Device Settings — the alert toggle with the stall subtitle:
![Device Settings stall
toggle](https://raw.githubusercontent.com/BasedHardware/omi/bf706226c6f7/.agent-artifacts/ui-evidence/cs-syncalerts/002-mobile-ios-device-stall-toggle.png)

Activity — the existing capture rows, unchanged by this PR:
![Activity capture rows
unchanged](https://raw.githubusercontent.com/BasedHardware/omi/bf706226c6f7/.agent-artifacts/ui-evidence/cs-syncalerts/003-mobile-ios-activity-capture-attention.png)

## Product invariants affected

- INV-DATA-1
- INV-CAP-1

INV-DATA-1 applies because `app/lib/main.dart` is a protected data-plane
routing authority; this PR only starts the watchdog's periodic timer
there after prefs init — no routing, endpoint, build define, or bundled
env value changes.

INV-CAP-1: touches `app/lib/services/capture/capture_wedge_monitor.dart`
only to add a slow-burn `CaptureSyncSilence` row (declared and ended
solely by the watchdog's evaluation) to Needs attention and the bell's
unseen logic. Ingress, retention and admission behavior are unchanged;
the new row makes no saved-audio claim.

## How it was verified

- `flutter test` (full hermetic `bash test.sh` CI suite): 7921 passed /
9 skipped; the single failure during the run was this PR's own new
visual-audit scenario before a fixture fix, green on re-run (`flutter
test test/visual_audit/visual_audit_smoke_test.dart`: 158 passed).
- New suites: `test/unit/sync_health_watchdog_test.dart` (device gating,
24h threshold arithmetic, daily cooldown, quiet hours, toggle, seeding,
throttle incl. no clock rewind, timer lifecycle) and new groups in
`test/services/capture/capture_wedge_monitor_test.dart` (sync-silence
row + bell) and `test/widgets/needs_attention_capture_test.dart` (row
render, absence, Offline Sync tap, ordering behind the upload row).
- `make mobile-verify ARGS="fast --all"`: 31/31 journeys passed
(full-suite fallback selected — cross-cutting `app/lib` change).
- `flutter analyze`: no new findings (pre-existing environment-only
errors in flavor-generated files aside, which `test.sh` bootstrap
generates).
- `python3 app/scripts/l10n.py check`: ok, 49 locales; 6 new keys via
the tool.
- `check_mobile_ux_contract.py` on the two touched UI files: clean.
- `tool/perf_shell.py --check`: PASS (shell regenerated for the
main.dart import).
- Renders above: `app/scripts/visual_audit.sh --head WORKTREE --only
activity-sync-stall,device-sync-stall-alerts,activity-capture-attention`.

## Not verified / limitations

- The watchdog evaluates only while the app process runs (the capture
foreground service keeps it alive on Android). If the OS kills the
process, no alert can fire until it runs again — the same boundary the
original PR had.
- The Activity row's hour count refreshes on watchdog ticks (30 min),
not per minute.
- No on-device run: all verification is hermetic (widget tests,
journeys, headless renders).

Line-Count-Exception: app/lib/backend/preferences.dart | 1612 -> 1630 |
three persisted sync-health keys: alert toggle, last activity timestamp,
last alert timestamp
Line-Count-Exception: app/lib/providers/device_provider.dart | 1624 ->
1663 | watchdog wiring, stall getters, connect/disconnect hooks
Line-Count-Exception: app/lib/services/capture/capture_controller.dart |
4602 -> 4609 | record sync-activity on BLE audio and batch finalization

Failure-Class: none

Closes #15501
Re-implements #17825 (original by @TheRWX, co-authored there)


<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/BasedHardware/omi/pull/23884?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/BasedHardware/omi/23884?returnTo=https%3A%2F%2Fgithub.com%2FBasedHardware%2Fomi%2Fpull%2F23884&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
Git-on-my-level added a commit that referenced this pull request Oct 28, 2026
…rst build, device perf harness (#23859)

* fix(mobile-perf): physical iOS batches die on macOS bash 3.2 before launch

HOST_GUARD is empty on every physical-device run, and macOS /usr/bin/env bash
(3.2) treats "${HOST_GUARD[@]}" of an empty array as unbound under set -u, so
every physical iPhone batch failed before receipt 1. Expand it only when set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mobile-perf): frame total spans go negative on iOS 27 devices

On an iPhone 15 Pro Max (iOS 27.0.1) about one frame in seven starts building
~0.85 ms before its recorded vsync, so FrameTiming.totalSpan (raster finish -
vsync) is negative. Total p50/p90 were contaminated in every window, and
cold_start's p50 went negative, failing report validation for every batch.
Measure total from the earlier of vsync and build start. This changes the total
metric: earlier total percentiles are not comparable; qualify a new baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(mobile-perf): live preview, live episode, Device Settings and On the Way windows

Three heavy-profile windows after the unchanged five, for the changes in
#23819, #23822 and #23853:

- home_live_preview: Home with the live episode row while a scripted on-device
  preview revises its newest line every 200 ms over a ~1,000-segment session
  (LivePreviewCoverage on every update and clock tick).
- live_episode_page: LiveEpisodePage on the transcript tab over the same
  session, scrolled while words arrive.
- device_settings_on_the_way: Device Settings (Catching up) and On the Way
  (day-grouped) over the 6,500-WAL backlog, each opened and scrolled.

The preview controller is installed through CaptureController.installLivePreview
and admitted through LivePreviewController.admit; only the socket admission flag
is asserted, since the fixture runs no listen socket. Device Settings gets an
interface fake: the production DeviceProvider subscribes to ServiceManager,
which this harness never initializes. Reports now require the eight windows;
comparisons list windows absent from the reference instead of failing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(changelog): heavy perf harness windows (internal only)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(mobile-perf): main-isolate stalls, field-shaped WAL index and an idle sync window

The field lag log that matters (iPhone 15 Pro Max, 10-26) is 133 main-isolate
stalls, mostly on idle Home and around sync phases, with a 71,536-record WAL
index. Drawn-frame timings could not see any of it.

- HeavyStallProbe: every window reports stalls over 50/100 ms, blocked time and
  the worst stall from a 20 ms timer's late ticks; reported, never variance-gated.
- --synced-wals N (0..80000): adds synced history on top of the pending
  backlog; such a batch changes the dataset and is never baseline-eligible.
- home_idle_sync_refresh: Home left alone while SyncProvider refreshes the
  inventory five times, as sync phase ends and upload callbacks do.
- Runner preflight: refuses a phone that is not on a wired USB connection
  before the multi-minute build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(home): live row computes uncovered preview once per clock tick

The Semantics label called previewText() twice per 1 s rebuild; each call runs
LivePreviewCoverage.uncovered over every final segment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(mobile): live transcript opens at the newest rows; Tasks first frame stays cheap

Measured on an iPhone 15 Pro Max (heavy profile harness):
- Opening LiveEpisodePage on the transcript tab took one 153 ms layout with
  1,775 builds (140-157 ms every run) and peaked RSS at 612 MB: following the
  latest jumped to maxScrollExtent on a variable-height lazy list, laying out
  every row. TranscriptWidget now anchors layout at the newest segment; a
  1,000-segment open builds 9 rows and lands on the newest.
- Preview revisions (every ~200 ms) no longer rebuild the transcript or rescan
  final history: LivePreviewCoverage prepares merged final spans once per
  transcript snapshot and answers each line by binary search, same results.
- The first Tasks tab switch spent 128 ms in one build (240-319 ms frame):
  ActionItemsPage now uses one lazy delegate across date sections, deferring
  offscreen sorting, headers and rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(sync): WAL index work scales with what changed, not with every record ever kept

A heavy phone's index held 71,536 records (66,222 synced). Its lag watchdog
logged 133 main-isolate stalls over ~10 h (24 s blocked, worst 1.6 s), mostly
on idle Home and ~20x more often right after sync phases and uploads. Every
index save copied, keyed and fingerprinted every record on the main isolate
and copied the full List<Wal> into an encoding isolate; sync screens rescanned
the inventory on every notify; reconcile polled ~2/s after a reconnect.

- Saves: Wal persisted-field setters mark dirty identities; a long-lived index
  worker owns the serialized rows and receives only changed JSON maps and
  removed tokens. Durable is acknowledged only after flush + atomic rename;
  failed deltas are retained and retried. No whole-index fingerprint.
- Load: decode and Wal construction run off-isolate; the worker seeds
  unchanged rows from disk, so the first save sends zero unchanged records.
- Format: unchanged version-1 wals.json, backup and quarantine recovery;
  retired/foreign/synced records and upload-order tie-breaks preserved.
- Views: SyncProvider derives pending/uploaded/attention/refusal lists and
  per-day totals incrementally (WalDerivedViews); Home, On the Way and Device
  Settings read them. Integration and widget fakes derive the same views from
  their raw inventory (WalDerivedSyncViews).
- Reconcile: visits uploaded rows only, joins an active pass, spaces passes by
  20 s; job backoff, fairness and acknowledgement unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(mobile): newest-anchored transcript renders short lists on the first frame

The newest-anchored layout needed extra frames for its anchor correction, so a
short live transcript (even one segment) was blank on its first frame;
capturing_empty_state_test failed 20 of 23. The viewport now measures its anchor
and initial latest position during layout, before paint. Short transcripts show
immediately; a 1,000-segment open still builds about one screen of rows. The
extra pumps added to episodes_ui_test are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mobile): live transcript keeps the reader's row fixed when rows above grow

Review finding (GLM): on a short scrollable live transcript, a photo group or
an older row growing during a drag or fling changed the newest-anchored
viewport's anchor with pixels unchanged, moving every visible row by the
inserted height under the reader's finger, and the correction bailed for the
whole gesture. The viewport now compensates the leading-edge shift with
offset.correctBy in the same layout pass. Regression tests cover short and
long histories, photo insertion/growth and older-row growth across drag, fling
and idle; first-frame rendering and 1,000-segment laziness still hold.

Failure-Class: none

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(sync): WAL index worker death, backup seeding and repair writes recover in-session

Review findings (GLM) on the delta index worker:
- A dead worker isolate (OOM/VM kill) left apply() awaiting forever, wedging
  every later save and the capture flush queue. Exit/error listeners now fail
  pending requests and release the queue; the replacement worker is seeded from
  a full snapshot of the in-memory index, preserving tokens, admitted deltas
  and every custody bucket.
- After loading from the backup because wals.json was corrupt and could not be
  quarantined, the worker seeded from the corrupt main file and every save
  failed for the session. Workers now receive the exact source the loader
  validated; a failed startup retries from memory.
- The load-time invalid-entry repair used the legacy whole-file writer on the
  same wals.json(.tmp) as the live worker with no shared lock. Loading,
  quarantine/repair, worker seeding and writes now share one lock.

Every failure fails closed: no durable acknowledgement before flush + rename.

Failure-Class: none

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(mobile-perf): field-shaped synced WALs survive the startup retention sweep

Dated syncedAt stamps were older than the synced-copy retention cutoff, so the
startup sweep removed all seeded synced records and Home saw only the pending
backlog. Legacy syncedAt 0 is never auto-removed, matching a long-lived phone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(mobile): move transcript row highlight and live viewport into a part file

Keeps transcript.dart under its line-count ratchet (1590) after the
newest-anchored viewport landed; no behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(dead-code): allowlist the derived sync views shared by SyncProvider stand-ins

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(mobile-perf): timeline runs name each widget build

Attribution-only runs (never baselines) enable debugProfileBuildsEnabled so a
slow first build, such as the first Tasks switch, names its widget in the trace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mobile): transcript scroll reporting stays bounded by the visible rows

Device regression (iPhone 15 Pro Max, 3 runs before/after): opening a saved
conversation's transcript and flinging it went from 0/101/100 ms of main-isolate
stalls per run to 223/257/116 ms. The trace put ~15% of samples in
_TranscriptPlaybackScrolling._reportTopVisibleSegment, which walked the sliver
tree on every scroll notification. Sliver discovery is now cached, each sliver
needs one reveal calculation, and remaining row positions use existing layout
offsets; first-frame layouts and reveal calls per frame are unchanged from 120
to 1,200 rows (test).

Failure-Class: none

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(mobile): kill switch for the delta WAL index writer

PostHog mobile-wal-delta-index-off (ops_kill, default false). True makes
the next launch save every index change through the pre-delta
full-snapshot writer; both writers produce the same version-1 file, so
flipping either way needs no migration. Read once per process with a 1 s
bound; offline or failed reads keep the delta writer on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant