feat: session tail rehydrate (#13) to main

openspec/changes/session-tail-rehydrate/design.md

@@ -0,0 +1,141 @@
+## Context
+
+- Subscribe is already incremental via `lastSeq` (`subscription-handler.ts`).
+- Strategy A persists raw events in IndexedDB and delta-subscribes on reload
+  (`replay-cache.ts`, `rehydrate-session.ts`, `App.tsx`).
+- Large sessions exceed `DEFAULT_MAX_BYTES_PER_SESSION` (5 MiB) → put deletes
+  the entry → every cold open is full replay.
+- Server cold path loads full JSONL then ships all stored events in batches of
+  50 with `MAX_REPLAY_EVENTS = 0` (unlimited).
+- ChatView `scrollStateMap` is in-module only; restore runs only on `sessionId`
+  change. Wipe→rebuild of the same session does not re-pin.
+
+## Goals
+
+1. Cold open of a large session transfers **O(budget)** events, not full history.
+2. Large sessions remain **cacheable** under a byte budget (newest-first).
+3. Users can **page older history** without losing the current viewport.
+4. After hydrate, default land position is **true bottom** (unless user escaped).
+5. Legacy clients (no `mode`) keep full-replay behavior.
+
+## Non-Goals
+
+See proposal. Notably: durable last-seen anchor and estimate-drift fixes stay
+out of this change.
+
+## Decisions
+
+### D1 — Byte budget, not turn count
+
+**Choice:** Keep newest events until ~**4 MiB** serialized payload.
+
+**Why:** Turn boundaries are uneven (one tool-heavy turn can be multi-MB). A
+byte budget gives a hard wire/IDB ceiling. Client and server share the same
+selection algorithm so cache tail ≈ first paint tail.
+
+**Default:** `DEFAULT_TAIL_WINDOW_BYTES = 4 * 1024 * 1024`. Client may pass
+`windowBytes` on subscribe; server clamps to `[256_KiB, 8_MiB]`.
+
+### D2 — Newest-first selection, whole events only
+
+```
+selectNewestEventsByBudget(eventsSortedBySeqAsc, budget):
+  walk from end → start
+  include event if size(event) + acc <= budget OR acc == 0 (always include newest)
+  stop when next would exceed
+  return included in ascending seq order
+```
+
+Size = `JSON.stringify({ seq, event }).length` (same as cache put today) for
+determinism between client trim and server wire estimate. Never split an event.
+
+**Incomplete oldest message:** If the oldest kept event is a `message_update`
+without its `message_start` in the window, still keep it — reducer already
+tolerates partial streams for display of finalized `message_end` when present.
+Do **not** expand the window past budget to complete pairs (budget is hard).
+
+### D3 — Additive protocol (no new message type for v1)
+
+**Subscribe (browser → server):**
+
+| Field | Type | Meaning |
+|---|---|---|
+| `mode?` | `"full" \| "tail"` | Default `"full"` (legacy). Cold open uses `"tail"`. |
+| `windowBytes?` | number | Budget hint; server clamps. |
+| `fromSeq?` | number | Load-older: exclusive upper bound (`seq < fromSeq`). |
+
+**Event replay (server → browser):**
+
+| Field | Type | Meaning |
+|---|---|---|
+| `hasMoreOlder?` | boolean | More history exists below `windowMinSeq`. |
+| `windowMinSeq?` | number | Lowest seq in the retained/delivered window. |
+| `windowMaxSeq?` | number | Highest seq in this delivery (usually = last event seq). |
+
+**Matrix:**
+
+| Client | Server |
+|---|---|
+| omit `mode` / `mode:"full"` | Today's full (or delta) path |
+| `mode:"tail"`, no `fromSeq`, `lastSeq` 0/absent | Newest budget window from store or disk |
+| `mode:"tail"`, `lastSeq > 0` | Delta `seq > lastSeq` only (mode ignored for delta) |
+| `fromSeq: N` | Older page: newest events with `seq < N` under budget |
+| store empty + disk | Load disk; apply same window before send |
+
+Warm reconnect with live `lastSeq` is unchanged.
+
+### D4 — Client cache: trim-to-tail, schema v2
+
+- On put: if full buffer serializes over budget, trim with
+  `selectNewestEventsByBudget` then put. **Never** delete solely for size.
+- Persist `{ maxSeq, windowMinSeq, payload, schemaVersion: 2 }`.
+- v1 entries: schema mismatch → miss → tail subscribe (safe).
+- `session_state_reset` still `drop()`s the entry.
+
+### D5 — Load-older preserves scroll anchor
+
+When prepending older rows:
+
+1. Snapshot first visible virtual row key + offset before prepend.
+2. Apply older events (prepend to reducer state / merge by seq).
+3. Restore anchor via `scrollToIndex` + offset (same CR-6 virtual coords as
+   session restore).
+
+Do not `session_state_reset` for older pages. Do not clear stick-to-bottom
+latch incorrectly: load-older implies user is at top → stick stays false.
+
+### D6 — Post-hydrate re-pin
+
+When `loadingHistory` goes `true → false` for the active session:
+
+- If `stickToBottomRef` is true **or** no user scroll-up occurred during this
+  hydrate (`!userEscapedDuringHydrate`), pin bottom once.
+- If user wheeled/touched away mid-hydrate, respect escape (existing
+  `cancelDescent` / scroll handler).
+
+This fixes wipe→rebuild same-`sessionId` without durable last-seen.
+
+### D7 — Shared pure helper location
+
+Prefer `packages/shared/src/event-window.ts` (or under existing shared event
+util path) so client IDB put and server subscribe use one implementation and
+one unit test file. Server-only copy is acceptable only if shared import is
+awkward for the worker thread; prefer shared.
+
+## Risks
+
+| Risk | Mitigation |
+|---|---|
+| Budget too small → thin context | 4 MiB default; clamp max 8 MiB; load-older |
+| Budget too large on mobile | Clamp; measure hydrate time in manual smoke |
+| `hasMoreOlder` wrong after store essential-trim | Compare `windowMinSeq` to buffer min seq; disk cold load may still have older not in memory — if only memory is searched, document that load-older may need disk path (reuse `loadSessionEvents` windowed) |
+| Double full load on disk cold | Load once; window in memory; page older from same buffer |
+| Fork / seq reset | Existing reset purge |
+| Prepend scroll jump | Anchor restore required in acceptance |
+
+## Open questions (resolve in tasks 1.x if needed)
+
+1. Should load-older use the same `subscribe` message or a one-shot
+   `load_history`? **Lean subscribe + `fromSeq`** (fewer types). Confirm if
+   re-entrancy with live subscribe set is awkward.
+2. Exact clamp bounds (256 KiB–8 MiB) — tune after first large-session smoke.

openspec/changes/session-tail-rehydrate/proposal.md

@@ -0,0 +1,79 @@
+## Why
+
+Strategy A (`reduce-session-replay-traffic`) persists the full raw event buffer
+per session in IndexedDB and delta-subscribes with `lastSeq = maxSeq` on reload.
+That fails for the sessions that matter most: **large chats**.
+
+The client cache is all-or-nothing:
+
+- `DEFAULT_MAX_BYTES_PER_SESSION = 5 MiB`
+- Over-cap → **delete entry and skip persist** → next cold open is always
+  `lastSeq: 0` full replay
+
+The server subscribe path still ships the **entire** in-memory (or cold-loaded
+JSONL) event stream when `lastSeq` is 0 (`MAX_REPLAY_EVENTS = 0`). Mobile return
+to a conversation therefore re-downloads and re-reduces most/all of history.
+
+A second, related UX bug: after wipe→rebuild of the same `sessionId`, ChatView
+restores scroll only on `sessionId` change. Multi-batch hydrate + sticky-scroll
+escape leaves the viewport mid-transcript on an arbitrary finished agent bubble
+instead of the true bottom / last-seen region.
+
+## What Changes
+
+1. **Byte-budget tail cache (client).** Persist only the **newest** events that
+   fit a ~4 MiB budget. Large sessions stay cacheable; cold open rehydrates a
+   tail and delta-subscribes from that `maxSeq`. Never all-or-nothing drop solely
+   because the full buffer is large.
+2. **Server tail-first subscribe (protocol additive).** Cold open uses
+   `subscribe { mode: "tail", windowBytes? }`. Server returns the newest events
+   under the budget with `hasMoreOlder` / `windowMinSeq` / `windowMaxSeq` on
+   `event_replay`. Legacy clients omit `mode` → full replay (unchanged).
+3. **Load-older.** When the user reaches the top of the window, request the next
+   older page via `fromSeq` (exclusive upper bound). Prepend without wiping;
+   preserve scroll anchor.
+4. **Post-hydrate re-pin.** When `loadingHistory` clears and the user has not
+   deliberately locked away from bottom, pin to the true bottom once so cold
+   open lands on the latest content.
+
+## Capabilities
+
+### New Capabilities
+
+- `session-history-window` — tail-mode subscribe, byte-budget event selection,
+  `hasMoreOlder` signaling, load-older paging.
+
+### Modified Capabilities
+
+- `session-replay-persistence` — over-cap behavior becomes **trim-to-tail and
+  persist**, not delete-and-miss; schemaVersion bump.
+- `chat-scroll-lock` — re-pin bottom after history hydrate completes (same
+  sessionId), without fighting deliberate scroll-up.
+
+## Non-Goals
+
+- Durable last-seen message id / scroll map across reloads (follow-up).
+- Content-aware virtual row size estimates (`fix-chat-scroll-to-top-estimate-drift`).
+- Persisting reduced `ChatMessage[]` instead of raw events.
+- Prefetch of non-selected sessions.
+- Raising server `DEFAULT_MAX_EVENTS_PER_SESSION` as a substitute for windowing.
+- Push / unread / `session_view` changes.
+
+## Impact
+
+- `packages/shared/src/browser-protocol.ts` — additive fields on `subscribe` and
+  `event_replay`.
+- `packages/server/src/browser-handlers/subscription-handler.ts` — tail / older
+  page selection before `sendEventBatches`.
+- `packages/server/src/` (+ optional shared) — pure
+  `selectNewestEventsByBudget`.
+- `packages/client/src/lib/replay-cache.ts` / `replay-persist.ts` — tail put,
+  schema v2.
+- `packages/client/src/App.tsx` — cold subscribe `mode: "tail"`.
+- `packages/client/src/components/ChatView.tsx` — re-pin + load-older trigger /
+  scroll-anchor preserve.
+- `packages/client/src/hooks/useMessageHandler.ts` — handle window metadata;
+  prepend older pages.
+
+Base branch: `develop` (post `omp-minimal` merge). Branch:
+`feat/session-tail-rehydrate`.

openspec/changes/session-tail-rehydrate/specs/chat-scroll-lock/spec.md

@@ -0,0 +1,57 @@
+## MODIFIED Requirements
+
+### Requirement: Sticky bottom and scroll-to-bottom control
+
+ChatView SHALL keep a stick-to-bottom latch: near-bottom follows new content;
+scroll-up escapes; the scroll-to-bottom control re-arms follow. Multi-batch
+`event_replay` SHALL not leave the user permanently stuck mid-list after a cold
+hydrate when they did not deliberately escape.
+
+#### Scenario: Escape while streaming
+
+- **WHEN** the user scrolls up away from the bottom during live content
+- **THEN** new content SHALL NOT yank the viewport to the bottom
+- **AND** the scroll-to-bottom control SHALL be visible
+
+#### Scenario: Re-arm at bottom
+
+- **WHEN** the user scrolls back within the near-bottom threshold
+- **THEN** stick-to-bottom SHALL re-arm and chase new content
+
+### Requirement: Post-hydrate bottom pin
+
+When history loading completes for the selected session, ChatView SHALL pin to
+the true bottom once if the user has not escaped during that hydrate. This
+applies even when `sessionId` did not change (wipe→rebuild / multi-batch cold
+open).
+
+#### Scenario: Cold open lands at bottom
+
+- **WHEN** a session opens empty with `loadingHistory` true and then receives
+  its first full history window with the user not scrolling away
+- **THEN** after `loadingHistory` becomes false the viewport SHALL be near the
+  bottom
+- **AND** the scroll-to-bottom control SHALL be hidden
+
+#### Scenario: Same-session wipe rebuild
+
+- **WHEN** the same `sessionId` is wiped to empty and rebuilt from replay without
+  a session switch
+- **AND** the user has not deliberately escaped during rebuild
+- **THEN** the viewport SHALL end near the bottom of the rebuilt transcript
+
+#### Scenario: User escape during hydrate is respected
+
+- **WHEN** the user wheels or touch-scrolls away from the bottom while history
+  is still loading
+- **THEN** post-hydrate pin SHALL NOT force them back to the bottom
+
+### Requirement: Load-older does not fight scroll lock
+
+#### Scenario: Prepend keeps anchor
+
+- **WHEN** older history is prepended while the user is at the top of the current
+  window
+- **THEN** the first visible row before prepend SHALL remain the first visible
+  row after prepend (within normal layout tolerance)
+- **AND** stick-to-bottom SHALL remain disarmed

openspec/changes/session-tail-rehydrate/specs/session-history-window/spec.md

@@ -0,0 +1,68 @@
+## ADDED Requirements
+
+### Requirement: Tail-mode subscribe
+
+The browser MAY send `subscribe` with `mode: "tail"` and optional `windowBytes`.
+When `mode` is absent or `"full"`, the server SHALL retain existing full/delta
+replay behavior. When `mode` is `"tail"` and `lastSeq` is absent or `0`, the
+server SHALL deliver only the newest events that fit the effective byte budget
+(default 4 MiB, clamped), not the entire session buffer.
+
+#### Scenario: Cold tail open under budget
+
+- **WHEN** a browser subscribes with `mode: "tail"` and `lastSeq` 0 to a session
+  whose full event buffer serializes under the budget
+- **THEN** the server SHALL deliver all events in ascending seq order
+- **AND** `hasMoreOlder` SHALL be false
+
+#### Scenario: Cold tail open over budget
+
+- **WHEN** a browser subscribes with `mode: "tail"` and `lastSeq` 0 to a session
+  whose full event buffer serializes over the budget
+- **THEN** the server SHALL deliver a newest-first subset under the budget
+- **AND** `hasMoreOlder` SHALL be true
+- **AND** `windowMinSeq` SHALL equal the lowest delivered seq
+- **AND** `windowMaxSeq` SHALL equal the highest delivered seq
+
+#### Scenario: Legacy client full replay
+
+- **WHEN** a browser subscribes without `mode` and `lastSeq` 0
+- **THEN** the server SHALL deliver the full available event buffer as today
+
+#### Scenario: Delta ignores tail mode
+
+- **WHEN** a browser subscribes with `mode: "tail"` and `lastSeq > 0` and the
+  server has events with `seq > lastSeq`
+- **THEN** the server SHALL delta-replay only those events (existing path)
+
+### Requirement: Load-older page
+
+The browser MAY send `subscribe` (or an equivalent same-session request) with
+`fromSeq: N` to request older history. The server SHALL return the newest events
+with `seq < N` that fit the budget, with updated `hasMoreOlder` / window fields.
+The client SHALL merge them without wiping already-reduced state.
+
+#### Scenario: Older page under budget
+
+- **WHEN** the client requests older history with `fromSeq` equal to the current
+  `windowMinSeq` and older events exist
+- **THEN** the server SHALL deliver events strictly older than `fromSeq`
+- **AND** the client SHALL prepend them into the session transcript
+- **AND** the visible scroll anchor SHALL remain stable
+
+#### Scenario: No older history
+
+- **WHEN** `fromSeq` is less than or equal to the oldest available seq
+- **THEN** the server SHALL deliver an empty (or terminal) page with
+  `hasMoreOlder: false`
+
+### Requirement: Byte-budget selection
+
+Event window selection SHALL walk newest→oldest, include whole events only, and
+always include at least the newest event when the buffer is non-empty. Client
+IDB tail trim and server wire selection SHALL use the same algorithm.
+
+#### Scenario: Deterministic trim
+
+- **WHEN** the same ordered event list and budget are passed to the shared helper
+- **THEN** client and server SHALL produce identical seq sets

openspec/changes/session-tail-rehydrate/specs/session-replay-persistence/spec.md

@@ -0,0 +1,50 @@
+## MODIFIED Requirements
+
+### Requirement: Durable replay cache is an optimization only
+
+The client SHALL persist a per-session raw event payload + cursor in IndexedDB so
+a reload can delta-subscribe. The cache remains an optimization: miss, schema
+mismatch, or `session_state_reset` SHALL fall back to a safe network path
+without rendering stale history as authoritative.
+
+#### Scenario: Reload with cache hit
+
+- **WHEN** the user reloads and a valid cache entry exists for the session
+- **THEN** the client SHALL pre-seed reduced state from the cached payload
+- **AND** SHALL subscribe with `lastSeq = persisted maxSeq` (and MAY set
+  `mode: "tail"`)
+- **AND** the server SHALL delta-replay only events after that cursor when present
+
+#### Scenario: Cache miss
+
+- **WHEN** no entry exists, schema mismatches, or IndexedDB errors
+- **THEN** the client SHALL subscribe without relying on cached state
+- **AND** for cold open SHALL use `mode: "tail"` so the server does not force a
+  full multi-megabyte replay when history is large
+
+### Requirement: Over-budget sessions remain cacheable
+
+When the live raw-event buffer exceeds the per-session byte budget, the client
+SHALL persist a **newest-first tail** that fits the budget rather than skipping
+persist or deleting the entry solely due to size.
+
+#### Scenario: Large session put trims to tail
+
+- **WHEN** the debounced persister flushes a buffer whose serialized size exceeds
+  the budget
+- **THEN** the cache SHALL store the newest events under the budget
+- **AND** SHALL record `maxSeq` as the highest seq in that tail
+- **AND** a subsequent reload SHALL be able to cache-hit that tail
+
+#### Scenario: Schema version bump invalidates old shape
+
+- **WHEN** `schemaVersion` on disk does not match the running client
+- **THEN** the get path SHALL treat the entry as a miss (full/tail network path)
+
+### Requirement: Reset purges cache
+
+#### Scenario: session_state_reset drops entry
+
+- **WHEN** the client receives `session_state_reset` for a session
+- **THEN** it SHALL delete that session's cache entry and in-memory persist buffer
+- **AND** subsequent subscribe SHALL not use the purged maxSeq