Show recovery scan health on the session dashboard

.scratch/multiplayer-janitor-implementation/issues/01-connect-team-identities.md

@@ -4,7 +4,7 @@
 
 **Blocked by:** None (can start immediately).
 
-**Status:** ready-for-agent
+**Status:** needs-triage
 
 Source: [Accepted implementation handoff](../spec.md). Implement this slice within its accepted contracts and evidence limits.
 
@@ -19,3 +19,5 @@ Source: [Accepted implementation handoff](../spec.md). Implement this slice with
 ## Comments
 
 2026-09-12: Implemented. Migration `0022_teammates.sql` adds teammates, links, single-use link attempts and audit. `Teammates` (admission, roles, removal/restoration, links, transactional `authorize`) and `AccountLinking` (Slack OIDC with bound state/nonce and workspace validation; GitHub App user authorization with numeric user lookup) sit behind the new membership middleware, which every human route now requires in addition to Access. Routes live under `/api/v1/account` and `/api/v1/team`; the web app gains an Account page with connected accounts and the admin roster. Tests cover callback replay, conflicting concurrent links, replacement, member/admin boundaries, last-admin races, removal/relinking and the authorization-versus-removal transaction; browser expiry is checked independently of links in the middleware tests. Setup instructions are in the README and `.env.example`. No verified deployed Access mapping supplies GitHub identity, so explicit user authorization is used. Real Slack and GitHub callbacks were exercised against stubbed platforms only; a live round trip still needs the apps described in the README.
+
+2026-09-13: Status set to needs-triage: the implementation is complete and awaits maintainer evaluation, as with the other implemented tickets.

.scratch/multiplayer-janitor-implementation/issues/10-platform-recovery.md

@@ -4,20 +4,22 @@
 
 **Blocked by:** 04: Observe sessions and recorded usage in Janitor; 09: Address GitHub review feedback automatically.
 
-**Status:** ready-for-agent
+**Status:** needs-triage
 
 Source: [Accepted implementation handoff](../spec.md). Implement this slice within its accepted contracts and evidence limits.
 
 - [x] Run five-minute rate-aware GitHub retained-delivery scans using App authentication, durable opaque cursors and per-page progress. Filter summaries before payload fetch; preserve exact large attempt IDs separately from delivery GUIDs.
 - [x] Scan known Slack home threads with overlap after the initiating boundary. Deduplicate against accepted/rejected inputs and initial context; apply current authorization only to previously unseen contributions.
-- [ ] Expose overdue scans, incomplete hydration and known unrecoverable gaps. Do not claim recovery of deleted uncaptured text, never-received starts or expired delivery history.
+- [x] Expose overdue scans, incomplete hydration and known unrecoverable gaps. Do not claim recovery of deleted uncaptured text, never-received starts or expired delivery history.
 - [x] Persist Retry-After deadlines and output ordering through process restart. Preserve substantive/final/error output, coalesce pending progress, split valid Unicode within real platform limits and avoid channel-rate floods.
 - [x] Reconcile ambiguous sends by positive marker/author/destination lookup across pages. Missing or deleted markers remain uncertain and cannot authorize reposting.
-- [ ] Loss of home-thread access retains work/output with an independent dashboard delivery warning. Restore the same queued replies when access returns without moving conversation to a DM or another channel.
+- [x] Loss of home-thread access retains work/output with an independent dashboard delivery warning. Restore the same queued replies when access returns without moving conversation to a DM or another channel.
 - [x] Verify actual bot removal/reinvite and retained delivery in a private test channel, with a participant performing platform-only steps. Test redelivery GUID deduplication, expired history limits and local crash/throttle cases; record which are live versus simulated.
 
 ## Comments
 
 2026-09-12: Backend recovery is implemented and committed. Session views expose overdue scans, pending capture, known gaps and independent delivery warnings. Browser visibility remains dependent on ticket 04, whose dashboard is absent in the starting tree. PostgreSQL tests verify retained replies through simulated membership loss and service restart, paginated positive reconciliation and missing/deleted-marker uncertainty. The live check ran last but Slack returned `account_inactive` before any writes. Removal/reinvite acceptance remains blocked on restoration of the fixture app/token. See [validation and evidence](../platform-recovery-validation.md).
 
 2026-09-13: Live removal/reinvite acceptance passed with the replacement bot in private `janitor-test`. Both saved replies retained their output IDs, arrived in their original thread after reinvite, and matched author/thread/metadata markers. The independent session delivery warning cleared. All three test messages were deleted. The remaining unchecked requirements concern browser visibility through ticket 04, not retained delivery or platform membership recovery. See the updated [live evidence](../platform-recovery-live.json).
+
+2026-09-13: Browser visibility is implemented now that ticket 04's dashboard exists. `RecoveryStatus` joins the session detail contract: one entry per platform with last completed scan time, overdue, incomplete, hydrating count, warning and gap, read through the shared recovery query that `AgentSessions.view` also uses. Overdue is a display threshold of two five-minute cycles since the last completed scan, so a healthy scan does not flicker at its own boundary; a Slack thread that is not ready yet is shown as not scanned rather than late or caught up. A pending GitHub payload marks a session incomplete only when it belongs to the session's repository. The GitHub warning falls back to the pending feedback hydration warning due to retry soonest when the scan itself is healthy. Migration `0029_recovery_observation.sql` invalidates the sessions channel when scan completion, warnings, gaps, captured attempts or feedback hydration state actually change; cursors, due times and leases stay silent even though the scans rewrite health columns alongside them. Time-based lateness reaches the browser through the existing fallback refresh. The session detail page gains a Recovery section that states each platform's scan status and its standing gap next to it, and never claims recovery of anything. The independent Slack delivery warning already reached the list and detail views through ticket 04. Tests cover the detail contract across unscanned, not-ready, paginated, hydrating, late and repository-scoped states, invalidation intent for changed health versus rewritten bookkeeping, and the rendered wording. Status is needs-triage for maintainer evaluation, as with the other implemented tickets. Review findings applied: numeric overdue threshold, branded session ID, one pending-hydration CTE, repository-scoped attempts, change-only triggers, and glossary and migration notes.

.scratch/multiplayer-janitor-implementation/platform-recovery-validation.md

@@ -28,7 +28,7 @@ This check does not exercise a deployed Janitor worker, browser dashboard, real
 
 ## Dependency and recovery limits
 
-Ticket 04's paginated dashboard and subscriptions are not implemented in the starting tree. This change exposes recovery health and independent Slack delivery warnings through `AgentSessions.view`; browser visibility remains dependent on ticket 04. Existing feedback rows expose incomplete hydration independently.
+Ticket 04's dashboard landed after the backend recovery work. Recovery health now reaches the browser through the session detail read: `SessionObservation.detail` and `AgentSessions.view` share one recovery query, and the session page shows a Recovery section with each platform's overdue, incomplete, hydrating, warning and gap facts. A scan counts as overdue after two five-minute cycles. The independent Slack delivery warning appears in both the list and the detail view.
 
 Expired GitHub delivery history, deleted uncaptured Slack text, edited originals that were never captured, and never-received start mentions cannot be reconstructed. A missing publication marker does not establish a failed send and never authorizes reposting.
 

CONTEXT.md

@@ -141,6 +141,15 @@ The persisted promise, kept per agent session, to read the runner's durable even
 **Automatic PR review handling**:
 Agent work prompted by an authorized teammate's PR review feedback, with changes and replies made on GitHub without another request in the session's home thread. Outside contributors' feedback requires an authorized teammate's request before the agent acts on it.
 
+**Recovery scan**:
+The periodic, per-platform read of retained GitHub delivery history or of a session's Slack home thread that catches up on feedback Janitor missed. A scan is overdue when it has not completed within two of its five-minute cycles and incomplete while a page or a retained payload is still being fetched; neither says anything was lost.
+
+**Recovery gap**:
+What a recovery scan can never bring back: expired GitHub delivery history, deleted text that was never captured, edited originals that were never captured, and start mentions that never arrived. The dashboard states the gap next to the scan's status rather than claiming recovery.
+
+**Feedback hydration**:
+Fetching the comments of an accepted GitHub review contribution page by page before the agent acts on it. A contribution is hydrating until every page is in; the dashboard counts hydrating contributions and shows the problem the fetch is retrying past.
+
 **Janitor dashboard**:
 The team-wide observation view of agent sessions and their usage, including token usage, independent of home-channel membership. Collaboration with agents takes place in their home threads.
 

apps/cluster/migrations/0029_recovery_observation.sql

@@ -0,0 +1,28 @@
+-- Recovery health on the sessions live channel.
+--
+-- The dashboard states when each platform's recovery scan last completed,
+-- whether it is overdue or incomplete, what it is retrying past, and what it
+-- can never bring back. Those facts change when a scan completes, records a
+-- warning or gap, or captures a retained payload, and when feedback hydration
+-- changes state. Scan cursors, due times and leases are bookkeeping and stay
+-- silent: the scans write them together with health columns on every page,
+-- so each trigger fires only when a health column actually changed. Time-based
+-- lateness has no trigger; the dashboard's fallback refresh picks it up.
+CREATE TRIGGER live_session_recovery AFTER UPDATE ON platform_recovery
+  FOR EACH ROW WHEN (OLD.completed_at IS DISTINCT FROM NEW.completed_at
+    OR OLD.warning IS DISTINCT FROM NEW.warning OR OLD.gap IS DISTINCT FROM NEW.gap)
+  EXECUTE FUNCTION enqueue_session_notification('sessions');
+CREATE TRIGGER live_session_recovery_attempt AFTER INSERT OR DELETE ON github_recovery_attempt
+  FOR EACH ROW EXECUTE FUNCTION enqueue_session_notification('sessions');
+CREATE TRIGGER live_session_recovery_attempt_state AFTER UPDATE ON github_recovery_attempt
+  FOR EACH ROW WHEN (OLD.state IS DISTINCT FROM NEW.state)
+  EXECUTE FUNCTION enqueue_session_notification('sessions');
+CREATE TRIGGER live_session_thread_recovery AFTER UPDATE ON slack_thread
+  FOR EACH ROW WHEN (OLD.recovery_completed_at IS DISTINCT FROM NEW.recovery_completed_at
+    OR OLD.recovery_warning IS DISTINCT FROM NEW.recovery_warning)
+  EXECUTE FUNCTION enqueue_session_notification('sessions');
+CREATE TRIGGER live_session_feedback AFTER INSERT OR DELETE ON github_feedback
+  FOR EACH ROW EXECUTE FUNCTION enqueue_session_notification('sessions');
+CREATE TRIGGER live_session_feedback_state AFTER UPDATE ON github_feedback
+  FOR EACH ROW WHEN (OLD.state IS DISTINCT FROM NEW.state OR OLD.warning IS DISTINCT FROM NEW.warning)
+  EXECUTE FUNCTION enqueue_session_notification('sessions');

apps/cluster/migrations/README.md

@@ -232,3 +232,15 @@ caused it; catch-up reads and delivery cron wakes then forward it. The
 projection's own `freshness_at` heartbeat, thread cursors and leases are
 bookkeeping and do not notify. No data changes; existing sessions appear on the dashboard
 once their next projection read commits.
+
+## Recovery observation
+
+`0029_recovery_observation.sql` puts recovery health on the `sessions` live
+channel. The dashboard states when each platform's recovery scan last
+completed, whether it is overdue or incomplete, what it is retrying past and
+what it can never bring back. Triggers on the GitHub scan record, retained
+delivery attempts, the thread's Slack scan columns and feedback hydration fire
+only when a health column actually changes, because the scans rewrite those
+columns together with cursors, due times and leases on every page. Lateness
+that comes from time alone has no trigger; the browser's fallback refresh
+picks it up. No data changes.

apps/cluster/src/Agent/Observation.ts

@@ -21,6 +21,7 @@ import { homeThreadUrl } from "../Slack/HomeThread.ts"
 import { describeError } from "../SqlErrors.ts"
 import { AgentSessionError, AgentSessionNotFound } from "./Sessions.ts"
 import { AgentSessionId } from "./RunnerProtocol.ts"
+import { recoveryStatus } from "./RecoveryStatus.ts"
 
 export const DEFAULT_PAGE_SIZE = 25
 export const MAX_PAGE_SIZE = 100
@@ -224,6 +225,7 @@ export class SessionObservation extends Context.Service<
           WHERE session_id = ${sessionId} AND state <> 'sent'
           ORDER BY platform
         `.pipe(wrap("detail"))
+        const recovery = yield* recoveryStatus(sql, sessionId).pipe(wrap("detail"))
         const summary = summarize(row)
         // Later accepted work supersedes an older rejection, as it does an older failed turn.
         const latestError =
@@ -239,6 +241,7 @@ export class SessionObservation extends Context.Service<
           lastInputAt: counts.last_input_at,
           latestError,
           pendingDelivery,
+          recovery,
         } satisfies SessionDetail
       })
 

apps/cluster/src/Agent/RecoveryStatus.ts

@@ -0,0 +1,64 @@
+// Recovery health for one session, read the same way by every viewer.
+//
+// The scans own their bookkeeping (cursors, leases, due times); this reads
+// only what a teammate can act on: when a scan last completed, whether it is
+// late, whether its results are all in, what it is retrying past, and what it
+// can never bring back.
+import * as Effect from "effect/Effect"
+import * as Schema from "effect/Schema"
+import type * as SqlClient from "effect/unstable/sql/SqlClient"
+import type { RecoveryStatus } from "@janitor/domain/Agent/Observation"
+import type { AgentSessionId } from "./RunnerProtocol.ts"
+
+/**
+ * Scans run every five minutes; a scan more than two cycles behind is late
+ * enough to say so without flickering at the boundary of a normal cycle.
+ */
+export const OVERDUE_AFTER_SECONDS = 600
+
+/** The Slack scan's standing limit: it can only read what is still in the thread. */
+export const SLACK_GAP =
+  "Deleted uncaptured text and never-received start mentions cannot be recovered"
+
+const Row = Schema.Struct({
+  platform: Schema.Literals(["slack", "github"]),
+  completed_at: Schema.NullOr(Schema.DateTimeUtcFromDate),
+  overdue: Schema.Boolean,
+  incomplete: Schema.Boolean,
+  hydrating: Schema.Int,
+  warning: Schema.NullOr(Schema.String),
+  gap: Schema.NullOr(Schema.String),
+})
+const decodeRows = Schema.decodeUnknownEffect(Schema.Array(Row))
+
+export const recoveryStatus = (
+  sql: SqlClient.SqlClient,
+  sessionId: AgentSessionId,
+): Effect.Effect<ReadonlyArray<RecoveryStatus>, { readonly message: string }> =>
+  sql`
+    WITH hydration AS (
+      -- Feedback contributions whose comments are still being fetched; the
+      -- warning shown is the one due to retry soonest.
+      SELECT COUNT(*)::int AS pending,
+        (array_agg(warning ORDER BY due_at) FILTER (WHERE warning IS NOT NULL))[1] AS warning
+      FROM github_feedback WHERE session_id = ${sessionId} AND state = 'pending'
+    )
+    SELECT 'github' AS platform, r.completed_at,
+      r.completed_at IS NULL OR r.completed_at < CLOCK_TIMESTAMP() - make_interval(secs => ${OVERDUE_AFTER_SECONDS}) AS overdue,
+      r.cursor <> '' OR EXISTS (
+        SELECT 1 FROM github_recovery_attempt a JOIN agent_session s ON s.repository_id = a.repository_id
+        WHERE s.session_id = ${sessionId} AND a.state = 'pending'
+      ) AS incomplete,
+      h.pending AS hydrating, COALESCE(r.warning, h.warning) AS warning, r.gap
+    FROM platform_recovery r, hydration h WHERE r.scan_id = 'github'
+    UNION ALL
+    SELECT 'slack', recovery_completed_at,
+      state = 'ready' AND (recovery_completed_at IS NULL OR recovery_completed_at < CLOCK_TIMESTAMP() - make_interval(secs => ${OVERDUE_AFTER_SECONDS})),
+      recovery_cursor <> '', 0, recovery_warning, ${SLACK_GAP}
+    FROM slack_thread WHERE session_id = ${sessionId}
+  `.pipe(
+    Effect.flatMap(decodeRows),
+    Effect.map((rows) =>
+      rows.map(({ completed_at, ...row }) => ({ ...row, completedAt: completed_at })),
+    ),
+  )

apps/cluster/src/Agent/Sessions.ts

@@ -9,9 +9,11 @@ import * as Effect from "effect/Effect"
 import * as Layer from "effect/Layer"
 import * as Schema from "effect/Schema"
 import * as SqlClient from "effect/unstable/sql/SqlClient"
+import type { RecoveryStatus } from "@janitor/domain/Agent/Observation"
 import { describeError } from "../SqlErrors.ts"
 import { WorkflowOutbox, type WorkflowOutboxError } from "../WorkflowOutbox.ts"
 import { handoffRequest } from "./Handoff.ts"
+import { recoveryStatus } from "./RecoveryStatus.ts"
 import {
   AgentSessionId,
   InputSource,
@@ -99,13 +101,7 @@ export interface AcceptInput {
 }
 
 export interface SessionView {
-  readonly recovery: ReadonlyArray<{
-    platform: string
-    overdue: boolean
-    incomplete: boolean
-    warning: string | null
-    gap: string | null
-  }>
+  readonly recovery: ReadonlyArray<RecoveryStatus>
   readonly slackDelivery: ReadonlyArray<{ id: string; state: string; error: string | null }>
   readonly deliveryWarning: string | null
   readonly feedback: ReadonlyArray<{ key: string; state: string; warning: string | null }>
@@ -342,18 +338,7 @@ export class AgentSessions extends Context.Service<
         }>`SELECT output_id AS id,state,error FROM github_feedback_output WHERE session_id=${sessionId} AND state<>'sent' ORDER BY sequence`.pipe(
           wrap("view"),
         )
-        const recovery = yield* sql<{
-          platform: string
-          overdue: boolean
-          incomplete: boolean
-          warning: string | null
-          gap: string | null
-        }>`
-          SELECT 'github' AS platform,completed_at IS NULL OR completed_at<CLOCK_TIMESTAMP()-interval '5 minutes' AS overdue,
-            cursor<>'' OR EXISTS(SELECT 1 FROM github_recovery_attempt WHERE state='pending') AS incomplete,warning,gap FROM platform_recovery WHERE scan_id='github'
-          UNION ALL SELECT 'slack',recovery_completed_at IS NULL OR recovery_completed_at<CLOCK_TIMESTAMP()-interval '5 minutes',recovery_cursor<>'',recovery_warning,
-            'Deleted uncaptured text and never-received start mentions cannot be recovered' FROM slack_thread WHERE session_id=${sessionId}
-        `.pipe(wrap("view"))
+        const recovery = yield* recoveryStatus(sql, sessionId).pipe(wrap("view"))
         const slackDelivery = yield* sql<{
           id: string
           state: string