From 7fe198bc7fa36860102c9d8f70c4f17c81b607aa Mon Sep 17 00:00:00 2001 From: John Zila Date: Thu, 10 Sep 2026 15:58:27 -0500 Subject: [PATCH] feat(codex): track Codex rate-limit windows and show them on the Usage page Codex CLI writes a `rate_limits` object into its `token_count` events (up to two windows, a short `primary` window and a longer `secondary` one, e.g. 5 hours and 7 days) alongside the plan type and credit balance. agentsview now extracts that object during Codex parsing, persists each observation, and surfaces it on the Usage page as a "Rate limits" section: one card per window showing used percent, time to reset, plan type, credit balance, and a small history chart of used percent over the selected date range. Schema and identity Observations are stored in a new `rate_limit_snapshots` table (SQLite only, alongside cursor_usage_events and the Codex incremental-import tables -- out of scope for the SQLite/PostgreSQL/ DuckDB parity rule). The table is vendor-keyed (`vendor`, `'codex'` for every row today) from the start rather than Codex-specific, because a second PR building on this one adds a Claude rate-limit poller that needs the same table: shipping the vendor-neutral shape now means that PR only adds rows, instead of carrying a migration that renames the table and backfills a `vendor` column for a table that had not even shipped to users yet. `account_id`, `account_label`, `scope_label`, and `details` are reserved the same way for a vendor whose rate-limit source has that shape, and stay empty on every Codex row. Codex rollouts carry no stable per-account identifier -- no account id, user id, email, or org field appears in session_meta or token_count payloads (see docs/internal/session-format-sources.md) -- so a Codex snapshot's identity is (machine, limit_id, plan_type, window_kind) rather than an account. Rows are written with INSERT OR IGNORE against a unique dedup_key (source session id + observed timestamp + limit id + window kind), never a delete-then-reinsert, so both a full parse and an incremental parse converge on the same rows without duplicating or losing history. session_id is nullable (ON DELETE SET NULL) so a row survives its source session being deleted. resets_at is also nullable end to end -- the parser type, the table, and the API response all preserve a genuinely unknown reset time as null rather than flattening it to 0, which previously would have shown as a bogus countdown to the unix epoch. Because this backfills history from existing rollout files, the parser data version is bumped (to 108) so an archive written by an older binary gets a full reparse and picks up the rate_limits history its rollout files already contained, instead of silently staying empty until the next observation. The "current" query (LatestRateLimitSnapshots) resolves "latest" per (vendor, machine, account_id, limit_id, plan_type) bucket, one level above window_kind, and returns every window belonging to that bucket's single newest observation, rather than ranking each window_kind independently: the latter let a window that stopped being reported (e.g. a session moving from primary+secondary windows to primary-only) keep showing its last-known row indefinitely. The winning observation is further disambiguated by a persisted observation_key column, stored once at insert time from the then-current session_id and independent of the nullable session_id foreign key: two different sessions on one machine and account can legitimately report the same limit/plan at the exact same instant (Codex's rate limit is account-wide), and using session_id itself for this at query time broke either when two such sessions were both deleted (their now-NULL session_ids would compare equal again) or, if that NULL case was special-cased per row instead, when one deleted session's own primary+secondary windows could no longer be told apart from each other. A key fixed at write time and never recomputed avoids both failure modes. The history endpoint downsamples a wide date range instead of returning every matching observation: `max_points` (default 500) caps the response, dividing the range into that many equal-width time buckets and keeping only the most recently observed row per bucket, so a long-lived window's full history does not grow the response (or RateLimitHistoryChart's point count) unboundedly. The chart itself requests a smaller point budget sized for its own sparkline-sized rendering. API and frontend New routes `GET /api/v1/rate-limits/current` and `.../history` accept `vendor`, `account_id`, and `machine` filters (`agent` is kept as a deprecated alias for `vendor`, matching the shared session filter's comma-separated selection semantics via `RateLimitAgentMatchesVendor`); `history` additionally accepts `limit_id`, `window`, `since`, `until`, and `max_points`. PostgreSQL and DuckDB implement the read side as no-ops, so the section is simply hidden when either backend is the active read store. The frontend groups cards by vendor and then by account; since Codex reports no account identity, its account group is keyed by machine instead, so two machines syncing the same account still render as two groups. A window with no known reset time shows "Reset time unknown" instead of a countdown. The history chart now plots observed time on a real time scale (it previously used a categorical point scale that spaced every observation evenly by index, which visually flattened bursts and gaps in the actual observation cadence). Robustness fixes folded in from post-implementation review - A full resync now copies existing rate_limit_snapshots rows into the replacement archive, the same way model pricing is copied, so previously observed windows survive the swap instead of going empty until the next observation. This copy runs after orphaned sessions are restored, not before: session_id is a foreign key, and a snapshot belonging to an orphaned session would otherwise violate it during the copy (INSERT OR IGNORE does not suppress a foreign-key violation the way it suppresses a duplicate), silently losing every row in the table rather than just that session's. The copy also NULLs session_id for any row whose session still does not exist in the destination even after orphan restoration -- one superseded by a reparse under a different id, or one excluded as parser-excluded -- rather than copying that id unchanged and hitting the same constraint; dedup_key and observation_key are preserved from the source unchanged either way. This history cannot be reconstructed once lost, so a failed copy aborts the swap instead of merely warning. - A rate_limits payload missing limit_id is skipped rather than failing the whole write -- since this write is normally part of a larger session write, the old behavior could fail an entire session's ingestion over one malformed rate-limit entry. - The machine and agent/vendor filters use the same comma-separated IN/contains semantics as the rest of the shared session filters, and rate-limit history queries parse and normalize since/until as RFC3339Nano rather than comparing raw request strings against the stored UTC column. - The incremental-parse cursor seed (a prefix scan that reconstructs cursor state without ever returning a session result) no longer accumulates rate-limit observations while scanning: they were never read out of that path, so collecting them grew memory with the whole scanned prefix on every incremental-parse cache miss instead of staying bounded. docs/agents/storage.md and docs/token-usage.md are updated to describe the new table and endpoints. --- docs/agents/storage.md | 112 ++ docs/internal/session-format-sources.md | 24 + docs/token-usage.md | 42 + frontend/messages/en.json | 16 +- frontend/messages/fr.json | 16 +- frontend/messages/ja.json | 16 +- frontend/messages/ko.json | 16 +- frontend/messages/zh-CN.json | 16 +- frontend/messages/zh-TW.json | 16 +- frontend/src/lib/api/generated/index.ts | 1 + .../models/getApiV1RateLimitsCurrentParams.ts | 23 + .../models/getApiV1RateLimitsCurrentVendor.ts | 10 + .../models/getApiV1RateLimitsHistoryParams.ts | 46 + .../models/getApiV1RateLimitsHistoryVendor.ts | 10 + .../models/getApiV1RateLimitsHistoryWindow.ts | 11 + .../src/lib/api/generated/models/index.ts | 6 + .../models/serviceRateLimitWindow.ts | 25 + .../api/generated/rate-limits/rate-limits.ts | 68 ++ .../ratelimits/RateLimitCard.svelte | 252 +++++ .../ratelimits/RateLimitHistoryChart.svelte | 77 ++ .../ratelimits/RateLimitHistoryChart.test.ts | 50 + .../ratelimits/RateLimitsSection.svelte | 148 +++ .../ratelimits/RateLimitsSection.test.ts | 86 ++ .../src/lib/components/usage/UsagePage.svelte | 13 +- .../src/lib/stores/ratelimits.svelte.test.ts | 100 ++ frontend/src/lib/stores/ratelimits.svelte.ts | 244 +++++ frontend/src/lib/utils/rateLimitFormat.ts | 66 ++ internal/db/db.go | 15 +- internal/db/messages.go | 5 + internal/db/rate_limit_snapshots.go | 998 ++++++++++++++++++ internal/db/rate_limit_snapshots_test.go | 272 +++++ internal/db/read_only_test.go | 11 + internal/db/schema.sql | 98 ++ internal/db/session_batch.go | 19 + internal/db/sessions.go | 6 + internal/db/store.go | 5 + internal/duckdb/rate_limits.go | 29 + internal/parser/codex.go | 175 ++- internal/parser/codex_cursor.go | 17 +- internal/parser/codex_parser_test.go | 21 + internal/parser/codex_provider.go | 13 + internal/parser/provider.go | 3 + internal/parser/traex_test.go | 13 + internal/parser/types.go | 67 +- internal/postgres/rate_limits.go | 28 + internal/server/huma_route_groups.go | 1 + internal/server/huma_routes_ratelimits.go | 93 ++ internal/service/rate_limits.go | 169 +++ internal/service/rate_limits_test.go | 58 + internal/sync/codex_staging.go | 1 + internal/sync/engine.go | 203 +++- internal/sync/engine_integration_test.go | 58 + internal/sync/engine_staged_contract_test.go | 2 +- internal/sync/engine_test.go | 8 +- internal/sync/parsediff.go | 1 + .../sync/rate_limit_snapshot_write_test.go | 240 +++++ internal/sync/s3.go | 5 + internal/testjsonl/testjsonl.go | 76 ++ 58 files changed, 4181 insertions(+), 39 deletions(-) create mode 100644 frontend/src/lib/api/generated/models/getApiV1RateLimitsCurrentParams.ts create mode 100644 frontend/src/lib/api/generated/models/getApiV1RateLimitsCurrentVendor.ts create mode 100644 frontend/src/lib/api/generated/models/getApiV1RateLimitsHistoryParams.ts create mode 100644 frontend/src/lib/api/generated/models/getApiV1RateLimitsHistoryVendor.ts create mode 100644 frontend/src/lib/api/generated/models/getApiV1RateLimitsHistoryWindow.ts create mode 100644 frontend/src/lib/api/generated/models/serviceRateLimitWindow.ts create mode 100644 frontend/src/lib/api/generated/rate-limits/rate-limits.ts create mode 100644 frontend/src/lib/components/ratelimits/RateLimitCard.svelte create mode 100644 frontend/src/lib/components/ratelimits/RateLimitHistoryChart.svelte create mode 100644 frontend/src/lib/components/ratelimits/RateLimitHistoryChart.test.ts create mode 100644 frontend/src/lib/components/ratelimits/RateLimitsSection.svelte create mode 100644 frontend/src/lib/components/ratelimits/RateLimitsSection.test.ts create mode 100644 frontend/src/lib/stores/ratelimits.svelte.test.ts create mode 100644 frontend/src/lib/stores/ratelimits.svelte.ts create mode 100644 frontend/src/lib/utils/rateLimitFormat.ts create mode 100644 internal/db/rate_limit_snapshots.go create mode 100644 internal/db/rate_limit_snapshots_test.go create mode 100644 internal/duckdb/rate_limits.go create mode 100644 internal/postgres/rate_limits.go create mode 100644 internal/server/huma_routes_ratelimits.go create mode 100644 internal/service/rate_limits.go create mode 100644 internal/service/rate_limits_test.go create mode 100644 internal/sync/rate_limit_snapshot_write_test.go diff --git a/docs/agents/storage.md b/docs/agents/storage.md index 95e49d694e..0b78e507ab 100644 --- a/docs/agents/storage.md +++ b/docs/agents/storage.md @@ -73,6 +73,118 @@ hash state can contain raw trailing transcript bytes. They retain staged parsing but publish projected messages and tool metadata without staged output. Late result updates use the same projection as newly inserted messages. +### Rate-limit snapshots + +`rate_limit_snapshots` is a SQLite-only vendor-data table, in the same +category as `cursor_usage_events` and the four Codex incremental-import +tables above: it is out of scope for the SQLite/PostgreSQL/DuckDB parity +rule below. It is vendor-keyed (`vendor`, `'codex'` today) from the start +so a future vendor can add rows without a schema change or a migration: +`account_id`, `account_label`, `scope_label`, and `details` are reserved +for a vendor whose rate-limit source has that shape, and stay `''` on +every Codex row -- verified against `~/.codex/sessions` rollouts: no +account id, user id, email, or org field appears in `session_meta` or +`token_count` payloads, so a Codex snapshot's identity omits an account +entirely (see `docs/internal/session-format-sources.md`). An +`account_id` filter scopes vendors that have +accounts, so both `LatestRateLimitSnapshots` and +`RateLimitSnapshotHistory` match a nonempty `account_id` against +`(account_id = '' OR account_id = ?)` rather than a bare equality, +letting an account-less vendor's rows (Codex today) pass through +instead of being excluded by someone else's account filter. + +It stores each `rate_limits` observation a Codex `token_count` event +carries beside `info.last_token_usage` (one row per rate-limit window). +It is written with an upsert against a unique `dedup_key` (source +session id + observed timestamp + limit id + window kind + `ordinal`), +never a delete-then-reinsert, so both a full parse (which sees the +whole transcript every time) and an incremental parse (which only sees +the appended tail) can write to it without duplicating or losing rows. +A `dedup_key` collision against a row whose `session_id` is already +NULL (a resync copy for a session absent from the destination -- see +`CopyRateLimitSnapshotsFrom`) reattaches that row to the incoming +session and refreshes its other fields instead of being ignored, since +the copy preserves `dedup_key` unchanged and the session reappearing +later (a fresh parse reproducing the identical key) would otherwise +collide with, and lose to, the stale detached row forever. A collision +against a row that already has a session attached is still a no-op. +`ordinal` is the source `token_count` event's 0-based +position among every `token_count` event in the rollout file, assigned +by the parser (`ParsedRateLimitSnapshot.Ordinal`) and stable across a +full parse, an incremental tail parse resuming from a cached or +reseeded cursor, and any later re-parse of the same file: without it, +two distinct `token_count` events landing on the same observed-at +second would produce the same `dedup_key`, and the second event's row +would be silently dropped by `INSERT OR IGNORE` instead of persisted. +It also participates in `observation_key` for the same reason -- two +such events would otherwise merge their sibling windows into one +`LatestRateLimitSnapshots` bucket. A single malformed +observation (e.g. one missing `limit_id`) is skipped rather than failing +the whole write, so it cannot take down the rest of the batch or the +session ingestion it rode in on. `session_id` is nullable (`ON DELETE SET +NULL`) so a row survives its source session being deleted. `resets_at` is +also nullable: Codex can report a window with no reset time, and that is +kept distinct from a window that resets at the unix epoch all the way +through the Go types and the API response (an absent field, not `0`), so +the Usage page can tell "no known reset time" apart from "resets right +now". A full resync copies existing rows into the replacement archive +the same way model pricing is copied (`CopyRateLimitSnapshotsFrom`), but +only after orphaned sessions are restored: `session_id` is a foreign +key, and copying before restoration could violate it for a snapshot +belonging to an orphaned session, aborting the whole copy. The copy also +NULLs `session_id` for any row whose session does not exist in the +destination even after restoration -- a session resync intentionally +does not restore, such as one superseded by a reparse under a different +id, or one excluded as parser-excluded -- rather than copying that id +unchanged and violating the same foreign key; `dedup_key` is preserved +from the source unchanged either way. The copy also skips any row whose +session was rebuilt by the resync's own reparse rather than merely +restored: it copies a row only when `session_id` is NULL, absent from +the destination's `sessions` table, or one of the ids the orphan copy +restored without reparsing, so a rebuilt session's superseded rows +cannot resurrect on top of its fresh, current ones. +`LatestRateLimitSnapshots` (the +`/current` endpoint) resolves "latest" per (vendor, machine, account_id, +limit_id) bucket -- one level above window_kind -- and returns every +window belonging to that bucket's single newest observation. Ranking +each window_kind independently instead would let a window that stops +being reported (e.g. a session moving from primary+secondary to +primary-only) keep surfacing its last-known row forever, since no newer +row for that window_kind ever arrives to supersede it. `plan_type` is +deliberately not part of this bucket, or of any other window identity in +this codebase: Codex reports it as a label that can flip between +`"pro"` and empty for the same window from one observation to the next, +not a stable identity component, so partitioning on it would let a stale +plan-keyed bucket coexist alongside the newest observation instead of +being superseded by it, and (on the history/frontend side) would split +one window's history across two chart series or silently drop half of +it. `plan_type` and `limit_name` are instead resolved independently as +the latest non-empty value ever observed for the bucket -- so a bucket +whose newest observation happens to omit one or both labels still +displays the last value seen for it rather than blanking the card -- +and are otherwise pure display labels carried on each row, never a +grouping key. `RateLimitSnapshotHistory` still returns every row over +time regardless of the current-snapshot grouping, and its query, +downsampling, and frontend cache key never filter or key by `plan_type` +either. A window's identity, for both the current-snapshot and history +paths, is (vendor, machine, account_id, limit_id, window_kind) -- the +same fields `RateLimitCardIdentity` on the frontend groups by. PostgreSQL +and DuckDB implement the read-side +`Store` methods as no-ops returning an empty result, so the Usage page's +rate-limits section is simply hidden when either backend is the active +read store. `LatestRateLimitSnapshots` and `RateLimitSnapshotHistory` both +probe once (cached per `*DB`) whether `rate_limit_snapshots` exists and +return an empty result instead of erroring when it does not, since +`OpenReadOnly` tolerates an older, otherwise-compatible archive that +predates the table. A write that replaces a session's messages wholesale +(an authoritative reparse superseding a fallback parsed at +`parser.DataVersionNeedsRetry`, or any other full delete-and-reinsert) +must delete that session's `rate_limit_snapshots` rows in the same +transaction before inserting the new set via +`InsertRateLimitSnapshotsReplacingSession`, while a normal incremental +parse keeps appending through `InsertRateLimitSnapshots` without +deleting. + ## Archive Content Policy `archive_content` (`internal/config.ArchiveContent`) narrows what the SQLite diff --git a/docs/internal/session-format-sources.md b/docs/internal/session-format-sources.md index 59f9cb63d8..1985dc9351 100644 --- a/docs/internal/session-format-sources.md +++ b/docs/internal/session-format-sources.md @@ -455,6 +455,30 @@ add an archived or maintained mirror without replacing the original identity. [LiteLLM's Bedrock row](https://github.com/BerriAI/litellm/blob/fbed17d567a62b14b8fc7d9ef13c5cd61a8d1ae0/model_prices_and_context_window.json). Remove that supplemental row when the shared snapshot includes it. +- **Rate limits (2026-09-09):** `token_count` events carry a `rate_limits` + object beside `info.last_token_usage`, confirmed against the pinned + [protocol types](https://github.com/openai/codex/blob/406dc9239492aff6d295cca5eebe2a548548d42f/codex-rs/protocol/src/protocol.rs): + a `RateLimitSnapshot` (`limit_id`, `limit_name`, `primary`/`secondary` + `RateLimitWindow`s, `credits`, `plan_type`, `rate_limit_reached_type`, + `individual_limit`, `spend_control_reached`) nested in `TokenCountEvent`. + Each `RateLimitWindow` is `{used_percent: f64, window_minutes: Option, + resets_at: Option}`; `CreditsSnapshot` is `{has_credits: bool, + unlimited: bool, balance: Option}`. `rate_limits` itself, and both + windows independently, are nullable — observed as null for events with no + rate-limit data and for limit ids (e.g. `premium`) that report credits only. + Cross-checked against a local corpus of 218,803 `token_count` events from + 2026-09 rollouts: 216,394 carried only a `primary` window, 2,408 carried + both `primary` and `secondary` (e.g. a 5h window plus a 7-day window), and + 1 carried neither. No account, user, or organization identifier field + appears anywhere in this protocol file, nor in any `session_meta` or + `token_count` payload across that corpus (also grepped for `account_id`, + `chatgpt_account`, `account_email`, `user_email`, and `org_id`; every hit + was inside transcript content, not a real identity field) — so + `rate_limit_snapshots` keys a Codex snapshot by (machine, limit_id, + window_kind) with an `account_id` column reserved for a future + Codex release that starts reporting one. See + `internal/db/schema.sql` and `docs/token-usage.md`. + - **Agentsview:** `internal/parser/codex.go` and `internal/parser/codex_provider.go`; usage is taken from the last-turn counters rather than repeatedly counting cumulative totals. Fork and diff --git a/docs/token-usage.md b/docs/token-usage.md index 2d60cd36d4..10d071851e 100644 --- a/docs/token-usage.md +++ b/docs/token-usage.md @@ -232,6 +232,48 @@ cache creation without earning the reads back. ![Cache efficiency panel](/docs/assets/generated/screenshots/usage-cache-efficiency.png) +### Rate Limits + +When the archive contains sessions from a vendor whose rate limits agentsview +tracks (Codex CLI today), the Usage page adds a **Rate limits** section below +the summary cards, grouped by vendor and then by account (Codex reports no +account identity, so its group is keyed by machine instead): one card per +rate-limit window (Codex reports up to two — a short `primary` window and a +longer `secondary` window, e.g. 5 hours and 7 days), showing how much of the +window is used, when it resets, the plan type, and the current credit +balance, plus a small history chart of used-percent over the selected date +range, plotted on a real time axis so the spacing between points reflects how +much time actually elapsed. The section is hidden entirely for archives with +no rate-limit data. + +Codex CLI writes a `rate_limits` object into its `token_count` events; +agentsview persists each observation into a vendor-keyed +`rate_limit_snapshots` table (SQLite only — see `docs/agents/storage.md`) and +serves it over: + +```http +GET /api/v1/rate-limits/current +GET /api/v1/rate-limits/history +``` + +Both accept `vendor`, `account_id`, and `machine` filters (`agent` is a +deprecated alias for `vendor`); `history` additionally accepts `limit_id`, +`window`, `since`, `until`, and `max_points` (default 500), but not +`plan_type`: a window's identity for history purposes is (vendor, +machine, account_id, limit_id, window_kind), and the returned series +(and its downsampling) covers the whole window regardless of the +`plan_type` label each observation happens to carry. A `history` +request spanning more observations than `max_points` is downsampled: the +range is divided into `max_points` equal-width time buckets and only the +most recently observed row in each bucket is kept, so a wide date range +does not grow the response (or the chart's point count) unboundedly. +Codex rollouts do not currently report a +stable per-account identifier, so Codex snapshots are grouped by machine, +`limit_id`, and window kind rather than by account — see the Codex +entry in `docs/internal/session-format-sources.md` for the evidence and +`internal/db/schema.sql` for the exact column set, including the columns +reserved for a future account-keyed vendor. + The dashboard reads from the same `model_pricing` table that backs the CLI commands below, so the numbers line up exactly with what `agentsview usage daily` prints. diff --git a/frontend/messages/en.json b/frontend/messages/en.json index 71c3ee6a41..a60e0b98e3 100644 --- a/frontend/messages/en.json +++ b/frontend/messages/en.json @@ -2038,5 +2038,19 @@ "settings_tool_result_images_keep": "Keep", "settings_tool_result_images_drop": "Drop", "settings_tool_result_images_hint": "Stored results keep the policy they were written under, and switching back to Keep cannot restore payloads already removed. Run agentsview db strip --images to project existing rows.", - "settings_tool_result_images_restart_notice": "Restart the AgentsView daemon to apply this change to newly ingested sessions." + "settings_tool_result_images_restart_notice": "Restart the AgentsView daemon to apply this change to newly ingested sessions.", + "rate_limits_section_title": "Rate limits", + "rate_limits_plan_label": "Plan", + "rate_limits_credits_label": "Credits", + "rate_limits_credits_unlimited": "Unlimited", + "rate_limits_resets_in": "Resets in {value}", + "rate_limits_resets_now": "Resets now", + "rate_limits_used_percent_aria": "Rate limit usage", + "rate_limits_history_aria": "Rate limit usage over time", + "rate_limits_window_weekly": "Weekly limit", + "rate_limits_window_session": "Session limit", + "rate_limits_window_generic": "{duration} limit", + "rate_limits_card_header": "{window} [{name}]", + "rate_limits_vendor_codex": "Codex", + "rate_limits_resets_unknown": "Reset time unknown" } diff --git a/frontend/messages/fr.json b/frontend/messages/fr.json index 99a955880a..fd5f0a21f7 100644 --- a/frontend/messages/fr.json +++ b/frontend/messages/fr.json @@ -2037,5 +2037,19 @@ "settings_tool_result_images_keep": "Conserver", "settings_tool_result_images_drop": "Supprimer", "settings_tool_result_images_hint": "Les résultats déjà stockés gardent la règle appliquée lors de leur écriture, et revenir à Conserver ne restaure pas les données déjà supprimées. Lancez agentsview db strip --images pour traiter les lignes existantes.", - "settings_tool_result_images_restart_notice": "Redémarrez le démon AgentsView pour appliquer ce changement aux nouvelles sessions ingérées." + "settings_tool_result_images_restart_notice": "Redémarrez le démon AgentsView pour appliquer ce changement aux nouvelles sessions ingérées.", + "rate_limits_section_title": "Limites de débit", + "rate_limits_plan_label": "Forfait", + "rate_limits_credits_label": "Crédits", + "rate_limits_credits_unlimited": "Illimité", + "rate_limits_resets_in": "Réinitialisation dans {value}", + "rate_limits_resets_now": "Réinitialisation immédiate", + "rate_limits_used_percent_aria": "Utilisation de la limite de débit", + "rate_limits_history_aria": "Historique d'utilisation de la limite de débit", + "rate_limits_window_weekly": "Limite hebdomadaire", + "rate_limits_window_session": "Limite de session", + "rate_limits_window_generic": "Limite {duration}", + "rate_limits_card_header": "{window} [{name}]", + "rate_limits_vendor_codex": "Codex", + "rate_limits_resets_unknown": "Heure de réinitialisation inconnue" } diff --git a/frontend/messages/ja.json b/frontend/messages/ja.json index ddd02bba9b..769fa4b314 100644 --- a/frontend/messages/ja.json +++ b/frontend/messages/ja.json @@ -2038,5 +2038,19 @@ "settings_tool_result_images_keep": "保持", "settings_tool_result_images_drop": "破棄", "settings_tool_result_images_hint": "保存済みの結果は書き込み時のポリシーを保ち、保持に戻しても削除済みのデータは復元できません。既存の行は agentsview db strip --images で処理してください。", - "settings_tool_result_images_restart_notice": "この変更を新しく取り込むセッションに適用するには、AgentsView デーモンを再起動してください。" + "settings_tool_result_images_restart_notice": "この変更を新しく取り込むセッションに適用するには、AgentsView デーモンを再起動してください。", + "rate_limits_section_title": "レート制限", + "rate_limits_plan_label": "プラン", + "rate_limits_credits_label": "クレジット", + "rate_limits_credits_unlimited": "無制限", + "rate_limits_resets_in": "{value}後にリセット", + "rate_limits_resets_now": "今すぐリセット", + "rate_limits_used_percent_aria": "レート制限の使用状況", + "rate_limits_history_aria": "レート制限の使用状況の推移", + "rate_limits_window_weekly": "週間上限", + "rate_limits_window_session": "セッション上限", + "rate_limits_window_generic": "{duration}上限", + "rate_limits_card_header": "{window} [{name}]", + "rate_limits_vendor_codex": "Codex", + "rate_limits_resets_unknown": "リセット時刻不明" } diff --git a/frontend/messages/ko.json b/frontend/messages/ko.json index cd8cf0f040..a499cb8970 100644 --- a/frontend/messages/ko.json +++ b/frontend/messages/ko.json @@ -1973,5 +1973,19 @@ "settings_tool_result_images_keep": "유지", "settings_tool_result_images_drop": "삭제", "settings_tool_result_images_hint": "이미 저장된 결과는 기록될 당시의 정책을 유지하며, 유지로 되돌려도 이미 삭제된 데이터는 복원되지 않습니다. 기존 행은 agentsview db strip --images 로 처리합니다.", - "settings_tool_result_images_restart_notice": "이 변경을 새로 수집되는 세션에 적용하려면 AgentsView 데몬을 다시 시작하세요." + "settings_tool_result_images_restart_notice": "이 변경을 새로 수집되는 세션에 적용하려면 AgentsView 데몬을 다시 시작하세요.", + "rate_limits_section_title": "속도 제한", + "rate_limits_plan_label": "요금제", + "rate_limits_credits_label": "크레딧", + "rate_limits_credits_unlimited": "무제한", + "rate_limits_resets_in": "{value} 후 초기화", + "rate_limits_resets_now": "지금 초기화", + "rate_limits_used_percent_aria": "속도 제한 사용량", + "rate_limits_history_aria": "속도 제한 사용량 추이", + "rate_limits_window_weekly": "주간 한도", + "rate_limits_window_session": "세션 한도", + "rate_limits_window_generic": "{duration} 한도", + "rate_limits_card_header": "{window} [{name}]", + "rate_limits_vendor_codex": "Codex", + "rate_limits_resets_unknown": "초기화 시간 알 수 없음" } diff --git a/frontend/messages/zh-CN.json b/frontend/messages/zh-CN.json index cbc6e983f6..2e2b5e598e 100644 --- a/frontend/messages/zh-CN.json +++ b/frontend/messages/zh-CN.json @@ -1973,5 +1973,19 @@ "settings_tool_result_images_keep": "保留", "settings_tool_result_images_drop": "丢弃", "settings_tool_result_images_hint": "已存储的结果保持写入时的策略,改回「保留」也无法恢复已删除的数据。运行 agentsview db strip --images 可处理已有记录。", - "settings_tool_result_images_restart_notice": "重启 AgentsView 守护进程后,此更改才会应用到新采集的会话。" + "settings_tool_result_images_restart_notice": "重启 AgentsView 守护进程后,此更改才会应用到新采集的会话。", + "rate_limits_section_title": "速率限制", + "rate_limits_plan_label": "套餐", + "rate_limits_credits_label": "余额", + "rate_limits_credits_unlimited": "不限量", + "rate_limits_resets_in": "{value} 后重置", + "rate_limits_resets_now": "现在重置", + "rate_limits_used_percent_aria": "速率限制使用情况", + "rate_limits_history_aria": "速率限制使用趋势", + "rate_limits_window_weekly": "每周限额", + "rate_limits_window_session": "会话限额", + "rate_limits_window_generic": "{duration} 限额", + "rate_limits_card_header": "{window} [{name}]", + "rate_limits_vendor_codex": "Codex", + "rate_limits_resets_unknown": "重置时间未知" } diff --git a/frontend/messages/zh-TW.json b/frontend/messages/zh-TW.json index 0663332de2..77209014b6 100644 --- a/frontend/messages/zh-TW.json +++ b/frontend/messages/zh-TW.json @@ -1973,5 +1973,19 @@ "settings_tool_result_images_keep": "保留", "settings_tool_result_images_drop": "捨棄", "settings_tool_result_images_hint": "已儲存的結果維持寫入時的政策,改回「保留」也無法還原已刪除的資料。執行 agentsview db strip --images 可處理既有記錄。", - "settings_tool_result_images_restart_notice": "重新啟動 AgentsView 常駐程式後,此變更才會套用到新擷取的對話。" + "settings_tool_result_images_restart_notice": "重新啟動 AgentsView 常駐程式後,此變更才會套用到新擷取的對話。", + "rate_limits_section_title": "速率限制", + "rate_limits_plan_label": "方案", + "rate_limits_credits_label": "餘額", + "rate_limits_credits_unlimited": "無限制", + "rate_limits_resets_in": "{value} 後重置", + "rate_limits_resets_now": "現在重置", + "rate_limits_used_percent_aria": "速率限制使用情況", + "rate_limits_history_aria": "速率限制使用趨勢", + "rate_limits_window_weekly": "每週限額", + "rate_limits_window_session": "工作階段限額", + "rate_limits_window_generic": "{duration} 限額", + "rate_limits_card_header": "{window} [{name}]", + "rate_limits_vendor_codex": "Codex", + "rate_limits_resets_unknown": "重置時間未知" } diff --git a/frontend/src/lib/api/generated/index.ts b/frontend/src/lib/api/generated/index.ts index c34fc68a2d..4bb4da5122 100644 --- a/frontend/src/lib/api/generated/index.ts +++ b/frontend/src/lib/api/generated/index.ts @@ -11,6 +11,7 @@ export * as MetadataService from "./metadata/metadata.ts"; export * as OpenersService from "./openers/openers.ts"; export * as PinsService from "./pins/pins.ts"; export * as PushService from "./push/push.ts"; +export * as RateLimitsService from "./rate-limits/rate-limits.ts"; export * as RawSyncService from "./raw-sync/raw-sync.ts"; export * as RecentEditsService from "./recent-edits/recent-edits.ts"; export * as RemoteSyncService from "./remote-sync/remote-sync.ts"; diff --git a/frontend/src/lib/api/generated/models/getApiV1RateLimitsCurrentParams.ts b/frontend/src/lib/api/generated/models/getApiV1RateLimitsCurrentParams.ts new file mode 100644 index 0000000000..0f0a921cf4 --- /dev/null +++ b/frontend/src/lib/api/generated/models/getApiV1RateLimitsCurrentParams.ts @@ -0,0 +1,23 @@ +/** + * Generated by Orval. Do not edit manually. + */ +import type { GetApiV1RateLimitsCurrentVendor } from "./getApiV1RateLimitsCurrentVendor.ts"; + +export type GetApiV1RateLimitsCurrentParams = { + /** + * Filter by vendor + */ + vendor?: GetApiV1RateLimitsCurrentVendor; + /** + * Filter by account id; scopes vendors that have accounts and never excludes an account-less vendor's rows (Codex today) + */ + account_id?: string; + /** + * Filter by machine (comma-separated) + */ + machine?: string; + /** + * Deprecated alias for vendor (comma-separated) + */ + agent?: string; +}; diff --git a/frontend/src/lib/api/generated/models/getApiV1RateLimitsCurrentVendor.ts b/frontend/src/lib/api/generated/models/getApiV1RateLimitsCurrentVendor.ts new file mode 100644 index 0000000000..7b022366a9 --- /dev/null +++ b/frontend/src/lib/api/generated/models/getApiV1RateLimitsCurrentVendor.ts @@ -0,0 +1,10 @@ +/** + * Generated by Orval. Do not edit manually. + */ + +export type GetApiV1RateLimitsCurrentVendor = + (typeof GetApiV1RateLimitsCurrentVendor)[keyof typeof GetApiV1RateLimitsCurrentVendor]; + +export const GetApiV1RateLimitsCurrentVendor = { + codex: "codex", +} as const; diff --git a/frontend/src/lib/api/generated/models/getApiV1RateLimitsHistoryParams.ts b/frontend/src/lib/api/generated/models/getApiV1RateLimitsHistoryParams.ts new file mode 100644 index 0000000000..d742f5a9aa --- /dev/null +++ b/frontend/src/lib/api/generated/models/getApiV1RateLimitsHistoryParams.ts @@ -0,0 +1,46 @@ +/** + * Generated by Orval. Do not edit manually. + */ +import type { GetApiV1RateLimitsHistoryVendor } from "./getApiV1RateLimitsHistoryVendor.ts"; +import type { GetApiV1RateLimitsHistoryWindow } from "./getApiV1RateLimitsHistoryWindow.ts"; + +export type GetApiV1RateLimitsHistoryParams = { + /** + * Filter by vendor + */ + vendor?: GetApiV1RateLimitsHistoryVendor; + /** + * Filter by account id; scopes vendors that have accounts and never excludes an account-less vendor's rows (Codex today) + */ + account_id?: string; + /** + * Filter by machine (comma-separated) + */ + machine?: string; + /** + * Deprecated alias for vendor (comma-separated) + */ + agent?: string; + /** + * Filter by limit id (e.g. codex) + */ + limit_id?: string; + /** + * Filter by rate-limit window kind + */ + window?: GetApiV1RateLimitsHistoryWindow; + /** + * Return snapshots observed at or after this RFC3339 timestamp + */ + since?: string; + /** + * Return snapshots observed strictly before this RFC3339 timestamp + */ + until?: string; + /** + * Maximum points returned; a wider range is downsampled to this many, keeping the most recent observation per time bucket + * @minimum 1 + * @maximum 2000 + */ + max_points?: number; +}; diff --git a/frontend/src/lib/api/generated/models/getApiV1RateLimitsHistoryVendor.ts b/frontend/src/lib/api/generated/models/getApiV1RateLimitsHistoryVendor.ts new file mode 100644 index 0000000000..9ca8c3724b --- /dev/null +++ b/frontend/src/lib/api/generated/models/getApiV1RateLimitsHistoryVendor.ts @@ -0,0 +1,10 @@ +/** + * Generated by Orval. Do not edit manually. + */ + +export type GetApiV1RateLimitsHistoryVendor = + (typeof GetApiV1RateLimitsHistoryVendor)[keyof typeof GetApiV1RateLimitsHistoryVendor]; + +export const GetApiV1RateLimitsHistoryVendor = { + codex: "codex", +} as const; diff --git a/frontend/src/lib/api/generated/models/getApiV1RateLimitsHistoryWindow.ts b/frontend/src/lib/api/generated/models/getApiV1RateLimitsHistoryWindow.ts new file mode 100644 index 0000000000..5f0e2bc62b --- /dev/null +++ b/frontend/src/lib/api/generated/models/getApiV1RateLimitsHistoryWindow.ts @@ -0,0 +1,11 @@ +/** + * Generated by Orval. Do not edit manually. + */ + +export type GetApiV1RateLimitsHistoryWindow = + (typeof GetApiV1RateLimitsHistoryWindow)[keyof typeof GetApiV1RateLimitsHistoryWindow]; + +export const GetApiV1RateLimitsHistoryWindow = { + primary: "primary", + secondary: "secondary", +} as const; diff --git a/frontend/src/lib/api/generated/models/index.ts b/frontend/src/lib/api/generated/models/index.ts index 534b1be1f4..0bbb81c432 100644 --- a/frontend/src/lib/api/generated/models/index.ts +++ b/frontend/src/lib/api/generated/models/index.ts @@ -235,6 +235,11 @@ export * from "./getApiV1InsightsType.ts"; export * from "./getApiV1MachinesParams.ts"; export * from "./getApiV1PinsParams.ts"; export * from "./getApiV1ProjectsParams.ts"; +export * from "./getApiV1RateLimitsCurrentParams.ts"; +export * from "./getApiV1RateLimitsCurrentVendor.ts"; +export * from "./getApiV1RateLimitsHistoryParams.ts"; +export * from "./getApiV1RateLimitsHistoryVendor.ts"; +export * from "./getApiV1RateLimitsHistoryWindow.ts"; export * from "./getApiV1RecentEditsParams.ts"; export * from "./getApiV1SearchContentMode.ts"; export * from "./getApiV1SearchContentParams.ts"; @@ -343,6 +348,7 @@ export * from "./resumeResponse.ts"; export * from "./searchResponse.ts"; export * from "./serviceContentSearchResult.ts"; export * from "./serviceMessageList.ts"; +export * from "./serviceRateLimitWindow.ts"; export * from "./serviceSecretFindingList.ts"; export * from "./serviceSessionDetail.ts"; export * from "./serviceSessionDetailHealthPenalties.ts"; diff --git a/frontend/src/lib/api/generated/models/serviceRateLimitWindow.ts b/frontend/src/lib/api/generated/models/serviceRateLimitWindow.ts new file mode 100644 index 0000000000..82b1965078 --- /dev/null +++ b/frontend/src/lib/api/generated/models/serviceRateLimitWindow.ts @@ -0,0 +1,25 @@ +/** + * Generated by Orval. Do not edit manually. + */ + +export interface ServiceRateLimitWindow { + accountId?: string; + accountLabel?: string; + creditsBalance?: string; + creditsHas: boolean; + creditsUnlimited: boolean; + details?: string; + limitId: string; + limitName?: string; + machine: string; + observedAt: string; + planType?: string; + rateLimitReachedType?: string; + resetsAt?: number; + scopeLabel?: string; + sessionId?: string; + usedPercent: number; + vendor: string; + windowKind: string; + windowMinutes?: number; +} diff --git a/frontend/src/lib/api/generated/rate-limits/rate-limits.ts b/frontend/src/lib/api/generated/rate-limits/rate-limits.ts new file mode 100644 index 0000000000..cc6ceab78e --- /dev/null +++ b/frontend/src/lib/api/generated/rate-limits/rate-limits.ts @@ -0,0 +1,68 @@ +/** + * Generated by Orval. Do not edit manually. + */ +import type { + GetApiV1RateLimitsCurrentParams, + GetApiV1RateLimitsHistoryParams, + ServiceRateLimitWindow, +} from "../models"; + +import { orvalFetch } from "../../runtime.ts"; + +export const getGetApiV1RateLimitsCurrentUrl = (params?: GetApiV1RateLimitsCurrentParams) => { + const normalizedParams = new URLSearchParams(); + + Object.entries(params || {}).forEach(([key, value]) => { + if (value !== undefined) { + normalizedParams.append(key, value === null ? "null" : String(value)); + } + }); + + const stringifiedParams = normalizedParams.toString(); + + return stringifiedParams.length > 0 + ? `/api/v1/rate-limits/current?${stringifiedParams}` + : `/api/v1/rate-limits/current`; +}; + +/** + * @summary Get current rate limits + */ +export const getApiV1RateLimitsCurrent = async ( + params?: GetApiV1RateLimitsCurrentParams, + options?: Parameters[1], +): Promise => { + return orvalFetch(getGetApiV1RateLimitsCurrentUrl(params), { + ...options, + method: "GET", + }); +}; + +export const getGetApiV1RateLimitsHistoryUrl = (params?: GetApiV1RateLimitsHistoryParams) => { + const normalizedParams = new URLSearchParams(); + + Object.entries(params || {}).forEach(([key, value]) => { + if (value !== undefined) { + normalizedParams.append(key, value === null ? "null" : String(value)); + } + }); + + const stringifiedParams = normalizedParams.toString(); + + return stringifiedParams.length > 0 + ? `/api/v1/rate-limits/history?${stringifiedParams}` + : `/api/v1/rate-limits/history`; +}; + +/** + * @summary Get rate limit history + */ +export const getApiV1RateLimitsHistory = async ( + params?: GetApiV1RateLimitsHistoryParams, + options?: Parameters[1], +): Promise => { + return orvalFetch(getGetApiV1RateLimitsHistoryUrl(params), { + ...options, + method: "GET", + }); +}; diff --git a/frontend/src/lib/components/ratelimits/RateLimitCard.svelte b/frontend/src/lib/components/ratelimits/RateLimitCard.svelte new file mode 100644 index 0000000000..bd6a9f6f35 --- /dev/null +++ b/frontend/src/lib/components/ratelimits/RateLimitCard.svelte @@ -0,0 +1,252 @@ + + +
+
+ {cardHeader} + {formatWindowLength(snapshot.windowMinutes)} +
+ +
+
+
+
+ + {usedPercent.toLocaleString(undefined, { maximumFractionDigits: 1 })}% + + + {#if !hasKnownReset} + {m.rate_limits_resets_unknown()} + {:else if resetCountdown !== null} + {m.rate_limits_resets_in({ value: resetCountdown })} + {:else} + {m.rate_limits_resets_now()} + {/if} + +
+ +
+ {#if snapshot.planType} + + {m.rate_limits_plan_label()} + {snapshot.planType} + + {/if} + {#if snapshot.creditsHas} + + {m.rate_limits_credits_label()} + + {snapshot.creditsUnlimited + ? m.rate_limits_credits_unlimited() + : snapshot.creditsBalance} + + + {/if} +
+ + {#if history.length > 1} +
+ +
+ {/if} +
+ + diff --git a/frontend/src/lib/components/ratelimits/RateLimitHistoryChart.svelte b/frontend/src/lib/components/ratelimits/RateLimitHistoryChart.svelte new file mode 100644 index 0000000000..15af3a1619 --- /dev/null +++ b/frontend/src/lib/components/ratelimits/RateLimitHistoryChart.svelte @@ -0,0 +1,77 @@ + + +{#if points.length > 1} + + + labelFor(Number(value))} + formatY={(value) => `${value}%`} + > + + + + +{/if} + + diff --git a/frontend/src/lib/components/ratelimits/RateLimitHistoryChart.test.ts b/frontend/src/lib/components/ratelimits/RateLimitHistoryChart.test.ts new file mode 100644 index 0000000000..cfae1448df --- /dev/null +++ b/frontend/src/lib/components/ratelimits/RateLimitHistoryChart.test.ts @@ -0,0 +1,50 @@ +// @vitest-environment jsdom +import { afterEach, beforeEach, describe, expect, it } from "vite-plus/test"; +import { mount, tick, unmount } from "svelte"; +import RateLimitHistoryChart from "./RateLimitHistoryChart.svelte"; +import type { RateLimitWindow } from "../../stores/ratelimits.svelte.js"; + +class ImmediateResizeObserver implements ResizeObserver { + constructor(private readonly cb: ResizeObserverCallback) {} + observe(target: Element): void { + this.cb([{ target, contentRect: { width: 400, height: 72 } }] as unknown as ResizeObserverEntry[], this); + } + unobserve(): void {} + disconnect(): void {} +} + +function snapshot(observedAt: string, usedPercent: number): RateLimitWindow { + return { vendor: "codex", machine: "laptop", limitId: "codex", planType: "pro", windowKind: "primary", usedPercent, windowMinutes: 10080, resetsAt: 0, creditsHas: false, creditsUnlimited: false, observedAt }; +} +// x coordinates of every point on the spline's "d" attribute ("M x,y" / "L x,y"). +function splineX(): number[] { + const el = document.querySelector(".rate-limit-history-chart path.lc-path"); + return [...(el?.getAttribute("d") ?? "").matchAll(/[ML]\s*(-?[\d.]+)/g)].map((m) => Number(m[1])); +} + +let component: ReturnType | undefined; + +describe("RateLimitHistoryChart", () => { + beforeEach(() => { + globalThis.ResizeObserver = ImmediateResizeObserver as typeof ResizeObserver; + }); + afterEach(() => { + if (component) void unmount(component); + component = undefined; + document.body.innerHTML = ""; + }); + // A ~1-year gap must render far wider than a 1-minute gap: points space by elapsed time, not index. + it("spaces points proportionally to elapsed time, not by index", async () => { + const snapshots = [ + snapshot("2026-01-01T00:00:00Z", 10), + snapshot("2026-01-01T00:01:00Z", 20), + snapshot("2026-12-31T00:01:00Z", 30), + ]; + component = mount(RateLimitHistoryChart, { target: document.body, props: { snapshots, color: "var(--accent-blue)" } }); + await tick(); + await tick(); + const xs = splineX(); + expect(xs).toHaveLength(3); + expect(Math.abs(xs[2]! - xs[1]!)).toBeGreaterThan(Math.abs(xs[1]! - xs[0]!) * 10); + }); +}); diff --git a/frontend/src/lib/components/ratelimits/RateLimitsSection.svelte b/frontend/src/lib/components/ratelimits/RateLimitsSection.svelte new file mode 100644 index 0000000000..cf56ef43be --- /dev/null +++ b/frontend/src/lib/components/ratelimits/RateLimitsSection.svelte @@ -0,0 +1,148 @@ + + +{#if rateLimits.hasData && visibleVendorGroups.length > 0} + +
+

{m.rate_limits_section_title()}

+
+ {#each visibleVendorGroups as vendorGroup (vendorGroup.vendor)} +
+ {#if visibleVendorGroups.length > 1} +

{vendorLabel(vendorGroup.vendor)}

+ {/if} + {#each vendorGroup.accounts as accountGroup (accountGroup.key)} + + {/each} +
+ {/each} +
+{/if} + + diff --git a/frontend/src/lib/components/ratelimits/RateLimitsSection.test.ts b/frontend/src/lib/components/ratelimits/RateLimitsSection.test.ts new file mode 100644 index 0000000000..ffd107ea6e --- /dev/null +++ b/frontend/src/lib/components/ratelimits/RateLimitsSection.test.ts @@ -0,0 +1,86 @@ +// @vitest-environment jsdom +import { afterEach, describe, expect, it, vi } from "vite-plus/test"; +import { mount, tick, unmount } from "svelte"; +import type { ServiceRateLimitWindow } from "../../api/generated/index"; + +const rateLimitsServiceMocks = vi.hoisted(() => ({ + getApiV1RateLimitsCurrent: vi.fn(), + getApiV1RateLimitsHistory: vi.fn(), +})); + +vi.mock("../../api/runtime.js", async (importOriginal) => { + const orig = await importOriginal(); + return { + ...orig, + callGenerated: vi.fn((request: (o?: { signal?: AbortSignal }) => Promise) => request()), + }; +}); +vi.mock("../../api/generated/index", async (importOriginal) => { + const orig = await importOriginal(); + return { ...orig, RateLimitsService: rateLimitsServiceMocks }; +}); + +const { rateLimits } = await import("../../stores/ratelimits.svelte.js"); +const { default: RateLimitsSection } = await import("./RateLimitsSection.svelte"); + +// snapshot builds a fixture with a limitName distinct from limitId, so a +// card that fell back to rendering limitId would be caught. +function snapshot(overrides: Partial = {}): ServiceRateLimitWindow { + return { + vendor: "codex", machine: "laptop", limitId: "codex", limitName: "GPT-5.3-Codex-Spark", + planType: "pro", windowKind: "primary", usedPercent: 95, windowMinutes: 10080, + creditsHas: true, creditsUnlimited: false, observedAt: "2026-09-09T10:00:00Z", ...overrides, + }; +} + +let component: ReturnType | undefined; + +afterEach(() => { + if (component) { + void unmount(component); + component = undefined; + } + vi.clearAllMocks(); + rateLimits.current = []; + rateLimits.history = {}; + document.body.innerHTML = ""; +}); + +async function mountSection(from = "2026-09-01", to = "2026-09-09") { + component = mount(RateLimitsSection, { target: document.body, props: { from, to } }); + await tick(); + await tick(); + await tick(); +} + +// A card's header must show the vendor-reported limit_name, not fall +// back to limit_id, and Codex snapshots (no account identity) must group +// into one account per machine within the vendor -- two machines render +// as two separate account groups. +describe("RateLimitsSection", () => { + it("shows each card's limit_name and groups snapshots by vendor then account", async () => { + rateLimitsServiceMocks.getApiV1RateLimitsCurrent.mockResolvedValue([ + snapshot({ machine: "laptop" }), + snapshot({ machine: "desktop" }), + ]); + rateLimitsServiceMocks.getApiV1RateLimitsHistory.mockResolvedValue([]); + await mountSection(); + + const header = document.querySelector(".limit-id")?.textContent; + expect(header).toContain("GPT-5.3-Codex-Spark"); + expect(header).not.toContain("[codex]"); + expect(document.querySelectorAll(".account-group")).toHaveLength(2); + const accountTitles = [...document.querySelectorAll(".account-title")].map((el) => el.textContent); + expect(accountTitles).toEqual(expect.arrayContaining(["laptop", "desktop"])); + }); + + // No windowMinutes: fall back to a window-kind label, not "— limit". + it("falls back to a window-kind label when windowMinutes is unknown", async () => { + rateLimitsServiceMocks.getApiV1RateLimitsCurrent.mockResolvedValue([ + snapshot({ windowMinutes: undefined, windowKind: "primary" }), + ]); + rateLimitsServiceMocks.getApiV1RateLimitsHistory.mockResolvedValue([]); + await mountSection(); + expect(document.querySelector(".limit-id")?.textContent).toContain("Session limit"); + }); +}); diff --git a/frontend/src/lib/components/usage/UsagePage.svelte b/frontend/src/lib/components/usage/UsagePage.svelte index 25c4303890..84cf659df0 100644 --- a/frontend/src/lib/components/usage/UsagePage.svelte +++ b/frontend/src/lib/components/usage/UsagePage.svelte @@ -25,6 +25,8 @@ type RangeSelection, } from "../shared/rangeSelection.js"; import UsageSummaryCards from "./UsageSummaryCards.svelte"; + import { rateLimits } from "../../stores/ratelimits.svelte.js"; + import RateLimitsSection from "../ratelimits/RateLimitsSection.svelte"; import UsagePairwiseComparisonPanel from "./UsagePairwiseComparisonPanel.svelte"; import CostTimeSeriesChart from "./CostTimeSeriesChart.svelte"; import AttributionPanel from "./AttributionPanel.svelte"; @@ -518,7 +520,14 @@ usage.fetchAll({ preserveTimeRange: true })} + onRefresh={() => { + usage.fetchAll({ preserveTimeRange: true }); + // Manual clicks and kit-ui's periodic auto-refresh both call + // this one callback; without this, rate-limit cards only ever + // refreshed on mount or a machine-filter change and otherwise + // went stale while the page stayed open. + rateLimits.fetchCurrent(); + }} label={m.usage_refresh()} title={m.shared_refresh()} /> @@ -551,6 +560,8 @@ + + ({ + getApiV1RateLimitsCurrent: vi.fn(), + getApiV1RateLimitsHistory: vi.fn(), +})); + +vi.mock("../api/runtime.js", async (importOriginal) => { + const orig = await importOriginal(); + return { + ...orig, + callGenerated: vi.fn((request: (o?: { signal?: AbortSignal }) => Promise) => request()), + }; +}); +vi.mock("../api/generated/index", async (importOriginal) => { + const orig = await importOriginal(); + return { ...orig, RateLimitsService: rateLimitsServiceMocks }; +}); +vi.mock("./sessions.svelte.js", () => ({ sessions: { filters: { machine: "", agent: "" } } })); + +// identity builds a full RateLimitCardIdentity fixture; overrides pick +// which axis differs between two cards under test. +function identity(overrides: Partial = {}): RateLimitCardIdentity { + return { + vendor: "codex", accountId: "", machine: "laptop", limitId: "codex", + windowKind: "primary", ...overrides, + }; +} +function snapshot(overrides: Partial = {}): ServiceRateLimitWindow { + return { + vendor: "codex", machine: "laptop", limitId: "codex", planType: "pro", windowKind: "primary", + usedPercent: 42, windowMinutes: 10080, resetsAt: 1789435448, creditsHas: true, + creditsUnlimited: false, creditsBalance: "100.0", observedAt: "2026-09-09T10:00:00Z", ...overrides, + }; +} + +let rateLimits: (typeof import("./ratelimits.svelte.js"))["rateLimits"]; + +beforeEach(async () => { + vi.clearAllMocks(); + ({ rateLimits } = await import("./ratelimits.svelte.js")); + rateLimits.history = {}; +}); + +// fetchHistory keys both its cache and its in-flight abort by the card's +// full identity: a request for one identity must not cancel or clobber +// another's, and a failed refetch for an identity already cached must +// leave that identity's prior history in place. +describe("rateLimits store", () => { + it("dedups history requests per identity without cross-identity interference", async () => { + let resolveLaptop: ((v: ServiceRateLimitWindow[]) => void) | undefined; + rateLimitsServiceMocks.getApiV1RateLimitsHistory.mockImplementationOnce( + () => new Promise((resolve) => { resolveLaptop = resolve; }), + ); + const laptopPromise = rateLimits.fetchHistory(identity({ machine: "laptop" }), "2026-09-08T00:00:00Z", "2026-09-09T23:59:59Z"); + rateLimitsServiceMocks.getApiV1RateLimitsHistory.mockResolvedValueOnce([snapshot({ machine: "desktop" })]); + await rateLimits.fetchHistory(identity({ machine: "desktop" }), "2026-09-08T00:00:00Z", "2026-09-09T23:59:59Z"); + resolveLaptop?.([snapshot({ machine: "laptop" })]); + await laptopPromise; + expect(rateLimits.historyFor(identity({ machine: "laptop" }))).toHaveLength(1); + expect(rateLimits.historyFor(identity({ machine: "desktop" }))).toHaveLength(1); + + rateLimitsServiceMocks.getApiV1RateLimitsHistory.mockRejectedValueOnce(new Error("boom")); + await rateLimits.fetchHistory(identity({ machine: "laptop" }), "2026-09-08T00:00:00Z", "2026-09-09T23:59:59Z"); + expect(rateLimits.historyFor(identity({ machine: "laptop" }))).toHaveLength(1); + }); + + // A stale in-flight request for the same identity can still resolve + // after a newer one already landed (abort does not guarantee the + // underlying promise never settles); the newer request's result must + // win regardless of resolution order. + it("ignores a same-identity response that resolves after a newer request for it", async () => { + let resolveFirst: ((v: ServiceRateLimitWindow[]) => void) | undefined; + rateLimitsServiceMocks.getApiV1RateLimitsHistory.mockImplementationOnce( + () => new Promise((resolve) => { resolveFirst = resolve; }), + ); + const firstPromise = rateLimits.fetchHistory(identity(), "2026-09-08T00:00:00Z", "2026-09-09T23:59:59Z"); + rateLimitsServiceMocks.getApiV1RateLimitsHistory.mockResolvedValueOnce([snapshot({ usedPercent: 99 })]); + await rateLimits.fetchHistory(identity(), "2026-09-08T00:00:00Z", "2026-09-09T23:59:59Z"); + resolveFirst?.([snapshot({ usedPercent: 1 })]); + await firstPromise; + expect(rateLimits.historyFor(identity())).toEqual([snapshot({ usedPercent: 99 })]); + }); + + // Codex reports plan_type as a label that can flip between "pro" and + // empty for the same window from one observation to the next, not a + // stable identity component (see RateLimitCardIdentity), so + // fetchHistory must keep every row for an identity regardless of each + // row's own plan_type rather than dropping the ones that disagree. + it("keeps every row for an identity regardless of differing plan_type", async () => { + rateLimitsServiceMocks.getApiV1RateLimitsHistory.mockResolvedValueOnce([ + snapshot({ planType: "pro" }), + snapshot({ planType: "" }), + ]); + await rateLimits.fetchHistory(identity(), "2026-09-08T00:00:00Z", "2026-09-09T23:59:59Z"); + expect(rateLimits.historyFor(identity())).toHaveLength(2); + }); +}); diff --git a/frontend/src/lib/stores/ratelimits.svelte.ts b/frontend/src/lib/stores/ratelimits.svelte.ts new file mode 100644 index 0000000000..cdb965a884 --- /dev/null +++ b/frontend/src/lib/stores/ratelimits.svelte.ts @@ -0,0 +1,244 @@ +import { + RateLimitsService, + type GetApiV1RateLimitsHistoryVendor, + type GetApiV1RateLimitsHistoryWindow, + type ServiceRateLimitWindow, +} from "../api/generated/index"; +import { callGenerated, isAbortError } from "../api/runtime.js"; +import { sessions } from "./sessions.svelte.js"; + +export type RateLimitWindow = ServiceRateLimitWindow; + +// Bounds each card's history request to a point count sized for its own +// small sparkline chart, well under the /rate-limits/history endpoint's +// own (much larger) default -- see RateLimitCard's history fetch. +const RATE_LIMIT_HISTORY_MAX_POINTS = 200; + +/** + * Identifies one Usage-page rate-limit card. This must be the full + * identity a card is grouped by (see LatestRateLimitSnapshots on the + * backend) -- vendor, account id, machine, limit id, and window kind -- + * not just a subset. Two cards can share a limit id and window kind + * while differing in machine (e.g. the same account synced from two + * machines), and each needs its own history request and cache entry so + * their charts do not merge. Codex snapshots carry no account identity + * (accountId always ""). + * + * planType is deliberately not part of this identity: Codex reports it + * as a label that can flip between "pro" and empty for the same window + * from one observation to the next (see docs/agents/storage.md), not a + * stable identity component, so keying or filtering history by it would + * split one window's history across two card identities, or defensively + * drop half of it from the chart. + */ +export interface RateLimitCardIdentity { + vendor: string; + accountId: string; + machine: string; + limitId: string; + windowKind: string; +} + +function historyKey(identity: RateLimitCardIdentity): string { + return [ + identity.vendor, + identity.accountId, + identity.machine, + identity.limitId, + identity.windowKind, + ].join(" "); +} + +/** Groups current snapshots by vendor, then by account (Codex carries no + * account identity, so its rows group by machine instead), matching the + * Usage page's vendor-then-account layout. + * + * `key` is the group's real identity for a keyed `{#each}` loop: + * `accountId` alone is not unique for Codex, which is always "" there, + * so two different Codex machines would otherwise render under the same + * key. `key` includes `machine` precisely for that case. + */ +export interface RateLimitAccountGroup { + vendor: string; + accountId: string; + accountLabel: string; + machine: string; + key: string; + windows: RateLimitWindow[]; +} + +export interface RateLimitVendorGroup { + vendor: string; + accounts: RateLimitAccountGroup[]; +} + +function groupByVendorThenAccount(snapshots: RateLimitWindow[]): RateLimitVendorGroup[] { + const vendorOrder: string[] = []; + const byVendor = new Map>(); + + for (const snapshot of snapshots) { + const vendor = snapshot.vendor; + if (!byVendor.has(vendor)) { + byVendor.set(vendor, new Map()); + vendorOrder.push(vendor); + } + const accounts = byVendor.get(vendor)!; + // Codex rows carry no account identity, so machine is the closest + // grouping key it has; a future account-keyed vendor would group by + // its real account id instead. + const machine = snapshot.machine ?? ""; + const accountId = snapshot.accountId ?? ""; + const accountKey = accountId || machine; + if (!accounts.has(accountKey)) { + accounts.set(accountKey, { + vendor, + accountId, + accountLabel: snapshot.accountLabel || machine, + machine, + key: `${vendor} ${accountKey}`, + windows: [], + }); + } + accounts.get(accountKey)!.windows.push(snapshot); + } + + return vendorOrder.map((vendor) => ({ + vendor, + accounts: [...byVendor.get(vendor)!.values()], + })); +} + +/** + * Rate-limit snapshots for the Usage page's "Rate limits" section + * (Codex today; the table is vendor-keyed so another vendor can add rows + * without a frontend change). Mirrors the shape of stores/usage.svelte.ts: + * reactive state written by callGenerated-wrapped fetches, one + * AbortController per in-flight request so a filter change cannot let a + * stale response overwrite a newer one. + */ +class RateLimitsStore { + current: RateLimitWindow[] = $state([]); + history: Record = $state({}); + loading = $state(false); + loaded = $state(false); + error: string | null = $state(null); + /** + * Bumped after every successful fetchCurrent(). Cards watch this (see + * RateLimitCard.svelte) to re-fetch their own history whenever current + * snapshots refresh -- mount, the Usage page's manual refresh button, or + * kit-ui RefreshControl's periodic timer -- so a chart does not go stale + * while newer observations are already showing in the card above it. + */ + refreshToken = $state(0); + + private currentAbort?: AbortController; + private historyAbort = new Map(); + // Per-identity request generation: aborting the previous controller for + // a key does not guarantee its in-flight promise never resolves (the + // underlying request may not honor AbortSignal), so a late response + // must still be recognized as superseded rather than overwriting a + // newer one for the same card. + private historyVersion = new Map(); + private version = 0; + + /** True once the first fetch has completed with at least one snapshot. */ + get hasData(): boolean { + return this.current.length > 0; + } + + /** Current snapshots grouped by vendor, then by account, in the order + * the Usage page renders them (vendors and accounts appear in the + * order their first snapshot was returned by the API). */ + get groupedByVendor(): RateLimitVendorGroup[] { + return groupByVendorThenAccount(this.current); + } + + historyFor(identity: RateLimitCardIdentity): RateLimitWindow[] { + return this.history[historyKey(identity)] ?? []; + } + + async fetchCurrent(): Promise { + const v = ++this.version; + this.currentAbort?.abort(); + const controller = new AbortController(); + this.currentAbort = controller; + this.loading = true; + try { + const machine = sessions.filters.machine || undefined; + // sessions.filters.agent is the shared, comma-separated agent + // selection; forwarded as-is so the backend's agent-list match + // (see db.RateLimitAgentMatchesVendor) naturally empties the + // result -- hiding the section -- once a non-Codex-only selection + // is active. + const agent = sessions.filters.agent || undefined; + const data = await callGenerated( + (options) => RateLimitsService.getApiV1RateLimitsCurrent({ machine, agent }, options), + controller.signal, + ); + if (v !== this.version) return; + this.current = data; + this.error = null; + this.loaded = true; + this.refreshToken++; + } catch (err) { + if (isAbortError(err)) return; + this.error = err instanceof Error ? err.message : String(err); + this.loaded = true; + } finally { + if (v === this.version) this.loading = false; + } + } + + async fetchHistory(identity: RateLimitCardIdentity, since: string, until: string): Promise { + const key = historyKey(identity); + this.historyAbort.get(key)?.abort(); + const controller = new AbortController(); + this.historyAbort.set(key, controller); + const requestVersion = (this.historyVersion.get(key) ?? 0) + 1; + this.historyVersion.set(key, requestVersion); + try { + const data = await callGenerated( + (options) => + RateLimitsService.getApiV1RateLimitsHistory( + { + vendor: identity.vendor as GetApiV1RateLimitsHistoryVendor, + account_id: identity.accountId || undefined, + // The card's own machine, not the page's (possibly + // "all machines") filter: history must stay scoped to + // exactly the window this card renders. + machine: identity.machine, + limit_id: identity.limitId, + window: identity.windowKind as GetApiV1RateLimitsHistoryWindow, + since, + until, + // RateLimitHistoryChart renders a small sparkline-sized + // chart, not a full-page one, so a much smaller point + // budget than the endpoint's own default keeps a wide + // date range from returning (and the chart from laying + // out) far more points than the chart can usefully show. + max_points: RATE_LIMIT_HISTORY_MAX_POINTS, + }, + options, + ), + controller.signal, + ); + // A superseded request (a newer fetchHistory for this same + // identity started after this one) must not overwrite the newer + // one's result just because it happened to resolve later. + if (this.historyVersion.get(key) !== requestVersion) return; + // The request above already scopes the response to this card's + // exact vendor/account/machine/limit/window, so the whole result + // is cached as-is -- including every plan_type label the window's + // history carries, so the chart is never missing points because a + // plan_type happened to differ between observations. + this.history = { ...this.history, [key]: data }; + } catch (err) { + if (isAbortError(err)) return; + // Leave any previously loaded history in place; a failed history + // fetch just leaves that card's chart empty rather than surfacing a + // page-level error for a secondary panel. + } + } +} + +export const rateLimits = new RateLimitsStore(); diff --git a/frontend/src/lib/utils/rateLimitFormat.ts b/frontend/src/lib/utils/rateLimitFormat.ts new file mode 100644 index 0000000000..5cbb5f1173 --- /dev/null +++ b/frontend/src/lib/utils/rateLimitFormat.ts @@ -0,0 +1,66 @@ +// Compact, non-localized duration abbreviations for rate-limit windows and +// reset countdowns (e.g. "5h", "7d", "2h 15m"), mirroring the existing +// non-localized unit abbreviations in utils/duration.ts (formatDuration). +// These are short technical units, not sentences, so they are not routed +// through the Paraglide message catalogues; the surrounding sentence +// ("Resets in {value}") is. + +const MINUTES_PER_HOUR = 60; +const MINUTES_PER_DAY = MINUTES_PER_HOUR * 24; + +/** Formats a rate-limit window length (in minutes) as a compact unit, e.g. "5h", "7d", "30m". Undefined (Codex reported no duration) renders as "—". */ +export function formatWindowLength(minutes: number | undefined): string { + if (minutes === undefined || !Number.isFinite(minutes) || minutes <= 0) return "—"; + if (minutes % MINUTES_PER_DAY === 0) return `${minutes / MINUTES_PER_DAY}d`; + if (minutes % MINUTES_PER_HOUR === 0) return `${minutes / MINUTES_PER_HOUR}h`; + if (minutes < MINUTES_PER_HOUR) return `${minutes}m`; + const hours = Math.floor(minutes / MINUTES_PER_HOUR); + const mins = minutes % MINUTES_PER_HOUR; + return `${hours}h ${mins}m`; +} + +/** + * Converts a browser-local calendar date range (YYYY-MM-DD, inclusive on + * both ends -- the shape the Usage page's date pickers and time-range + * brush use) into UTC instant bounds for a rate-limit history request: + * `since` is local midnight of `from`, and `until` is local midnight of + * the day AFTER `to` (an exclusive upper bound). Building local Date + * objects from the calendar fields (rather than parsing the strings as if + * they were already UTC) ties the boundary to the viewer's own calendar + * day; the exclusive next-day upper bound means a sub-second observation + * on `to`'s last moment is never dropped by a same-day 23:59:59Z cutoff + * that has no fractional part. + */ +export function localDateRangeToUTCBounds(from: string, to: string): { since: string; until: string } { + const parseYMD = (date: string): [number, number, number] => { + const [y, m, d] = date.split("-"); + return [Number(y), Number(m), Number(d)]; + }; + const [fy, fm, fd] = parseYMD(from); + const [ty, tm, td] = parseYMD(to); + return { + since: new Date(fy, fm - 1, fd).toISOString(), + until: new Date(ty, tm - 1, td + 1).toISOString(), + }; +} + +/** + * Formats the time remaining until a unix-seconds reset timestamp as a + * compact countdown ("2h 15m", "3d 4h"), or null once the reset has passed + * (the caller should show a "resets now"-style message instead). + */ +export function formatResetCountdown( + resetsAtSeconds: number, + nowMs: number = Date.now(), +): string | null { + if (!Number.isFinite(resetsAtSeconds)) return null; + const deltaMs = resetsAtSeconds * 1000 - nowMs; + if (deltaMs <= 0) return null; + const totalMinutes = Math.round(deltaMs / 60_000); + const days = Math.floor(totalMinutes / MINUTES_PER_DAY); + const hours = Math.floor((totalMinutes % MINUTES_PER_DAY) / MINUTES_PER_HOUR); + const mins = totalMinutes % MINUTES_PER_HOUR; + if (days > 0) return `${days}d ${hours}h`; + if (hours > 0) return `${hours}h ${mins}m`; + return `${Math.max(mins, 1)}m`; +} diff --git a/internal/db/db.go b/internal/db/db.go index 6508dcf3d5..c0f6f6f41b 100644 --- a/internal/db/db.go +++ b/internal/db/db.go @@ -487,7 +487,11 @@ CREATE INDEX IF NOT EXISTS idx_provider_freshness_updated_at // Subagent tool calls from Other to Task so delegation renders as a task call // and leaves the Other analytics bucket; subagent transcripts themselves are // new sources and need no re-parse.) -const dataVersion = 107 +// (108: Codex token_count events now also extract the rate_limits object +// into rate_limit_snapshots. An unchanged rollout file byte-for-byte +// still needs re-parsing to backfill this history, because a fingerprint +// change alone cannot repair a session parsed before this field existed.) +const dataVersion = 108 const tokenCoverageRepairStatsKey = "token_coverage_repair_v1" @@ -730,6 +734,15 @@ type DB struct { // the incremental signal path: a maintained delta must not load // session history. messagesLoadCount atomic.Int64 + + // rateLimitSnapshotsTableMu guards the cached sqlite_master probe for + // whether this archive's rate_limit_snapshots table exists (see + // docs/agents/storage.md): a read-only open of an archive that + // predates this table leaves it absent, and Latest/History must + // tolerate that without probing on every call. + rateLimitSnapshotsTableMu sync.Mutex + rateLimitSnapshotsTableProbed bool + rateLimitSnapshotsTableFound bool } // MessagesLoadCount returns the total number of GetAllMessages calls the diff --git a/internal/db/messages.go b/internal/db/messages.go index fa15e812df..ab200bb2d9 100644 --- a/internal/db/messages.go +++ b/internal/db/messages.go @@ -1709,6 +1709,11 @@ func (db *DB) WriteSessionIncremental( if err := updateSessionIncrementalTx(tx, sessionID, update); err != nil { return false, err } + if err := insertRateLimitSnapshotsTx( + context.Background(), tx, update.RateLimitSnapshots, + ); err != nil { + return false, err + } if update.Checkpoint != nil && update.CheckpointBlobs != nil { if err := upsertParserCheckpointTx( tx, *update.Checkpoint, *update.CheckpointBlobs, diff --git a/internal/db/rate_limit_snapshots.go b/internal/db/rate_limit_snapshots.go new file mode 100644 index 0000000000..549d22c2f3 --- /dev/null +++ b/internal/db/rate_limit_snapshots.go @@ -0,0 +1,998 @@ +package db + +import ( + "context" + "crypto/sha256" + "database/sql" + "encoding/hex" + "errors" + "fmt" + "slices" + "strings" + "time" +) + +// RateLimitSnapshot is one rate-limit window observed for a vendor at a +// point in time: today, always a Codex token_count event's rate_limits +// payload, alongside the plan type and credit balance reported at the +// same instant. The table is vendor-keyed from the start so a future +// vendor can add rows without a schema change: Vendor distinguishes +// them, and fields a given vendor never populates (AccountID, +// AccountLabel, ScopeLabel, Details for Codex) simply stay empty on that +// vendor's rows. +// +// A window's identity is (Vendor, Machine, AccountID, LimitID, +// WindowKind); PlanType and LimitName are display labels, not part of +// it. See docs/agents/storage.md for the full rationale, including why +// Codex rows carry no AccountID. +type RateLimitSnapshot struct { + ID int64 + + // Vendor is "codex" today. Defaults to "codex" on write for + // backward compatibility with callers that predate this field. + Vendor string + + SessionID string // Codex only; empty when the source session has been deleted + Machine string // Codex only + + AccountID string // reserved for a future account-keyed vendor; always "" for Codex + AccountLabel string // reserved for a future account-keyed vendor; always "" for Codex + + LimitID string // Codex only + LimitName string + PlanType string + WindowKind string // Codex: "primary" or "secondary" + + UsedPercent float64 + // WindowMinutes is 0 both for a window Codex reported with no + // duration (parser.ParsedRateLimitSnapshot.WindowMinutes nil) and, + // in principle, a genuine zero-minute window -- the latter never + // occurs in practice, so 0 doubles as "unknown" without a nullable + // column; the API layer treats it that way (see + // service.RateLimitWindow.WindowMinutes). Codex only. + WindowMinutes int + // ResetsAt is unix seconds, or nil when the vendor did not report a + // reset time for this window. Kept nullable rather than flattened to + // 0, so a genuinely unknown reset time is never displayed as if the + // window resets at the unix epoch. + ResetsAt *int64 + + CreditsHas bool // Codex only + CreditsUnlimited bool // Codex only + CreditsBalance string + + RateLimitReachedType string // Codex only + + // ScopeLabel and Details are reserved for a future vendor whose + // rate-limit source reports a scoped or free-form extra shape (see + // docs/agents/storage.md); always "" for Codex. + ScopeLabel string + Details string + + ObservedAt string // RFC3339Nano + // Ordinal is the source token_count event's stable per-file position + // (see parser.ParsedRateLimitSnapshot.Ordinal); folded into DedupKey + // and ObservationKey so two events sharing an ObservedAt second stay + // distinct. Codex only; always 0 for a vendor without per-event + // ordinals. + Ordinal int + DedupKey string + + // ObservationKey identifies the single source observation a row + // came from (e.g. one Codex token_count event's rate_limits + // payload), shared across the up-to-two window rows (primary, + // secondary) that observation produced. Computed once from + // SessionID+ObservedAt at insert time and stored independently of + // the nullable SessionID column, so it keeps sibling windows + // grouped together for LatestRateLimitSnapshots even after the + // source session is deleted or excluded from a resync -- unlike + // SessionID itself, which is exactly what goes away in that case. + ObservationKey string +} + +// RateLimitFilter selects rate-limit snapshots by vendor, account, and +// (Codex-only) machine. +type RateLimitFilter struct { + // Vendor narrows to one vendor ("codex" today); empty matches every + // vendor the table holds. + Vendor string + AccountID string + Machine string + // Agent is the same comma-separated agent selection the shared + // session filters use (sessions.filters.agent), accepted for + // backward compatibility with callers that filtered by agent before + // Vendor existed. When Vendor is unset, Agent is applied directly as + // the vendor restriction (see rateLimitEffectiveVendor): an agent + // slug and a rate-limit vendor name are the same value. + Agent string +} + +// RateLimitHistoryFilter selects a time series of rate-limit snapshots, +// optionally narrowed to vendor, account, one limit_id, and/or window +// kind. Machine supports the same comma-separated IN semantics as +// RateLimitFilter.Machine, and Agent is resolved into the vendor +// restriction the same way as RateLimitFilter.Agent (see +// rateLimitEffectiveVendor). +type RateLimitHistoryFilter struct { + Vendor string + AccountID string + Machine string + Agent string + LimitID string + WindowKind string + Since string // RFC3339Nano, inclusive lower bound; empty means unbounded + Until string // RFC3339Nano, exclusive upper bound; empty means unbounded + // MaxPoints bounds the number of rows returned: the matching range is + // bucketed in SQL (see RateLimitSnapshotHistory) into at most + // MaxPoints equal-width time buckets, keeping only the most recent + // observation in each so the series stays representative of the + // window's end state without growing the response (and the + // resulting chart's point count) with the query range. <= 0 uses + // defaultRateLimitHistoryMaxPoints. + MaxPoints int +} + +// rateLimitMachineClause returns a SQL fragment (starting with " AND") +// and its bind args for filtering rate_limit_snapshots by machine, using +// the same single-value "=" / multi-value "IN" split the other usage +// filters use (see UsageFilter.appendUsageSessionFilterClauses): a lone +// value compares with "=" so the query plan can still use an equality +// index lookup, and two or more values compare with "IN". An empty filter +// returns no clause at all. +func rateLimitMachineClause(machine string) (string, []any) { + if machine == "" { + return "", nil + } + vals := strings.Split(machine, ",") + if len(vals) == 1 { + return " AND machine = ?", []any{vals[0]} + } + placeholders := make([]string, len(vals)) + args := make([]any, len(vals)) + for i, v := range vals { + placeholders[i] = "?" + args[i] = v + } + return " AND machine IN (" + strings.Join(placeholders, ",") + ")", args +} + +// rateLimitVendorClause returns a SQL fragment (starting with " AND") +// and its bind args restricting rate_limit_snapshots to one or more +// vendors, using the same single-value "=" / multi-value "IN" split as +// rateLimitMachineClause. An empty vendor returns no clause at all, +// matching every vendor the table holds. +func rateLimitVendorClause(vendor string) (string, []any) { + if vendor == "" { + return "", nil + } + vals := strings.Split(vendor, ",") + if len(vals) == 1 { + return " AND vendor = ?", []any{vals[0]} + } + placeholders := make([]string, len(vals)) + args := make([]any, len(vals)) + for i, v := range vals { + placeholders[i] = "?" + args[i] = v + } + return " AND vendor IN (" + strings.Join(placeholders, ",") + ")", args +} + +// rateLimitEffectiveVendor resolves the vendor restriction a filter's +// Vendor and Agent fields together imply: an explicit Vendor takes +// precedence; otherwise Agent's comma-separated selection is applied +// directly, since an agent slug (e.g. "codex") and a rate-limit vendor +// name are the same value domain (see RateLimitAgentMatchesVendor). +// Empty when neither is set, matching every vendor the table holds. +func rateLimitEffectiveVendor(vendor, agent string) string { + if vendor != "" { + return vendor + } + return agent +} + +// normalizeRateLimitBoundary parses an RFC3339 (or RFC3339Nano) history +// since/until bound and reformats it as a UTC RFC3339Nano string. Request +// bounds cannot be compared against the stored observed_at column as raw +// text: Go's RFC3339Nano formatting trims trailing fractional-second +// zeros to a variable width, so two otherwise-adjacent instants (e.g. +// "...T23:59:59Z" and "...T23:59:59.5Z") do not sort the way their times +// do, and a non-UTC request offset would not compare correctly against +// the always-UTC stored value either. Normalizing both sides to the same +// UTC instant here, and comparing them with SQL julianday() rather than a +// text comparison (see internal/db/search.go's identical julianday() +// rationale) makes the comparison correct regardless of either string's +// formatting or offset. julianday() resolves observed_at to roughly +// millisecond precision (SQLite's ISO8601 parsing keeps at most three +// fractional-second digits), which is well beyond Codex's own +// whole-second event timestamps and the day-boundary bounds this +// endpoint is queried with; it is not exact at sub-millisecond +// resolution. Empty input means unbounded and is returned unchanged. +func normalizeRateLimitBoundary(s string) (string, error) { + if s == "" { + return "", nil + } + t, err := time.Parse(time.RFC3339Nano, s) + if err != nil { + return "", fmt.Errorf("parsing rate limit history bound %q: %w", s, err) + } + return t.UTC().Format(time.RFC3339Nano), nil +} + +// RateLimitAgentMatchesVendor reports whether agent -- a comma-separated +// agent selection such as sessions.filters.agent, or "" for no filter -- +// includes vendor. rate_limit_snapshots only ever holds Codex rows +// today, so every current caller passes vendor "codex", but the check +// itself does not hardcode that: an empty selection (no filter active) +// matches everything, and any non-empty selection must name vendor +// somewhere in the list; a selection of one or more other agents that +// omits vendor (e.g. "claude") matches nothing. A naive exact-equality +// check here would wrongly hide a vendor's rate-limit cards whenever +// more than one agent is selected together with it (e.g. +// "codex,claude"). +func RateLimitAgentMatchesVendor(agent, vendor string) bool { + if agent == "" { + return true + } + return slices.Contains(strings.Split(agent, ","), vendor) +} + +// RateLimitSnapshotDedupKey returns the stable identity for one snapshot +// row: the source session id, its observed timestamp, the limit id, the +// window kind, and the source event's ordinal. Re-parsing a Codex +// rollout (full or incremental) produces the same key for the same +// event, so the unique dedup_key index makes re-parsing an upsert +// instead of duplicating rows (see insertRateLimitSnapshotsTx). Vendor does not +// participate in the key: session id is already vendor-specific, so +// adding vendor here would be redundant. Ordinal disambiguates two +// distinct token_count events that happen to share an ObservedAt +// second, which SessionID+ObservedAt+LimitID+WindowKind alone cannot: +// without it, the second event's row would collide with -- and be +// silently dropped in favor of -- the first's. +func RateLimitSnapshotDedupKey(s RateLimitSnapshot) string { + sum := sha256.Sum256([]byte(fmt.Sprintf( + "%s|%s|%s|%s|%d", + s.SessionID, s.ObservedAt, s.LimitID, s.WindowKind, s.Ordinal, + ))) + return hex.EncodeToString(sum[:]) +} + +// InsertRateLimitSnapshots appends new rate-limit snapshot rows and +// ignores duplicates with the same dedup key. It never deletes existing +// rows, so it is safe to call from both a full parse (which sees the +// whole transcript on every run) and an incremental parse (which only +// sees the appended tail): both converge on the same set of rows. A +// write that instead replaces a session's prior content wholesale -- +// most notably an authoritative reparse superseding a fallback marked +// parser.DataVersionNeedsRetry -- must use +// InsertRateLimitSnapshotsReplacingSession instead, or that fallback's +// rows outlive the parse that superseded them. +func (db *DB) InsertRateLimitSnapshots( + snapshots []RateLimitSnapshot, +) error { + return db.InsertRateLimitSnapshotsContext( + context.Background(), snapshots, + ) +} + +// InsertRateLimitSnapshotsContext is InsertRateLimitSnapshots bound to +// ctx. It opens and commits its own transaction. +func (db *DB) InsertRateLimitSnapshotsContext( + ctx context.Context, snapshots []RateLimitSnapshot, +) error { + if len(snapshots) == 0 { + return nil + } + db.mu.Lock() + defer db.mu.Unlock() + + tx, err := db.getWriter().BeginTx(ctx, nil) + if err != nil { + return fmt.Errorf("beginning rate limit snapshots tx: %w", err) + } + defer func() { _ = tx.Rollback() }() + + if err := insertRateLimitSnapshotsTx(ctx, tx, snapshots); err != nil { + return err + } + return tx.Commit() +} + +// InsertRateLimitSnapshotsReplacingSession deletes every existing +// rate_limit_snapshots row for sessionID and inserts snapshots in the +// same transaction. Use this instead of InsertRateLimitSnapshots at a +// write that replaces a session's prior content wholesale (the same +// path ReplaceSessionMessages/ReplaceSessionContent use for an +// authoritative reparse): rows are otherwise append-only (see +// InsertRateLimitSnapshots), so a fallback parse marked +// parser.DataVersionNeedsRetry can leave stale observations behind once +// a later authoritative parse supersedes it -- neither dedup_key nor a +// plain append ever removes them. A normal incremental parse (the same +// data version, appending only the newly parsed tail) must keep calling +// InsertRateLimitSnapshots, which never deletes. +func (db *DB) InsertRateLimitSnapshotsReplacingSession( + sessionID string, snapshots []RateLimitSnapshot, +) error { + return db.InsertRateLimitSnapshotsReplacingSessionContext( + context.Background(), sessionID, snapshots, + ) +} + +// InsertRateLimitSnapshotsReplacingSessionContext is +// InsertRateLimitSnapshotsReplacingSession bound to ctx. +func (db *DB) InsertRateLimitSnapshotsReplacingSessionContext( + ctx context.Context, sessionID string, snapshots []RateLimitSnapshot, +) error { + if sessionID == "" { + return db.InsertRateLimitSnapshotsContext(ctx, snapshots) + } + db.mu.Lock() + defer db.mu.Unlock() + + tx, err := db.getWriter().BeginTx(ctx, nil) + if err != nil { + return fmt.Errorf( + "beginning rate limit snapshots replace tx: %w", err, + ) + } + defer func() { _ = tx.Rollback() }() + + if err := deleteRateLimitSnapshotsForSessionTx(tx, sessionID); err != nil { + return err + } + if err := insertRateLimitSnapshotsTx(ctx, tx, snapshots); err != nil { + return err + } + return tx.Commit() +} + +// deleteRateLimitSnapshotsForSessionTx deletes every rate_limit_snapshots +// row belonging to sessionID. Called before inserting a fresh set from a +// parse that replaces a session's prior content wholesale, so a +// superseded fallback parse's rows (e.g. one marked +// parser.DataVersionNeedsRetry) cannot survive alongside the later +// authoritative parse's rows. +func deleteRateLimitSnapshotsForSessionTx( + tx transactionQueries, sessionID string, +) error { + if _, err := tx.Exec( + "DELETE FROM rate_limit_snapshots WHERE session_id = ?", sessionID, + ); err != nil { + return fmt.Errorf( + "deleting rate limit snapshots for %s: %w", sessionID, err, + ) + } + return nil +} + +// insertRateLimitSnapshotsTx inserts every row in snapshots that carries +// its required identity fields (limit_id, window_kind, observed_at), +// skipping -- rather than failing the whole batch, and with it the +// session write this batch is usually part of -- any single entry +// missing one. A source-format quirk or parser bug that produces one +// malformed rate_limits observation must not take down every other +// snapshot in the same batch, nor the session ingestion it rode in on; +// see docs/agents/storage.md. +// +// A dedup_key collision against an existing row whose session_id is NULL +// (a resync copy for a session absent from the destination -- see +// CopyRateLimitSnapshotsFrom) reattaches that row to the incoming +// session_id and refreshes its other fields instead of being ignored: +// dedup_key is preserved unchanged by that copy, so the session's later +// reappearance (a fresh parse producing the identical dedup_key) would +// otherwise collide with, and be silently discarded in favor of, the +// stale detached row forever. A collision against a row that already +// has a non-NULL session_id is unaffected and still a no-op, matching +// plain INSERT OR IGNORE. +func insertRateLimitSnapshotsTx( + ctx context.Context, tx transactionQueries, snapshots []RateLimitSnapshot, +) error { + for _, snap := range snapshots { + if err := ctx.Err(); err != nil { + return err + } + if snap.Vendor == "" { + snap.Vendor = "codex" + } + if snap.LimitID == "" || snap.WindowKind == "" || snap.ObservedAt == "" { + continue + } + if snap.DedupKey == "" { + snap.DedupKey = RateLimitSnapshotDedupKey(snap) + } + if snap.ObservationKey == "" { + snap.ObservationKey = RateLimitObservationKey(snap) + } + + var sessionID any + if snap.SessionID != "" { + sessionID = snap.SessionID + } + var resetsAt any + if snap.ResetsAt != nil { + resetsAt = *snap.ResetsAt + } + creditsHas := 0 + if snap.CreditsHas { + creditsHas = 1 + } + creditsUnlimited := 0 + if snap.CreditsUnlimited { + creditsUnlimited = 1 + } + + if _, err := tx.Exec(` + INSERT INTO rate_limit_snapshots ( + vendor, session_id, machine, account_id, account_label, + limit_id, limit_name, plan_type, window_kind, + used_percent, window_minutes, resets_at, + credits_has, credits_unlimited, credits_balance, + rate_limit_reached_type, scope_label, details, + observed_at, ordinal, dedup_key, observation_key + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(dedup_key) WHERE dedup_key != '' DO UPDATE SET + session_id = excluded.session_id, + machine = excluded.machine, + account_id = excluded.account_id, + account_label = excluded.account_label, + limit_name = excluded.limit_name, + plan_type = excluded.plan_type, + used_percent = excluded.used_percent, + window_minutes = excluded.window_minutes, + resets_at = excluded.resets_at, + credits_has = excluded.credits_has, + credits_unlimited = excluded.credits_unlimited, + credits_balance = excluded.credits_balance, + rate_limit_reached_type = excluded.rate_limit_reached_type, + scope_label = excluded.scope_label, + details = excluded.details, + observation_key = excluded.observation_key + WHERE rate_limit_snapshots.session_id IS NULL`, + snap.Vendor, sessionID, SanitizeUTF8(snap.Machine), + SanitizeUTF8(snap.AccountID), SanitizeUTF8(snap.AccountLabel), + SanitizeUTF8(snap.LimitID), SanitizeUTF8(snap.LimitName), + SanitizeUTF8(snap.PlanType), snap.WindowKind, + snap.UsedPercent, snap.WindowMinutes, resetsAt, + creditsHas, creditsUnlimited, SanitizeUTF8(snap.CreditsBalance), + SanitizeUTF8(snap.RateLimitReachedType), SanitizeUTF8(snap.ScopeLabel), + SanitizeUTF8(snap.Details), snap.ObservedAt, snap.Ordinal, snap.DedupKey, + snap.ObservationKey, + ); err != nil { + return fmt.Errorf("inserting rate limit snapshot: %w", err) + } + } + return nil +} + +// RateLimitObservationKey returns the stable identity of the single +// source observation s came from (e.g. one Codex token_count event's +// rate_limits payload), shared by the up-to-two window rows (primary, +// secondary) that observation produces. It is computed once at insert +// time from fields that do not change afterward -- unlike SessionID, +// which InsertRateLimitSnapshots/CopyRateLimitSnapshotsFrom can later +// leave NULL for a deleted or unrestored session -- so storing it in +// its own column keeps sibling windows grouped for +// LatestRateLimitSnapshots regardless of what later happens to the +// source session. Ordinal participates for the same reason it does in +// RateLimitSnapshotDedupKey: two distinct token_count events at the same +// SessionID+ObservedAt+LimitID+PlanType would otherwise share one +// observation_key, merging their sibling windows into a single bucket +// for LatestRateLimitSnapshots. Empty when SessionID is empty (no vendor +// writes rows that way today); callers fall back to a per-row key in +// that case. +func RateLimitObservationKey(s RateLimitSnapshot) string { + if s.SessionID == "" { + return "" + } + return fmt.Sprintf( + "%s|%s|%s|%s|%d", + s.SessionID, s.ObservedAt, s.LimitID, s.PlanType, s.Ordinal, + ) +} + +const rateLimitSnapshotColumns = ` + id, vendor, session_id, machine, account_id, account_label, + limit_id, limit_name, plan_type, window_kind, + used_percent, window_minutes, resets_at, + credits_has, credits_unlimited, credits_balance, + rate_limit_reached_type, scope_label, details, observed_at, ordinal, + dedup_key, observation_key` + +func scanRateLimitSnapshot(row interface{ Scan(...any) error }) (RateLimitSnapshot, error) { + var s RateLimitSnapshot + var sessionID *string + var resetsAt *int64 + var creditsHas, creditsUnlimited int + if err := row.Scan( + &s.ID, &s.Vendor, &sessionID, &s.Machine, &s.AccountID, &s.AccountLabel, + &s.LimitID, &s.LimitName, &s.PlanType, &s.WindowKind, + &s.UsedPercent, &s.WindowMinutes, &resetsAt, + &creditsHas, &creditsUnlimited, &s.CreditsBalance, + &s.RateLimitReachedType, &s.ScopeLabel, &s.Details, + &s.ObservedAt, &s.Ordinal, &s.DedupKey, &s.ObservationKey, + ); err != nil { + return RateLimitSnapshot{}, err + } + if sessionID != nil { + s.SessionID = *sessionID + } + s.ResetsAt = resetsAt + s.CreditsHas = creditsHas != 0 + s.CreditsUnlimited = creditsUnlimited != 0 + return s, nil +} + +// hasRateLimitSnapshotsTable reports whether this archive's +// rate_limit_snapshots table exists, probing sqlite_master once per DB +// instance and caching the result. A writable Open always creates the +// table (schema.sql's CREATE TABLE IF NOT EXISTS), but OpenReadOnly runs +// no migrations, so an older, otherwise-compatible archive opened +// read-only can legitimately predate the table (see +// docs/agents/storage.md and CopyRateLimitSnapshotsFrom's identical +// sqlite_master check). A failed probe is not cached, so a transient +// error is retried on the next call rather than permanently treated as +// "missing". +func (db *DB) hasRateLimitSnapshotsTable() bool { + db.rateLimitSnapshotsTableMu.Lock() + defer db.rateLimitSnapshotsTableMu.Unlock() + if db.rateLimitSnapshotsTableProbed { + return db.rateLimitSnapshotsTableFound + } + var exists bool + if err := db.getReader().QueryRow(`SELECT EXISTS( + SELECT 1 FROM sqlite_master + WHERE type = 'table' AND name = 'rate_limit_snapshots' + )`).Scan(&exists); err != nil { + return false + } + db.rateLimitSnapshotsTableProbed = true + db.rateLimitSnapshotsTableFound = exists + return exists +} + +// rateLimitBucketKey identifies one LatestRateLimitSnapshots grouping +// bucket. +type rateLimitBucketKey struct { + vendor, machine, accountID, limitID string +} + +// rateLimitBuckets discovers every distinct (vendor, machine, account_id, +// limit_id) bucket matching whereClause/whereArgs by repeatedly seeking +// the next key greater than the last one found, via the row-value +// comparison SQLite compiles into an index seek against +// idx_rate_limit_snapshots_bucket_observed. A single "GROUP BY ... +// MAX(observed_at)" query over the same index costs one comparison per +// matching row -- confirmed via EXPLAIN QUERY PLAN to be a full covering +// index scan, not a seek -- which dominates LatestRateLimitSnapshots' cost +// at scale even though the number of distinct buckets a real archive +// holds stays small (one per machine/limit family) regardless of how much +// history has accumulated. This walk instead costs one O(log n) seek per +// bucket. +func (db *DB) rateLimitBuckets( + ctx context.Context, whereClause string, whereArgs []any, +) ([]rateLimitBucketKey, error) { + var out []rateLimitBucketKey + var vendor, machine, accountID, limitID string + for { + args := append([]any{vendor, machine, accountID, limitID}, whereArgs...) + row := db.getReader().QueryRowContext(ctx, ` + SELECT vendor, machine, account_id, limit_id + FROM rate_limit_snapshots + WHERE (vendor, machine, account_id, limit_id) > (?, ?, ?, ?)`+whereClause+` + ORDER BY vendor, machine, account_id, limit_id + LIMIT 1`, + args..., + ) + var next rateLimitBucketKey + if err := row.Scan(&next.vendor, &next.machine, &next.accountID, &next.limitID); err != nil { + if errors.Is(err, sql.ErrNoRows) { + break + } + return nil, fmt.Errorf("discovering rate limit buckets: %w", err) + } + out = append(out, next) + vendor, machine, accountID, limitID = next.vendor, next.machine, next.accountID, next.limitID + } + return out, nil +} + +// LatestRateLimitSnapshots returns the most recently observed row per +// (vendor, machine, account_id, limit_id, window_kind) group, matching f. +// "Latest" is resolved per (vendor, machine, account_id, limit_id) bucket +// -- one level above window_kind -- with plan_type and limit_name each +// independently backfilled from the latest observation that reported them +// non-empty; see docs/agents/storage.md for the rationale (why plan_type +// is not part of the bucket, why a window that stops being reported must +// still be superseded, and the observation_key tie-break for two sessions +// reporting at the same instant). "Most recently observed" is decided by +// julianday(observed_at); see normalizeRateLimitBoundary's doc comment +// for why a raw text comparison is not safe. Buckets are discovered with +// rateLimitBuckets, then each bucket's winning row and its plan_type/ +// limit_name labels are resolved with one small indexed query per bucket, +// so cost stays close to the number of buckets rather than the number of +// rows. An archive opened read-only from before this table existed +// returns an empty result instead of a "no such table" error (see +// hasRateLimitSnapshotsTable). +func (db *DB) LatestRateLimitSnapshots( + ctx context.Context, f RateLimitFilter, +) ([]RateLimitSnapshot, error) { + if !db.hasRateLimitSnapshotsTable() { + return nil, nil + } + machineClause, machineArgs := rateLimitMachineClause(f.Machine) + vendorClause, vendorArgs := rateLimitVendorClause(rateLimitEffectiveVendor(f.Vendor, f.Agent)) + accountClause := "" + var accountArgs []any + if f.AccountID != "" { + // account_id scopes vendors that have accounts; a vendor whose + // rows always carry account_id '' (Codex today) has nothing to + // scope, so its rows must pass through an account_id filter + // rather than being excluded by it. + accountClause = " AND (account_id = '' OR account_id = ?)" + accountArgs = []any{f.AccountID} + } + whereClause := machineClause + vendorClause + accountClause + var whereArgs []any + whereArgs = append(whereArgs, machineArgs...) + whereArgs = append(whereArgs, vendorArgs...) + whereArgs = append(whereArgs, accountArgs...) + + buckets, err := db.rateLimitBuckets(ctx, whereClause, whereArgs) + if err != nil { + return nil, err + } + + var out []RateLimitSnapshot + for _, bk := range buckets { + // winner resolves the single winning observation for this bucket: + // the newest observed_at, and among rows tied on that timestamp + // (two sessions can legitimately report the same account-wide + // limit at the same instant) the highest id, matching + // RateLimitObservationKey's tie-break. That key, not session_id + // (nullable once a session is deleted), is what the final query + // matches sibling windows on. limit_name/plan_type are resolved + // the same way, independently, as the latest non-empty value ever + // observed for the bucket. + bkArgs := []any{bk.vendor, bk.machine, bk.accountID, bk.limitID} + args := append([]any{}, bkArgs...) + args = append(args, bkArgs...) + args = append(args, bkArgs...) + args = append(args, bkArgs...) + rows, err := db.getReader().QueryContext(ctx, ` + WITH winner AS ( + SELECT observed_at, + CASE WHEN observation_key != '' THEN observation_key ELSE 'row:' || id END AS obs_key + FROM rate_limit_snapshots + WHERE vendor = ? AND machine = ? AND account_id = ? AND limit_id = ? + ORDER BY julianday(observed_at) DESC, id DESC + LIMIT 1 + ) + SELECT r.id, r.vendor, r.session_id, r.machine, r.account_id, r.account_label, + r.limit_id, + COALESCE(( + SELECT ln.limit_name FROM rate_limit_snapshots ln + WHERE ln.vendor = ? AND ln.machine = ? AND ln.account_id = ? AND ln.limit_id = ? + AND ln.limit_name != '' + ORDER BY julianday(ln.observed_at) DESC, ln.id DESC + LIMIT 1 + ), r.limit_name) AS limit_name, + COALESCE(( + SELECT pt.plan_type FROM rate_limit_snapshots pt + WHERE pt.vendor = ? AND pt.machine = ? AND pt.account_id = ? AND pt.limit_id = ? + AND pt.plan_type != '' + ORDER BY julianday(pt.observed_at) DESC, pt.id DESC + LIMIT 1 + ), r.plan_type) AS plan_type, + r.window_kind, + r.used_percent, r.window_minutes, r.resets_at, + r.credits_has, r.credits_unlimited, r.credits_balance, + r.rate_limit_reached_type, r.scope_label, r.details, + r.observed_at, r.ordinal, r.dedup_key, r.observation_key + FROM rate_limit_snapshots r, winner w + WHERE r.vendor = ? AND r.machine = ? AND r.account_id = ? AND r.limit_id = ? + AND r.observed_at = w.observed_at + AND (CASE WHEN r.observation_key != '' THEN r.observation_key ELSE 'row:' || r.id END) = w.obs_key`, + args..., + ) + if err != nil { + return nil, fmt.Errorf("querying latest rate limit snapshot: %w", err) + } + for rows.Next() { + s, err := scanRateLimitSnapshot(rows) + if err != nil { + _ = rows.Close() + return nil, fmt.Errorf("scanning rate limit snapshot: %w", err) + } + out = append(out, s) + } + if err := rows.Err(); err != nil { + _ = rows.Close() + return nil, fmt.Errorf("iterating rate limit snapshots: %w", err) + } + _ = rows.Close() + } + + // rateLimitBuckets already yields buckets ordered by (vendor, machine, + // account_id, limit_id); re-key to the documented (vendor, account_id, + // machine, limit_id, window_kind) result order. + slices.SortFunc(out, func(a, b RateLimitSnapshot) int { + if c := strings.Compare(a.Vendor, b.Vendor); c != 0 { + return c + } + if c := strings.Compare(a.AccountID, b.AccountID); c != 0 { + return c + } + if c := strings.Compare(a.Machine, b.Machine); c != 0 { + return c + } + if c := strings.Compare(a.LimitID, b.LimitID); c != 0 { + return c + } + return strings.Compare(a.WindowKind, b.WindowKind) + }) + return out, nil +} + +// defaultRateLimitHistoryMaxPoints bounds RateLimitSnapshotHistory's +// result size when a caller does not set RateLimitHistoryFilter.MaxPoints. +// A wide query range on a long-lived rate limit window can otherwise +// accumulate thousands of observations, bloating the API response and +// giving RateLimitHistoryChart that many SVG points to lay out for what +// is, visually, a small sparkline-sized chart. +const defaultRateLimitHistoryMaxPoints = 500 + +// RateLimitSnapshotHistory returns a time-ordered series of snapshots +// matching f, bounded to at most f.MaxPoints (or +// defaultRateLimitHistoryMaxPoints) entries, for charting used_percent +// over time. A range with more matching rows than that is bucketed +// entirely in SQL: divided into that many equal-width time buckets, each +// contributing only its most recently observed row (the bucket's end +// state, not an average or first observation), so the result stays +// bounded and representative without pulling every matching row into Go +// first. The series is ordered by julianday(observed_at), not a raw text +// comparison; see normalizeRateLimitBoundary's doc comment for why. An +// archive opened read-only from before this table existed returns an +// empty result instead of a "no such table" error (see +// hasRateLimitSnapshotsTable). +func (db *DB) RateLimitSnapshotHistory( + ctx context.Context, f RateLimitHistoryFilter, +) ([]RateLimitSnapshot, error) { + if !db.hasRateLimitSnapshotsTable() { + return nil, nil + } + since, err := normalizeRateLimitBoundary(f.Since) + if err != nil { + return nil, err + } + until, err := normalizeRateLimitBoundary(f.Until) + if err != nil { + return nil, err + } + maxPoints := f.MaxPoints + if maxPoints <= 0 { + maxPoints = defaultRateLimitHistoryMaxPoints + } + machineClause, machineArgs := rateLimitMachineClause(f.Machine) + vendorClause, vendorArgs := rateLimitVendorClause(rateLimitEffectiveVendor(f.Vendor, f.Agent)) + args := append([]any{}, machineArgs...) + args = append(args, vendorArgs...) + accountClause := "" + if f.AccountID != "" { + // See LatestRateLimitSnapshots: account_id scopes vendors that + // have accounts, so an account-less vendor's rows (Codex today) + // must pass through rather than being excluded. + accountClause = " AND (account_id = '' OR account_id = ?)" + args = append(args, f.AccountID) + } + args = append(args, + f.LimitID, f.LimitID, + f.WindowKind, f.WindowKind, + since, since, + until, until, + // bucket's CASE below: the "keep everything" threshold, the + // clip ceiling (maxPoints-1), and the bucket-width divisor. + maxPoints, maxPoints, maxPoints, + maxPoints, // final LIMIT + ) + // filtered applies every predicate once. bounds computes the + // matching range's row count and observed_at span so bucket can + // decide, per row, which of three regimes applies: fewer rows than + // maxPoints keeps every row (bucket = its own id, so it is the sole + // member of its partition below); a zero-width span (every matching + // row shares one observed_at) collapses to a single bucket, since + // there is no time axis left to divide; otherwise each row's bucket + // is its fractional position in the span scaled to maxPoints buckets + // and clipped to the last one. ranked then keeps only the most + // recent row (ORDER BY jd DESC, id DESC) per bucket, and the outer + // query restores chronological order. + rows, err := db.getReader().QueryContext(ctx, ` + WITH filtered AS ( + SELECT `+rateLimitSnapshotColumns+`, julianday(observed_at) AS jd + FROM rate_limit_snapshots + WHERE 1=1`+machineClause+vendorClause+accountClause+` + AND (? = '' OR limit_id = ?) + AND (? = '' OR window_kind = ?) + AND (? = '' OR julianday(observed_at) >= julianday(?)) + AND (? = '' OR julianday(observed_at) < julianday(?)) + ), + bounds AS ( + SELECT MIN(jd) AS min_jd, MAX(jd) AS max_jd, COUNT(*) AS cnt FROM filtered + ), + bucketed AS ( + SELECT f.*, + CASE + WHEN b.cnt <= ? THEN f.id + WHEN b.max_jd <= b.min_jd THEN 0 + ELSE MIN(? - 1, CAST((f.jd - b.min_jd) * ? / (b.max_jd - b.min_jd) AS INTEGER)) + END AS bucket + FROM filtered f, bounds b + ), + ranked AS ( + SELECT *, ROW_NUMBER() OVER ( + PARTITION BY bucket ORDER BY jd DESC, id DESC + ) AS rn + FROM bucketed + ) + SELECT `+rateLimitSnapshotColumns+` + FROM ranked + WHERE rn = 1 + ORDER BY jd ASC, id ASC + LIMIT ?`, + args..., + ) + if err != nil { + return nil, fmt.Errorf("querying rate limit snapshot history: %w", err) + } + defer rows.Close() + + var out []RateLimitSnapshot + for rows.Next() { + s, err := scanRateLimitSnapshot(rows) + if err != nil { + return nil, fmt.Errorf("scanning rate limit snapshot: %w", err) + } + out = append(out, s) + } + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("iterating rate limit snapshot history: %w", err) + } + return out, nil +} + +// CopyRateLimitSnapshotsFrom copies rate_limit_snapshots rows from the +// database file at sourcePath into this archive during a resync, the +// same way CopyModelPricingFrom copies model_pricing; INSERT OR IGNORE +// against the unique dedup_key index makes re-running a resync (or +// copying from a source that shares rows with the destination) safe. +// retainedSessionIDs is the union of CopyTrashedDataFrom's and +// CopyOrphanedDataFromExcluding's return values -- sessions copied here +// verbatim, without a reparse. See docs/agents/storage.md for the full +// row-eligibility and session_id-nulling rules this implements. +func (db *DB) CopyRateLimitSnapshotsFrom(sourcePath string, retainedSessionIDs []string) error { + db.mu.Lock() + defer db.mu.Unlock() + + // Pin a single connection: ATTACH is connection-scoped and + // database/sql's pool doesn't guarantee the same underlying + // connection across separate Exec calls. + ctx := context.Background() + conn, err := db.getWriter().Conn(ctx) + if err != nil { + return fmt.Errorf("acquiring connection: %w", err) + } + defer conn.Close() + + if _, err := conn.ExecContext( + ctx, "ATTACH DATABASE ? AS old_rate_limits_db", sourcePath, + ); err != nil { + return fmt.Errorf("attaching source db: %w", err) + } + defer func() { + _, _ = conn.ExecContext(ctx, "DETACH DATABASE old_rate_limits_db") + }() + + var hasTable bool + if err := conn.QueryRowContext(ctx, `SELECT EXISTS( + SELECT 1 FROM old_rate_limits_db.sqlite_master + WHERE type = 'table' AND name = 'rate_limit_snapshots' + )`).Scan(&hasTable); err != nil { + return fmt.Errorf("checking rate limit snapshots storage: %w", err) + } + if !hasTable { + return nil + } + + // _retained_rate_limit_session_ids holds retainedSessionIDs, the same + // way CopyOrphanedDataFromExcluding stages caller-supplied ids in + // _extra_excluded_orphan_ids: a temp table so the id list can + // participate in the SQL below without a per-id round trip or a + // giant IN (...) literal. + if _, err := conn.ExecContext(ctx, ` + CREATE TEMP TABLE _retained_rate_limit_session_ids ( + id TEXT PRIMARY KEY + )`, + ); err != nil { + return fmt.Errorf("creating retained session ids: %w", err) + } + defer func() { + _, _ = conn.ExecContext(ctx, "DROP TABLE IF EXISTS _retained_rate_limit_session_ids") + }() + if len(retainedSessionIDs) > 0 { + idsTx, err := conn.BeginTx(ctx, nil) + if err != nil { + return fmt.Errorf("begin retained session ids: %w", err) + } + stmt, err := idsTx.PrepareContext(ctx, + "INSERT OR IGNORE INTO _retained_rate_limit_session_ids (id) VALUES (?)", + ) + if err != nil { + _ = idsTx.Rollback() + return fmt.Errorf("prepare retained session ids: %w", err) + } + for _, id := range retainedSessionIDs { + if id == "" { + continue + } + if _, err := stmt.ExecContext(ctx, id); err != nil { + _ = stmt.Close() + _ = idsTx.Rollback() + return fmt.Errorf("insert retained session id %s: %w", id, err) + } + } + if err := stmt.Close(); err != nil { + _ = idsTx.Rollback() + return fmt.Errorf("close retained session ids: %w", err) + } + if err := idsTx.Commit(); err != nil { + return fmt.Errorf("commit retained session ids: %w", err) + } + } + + tx, err := conn.BeginTx(ctx, nil) + if err != nil { + return fmt.Errorf("beginning rate limit snapshots copy: %w", err) + } + defer func() { _ = tx.Rollback() }() + + // session_id is NULLed unless the session exists in the destination + // (see docs/agents/storage.md); dedup_key and observation_key are + // copied unchanged, never recomputed. The WHERE clause below is what + // keeps a rebuilt session's own fresh rows from being overwritten by + // its stale source rows. + if _, err := tx.ExecContext(ctx, ` + INSERT OR IGNORE INTO rate_limit_snapshots ( + vendor, session_id, machine, account_id, account_label, + limit_id, limit_name, plan_type, window_kind, + used_percent, window_minutes, resets_at, + credits_has, credits_unlimited, credits_balance, + rate_limit_reached_type, scope_label, details, + observed_at, ordinal, dedup_key, observation_key + ) + SELECT o.vendor, + CASE + WHEN o.session_id IS NOT NULL + AND EXISTS (SELECT 1 FROM sessions s WHERE s.id = o.session_id) + THEN o.session_id + ELSE NULL + END, + o.machine, o.account_id, o.account_label, + o.limit_id, o.limit_name, o.plan_type, o.window_kind, + o.used_percent, o.window_minutes, o.resets_at, + o.credits_has, o.credits_unlimited, o.credits_balance, + o.rate_limit_reached_type, o.scope_label, o.details, + o.observed_at, o.ordinal, o.dedup_key, o.observation_key + FROM old_rate_limits_db.rate_limit_snapshots o + WHERE o.session_id IS NULL + OR NOT EXISTS (SELECT 1 FROM sessions s WHERE s.id = o.session_id) + OR EXISTS ( + SELECT 1 FROM _retained_rate_limit_session_ids r + WHERE r.id = o.session_id + )`, + ); err != nil { + return fmt.Errorf("copying rate limit snapshots: %w", err) + } + if err := tx.Commit(); err != nil { + return fmt.Errorf("committing rate limit snapshots copy: %w", err) + } + return nil +} diff --git a/internal/db/rate_limit_snapshots_test.go b/internal/db/rate_limit_snapshots_test.go new file mode 100644 index 0000000000..3d376cebfa --- /dev/null +++ b/internal/db/rate_limit_snapshots_test.go @@ -0,0 +1,272 @@ +package db + +import ( + "context" + "path/filepath" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// rlSnap builds a minimal RateLimitSnapshot fixture; callers override +// additional fields on the returned value. +func rlSnap(sessionID, machine, limitID, windowKind, observedAt string, usedPercent float64) RateLimitSnapshot { + return RateLimitSnapshot{ + SessionID: sessionID, Machine: machine, LimitID: limitID, PlanType: "pro", + WindowKind: windowKind, WindowMinutes: 10080, UsedPercent: usedPercent, ObservedAt: observedAt, + } +} + +// TestInsertRateLimitSnapshots_NullableResetsAt: an unknown reset time +// round-trips as nil, not flattened to 0. +func TestInsertRateLimitSnapshots_NullableResetsAt(t *testing.T) { + d := testDB(t) + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-1", Agent: "codex"})) + require.NoError(t, d.InsertRateLimitSnapshots([]RateLimitSnapshot{ + rlSnap("codex:sess-1", "laptop", "codex", "primary", "2026-09-09T10:00:00Z", 95), + })) + rows, err := d.RateLimitSnapshotHistory(context.Background(), RateLimitHistoryFilter{Machine: "laptop"}) + require.NoError(t, err) + require.Len(t, rows, 1) + assert.Nil(t, rows[0].ResetsAt, "a null resets_at must not be flattened to 0") +} + +// TestLatestRateLimitSnapshots_ObservationResolution pins the +// observation_key invariant: "latest" resolves per whole observation, not +// per window_kind, and a NULL session_id left by a deleted session must +// never merge or split distinct observations. +func TestLatestRateLimitSnapshots_ObservationResolution(t *testing.T) { + const at = "2026-09-09T10:00:00Z" + cases := []struct { + name string + build func(t *testing.T, d *DB) + check func(t *testing.T, rows []RateLimitSnapshot) + }{ + {"newer observation omitting a window supersedes it", func(t *testing.T, d *DB) { + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-1", Agent: "codex"})) + require.NoError(t, d.InsertRateLimitSnapshots([]RateLimitSnapshot{ + rlSnap("codex:sess-1", "laptop", "codex", "primary", "2026-09-09T09:00:00Z", 80), + rlSnap("codex:sess-1", "laptop", "codex", "secondary", "2026-09-09T09:00:00Z", 10), + rlSnap("codex:sess-1", "laptop", "codex", "primary", at, 95), + })) + }, func(t *testing.T, rows []RateLimitSnapshot) { + require.Len(t, rows, 1) + assert.Equal(t, "primary", rows[0].WindowKind) + }}, + {"two sessions observing at the same instant do not merge", func(t *testing.T, d *DB) { + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-1", Agent: "codex"})) + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-2", Agent: "codex"})) + require.NoError(t, d.InsertRateLimitSnapshots([]RateLimitSnapshot{ + rlSnap("codex:sess-1", "laptop", "codex", "secondary", at, 5), + rlSnap("codex:sess-2", "laptop", "codex", "primary", at, 77), + })) + }, func(t *testing.T, rows []RateLimitSnapshot) { + require.Len(t, rows, 1) + assert.Equal(t, "codex:sess-2", rows[0].SessionID) + }}, + {"deleting both colliding sessions keeps observations apart", func(t *testing.T, d *DB) { + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-1", Agent: "codex"})) + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-2", Agent: "codex"})) + require.NoError(t, d.InsertRateLimitSnapshots([]RateLimitSnapshot{ + rlSnap("codex:sess-1", "laptop", "codex", "secondary", at, 5), + rlSnap("codex:sess-2", "laptop", "codex", "primary", at, 77), + })) + require.NoError(t, d.DeleteSession("codex:sess-1")) + require.NoError(t, d.DeleteSession("codex:sess-2")) + }, func(t *testing.T, rows []RateLimitSnapshot) { + require.Len(t, rows, 1) + assert.Empty(t, rows[0].SessionID) + }}, + {"deleting one session keeps its sibling windows together", func(t *testing.T, d *DB) { + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-1", Agent: "codex"})) + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-2", Agent: "codex"})) + require.NoError(t, d.InsertRateLimitSnapshots([]RateLimitSnapshot{ + rlSnap("codex:sess-1", "laptop", "codex", "primary", at, 60), + rlSnap("codex:sess-1", "laptop", "codex", "secondary", at, 15), + rlSnap("codex:sess-2", "desktop", "codex", "primary", at, 30), + })) + require.NoError(t, d.DeleteSession("codex:sess-1")) + }, func(t *testing.T, rows []RateLimitSnapshot) { + require.Len(t, rows, 3) + empty := 0 + for _, r := range rows { + if r.SessionID == "" { + empty++ + } + } + assert.Equal(t, 2, empty, "both of the deleted session's windows survive") + }}, + {"a newer observation with empty plan_type keeps the latest non-empty label", func(t *testing.T, d *DB) { + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-1", Agent: "codex"})) + newer := rlSnap("codex:sess-1", "laptop", "codex", "primary", at, 95) + newer.PlanType = "" + require.NoError(t, d.InsertRateLimitSnapshots([]RateLimitSnapshot{rlSnap("codex:sess-1", "laptop", "codex", "primary", "2026-09-09T09:00:00Z", 80), newer})) + }, func(t *testing.T, rows []RateLimitSnapshot) { + require.Len(t, rows, 1) + assert.Equal(t, "pro", rows[0].PlanType, "plan_type falls back to the latest non-empty label") + }}, + {"an authoritative reparse supersedes a fallback parse's rows", func(t *testing.T, d *DB) { + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-1", Agent: "codex"})) + require.NoError(t, d.InsertRateLimitSnapshots([]RateLimitSnapshot{rlSnap("codex:sess-1", "laptop", "codex", "primary", "2026-09-09T09:00:00Z", 40)})) + require.NoError(t, d.InsertRateLimitSnapshotsReplacingSession("codex:sess-1", []RateLimitSnapshot{rlSnap("codex:sess-1", "laptop", "codex", "primary", at, 95)})) + }, func(t *testing.T, rows []RateLimitSnapshot) { + require.Len(t, rows, 1, "the superseded fallback row must not survive") + assert.Equal(t, 95.0, rows[0].UsedPercent) + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + d := testDB(t) + tc.build(t, d) + rows, err := d.LatestRateLimitSnapshots(context.Background(), RateLimitFilter{}) + require.NoError(t, err) + tc.check(t, rows) + }) + } +} + +// TestLatestRateLimitSnapshots_AccountIDPassesThroughAccountlessVendor: an account_id filter must not exclude a vendor whose rows carry no account. +func TestLatestRateLimitSnapshots_AccountIDPassesThroughAccountlessVendor(t *testing.T) { + d := testDB(t) + require.NoError(t, d.UpsertSession(Session{ID: "claude:sess-a", Agent: "claude"})) + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-1", Agent: "codex"})) + acctA := rlSnap("claude:sess-a", "laptop", "5h", "primary", "2026-09-09T10:00:00Z", 30) + acctA.Vendor, acctA.AccountID = "claude", "acct-a" + require.NoError(t, d.InsertRateLimitSnapshots([]RateLimitSnapshot{acctA, rlSnap("codex:sess-1", "laptop", "codex", "primary", "2026-09-09T10:00:00Z", 50)})) + rows, err := d.LatestRateLimitSnapshots(context.Background(), RateLimitFilter{AccountID: "acct-a"}) + require.NoError(t, err) + assert.Len(t, rows, 2, "account A's row plus the account-less Codex row must both return") +} + +// TestRateLimitFilter_AgentAloneScopesVendorInSQL: Agent alone (Vendor +// unset) must scope the SQL itself, not just an early-return check. +func TestRateLimitFilter_AgentAloneScopesVendorInSQL(t *testing.T) { + d := testDB(t) + require.NoError(t, d.UpsertSession(Session{ID: "claude:sess-a", Agent: "claude"})) + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-1", Agent: "codex"})) + claude := rlSnap("claude:sess-a", "laptop", "5h", "primary", "2026-09-09T10:00:00Z", 30) + claude.Vendor = "claude" + codex := rlSnap("codex:sess-1", "laptop", "codex", "primary", "2026-09-09T10:00:00Z", 50) + require.NoError(t, d.InsertRateLimitSnapshots([]RateLimitSnapshot{claude, codex})) + latest, err := d.LatestRateLimitSnapshots(context.Background(), RateLimitFilter{Agent: "codex"}) + require.NoError(t, err) + require.Len(t, latest, 1, "codex only, not every vendor") + history, err := d.RateLimitSnapshotHistory(context.Background(), RateLimitHistoryFilter{Agent: "codex"}) + require.NoError(t, err) + require.Len(t, history, 1, "history must scope by Agent too") +} + +// TestRateLimitSnapshotHistory_MachineFilter: a comma-separated machine +// list must OR-match, not compare as one opaque string. +func TestRateLimitSnapshotHistory_MachineFilter(t *testing.T) { + d := testDB(t) + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-1", Agent: "codex"})) + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-2", Agent: "codex"})) + require.NoError(t, d.InsertRateLimitSnapshots([]RateLimitSnapshot{ + rlSnap("codex:sess-1", "laptop", "codex", "primary", "2026-09-08T00:00:00Z", 20), + rlSnap("codex:sess-2", "desktop", "codex", "primary", "2026-09-08T00:00:00Z", 50), + rlSnap("codex:sess-2", "phone", "codex", "primary", "2026-09-08T00:00:00Z", 50), + })) + rows, err := d.RateLimitSnapshotHistory(context.Background(), RateLimitHistoryFilter{ + Machine: "laptop,desktop", LimitID: "codex", WindowKind: "primary", + }) + require.NoError(t, err) + assert.Len(t, rows, 2, "a comma-separated machine list must use IN semantics") +} + +// TestRateLimitSnapshotHistory_DownsamplesWideRange: more matching rows +// than MaxPoints must come back capped at MaxPoints and still ascending. +func TestRateLimitSnapshotHistory_DownsamplesWideRange(t *testing.T) { + d := testDB(t) + require.NoError(t, d.UpsertSession(Session{ID: "codex:sess-1", Agent: "codex"})) + const totalRows, maxPoints = 50, 10 + base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) + snapshots := make([]RateLimitSnapshot, totalRows) + for i := range snapshots { + snapshots[i] = rlSnap("codex:sess-1", "laptop", "codex", "primary", + base.Add(time.Duration(i)*time.Hour).Format(time.RFC3339Nano), float64(i)) + } + require.NoError(t, d.InsertRateLimitSnapshots(snapshots)) + rows, err := d.RateLimitSnapshotHistory(context.Background(), + RateLimitHistoryFilter{Machine: "laptop", MaxPoints: maxPoints}) + require.NoError(t, err) + require.LessOrEqual(t, len(rows), maxPoints, "a wide range must be downsampled to at most MaxPoints") + for i := 1; i < len(rows); i++ { + prev, _ := time.Parse(time.RFC3339Nano, rows[i-1].ObservedAt) + cur, _ := time.Parse(time.RFC3339Nano, rows[i].ObservedAt) + assert.True(t, cur.After(prev), "downsampled rows must stay strictly ascending") + } +} + +// TestRateLimitAgentMatchesVendor: an exact-equality check would wrongly +// reject "codex" named alongside another agent. +func TestRateLimitAgentMatchesVendor(t *testing.T) { + assert.True(t, RateLimitAgentMatchesVendor("", "codex"), "no filter matches everything") + assert.True(t, RateLimitAgentMatchesVendor("claude,codex", "codex"), "codex named alongside another agent") + assert.False(t, RateLimitAgentMatchesVendor("claude", "codex"), "a single other agent matches nothing") +} + +// TestCopyRateLimitSnapshotsFrom_NullsSessionIDForUnrestoredSession pins the resync-copy invariants: an unrestored source row is copied with +// session_id NULLed and its dedup_key preserved, while a session the resync's own reparse already rebuilt keeps only its fresh row. +func TestCopyRateLimitSnapshotsFrom_NullsSessionIDForUnrestoredSession(t *testing.T) { + dir := t.TempDir() + srcPath := filepath.Join(dir, "src.db") + srcDB := testDBAtPath(t, srcPath, "src") + require.NoError(t, srcDB.UpsertSession(Session{ID: "codex:superseded", Agent: "codex"})) + require.NoError(t, srcDB.UpsertSession(Session{ID: "codex:rebuilt", Agent: "codex"})) + superseded := rlSnap("codex:superseded", "laptop", "codex", "primary", "2026-09-09T10:00:00Z", 0) + require.NoError(t, srcDB.InsertRateLimitSnapshots([]RateLimitSnapshot{ + superseded, rlSnap("codex:rebuilt", "laptop", "codex", "primary", "2026-09-09T09:00:00Z", 40), + })) + srcDB.Close() + wantDedupKey := RateLimitSnapshotDedupKey(superseded) + + // The destination does not restore "codex:superseded" but already reparsed "codex:rebuilt" with its own current row before the copy runs. + dstPath := filepath.Join(dir, "dst.db") + dstDB := testDBAtPath(t, dstPath, "dst") + defer dstDB.Close() + require.NoError(t, dstDB.UpsertSession(Session{ID: "codex:rebuilt", Agent: "codex"})) + require.NoError(t, dstDB.InsertRateLimitSnapshots([]RateLimitSnapshot{rlSnap("codex:rebuilt", "laptop", "codex", "primary", "2026-09-09T11:00:00Z", 95)})) + require.NoError(t, dstDB.CopyRateLimitSnapshotsFrom(srcPath, nil)) + + copied, err := dstDB.RateLimitSnapshotHistory(context.Background(), RateLimitHistoryFilter{}) + require.NoError(t, err) + require.Len(t, copied, 2, "unrestored row copied, rebuilt session's stale row skipped") + for _, r := range copied { + if r.SessionID == "codex:rebuilt" { + assert.Equal(t, 95.0, r.UsedPercent, "the rebuilt session's stale row must not resurface") + continue + } + assert.Empty(t, r.SessionID, "session_id must be NULLed for a session absent from the destination") + assert.Equal(t, wantDedupKey, r.DedupKey, "dedup_key must be preserved from the source, not recomputed") + } +} + +// TestInsertRateLimitSnapshots_ReattachesDetachedRow: a session copied as +// NULL-session by a resync must be reattached, not ignored, once that +// session's dedup_key reappears via a normal insert. +func TestInsertRateLimitSnapshots_ReattachesDetachedRow(t *testing.T) { + dir := t.TempDir() + srcPath := filepath.Join(dir, "src.db") + srcDB := testDBAtPath(t, srcPath, "src") + require.NoError(t, srcDB.UpsertSession(Session{ID: "codex:missing", Agent: "codex"})) + snap := rlSnap("codex:missing", "laptop", "codex", "primary", "2026-09-09T10:00:00Z", 40) + require.NoError(t, srcDB.InsertRateLimitSnapshots([]RateLimitSnapshot{snap})) + srcDB.Close() + + dstDB := testDB(t) + require.NoError(t, dstDB.CopyRateLimitSnapshotsFrom(srcPath, nil)) + detached, err := dstDB.RateLimitSnapshotHistory(context.Background(), RateLimitHistoryFilter{}) + require.NoError(t, err) + require.Len(t, detached, 1) + assert.Empty(t, detached[0].SessionID, "copied row starts detached") + + require.NoError(t, dstDB.UpsertSession(Session{ID: "codex:missing", Agent: "codex"})) + require.NoError(t, dstDB.InsertRateLimitSnapshots([]RateLimitSnapshot{snap})) + reattached, err := dstDB.RateLimitSnapshotHistory(context.Background(), RateLimitHistoryFilter{}) + require.NoError(t, err) + require.Len(t, reattached, 1, "reattachment must not duplicate the row") + assert.Equal(t, "codex:missing", reattached[0].SessionID, "the detached row must be reattached, not left stale") +} diff --git a/internal/db/read_only_test.go b/internal/db/read_only_test.go index f4a6e3bed4..d481c9d20b 100644 --- a/internal/db/read_only_test.go +++ b/internal/db/read_only_test.go @@ -442,6 +442,17 @@ func TestOpenReadOnlyAllowsMissingFTSTable(t *testing.T) { assert.False(t, readonly.HasFTS()) } +// TestOpenReadOnlyToleratesMissingRateLimitSnapshotsTable: a pre-table archive opened read-only must not fail Latest/History with "no such table". +func TestOpenReadOnlyToleratesMissingRateLimitSnapshotsTable(t *testing.T) { + path := createClosedTestDB(t, tempDBPath(t, "sessions.db"), nil) + execRawSQLite(t, path, "DROP TABLE IF EXISTS rate_limit_snapshots") + readonly := openReadOnlyTestDB(t, path) + _, err := readonly.LatestRateLimitSnapshots(context.Background(), RateLimitFilter{}) + require.NoError(t, err) + _, err = readonly.RateLimitSnapshotHistory(context.Background(), RateLimitHistoryFilter{}) + require.NoError(t, err) +} + func TestOpenReadOnlyCopyHelpersReturnErrReadOnly(t *testing.T) { dir := t.TempDir() srcPath := filepath.Join(dir, "source.db") diff --git a/internal/db/schema.sql b/internal/db/schema.sql index 187d8946a0..cc265d6475 100644 --- a/internal/db/schema.sql +++ b/internal/db/schema.sql @@ -311,6 +311,104 @@ CREATE INDEX IF NOT EXISTS idx_cursor_usage_events_occurred CREATE INDEX IF NOT EXISTS idx_cursor_usage_events_model ON cursor_usage_events(model); +-- Rate-limit snapshots. Each row is one rate-limit window (5h "primary", +-- weekly "secondary", ...) observed for a vendor at a point in time -- +-- today, always a Codex token_count event's rate_limits payload, +-- alongside the plan type and credit balance reported at the same +-- instant. SQLite-only, following the same vendor-data precedent as +-- cursor_usage_events above and the Codex incremental-import tables +-- documented in docs/agents/storage.md: it is not part of the +-- SQLite/PostgreSQL/DuckDB parity contract. +-- +-- `vendor` is NOT NULL from the start (always 'codex' today) so a future +-- vendor's rows never need a backfill or a migration to add the column; +-- `account_id`/`account_label`/`scope_label`/`details` are reserved the +-- same way and always '' for Codex. +-- +-- Codex rollouts carry no stable per-account identifier (no account id, +-- user id, email, or org field appears in session_meta or token_count +-- payloads as of 2026-09 -- see docs/internal/session-format-sources.md), +-- so a Codex snapshot's identity is (machine, limit_id, plan_type, +-- window_kind) rather than an account. +CREATE TABLE IF NOT EXISTS rate_limit_snapshots ( + id INTEGER PRIMARY KEY, + vendor TEXT NOT NULL DEFAULT 'codex', + session_id TEXT REFERENCES sessions(id) ON DELETE SET NULL, + machine TEXT NOT NULL DEFAULT '', + account_id TEXT NOT NULL DEFAULT '', + account_label TEXT NOT NULL DEFAULT '', + limit_id TEXT NOT NULL DEFAULT '', + limit_name TEXT NOT NULL DEFAULT '', + plan_type TEXT NOT NULL DEFAULT '', + window_kind TEXT NOT NULL, + used_percent REAL NOT NULL DEFAULT 0, + window_minutes INTEGER NOT NULL DEFAULT 0, + resets_at INTEGER, + credits_has INTEGER NOT NULL DEFAULT 0, + credits_unlimited INTEGER NOT NULL DEFAULT 0, + credits_balance TEXT NOT NULL DEFAULT '', + rate_limit_reached_type TEXT NOT NULL DEFAULT '', + scope_label TEXT NOT NULL DEFAULT '', + details TEXT NOT NULL DEFAULT '', + observed_at TEXT NOT NULL, + -- ordinal is the source token_count event's stable per-file position + -- (see docs/agents/storage.md and parser.ParsedRateLimitSnapshot). + -- Folded into dedup_key and observation_key below so two distinct + -- token_count events sharing an observed_at second stay distinct + -- rows instead of colliding. + ordinal INTEGER NOT NULL DEFAULT 0, + dedup_key TEXT NOT NULL DEFAULT '', + -- observation_key identifies the single source observation a row + -- came from (shared by the up-to-two window rows -- primary, + -- secondary -- one Codex rate_limits payload produces). Computed + -- from session_id+observed_at at insert time and stored + -- independently of the nullable session_id column, so sibling + -- windows stay grouped for LatestRateLimitSnapshots even after the + -- source session is deleted or excluded from a resync. + observation_key TEXT NOT NULL DEFAULT '' +); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_rate_limit_snapshots_dedup + ON rate_limit_snapshots(dedup_key) + WHERE dedup_key != ''; +CREATE INDEX IF NOT EXISTS idx_rate_limit_snapshots_machine_limit_observed + ON rate_limit_snapshots(machine, limit_id, observed_at); +CREATE INDEX IF NOT EXISTS idx_rate_limit_snapshots_session + ON rate_limit_snapshots(session_id); +-- Backs LatestRateLimitSnapshots' bucket-max lookup and its per-bucket +-- plan_type/limit_name correlated subqueries, and RateLimitSnapshotHistory's +-- range scan: all three key on this same (vendor, machine, account_id, +-- limit_id) bucket with observed_at trailing, so SQLite can seek a bucket's +-- newest row directly off the index instead of ranking the whole filtered +-- set with a window function. Applied on every writable open (this file +-- re-runs in full; see execSchemaScriptLocked), so an existing archive +-- picks it up without a separate migration. +CREATE INDEX IF NOT EXISTS idx_rate_limit_snapshots_bucket_observed + ON rate_limit_snapshots(vendor, machine, account_id, limit_id, observed_at); +-- julianday(observed_at), not the raw column, because every "most recent +-- observation" lookup orders by julianday(observed_at) -- raw RFC3339Nano +-- text does not sort chronologically once two timestamps differ in +-- fractional-second width (see normalizeRateLimitBoundary) -- and without +-- a matching expression index SQLite falls back to a temp-b-tree sort of +-- the whole bucket for that ORDER BY ... LIMIT 1, even though the bucket +-- itself is reached by an index seek on the leading columns. +CREATE INDEX IF NOT EXISTS idx_rate_limit_snapshots_bucket_jd + ON rate_limit_snapshots(vendor, machine, account_id, limit_id, julianday(observed_at), id); +-- Partial indexes backing the "latest non-empty plan_type/limit_name" +-- lookups: without a partial index whose WHERE clause matches the +-- subquery's own "plan_type != ''" (or "limit_name != ''") filter, +-- idx_rate_limit_snapshots_bucket_jd can seek to the bucket's newest row +-- but then has to walk backward past every row that fails the filter -- +-- the whole bucket, in the worst case a label that is empty on every +-- observation ever recorded for it -- before concluding there is no +-- match. These let that search land only on rows that could qualify. +CREATE INDEX IF NOT EXISTS idx_rate_limit_snapshots_bucket_jd_plan_type + ON rate_limit_snapshots(vendor, machine, account_id, limit_id, julianday(observed_at), id) + WHERE plan_type != ''; +CREATE INDEX IF NOT EXISTS idx_rate_limit_snapshots_bucket_jd_limit_name + ON rate_limit_snapshots(vendor, machine, account_id, limit_id, julianday(observed_at), id) + WHERE limit_name != ''; + -- Tool calls table CREATE TABLE IF NOT EXISTS tool_calls ( id INTEGER PRIMARY KEY, diff --git a/internal/db/session_batch.go b/internal/db/session_batch.go index 3b07b8be34..feaa9a7bf5 100644 --- a/internal/db/session_batch.go +++ b/internal/db/session_batch.go @@ -17,6 +17,7 @@ type SessionBatchWrite struct { Session Session Messages []Message UsageEvents []UsageEvent + RateLimitSnapshots []RateLimitSnapshot IdentityObservation export.ProjectIdentityObservation // IdentitySnapshotProject distinguishes legacy omission (nil, use the // aggregate project) from an explicit empty parser source (omit snapshot). @@ -544,6 +545,24 @@ func writeOneSessionBatchTx( ); err != nil { return 0, err } + if replaceMessages { + // This is the same full-replacement path used for an + // authoritative reparse superseding a fallback marked + // parser.DataVersionNeedsRetry: without this delete, that + // fallback's rate_limit_snapshots rows would outlive the parse + // that superseded them (see docs/agents/storage.md). An + // incremental append (replaceMessages false) must not delete. + if err := deleteRateLimitSnapshotsForSessionTx( + queries, write.Session.ID, + ); err != nil { + return 0, err + } + } + if err := insertRateLimitSnapshotsTx( + ctx, queries, write.RateLimitSnapshots, + ); err != nil { + return 0, err + } msgs := write.Messages var pins []savedPin diff --git a/internal/db/sessions.go b/internal/db/sessions.go index 7a98f4883c..0860b1ad31 100644 --- a/internal/db/sessions.go +++ b/internal/db/sessions.go @@ -2761,6 +2761,12 @@ type IncrementalSessionUpdate struct { SubagentLinks []ToolCallSubagentLink ToolCallResultUpdates []ToolCallResultUpdate MessageTokenUsageUpdates []MessageTokenUsageUpdate + // RateLimitSnapshots carries Codex rate_limits observations parsed + // from this delta's appended tail. Rows are inserted with INSERT OR + // IGNORE against a unique dedup_key rather than replaced, so this can + // be a non-exhaustive incremental slice without risking duplicates or + // losing earlier history. + RateLimitSnapshots []RateLimitSnapshot // Checkpoint/CheckpointBlobs are the machine-local parser checkpoint // metadata and lazy payload to persist in the same transaction as this // delta. nil keeps any existing checkpoint. diff --git a/internal/db/store.go b/internal/db/store.go index 4127c84859..845b1dba3f 100644 --- a/internal/db/store.go +++ b/internal/db/store.go @@ -99,6 +99,11 @@ type Store interface { GetUsageMatchingSessionCount(ctx context.Context, f UsageFilter) (int, error) GetSessionUsage(ctx context.Context, sessionID string, includeBreakdown bool) (*SessionUsage, error) + // Rate limits. SQLite-only (see docs/agents/storage.md); other + // backends return an empty slice so the Usage page hides the section. + LatestRateLimitSnapshots(ctx context.Context, f RateLimitFilter) ([]RateLimitSnapshot, error) + RateLimitSnapshotHistory(ctx context.Context, f RateLimitHistoryFilter) ([]RateLimitSnapshot, error) + // Stars. StarSession(sessionID string) (bool, error) UnstarSession(sessionID string) error diff --git a/internal/duckdb/rate_limits.go b/internal/duckdb/rate_limits.go new file mode 100644 index 0000000000..e105324a6d --- /dev/null +++ b/internal/duckdb/rate_limits.go @@ -0,0 +1,29 @@ +package duckdb + +import ( + "context" + + "go.kenn.io/agentsview/internal/db" +) + +// LatestRateLimitSnapshots and RateLimitSnapshotHistory are SQLite-only +// (see docs/agents/storage.md): rate_limit_snapshots is a local +// vendor-data table, not part of the SQLite/PostgreSQL/DuckDB parity +// contract, and DuckDB is a disposable read mirror besides. The DuckDB +// reader returns an empty result rather than an error so the Usage page's +// rate-limits section simply stays hidden when DuckDB is the active read +// backend. + +// LatestRateLimitSnapshots is not supported by the DuckDB backend. +func (s *Store) LatestRateLimitSnapshots( + _ context.Context, _ db.RateLimitFilter, +) ([]db.RateLimitSnapshot, error) { + return nil, nil +} + +// RateLimitSnapshotHistory is not supported by the DuckDB backend. +func (s *Store) RateLimitSnapshotHistory( + _ context.Context, _ db.RateLimitHistoryFilter, +) ([]db.RateLimitSnapshot, error) { + return nil, nil +} diff --git a/internal/parser/codex.go b/internal/parser/codex.go index a35fa55327..622fdf55e9 100644 --- a/internal/parser/codex.go +++ b/internal/parser/codex.go @@ -79,7 +79,17 @@ type codexSessionBuilder struct { committedUsageTarget *int committedUsageBlockedByUser bool messageUsageUpdates []ParsedMessageTokenUsageUpdate - checkpointUnsafe bool + rateLimitSnapshots []ParsedRateLimitSnapshot + // discardRateLimitSnapshots skips accumulating rateLimitSnapshots + // entirely. Set for builders that reconstruct cursor state from a + // full prefix scan (seedCodexIncrementalStateFromReader) without + // ever returning a session result: those observations are neither + // read nor persisted, so collecting them would grow unboundedly + // with the scanned prefix -- proportional to the whole rollout, not + // the incremental tail a cache hit would otherwise read -- for no + // benefit on every incremental-parse cache miss. + discardRateLimitSnapshots bool + checkpointUnsafe bool // Calls beyond the persisted cursor's capacity remain parse-local until // enough results arrive to fit the bounded checkpoint again. overflowPendingCalls map[string]codexPendingToolCall @@ -400,7 +410,7 @@ func (b *codexSessionBuilder) processLine( if b.suppresses(codexTypeEventMsg, payload) { return false } - b.handleEventMsg(payload) + b.handleEventMsg(payload, ts) } return false } @@ -538,13 +548,15 @@ func (b *codexSessionBuilder) handleAgentMessage( }) } -func (b *codexSessionBuilder) handleEventMsg(payload gjson.Result) { +func (b *codexSessionBuilder) handleEventMsg( + payload gjson.Result, ts time.Time, +) { eventType := payload.Get("type").Str switch eventType { case "task_started", "task_complete", "turn_aborted": b.observeTaskEvent(eventType) case "token_count": - b.handleTokenCountEvent(payload) + b.handleTokenCountEvent(payload, ts) case "collab_agent_spawn_end": b.handleCollabAgentSpawnEnd(payload) case "sub_agent_activity": @@ -557,8 +569,24 @@ func (b *codexSessionBuilder) markFirstUserReplayPossible() { } func (b *codexSessionBuilder) handleTokenCountEvent( - payload gjson.Result, + payload gjson.Result, ts time.Time, ) { + // ordinal is this token_count event's 0-based position among every + // token_count event in the file, counted unconditionally -- even + // when discardRateLimitSnapshots is set, or rate_limits itself is + // absent -- so a prefix rescan that seeds an incremental parse's + // cursor keeps the counter in lockstep with what a full parse would + // have counted by the same offset. See ParsedRateLimitSnapshot.Ordinal. + ordinal := int(b.tokenCountOrdinal) + b.tokenCountOrdinal++ + + // rate_limits is a sibling of info.last_token_usage, not nested under + // it, and must be captured even when the usage payload itself is + // empty or a duplicate (observeTokenUsage dedups by content below); + // codex_exec heartbeats can repeat identical token usage while still + // reporting a freshly advanced rate-limit window. + b.observeRateLimits(payload.Get("rate_limits"), ts, ordinal) + raw := payload.Get("info.last_token_usage").Raw if raw == "" || b.observeTokenUsage(raw) { return @@ -593,6 +621,112 @@ func (b *codexSessionBuilder) handleTokenCountEvent( b.committedUsageTarget = nil } +// observeRateLimits extracts the rate_limits object carried beside +// info.last_token_usage in a Codex token_count event. rate_limits is +// nullable (older releases and non-primary limit_id rows such as +// "premium" report it as null); a present object nests up to two nullable +// windows, "primary" and "secondary" (e.g. a 5h window and a weekly +// window), each shaped {used_percent, window_minutes, resets_at}. One +// ParsedRateLimitSnapshot is appended per non-null window so the schema +// can key rows by window kind; a rate_limits object with both windows +// null (seen for limit_id "premium", which currently reports credits +// only) carries no window to record and is skipped. SessionID and +// Machine are left blank here — the builder does not reliably know the +// final session identity yet during an incremental tail parse — and are +// filled in by the caller once the enclosing ParsedSession/ +// IncrementalOutcome is assembled. ordinal is the source token_count +// event's stable per-file position (see ParsedRateLimitSnapshot.Ordinal) +// and is stamped onto every window this one event produces. +func (b *codexSessionBuilder) observeRateLimits( + rl gjson.Result, ts time.Time, ordinal int, +) { + if !rl.Exists() || rl.Type == gjson.Null || ts.IsZero() { + return + } + if b.discardRateLimitSnapshots { + return + } + limitID := rl.Get("limit_id").Str + limitName := rl.Get("limit_name").Str + planType := rl.Get("plan_type").Str + reachedType := rl.Get("rate_limit_reached_type").Str + creditsHas := rl.Get("credits.has_credits").Bool() + creditsUnlimited := rl.Get("credits.unlimited").Bool() + creditsBalance := rl.Get("credits.balance").String() + + base := ParsedRateLimitSnapshot{ + LimitID: limitID, + LimitName: limitName, + PlanType: planType, + CreditsHas: creditsHas, + CreditsUnlimited: creditsUnlimited, + CreditsBalance: creditsBalance, + RateLimitReachedType: reachedType, + ObservedAt: ts, + Ordinal: ordinal, + } + + if win, ok := codexRateLimitWindow(rl.Get("primary")); ok { + snap := base + snap.WindowKind = "primary" + snap.UsedPercent = win.usedPercent + snap.WindowMinutes = win.windowMinutes + snap.ResetsAt = win.resetsAt + b.rateLimitSnapshots = append(b.rateLimitSnapshots, snap) + } + if win, ok := codexRateLimitWindow(rl.Get("secondary")); ok { + snap := base + snap.WindowKind = "secondary" + snap.UsedPercent = win.usedPercent + snap.WindowMinutes = win.windowMinutes + snap.ResetsAt = win.resetsAt + b.rateLimitSnapshots = append(b.rateLimitSnapshots, snap) + } +} + +type codexRateLimitWindowValue struct { + usedPercent float64 + // windowMinutes is nil when the window's window_minutes field is + // absent or JSON null, which the Codex protocol allows independently + // of the window object itself being present -- distinct from a + // reported duration of zero, which Codex never sends. Preserving + // that as nil (rather than flattening it to 0) keeps a window with + // an unknown duration from being displayed with a bogus "0m" length; + // the card falls back to a window-kind label instead (see + // RateLimitWindow.WindowMinutes). + windowMinutes *int + // resetsAt is nil when the window's resets_at field is absent or + // JSON null, which the Codex protocol allows independently of the + // window object itself being present. + resetsAt *int64 +} + +// codexRateLimitWindow decodes one nullable rate_limits window object +// ({used_percent, window_minutes, resets_at}). resets_at is itself +// independently nullable, distinct from the window as a whole being +// null: preserving that as a nil resetsAt (rather than flattening it to +// 0) keeps a window with an unknown reset time from being displayed as +// if it resets at the unix epoch. +func codexRateLimitWindow( + win gjson.Result, +) (codexRateLimitWindowValue, bool) { + if !win.Exists() || win.Type == gjson.Null { + return codexRateLimitWindowValue{}, false + } + value := codexRateLimitWindowValue{ + usedPercent: win.Get("used_percent").Float(), + } + if windowMinutes := win.Get("window_minutes"); windowMinutes.Exists() && windowMinutes.Type != gjson.Null { + v := int(windowMinutes.Int()) + value.windowMinutes = &v + } + if resetsAt := win.Get("resets_at"); resetsAt.Exists() && resetsAt.Type != gjson.Null { + v := resetsAt.Int() + value.resetsAt = &v + } + return value, true +} + func (b *codexSessionBuilder) handleCollabAgentSpawnEnd( payload gjson.Result, ) { @@ -1771,6 +1905,12 @@ func (p *codexProvider) parseCodexSessionSnapshotStreaming( b := newCodexSessionBuilder( ctx, includeExec, p.parentTurnResolver(ctx, path), sink, ) + if p.spec.agent != AgentCodex { + // TraeX shares this parser (same rollout format) but is not a + // supported rate-limit source; its rate_limits payloads are + // left uncollected rather than persisted under the wrong agent. + b.discardRateLimitSnapshots = true + } malformedLines := 0 for { @@ -1897,6 +2037,16 @@ func (p *codexProvider) parseCodexSessionSnapshotStreaming( ChangeTime: changeTime, }, } + if len(b.rateLimitSnapshots) > 0 { + sess.RateLimitSnapshots = make( + []ParsedRateLimitSnapshot, len(b.rateLimitSnapshots), + ) + for i, snap := range b.rateLimitSnapshots { + snap.SessionID = sessionID + snap.Machine = machine + sess.RateLimitSnapshots[i] = snap + } + } if err := accumulateMessageTokenUsageContext(ctx, sess, msgs); err != nil { return nil, nil, codexCursorState{}, false, nil, "", "", err @@ -2172,6 +2322,12 @@ func seedCodexIncrementalStateFromReader( b := newCodexSessionBuilder( context.Background(), false, resolveParentTurns, sink, ) + // This scan reconstructs cursor state from a (potentially large) + // prefix and never returns a session result -- codexIncrementalSeed + // carries only cursor/pending-call state, not rate-limit history -- + // so collecting rate-limit snapshots here would grow unboundedly + // with the scanned prefix for no benefit. + b.discardRateLimitSnapshots = true lr := newLineReader(r, maxLineSize) defer releaseLineReader(lr) for { @@ -2359,6 +2515,7 @@ type codexIncrementalParseResult struct { messages []ParsedMessage toolCallUpdates []ParsedToolCallUpdate messageUsageUpdates []ParsedMessageTokenUsageUpdate + rateLimitSnapshots []ParsedRateLimitSnapshot endedAt time.Time consumedBytes int64 initialCursor codexCursorState @@ -2531,6 +2688,11 @@ func (p *codexProvider) parseSessionFromWithSources( p.parentTurnResolver(context.Background(), path), NewCodexCollectingSink(startOrdinal), ) + if p.spec.agent != AgentCodex { + // See parseCodexSessionSnapshotStreaming: TraeX is not a + // supported rate-limit source. + b.discardRateLimitSnapshots = true + } b.codexCursorState = seed.codexCursorState b.overflowPendingCalls = seed.overflowPendingCalls if committedUsageTarget != nil { @@ -2588,6 +2750,9 @@ func (p *codexProvider) parseSessionFromWithSources( messageUsageUpdates: append( []ParsedMessageTokenUsageUpdate(nil), b.messageUsageUpdates..., ), + rateLimitSnapshots: append( + []ParsedRateLimitSnapshot(nil), b.rateLimitSnapshots..., + ), endedAt: b.endedAt, consumedBytes: consumed, initialCursor: seed.codexCursorState, diff --git a/internal/parser/codex_cursor.go b/internal/parser/codex_cursor.go index 8304018d63..8c2eaabdda 100644 --- a/internal/parser/codex_cursor.go +++ b/internal/parser/codex_cursor.go @@ -20,12 +20,17 @@ const ( // codexCursorCheckpointVersion is the wire version for the persisted // cursor encoding. Bump when the encoding changes; decode failures fall // back to a full parse. + // Version 5 additionally stores the running token_count event + // ordinal (see codexCursorState.tokenCountOrdinal), so an incremental + // parse resuming from a persisted or cached cursor assigns the same + // ordinal to a later token_count event that a full reparse of the + // whole file would. // Version 4 stores the current reasoning effort alongside the model. // Version 3 replaces duplicate IDs with their latest occurrence, matching // full parsing; version 2 retained the oldest unresolved occurrence. // The fork replay gate is process-only state: it is re-armed from the // transcript on every parse and is not part of the persisted cursor. - codexCursorCheckpointVersion = 4 + codexCursorCheckpointVersion = 5 codexCursorCheckpointMaxString = 1 << 20 // Account for the map bucket, list element, pointers, string headers, and @@ -65,6 +70,10 @@ type codexCursorState struct { pendingCalls [codexCursorMaxPendingCalls]codexPendingToolCall pendingCallCount uint8 pendingCallsOverflow bool + // tokenCountOrdinal is the number of token_count events processed so + // far in this file (equivalently, the 0-based ordinal the next one + // will receive). See ParsedRateLimitSnapshot.Ordinal. + tokenCountOrdinal uint32 } // MarshalBinary encodes the compact continuation state for persistence. @@ -92,6 +101,9 @@ func (s *codexCursorState) MarshalBinary() ([]byte, error) { if err := write(uint8(codexCursorCheckpointVersion)); err != nil { return nil, err } + if err := write(s.tokenCountOrdinal); err != nil { + return nil, err + } for _, str := range []string{s.model, s.reasoningEffort, s.cwd, s.agentPath} { if err := writeStr(str); err != nil { return nil, err @@ -186,6 +198,9 @@ func (s *codexCursorState) UnmarshalBinary(data []byte) error { ) } *s = codexCursorState{} + if err := read(&s.tokenCountOrdinal); err != nil { + return err + } var err error if s.model, err = readStr(); err != nil { return err diff --git a/internal/parser/codex_parser_test.go b/internal/parser/codex_parser_test.go index 043ca7527d..08d5a64ae3 100644 --- a/internal/parser/codex_parser_test.go +++ b/internal/parser/codex_parser_test.go @@ -1676,6 +1676,27 @@ func TestParseCodexSession_TokenUsage(t *testing.T) { }) } +// TestParseCodexSession_RateLimits_NullResetsAt pins that a null +// resets_at/window_minutes parses as nil, not 0. +func TestParseCodexSession_RateLimits_NullResetsAt(t *testing.T) { + content := testjsonl.JoinJSONL( + testjsonl.CodexSessionMetaJSON("rl-sess", "/tmp", "user", tsEarly), + testjsonl.CodexTurnContextJSON("gpt-5.4", tsEarlyS1), + testjsonl.CodexMsgJSON("user", "hello", tsEarlyS1), + testjsonl.CodexMsgJSON("assistant", "hi", tsEarlyS5), + testjsonl.CodexTokenCountWithRateLimitsJSON( + tsEarlyS5, 10000, 500, 6000, "codex", "pro", + &testjsonl.CodexRateLimitWindow{UsedPercent: 40, WindowMinutesNull: true, ResetsAtNull: true}, + nil, "100.0", + ), + ) + sess, _ := runCodexParserTest(t, "test.jsonl", content, false) + require.NotNil(t, sess) + require.Len(t, sess.RateLimitSnapshots, 1) + assert.Nil(t, sess.RateLimitSnapshots[0].ResetsAt, "a null resets_at must not be flattened to 0") + assert.Nil(t, sess.RateLimitSnapshots[0].WindowMinutes, "a null window_minutes must not be flattened to 0") +} + // testUUIDv7 builds a syntactically valid UUIDv7 whose embedded // timestamp is the given unix-millisecond value. func testUUIDv7(ms int64, seq byte) string { diff --git a/internal/parser/codex_provider.go b/internal/parser/codex_provider.go index 37462c1e1b..2e6cf081b5 100644 --- a/internal/parser/codex_provider.go +++ b/internal/parser/codex_provider.go @@ -502,6 +502,7 @@ func (p *codexProvider) Parse( Result: ParseResult{ Session: *sess, Messages: msgs, + RateLimitSnapshots: sess.RateLimitSnapshots, Checkpoint: checkpoint, CheckpointHashState: hashState, CheckpointAnchorDigest: anchorDigest, @@ -714,11 +715,23 @@ func (p *codexProvider) ParseIncremental( ) } } + rateLimitSnapshots := result.rateLimitSnapshots + if len(rateLimitSnapshots) > 0 { + machine := firstNonEmptyJSONLString(req.Machine, p.Config.Machine) + filled := make([]ParsedRateLimitSnapshot, len(rateLimitSnapshots)) + for i, snap := range rateLimitSnapshots { + snap.SessionID = req.SessionID + snap.Machine = machine + filled[i] = snap + } + rateLimitSnapshots = filled + } return IncrementalOutcome{ SessionID: req.SessionID, Messages: result.messages, ToolCallUpdates: result.toolCallUpdates, MessageTokenUsageUpdates: result.messageUsageUpdates, + RateLimitSnapshots: rateLimitSnapshots, NextCursor: nextCursor, EndedAt: result.endedAt, ConsumedBytes: result.consumedBytes, diff --git a/internal/parser/provider.go b/internal/parser/provider.go index 1a823344b7..f63fc413b1 100644 --- a/internal/parser/provider.go +++ b/internal/parser/provider.go @@ -1091,6 +1091,9 @@ type IncrementalOutcome struct { SubagentLinks []ClaudeSubagentLink ToolCallUpdates []ParsedToolCallUpdate MessageTokenUsageUpdates []ParsedMessageTokenUsageUpdate + // RateLimitSnapshots carries Codex rate_limits observations parsed from + // the appended tail. Empty for providers that do not emit them. + RateLimitSnapshots []ParsedRateLimitSnapshot // NextCursor is the provider's continuation state after consuming the // appended tail, for persistence alongside the committed offset. NextCursor []byte diff --git a/internal/parser/traex_test.go b/internal/parser/traex_test.go index f525d78d15..afa8762ac2 100644 --- a/internal/parser/traex_test.go +++ b/internal/parser/traex_test.go @@ -190,6 +190,19 @@ func TestTraeXProviderParseRelabelsCodexSession(t *testing.T) { assert.Equal(t, []string{"traex:" + spawned}, subagentIDs) } +// TestTraeXProviderDiscardsRateLimitSnapshots: TraeX is not a supported rate-limit source, though it shares the Codex-format parser. +func TestTraeXProviderDiscardsRateLimitSnapshots(t *testing.T) { + path := filepath.Join(t.TempDir(), "test.jsonl") + content := testjsonl.JoinJSONL( + testjsonl.CodexSessionMetaJSON("s1", "/tmp", "codex-tui", "2026-08-01T18:07:03Z"), testjsonl.CodexMsgJSON("user", "hi", "2026-08-01T18:07:04Z"), + testjsonl.CodexTokenCountWithRateLimitsJSON("2026-08-01T18:07:05Z", 100, 50, 0, "codex", "pro", &testjsonl.CodexRateLimitWindow{UsedPercent: 40, WindowMinutes: 10080}, nil, "1.0")) + require.NoError(t, os.WriteFile(path, []byte(content), 0o644)) + provider, _ := NewProvider(AgentTraeX, ProviderConfig{Roots: []string{filepath.Dir(path)}}) + sess, _, err := provider.(*codexProvider).parseSession(path, "devbox", false) + require.NoError(t, err) + assert.Empty(t, sess.RateLimitSnapshots, "TraeX must not persist rate-limit snapshots") +} + func TestTraeXProviderIgnoresCopiedCodexSessionIndex(t *testing.T) { root := t.TempDir() sessionsRoot := filepath.Join(root, "sessions") diff --git a/internal/parser/types.go b/internal/parser/types.go index 34769b7519..44f4eb1329 100644 --- a/internal/parser/types.go +++ b/internal/parser/types.go @@ -1303,6 +1303,11 @@ type ParsedSession struct { // the usage_events table for catalog-based cost pricing. UsageEvents []ParsedUsageEvent + // RateLimitSnapshots carries Codex rate_limits observations extracted + // from token_count events. The sync engine forwards these into the + // rate_limit_snapshots table. + RateLimitSnapshots []ParsedRateLimitSnapshot + // CountsAuthoritative marks parsers that own MessageCount and // UserMessageCount even when they intentionally emit no transcript rows. CountsAuthoritative bool @@ -1462,6 +1467,61 @@ type ParsedUsageEvent struct { DedupKey string } +// ParsedRateLimitSnapshot records one Codex rate-limit window observed in a +// token_count event's rate_limits payload. Codex rollouts carry no stable +// account identifier (no account id, user id, email, or org field appears +// anywhere in session_meta or token_count payloads as of 2026-09), so a +// snapshot is identified by (Machine, LimitID, PlanType, WindowKind) rather +// than by account; see docs/internal/session-format-sources.md for the +// evidence entry and docs/token-usage.md for the resulting design note. +// SessionID and Machine are filled in by the caller once the enclosing +// session's identity is known (the builder that emits these during +// line-by-line parsing does not always know either yet, e.g. during an +// incremental parse of a tail chunk that starts after session_meta). +// ObservedAt is the event's envelope timestamp; the db layer derives +// its own dedup key from SessionID + ObservedAt + LimitID + WindowKind + +// Ordinal once the caller has filled in the final SessionID, so +// re-parsing a file cannot duplicate rows. +type ParsedRateLimitSnapshot struct { + SessionID string + Machine string + LimitID string + LimitName string + PlanType string + WindowKind string // "primary" or "secondary" + + // Ordinal is the source token_count event's 0-based position among + // every token_count event in the rollout file, counted in file + // order regardless of whether an event carries a rate_limits + // payload. It is stable across a full parse, an incremental tail + // parse resuming from a cached or seeded cursor, and any later + // re-parse of the same file (see codexCursorState.tokenCountOrdinal), + // so the db layer folds it into dedup_key/observation_key to tell + // apart two token_count events that land on the same observed_at + // second -- otherwise indistinguishable by SessionID+ObservedAt+ + // LimitID+WindowKind alone. + Ordinal int + + UsedPercent float64 + // WindowMinutes is nil when Codex did not report a duration for this + // window, distinct from a reported duration of zero (which Codex + // never sends). See docs/agents/storage.md. + WindowMinutes *int + // ResetsAt is unix seconds, or nil when Codex did not report a + // reset time for this window. A window with no known reset time is + // distinct from one that resets at the unix epoch -- flattening the + // two let a null resets_at surface as a bogus "resets in 0m" instead + // of an unknown reset time. + ResetsAt *int64 + + CreditsHas bool + CreditsUnlimited bool + CreditsBalance string + + RateLimitReachedType string + ObservedAt time.Time +} + // accumulateMessageTokenUsage rolls up explicit per-message token // metadata into session totals without inferring presence from raw // numeric values alone. @@ -1665,9 +1725,10 @@ func (s ParsedSession) TokenCoverageContext( // ParseResult pairs a parsed session with its messages. type ParseResult struct { - Session ParsedSession - Messages []ParsedMessage - UsageEvents []ParsedUsageEvent + Session ParsedSession + Messages []ParsedMessage + UsageEvents []ParsedUsageEvent + RateLimitSnapshots []ParsedRateLimitSnapshot // Checkpoint is opaque provider continuation state (a parser // checkpoint) that the sync engine persists after this result's // session rows commit, so later appends can resume without rescanning diff --git a/internal/postgres/rate_limits.go b/internal/postgres/rate_limits.go new file mode 100644 index 0000000000..620c1e2343 --- /dev/null +++ b/internal/postgres/rate_limits.go @@ -0,0 +1,28 @@ +package postgres + +import ( + "context" + + "go.kenn.io/agentsview/internal/db" +) + +// LatestRateLimitSnapshots and RateLimitSnapshotHistory are SQLite-only +// (see docs/agents/storage.md): rate_limit_snapshots is a local +// vendor-data table, not part of the SQLite/PostgreSQL/DuckDB parity +// contract. The PostgreSQL reader returns an empty result rather than an +// error so the Usage page's rate-limits section simply stays hidden when +// PostgreSQL is the active read backend. + +// LatestRateLimitSnapshots is not supported by the PostgreSQL backend. +func (s *Store) LatestRateLimitSnapshots( + _ context.Context, _ db.RateLimitFilter, +) ([]db.RateLimitSnapshot, error) { + return nil, nil +} + +// RateLimitSnapshotHistory is not supported by the PostgreSQL backend. +func (s *Store) RateLimitSnapshotHistory( + _ context.Context, _ db.RateLimitHistoryFilter, +) ([]db.RateLimitSnapshot, error) { + return nil, nil +} diff --git a/internal/server/huma_route_groups.go b/internal/server/huma_route_groups.go index d36da05c45..5b4ffa67c6 100644 --- a/internal/server/huma_route_groups.go +++ b/internal/server/huma_route_groups.go @@ -14,6 +14,7 @@ func (s *Server) registerTypedAPIRoutes() { s.registerRecentEditsRoutes() s.registerTrendsRoutes() s.registerUsageRoutes() + s.registerRateLimitRoutes() s.registerInsightsRoutes() s.registerSearchRoutes() s.registerSecretsRoutes() diff --git a/internal/server/huma_routes_ratelimits.go b/internal/server/huma_routes_ratelimits.go new file mode 100644 index 0000000000..bb509e03af --- /dev/null +++ b/internal/server/huma_routes_ratelimits.go @@ -0,0 +1,93 @@ +package server + +import ( + "context" + + "go.kenn.io/agentsview/internal/service" +) + +func (s *Server) registerRateLimitRoutes() { + group := newRouteGroup(s.api, "/api/v1/rate-limits", "RateLimits") + + s.get(group, "/current", "Get current rate limits", s.humaRateLimitsCurrent) + s.get(group, "/history", "Get rate limit history", s.humaRateLimitsHistory) +} + +// RateLimitFilterInput is the shared vendor/account/machine filter for +// both rate-limits endpoints. The table is SQLite-only (see +// docs/agents/storage.md) and Codex-only today; Agent takes the same +// comma-separated selection as the shared session filters, and a +// selection that omits the resolved vendor (Vendor, or "codex" when +// unset) matches nothing. +type RateLimitFilterInput struct { + Vendor string `query:"vendor" enum:"codex" doc:"Filter by vendor"` + AccountID string `query:"account_id" doc:"Filter by account id; scopes vendors that have accounts and never excludes an account-less vendor's rows (Codex today)"` + Machine string `query:"machine" doc:"Filter by machine (comma-separated)"` + // Agent is accepted for backward compatibility with the original + // Codex-only filter name; it behaves like Vendor when Vendor is + // unset. + Agent string `query:"agent" doc:"Deprecated alias for vendor (comma-separated)"` +} + +type rateLimitsHistoryInput struct { + RateLimitFilterInput + LimitID string `query:"limit_id" doc:"Filter by limit id (e.g. codex)"` + WindowKind string `query:"window" enum:"primary,secondary" doc:"Filter by rate-limit window kind"` + Since string `query:"since" format:"date-time" doc:"Return snapshots observed at or after this RFC3339 timestamp"` + Until string `query:"until" format:"date-time" doc:"Return snapshots observed strictly before this RFC3339 timestamp"` + // MaxPoints bounds the response size for a wide date range: a range + // with more matching observations than this is downsampled to at + // most this many points, keeping the most recently observed row per + // time bucket rather than every observation. + MaxPoints int `query:"max_points" minimum:"1" maximum:"2000" default:"500" doc:"Maximum points returned; a wider range is downsampled to this many, keeping the most recent observation per time bucket"` +} + +func rateLimitFilterRequestFromInput(in RateLimitFilterInput) service.RateLimitFilterRequest { + return service.RateLimitFilterRequest{ + Vendor: in.Vendor, + AccountID: in.AccountID, + Machine: in.Machine, + Agent: in.Agent, + } +} + +func (s *Server) humaRateLimitsCurrent( + ctx context.Context, + in *RateLimitFilterInput, +) (*jsonOutput[[]service.RateLimitWindow], error) { + rows, err := service.RateLimitCurrent(ctx, s.db, rateLimitFilterRequestFromInput(*in)) + if err != nil { + if handled := handleHumaContextError(err); handled != nil { + return nil, handled + } + if handled := handleHumaReadOnly(err); handled != nil { + return nil, handled + } + return nil, internalError("rate limits current error", err) + } + return &jsonOutput[[]service.RateLimitWindow]{Body: rows}, nil +} + +func (s *Server) humaRateLimitsHistory( + ctx context.Context, + in *rateLimitsHistoryInput, +) (*jsonOutput[[]service.RateLimitWindow], error) { + rows, err := service.RateLimitHistory(ctx, s.db, service.RateLimitHistoryRequest{ + RateLimitFilterRequest: rateLimitFilterRequestFromInput(in.RateLimitFilterInput), + LimitID: in.LimitID, + WindowKind: in.WindowKind, + Since: in.Since, + Until: in.Until, + MaxPoints: in.MaxPoints, + }) + if err != nil { + if handled := handleHumaContextError(err); handled != nil { + return nil, handled + } + if handled := handleHumaReadOnly(err); handled != nil { + return nil, handled + } + return nil, internalError("rate limits history error", err) + } + return &jsonOutput[[]service.RateLimitWindow]{Body: rows}, nil +} diff --git a/internal/service/rate_limits.go b/internal/service/rate_limits.go new file mode 100644 index 0000000000..d28fd1be9f --- /dev/null +++ b/internal/service/rate_limits.go @@ -0,0 +1,169 @@ +// ABOUTME: Rate-limit request/response types and the thin translation +// ABOUTME: between db.RateLimitSnapshot rows and the API shape. +package service + +import ( + "context" + "strings" + + "go.kenn.io/agentsview/internal/db" +) + +// RateLimitFilterRequest is the transport-neutral filter shared by the +// current-snapshot and history endpoints. +type RateLimitFilterRequest struct { + // Vendor narrows to one vendor ("codex" today); empty matches every + // vendor the table holds. + Vendor string `json:"vendor,omitempty"` + // AccountID narrows to one account. Reserved for a future + // account-keyed vendor; ignored for Codex rows, which carry no + // account identity. + AccountID string `json:"accountId,omitempty"` + Machine string `json:"machine,omitempty"` + // Agent is the same comma-separated agent selection the shared + // session filters use (see db.RateLimitAgentMatchesVendor). It is + // accepted for backward compatibility with the original Codex-only + // filter name; it behaves like Vendor when Vendor is unset. + Agent string `json:"agent,omitempty"` +} + +// RateLimitHistoryRequest narrows RateLimitFilterRequest to a time range +// and optionally one limit_id/window_kind, for a used_percent time series. +// Since and Until are RFC3339 (or RFC3339Nano) timestamps, parsed and +// normalized to UTC by db.RateLimitSnapshotHistory: Since is an inclusive +// lower bound and Until is an EXCLUSIVE upper bound, so a caller wanting +// a whole local day sends the next day's local midnight as Until rather +// than that day's 23:59:59. +type RateLimitHistoryRequest struct { + RateLimitFilterRequest + LimitID string `json:"limit_id,omitempty"` + WindowKind string `json:"window_kind,omitempty"` + Since string `json:"since,omitempty"` + Until string `json:"until,omitempty"` + // MaxPoints bounds the number of points returned; a range with more + // matching observations than this is downsampled to one per time + // bucket (see db.RateLimitHistoryFilter.MaxPoints). <= 0 uses the + // database layer's default. + MaxPoints int `json:"max_points,omitempty"` +} + +// RateLimitWindow is the API shape for one rate-limit window snapshot: +// which vendor reported it, how much of the window is used, when it +// resets, and (Codex only) the plan type and credit balance reported +// alongside it. +type RateLimitWindow struct { + Vendor string `json:"vendor"` + AccountID string `json:"accountId,omitempty"` + AccountLabel string `json:"accountLabel,omitempty"` + Machine string `json:"machine"` + LimitID string `json:"limitId"` + LimitName string `json:"limitName,omitempty"` + PlanType string `json:"planType,omitempty"` + WindowKind string `json:"windowKind"` + + UsedPercent float64 `json:"usedPercent"` + // WindowMinutes is omitted when Codex did not report this window's + // duration on any observation on record (db.RateLimitSnapshot. + // WindowMinutes == 0, which never occurs for a genuine duration): the + // frontend card falls back to a window-kind label instead of showing + // a placeholder duration. + WindowMinutes *int `json:"windowMinutes,omitempty"` + // ResetsAt is unix seconds, omitted when the vendor did not report a + // reset time for this window (kept nullable rather than flattened + // to 0, so an unknown reset time is never displayed as if the + // window resets at the unix epoch). + ResetsAt *int64 `json:"resetsAt,omitempty"` + + CreditsHas bool `json:"creditsHas"` + CreditsUnlimited bool `json:"creditsUnlimited"` + CreditsBalance string `json:"creditsBalance,omitempty"` + + // ScopeLabel and Details are reserved for a future vendor whose + // rate-limit source reports a scoped or free-form extra shape; both + // are always empty for Codex. + ScopeLabel string `json:"scopeLabel,omitempty"` + Details string `json:"details,omitempty"` + + RateLimitReachedType string `json:"rateLimitReachedType,omitempty"` + ObservedAt string `json:"observedAt"` + SessionID string `json:"sessionId,omitempty"` +} + +func rateLimitWindowFromRow(row db.RateLimitSnapshot) RateLimitWindow { + vendor := row.Vendor + if vendor == "" { + vendor = "codex" + } + var windowMinutes *int + if row.WindowMinutes != 0 { + windowMinutes = &row.WindowMinutes + } + return RateLimitWindow{ + Vendor: vendor, + AccountID: row.AccountID, + AccountLabel: row.AccountLabel, + Machine: row.Machine, + LimitID: row.LimitID, + LimitName: row.LimitName, + PlanType: row.PlanType, + WindowKind: row.WindowKind, + UsedPercent: row.UsedPercent, + WindowMinutes: windowMinutes, + ResetsAt: row.ResetsAt, + CreditsHas: row.CreditsHas, + CreditsUnlimited: row.CreditsUnlimited, + CreditsBalance: row.CreditsBalance, + ScopeLabel: row.ScopeLabel, + Details: row.Details, + RateLimitReachedType: row.RateLimitReachedType, + ObservedAt: row.ObservedAt, + SessionID: row.SessionID, + } +} + +// RateLimitCurrent returns the latest snapshot per (vendor, machine, +// account, limit_id, window_kind) group. +func RateLimitCurrent( + ctx context.Context, store db.Store, req RateLimitFilterRequest, +) ([]RateLimitWindow, error) { + rows, err := store.LatestRateLimitSnapshots(ctx, db.RateLimitFilter{ + Vendor: strings.TrimSpace(req.Vendor), + AccountID: strings.TrimSpace(req.AccountID), + Machine: strings.TrimSpace(req.Machine), + Agent: strings.TrimSpace(req.Agent), + }) + if err != nil { + return nil, err + } + out := make([]RateLimitWindow, len(rows)) + for i, row := range rows { + out[i] = rateLimitWindowFromRow(row) + } + return out, nil +} + +// RateLimitHistory returns a time-ordered series of snapshots for +// charting used_percent over the requested range. +func RateLimitHistory( + ctx context.Context, store db.Store, req RateLimitHistoryRequest, +) ([]RateLimitWindow, error) { + rows, err := store.RateLimitSnapshotHistory(ctx, db.RateLimitHistoryFilter{ + Vendor: strings.TrimSpace(req.Vendor), + AccountID: strings.TrimSpace(req.AccountID), + Machine: strings.TrimSpace(req.Machine), + Agent: strings.TrimSpace(req.Agent), + LimitID: strings.TrimSpace(req.LimitID), + WindowKind: strings.TrimSpace(req.WindowKind), + Since: strings.TrimSpace(req.Since), + Until: strings.TrimSpace(req.Until), + MaxPoints: req.MaxPoints, + }) + if err != nil { + return nil, err + } + out := make([]RateLimitWindow, len(rows)) + for i, row := range rows { + out[i] = rateLimitWindowFromRow(row) + } + return out, nil +} diff --git a/internal/service/rate_limits_test.go b/internal/service/rate_limits_test.go new file mode 100644 index 0000000000..a60bfc0bd6 --- /dev/null +++ b/internal/service/rate_limits_test.go @@ -0,0 +1,58 @@ +package service_test + +import ( + "context" + "encoding/json" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "go.kenn.io/agentsview/internal/db" + "go.kenn.io/agentsview/internal/dbtest" + "go.kenn.io/agentsview/internal/service" +) + +// TestRateLimitCurrent_ResetsAtNullability covers the service layer's +// happy path (a snapshot reaches the API shape) and the nullable +// resets_at invariant: a window with no known reset time must reach the +// wire as an omitted field, not the number 0, which the Usage page would +// otherwise render as a bogus "resets in 0m". +func TestRateLimitCurrent_ResetsAtNullability(t *testing.T) { + known := int64(1789435448) + for _, tc := range []struct { + name string + resetsAt *int64 + wantPresent bool + }{ + {"unknown reset time is omitted, not serialized as 0", nil, false}, + {"a known reset time is serialized", &known, true}, + } { + t.Run(tc.name, func(t *testing.T) { + d := dbtest.OpenTestDB(t) + require.NoError(t, d.UpsertSession(db.Session{ID: "codex:sess-1", Agent: "codex"})) + require.NoError(t, d.InsertRateLimitSnapshots([]db.RateLimitSnapshot{{ + SessionID: "codex:sess-1", Machine: "laptop", LimitID: "codex", + PlanType: "pro", WindowKind: "primary", + ObservedAt: "2026-09-09T10:00:00Z", ResetsAt: tc.resetsAt, + }})) + + rows, err := service.RateLimitCurrent( + context.Background(), d, service.RateLimitFilterRequest{}, + ) + require.NoError(t, err) + require.Len(t, rows, 1) + assert.Equal(t, tc.resetsAt == nil, rows[0].ResetsAt == nil) + + raw, err := json.Marshal(rows[0]) + require.NoError(t, err) + var decoded map[string]any + require.NoError(t, json.Unmarshal(raw, &decoded)) + value, present := decoded["resetsAt"] + assert.Equal(t, tc.wantPresent, present) + if tc.wantPresent { + assert.EqualValues(t, *tc.resetsAt, value) + } + }) + } +} diff --git a/internal/sync/codex_staging.go b/internal/sync/codex_staging.go index ebee19d8ba..5ef126ef72 100644 --- a/internal/sync/codex_staging.go +++ b/internal/sync/codex_staging.go @@ -374,6 +374,7 @@ func stagedCodexParseOutcome( Result: parser.ParseResult{ Session: *sess, Messages: msgs, + RateLimitSnapshots: sess.RateLimitSnapshots, Checkpoint: cursor, CheckpointHashState: hashState, CheckpointAnchorDigest: anchorDigest, diff --git a/internal/sync/engine.go b/internal/sync/engine.go index 62a322ef58..3d26e8fa60 100644 --- a/internal/sync/engine.go +++ b/internal/sync/engine.go @@ -3522,6 +3522,43 @@ func (e *Engine) resyncBuildLocked( } stats.OrphanedCopied = len(orphaned) copiedSessionIDs = append(copiedSessionIDs, orphaned...) + + // Copy rate-limit snapshots so previously observed Codex rate-limit + // windows survive the swap, the same way model pricing does above. + // This must run after orphaned sessions are restored, not before: + // rate_limit_snapshots.session_id is a foreign key, and a snapshot + // belonging to an orphaned or trashed session (its source file is + // gone, so it is absent from newDB until the copies above run) would + // otherwise violate that constraint -- INSERT OR IGNORE does not + // suppress a foreign-key violation the way it suppresses a duplicate, + // so copying too early silently loses every row in the table, not + // just that session's. Unlike model pricing, this history cannot be + // reconstructed once lost except by a full reparse of the source + // rollout, which neither an orphaned nor a trashed session still has, + // so a failure here aborts the swap instead of merely warning. + // copiedSessionIDs -- the union of CopyTrashedDataFrom's and + // CopyOrphanedDataFromExcluding's ids, both restored-without-reparse + // -- is passed through so the copy can tell those sessions apart from + // ones the fresh sync itself rebuilt, and only resurrect old rows for + // the former (see CopyRateLimitSnapshotsFrom's doc comment). Passing + // only orphaned here would silently drop a trashed session's + // rate-limit history on every resync. + if err := newDB.CopyRateLimitSnapshotsFrom(origPath, copiedSessionIDs); err != nil { + log.Printf("resync: copy rate limit snapshots: %v", err) + stats.Aborted = true + stats.Warnings = append(stats.Warnings, + "rate limit snapshots copy failed, aborting swap: "+ + err.Error(), + ) + newDB.Close() + removeTempDB(tempPath) + restoreSkipCache() + e.mu.Lock() + e.lastSyncStats = stats + e.mu.Unlock() + return stats, err + } + deferredCwdUpdated, err := e.applyDeferredSourceCwd( newDB, deferredSourceCwd, ) @@ -9318,10 +9355,11 @@ func (e *Engine) syncProviderDBBacked( pending := make([]pendingWrite, 0, len(outcome.Results)) for _, result := range outcome.Results { pending = append(pending, pendingWrite{ - sess: result.Result.Session, - msgs: result.Result.Messages, - usageEvents: result.Result.UsageEvents, - needsRetry: !complete, + sess: result.Result.Session, + msgs: result.Result.Messages, + usageEvents: result.Result.UsageEvents, + rateLimitSnapshots: result.Result.RateLimitSnapshots, + needsRetry: !complete, }) } if len(pending) > 0 && !flush(pending) { @@ -10367,6 +10405,7 @@ func (e *Engine) collectAndBatchWithOptions( sess: pr.Session, msgs: pr.Messages, usageEvents: pr.UsageEvents, + rateLimitSnapshots: pr.RateLimitSnapshots, sourceBytes: r.sourceBytes, checkpoint: pr.Checkpoint, checkpointHashState: pr.CheckpointHashState, @@ -10742,6 +10781,7 @@ type incrementalUpdate struct { links []parser.ClaudeSubagentLink toolCallUpdates []parser.ParsedToolCallUpdate messageUsageUpdates []parser.ParsedMessageTokenUsageUpdate + rateLimitSnapshots []parser.ParsedRateLimitSnapshot // checkpoint is the machine-local parser checkpoint to persist in the // same transaction as this incremental delta. nil keeps the existing // checkpoint (or leaves none). @@ -14848,7 +14888,7 @@ func (e *Engine) tryProviderIncrementalAppend( parseFn := func( _ string, inc *db.IncrementalInfo, - ) ([]parser.ParsedMessage, []parser.ClaudeSubagentLink, []parser.ParsedToolCallUpdate, []parser.ParsedMessageTokenUsageUpdate, time.Time, int64, *string, []byte, error) { + ) ([]parser.ParsedMessage, []parser.ClaudeSubagentLink, []parser.ParsedToolCallUpdate, []parser.ParsedMessageTokenUsageUpdate, []parser.ParsedRateLimitSnapshot, time.Time, int64, *string, []byte, error) { // The Claude parser needs the stored tail's provider message id // so its queued-command masking fallback fires only for a real // same-message.id continuation; without it, every routine queued @@ -14878,14 +14918,14 @@ func (e *Engine) tryProviderIncrementalAppend( }, ) if perr != nil { - return nil, nil, nil, nil, time.Time{}, 0, nil, nil, perr + return nil, nil, nil, nil, nil, time.Time{}, 0, nil, nil, perr } switch status { case parser.IncrementalNeedsFullParse: if outcome.ForceReplace { // Signal the shared helper to fall back to a // full parse that replaces stored messages. - return nil, nil, nil, nil, time.Time{}, 0, nil, nil, + return nil, nil, nil, nil, nil, time.Time{}, 0, nil, nil, parser.ErrIncrementalNeedsFullParse } // A plain full-parse fallback without a replace request. @@ -14893,9 +14933,9 @@ func (e *Engine) tryProviderIncrementalAppend( // fallbacks (a DAG fork can drop or re-branch stored // rows), so this branch serves providers that only need // an append-preserving full parse. - return nil, nil, nil, nil, time.Time{}, 0, nil, nil, parser.ErrDAGDetected + return nil, nil, nil, nil, nil, time.Time{}, 0, nil, nil, parser.ErrDAGDetected case parser.IncrementalNoNewData: - return nil, nil, nil, nil, time.Time{}, 0, nil, nil, nil + return nil, nil, nil, nil, nil, time.Time{}, 0, nil, nil, nil default: var terminationStatus *string if outcome.TerminationStatus != nil { @@ -14905,6 +14945,7 @@ func (e *Engine) tryProviderIncrementalAppend( return outcome.Messages, outcome.SubagentLinks, outcome.ToolCallUpdates, outcome.MessageTokenUsageUpdates, + outcome.RateLimitSnapshots, outcome.EndedAt, outcome.ConsumedBytes, terminationStatus, outcome.NextCursor, nil } @@ -14924,7 +14965,7 @@ func (e *Engine) tryProviderIncrementalAppend( // only complete, valid JSON lines so it can be used as a safe resume offset. type incrementalParseFunc func( path string, inc *db.IncrementalInfo, -) ([]parser.ParsedMessage, []parser.ClaudeSubagentLink, []parser.ParsedToolCallUpdate, []parser.ParsedMessageTokenUsageUpdate, time.Time, int64, *string, []byte, error) +) ([]parser.ParsedMessage, []parser.ClaudeSubagentLink, []parser.ParsedToolCallUpdate, []parser.ParsedMessageTokenUsageUpdate, []parser.ParsedRateLimitSnapshot, time.Time, int64, *string, []byte, error) // tryIncrementalJSONL attempts an incremental parse of an // append-only JSONL file by reading only bytes appended since @@ -15058,7 +15099,7 @@ func (e *Engine) tryIncrementalJSONL( return processResult{err: leaseErr}, true } - newMsgs, links, toolCallUpdates, messageUsageUpdates, endedAt, consumed, terminationStatus, cursor, err := parseFn( + newMsgs, links, toolCallUpdates, messageUsageUpdates, rateLimitSnapshots, endedAt, consumed, terminationStatus, cursor, err := parseFn( file.Path, inc, ) if err != nil { @@ -15233,6 +15274,7 @@ func (e *Engine) tryIncrementalJSONL( links: links, toolCallUpdates: toolCallUpdates, messageUsageUpdates: messageUsageUpdates, + rateLimitSnapshots: rateLimitSnapshots, checkpoint: nextCheckpoint, checkpointBlobs: nextCheckpointBlobs, endedAt: endedAt, @@ -15339,6 +15381,7 @@ func (e *Engine) tryIncrementalJSONL( links: links, toolCallUpdates: toolCallUpdates, messageUsageUpdates: messageUsageUpdates, + rateLimitSnapshots: rateLimitSnapshots, checkpoint: nextCheckpoint, checkpointBlobs: nextCheckpointBlobs, endedAt: endedAt, @@ -16189,9 +16232,10 @@ func (e *Engine) recomputeSignalsFromDBWithHook( } type pendingWrite struct { - sess parser.ParsedSession - msgs []parser.ParsedMessage - usageEvents []parser.ParsedUsageEvent + sess parser.ParsedSession + msgs []parser.ParsedMessage + usageEvents []parser.ParsedUsageEvent + rateLimitSnapshots []parser.ParsedRateLimitSnapshot // sourceBytes is the physical source size carried from the parse result; // collectAndBatch uses it to flush batches on estimated bytes as well as // session count. @@ -16912,6 +16956,21 @@ func (e *Engine) writeBatchWithOutcomeContext( outcome.failedSessions++ continue } + if err := e.writeRateLimitSnapshots( + replaceMessages, s.ID, + rateLimitSnapshotsForWrite(s.ID, s.Machine, pw.rateLimitSnapshots), + ); err != nil { + if ctx.Err() != nil { + return outcome + } + log.Printf( + "write rate limit snapshots for %s: %v", + s.ID, err, + ) + e.markStaleFailedMemberWrite(pw) + outcome.failedSessions++ + continue + } if ctx.Err() != nil { return outcome } @@ -18054,6 +18113,20 @@ func (e *Engine) writeBatchBulkWithOutcomeContext( outcome.failedSessions++ continue } + // A staged full parse always force-replaces (see the comment + // above), so its rate-limit snapshots replace the session's + // prior rows too. + if err := e.writeRateLimitSnapshots( + true, s.ID, + rateLimitSnapshotsForWrite(s.ID, s.Machine, pw.rateLimitSnapshots), + ); err != nil { + log.Printf( + "write rate limit snapshots for %s: %v", s.ID, err, + ) + e.markStaleFailedMemberWrite(pw) + outcome.failedSessions++ + continue + } if err := e.db.SetSessionDataVersion( s.ID, dataVersionForWrite(pw), ); err != nil { @@ -18119,9 +18192,10 @@ func (e *Engine) writeBatchBulkWithOutcomeContext( identityObservation, hasIdentityObservation := e.projectIdentityObservationForWrite(pw, s) writes = append(writes, db.SessionBatchWrite{ - Session: s, - Messages: msgs, - UsageEvents: usageEvents, + Session: s, + Messages: msgs, + UsageEvents: usageEvents, + RateLimitSnapshots: rateLimitSnapshotsForWrite(s.ID, s.Machine, pw.rateLimitSnapshots), IdentityObservation: identityObservationOrZero( identityObservation, hasIdentityObservation, ), @@ -18890,6 +18964,7 @@ func (e *Engine) writeIncremental( SubagentLinks: subagentLinks, ToolCallResultUpdates: toolCallResultUpdates, MessageTokenUsageUpdates: messageUsageUpdates, + RateLimitSnapshots: rateLimitSnapshotsForWrite(inc.sessionID, inc.machine, inc.rateLimitSnapshots), Checkpoint: inc.checkpoint, CheckpointBlobs: inc.checkpointBlobs, BlockedResultCategories: e.blockedResultCategories, @@ -19099,6 +19174,19 @@ func (e *Engine) writeSessionFullWithResolver( ) return err } + // writeSessionFullWithResolver always does a full delete+reinsert of + // messages (see its doc comment), so its rate-limit snapshots + // replace the session's prior rows too. + if err := e.writeRateLimitSnapshots( + true, s.ID, + rateLimitSnapshotsForWrite(s.ID, s.Machine, pw.rateLimitSnapshots), + ); err != nil { + log.Printf( + "write rate limit snapshots for %s: %v", + s.ID, err, + ) + return err + } // See writeBatch for why data_version is bumped here // rather than inside UpsertSession. @@ -19795,6 +19883,86 @@ func (e *Engine) usageEventsForWriteContext( return out, nil } +// writeRateLimitSnapshots inserts snapshots for sessionID, deleting the +// session's existing rate_limit_snapshots rows first when replaceMessages +// is true -- the same full-replacement condition used for the session's +// own messages at this call site, so an authoritative reparse superseding +// a fallback marked parser.DataVersionNeedsRetry cannot leave that +// fallback's rows behind (see docs/agents/storage.md). A normal +// incremental parse (replaceMessages false, appending only the newly +// parsed tail) must not delete. +func (e *Engine) writeRateLimitSnapshots( + replaceMessages bool, sessionID string, snapshots []db.RateLimitSnapshot, +) error { + if replaceMessages { + return e.db.InsertRateLimitSnapshotsReplacingSession( + sessionID, snapshots, + ) + } + return e.db.InsertRateLimitSnapshots(snapshots) +} + +// rateLimitSnapshotsForWrite converts parser-emitted Codex rate-limit +// snapshots into db rows for InsertRateLimitSnapshots, always +// stamping them with sessionID and machine -- the session id and machine +// this write is actually committing under -- rather than the parser- +// native snap.SessionID/snap.Machine. The parser fills those fields from +// its own raw scan before the engine can correct them: remote/S3 sync +// namespaces the session id with a machine prefix (see +// applyRemoteRewrites and applyIDPrefixToParsedResult), and +// normalizePendingWriteMachines can overwrite pw.sess.Machine with the +// archive's immutable stored machine for an existing session. Preferring +// the parser's own values here would attach a prefixed sync's rate-limit +// rows to a session id that was never written (failing the session_id +// foreign key), or store them under a machine that no longer matches the +// session they belong to, splitting one account's history across two +// machine labels. sessionID and machine are always the final, corrected +// values at every call site. Unlike usage events, rate-limit rows are +// normally inserted with INSERT OR IGNORE against a unique dedup_key +// rather than replaced per session, so calling this on every incremental +// write cannot duplicate rows; the dedup key is (re)computed from the +// corrected SessionID by InsertRateLimitSnapshotsContext. A write that +// replaces a session's messages wholesale instead deletes the session's +// prior rows first (see writeRateLimitSnapshots and +// InsertRateLimitSnapshotsReplacingSession) so a superseded fallback +// parse cannot leave stale rows behind. +func rateLimitSnapshotsForWrite( + sessionID, machine string, snapshots []parser.ParsedRateLimitSnapshot, +) []db.RateLimitSnapshot { + if len(snapshots) == 0 { + return nil + } + out := make([]db.RateLimitSnapshot, len(snapshots)) + for i, snap := range snapshots { + // windowMinutes: 0 doubles as "unknown" from the database onward + // (see db.RateLimitSnapshot.WindowMinutes), since Codex never + // reports a genuine zero-minute window; a nil snap.WindowMinutes + // collapses to that same sentinel here. + windowMinutes := 0 + if snap.WindowMinutes != nil { + windowMinutes = *snap.WindowMinutes + } + out[i] = db.RateLimitSnapshot{ + SessionID: sessionID, + Machine: machine, + LimitID: snap.LimitID, + LimitName: snap.LimitName, + PlanType: snap.PlanType, + WindowKind: snap.WindowKind, + UsedPercent: snap.UsedPercent, + WindowMinutes: windowMinutes, + ResetsAt: snap.ResetsAt, + CreditsHas: snap.CreditsHas, + CreditsUnlimited: snap.CreditsUnlimited, + CreditsBalance: snap.CreditsBalance, + RateLimitReachedType: snap.RateLimitReachedType, + ObservedAt: snap.ObservedAt.UTC().Format(time.RFC3339Nano), + Ordinal: snap.Ordinal, + } + } + return out +} + // postFilterCounts returns the total and user message counts // from a filtered message slice. System-injected messages // (e.g. Zencoder compaction, continuation notices) are excluded @@ -20891,6 +21059,7 @@ func (e *Engine) processAndWriteSessionFile( sess: pr.Session, msgs: pr.Messages, usageEvents: pr.UsageEvents, + rateLimitSnapshots: pr.RateLimitSnapshots, checkpoint: pr.Checkpoint, checkpointHashState: pr.CheckpointHashState, checkpointAnchorDigest: pr.CheckpointAnchorDigest, diff --git a/internal/sync/engine_integration_test.go b/internal/sync/engine_integration_test.go index a1e38f1da5..5e69879623 100644 --- a/internal/sync/engine_integration_test.go +++ b/internal/sync/engine_integration_test.go @@ -6341,6 +6341,64 @@ func TestCodexRequiredReparseWithoutIndexPreservesStoredTitle(t *testing.T) { } } +// TestUpgradingArchiveBackfillsCodexRateLimitSnapshots pins the +// data-version-backfill invariant: a session stamped at a data_version +// older than the one that added rate_limits extraction must get a full +// reparse (not skip via the incremental fast path) once +// db.CurrentDataVersion() moves past it, backfilling its history from the +// unchanged source file. +func TestUpgradingArchiveBackfillsCodexRateLimitSnapshots(t *testing.T) { + root := t.TempDir() + codexDir := filepath.Join(root, "sessions") + require.NoError(t, os.MkdirAll(codexDir, 0o755)) + env := setupTestEnv(t, WithCodexDirs([]string{codexDir})) + + uuid := "019eb791-cf7d-75c1-8439-9ed74c1229f5" + sessionID := "codex:" + uuid + content := testjsonl.NewSessionBuilder(). + AddCodexMeta(tsEarly, uuid, "/repo", "user"). + AddCodexMessage(tsEarlyS1, "user", "hello"). + AddCodexMessage(tsEarlyS5, "assistant", "hi"). + AddRaw(testjsonl.CodexTokenCountWithRateLimitsJSON( + tsEarlyS5, 10000, 500, 6000, "codex", "pro", + &testjsonl.CodexRateLimitWindow{UsedPercent: 33, WindowMinutes: 10080, ResetsAt: 1789435448}, + nil, "100.0", + )). + String() + env.writeCodexSession(t, filepath.Join("2026", "06", "11"), + "rollout-2026-06-11T12-44-06-"+uuid+".jsonl", content) + + require.Equal(t, 1, env.engine.SyncAll(context.Background(), nil).Synced) + require.Equal(t, db.CurrentDataVersion(), env.db.GetSessionDataVersion(sessionID)) + + // Simulate the pre-upgrade archive state this binary predates: an + // older data_version and no rate-limit rows, the file on disk + // untouched. The literal pre-rate-limits version (107), not + // CurrentDataVersion()-1, so the test still proves the bump matters + // if it were reverted. + const preRateLimitsDataVersion = 107 + require.Less(t, preRateLimitsDataVersion, db.CurrentDataVersion()) + require.NoError(t, env.db.SetSessionDataVersion(sessionID, preRateLimitsDataVersion)) + raw, err := sql.Open("sqlite3", env.db.Path()) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, raw.Close()) }) + _, err = raw.Exec(`DELETE FROM rate_limit_snapshots WHERE session_id = ?`, sessionID) + require.NoError(t, err) + + require.Equal(t, 1, env.engine.SyncAll(context.Background(), nil).Synced, + "a stale data version must force a full reparse of the unchanged file") + + sess, err := env.db.GetSessionFull(context.Background(), sessionID) + require.NoError(t, err) + require.NotNil(t, sess) + assert.Equal(t, db.CurrentDataVersion(), sess.DataVersion) + + rows, err := env.db.RateLimitSnapshotHistory(context.Background(), db.RateLimitHistoryFilter{}) + require.NoError(t, err) + require.Len(t, rows, 1, "upgrading the archive must backfill rate-limit history from the unchanged file") + assert.InDelta(t, 33.0, rows[0].UsedPercent, 0.001) +} + func TestCodexExplicitBlankIndexTitleClearsStoredTitle(t *testing.T) { root := t.TempDir() codexDir := filepath.Join(root, "sessions") diff --git a/internal/sync/engine_staged_contract_test.go b/internal/sync/engine_staged_contract_test.go index 84d677549e..e894d6d466 100644 --- a/internal/sync/engine_staged_contract_test.go +++ b/internal/sync/engine_staged_contract_test.go @@ -1,7 +1,6 @@ package sync import ( - "go.kenn.io/agentsview/internal/testjsonl" "os" "path/filepath" "testing" @@ -9,6 +8,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "go.kenn.io/agentsview/internal/parser" + "go.kenn.io/agentsview/internal/testjsonl" ) func TestStagedImportHonorsDisabledSignalRecomputation(t *testing.T) { diff --git a/internal/sync/engine_test.go b/internal/sync/engine_test.go index 086ce1f36e..73d07a5c10 100644 --- a/internal/sync/engine_test.go +++ b/internal/sync/engine_test.go @@ -6880,11 +6880,11 @@ func TestProjectIdentityIncrementalStatePreservesExplicitSourceProject( func( _ string, inc *db.IncrementalInfo, - ) ([]parser.ParsedMessage, []parser.ClaudeSubagentLink, []parser.ParsedToolCallUpdate, []parser.ParsedMessageTokenUsageUpdate, time.Time, int64, *string, []byte, error) { + ) ([]parser.ParsedMessage, []parser.ClaudeSubagentLink, []parser.ParsedToolCallUpdate, []parser.ParsedMessageTokenUsageUpdate, []parser.ParsedRateLimitSnapshot, time.Time, int64, *string, []byte, error) { return []parser.ParsedMessage{{ Role: parser.RoleAssistant, Content: "appended", Ordinal: inc.NextOrdinal, - }}, nil, nil, nil, appendedInfo.ModTime(), int64(len(appended)), nil, nil, nil + }}, nil, nil, nil, nil, appendedInfo.ModTime(), int64(len(appended)), nil, nil, nil }, nil, "", nil, ) @@ -6982,9 +6982,9 @@ func TestProjectIdentityLegacyMappedSnapshotReparsesBeforeIncrementalAppend( func( _ string, _ *db.IncrementalInfo, - ) ([]parser.ParsedMessage, []parser.ClaudeSubagentLink, []parser.ParsedToolCallUpdate, []parser.ParsedMessageTokenUsageUpdate, time.Time, int64, *string, []byte, error) { + ) ([]parser.ParsedMessage, []parser.ClaudeSubagentLink, []parser.ParsedToolCallUpdate, []parser.ParsedMessageTokenUsageUpdate, []parser.ParsedRateLimitSnapshot, time.Time, int64, *string, []byte, error) { parseCalled = true - return nil, nil, nil, nil, time.Time{}, 0, nil, nil, nil + return nil, nil, nil, nil, nil, time.Time{}, 0, nil, nil, nil }, nil, "", nil, ) diff --git a/internal/sync/parsediff.go b/internal/sync/parsediff.go index 939496bf22..b5385bffd3 100644 --- a/internal/sync/parsediff.go +++ b/internal/sync/parsediff.go @@ -716,6 +716,7 @@ func (e *Engine) parseDiffCollectFile( sess: pr.Session, msgs: pr.Messages, usageEvents: pr.UsageEvents, + rateLimitSnapshots: pr.RateLimitSnapshots, needsRetry: job.needsRetryForSession(pr.Session.ID), sourceCwdResolution: job.sourceCwdResolution, sourceCwdStored: job.sourceCwdStored, diff --git a/internal/sync/rate_limit_snapshot_write_test.go b/internal/sync/rate_limit_snapshot_write_test.go new file mode 100644 index 0000000000..ab5bead298 --- /dev/null +++ b/internal/sync/rate_limit_snapshot_write_test.go @@ -0,0 +1,240 @@ +package sync + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.kenn.io/agentsview/internal/db" + "go.kenn.io/agentsview/internal/parser" + "go.kenn.io/agentsview/internal/testjsonl" +) + +// codexRLTranscript builds a single Codex rollout with one rate-limit +// observation, shared by both tests in this file. +func codexRLTranscript(uuid, dir string, usedPercent float64) string { + return testjsonl.JoinJSONL( + testjsonl.CodexSessionMetaJSON(uuid, dir, "codex_cli_rs", "2026-06-11T12:44:06Z"), + testjsonl.CodexMsgJSON("user", "hello", "2026-06-11T12:44:07Z"), + testjsonl.CodexMsgJSON("assistant", "hi", "2026-06-11T12:44:08Z"), + testjsonl.CodexTokenCountWithRateLimitsJSON( + "2026-06-11T12:44:09Z", 10000, 500, 6000, "codex", "pro", + &testjsonl.CodexRateLimitWindow{UsedPercent: usedPercent, WindowMinutes: 10080, ResetsAt: 1789435448}, + nil, "100.0", + ), + ) +} + +// TestCodexEngineRateLimitSnapshotIDPrefix pins the prefixed-session-id +// invariant: the parser stamps each snapshot with its own raw session id, +// but a remote sync's applyRemoteRewrites renames the session itself, so +// rateLimitSnapshotsForWrite must attach the snapshot to the final, +// prefixed id. Covers both the collecting and staged parse paths. +func TestCodexEngineRateLimitSnapshotIDPrefix(t *testing.T) { + const uuid = "019eb791-cf7d-75c1-8439-9ed74c122b20" + transcript := codexRLTranscript(uuid, "/workspace/project-a", 42) + const wantSessionID = "remote-host~codex:" + uuid + + for _, stagedMin := range []int64{0, 1} { + database := openTestDB(t) + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentCodex: {writeCodexTranscriptRoot(t, uuid, transcript)}, + }, + Machine: "remote", IDPrefix: "remote-host~", StagedCodexParseMinBytes: stagedMin, + }) + t.Cleanup(engine.Close) + stats := engine.SyncAll(t.Context(), nil) + require.Zero(t, stats.Failed) + require.Equal(t, 1, stats.Synced) + + rows, err := database.RateLimitSnapshotHistory(t.Context(), db.RateLimitHistoryFilter{Machine: "remote"}) + require.NoError(t, err) + require.Len(t, rows, 1) + assert.Equal(t, wantSessionID, rows[0].SessionID, + "snapshot must carry the final prefixed session id, not the parser's native one") + } +} + +// writeCodexRolloutInto writes a single Codex rollout file for uuid under +// root, so a test can later remove it to simulate an orphaned session. +func writeCodexRolloutInto(t *testing.T, root, uuid, transcript string) string { + t.Helper() + day := filepath.Join(root, "2026", "06", "11") + require.NoError(t, os.MkdirAll(day, 0o755)) + path := filepath.Join(day, "rollout-2026-06-11T12-44-06-"+uuid+".jsonl") + require.NoError(t, os.WriteFile(path, []byte(transcript), 0o644)) + return path +} + +// TestResyncAllPreservesOrphanedSessionRateLimitSnapshots pins ResyncAll's +// copy ordering: CopyRateLimitSnapshotsFrom must run after +// CopyOrphanedDataFromExcluding, or an orphaned session's snapshot would +// trip the table's session_id foreign key and abort the whole copy. +func TestResyncAllPreservesOrphanedSessionRateLimitSnapshots(t *testing.T) { + const uuid = "019eb791-cf7d-75c1-8439-9ed74c122b21" + sessionID := "codex:" + uuid + root := t.TempDir() + rolloutPath := writeCodexRolloutInto(t, root, uuid, codexRLTranscript(uuid, "/repo", 55)) + + // A second, still-present session, so removing the first below does + // not trip ResyncAll's empty-discovery guard instead. + const keptUUID = "019eb791-cf7d-75c1-8439-9ed74c122b22" + writeCodexRolloutInto(t, root, keptUUID, testjsonl.JoinJSONL( + testjsonl.CodexSessionMetaJSON(keptUUID, "/repo", "codex_cli_rs", "2026-06-11T13:00:00Z"), + testjsonl.CodexMsgJSON("user", "hello", "2026-06-11T13:00:01Z"), + testjsonl.CodexMsgJSON("assistant", "hi", "2026-06-11T13:00:02Z"), + )) + + database := openTestDB(t) + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{parser.AgentCodex: {root}}, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 2, engine.SyncAll(t.Context(), nil).Synced) + + require.NoError(t, os.Remove(rolloutPath), "remove orphan source") + stats := engine.ResyncAll(t.Context(), nil) + require.False(t, stats.Aborted, "ResyncAll aborted: %+v", stats) + sess, err := database.GetSessionFull(t.Context(), sessionID) + require.NoError(t, err) + require.NotNil(t, sess, "the orphaned session itself must survive the resync") + + rows, err := database.RateLimitSnapshotHistory(t.Context(), db.RateLimitHistoryFilter{}) + require.NoError(t, err) + require.Len(t, rows, 1, "the orphaned session's rate-limit snapshot must survive the resync") + assert.Equal(t, sessionID, rows[0].SessionID) +} + +// TestResyncAllPreservesTrashedSessionRateLimitSnapshots: CopyRateLimitSnapshotsFrom must get copiedSessionIDs (trashed + orphaned), not orphaned alone. +func TestResyncAllPreservesTrashedSessionRateLimitSnapshots(t *testing.T) { + const uuid = "019eb791-cf7d-75c1-8439-9ed74c122b25" + sessionID := "codex:" + uuid + root := t.TempDir() + writeCodexRolloutInto(t, root, uuid, codexRLTranscript(uuid, "/repo", 65)) + database := openTestDB(t) + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{parser.AgentCodex: {root}}, Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 1, engine.SyncAll(t.Context(), nil).Synced) + require.NoError(t, database.SoftDeleteSession(sessionID), "trash the session") + stats := engine.ResyncAll(t.Context(), nil) + require.False(t, stats.Aborted, "ResyncAll aborted: %+v", stats) + rows, err := database.RateLimitSnapshotHistory(t.Context(), db.RateLimitHistoryFilter{}) + require.NoError(t, err) + require.Len(t, rows, 1, "the trashed session's rate-limit snapshot must survive the resync") +} + +// TestCodexEngineRateLimitSnapshotSameTimestampOrdinal pins the parser's +// per-event ordinal (see parser.ParsedRateLimitSnapshot.Ordinal): two +// token_count events landing on the same observed_at second must both +// persist instead of the second colliding with, and being dropped +// against, the first's dedup_key. Covers both the collecting and staged +// parse paths, which share the same builder and so must assign the same +// ordinals. +func TestCodexEngineRateLimitSnapshotSameTimestampOrdinal(t *testing.T) { + const uuid = "019eb791-cf7d-75c1-8439-9ed74c122b23" + const at = "2026-06-11T12:44:09Z" + transcript := testjsonl.JoinJSONL( + testjsonl.CodexSessionMetaJSON(uuid, "/workspace/project-a", "codex_cli_rs", "2026-06-11T12:44:06Z"), + testjsonl.CodexMsgJSON("user", "hello", "2026-06-11T12:44:07Z"), + testjsonl.CodexMsgJSON("assistant", "hi", "2026-06-11T12:44:08Z"), + testjsonl.CodexTokenCountWithRateLimitsJSON( + at, 10000, 500, 6000, "codex", "pro", + &testjsonl.CodexRateLimitWindow{UsedPercent: 40, WindowMinutes: 10080, ResetsAt: 1789435448}, + nil, "100.0", + ), + testjsonl.CodexTokenCountWithRateLimitsJSON( + at, 10200, 520, 6100, "codex", "pro", + &testjsonl.CodexRateLimitWindow{UsedPercent: 60, WindowMinutes: 10080, ResetsAt: 1789435448}, + nil, "100.0", + ), + ) + + for _, stagedMin := range []int64{0, 1} { + database := openTestDB(t) + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentCodex: {writeCodexTranscriptRoot(t, uuid, transcript)}, + }, + Machine: "local", StagedCodexParseMinBytes: stagedMin, + }) + t.Cleanup(engine.Close) + stats := engine.SyncAll(t.Context(), nil) + require.Zero(t, stats.Failed) + require.Equal(t, 1, stats.Synced) + + rows, err := database.RateLimitSnapshotHistory(t.Context(), db.RateLimitHistoryFilter{Machine: "local"}) + require.NoError(t, err) + require.Len(t, rows, 2, "two token_count events sharing a timestamp must both persist") + } +} + +// TestResyncAllRateLimitSnapshotReparseAddsZeroRows pins ordinal +// reparse-stability across an incremental append: the second +// token_count event's ordinal, assigned during an incremental tail +// parse from a cached (or reseeded) cursor, must match what a later +// full reparse of the whole file assigns it. ResyncAll both reparses +// every still-present rollout from scratch and copies the old archive's +// rows via CopyRateLimitSnapshotsFrom, so a drifted ordinal would +// duplicate the second event's row under a new dedup_key instead of +// converging on the row already there. +func TestResyncAllRateLimitSnapshotReparseAddsZeroRows(t *testing.T) { + const uuid = "019eb791-cf7d-75c1-8439-9ed74c122b25" + sessionID := "codex:" + uuid + root := t.TempDir() + path := writeCodexRolloutInto(t, root, uuid, codexRLTranscript(uuid, "/repo", 42)) + + database := openTestDB(t) + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{parser.AgentCodex: {root}}, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 1, engine.SyncAll(t.Context(), nil).Synced) + + rows, err := database.RateLimitSnapshotHistory(t.Context(), db.RateLimitHistoryFilter{}) + require.NoError(t, err) + require.Len(t, rows, 1) + + // Append a second turn (its own token_count event) and sync it + // incrementally (not via a fresh full parse), so the new event's + // ordinal comes from the builder's cached/reseeded cursor rather + // than a from-scratch count. The appended assistant message gives + // the token_count event's usage somewhere to attach, so the + // incremental parser does not itself fall back to a full reparse. + f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644) + require.NoError(t, err) + _, err = f.WriteString(testjsonl.JoinJSONL( + testjsonl.CodexMsgJSON("user", "again", "2026-06-11T12:44:10Z"), + testjsonl.CodexMsgJSON("assistant", "sure", "2026-06-11T12:44:11Z"), + testjsonl.CodexTokenCountWithRateLimitsJSON( + "2026-06-11T12:44:12Z", 10200, 520, 6100, "codex", "pro", + &testjsonl.CodexRateLimitWindow{UsedPercent: 44, WindowMinutes: 10080, ResetsAt: 1789435448}, + nil, "100.0", + ), + ) + "\n") + require.NoError(t, f.Close()) + require.NoError(t, err) + engine.SyncPaths([]string{path}) + + sess, err := database.GetSessionFull(t.Context(), sessionID) + require.NoError(t, err) + require.NotNil(t, sess) + require.True(t, sess.LastWriteIncremental, "the append must take the incremental parse path, not a full reparse") + + rows, err = database.RateLimitSnapshotHistory(t.Context(), db.RateLimitHistoryFilter{}) + require.NoError(t, err) + require.Len(t, rows, 2, "the incrementally-parsed second event must persist its own row") + + stats := engine.ResyncAll(t.Context(), nil) + require.False(t, stats.Aborted, "ResyncAll aborted: %+v", stats) + + rows, err = database.RateLimitSnapshotHistory(t.Context(), db.RateLimitHistoryFilter{}) + require.NoError(t, err) + assert.Len(t, rows, 2, "reparsing the file from scratch must not duplicate the incrementally-parsed row") +} diff --git a/internal/sync/s3.go b/internal/sync/s3.go index 9e825310b1..c7c9a4a761 100644 --- a/internal/sync/s3.go +++ b/internal/sync/s3.go @@ -67,6 +67,11 @@ func applyIDPrefixToParsedResult( } } } + for i := range result.RateLimitSnapshots { + result.RateLimitSnapshots[i].SessionID = applyIDPrefixToID( + prefix, result.RateLimitSnapshots[i].SessionID, + ) + } } func safeS3TempRelPath( diff --git a/internal/testjsonl/testjsonl.go b/internal/testjsonl/testjsonl.go index ef54c04fd1..13fa361d91 100644 --- a/internal/testjsonl/testjsonl.go +++ b/internal/testjsonl/testjsonl.go @@ -451,6 +451,82 @@ func CodexTokenCountJSON( return mustMarshal(m) } +// CodexRateLimitWindow describes one nullable rate_limits window +// ({used_percent, window_minutes, resets_at}) for test fixtures. +// ResetsAtNull encodes resets_at as JSON null (the Codex protocol allows +// this independently of the window itself being present) regardless of +// ResetsAt's value; leave it false to encode ResetsAt normally. +// WindowMinutesNull does the same for window_minutes. +type CodexRateLimitWindow struct { + UsedPercent float64 + WindowMinutes int + WindowMinutesNull bool + ResetsAt int64 + ResetsAtNull bool +} + +func (w *CodexRateLimitWindow) toMap() any { + if w == nil { + return nil + } + var resetsAt any = w.ResetsAt + if w.ResetsAtNull { + resetsAt = nil + } + var windowMinutes any = w.WindowMinutes + if w.WindowMinutesNull { + windowMinutes = nil + } + return map[string]any{ + "used_percent": w.UsedPercent, + "window_minutes": windowMinutes, + "resets_at": resetsAt, + } +} + +// CodexTokenCountWithRateLimitsJSON returns a Codex event_msg with +// payload.type=token_count carrying both last_token_usage and a +// rate_limits object shaped like a real Codex CLI payload: limit_id, +// plan_type, up to two nullable windows (primary/secondary), and a +// credits object. Pass primary/secondary as nil to encode a null window. +// Use CodexTokenCountJSON instead when the fixture needs rate_limits to +// be entirely absent or null. +func CodexTokenCountWithRateLimitsJSON( + timestamp string, + inputTokens, outputTokens, cachedInputTokens int, + limitID, planType string, + primary, secondary *CodexRateLimitWindow, + creditsBalance string, +) string { + m := map[string]any{ + "type": "event_msg", + "timestamp": timestamp, + "payload": map[string]any{ + "type": "token_count", + "info": map[string]any{ + "last_token_usage": map[string]any{ + "input_tokens": inputTokens, + "output_tokens": outputTokens, + "cached_input_tokens": cachedInputTokens, + "total_tokens": inputTokens + outputTokens, + }, + }, + "rate_limits": map[string]any{ + "limit_id": limitID, + "limit_name": nil, + "primary": primary.toMap(), + "secondary": secondary.toMap(), + "credits": map[string]any{"has_credits": true, "unlimited": false, "balance": creditsBalance}, + "individual_limit": nil, + "spend_control_reached": nil, + "plan_type": planType, + "rate_limit_reached_type": nil, + }, + }, + } + return mustMarshal(m) +} + // ClaudeEntryJSON returns a Claude JSONL entry with uuid and // parentUuid fields. func ClaudeEntryJSON(