dashboard: cancelled / client_disconnected requests inflate error rate metric

[cat-thats-fat]

Pre-flight checklist

• I searched existing issues and confirmed this is not a duplicate.
• I am running a supported codex-lb version (see Releases).
• I can reproduce the problem with the steps below.

codex-lb version

1.22.0 (latest tag)

Deployment method

Docker

Client used against codex-lb

Codex CLI via Hermes Agent

ChatGPT account plan(s) involved

3x Plus

Model(s) involved

gpt-5.6-sol

What happened?

The dashboard overview error rate is dramatically inflated because it counts status=cancelled / error_code=client_disconnected requests as errors alongside genuine upstream failures.

The error rate is computed in app/modules/usage/builders.py (line ~307):

error_logs = [log for log in logs_secondary if log.status != "success"]

This treats any request that didn't end with a clean terminal event as an error. status=cancelled requests are normal agent lifecycle — they occur whenever a Codex CLI client disconnects before the final SSE event lands (e.g. agent finishes a turn, user stops a session, parallel agents cycle). They are not upstream errors.

On an instance running multiple parallel Codex CLI agents, cancelled requests dominate the total:

SELECT status, COUNT(*) as cnt FROM request_logs
WHERE requested_at > datetime('now', '-1 hour')
GROUP BY status ORDER BY cnt DESC;

success    | 4
cancelled  | 204
error      | 0

Real upstream error rate: 0%
Dashboard displayed error rate: 98%

A user with previously stable error rate (0.2-0.7%) sees it spike to 60-98% due to normal agent activity, making the dashboard metric misleading for operational monitoring.

What did you expect to happen?

cancelled requests should be excluded from the error rate calculation, or the dashboard should display a breakdown (success / cancelled / error) so users can distinguish upstream failures from normal client lifecycle disconnects.

Suggested: error_rate = error_count / (success_count + error_count)

Steps to reproduce

1. Run multiple parallel Codex CLI sessions through codex-lb
2. Let agents start/stop naturally over an hour
3. Observe dashboard error rate climbing due to cancelled requests
4. Query request_logs table to confirm cancelled dominates

Relevant logs

N/A — reproducible purely from the request_logs table.

Configuration / environment

Default configuration, bridge disabled (CODEX_LB_HTTP_RESPONSES_SESSION_BRIDGE_ENABLED=false), but issue reproduces regardless.

Additional context

Issue #1323 documents the cancelled/client_disconnected metadata recording. This is a separate issue about how that status is then used in the error rate calculation.

[Soju06]

Confirmed. _usage_metrics counts status != "success" (app/modules/usage/builders.py:355), and cancelled is a normal terminal status written on client disconnect (app/modules/proxy/_service/streaming/helpers.py:616-620, app/modules/proxy/_service/websocket/mixin.py:1852). Two things a fix needs beyond that one line:

• The same fold is materialized elsewhere: build_usage_metrics_from_aggregate divides aggregate.error_count (builders.py:322), and error_count is baked into the time-rollup tables as sum(status != 'success') (app/modules/accounts/usage_time_rollup.py:440, app/db/models.py:266) plus app/modules/reports/repository.py:159,547 and app/modules/fleet/observability.py:137. The dashboard error-rate trend reads those rollup rows (builders.py:136-146), so fixing only the builder leaves the trend and aggregate paths inflated — and historical rollup rows can't be re-split without a backfill or a separate cancelled_count measure going forward.
• top_error derives from the same filtered set (builders.py:359), so client_disconnected dominates it too.

Classification-wise the current metric is also inconsistent with routing health, which already treats client disconnects as non-penalizing (penalize_account=not client_disconnected, websocket/mixin.py:1852-1853). Excluding cancelled from the error numerator is right; numerator-only vs. also dropping it from the denominator is a display choice — a success/cancelled/error breakdown preserves the most signal.