[ducanh3005]

Closes #21461

Failure-Class: none
Line-Count-Exception: backend/utils/metrics.py | 1523 -> 1592 | Central Prometheus registry owns bounded WS handshake, API 429, and direct LLM quota counters (#21461).

Summary

Adds bounded, startup zero-initialized Prometheus counters to cover the three alerting gaps after pausing raw-line Loki log rules (#21461):

1. omi_listen_ws_handshake_failures_total{surface,reason} (backend/utils/metrics.py, backend/routers/transcribe.py):
   • Bounded surface (mobile, web, fallback other) and reason (accept_runtime_error, fallback other) labels, startup zero-initialized.
   • Wired into websocket.accept() RuntimeError handlers in _listen (surface='mobile') and web_listen_handler (surface='web').
2. omi_api_rate_limit_throttled_total{scope,policy_family} (backend/utils/metrics.py, backend/utils/other/endpoints.py):
   • Bounded scope (redis_policy, ip_window, fallback other) and policy_family (derived directly from RATE_POLICIES in utils/rate_limit_config.py plus fallback custom), startup zero-initialized across all families.
   • Wired into rate_limit_custom (scope='ip_window') and live (non-shadow) _enforce_rate_limit (scope='redis_policy').
3. omi_direct_llm_rate_limit_errors_total{provider,source,reason} (backend/utils/metrics.py, backend/utils/llm/byok_errors.py):
   • Bounded provider (openai, anthropic, gemini, openrouter, groq, other), source (platform, byok, other), and reason (quota, rate_limit), startup zero-initialized across all 6 providers.
   • Wired into handle_llm_error via _classify_rate_limit_or_quota_error while keeping classify_byok_llm_error behavior unchanged.

Verification

• pytest backend/tests/unit/test_listen_quota_handshake_metrics.py -v (6/6 passing, including RATE_POLICIES family parity check)

[ducanh3005]

Payout & Verification Info (@ducanh3005)

• Author GitHub: @ducanh3005 (ynsuperit2@gmail.com)
• Bounty Recipient Wallet (EVM - Base / Polygon / Arbitrum): 0xdF4Ff9D2f45f99f11A90b16F277D4B7EddDB72A5
• Solana Wallet: 3peh7eRyemZQ3ppVC9CeUeEbHq6aADmwqvSTRnQf17b7
• Preferred Token: USDC

[Git-on-my-level]

Verdict: the counters themselves are well designed — bounded label sets, startup zero-initialization, real tests — and one mechanical CI blocker (Hygiene line-count ratchet) needs clearing before merge.

Findings:

• backend/utils/metrics.py — all three counters pre-create their full label cross-product and the record_* helpers collapse unknown values to other/custom, so there is no cardinality-growth path. Maintainability note: _API_RATE_LIMIT_POLICY_FAMILIES hardcodes the family list instead of deriving it from RATE_POLICIES in utils/rate_limit_config.py — a new policy family added there will silently land in policy_family="custom". Consider deriving the set from RATE_POLICIES or adding a parity test.
• backend/utils/other/endpoints.py — placement is right on both paths: the redis_policy increment sits after the RATE_LIMIT_SHADOW early-return (shadow traffic is not counted as throttling), and rate_limit_custom increments only on the actual 429 raise. Keys/uids are never used as labels (family prefix only) — no identifiers leak into metrics.
• backend/routers/transcribe.py — increments inside the existing except RuntimeError handlers before return; no behavior change beyond the metric.
• backend/utils/llm/byok_errors.py — _classify_rate_limit_or_quota_error mirrors the existing classifier without changing classify_byok_llm_error, and records platform as well as byok sources. Minor: any 429 whose message merely contains "quota" (e.g. "quota resets at midnight") is classified reason="quota" — fine for alerting, just be aware the quota/rate_limit split is heuristic.
• backend/tests/unit/test_listen_quota_handshake_metrics.py — good coverage: zero-init children, label collapse, both WS surfaces, both rate-limit scopes, platform/byok LLM paths; the endpoints.cached mutation is cleaned up in a finally block.

Blocking (Hygiene): the product-file-line-count ratchet fails because backend/utils/metrics.py grew 1523 -> 1594. Either add a line like:

Line-Count-Exception: backend/utils/metrics.py | 1523 -> 1594 |

to the PR body, or move the new counters into their own module if you prefer a split.

Not your fault (Backend unit suite): all 6 shards failed on 5 pre-existing pyright errors in utils/retrieval/tools/omi_tools.py, utils/retrieval/tools/result_bounds.py, and utils/wrapped/generate_2025.py — the same errors failed main shortly before your CI ran, and main has since gone green. Rebasing onto current main (or re-running after a rebase) should clear it.

Performance: negligible — roughly a hundred pre-created series, increments only on failure paths. Blast radius: backend metrics/alerting only; a wrong label merely misroutes an alert.

USER IMPACT: none — internal observability change; no user-visible behavior differs.

What I'd still want as a reviewer: the policy-family parity guard (or derivation from RATE_POLICIES), and a green Backend unit suite on the rebased head. Beyond that, the new metric names and label contracts are what #21461's alerting rules will key on, so a maintainer should sign off on those names before merge.

---

by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.

[ducanh3005]

Thanks for the review @Git-on-my-level! Addressed all points in 82314cf647:

• Derived _API_RATE_LIMIT_POLICY_FAMILIES directly from RATE_POLICIES in backend/utils/rate_limit_config.py (frozenset({k.split(':', 1)[0] for k in RATE_POLICIES} | {'custom'})) so newly added policy families are automatically registered and zero-initialized at startup.
• Added test_api_rate_limit_policy_families_match_rate_policies parity check in backend/tests/unit/test_listen_quota_handshake_metrics.py.
• Added Line-Count-Exception: backend/utils/metrics.py | 1523 -> 1592 | ... to the PR description so PR Metadata Preflight and Hygiene pass, and triggered a fresh CI run against current main.