feat: telemetry surface (/status extras, /metrics, /usage)#1
Merged
Conversation
Expose the router-only telemetry no single backend can produce: queue
backlog, swap economics, rolling throughput, cumulative counters, a
recent-request log, and error/health signal. Jano already sees every
request/response and invokes every swap, so this is mostly counters and
small ring buffers in existing code paths.
- extend GET /status with inFlight, oldestWaitingMs, currentModelLoadedAt,
lastSwapDurationMs, swaps{15m,Hour}, recent{Gen,Prompt}TokS, tokSByModel,
uptime, cumulative counters, requestsPerMinute, errors, backendHealth
- add GET /metrics (Prometheus: counters, queue gauges, swap-duration
histogram + per-transition cost table, backend health)
- add GET /usage?limit=N (recent per-request token/timing records)
- telemetry.ts: bounded ring buffers + counters, clock-injectable
- responseStats.ts: normalize OpenAI usage / llama.cpp timings / Ollama
nanosecond fields to tok/s; bodyTap.ts captures stats without buffering
- USAGE_RECORDS_MAX env (default 500)
Hardware metrics and relay heartbeat stay out of scope (sidecar's job).
35 new tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the in-process telemetry surface that makes "RigPilot + Jano" strictly richer than "RigPilot + plain Ollama." Because Jano sits above the backends and owns every request, response, and swap, it can expose telemetry no single backend (Ollama, llama-server, mlx-lm.server) can produce on its own.
All of it is bounded ring buffers and counters in code paths that already exist. Existing
/healthand/statuscallers keep working.Extended
GET /statusQueue/concurrency (
inFlight,oldestWaitingMs), swap economics (currentModelLoadedAt,lastSwapDurationMs,swapsLast15m/Hour), rolling throughput across all callers (recentGenTokS,recentPromptTokS,tokSByModel), cumulative counters (requestsServedTotal,tokensGeneratedTotal, both per-model,requestsPerMinute,uptimeSeconds), and error/health signal (errorsLast15m,lastError,backendHealth).New
GET /metricsPrometheus text exposition: counters, queue gauges, swap-duration histogram + per-transition cost table, backend-up gauges.
New
GET /usage?limit=NRing of recent per-request records
{ts, model, status, total_ms, ttfb_ms, prompt_tokens, gen_tokens, gen_tok_s, prompt_tok_s}. Size viaUSAGE_RECORDS_MAX(default 500).How
telemetry.ts— bounded aggregator, clock-injectable (likeFailureBudget)responseStats.ts— normalizes OpenAIusage, llama.cpptimings, and Ollama nanosecond fields to tok/sbodyTap.ts— captures just enough of a forwarded body to read stats without buffering the streammetrics.ts— the live singleton; dispatcher records swaps/errors/in-flight, HTTP layer records per-request token/timingOut of scope
Hardware metrics (GPU temp/util/fan, system RAM/CPU/power) and a relay heartbeat stay the sidecar's job.
Tests
35 new tests (86 total); all four CI gates pass (format, lint, typecheck, test).