server: emit GPU ttft/generation/decode_tps in stream usage

[ai-hpc]

Summary

• Measure GPU decode on the continuous-batch worker around forward_token (excluding SSE/network backpressure).
• Expose ttft_ms, generation_ms, and decode_tps on the final streaming usage chunk so UIs (e.g. sparkinfer-web) can show accurate tok/s.
• Additive OpenAI-compatible fields only; existing prompt_tokens / completion_tokens / total_tokens unchanged.

Pipeline impact

• Safe for existing clients — extra JSON fields are ignored by OpenAI SDKs and current bench scripts (they compute their own client-side decode_tps).
• No change to scheduling, KV, sampling, eval bots, or native bench binaries.
• Negligible overhead — one steady_clock sample around each decode forward.
• Left kMaxOutputTokens at the upstream default (4096); deployment max-out remains an init/CLI concern.

Test plan

• Rebuild sparkinfer_server and stream a short chat; confirm usage includes decode_tps / ttft_ms / generation_ms
• Smoke existing API benches still parse usage (bench_api_vs_native.py)
• Confirm non-stream /v1/chat/completions still returns standard usage
• sparkinfer-web chat shows non-zero GPU tok/s

[skyrocket2026]

Review

Nice additive change — streaming usage with ttft_ms / generation_ms / decode_tps is the right shape for the chat UI, and measuring around forward_token (before on_token) correctly excludes SSE backpressure.

Blocker: last_timing_ is racy under continuous batching

ModelEngine::complete_streaming writes last_timing_ after the job finishes, then sparkinfer_server reads engine.last_timing() on a later line. httplib serves concurrent requests and the batch engine interleaves them, so this can happen:

1. Request A finishes → sets last_timing_
2. Request B finishes → overwrites last_timing_
3. Request A’s handler reads last_timing() → B’s timings

Worse: if the stolen values are >= 0, the per-request wall-clock fallback in sparkinfer_server.cpp is skipped, so the UI can show another session’s tok/s.

last_timing() also returns an unlocked reference (unlike last_error() which takes mu_).

Suggested fix: return timings with the completion result (extend the return of complete_streaming, or an out-param / pair), and use that local value in the stream handler. Avoid a process-global “last timing” side channel.

Non-blocking notes

• Non-stream /v1/chat/completions still emits standard usage only (fine if intentional).
• Single-token completions fall back to wall-clock decode_tps (decode_forwards == 0); OK.
• Prefill seed token correctly excluded from decode_forwards (decode tok/s after first token).

Please fix the timing handoff, then this looks good to merge.

[skyrocket2026]

Maintainer hold: non-speed runtime/server change — keep open; do not auto-close for missing RTX 5090 greenlight.

[skyrocket2026]

Closed — RTX 5090 checkbox not ticked

This PR was auto-closed because the template includes Tested on RTX 5090 as - [ ] (unchecked). Evaluation is opt-in — tick the box only after a real 5090 run.

To submit for review:

1. Edit this PR description (you can edit while closed): change to - [x] Tested on RTX 5090
2. Fill the decode and/or prefill before → after tables with real bench/scripts/bench.sh numbers showing improvement
3. Reopen this PR

If this PR does not need GPU eval (e.g. docs-only), remove the proof-of-speedup section from the description instead of leaving an unchecked box.

CONTRIBUTING.md

_{Automated by eval bot / rtx5090-required CI.}

[skyrocket2026]

closed

[skyrocket2026]

Reopened + hold. This is a non-speed runtime/server PR — do not auto-close for missing RTX 5090 greenlight. Eval bot incorrectly closed it despite hold; fixing that path.