Skip to content

feat: show token usage and rate-limit footer per turn - #111

Merged
shauryagangrade merged 2 commits into
shauryagangrade:mainfrom
aryansk:feat-token-usage-70
Aug 24, 2026
Merged

feat: show token usage and rate-limit footer per turn#111
shauryagangrade merged 2 commits into
shauryagangrade:mainfrom
aryansk:feat-token-usage-70

Conversation

@aryansk

@aryansk aryansk commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Fixes #70

Problem

Free OpenRouter models are rate-limited and users get no feedback on how much context they're consuming. ChatOpenAI streams carry usage metadata, but GCode drops it. Issue #70 proposes capturing usage/rate-limit info from stream chunks and printing a compact footer after each turn: tokens in/out, approximate cost, and remaining rate limit, plus a /usage surface.

Change

  • Add gcode/agent.py:_print_usage(response, ui) that extracts usage from response.usage_metadata (LangChain 0.3+) with fallback to response.response_metadata['token_usage'/'usage']
  • Fix _stream to carry usage_metadata/response_metadata onto the canonical AIMessage it returns — without this the rebuilt message discarded the provider's token counts and the footer never printed
  • Call _print_usage after each _stream in run_turn (initial + each tool-loop iteration, so multi-tool turns show one line per model call); silently omits when provider supplies no usage
  • Footer format: Usage: <in> in / <out> out · total <n> · ~$<cost> (only present fields shown; zero counts are preserved via explicit None checks)
  • Rate-limit header parsing dropped: ChatOpenAI never exposes raw response headers in response_metadata, so that path was unreachable. Surfacing rate-limit state needs an httpx hook on the underlying client — tracked as a follow-up
  • Never breaks the turn on footer errors; unit tests added for metadata preservation, both usage sources, zero-count handling, cost display, and silent omission

Why this approach

Reuses the existing streaming path; the footer is the minimal user-visible fix matching the acceptance criterion "A turn prints usage when the provider supplies it; silently omits otherwise." No new /usage command needed because the footer is automatic; folding into /history would be redundant.

Testing

command: uv run ruff check gcode tests
result: all checks passed

command: uv run mypy gcode
result: success, no issues

command: uv run pytest tests/
result: 173 passed

Unit tests in tests/test_agent.py: _stream preserves usage_metadata; footer renders from usage_metadata, from response_metadata['token_usage'], keeps zero counts, shows cost, and stays silent with no usage.

Manual (with API key): start a session, trigger a turn, and observe footer like Usage: 842 in / 128 out · total 970 when OpenRouter returns usage; no footer when Ollama/local returns no usage.

Documentation and release impact

  • User-facing documentation updated: footer is visible in terminal
  • Changelog/release note needed: feature
  • Migration or compatibility note needed
  • No documentation impact

Review notes

  • Known limitations: cost is shown only when the provider returns it; rate-limit remaining is not shown (see follow-up)
  • Follow-up issue: surface OpenRouter rate-limit headers requires a custom httpx event hook — not reachable through LangChain's ChatOpenAI today
  • Security/licensing considerations: none

aryansk and others added 2 commits August 24, 2026 20:43
Fixes shauryagangrade#70

Free OpenRouter models are rate-limited and context consumption is
invisible. ChatOpenAI streams carry usage metadata but GCode dropped
it. Capture usage from response.usage_metadata and
response_metadata.token_usage/usage, plus X-RateLimit-Remaining
headers when present, and print a compact footer after each turn:
tokens in/out, total, approximate cost, and remaining rate limit.
Silently omits when provider supplies no usage. Rate-limit remaining
is surfaced before a 429, matching the acceptance criteria.

Validation: py_compile passes, git diff --check clean; footer prints
only when usage present, never breaks the turn.
_stream previously rebuilt the AIMessage without usage_metadata /
response_metadata, discarding the provider's token counts before
_print_usage ever saw them. Also drop the rate-limit header parsing
(ChatOpenAI never exposes response headers), keep zero token counts
via explicit None checks, and add unit tests.
@shauryagangrade
shauryagangrade marked this pull request as ready for review August 24, 2026 16:07
@shauryagangrade
shauryagangrade merged commit 05af73f into shauryagangrade:main Aug 24, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Show token usage and rate-limit warnings per turn

2 participants