Skip to content

ci(changelog-review): track per-call token efficiency over time #1305

@claude

Description

@claude

Context

From PR #1302 follow-up (not blocking):

The show_full_output: false setting in anthropics/claude-code-action keeps the per-call token counts hidden. If we ever want to track token-level efficiency over time, we could either flip that on for the triage job specifically, or scrape the OpenTelemetry log events that the SDK emits (richer detail than the summary result).

Background

The PR analysis used total_cost_usd as a proxy for token volume (since per-call counts aren't exposed at the result level with show_full_output: false). Per-run cost comparison:

Run Turns Duration Notional $ Notes
2026-04-27 4 47m $57.45 Big-context first-hit
2026-05-11 (failed) 51 2h 7m $63.69 62-version gap
2026-05-04 41 8m 31s $2.01 Normal weekly run

Options

Option A: show_full_output: true on the triage job

  • Exposes per-call token counts in CI logs
  • Increases log verbosity significantly
  • Easy to implement: one line change in the workflow

Option B: Scrape OpenTelemetry events from SDK

  • Richer detail (per-call breakdown, cache hit rates)
  • Requires parsing structured log output
  • More setup complexity

Option C: Status quo — use total_cost_usd as proxy

  • Already working; cost is a reasonable efficiency signal
  • No action needed unless finer granularity is wanted

Acceptance criteria (if pursued)

  • Per-call token counts are captured in CI artifacts or a summary step
  • Cache hit rate is visible (useful for tuning prompt structure)

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions