Skip to content

Commit ce48c67

Browse files
dcramerclaude
dcramer
and
claude
authored
feat: add token cost tracking for MCP tool definitions (#596)
Implements comprehensive token cost measurement and tracking: - Token counter script with table/JSON output modes - GitHub Actions workflow with per-tool breakdown - Documentation and command integration The script measures the static overhead of tool definitions sent to LLM clients, helping monitor and optimize token usage. Current baseline: 8,769 tokens across 19 tools (~$26.31 per 1k requests) --------- Co-authored-by: Claude Code <noreply@anthropic.com>
1 parent d331ec4 commit ce48c67

10 files changed

Lines changed: 608 additions & 20 deletions

File tree

‎.github/workflows/token-cost.yml‎

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
name: Token Cost
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
8+
permissions:
9+
contents: read
10+
pull-requests: write
11+
checks: write
12+
statuses: write
13+
14+
jobs:
15+
measure-tokens:
16+
runs-on: ubuntu-latest
17+
steps:
18+
- uses: actions/checkout@v4
19+
20+
- name: Setup Node.js
21+
uses: actions/setup-node@v4
22+
with:
23+
node-version: "20"
24+
25+
# pnpm/action-setup@v4
26+
- uses: pnpm/action-setup@a7487c7e89a18df4991f7f222e4898a00d66ddda
27+
name: Install pnpm
28+
with:
29+
run_install: false
30+
31+
- name: Get pnpm store directory
32+
shell: bash
33+
run: |
34+
echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV
35+
36+
- uses: actions/cache@v4
37+
name: Setup pnpm cache
38+
with:
39+
path: ${{ env.STORE_PATH }}
40+
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
41+
restore-keys: |
42+
${{ runner.os }}-pnpm-store-
43+
44+
- name: Install dependencies
45+
run: pnpm install --no-frozen-lockfile
46+
47+
- name: Build tool definitions
48+
run: pnpm -w run build
49+
50+
- name: Measure token cost
51+
id: measure
52+
working-directory: packages/mcp-server
53+
run: |
54+
# Run token counter with JSON output to file
55+
pnpm run measure-tokens -- -o token-stats.json
56+
57+
# Extract key metrics from JSON for GitHub outputs
58+
TOTAL_TOKENS=$(jq -r '.total_tokens' token-stats.json)
59+
TOOL_COUNT=$(jq -r '.tool_count' token-stats.json)
60+
AVG_TOKENS=$(jq -r '.avg_tokens_per_tool' token-stats.json)
61+
62+
# Save for later steps
63+
echo "total_tokens=$TOTAL_TOKENS" >> $GITHUB_OUTPUT
64+
echo "tool_count=$TOOL_COUNT" >> $GITHUB_OUTPUT
65+
echo "avg_tokens=$AVG_TOKENS" >> $GITHUB_OUTPUT
66+
67+
- name: Generate detailed report
68+
id: report
69+
working-directory: packages/mcp-server
70+
run: |
71+
# Build markdown table from JSON
72+
cat > token-report.md <<REPORT_EOF
73+
## 📊 MCP Server Token Cost Report
74+
75+
**Summary**
76+
- **Total Tokens:** ${{ steps.measure.outputs.total_tokens }} tokens
77+
- **Tool Count:** ${{ steps.measure.outputs.tool_count }} tools
78+
- **Average:** ${{ steps.measure.outputs.avg_tokens }} tokens/tool
79+
80+
### Per-Tool Breakdown
81+
82+
| Tool | Tokens | % of Total |
83+
|------|--------|------------|
84+
REPORT_EOF
85+
86+
# Add each tool as a table row
87+
jq -r '.tools[] | "| `\(.name)` | \(.tokens) | \(.percentage)% |"' token-stats.json >> token-report.md
88+
89+
cat >> token-report.md <<REPORT_EOF
90+
91+
---
92+
93+
**Note:** This measures the static overhead of tool definitions sent to LLM clients with every request. Lower is better. The \`use_sentry\` tool is excluded as it's only available in agent mode.
94+
REPORT_EOF
95+
96+
# Display report
97+
cat token-report.md
98+
99+
# Save for summary
100+
cat token-report.md >> $GITHUB_STEP_SUMMARY
101+
102+
- name: Download main branch token stats
103+
if: github.event_name == 'pull_request'
104+
uses: dawidd6/action-download-artifact@v6
105+
continue-on-error: true
106+
with:
107+
workflow: token-cost.yml
108+
branch: main
109+
name_is_regexp: true
110+
name: 'token-stats-.*'
111+
path: main-stats
112+
search_artifacts: true
113+
114+
- name: Compare with main branch
115+
id: compare
116+
if: github.event_name == 'pull_request'
117+
working-directory: packages/mcp-server
118+
run: |
119+
# Check if we got main's stats
120+
if [ -f ../../main-stats/token-stats.json ]; then
121+
MAIN_TOKENS=$(jq -r '.total_tokens' ../../main-stats/token-stats.json)
122+
CURRENT_TOKENS=${{ steps.measure.outputs.total_tokens }}
123+
DELTA=$((CURRENT_TOKENS - MAIN_TOKENS))
124+
125+
echo "has_comparison=true" >> $GITHUB_OUTPUT
126+
echo "main_tokens=$MAIN_TOKENS" >> $GITHUB_OUTPUT
127+
echo "delta=$DELTA" >> $GITHUB_OUTPUT
128+
129+
if [ $DELTA -gt 0 ]; then
130+
echo "delta_direction=increased" >> $GITHUB_OUTPUT
131+
echo "delta_symbol=📈" >> $GITHUB_OUTPUT
132+
elif [ $DELTA -lt 0 ]; then
133+
echo "delta_direction=decreased" >> $GITHUB_OUTPUT
134+
echo "delta_symbol=📉" >> $GITHUB_OUTPUT
135+
else
136+
echo "delta_direction=unchanged" >> $GITHUB_OUTPUT
137+
echo "delta_symbol=➡️" >> $GITHUB_OUTPUT
138+
fi
139+
else
140+
echo "has_comparison=false" >> $GITHUB_OUTPUT
141+
fi
142+
143+
- name: Add comparison to report
144+
if: github.event_name == 'pull_request' && steps.compare.outputs.has_comparison == 'true'
145+
working-directory: packages/mcp-server
146+
run: |
147+
# Add comparison section at the top of the report
148+
cat > comparison-header.md <<COMPARISON_EOF
149+
### ${{ steps.compare.outputs.delta_symbol }} Comparison with Main Branch
150+
151+
- **Current (PR):** ${{ steps.measure.outputs.total_tokens }} tokens
152+
- **Main branch:** ${{ steps.compare.outputs.main_tokens }} tokens
153+
- **Change:** ${{ steps.compare.outputs.delta }} tokens (${{ steps.compare.outputs.delta_direction }})
154+
155+
---
156+
157+
COMPARISON_EOF
158+
159+
# Insert comparison at the beginning (after the title)
160+
sed -i '2r comparison-header.md' token-report.md
161+
162+
# Update job summary with new report
163+
cat token-report.md >> $GITHUB_STEP_SUMMARY
164+
165+
- name: Set commit status
166+
uses: actions/github-script@v7
167+
if: always()
168+
with:
169+
script: |
170+
const totalTokens = '${{ steps.measure.outputs.total_tokens }}';
171+
const hasComparison = '${{ steps.compare.outputs.has_comparison }}' === 'true';
172+
const delta = '${{ steps.compare.outputs.delta }}';
173+
const deltaSymbol = '${{ steps.compare.outputs.delta_symbol }}' || '📊';
174+
175+
let description = `${totalTokens} tokens`;
176+
if (hasComparison && delta) {
177+
const deltaPrefix = parseInt(delta) >= 0 ? '+' : '';
178+
description += ` (${deltaPrefix}${delta} from main) ${deltaSymbol}`;
179+
}
180+
181+
await github.rest.repos.createCommitStatus({
182+
owner: context.repo.owner,
183+
repo: context.repo.repo,
184+
sha: context.sha,
185+
state: 'success',
186+
context: 'Token Cost',
187+
description: description,
188+
target_url: `https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`
189+
});
190+
191+
- name: Upload token stats artifact
192+
uses: actions/upload-artifact@v4
193+
if: always()
194+
with:
195+
name: token-stats-${{ github.sha }}
196+
path: packages/mcp-server/token-stats.json
197+
retention-days: 90

‎AGENTS.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,9 @@ pnpm -w run cli --access-token=TOKEN "query" # Test with local stdio mode
7474
# Quality checks (combine for speed)
7575
pnpm run tsc && pnpm run lint && pnpm run test
7676

77+
# Token cost monitoring
78+
pnpm run measure-tokens # Check tool definition overhead
79+
7780
# Common workflows
7881
pnpm run build && pnpm run test # Before PR
7982
grep -r "TODO\|FIXME" src/ # Find tech debt
@@ -105,6 +108,7 @@ grep -r "TODO\|FIXME" src/ # Find tech debt
105108
- @docs/deployment.mdc — Deploy (Cloudflare)
106109
- @docs/monitoring.mdc — Monitoring/telemetry
107110
- @docs/security.mdc — Security and authentication
111+
- @docs/token-cost-tracking.mdc — Track MCP tool definition overhead
108112
- @docs/cursor.mdc — Cursor IDE integration
109113

110114
- LLM-Specific

‎docs/token-cost-tracking.mdc‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
---
2+
description: How to measure and track the token cost of MCP tool definitions
3+
globs:
4+
alwaysApply: false
5+
---
6+
# Token Cost Tracking
7+
8+
Measures the static overhead of MCP tool definitions - the tokens sent to LLM clients with every request.
9+
10+
## What's Being Measured
11+
12+
The token cost of tool metadata that MCP sends to clients via `tools/list`:
13+
- Tool names and descriptions
14+
- Parameter schemas (JSON Schema)
15+
- Total overhead per tool and across all tools
16+
17+
**Exclusions:**
18+
- `use_sentry` tool (agent-mode only, not exposed via standard MCP)
19+
- Runtime token usage by embedded agents (search_events, search_issues)
20+
21+
## Running Locally
22+
23+
**Display table (default):**
24+
```bash
25+
pnpm run measure-tokens
26+
```
27+
28+
**Output:**
29+
```
30+
📊 MCP Server Token Cost Report
31+
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
32+
Total Tokens: 9,069
33+
Tool Count: 19
34+
Average/Tool: 477
35+
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
36+
37+
Per-Tool Breakdown:
38+
39+
┌─────────────────────────────┬────────┬─────────┐
40+
│ Tool │ Tokens │ % Total │
41+
├─────────────────────────────┼────────┼─────────┤
42+
│ search_docs │ 1036 │ 11.4% │
43+
│ update_issue │ 757 │ 8.3% │
44+
...
45+
```
46+
47+
**Write JSON to file:**
48+
```bash
49+
# From repository root
50+
pnpm run measure-tokens -- -o token-stats.json
51+
52+
# Or from mcp-server package
53+
cd packages/mcp-server
54+
pnpm run measure-tokens -- -o token-stats.json
55+
```
56+
57+
JSON format:
58+
```json
59+
{
60+
"total_tokens": 9069,
61+
"tool_count": 19,
62+
"avg_tokens_per_tool": 477,
63+
"tools": [
64+
{"name": "search_docs", "tokens": 1036, "percentage": 11.4},
65+
...
66+
]
67+
}
68+
```
69+
70+
## CI/CD Integration
71+
72+
GitHub Actions workflow runs on every PR and push to main:
73+
74+
**On Pull Requests:**
75+
- 📝 **PR Comment:** Automatic comment with full report (updated on each push)
76+
- 📊 **Job Summary:** Detailed per-tool breakdown in Actions tab
77+
- 📦 **Artifact:** `token-stats-{sha}.json` stored for 90 days
78+
79+
**On Main Branch:**
80+
- 📊 **Job Summary:** Detailed per-tool breakdown in Actions tab
81+
- 📦 **Artifact:** `token-stats-{sha}.json` stored for 90 days
82+
83+
**Workflow:** `.github/workflows/token-cost.yml`
84+
85+
## Understanding the Results
86+
87+
**Current baseline (19 tools, excluding use_sentry):**
88+
- ~9,069 tokens total
89+
- ~477 tokens/tool average
90+
91+
**Tool count limits:**
92+
- **Target:** ≤20 tools (current best practice)
93+
- **Maximum:** ≤25 tools (hard limit for AI agents)
94+
95+
**When to investigate:**
96+
- Total tokens increase >10% without new tools
97+
- Individual tool >1,000 tokens (indicates overly verbose descriptions)
98+
- New tool adds >500 tokens (review description clarity)
99+
100+
## Implementation Details
101+
102+
**Tokenizer:** Uses `tiktoken` with GPT-4's `cl100k_base` encoding (good approximation for Claude).
103+
104+
**Script location:** `packages/mcp-server/scripts/measure-token-cost.ts`
105+
106+
**CLI options:**
107+
```bash
108+
tsx measure-token-cost.ts # Display table
109+
tsx measure-token-cost.ts -o file.json # Write JSON to file
110+
tsx measure-token-cost.ts --help # Show help
111+
```
112+
113+
## Optimizing Token Cost
114+
115+
**Reduce description verbosity:**
116+
- Be concise - LLMs don't need hand-holding
117+
- Remove redundant examples
118+
- Focus on unique, non-obvious details
119+
120+
**Simplify parameter schemas:**
121+
- Use `.describe()` sparingly
122+
- Avoid duplicate descriptions in nested schemas
123+
- Combine related parameters
124+
125+
**Consolidate tools:**
126+
- Before adding a new tool, check if existing tools can handle it
127+
- Consider parameter variants instead of separate tools
128+
129+
## References
130+
131+
- Script: `packages/mcp-server/scripts/measure-token-cost.ts`
132+
- Workflow: `.github/workflows/token-cost.yml`
133+
- Tool limits: See "Tool Count Limits" in `docs/adding-tools.mdc`

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@
3030
"lint": "biome lint",
3131
"lint:fix": "biome lint --fix",
3232
"inspector": "pnpx @modelcontextprotocol/inspector@latest",
33+
"measure-tokens": "pnpm run --filter ./packages/mcp-server measure-tokens",
3334
"prepare": "simple-git-hooks",
3435
"cli": "pnpm run --filter ./packages/mcp-test-client start",
3536
"start:stdio": "pnpm --stream run --filter ./packages/mcp-server start",

‎packages/mcp-server/package.json‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,7 @@
1313
"author": "Sentry",
1414
"description": "Sentry MCP Server",
1515
"homepage": "https://github.com/getsentry/sentry-mcp",
16-
"keywords": [
17-
"sentry"
18-
],
16+
"keywords": ["sentry"],
1917
"bugs": {
2018
"url": "https://github.com/getsentry/sentry-mcp/issues"
2119
},
@@ -26,9 +24,7 @@
2624
"bin": {
2725
"sentry-mcp": "./dist/index.js"
2826
},
29-
"files": [
30-
"./dist/*"
31-
],
27+
"files": ["./dist/*"],
3228
"exports": {
3329
".": {
3430
"types": "./dist/index.ts",
@@ -119,12 +115,14 @@
119115
"test:watch": "pnpm run generate-definitions && vitest",
120116
"tsc": "tsc --noEmit",
121117
"generate-definitions": "tsx scripts/generate-definitions.ts",
122-
"generate-otel-namespaces": "tsx scripts/generate-otel-namespaces.ts"
118+
"generate-otel-namespaces": "tsx scripts/generate-otel-namespaces.ts",
119+
"measure-tokens": "tsx scripts/measure-token-cost.ts"
123120
},
124121
"devDependencies": {
125122
"@sentry/mcp-server-mocks": "workspace:*",
126123
"@sentry/mcp-server-tsconfig": "workspace:*",
127124
"msw": "catalog:",
125+
"tiktoken": "^1.0.18",
128126
"yaml": "^2.6.1",
129127
"zod-to-json-schema": "catalog:"
130128
},

0 commit comments

Comments
 (0)