Skip to content

feat: add configurable provider RPM limiting - #259

Open
BelugaRex wants to merge 9 commits into
smallmain:mainfrom
BelugaRex:main
Open

feat: add configurable provider RPM limiting#259
BelugaRex wants to merge 9 commits into
smallmain:mainfrom
BelugaRex:main

Conversation

@BelugaRex

Copy link
Copy Markdown

Summary

  • add global default and per-provider RPM configuration; set rpm: 0 to disable a provider's limiter
  • use a token bucket at the logical chat-request entry point so HTTP, SSE, and Responses WebSocket traffic share the same limit
  • serialize concurrent token acquisition FIFO to prevent negative balances or duplicate consumption
  • display the post-acquire token snapshot with two decimal places and document the per-window scope

Validation

  • npm run compile
  • npm run l10n:check
  • concurrent RateLimiter(60) harness: 52 requests completed FIFO; capacity was 48; the final four completed at approximately 1-second intervals; no snapshot became negative

Fixes #242

Copilot AI review requested due to automatic review settings July 13, 2026 19:44
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@BelugaRex

Copy link
Copy Markdown
Author

希望大佬见谅,我是vide的,但是我有本地测试过,好像是能有效果的。(比较悲哀的是,就算我卡rpm在4,也会被老黄429,唉)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds configurable RPM (requests per minute) rate limiting for logical chat requests, with a global default and per-provider overrides, and applies the limiter at the service entry point so all transports (HTTP/SSE/WebSocket) share the same limit within a VS Code window.

Changes:

  • Introduces a token-bucket RateLimiter with FIFO token acquisition and a createRateLimiter helper.
  • Wires rate-limit token acquisition + status logging into UnifyChatService and the various provider clients via ApiProvider optional methods.
  • Adds UI/config plumbing (schema, forms, descriptions) and localization strings for global and per-provider rateLimit.rpm.

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/ui/screens/timeout-form-screen.ts Adds rate-limit RPM field editing to the timeout/network settings UI flow.
src/ui/screens/provider-form-screen.ts Passes rateLimit into the timeout form route for provider editing.
src/ui/screens/provider-draft-form-screen.ts Passes rateLimit into the timeout form route for draft editing.
src/ui/router/types.ts Extends TimeoutFormRoute to include rateLimit.
src/ui/provider-fields.ts Includes rate-limit presence/value in provider “timeout” field description string.
src/types.ts Adds rateLimit?: RateLimitConfig to ProviderConfig with documented semantics.
src/service.ts Acquires a rate-limit token before transport selection and logs the limiter snapshot.
src/rate-limit.ts New token-bucket rate limiter implementation + config type + factory.
src/logger.ts Adds formatting and log suffix for rate-limit token snapshots.
src/config-store.ts Reads global networkSettings.rateLimit and applies as a provider default.
src/config-ops.ts Persists rateLimit as part of provider config keys.
src/client/openai/responses-client.ts Creates and exposes rate limiter for OpenAI Responses provider.
src/client/openai/chat-completion-client.ts Creates and exposes rate limiter for OpenAI Chat Completions provider.
src/client/ollama/client.ts Creates and exposes rate limiter for Ollama provider.
src/client/interface.ts Adds optional getRateLimitStatus / acquireRateLimitToken hooks to ApiProvider.
src/client/google/ai-studio-client.ts Creates and exposes rate limiter for Google AI Studio provider.
src/client/github-copilot/client.ts Creates and exposes rate limiter for GitHub Copilot provider.
src/client/anthropic/client.ts Creates and exposes rate limiter for Anthropic provider.
package.nls.zh-cn.json Adds Chinese configuration descriptions for rate-limit settings.
package.nls.json Adds English configuration descriptions for rate-limit settings.
package.json Adds configuration schema entries for global and per-provider rateLimit.rpm; bumps version.
package-lock.json Updates lockfile version metadata to match package version bump.
l10n/bundle.l10n.zh-cn.json Adds UI strings for RPM rate limit editing (Chinese).
l10n/bundle.l10n.json Adds UI strings for RPM rate limit editing (English).
CHANGELOG.md Adds v7.12.4 changelog entry covering the rate limiter changes.
.vscode/指北.md Adds internal guidance doc describing RPM limiter semantics and maintenance boundaries.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/config-store.ts
Comment on lines +390 to +397
// Apply global networkSettings.rateLimit as default when the endpoint
// does not define its own rate-limit configuration.
if (provider.rateLimit === undefined) {
const globalRateLimit = this.readNetworkRateLimit();
if (globalRateLimit) {
provider.rateLimit = globalRateLimit;
}
}
Comment thread src/service.ts
Comment on lines +772 to +775
const client = this.getClient(resolvedProvider);
await client.acquireRateLimitToken?.();
const rateLimitStatus = client.getRateLimitStatus?.();

Comment thread src/rate-limit.ts
Comment on lines +88 to +97
/**
* Peek at the current token count without consuming any.
*
* Refills first so the returned value reflects tokens earned since
* the last access. Used for logging / status display.
*/
getAvailableTokens(): { available: number; capacity: number } {
this.refill();
return { available: this.tokens, capacity: this.maxTokens };
}
- config-store: normalize provider rateLimit so invalid/empty values (e.g. {}
  or non-integer rpm) fall back to the global default, while still honoring an
  explicit rpm: 0 override
- rate-limit: accept an optional AbortSignal in acquire(); abort a cancelled
  waiter without consuming a token
- service: wire the chat CancellationToken into rate-limit token acquisition so
  cancel-while-waiting aborts early
- rate-limit: getAvailableTokens no longer refills, so the logged post-acquire
  snapshot is exact instead of slightly inflated
@BelugaRex

Copy link
Copy Markdown
Author

Addressed Copilot review suggestions

Commit f6982d1 tackles the three Copilot review comments:

1. config-store.ts — Global rateLimit fallback skipped for invalid per-provider config (Medium)

  • Added normalizeRateLimitConfig(): invalid/empty per-provider rateLimit (e.g. {} or non-integer rpm) is now normalized to undefined, so it falls back to the global networkSettings.rateLimit default. Explicit rpm: 0 (disable limiter) is still preserved — it does not fall back.

2. rate-limit.ts — getAvailableTokens inflates the logged post-acquire snapshot (Low)

  • getAvailableTokens() no longer refills before returning. Since acquire() already refills internally before consuming, the snapshot taken right after an acquisition is the exact post-consumption bucket state instead of being slightly inflated by freshly-earned tokens.

3. service.ts — Rate-limit acquisition is not cancellation-aware (Medium)

  • RateLimiter.acquire(signal?: AbortSignal) now accepts an AbortSignal; if aborted before or while waiting, it rejects with an AbortError without consuming a token.
  • ApiProvider.acquireRateLimitToken(signal?) and all 6 provider implementations forward the signal.
  • service.ts bridges the VS Code CancellationToken into an AbortController passed to acquireRateLimitToken, so a user cancellation while waiting for a token aborts early and doesn't waste a rate-limit slot.

Also updated CHANGELOG.md and .vscode/指北.md, and npm run compile passes clean.

# Conflicts:
#	package-lock.json
#	src/client/interface.ts
#	src/config-ops.ts
#	src/config-store.ts
#	src/types.ts
# Conflicts:
#	CHANGELOG.md
#	package-lock.json
#	package.json
#	src/client/anthropic/client.ts
#	src/config-store.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

增加 RPM 等常见限流配置

2 participants