Add model context length and manual model deletion#520
Open
roiding wants to merge 2 commits intocita-777:mainfrom
Open
Add model context length and manual model deletion#520roiding wants to merge 2 commits intocita-777:mainfrom
roiding wants to merge 2 commits intocita-777:mainfrom
Conversation
added 2 commits
April 29, 2026 17:04
- New modelContextLengthCache service for in-memory model→context_length mapping - Platform adapters (newApi, standardApiProvider) extract context_length from upstream /v1/models - modelsSurface injects context_length into both OpenAI and Claude response formats - Default 1,000,000 when upstream does not provide context_length - Supports field names: context_length, contextLength, max_context_length, contextWindow, etc.
- Backend: DELETE /api/accounts/:id/models/manual endpoint - Only deletes models where isManual=true (safe against auto-discovered models) - Frontend: AccountModelsModal shows '✕ 删除' button next to each manual model - Frontend: api.removeAccountManualModels() function - Accounts.tsx: wires up delete handler with toast feedback
📝 WalkthroughWalkthroughThe pull request introduces model context length caching across the platform infrastructure and adds a new backend endpoint to remove manually configured models from accounts, along with corresponding frontend API and UI components to support deletion workflows. Changes
Sequence Diagram(s)sequenceDiagram
participant Client as Client
participant Server as Server<br/>(Model Listing)
participant Cache as Context Length<br/>Cache
participant Response as API Response
Client->>Server: GET /models
activate Server
Server->>Cache: getModelContextLength(modelId)
activate Cache
Cache-->>Server: contextLength (or default 1M)
deactivate Cache
Server->>Response: Return models with<br/>context_length field
deactivate Server
Response-->>Client: Model list with context<br/>length metadata
sequenceDiagram
participant Client as Client
participant Server as Server<br/>(Accounts Route)
participant Validator as Request<br/>Validator
participant DB as Database
participant Router as Route<br/>Rebuilder
Client->>Server: DELETE /api/accounts/:id/models/manual<br/>{models: [...]}
activate Server
Server->>Validator: Validate payload &<br/>accountId
activate Validator
Validator-->>Server: Valid payload &<br/>account exists
deactivate Validator
Server->>DB: Delete modelAvailability<br/>records where isManual=true
activate DB
DB-->>Server: Deletion complete
deactivate DB
Server->>Router: rebuildRoutesBestEffort()
activate Router
Router-->>Server: Rebuild complete
deactivate Router
Server-->>Client: {success: true}
deactivate Server
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 7/8 reviews remaining, refill in 7 minutes and 30 seconds.Comment |
github-actions
Bot
added
area: server
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: af9aac3ad7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
||
| const DEFAULT_CONTEXT_LENGTH = 1_000_000; | ||
|
|
||
| const cache = new Map<string, number>(); |
There was a problem hiding this comment.
modelContextLengthCache stores values in a single process-wide map keyed only by model name, but the values are populated from per-account upstream /v1/models payloads (via setModelContextLengths in platform adapters). In a multi-account deployment where the same model ID has different limits (or one provider omits the field), the most recently refreshed account overwrites the value for all others, so /v1/models can return incorrect context_length metadata and cause clients to size prompts incorrectly for the actual selected channel.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
src/server/services/modelContextLengthCache.test.ts (1)
33-45: Add a whitespace-only model-name test case.Current invalid-name assertions cover empty string but not whitespace-only input (e.g.
' '). Adding that case would lock in the expected behavior and catch empty-key regressions.Also applies to: 67-78
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/server/services/modelContextLengthCache.test.ts` around lines 33 - 45, Add tests that assert whitespace-only model names are treated as invalid: update the 'ignores invalid values' case to call setModelContextLength(' ', 128000) and expect hasModelContextLength(' ') toBe(false), and likewise add the same whitespace-only check in the other invalid-name test block (the one around lines 67-78). This ensures setModelContextLength and hasModelContextLength both trim/validate names (or reject whitespace-only keys) consistent with the empty-string behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/server/routes/api/accounts.ts`:
- Around line 1969-1983: Extract the deletion loop into a new service function
(e.g., removeManualModelsFromAccount in
src/server/services/accountManualModelsService.ts) that accepts { accountId,
modelNames } and runs the deletes inside a single db.transaction using the
transaction handle (tx) for the .delete(schema.modelAvailability) calls and the
same where(...) predicate (use input.accountId and each modelName); then replace
the loop in the route with a call to this service and only call
rebuildRoutesBestEffort() after the service resolves successfully; ensure errors
are propagated so the route can return the appropriate error response.
In `@src/server/services/modelContextLengthCache.ts`:
- Around line 13-43: The cache is currently process-global and keyed only by
model name, causing cross-source collisions; change the cache scheme to be
scoped by source by updating the cache variable and normalizeKey to incorporate
a source identifier (e.g., normalizeKey(source, modelName) or use a compound key
`${source}:${modelName}`), update setModelContextLength(source, modelName,
contextLength) and getModelContextLength(source, modelName) signatures to use
the scoped key, and modify setModelContextLengths to accept a source and
replace/refresh only that source's entries (or clear existing entries for that
source before bulk-setting) so stale or missing upstream context_length values
from one source don't affect others.
- Around line 22-34: The validation currently checks raw names but not the
normalized result, allowing whitespace-only names to become an empty key; update
setModelContextLength to compute const key = normalizeKey(modelName) and only
call cache.set(key, ...) if key is non-empty (truthy) and contextLength is
valid, and similarly update setModelContextLengths to compute const key =
normalizeKey(name) inside the loop and only write cache.set(key, ...) when key
is non-empty and length is a finite positive number.
In `@src/web/pages/Accounts.tsx`:
- Around line 3480-3486: The async handler onRemoveManualModel can reopen the
modal after the user has closed or changed it; to fix, capture the current modal
state/account (e.g., const currentAccountId = modelModal?.account?.id and const
wasOpen = modelModal?.open) before awaiting api.removeAccountManualModels, then
after the await verify that modelModal is still open and modelModal.account.id
=== currentAccountId (or wasOpen is true) before calling
loadModelModalModels(modelModal.account, {}); return early if the modal was
closed or the account changed to avoid re-opening stale UI.
---
Nitpick comments:
In `@src/server/services/modelContextLengthCache.test.ts`:
- Around line 33-45: Add tests that assert whitespace-only model names are
treated as invalid: update the 'ignores invalid values' case to call
setModelContextLength(' ', 128000) and expect hasModelContextLength(' ')
toBe(false), and likewise add the same whitespace-only check in the other
invalid-name test block (the one around lines 67-78). This ensures
setModelContextLength and hasModelContextLength both trim/validate names (or
reject whitespace-only keys) consistent with the empty-string behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 2db3f102-e339-4deb-ac30-214a3b75022e
📒 Files selected for processing (9)
src/server/proxy-core/surfaces/modelsSurface.tssrc/server/routes/api/accounts.tssrc/server/services/modelContextLengthCache.test.tssrc/server/services/modelContextLengthCache.tssrc/server/services/platforms/newApi.tssrc/server/services/platforms/standardApiProvider.tssrc/web/api.tssrc/web/pages/Accounts.tsxsrc/web/pages/accounts/AccountModelsModal.tsx
Comment on lines
+1969
to
+1983
| try { | ||
| for (const modelName of normalizedModels) { | ||
| await db | ||
| .delete(schema.modelAvailability) | ||
| .where( | ||
| and( | ||
| eq(schema.modelAvailability.accountId, accountId), | ||
| eq(schema.modelAvailability.modelName, modelName), | ||
| eq(schema.modelAvailability.isManual, true), | ||
| ), | ||
| ) | ||
| .run(); | ||
| } | ||
| await rebuildRoutesBestEffort(); | ||
|
|
There was a problem hiding this comment.
Extract deletion workflow to a service and execute it atomically.
Line 1970-Line 1981 performs DB mutation orchestration directly in the route, and a mid-loop failure can leave partial deletions while returning an error.
Proposed refactor direction
- try {
- for (const modelName of normalizedModels) {
- await db
- .delete(schema.modelAvailability)
- .where(
- and(
- eq(schema.modelAvailability.accountId, accountId),
- eq(schema.modelAvailability.modelName, modelName),
- eq(schema.modelAvailability.isManual, true),
- ),
- )
- .run();
- }
+ try {
+ await removeManualModelsFromAccount({
+ accountId,
+ modelNames: normalizedModels,
+ });
await rebuildRoutesBestEffort();
return { success: true };
} catch (err: any) {// src/server/services/accountManualModelsService.ts
export async function removeManualModelsFromAccount(input: {
accountId: number;
modelNames: string[];
}) {
await db.transaction(async (tx) => {
for (const modelName of input.modelNames) {
await tx
.delete(schema.modelAvailability)
.where(
and(
eq(schema.modelAvailability.accountId, input.accountId),
eq(schema.modelAvailability.modelName, modelName),
eq(schema.modelAvailability.isManual, true),
),
)
.run();
}
});
}As per coding guidelines: "Route files in src/server/routes/** are adapters, not owners... must not own ... persistence."
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/server/routes/api/accounts.ts` around lines 1969 - 1983, Extract the
deletion loop into a new service function (e.g., removeManualModelsFromAccount
in src/server/services/accountManualModelsService.ts) that accepts { accountId,
modelNames } and runs the deletes inside a single db.transaction using the
transaction handle (tx) for the .delete(schema.modelAvailability) calls and the
same where(...) predicate (use input.accountId and each modelName); then replace
the loop in the route with a call to this service and only call
rebuildRoutesBestEffort() after the service resolves successfully; ensure errors
are propagated so the route can return the appropriate error response.
Comment on lines
+13
to
+43
| const cache = new Map<string, number>(); | ||
|
|
||
| function normalizeKey(modelName: string): string { | ||
| return modelName.trim().toLowerCase(); | ||
| } | ||
|
|
||
| /** | ||
| * Store context length for a single model. | ||
| */ | ||
| export function setModelContextLength(modelName: string, contextLength: number): void { | ||
| if (!modelName || !Number.isFinite(contextLength) || contextLength <= 0) return; | ||
| cache.set(normalizeKey(modelName), Math.round(contextLength)); | ||
| } | ||
|
|
||
| /** | ||
| * Bulk-store context lengths from a map (e.g. extracted from upstream payload). | ||
| */ | ||
| export function setModelContextLengths(entries: Map<string, number>): void { | ||
| for (const [name, length] of entries) { | ||
| if (name && Number.isFinite(length) && length > 0) { | ||
| cache.set(normalizeKey(name), Math.round(length)); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Get context length for a model. Returns the default if not found. | ||
| */ | ||
| export function getModelContextLength(modelName: string): number { | ||
| return cache.get(normalizeKey(modelName)) ?? DEFAULT_CONTEXT_LENGTH; | ||
| } |
There was a problem hiding this comment.
Scope the cache by source to prevent wrong context_length values.
At Line 13 the cache is process-global and keyed only by model name. If different accounts/providers share model IDs, later fetches can overwrite earlier values; stale values can also linger when upstream omits context fields later. This can return incorrect context_length downstream.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/server/services/modelContextLengthCache.ts` around lines 13 - 43, The
cache is currently process-global and keyed only by model name, causing
cross-source collisions; change the cache scheme to be scoped by source by
updating the cache variable and normalizeKey to incorporate a source identifier
(e.g., normalizeKey(source, modelName) or use a compound key
`${source}:${modelName}`), update setModelContextLength(source, modelName,
contextLength) and getModelContextLength(source, modelName) signatures to use
the scoped key, and modify setModelContextLengths to accept a source and
replace/refresh only that source's entries (or clear existing entries for that
source before bulk-setting) so stale or missing upstream context_length values
from one source don't affect others.
Comment on lines
+22
to
+34
| export function setModelContextLength(modelName: string, contextLength: number): void { | ||
| if (!modelName || !Number.isFinite(contextLength) || contextLength <= 0) return; | ||
| cache.set(normalizeKey(modelName), Math.round(contextLength)); | ||
| } | ||
|
|
||
| /** | ||
| * Bulk-store context lengths from a map (e.g. extracted from upstream payload). | ||
| */ | ||
| export function setModelContextLengths(entries: Map<string, number>): void { | ||
| for (const [name, length] of entries) { | ||
| if (name && Number.isFinite(length) && length > 0) { | ||
| cache.set(normalizeKey(name), Math.round(length)); | ||
| } |
There was a problem hiding this comment.
Reject whitespace-only model names after normalization.
At Line 23 and Line 32, ' ' passes validation and gets stored under an empty normalized key. Validate the normalized key before writing.
💡 Suggested fix
function normalizeKey(modelName: string): string {
return modelName.trim().toLowerCase();
}
export function setModelContextLength(modelName: string, contextLength: number): void {
- if (!modelName || !Number.isFinite(contextLength) || contextLength <= 0) return;
- cache.set(normalizeKey(modelName), Math.round(contextLength));
+ const key = normalizeKey(modelName);
+ if (!key || !Number.isFinite(contextLength) || contextLength <= 0) return;
+ cache.set(key, Math.round(contextLength));
}
export function setModelContextLengths(entries: Map<string, number>): void {
for (const [name, length] of entries) {
- if (name && Number.isFinite(length) && length > 0) {
- cache.set(normalizeKey(name), Math.round(length));
+ const key = normalizeKey(name);
+ if (key && Number.isFinite(length) && length > 0) {
+ cache.set(key, Math.round(length));
}
}
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/server/services/modelContextLengthCache.ts` around lines 22 - 34, The
validation currently checks raw names but not the normalized result, allowing
whitespace-only names to become an empty key; update setModelContextLength to
compute const key = normalizeKey(modelName) and only call cache.set(key, ...) if
key is non-empty (truthy) and contextLength is valid, and similarly update
setModelContextLengths to compute const key = normalizeKey(name) inside the loop
and only write cache.set(key, ...) when key is non-empty and length is a finite
positive number.
Comment on lines
+3480
to
+3486
| onRemoveManualModel={async (modelName) => { | ||
| if (!modelModal.account) return; | ||
| try { | ||
| await api.removeAccountManualModels(modelModal.account.id, [modelName]); | ||
| toast.success(`已删除模型 ${modelName}`); | ||
| await loadModelModalModels(modelModal.account, {}); | ||
| } catch (err: any) { |
There was a problem hiding this comment.
Guard against stale async completion before reloading the modal.
Line 3485 can reopen the modal after the user already closed it (request finishes late, loadModelModalModels forces open: true).
Suggested fix
onRemoveManualModel={async (modelName) => {
- if (!modelModal.account) return;
+ const targetAccount = modelModal.account;
+ if (!targetAccount) return;
+ const requestSeq = modelModalRequestSeqRef.current;
try {
- await api.removeAccountManualModels(modelModal.account.id, [modelName]);
+ await api.removeAccountManualModels(targetAccount.id, [modelName]);
toast.success(`已删除模型 ${modelName}`);
- await loadModelModalModels(modelModal.account, {});
+ if (modelModalRequestSeqRef.current !== requestSeq) return;
+ await loadModelModalModels(targetAccount, {});
} catch (err: any) {
toast.error(err?.message || "删除失败");
}
}}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| onRemoveManualModel={async (modelName) => { | |
| if (!modelModal.account) return; | |
| try { | |
| await api.removeAccountManualModels(modelModal.account.id, [modelName]); | |
| toast.success(`已删除模型 ${modelName}`); | |
| await loadModelModalModels(modelModal.account, {}); | |
| } catch (err: any) { | |
| onRemoveManualModel={async (modelName) => { | |
| const targetAccount = modelModal.account; | |
| if (!targetAccount) return; | |
| const requestSeq = modelModalRequestSeqRef.current; | |
| try { | |
| await api.removeAccountManualModels(targetAccount.id, [modelName]); | |
| toast.success(`已删除模型 ${modelName}`); | |
| if (modelModalRequestSeqRef.current !== requestSeq) return; | |
| await loadModelModalModels(targetAccount, {}); | |
| } catch (err: any) { | |
| toast.error(err?.message || "删除失败"); | |
| } | |
| }} |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/web/pages/Accounts.tsx` around lines 3480 - 3486, The async handler
onRemoveManualModel can reopen the modal after the user has closed or changed
it; to fix, capture the current modal state/account (e.g., const
currentAccountId = modelModal?.account?.id and const wasOpen = modelModal?.open)
before awaiting api.removeAccountManualModels, then after the await verify that
modelModal is still open and modelModal.account.id === currentAccountId (or
wasOpen is true) before calling loadModelModalModels(modelModal.account, {});
return early if the modal was closed or the account changed to avoid re-opening
stale UI.
Owner
|
请处理coderabbit ai给出的comments和CI错误 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title
Add model context length and manual model deletion
Body
Summary
context_lengthto/v1/modelsresponsesTest plan
/v1/modelsincludescontext_lengthSummary by CodeRabbit
Release Notes