Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions server/src/__tests__/db/ollama-410-v19.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import { describe, it, expect, beforeAll } from 'vitest';
import { initDb, getDb } from '../../db/index.js';

// Live-verified 2026-07-22: Ollama Cloud answers `410 Gone` for
// qwen3-coder:480b — the hosted model was removed upstream. With
// intelligence_rank 2 the row sat at the top of the auto-route chain, so
// every `model: "auto"` request died on it before the 410 classifier fix.
const RETIRED_OLLAMA_MODELS = ['qwen3-coder:480b'];

// Probed the same day through /v1/chat/completions: these still answer 200.
const LIVE_OLLAMA_MODELS = ['gemma4:31b', 'gpt-oss:120b', 'gpt-oss:20b'];

describe('V19 Ollama Cloud retirement', () => {
beforeAll(() => {
process.env.ENCRYPTION_KEY = '0'.repeat(64);
initDb(':memory:');
});
Comment on lines +14 to +17

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Files matching ollama test / db init:"
fd -i 'ollama-410-v19\.test\.ts|index\.ts$' server/src 2>/dev/null || true

echo
echo "Target test excerpt:"
if [ -f server/src/__tests__/db/ollama-410-v19.test.ts ]; then
  nl -ba server/src/__tests__/db/ollama-410-v19.test.ts | sed -n '1,80p'
fi

echo
echo "Find initDb declarations/usages:"
rg -n "function initDb|const initDb|initDb\(" server/src --glob '*.ts' || true

echo
echo "Relevant db index excerpts:"
for f in $(fd -i 'index\.ts$' server/src/db 2>/dev/null); do
  echo "--- $f"
  nl -ba "$f" | sed -n '1,220p'
done

echo
echo "Git status/diff stat:"
git diff --stat || true
git diff -- server/src/__tests__/db/ollama-410-v19.test.ts server/src/db/index.ts | sed -n '1,220p' || true

Repository: aliihsaad/LLM_HUB

Length of output: 376


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Target test excerpt:"
if [ -f server/src/__tests__/db/ollama-410-v19.test.ts ]; then
  awk '{printf "%6d\t%s\n", NR, $0}' server/src/__tests__/db/ollama-19.test.ts 2>/dev/null | sed -n '1,120p' || true
  awk '{printf "%6d\t%s\n", NR, $0}' server/src/__tests__/db/ollama-410-v19.test.ts 2>/dev/null | sed -n '1,140p' || true
fi

echo
echo "Find initDb declarations/usages:"
rg -n "function initDb|const initDb|initDb\(" server/src --glob '*.ts' || true

echo
echo "Relevant db index excerpts:"
if [ -f server/src/db/index.ts ]; then
  awk '{printf "%6d\t%s\n", NR, $0}' server/src/db/index.ts | sed -n '1,260p'
fi

echo
echo "Relevant server index excerpts around db initialization imports:"
if [ -f server/src/index.ts ]; then
  awk '{printf "%6d\t%s\n", NR, $0}' server/src/index.ts | sed -n '1,220p'
fi

echo
echo "Git diff stat:"
git diff --stat || true
git diff -- server/src/__tests__/db/ollama-410-v19.test.ts server/src/db/index.ts server/src/index.ts | sed -n '1,260p' || true

Repository: aliihsaad/LLM_HUB

Length of output: 19783


Await the asynchronous database initialization.

initDb returns a promise, but this synchronous beforeAll hook does not wait for it. The current setup may run the synchronous seed/migrate phase before the first async import, while later initialization and failures can still occur outside Vitest’s setup lifecycle.

Proposed fix
-  beforeAll(() => {
+  beforeAll(async () => {
     process.env.ENCRYPTION_KEY = '0'.repeat(64);
-    initDb(':memory:');
+    await initDb(':memory:');
   });
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
beforeAll(() => {
process.env.ENCRYPTION_KEY = '0'.repeat(64);
initDb(':memory:');
});
beforeAll(async () => {
process.env.ENCRYPTION_KEY = '0'.repeat(64);
await initDb(':memory:');
});
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server/src/__tests__/db/ollama-410-v19.test.ts` around lines 14 - 17, Update
the beforeAll hook around initDb to be asynchronous and await its returned
promise, ensuring database initialization and any failures complete within
Vitest’s setup lifecycle before tests run.


it('disables the Ollama Cloud rows the upstream deleted (410 Gone)', () => {
const db = getDb();
for (const modelId of RETIRED_OLLAMA_MODELS) {
const row = db
.prepare("SELECT enabled FROM models WHERE platform = 'ollama' AND model_id = ?")
.get(modelId) as { enabled: number } | undefined;
expect(row, `${modelId} should exist in the catalog`).toBeDefined();
expect(row!.enabled, `${modelId} must be disabled — upstream returns 410`).toBe(0);
}
});

it('keeps the still-hosted Ollama Cloud models enabled', () => {
const db = getDb();
for (const modelId of LIVE_OLLAMA_MODELS) {
const row = db
.prepare("SELECT enabled FROM models WHERE platform = 'ollama' AND model_id = ?")
.get(modelId) as { enabled: number } | undefined;
expect(row, `${modelId} should exist in the catalog`).toBeDefined();
expect(row!.enabled, `${modelId} should stay enabled`).toBe(1);
}
});
});
38 changes: 38 additions & 0 deletions server/src/__tests__/providers/google.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -556,6 +556,44 @@ describe('GoogleProvider', () => {
expect(result._routed_via).toEqual({ platform: 'google', model: 'gemini-3.1-flash-image' });
});

it('should transcribe uploaded audio through Gemini and return OpenAI-compatible JSON', async () => {
let capturedBody: any;
vi.spyOn(global, 'fetch').mockImplementation(async (_url, init) => {
capturedBody = JSON.parse((init as any).body);
return {
ok: true,
json: () => Promise.resolve({
candidates: [{
content: { parts: [{ text: 'Turn on the office light.' }] },
finishReason: 'STOP',
}],
}),
} as any;
});

const audio = Buffer.from('fake-wave');
const result = await provider.transcribeAudio(
'test-key',
{
file: {
filename: 'utterance.wav',
contentType: 'audio/wav',
data: audio,
},
response_format: 'json',
},
'gemini-2.5-flash',
);

expect(capturedBody.contents[0].parts[0].text).toContain('Transcribe');
expect(capturedBody.contents[0].parts[1]).toEqual({
inlineData: { mimeType: 'audio/wav', data: audio.toString('base64') },
});
expect(result.body).toEqual({ text: 'Turn on the office light.' });
expect(result.contentType).toBe('application/json');
expect(result._routed_via).toEqual({ platform: 'google', model: 'gemini-2.5-flash' });
});

it('should generate speech and return WAV audio bytes', async () => {
let capturedBody: any;
vi.spyOn(global, 'fetch').mockImplementation(async (_url, init) => {
Expand Down
30 changes: 30 additions & 0 deletions server/src/__tests__/services/provider-errors.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,36 @@ import { describe, expect, it } from 'vitest';
import { canRetryProviderFailure, classifyProviderError } from '../../services/provider-errors.js';

describe('provider error classification', () => {
it('rotates to another key when a provider organization is restricted', () => {
const failure = classifyProviderError(
new Error('Groq API error 400: Organization has been restricted. Please reach out to support.'),
);

expect(failure).toMatchObject({
category: 'auth',
retryable: true,
skipModel: false,
});
expect(failure.keyCooldownMs).toBeGreaterThan(0);
expect(canRetryProviderFailure(failure, 'whisper-large-v3-turbo')).toBe(true);
});

it('treats HTTP 410 Gone as a model-level unavailable error so routing moves on', () => {
const failure = classifyProviderError(
new Error('Ollama Cloud API error 410: Gone'),
);

expect(failure).toMatchObject({
category: 'model_unavailable',
retryable: true,
skipModel: true,
});
// auto-route: skip the retired model and continue down the chain
expect(canRetryProviderFailure(failure)).toBe(true);
// pinned request: fail honestly instead of silently switching models
expect(canRetryProviderFailure(failure, 'qwen3-coder:480b')).toBe(false);
});

it('treats provider terms acceptance gates as model-level unavailable errors', () => {
const failure = classifyProviderError(
new Error('Groq API error 400: The model `canopylabs/orpheus-v1-english` requires terms acceptance. Please have the org admin accept the terms.'),
Expand Down
18 changes: 18 additions & 0 deletions server/src/db/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ export async function initDb(dbPath?: string): Promise<Database.Database> {
migrateModelsV16(db);
migrateModelsV17(db);
migrateModelsV18(db);
migrateModelsV19(db);
seedModelCapabilities(db);
// Must follow seedModelCapabilities — that is where the image rows are seeded.
flagPaidGoogleModels(db);
Expand Down Expand Up @@ -1146,6 +1147,7 @@ function seedModelCapabilities(db: Database.Database) {
['google', 'gemini-2.5-flash-native-audio-preview-12-2025', 'realtime_audio', 1],
['google', 'gemini-3.1-flash-live-preview', 'realtime_audio', 2],
['groq', 'whisper-large-v3-turbo', 'transcription', 1],
['google', 'gemini-2.5-flash', 'transcription', 2],
['groq', 'whisper-large-v3-turbo', 'translation', 1],
['groq', 'whisper-large-v3', 'transcription', 2],
['groq', 'whisper-large-v3', 'translation', 2],
Expand Down Expand Up @@ -1695,6 +1697,22 @@ function migrateModelsV18(db: Database.Database) {
disable.run();
}

/**
* V19 (July 2026): retire the Ollama Cloud rows the upstream deleted.
*
* Ollama Cloud answers `410 Gone` for qwen3-coder:480b — the hosted model was
* removed outright (live-verified 2026-07-22). With intelligence_rank 2 the
* row sat at the top of the auto-route chain, so before the 410 classifier
* fix every `model: "auto"` request died on it with a 502. Disabled, never
* deleted, per V18 precedent; a future migration can re-enable it if Ollama
* brings the model back.
*/
function migrateModelsV19(db: Database.Database) {
db.prepare(
"UPDATE models SET enabled = 0 WHERE platform = 'ollama' AND model_id = 'qwen3-coder:480b'",
).run();
}

/**
* Mark Google models that have no free tier (verified 2026-07-22 against
* ai.google.dev pricing — image generation reads "Not available" in the Free
Expand Down
7 changes: 4 additions & 3 deletions server/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,15 @@ import { startHealthChecker } from './services/health.js';
import { startModelScout } from './services/model-scout.js';

const PORT = process.env.PORT ?? 3001;
const HOST = process.env.HOST ?? '127.0.0.1';

async function main() {
await initDb();
const app = createApp();

app.listen(Number(PORT), '0.0.0.0', () => {
console.log(`Server running on http://0.0.0.0:${PORT}`);
console.log(`Proxy endpoint: http://0.0.0.0:${PORT}/v1/chat/completions`);
app.listen(Number(PORT), HOST, () => {
console.log(`Server running on http://${HOST}:${PORT}`);
console.log(`Proxy endpoint: http://${HOST}:${PORT}/v1/chat/completions`);
Comment on lines +15 to +16

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Do not advertise every bind host as a directly usable URL.

For HOST=0.0.0.0 or HOST=::, the logged URL is not a reliable client destination; IPv6 literals also require brackets. Use a separate, normalized display host and avoid emitting a clickable endpoint for wildcard binds.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server/src/index.ts` around lines 15 - 16, Update the startup logging around
the server URL messages to derive a normalized display host separate from the
bind HOST: bracket IPv6 literals when they are usable destinations, and for
wildcard binds such as 0.0.0.0 or :: avoid emitting a clickable proxy endpoint
URL. Preserve the server-running message while ensuring it does not advertise
wildcard addresses as client destinations.

startHealthChecker();
startModelScout();
});
Expand Down
58 changes: 58 additions & 0 deletions server/src/providers/google.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ import type {
ChatToolCall,
ChatToolChoice,
ChatToolDefinition,
AudioTextResult,
AudioTranscriptionRequest,
ImageEditRequest,
ImageFileUpload,
ImageGenerationRequest,
Expand Down Expand Up @@ -715,6 +717,62 @@ export class GoogleProvider extends BaseProvider {
};
}

async transcribeAudio(
apiKey: string,
request: AudioTranscriptionRequest,
modelId: string,
): Promise<AudioTextResult> {
if (!request.file) {
throw new Error('Google transcription requires an uploaded audio file');
}

const instructions = [
'Transcribe the supplied audio exactly.',
'Return only the transcript without commentary, labels, quotation marks, or Markdown.',
request.language ? `The expected language code is ${request.language}.` : '',
request.prompt ? `Context that may help disambiguate names: ${request.prompt}` : '',
].filter(Boolean).join(' ');
const body = {
contents: [{
parts: [
{ text: instructions },
{
inlineData: {
mimeType: request.file.contentType || 'audio/wav',
data: Buffer.from(request.file.data).toString('base64'),
},
},
],
}],
generationConfig: { temperature: 0 },
};

const url = `${API_BASE}/models/${modelId}:generateContent?key=${apiKey}`;
const res = await this.fetchWithTimeout(url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body),
}, 120000);

if (!res.ok) {
const err = await res.json().catch(() => ({}));
throw new Error(`Google API error ${res.status}: ${(err as any).error?.message ?? res.statusText}`);
}

const data = await res.json() as GeminiResponse;
const transcript = extractText(data.candidates?.[0]?.content?.parts)?.trim();
if (!transcript) {
throw new Error('Google transcription returned no text');
}

const wantsText = request.response_format === 'text';
return {
body: wantsText ? transcript : { text: transcript },
contentType: wantsText ? 'text/plain; charset=utf-8' : 'application/json',
Comment on lines +762 to +771

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Do not return truncated Gemini output as a successful transcript.

finishReason can be MAX_TOKENS, but this method returns any extracted text as success. Long audio can therefore produce a partial transcript with HTTP 200. Check candidate.finishReason before returning; the API documents MAX_TOKENS as output truncation. (ai.google.dev)

Proposed guard
     const data = await res.json() as GeminiResponse;
-    const transcript = extractText(data.candidates?.[0]?.content?.parts)?.trim();
+    const candidate = data.candidates?.[0];
+    if (candidate?.finishReason && candidate.finishReason !== 'STOP') {
+      throw new Error(`Google transcription stopped with ${candidate.finishReason}`);
+    }
+    const transcript = extractText(candidate?.content?.parts)?.trim();
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const data = await res.json() as GeminiResponse;
const transcript = extractText(data.candidates?.[0]?.content?.parts)?.trim();
if (!transcript) {
throw new Error('Google transcription returned no text');
}
const wantsText = request.response_format === 'text';
return {
body: wantsText ? transcript : { text: transcript },
contentType: wantsText ? 'text/plain; charset=utf-8' : 'application/json',
const data = await res.json() as GeminiResponse;
const candidate = data.candidates?.[0];
if (candidate?.finishReason && candidate.finishReason !== 'STOP') {
throw new Error(`Google transcription stopped with ${candidate.finishReason}`);
}
const transcript = extractText(candidate?.content?.parts)?.trim();
if (!transcript) {
throw new Error('Google transcription returned no text');
}
const wantsText = request.response_format === 'text';
return {
body: wantsText ? transcript : { text: transcript },
contentType: wantsText ? 'text/plain; charset=utf-8' : 'application/json',
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server/src/providers/google.ts` around lines 762 - 771, Update the
transcription response handling around GeminiResponse and the extracted
transcript to inspect candidates[0].finishReason before returning success. Treat
MAX_TOKENS as an error by throwing instead of returning the partial transcript,
while preserving the existing no-text validation and successful response
formatting for other finish reasons.

Source: MCP tools

_routed_via: { platform: 'google', model: modelId },
};
}

async createSpeech(
apiKey: string,
request: SpeechRequest,
Expand Down
8 changes: 8 additions & 0 deletions server/src/services/provider-errors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,13 @@ export function classifyProviderError(err: unknown): ClassifiedProviderError {
return { category: 'zero_quota', retryable: true, skipModel: true, keyCooldownMs: 0 };
}

if (
msg.includes('organization has been restricted')
|| msg.includes('organization is restricted')
) {
return { category: 'auth', retryable: true, skipModel: false, keyCooldownMs: 24 * 60 * 60 * 1000 };
}

if (
msg.includes('401')
|| msg.includes('unauthorized')
Expand All @@ -29,6 +36,7 @@ export function classifyProviderError(err: unknown): ClassifiedProviderError {

if (
msg.includes('404')
|| msg.includes('410')
|| msg.includes('not found')
|| msg.includes('model does not exist')
|| msg.includes('unavailable_model')
Expand Down
Loading