Auto_load_model ignores ctx_size in chat request body

[indistinctTalk]

Auto-load ignores ctx_size in chat request body

Platform: Linux/Fedora (Bazzite)
Lemonade Version: 10.10.0
GPU / APU Model: AMD RYZEN AI MAX+ 395 (AMD Radeon 8060S Graphics, Integrated)
Component: lemond (server)

---

Bug Description

When the server auto-loads a model via the /v1/chat/completions endpoint, it ignores the ctx_size parameter in the request body. The model is loaded using recipe_options.json defaults, regardless of what ctx_size the client specifies in the chat request.

This causes failures for clients with large system prompts that exceed the auto-loaded context size. The client explicitly requests ctx_size: 128000 in its request body, but the model auto-loads at 4096, causing the request to be rejected with a context size exceeded error.

The /v1/load endpoint correctly accepts ctx_size as a request parameter and can override the default. However, the chat completions endpoint's auto-load path does not forward request body parameters to the loading phase.

Steps to Reproduce

1. Start Lemonade server with a model (e.g., Qwen3-Coder-Next-GGUF-Q4_K_M) that has a small default ctx_size (4096) in its recipe_options.json
2. Ensure the model is NOT pre-loaded (unload it or start fresh):

   curl -X POST http://localhost:13305/v1/unload \
     -d '{"model_name": "Qwen3-Coder-Next-GGUF-Q4_K_M"}'

3. Send a POST to /v1/chat/completions with "ctx_size": 128000 in the request body:

   curl -X POST http://localhost:13305/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "Qwen3-Coder-Next-GGUF-Q4_K_M",
       "messages": [{"role": "user", "content": "Hello"}],
       "max_tokens": 10,
       "ctx_size": 128000
     }'

4. Check the loaded model's context size:

   curl http://localhost:13305/v1/models/Qwen3-Coder-Next-GGUF-Q4_K_M | jq '.recipe_options.ctx_size'

5. Send a request with prompt_tokens > 4096 (e.g., a large system prompt) — it fails with context size exceeded

Expected vs Actual Behavior

Expected: The model should be auto-loaded with ctx_size: 128000 as specified in the chat request body, matching the behavior of explicitly calling /v1/load with ctx_size: 128000.

Actual: The model is auto-loaded with ctx_size: 4096 (the recipe_options.json default). The ctx_size parameter in the chat request body is not forwarded to the model loading phase.

Log Output

Debug logging enabled (lemonade config set log_level=debug). Logs from the auto-load at ctx_size=4096 followed by the request rejection:

2026-07-11 13:31:35.407 [Info] (Server) Auto-loading model: Qwen3-Coder-Next-GGUF-Q4_K_M
2026-07-11 13:31:35.407 [Debug] (Router) Loading model: user.Qwen3-Coder-Next-GGUF-Q4_K_M
2026-07-11 13:31:35.407 [Debug] (Router) Effective settings: ctx_size=4096, llamacpp_backend=vulkan, ...
2026-07-11 13:31:36.183 [Debug] (ProcessManager) Starting process: llama-server ... --ctx-size 4096 --port 8001 ...
2026-07-11 13:31:53.829 [Info] (Server) Model loaded successfully: Qwen3-Coder-Next-GGUF-Q4_K_M
2026-07-11 13:31:53.892 [Info] (Process) cache state: 0 prompts (limits: 8192.000 MiB, 4096 tokens)
2026-07-11 13:31:53.892 [Error] (Process) E srv send_error: task id = 0, error: request (8601 tokens) exceeds the available context size (4096 tokens), try increasing it
2026-07-11 13:31:53.893 [Error] (StreamingProxy) Backend returned error: 400

Key observation: The model auto-loaded with --ctx-size 4096 from recipe_options.json, then rejected an 8601-token request with the exact error: "request (8601 tokens) exceeds the available context size (4096 tokens)". The client had sent ctx_size: 128000 in the request body, but it was ignored during auto-load.

Additional Context

Source Code Analysis

The root cause is in auto_load_model_if_needed() in src/cpp/server/server.cpp (line ~1855). It calls:

router_->load_model(requested_model, info, RecipeOptions(info.recipe, json::object()), true);

The json::object() passes an empty options object to RecipeOptions, completely ignoring all parameters from the request body (including ctx_size). This is hardcoded and does not accept per-request overrides.

Compare with the /v1/load endpoint, which reads ctx_size from the request body and passes it correctly:

// /v1/load path:
auto options = RecipeOptions(info.recipe, request_body); // ← reads request body
router_->load_model(model_name, info, options, true);

Workaround

Users can pre-configure the model's ctx_size by loading it with save_options: true:

curl -X POST http://localhost:13305/v1/load \
  -d '{"model_name": "Qwen3-Coder-Next-GGUF-Q4_K_M", "ctx_size": 128000, "save_options": true}'

This is a manual step that shouldn't be necessary. The workaround persists the ctx_size to recipe_options.json, which then applies to all future auto-loads.

Affected Clients

Any client that:

• Uses the OpenAI-compatible /v1/chat/completions endpoint
• Sends a ctx_size parameter in the request body
• Relies on server-side model auto-load (no pre-loaded model)
• Has a system prompt that exceeds the default ctx_size (typically 4096)

Known affected client: pi (pi-coding-agent), which sends a ~8600 token system prompt + user message. When auto-loaded at ctx_size: 4096, the request fails, causing the client to see a "Stream ended without finish_reason" error with zero output tokens.

Testing Evidence

• curl + ctx_size: 128000 (model pre-loaded at correct size): Works perfectly — response returned with correct tokens.
• curl + ctx_size: 128000 (model auto-loaded at 4096): Model loads at 4096, large requests fail.
• pi (via OpenAI SDK) + ctx_size: 128000 (model auto-loaded at 4096): Fails with stopReason: error, input: 0, output: 0 — the stream returns zero chunks because the backend rejects the oversized prompt.
• pi + server-side recipe_options.json set to ctx_size: 128000: Works correctly — auto-load uses the persisted size.

[indistinctTalk]

So the agent fired off this bug before I had asked it to - we were preparing the draft, but I believe the issue reported is valid.

[bong-water-water-bong]

Fixed in PR #2673.

The root cause was in auto_load_model_if_needed() which hardcoded json::object() as the options to RecipeOptions, ignoring all request body parameters. The fix adds an optional request_options parameter to the function and threads request_json through from all call sites that have it available.

Also fixed OllamaApi::auto_load_model() for the same issue.