fix(server): forward request body params (ctx_size etc.) during auto-load#2673
Closed
bong-water-water-bong wants to merge 9 commits into
Closed
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Auto-syncs every 5AM UTC via fast-forward merge. Manual trigger available via workflow_dispatch.
…on (lemonade-sdk#2579) Adds get_hf_endpoint() helper that reads the HF_ENDPOINT environment variable at runtime and falls back to https://huggingface.co when unset. Trailing slashes are automatically stripped. Replaces all 11 hardcoded 'https://huggingface.co' URL constructions across 6 source files with calls to get_hf_endpoint(): - model_manager.cpp (5 sites): API model lookups, tree listing, file URL construction - hf_variants.cpp (2 sites): variant enumeration, manifest fetching - vllm_server.cpp (1 site): config.json fetch for quant method detection - whispercpp_server.cpp (1 site): whisper model download - hf_pull.cpp (1 site + prefix stripping): CLI URL prefix normalization also recognizes the configured mirror endpoint Usage: HF_ENDPOINT=https://hf-mirror.com lemond docker run -e HF_ENDPOINT=https://hf-mirror.com ghcr.io/lemonade-sdk/lemonade-server:latest Fully backward compatible when HF_ENDPOINT is unset.
…mits The daily sync used git merge --ff-only which fails when the fork has local commits (e.g. HF_ENDPOINT, MLX backend). Changed to a regular merge with a descriptive message so the daily sync creates a merge commit that keeps both upstream and fork history intact.
Avoids unnecessary merge commits when the fork has no local changes. Only creates a merge commit when local commits (like PR branches on main) prevent a clean fast-forward.
…king Two bugs in the Anthropic /v1/messages endpoint that break Claude Code: 1. System role messages inside the messages array were silently dropped. Claude Code puts sub-agent definitions as role:system messages. When skipped, the model received a malformed prompt and returned empty responses. Fix: Collect system messages from both the top-level system field AND inline role:system messages in a first pass, then emit them as a single combined system message. 2. thinking.type: 'adaptive' was treated as unsupported. This is the standard Anthropic mode that lets the model decide when to use thinking. Treated the same as 'enabled'. Fixes lemonade-sdk#2662
The scheduled Publish Containers workflow pushes build environment container images to ghcr.io/lemonade-sdk/lemonade/build-environment. When running from a fork, the auto-generated GITHUB_TOKEN lacks write permission to the upstream org's package namespace, causing: denied: permission_denied: The requested installation does not exist. Fix: gate the workflow with 'if: github.repository == "lemonade-sdk/lemonade"' so it only executes in the upstream repo. Issue #4
The auto-label workflow uses ANTHROPIC_API_KEY for LLM-based classification, which is only available in the upstream repo. Forks trigger the workflow on PR open/edit but fail immediately because the secret isn't set. Gate with if: github.repository == 'lemonade-sdk/lemonade' to match the build-container.yml fix.
…load When the server auto-loads a model via /v1/chat/completions (or any API endpoint), it was passing an empty json::object() to RecipeOptions, ignoring all per-request parameters like ctx_size from the request body. This caused models to always load with their recipe_options.json defaults (e.g. ctx_size=4096), even when the client explicitly requested a larger context like ctx_size=128000. Requests exceeding the default context would fail with 'request (X tokens) exceeds the available context size'. The /v1/load endpoint already correctly passes the request body to RecipeOptions -- this fix brings auto-load in line with that behavior. Fixes: lemonade-sdk#2663
bong-water-water-bong
force-pushed
the
fix/2662-anthropic-system-role-thinking
branch
from
0338d8d to
78ec27c
Compare
github-actions
Bot
added
bug
Collaborator
|
Thank you for your effort and the time you invested in this pull request. However, I am closing this PR in favor of #2664 as we need to take a different approach for this issue. Here is a breakdown of why this specific patch is not the preferred solution:
Thank you again for your time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2663
Problem
When the server auto-loads a model via
/v1/chat/completions(or any API endpoint),auto_load_model_if_needed()was passingjson::object()toRecipeOptions, ignoring all per-request parameters from the request body — includingctx_size.This caused the model to load with
recipe_options.jsondefaults (e.g.ctx_size=4096), even when the client explicitly requestedctx_size=128000. Any request with a prompt larger than the default context size would fail with:Fix
auto_load_model_if_needed()to accept an optionalrequest_optionsparameter (defaults to empty JSON for backward compatibility).request_jsonto pass it through as request options.RecipeOptionsconstructor already safely extracts only recipe-relevant keys from the options JSON — matching the pattern already used by the/v1/loadendpoint.OllamaApi::auto_load_model()for consistency.Testing
Build succeeds cleanly. The change is purely additive (default parameter) — no existing call site changes behavior unless explicitly updated.