Skip to content

refactor(provider): enable maxInputTokens to be set by context length on Lemonade Server#14

Open
matouka wants to merge 2 commits into
lemonade-sdk:mainfrom
matouka:feature/change-input-max-token-based-on-recipe
Open

refactor(provider): enable maxInputTokens to be set by context length on Lemonade Server#14
matouka wants to merge 2 commits into
lemonade-sdk:mainfrom
matouka:feature/change-input-max-token-based-on-recipe

Conversation

@matouka

@matouka matouka commented May 16, 2026

Copy link
Copy Markdown

Sorry, new here to using Lemonade server. I'm not 100% sure if this is valid, or if it's just my setting. When I set my context explicitly on Lemonade Server, it's returned something like this from the models endpoint:

{
  "data": [
    {
      "checkpoint": "unsloth/Qwen3.6-35B-A3B-GGUF:Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf",
      "checkpoints": {
        "main": "unsloth/Qwen3.6-35B-A3B-GGUF:Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf",
        "mmproj": "unsloth/Qwen3.6-35B-A3B-GGUF:mmproj-F16.gguf"
      },
      "created": 1234567890,
      "id": "Qwen3.6-35B-A3B-GGUF",
      "labels": [
        "vision",
        "tool-calling",
        "hot"
      ],
      "object": "model",
      "owned_by": "lemonade",
      "recipe": "llamacpp",
      "recipe_options": {
        "ctx_size": 200000
      },
      "size": 22.4,
      "suggested": true
    },
  ],
  "object": "list"
}

I removed some fields in the sample

So I thought why not at least attempt to pull in the allowable input tokens rather than having to set one single value, then used for all models, through an env setting.

Also, judging by the object returned, it looked like you should also be able to update the imageInput field using the labels. However, when I manually set it to true, the model didn't seem to make any use of it, so I haven't touched that.

@matouka matouka changed the title refactor(provider): to allow for the maxInputTokens to be set by the … refactor(provider): enable maxInputTokens to be set by context length on Lemonade Server May 16, 2026

@danielholanda danielholanda left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR looks good!

@danielholanda danielholanda left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please wait for fl0rianr to review before merging

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants