Skip to content

backends-reference.md

Latest commit

 

History

History
415 lines (357 loc) · 20.4 KB

backends-reference.md

File metadata and controls

415 lines (357 loc) · 20.4 KB
Raw

Backend reference

Backends

Recipe Name Selectable backend Uses ctx_size Backends
acestep ACE-Step yes no cuda, rocm, vulkan
flm FastFlowLM NPU no yes npu
kokoro Kokoro no no cpu, metal
llamacpp Llama.cpp GPU yes yes cpu, cuda, metal, rocm, system, vulkan
moonshine Moonshine no no cpu
openmoss OpenMOSS TTS yes no cuda, rocm, vulkan
ryzenai-llm Ryzen AI LLM no yes npu
sd-cpp StableDiffusion.cpp yes no cpu, cuda, metal, rocm, vulkan
thinksound ThinkSound yes no cuda, rocm, vulkan
trellis TRELLIS.2 yes no cuda, rocm, vulkan
vllm vLLM ROCm (experimental) yes yes rocm
whispercpp Whisper.cpp yes no cpu, metal, npu, rocm, vulkan

Support matrix

Recipe Backend OS Device families
acestep rocm linux, windows amd_gpu (gfx103X, gfx110X, gfx1150, gfx1151, gfx1152, gfx120X)
acestep cuda linux, windows nvidia_gpu
acestep vulkan linux, windows amd_gpu; cpu (x86_64); nvidia_gpu
flm npu linux, windows amd_npu (XDNA2)
kokoro cpu linux, windows cpu (x86_64)
kokoro metal macos metal
llamacpp system linux cpu (arm64, x86_64)
llamacpp metal macos metal
llamacpp cuda linux, windows nvidia_gpu (sm_100, sm_120, sm_121, sm_75, sm_80, sm_86, sm_89, sm_90)
llamacpp vulkan linux, windows amd_gpu; cpu (arm64, x86_64)
llamacpp rocm linux, windows amd_gpu (gfx103X, gfx110X, gfx1150, gfx1151, gfx1152, gfx120X, gfx942)
llamacpp cpu linux, windows cpu (arm64, x86_64)
moonshine cpu windows cpu (x86_64)
moonshine cpu linux cpu (arm64, x86_64)
moonshine cpu macos cpu (arm64)
openmoss rocm linux, windows amd_gpu
openmoss cuda linux, windows nvidia_gpu
openmoss vulkan linux, windows amd_gpu; cpu (x86_64); nvidia_gpu
ryzenai-llm npu windows amd_npu (XDNA2)
sd-cpp rocm linux, windows amd_gpu (gfx103X, gfx110X, gfx1150, gfx1151, gfx1152, gfx120X)
sd-cpp cuda linux, windows nvidia_gpu (sm_100, sm_120, sm_121, sm_75, sm_80, sm_86, sm_89, sm_90)
sd-cpp vulkan linux, windows amd_gpu; cpu (x86_64); nvidia_gpu
sd-cpp cpu linux, windows cpu (x86_64)
sd-cpp metal macos metal
thinksound rocm linux, windows amd_gpu (gfx103X, gfx110X, gfx1150, gfx1151, gfx1152, gfx120X)
thinksound cuda linux, windows nvidia_gpu
thinksound vulkan linux, windows amd_gpu; cpu (x86_64); nvidia_gpu
trellis rocm linux, windows amd_gpu (gfx103X, gfx110X, gfx1150, gfx1151, gfx1152, gfx120X)
trellis cuda linux, windows nvidia_gpu
trellis vulkan linux, windows amd_gpu; cpu (x86_64); nvidia_gpu
vllm rocm linux amd_gpu (gfx110X, gfx1150, gfx1151, gfx120X)
whispercpp npu windows amd_npu (XDNA2)
whispercpp rocm linux, windows amd_gpu (gfx110X, gfx1150, gfx1151, gfx120X)
whispercpp vulkan linux, windows amd_gpu; cpu (x86_64)
whispercpp cpu linux, windows cpu (x86_64)
whispercpp metal macos metal

Recipe options

acestep — ACE-Step

Option CLI flag Type Default Description
acestep_backend --acestep BACKEND "" ACE-Step backend to use

llamacpp — Llama.cpp GPU

Option CLI flag Type Default Description
ctx_size --ctx-size SIZE -1 Context size for the model
llamacpp_backend --llamacpp BACKEND "" LlamaCpp backend to use
llamacpp_device --llamacpp-device DEVICES "" Comma-separated list of accelerator devices to use (e.g. Vulkan0)
llamacpp_args --llamacpp-args ARGS "" Custom arguments to pass to llama-server

moonshine — Moonshine

Option CLI flag Type Default Description
moonshine_args --moonshine-args ARGS "" Custom arguments to pass to moonshine-server

openmoss — OpenMOSS TTS

Option CLI flag Type Default Description
openmoss_backend --openmoss BACKEND "" OpenMOSS TTS backend to use

sd-cpp — StableDiffusion.cpp

Option CLI flag Type Default Description
sd-cpp_backend --sdcpp BACKEND "" SD.cpp backend to use
sdcpp_args --sdcpp-args ARGS "" Custom arguments to pass to sd-server (must not conflict with managed args)
steps — SIZE 20 Number of diffusion steps
cfg_scale — SIZE 7.0 Classifier-free guidance scale
width — SIZE 512 Output image width
height — SIZE 512 Output image height
sampling_method — ARGS "" Sampling method
flow_shift — SIZE 0.0 Flow shift

thinksound — ThinkSound

Option CLI flag Type Default Description
thinksound_backend --thinksound BACKEND "" ThinkSound backend to use

trellis — TRELLIS.2

Option CLI flag Type Default Description
trellis_backend --trellis BACKEND "" Trellis backend to use

vllm — vLLM ROCm (experimental)

Option CLI flag Type Default Description
ctx_size --ctx-size SIZE -1 Context size for the model
vllm_backend --vllm BACKEND "" vLLM backend to use
vllm_args --vllm-args ARGS "" Custom arguments to pass to vllm-server

whispercpp — Whisper.cpp

Option CLI flag Type Default Description
whispercpp_backend --whispercpp BACKEND "" WhisperCpp backend to use
whispercpp_args --whispercpp-args ARGS "" Custom arguments to pass to whisper-server

Models

acestep — ACE-Step (1 models)

Model Size (GB) Labels
ACE-Step-Music 10.5 audio-generation

collection.omni — collection.omni (5 models)

Model Size (GB) Labels
LMX-Omni-5.5B-Lite 9.3 —
LMX-Omni-52B-Halo 44.77 —
Lite Collection —
RPG-HaloTales-V1 39.77 —
Ultra Collection —

kokoro — Kokoro (1 models)

Model Size (GB) Labels
kokoro-v1 0.354 tts

llamacpp — Llama.cpp GPU (77 models)

Model Size (GB) Labels
Bonsai-1.7B-gguf 0.25 llamacpp
Bonsai-4B-gguf 0.572 llamacpp
Bonsai-8B-gguf 1.16 llamacpp
Cogito-v2-llama-109B-MoE-GGUF 65.4 vision
DeepSeek-Qwen3-8B-GGUF 5.25 reasoning
Devstral-Small-2507-GGUF 14.3 coding, tool-calling
GLM-4.5-Air-UD-Q4K-XL-GGUF 67.7 reasoning
GLM-4.7-Flash-GGUF 17.5 tool-calling
Gemma-3-4b-it-GGUF 3.34 vision
Gemma-4-12B-it-GGUF 7.29 tool-calling, vision, llamacpp
Gemma-4-12B-it-MTP-GGUF 7.75 tool-calling, llamacpp, vision, mtp
Gemma-4-26B-A4B-it-GGUF 18.1 hot, tool-calling, vision, llamacpp
Gemma-4-26B-A4B-it-MTP-GGUF 18.5 hot, tool-calling, vision, llamacpp, mtp
Gemma-4-31B-it-GGUF 19.5 hot, tool-calling, vision, llamacpp
Gemma-4-31B-it-MTP-GGUF 20.0 hot, tool-calling, vision, llamacpp, mtp
Gemma-4-E2B-it-GGUF 4.09 tool-calling, vision, llamacpp
Gemma-4-E4B-it-GGUF 5.97 tool-calling, vision, llamacpp
Jan-nano-128k-GGUF 2.5 —
Jan-v1-4B-GGUF 2.5 —
LFM2-1.2B-GGUF 0.731 —
LFM2-24B-A2B-GGUF 14.4 —
LFM2-8B-A1B-GGUF 5.04 —
LFM2.5-1.2B-Instruct-GGUF 0.731 —
LFM2.5-8B-A1B 5.16 —
Llama-3.2-1B-Instruct-GGUF 0.834 —
Llama-3.2-3B-Instruct-GGUF 2.06 —
Llama-4-Scout-17B-16E-Instruct-GGUF 63.2 vision
Ministral-3-3B-Instruct-2512-GGUF 2.99 vision
Nemotron-3-Nano-30B-A3B-GGUF 22.8 —
Phi-4-mini-instruct-GGUF 2.49 —
Playable1-GGUF 4.68 coding
PromptBridge-0.6b-Alpha-GGUF 0.397 —
Qwen2.5-Coder-32B-Instruct-GGUF 19.9 coding
Qwen2.5-Omni-3B-GGUF 4.73 vision, chat-transcription
Qwen2.5-Omni-7B-GGUF 7.33 vision, chat-transcription
Qwen2.5-VL-3B-Instruct-GGUF 3.27 vision
Qwen2.5-VL-7B-Instruct-GGUF 6.04 vision
Qwen3-0.6B-GGUF 0.38 reasoning
Qwen3-1.7B-GGUF 1.06 reasoning
Qwen3-14B-GGUF 8.54 reasoning
Qwen3-30B-A3B-GGUF 17.4 reasoning
Qwen3-30B-A3B-Instruct-2507-GGUF 17.4 tool-calling
Qwen3-4B-GGUF 2.38 reasoning
Qwen3-4B-Instruct-2507-GGUF 2.5 tool-calling
Qwen3-8B-GGUF 5.25 reasoning
Qwen3-Coder-30B-A3B-Instruct-GGUF 18.6 coding, tool-calling, hot
Qwen3-Coder-Next-GGUF 48.0 coding, tool-calling, hot
Qwen3-Embedding-0.6B-GGUF 0.64 embeddings
Qwen3-Embedding-4B-GGUF 4.28 embeddings
Qwen3-Embedding-8B-GGUF 8.05 embeddings
Qwen3-Next-80B-A3B-Instruct-GGUF 46.1 tool-calling
Qwen3-VL-4B-Instruct-GGUF 3.33 vision
Qwen3-VL-8B-Instruct-GGUF 6.19 vision
Qwen3.5-0.8B-GGUF 0.764 vision, tool-calling
Qwen3.5-122B-A10B-GGUF 77.9 vision, tool-calling
Qwen3.5-122B-A10B-MTP-GGUF 79.6 vision, tool-calling, mtp
Qwen3.5-27B-GGUF 18.5 vision, tool-calling
Qwen3.5-2B-GGUF 2.01 vision, tool-calling
Qwen3.5-35B-A3B-GGUF 23.1 vision, tool-calling
Qwen3.5-4B-GGUF 3.58 vision, tool-calling, hot
Qwen3.5-4B-MTP-GGUF 3.66 vision, tool-calling, mtp
Qwen3.5-9B-GGUF 6.88 vision, tool-calling
Qwen3.6-27B-GGUF 18.5 vision, tool-calling
Qwen3.6-27B-MTP-GGUF 18.8 vision, tool-calling, mtp, hot
Qwen3.6-35B-A3B-GGUF 23.3 vision, tool-calling, hot
Qwen3.6-35B-A3B-MTP-GGUF 23.8 vision, tool-calling, mtp
SmolLM3-3B-GGUF 1.94 —
Tiny-Test-Model-GGUF 0.18 —
bge-reranker-v2-m3-GGUF 0.636 reranking
gpt-oss-120b-GGUF 62.8 reasoning, tool-calling
gpt-oss-120b-mxfp-GGUF 63.4 hot, reasoning, tool-calling
gpt-oss-20b-GGUF 11.6 reasoning, tool-calling
gpt-oss-20b-mxfp4-GGUF 12.1 hot, reasoning, tool-calling
granite-4.0-h-tiny-GGUF 4.25 tool-calling
jina-reranker-v1-tiny-en-GGUF 0.0367 reranking
nomic-embed-text-v1-GGUF 0.0781 embeddings
nomic-embed-text-v2-moe-GGUF 0.51 embeddings

moonshine — Moonshine (3 models)

Model Size (GB) Labels
Moonshine-Medium-Streaming 1.08 transcription, realtime-transcription, hot
Moonshine-Small-Streaming 0.431 transcription, realtime-transcription
Moonshine-Tiny-Streaming 0.202 transcription, realtime-transcription

openmoss — OpenMOSS TTS (2 models)

Model Size (GB) Labels
MOSS-VoiceGen 7.3 tts, voice-design
OpenMOSS-TTS 12.5 tts

ryzenai-llm — Ryzen AI LLM (79 models)

Model Size (GB) Labels
AMD-OLMo-1B-SFT-DPO-Hybrid 1.48 —
CodeLlama-7b-Instruct-hf-Hybrid 7.24 coding
CodeLlama-7b-Instruct-hf-NPU 7.54 coding
DeepSeek-R1-Distill-Llama-8B-CPU 6.2 reasoning
DeepSeek-R1-Distill-Llama-8B-Hybrid 9.09 reasoning
DeepSeek-R1-Distill-Llama-8B-NPU 9.3 reasoning
DeepSeek-R1-Distill-Qwen-1.5B-Hybrid 2.19 reasoning
DeepSeek-R1-Distill-Qwen-1.5B-NPU 2.3 reasoning
DeepSeek-R1-Distill-Qwen-7B-CPU 6.2 reasoning
DeepSeek-R1-Distill-Qwen-7B-Hybrid 8.67 reasoning
DeepSeek-R1-Distill-Qwen-7B-NPU 8.87 reasoning
Gemma-3-4b-it-mm-NPU 6.68 vision
Llama-2-7b-chat-hf-Hybrid 7.31 —
Llama-2-7b-chat-hf-NPU 7.47 —
Llama-2-7b-hf-Hybrid 7.31 —
Llama-2-7b-hf-NPU 7.47 —
Llama-3.1-8B-Hybrid 9.09 —
Llama-3.1-8B-NPU 9.3 —
Llama-3.2-1B-Hybrid 1.89 —
Llama-3.2-1B-Instruct-CPU 1.76 —
Llama-3.2-1B-Instruct-Hybrid 1.89 —
Llama-3.2-1B-Instruct-NPU 1.96 —
Llama-3.2-1B-NPU 1.96 —
Llama-3.2-3B-Hybrid 4.28 —
Llama-3.2-3B-Instruct-CPU 3.38 —
Llama-3.2-3B-Instruct-Hybrid 4.28 —
Meta-Llama-3-8B-Hybrid 9.06 —
Meta-Llama-3-8B-NPU 9.23 —
Meta-Llama-3.1-8B-Instruct-Hybrid 9.09 —
Meta-Llama-3.1-8B-Instruct-NPU 9.3 —
Mistral-7B-Instruct-v0.1-Hybrid 7.34 —
Mistral-7B-Instruct-v0.1-NPU 8.01 —
Mistral-7B-Instruct-v0.2-Hybrid 7.34 —
Mistral-7B-Instruct-v0.2-NPU 8.01 —
Mistral-7B-Instruct-v0.3-Hybrid 7.35 —
Mistral-7B-Instruct-v0.3-NPU 8.09 —
Mistral-7B-v0.3-Hybrid 7.35 —
Mistral-7B-v0.3-NPU 8.09 —
Phi-3-Mini-Instruct-CPU 2.39 —
Phi-3-mini-128k-instruct-Hybrid 4.21 —
Phi-3-mini-128k-instruct-NPU 4.35 —
Phi-3-mini-4k-instruct-Hybrid 4.19 —
Phi-3-mini-4k-instruct-NPU 4.3 —
Phi-3.5-mini-instruct-Hybrid 4.21 —
Phi-3.5-mini-instruct-NPU 4.35 —
Phi-4-mini-instruct-Hybrid 5.47 —
Phi-4-mini-instruct-NPU 5.59 —
Phi-4-mini-reasoning-Hybrid 5.47 reasoning
Qwen-1.5-7B-Chat-CPU 6.32 —
Qwen-2.5-1.5B-Instruct-Hybrid 2.17 —
Qwen-2.5-1.5B-Instruct-NPU 2.25 —
Qwen1.5-7B-Chat-Hybrid 8.83 —
Qwen1.5-7B-Chat-NPU 9.02 —
Qwen2-1.5B-Hybrid 2.19 —
Qwen2-1.5B-NPU 2.3 —
Qwen2-7B-Hybrid 8.68 —
Qwen2-7B-NPU 8.88 —
Qwen2.5-0.5B-Instruct-CPU 0.834 —
Qwen2.5-0.5B-Instruct-Hybrid 0.828 —
Qwen2.5-14B-instruct-Hybrid 16.5 —
Qwen2.5-3B-Instruct-Hybrid 3.97 —
Qwen2.5-3B-Instruct-NPU 4.1 —
Qwen2.5-7B-Instruct-Hybrid 8.65 —
Qwen2.5-7B-Instruct-NPU 8.83 —
Qwen2.5-Coder-0.5B-Instruct-Hybrid 0.828 coding
Qwen2.5-Coder-1.5B-Instruct-Hybrid 2.17 coding
Qwen2.5-Coder-1.5B-Instruct-NPU 2.25 coding
Qwen2.5-Coder-7B-Instruct-Hybrid 8.65 coding
Qwen2.5-Coder-7B-Instruct-NPU 8.83 coding
Qwen3-1.7B-Hybrid 2.55 reasoning
Qwen3-14B-Hybrid 16.5 reasoning
Qwen3-4B-Hybrid 5.17 reasoning
Qwen3-8B-Hybrid 9.42 reasoning
SmolLM-135M-Instruct-Hybrid 0.232 —
SmolLM2-135M-Instruct-Hybrid 0.233 —
chatglm3-6b-Hybrid 6.9 —
chatglm3-6b-NPU 7.04 —
gemma-2-2b-Hybrid 4.04 —
gpt-oss-20b-NPU 13.4 —

sd-cpp — StableDiffusion.cpp (12 models)

Model Size (GB) Labels
Flux-2-Klein-4B 16.1 image, edit
Flux-2-Klein-9B-GGUF 19.0 image, edit
Qwen-Image-2512-GGUF 19.4 image
Qwen-Image-GGUF 18.2 image
RealESRGAN-x4plus 0.064 upscaling, image
RealESRGAN-x4plus-anime 0.017 upscaling, image
SD-1.5 7.7 image
SD-Turbo 5.21 image
SD-Turbo-GGUF 2.02 image
SDXL-Base-1.0 6.94 image
SDXL-Turbo 6.94 image
Z-Image-Turbo 20.7 image

thinksound — ThinkSound (1 models)

Model Size (GB) Labels
ThinkSound-SFX 6.4 audio-generation

trellis — TRELLIS.2 (1 models)

Model Size (GB) Labels
TRELLIS-3D 15.4 3d

vllm — vLLM ROCm (experimental) (7 models)

Model Size (GB) Labels
GLM-4.7-Flash-FP16-vLLM 62.47 reasoning, tool-calling
Qwen3.5-0.8B-FP16-vLLM 1.77 reasoning
Qwen3.5-2B-FP16-vLLM 4.57 reasoning, tool-calling
Qwen3.5-4B-FP16-vLLM 9.34 reasoning, hot, tool-calling
Qwen3.5-9B-FP16-vLLM 19.3 reasoning, tool-calling
Qwen3.6-27B-FP16-vLLM 55.59 reasoning, tool-calling, vision
Qwen3.6-35B-A3B-FP16-vLLM 71.93 reasoning, tool-calling, vision

whispercpp — Whisper.cpp (6 models)

Model Size (GB) Labels
Whisper-Base 0.148 transcription, realtime-transcription
Whisper-Large-v3 3.1 transcription, realtime-transcription
Whisper-Large-v3-Turbo 1.62 transcription, realtime-transcription, hot
Whisper-Medium 1.53 transcription, realtime-transcription
Whisper-Small 0.488 transcription, realtime-transcription
Whisper-Tiny 0.075 transcription, realtime-transcription

Implementation notes

ACE-Step (acestep)

ace-server exposes an asynchronous job API: POST /lm or POST /synth returns a job id immediately, GET /job?id=N polls the status, and GET /job?id=N&result=1 fetches the finished result. AceStepServer::run_job wraps this submit/poll/fetch cycle with a ceiling of roughly 20 minutes per stage at a 1-second poll cadence. Synth results arrive as multipart/mixed (an audio part plus a latent part); Lemonade extracts the first audio part.

Vocals are a two-stage pipeline. POST /lm with lm_mode: "generate" runs the ACE-Step language model, which turns the caption and lyrics into audio codes plus LM-filled metadata, returned as a JSON array of enriched requests. That array is accepted by POST /synth verbatim, so Lemonade feeds it through unchanged. POST /synth on its own is the DiT-only instrumental path — it has no language model and cannot sing, so a lyrics value (other than the sentinel below) is what routes a request through /lm first.

[Instrumental] — any case, surrounding whitespace ignored — is ACE-Step's sentinel for the no-vocals path, matching the Python reference implementation. Instrumental requests send the sentinel explicitly rather than an empty string because the synth stage also feeds the lyrics text into its conditioning.

Errors from audio_generations are written into the response sink as a JSON error payload; the endpoint handler turns that into an HTTP error instead of shipping it as audio.

The model download fetches the DiT checkpoint variant plus three companions when present in the repo: the language model (acestep-5Hz-lm-4B-Q8_0.gguf, required for vocals and auto-lyrics), the Qwen3 text encoder, and the VAE. The checkpoint path handed to --models is the directory of GGUFs; ace-server scans it by architecture, and --keep-loaded keeps models resident across requests.

Backend auto-selection

When a recipe's backend is not pinned in config.json (the backend key is absent or "auto"), the default backend reported in system-info — and used by RecipeOptions when resolving *_backend options — is chosen as follows: the first supported backend in RECIPE_DEFS preference order wins, unless a later supported backend is already locally installed (state installed, update_available, or update_required) while the earlier candidates are merely installable. In that case the first installed one wins. This makes explicitly installing a variant (e.g. the Vulkan build of a GPU backend) an effective override: auto-selection uses what is on disk instead of downloading the preference-order winner. An explicit backend value in config.json always takes precedence over both rules. The llamacpp system variant is never auto-selected unless prefer_system is set.