This document records architectural conventions, project lessons learned, and guardrails to prevent regressions in future sessions.
externalBinBuild Guardrail:- Tauri 2 requires
src-tauri/binaries/llama-server-<target-triple>to physically exist beforetauri_build::build()executes insrc-tauri/build.rs. - Solution:
src-tauri/build.rsauto-generates an executable placeholder stub if the binary has not been fetched yet. - Provisioning: Real binaries are downloaded via
npm run sidecar:fetchorscripts/fetch-sidecar.sh.
- Tauri 2 requires
- The Pin Must Cover Every Architecture We Let Users Import:
- Ollama imports can carry architectures newer than the pin.
lfm3landed upstream on 2026-10-27 (first releaseb11935); the previousb10645pin failed withunknown model architecture: 'lfm3'even after a successful import. Current pin:b12032. - Newer builds renamed
--no-mmapto-lm none(--load-mode);server_managermust pass the new spelling. Checkllama-server --helpafter any pin bump.
- Ollama imports can carry architectures newer than the pin.
- Never Hardcode Soname Versions in the Fetch Script:
- Library versions rotate with every pin (
libggml0.22.0 → 0.28.0,libllama-server-implgained/lost soname links).scripts/fetch-sidecar.shderiveslibfoo.so.N/libfoo.N.dyliblinks from the staged payload and recordssidecars/binaries/.pinned-tag; a tag bump clears the previous libs before staging so stale links cannot shadow the new payload. - The executable itself must also be synced into
target/{debug,release}(and<target>/binaries/<triple>):resolve_sidecar_path()prefers those dev paths, and a stale 88-byte build.rs stub there reproduces the "Please run ./scripts/fetch-sidecar.sh" error even when the real payload is installed.
- Library versions rotate with every pin (
- Multi-Platform Target Detection:
- macOS targets:
x86_64-apple-darwin(Intel) oraarch64-apple-darwin(Apple Silicon). - Linux targets:
x86_64-unknown-linux-gnu. - Always inspect
uname -sanduname -mwhen writing download or launch scripts. - On Linux,
llama-serverdynamic dependencies (libggml.so,libllama.so,libmtmd.so,libllama-server-impl.so) must be placed insrc-tauri/binaries/withLD_LIBRARY_PATHconfigured on child spawn. - AppImage / linuxdeploy Guardrail: When bundling
.AppImage,linuxdeploytraces ELF dynamic dependencies. EnsureLD_LIBRARY_PATH="$PWD/src-tauri/binaries:$LD_LIBRARY_PATH"is set solinuxdeploydiscovers internal.sofiles and avoidsCould not find dependency: libllama-server-impl.so.
- macOS targets:
- macOS Dynamic Library Bundling & Dev Guardrail:
- Tauri 2 only copies the single binary in
externalBinintoContents/MacOS/llama-server(ortarget/debug/llama-server). - Because
llama-serveris dynamically linked against.dylibfiles (libllama-server-impl.dylib,libllama.dylib,libggml.dylib,libllama-common.dylib,libmtmd.dylib, etc.) resolving via@rpath/@loader_path, all.dylibfiles, metal shaders, and their version symlinks must be synchronized intoContents/MacOS/viascripts/post-bundle.shand intotarget/debug/viasrc-tauri/build.rsandserver_manager::ensure_sidecar_dependencies. - macOS Floor Is 15.0 — Do Not Lower It Without Rebuilding the Sidecar:
- The old
b10645prebuilt declaredminos 15.0(SDK 15.5) and linked Accelerate ILP64 symbols (_cblas_sgemm$NEWLAPACK$ILP64, present only in macOS 13.3+) plus newerlibc++/libSystem, sotauri.conf.jsonmacOS.minimumSystemVersionis"15.0"(enforced by the OS viaLSMinimumSystemVersion). The currentb12032prebuilt declaresminos 13.3, so 15.0 can be revisited, but never lower it without re-checkingLC_BUILD_VERSIONand theNEWLAPACK$ILP64symbols on both arch assets. - Supporting an older floor again would require either a natively rebuilt sidecar (
-DGGML_BLAS=OFF -DCMAKE_OSX_DEPLOYMENT_TARGET=<floor>, plus anm -m | grep NEWLAPACKgate inscripts/fetch-sidecar.sh) or a dual-variant bundle — all previously prototyped and deliberately removed in favor of the 15.0 floor.
- Tauri 2 only copies the single binary in
- Subcrate Dependency Declaration:
- Declaring a crate in root
[workspace.dependencies](e.g.chrono,tokio-util) makes the version available to the workspace, but does NOT automatically expose it to member crates. - Every member crate (e.g.,
src-tauri/Cargo.toml,crates/gateway/Cargo.toml) that imports a crate must explicitly declare:[dependencies] chrono.workspace = true tokio-util.workspace = true
- Declaring a crate in root
- Strict Port 13370 Binding:
- External developer tools (Cursor, Continue, Cline, Aider, OpenAI SDKs) hardcode
http://127.0.0.1:13370/v1. - The gateway must strictly bind to port 13370. Never loop or auto-increment to 13371 on collision.
- On port collision, return
AppError::ServerPortBindand surface an actionable error dialog.
- External developer tools (Cursor, Continue, Cline, Aider, OpenAI SDKs) hardcode
- Internal Sidecar Port:
llama-serversidecar binds to an ephemeral OS-assigned free port (e.g.,18080+). External clients never communicate directly with the sidecar.
- Grouped-Query Attention (GQA):
- Always use
n_kv_heads(e.g.,8for Llama 3.1 8B), never total attention heads (n_heads = 32), when calculating KV cache bytes. Usingn_headsinflates KV cache size by 4×.
- Always use
- Gemma 2
head_dim:- Gemma 2 decouples
head_dim(explicitly256) fromhidden_size / n_heads. Always respect explicithead_dim.
- Gemma 2 decouples
- Binary Header Verification:
crates/downloader/src/gguf.rsreads GGUF binary headers post-download to verifyn_layers,n_kv_heads, andhead_dimagainst catalog records.- Parser sanity limits must stay above real vocab sizes:
tokenizer.ggml.tokensarrays routinely hit 128k/256k entries. The old 100k array cap rejected valid files (Ollama's LFM3-1.7B blob failed to import with a misleading "not a valid GGUF binary" error); the cap is now 1M. Never lower it below 1M without checking real tokenizer sizes. - Per-layer scalar arrays: some architectures store per-layer scalars as GGUF arrays (LFM3 interleaved sliding-window attention stores
{arch}.attention.head_count_kvper layer). Numeric header lookups useGgufValue::as_u32_max, which resolves arrays to their max element (conservative for KV-cache math) instead of dropping them. - Ollama blobs: Ollama model layers are raw GGUF named
sha256-<digest>; the manifest's layerfromfield carries the original filename and is used as a quantization fallback when the GGUF has nogeneral.file_type.
- Zero Binaries in Git:
- Never commit
.so,.dylib,.dll,.exe,llama-server*, or.tar.gzarchives. - Never commit
.gguf,.bin, or.partmodel weights. - All binaries and weights are ignored via
.gitignoreand provisioned on demand.
- Never commit
- Release Profile Optimizations:
- Workspace root
Cargo.tomlmust declare[profile.release]withlto = "thin",codegen-units = 1,panic = "abort", andstrip = true. This cuts the core binary size from 25 MB down to 13 MB.
- Workspace root
- Tauri Packaging Deduplication:
- In
src-tauri/tauri.conf.json,"resources": ["binaries/*"]provisions dynamic libraries into the application resource path (/usr/lib/ModelSift/for current packages;/usr/lib/llm-advisor/remains a runtime fallback for pre-rename installs). - Do NOT add
"files": { "/usr/lib/ModelSift": "binaries" }inlinux.deborlinux.rpm. Doing so duplicates the 37 MB.sotree into a redundant 62 MB nested folder where version symlinks are dereferenced into full duplicate files.
- In
- Sidecar Auxiliary Library Pruning:
- Upstream
llama.cppreleases contain implementation shared libraries for unrelated CLI tools (libllama-bench-impl.so,libllama-cli-impl.so,libllama-perplexity-impl.so, etc.). scripts/fetch-sidecar.shfilters these out during download, preserving onlyllama-serverand its dynamic runtime dependencies.- Never filter
libggml-rpc*: the macOS x64llama-serverprebuilt links@rpath/libggml-rpc.0.dylib, so dropping it fails every Intel launch withLibrary not loadedand can trigger a multi-hour native-fallback job. Keep the--versionverification bounded by the script's watchdog as well.
- Upstream
- Tauri CLI auto-injects linked runtime deps — do not duplicate them manually:
- When bundling on Linux,
tauri-cliappends dependencies for the libraries it links (seesrc/interface/rust.rsin the publishedtauri-clicrate):deb:libwebkit2gtk-4.1-0,libgtk-3-0, pluslibayatana-appindicator3-1when thetray-iconfeature is enabled.rpm: the portable soname formlibwebkit2gtk-4.1.so.0()(64bit),libgtk-3.so.0()(64bit),libayatana-appindicator3.so.1()(64bit)(()(64bit)suffix on 64-bit targets).
- When bundling on Linux,
bundle.linux.rpm.dependsmust contain sonames, never distro package names:- Fedora-only names (
webkit2gtk4.1,gtk3,librsvg2,openssl) do not resolve on openSUSE, and openSUSE source-package names such aslibayatana-appindicatorresolve nowhere. On Fedora the latter surfaces asnothing provides libayatana-appindicator needed by llm-advisoranddnfrefuses the whole transaction. - Use
libfoo.so.N()(64bit)(with the 64-bit suffix — all release targets are 64-bit). Fedora mapping examples:libayatana-appindicator3.so.1()(64bit)→libayatana-appindicator-gtk3,libcrypto.so.3()(64bit)→openssl-libs. - v0.2.3 shipped the broken
libayatana-appindicatorrequires;install.shdetects that metadata on failure and prints AppImage / force-install remedies until users upgrade.
- Fedora-only names (
bundle.linux.deb.dependskeeps Debian/Ubuntu binary package names (auto-injected duplicates are harmless).
- Linux floor = release build host:
release.ymlbuilds onubuntu-24.04, so the binaries require glibc >= 2.39 (verified withobjdump -T: the app binary importsGLIBC_2.39; the pinned llama.cpp sidecar only needs 2.34, so the app governs).- Measure per release:
objdump -T target/release/llm-advisor | grep -oE 'GLIBC_[0-9]+\.[0-9]+' | sort -Vu | tail -1. install.shenforces it viaMIN_GLIBC;release.ymlfails the release when the built app or sidecar libs require a newer glibc thaninstall.shadvertises (guards silent runner-image bumps). musl systems (Alpine) are refused by the installer.- Supported minimums: Ubuntu 24.04+, Debian 13+ (glibc 2.41), Fedora 40+ (2.39), RHEL/Rocky/Alma 10+ (2.39; EPEL for the tray dependency), openSUSE Leap 16+ (2.40)/Tumbleweed.
- Measure per release:
- macOS floor = 15.0, declared in
src-tauri/tauri.conf.json(bundle.macOS.minimumSystemVersion) and dictated by the upstream llama.cpp prebuilt (minos 15.0, Accelerate ILP64 symbols — see section 1).install.shenforces the same value viaMIN_MACOS;release.ymlverifies both stay in sync. - Bumping the runner image or the sidecar pin can raise these floors — update
MIN_GLIBC/MIN_MACOSininstall.sh, the README note, and this section in the same commit.
- Catalog contract: embedding entries must set
output_type: "embedding"and an explicitpooling(mean,cls,last, orrank).catalog::parse_catalog_jsonrejects invalid pooling values and pooling on non-embedding entries;scripts/update-catalog.pyandscripts/build-catalog.mjsenforce the same rules. - Sidecar flags: embedding instances launch with
--embeddings, plus--pooling <value>when the catalog declares one. They must not receive-ctk/-ctv(there is no KV cache). Text instances keep the existing KV flags. - Context clamp:
-cfor an embedding model is the maximum input length, sosrc-tauri/src/lib.rsclamps it toentry.context_trainwith a floor of 256 tokens (e.g. MiniLM stays at 512). - Capability-aware routing:
ServerManager::route_model(requested, ModelCapability)is the single routing entry point. Chat (/v1/chat/completions,/v1/completions) and embeddings (/v1/embeddings) must never cross capabilities. Explicitly naming the wrong type returnsWrongCapability→ HTTP 400invalid_request_error; an empty capability pool returnsNoMatchingModel→ HTTP 503. - Runtime metadata:
RunningProcess/RunningInstanceInfo,GET /v1/models, andServerState::Serving.instancesexposeoutput_typeso the UI can badge instances and the gateway can keep capabilities separate. - Fit math: the fit engine still returns zero KV bytes for embedding models. BERT-family GGUFs often omit
head_count_kv/key_length; the downloader skips those header checks rather than failing verification. - Nomic v2 MoE:
nomic-embed-text-v2-moe-q8_0has a real 512-token context limit and is documented as such in the README.
- Frameless by design:
src-tauri/tauri.conf.jsonsetsdecorations: falseon every OS.src/components/layout/TitleBar.tsxis the only title bar; do not reintroduce OS-specific native chrome (the old GTK header-bar CSS override and itsgtkcrate were removed as obsolete). - Drag region: the title bar uses
data-tauri-drag-region="deep"(Tauri 2.11+) so the entire bar drags, while buttons/inputs opt out automatically. Double-click toggling comes from Tauri's injected script (core:window:allow-internal-toggle-maximize, included incore:default). - Window permissions in
src-tauri/capabilities/default.jsonmust keepallow-minimize,allow-toggle-maximize,allow-close,allow-start-dragging, andallow-start-resize-dragging;allow-is-maximizedalso comes fromcore:default. - Resize border:
decorations: falseremoves the native resize frame, soWindowResizeHandles.tsxprovides invisible 8-direction hit areas backed bystartResizeDragging. Keep them mounted at the app shell (hidden when maximized); without them edge resizing breaks on Linux/X11. - Trade-off: macOS loses native traffic lights and system window-menu behaviors (move/align) in exchange for identical chrome across OSes. This is intentional.
- Display brand is ModelSift (one word, capital S), tagline The local model advisor. The UI, docs, installer banners, tray menu, release name, and package
productNameuse this spelling. Do not reintroduce "LLM Advisor" in user-facing copy. - Deliberately retained technical identifiers (do not rename without a migration plan):
- Tauri
identifier:dev.yoel3imari.llm-advisor— controls the app-data directory, single-instance identity, and updater continuity. - Cargo crate / binary:
llm-advisor; npm package:llm-advisor. Release assets andinstall.sh APP_BINstill use it. - Data directories:
~/.local/share/dev.yoel3imari.llm-advisor/,~/Library/Application Support/dev.yoel3imari.llm-advisor/,%APPDATA%\dev.yoel3imari.llm-advisor\. localStoragekeys:llm-advisor-sidebar-collapsed,llm-advisor.chat.sessions.v1,llm_advisor_theme, and the demo'sllm_advisor_*keys.
- Tauri
- GitHub repository: renamed to
yoel3imari/modelsift. GitHub redirects the oldyoel3imari/llm-advisorslug indefinitely, unless a new repo is created with the old name. Never reusellm-advisorfor a different repository; old installers, updater endpoints, catalog sync, and raw install URLs depend on that redirect. - Packaging paths:
productNameis nowModelSift, so Tauri uses/usr/lib/ModelSiftfor Linux resources,ModelSift.desktop/ModelSift.app, and kebab-cases the deb/rpm package name tomodel-sift; Windows installs underModelSift. Every sidecar lookup and uninstall path keeps the oldllm-advisorlocations as fallbacks so upgrades from 0.2.x keep working.scripts/post-bundle.shand both uninstall scripts must continue to handle both names. The deb declaresreplaces/conflictsand the rpm declaresobsoletesforllm-advisorso the package manager replaces the pre-rename package instead of failing on the shared/usr/bin/llm-advisorpath. - Updater note: changing
productNamechanges the macOS.appdirectory name, the Windows install/registry identity, and the Linux package name. Keep the identifier stable and verify an in-place upgrade from the lastLLM Advisorrelease on every target before shipping the rename; users may otherwise end up with both builds installed.
- Shared core:
crates/app_coreowns settings, library, downloads, inference lifecycle, gateway lifecycle, catalog sync, and factory reset.src-tauriis a thin adapter (AppStatederefs toAppCore) andcrates/cliis the second consumer. New orchestration logic belongs inapp_core, not in Tauri commands or the CLI. - Frontend contract: the Tauri event bridge must keep emitting the existing names and payloads:
download-complete(entry_id,filename,size_bytes),download-failed(entry_id,reason),catalog:updated(count,etag), andsettings-changed(the settings object). The tray tooltip is updated fromCoreEvent::GatewayStarted. - Admin API: routes live under
/__modelsifton the same listener and are disabled when noAdminBackendis installed (404). Authentication is a bearer token stored in<data>/admin.token(0600, 32 random bytes hex-encoded) and compared in constant time. Keep it loopback-only; never log the token. - Ownership:
<data>/runtime.lockis an exclusivefs2advisory lock held for the lifetime of the gateway;<data>/runtime.jsonpublishesowner,pid,gateway_port,version, andstarted_at.stop_gateway()removesruntime.jsononly while the process still holds the lock, so a desktop app that failed to acquire the gateway never deletes a CLI daemon's state. OnlyRuntimeOwner::Clidaemons acceptPOST /__modelsift/shutdown; desktop-owned gateways return HTTP 409. - Ports: external clients still use the strict
13370default; do not auto-increment on collision.--port 0exists for tests and binds an ephemeral port, which must be read back fromruntime.json(the gateway returns the actual bound port). - CLI:
crates/clibuilds a singlemodelsiftbinary. A second[[bin]] name = "llm-advisor"collides with Tauri'smainBinaryName, so the compatibility alias is an install-time symlink created byinstall.sh --with-clionly whenllm-advisoris not already the desktop binary. Exit codes are part of the contract:0success,1general error,3not running,4port/ownership conflict,5model error. - Detached mode:
modelsift serve --detachspawns itself in a new process group, redirects stdout/stderr to<data>/logs/modelsift.log, and waits for the admin API (and--model, when given) before returning. The child must clean upruntime.jsonon every exit path, including failed initial model loads and SIGTERM. - Release plumbing:
.github/workflows/release.ymlpublishes raw CLI binaries namedmodelsift-<target-triple>(.exeon Windows).install.sh --with-cliresolves that asset and installs it to~/.local/bin(or--cli-dir). Keep the asset name and installer lookup in sync.
- Two execution models, one core:
- Long-lived chat and embedding servers stay in
server_manager(ServerManager::route_modelremains the capability-aware entry point). Do not move them into the job queue. - One-shot engine work (verification today; transcription, image generation, speech next) goes through
crates/runtime::JobQueue.run_processspawns the engine binary, streams stdout/stderr into the job log, parses progress, and kills the process group on cancellation.
- Long-lived chat and embedding servers stay in
- Engine detection:
crates/runtime/src/engine.rsdefinesEngineKind/EngineCapability/EngineRegistry::detect. Detection looks for bundled sidecars and well-known binary names. Missing engines must remain visible with an actionablehint; never hide a capability because its binary is absent. - Durable queue:
JobQueuepersists records to<data>/jobs.json. Statuses serialize lowercase (queued,running,succeeded,failed,cancelled); kinds aremodel_verify,engine_run, andhardware_benchmark. Concurrency is bounded, history is pruned, and jobs leftrunningby a crash are recovered as failed on startup.JobQueue::submitreceives anArc<JobContext>and returns artifacts. - Paths: job logs live in
<data>/logs/jobs/<job-id>.log, artifacts in<data>/artifacts/<job-id>/. Never write job output into the model library or the workspace tree directly; register it through the workspace store when the job belongs to a workspace. - First real job:
AppCore::start_library_verifyre-hashes installed models, parses GGUF headers, and writesverify-report.jsonas an artifact. It is exposed asstart_library_verify(Tauri),POST /__modelsift/jobswith{"kind":"model_verify"}(admin), andmodelsift jobs verify [--wait](CLI). - Hardware benchmark:
JobKind::HardwareBenchmarkruns three fixed prompts (latency, throughput, long-context prefill) per selected chat model through the running sidecar, measuring tok/s, TTFT, prefill speed, RSS and memory headroom. Results are written to<data>/benchmarks/<job-id>.json, mirrored to<data>/benchmarks/latest.json, and published as thebenchmark-report.jsonjob artifact. The composite score is0.5*speed + 0.3*latency + 0.2*headroom(120 tok/s and 100 ms TTFT are full marks; headroom is a percentage of VRAM or RAM) with S/A/B/C/D/F grades at 90/80/70/60/50. Exposed asstart_benchmark/get_latest_benchmark_report(Tauri),POST /__modelsift/benchmarkandGET /__modelsift/benchmark/latest(admin), andmodelsift benchmark [--model <id>] [--wait|--latest](CLI). - Workspaces:
crates/workspace::WorkspaceStoreowns workspaces, sessions, artifacts and presets under<data>/workspaces/, plus<data>/workspace-presets.json. IDs and artifact filenames are validated for path traversal; keep those guards on every new entry point.WorkspaceKindis lowercase serde (general,notebook,writing,tts,image,video). - Cleanup:
CleanupPolicysupportsmax_age_days,max_total_bytes, anddelete_orphan_files(default true).gc()only walks registered workspace directories and reportsworkspaces_scanned,artifacts_deleted,orphan_files_deleted, andreclaimed_bytes. Factory reset clears jobs and workspaces throughAppCore. - App core ownership:
AppCoreholdsArc<JobQueue>,Arc<WorkspaceStore>, andArc<EngineRegistry>and implementsgateway::AdminBackend. Tauri commands are thin adapters; new orchestration belongs inapp_coreso the CLI inherits it.WorkspaceStore::add_artifactexists but is not yet wired intoAppCore; future engine jobs with aworkspace_idmust register their artifacts there. - Frontend contract: the Tauri app emits
job-updatedwith the fullJobRecordpayload. The Workspaces view (src/views/WorkspacesView.tsx) polls while work is active and also listens for the event. The Benchmark view (src/views/BenchmarkView.tsx) pollsget_jobfor the active benchmark, then loadsget_latest_benchmark_reportwhen it succeeds. Keepsrc/ipc/mock.tsin sync with new commands sonpm run dev/Vitest remain functional;mockFactoryReset()must clear jobs, workspaces, sessions, presets and the latest benchmark report. - Admin routes (all under
/__modelsift, bearer-token guarded):GET /engines,GET|POST /jobs,GET /jobs/{id},GET /jobs/{id}/logs,POST /jobs/{id}/cancel,POST /benchmark,GET /benchmark/latest,GET|POST /workspaces,POST /workspaces/gc,GET|PATCH|DELETE /workspaces/{id},GET|POST /workspaces/{id}/sessions,DELETE /workspaces/{id}/sessions/{sid},GET /workspaces/{id}/artifacts,DELETE /workspaces/{id}/artifacts/{aid}?delete_file=,GET|POST /presets,DELETE /presets/{id}. Collection routes return{ "engines": [...] },{ "jobs": [...] },{ "workspaces": [...] }, etc.; single-resource routes return the object directly. - CLI surface:
modelsift jobs list|show|logs|verify|cancel,modelsift workspaces list|show|create|rm|gc,modelsift benchmark [--model <id>] [--context <n>] [--max-tokens <n>] [--wait|--latest], plusmodelsift engines.jobs verify --waitandbenchmark --waitpoll the job and exit5on failure. Keep text output and--jsonenvelopes ({"ok":true,"data":...}) stable. - Tests:
cargo test -p runtime -p workspace -p app_core -p gateway -p modelsift-clicovers the queue, store, admin routes and CLI lifecycle;npm testcovers the Workspaces view. Runtime process tests are Unix-gated (sh -c).
enginefield:CatalogEntry.engine(domain::ModelEngine, serde snake_case, defaultllama_cpp) discriminates asset loaders:llama_cpp(GGUF),stable_diffusion(safetensors),whisper_cpp(legacy GGML.bin),piper(.onnx). Missingenginealways meansllama_cpp— keep the#[serde(default)]for backward compat with old catalogs and library records.- Relaxed architecture fields:
n_layers/n_kv_heads/head_dim/context_train(andparams_billionsin the scripts) are only required forllama_cppentries; image/audio entries may carry zeros. Relaxations live incatalog::parse_catalog_json,scripts/update-catalog.py::validate_entry, andscripts/build-catalog.mjs::validateCatalog— update all three together.quality_tierstays 1–5 everywhere. - GGUF-only checks: header cross-checks in the downloader and
verify_model_filemust be gated onentry.uses_gguf_header()(engine == llama_cpp). Whisper.binfiles are GGML (lmggmagic), not GGUF;.onnx/.safetensorshave no header at all. Those assets verify on size + sha256 alone. - Download naming: single-file destinations are
{id}.{real extension from entry.filename}viadownloader::asset_extension— never hardcode.ggufagain. Existing GGUF installs keep their names. - Companion files:
CatalogEntry.companion(CompanionFile) downloads a config next to the weights under its real basename (Piper requires<voice>.onnx.jsonbeside<voice>.onnx).verify_model_filefolds a missing/corrupt companion into the verify result. - Routing:
ModelCapability::Chatmatches onlyText | Multimodaloutputs; image/audio models are never chat-routable.AppCore::load_modelrejects non-llama_cppengines with an actionable "downloadable only" error — these engines have no run pipeline yet (noJobKind::EngineRunproducers). - Frontend:
src/types/domain.tsmirrorsengine/companionas optional (older payloads default tollama_cpp). Tables render—for zero params/context, and the output-type filter already includes Image/Audio. - Catalog content: entries added so far — 10 whisper models (
ggerganov/whisper.cpp, GGML.bin, metadata parsed from the GGML header), 7 Piper voices (rhasspy/piper-voices,.onnx+.onnx.jsoncompanion; configs are non-LFS so their tree API returns git oids, not sha256 — download and hash them to catalogue), 4 Stable Diffusion checkpoints (stabilityai/*+stable-diffusion-v1-5/*, safetensors; sizes/sha256 from HF LFS oids). Catalog total is 133 before the OCR entries.
- Source of truth is per-entry:
crates/catalog/entries/<id>.jsonholds exactly oneCatalogEntryobject. The filename must equal<id>.json. Never hand-editcrates/catalog/catalog.json— it is generated. - Distribution artifact:
crates/catalog/catalog.jsonis regenerated bynode scripts/build-catalog.mjs(sorted by id, canonical key order shared withdomain::CatalogEntry). It stays committed because it is embedded viainclude_str!and served for remote etag sync. - Commands:
node scripts/split-catalog.mjs [--force]— one-time migration from the monolith; refuses to clobber an existing entries directory unless forced.node scripts/build-catalog.mjs— validate entries and regeneratecatalog.json.node scripts/build-catalog.mjs --verify-only— validate only, no writes.node scripts/build-catalog.mjs --check— CI guard; fails whencatalog.jsonis stale. Wired into.github/workflows/ci.yml.python3 scripts/update-catalog.py --online [--only <id>] [--force]— refresh HF metadata/benchmarks into entry files, then regeneratecatalog.json. Entries verified within--max-age-days(default 7) are skipped; the cache lives atcrates/catalog/.catalog-cache.jsonand is gitignored.
- Curated fields are never clobbered:
update-catalog.pymerges HF size/sha256/architecture/benchmarks into existing entries;tags,quality_tier,prompt_preset,pooling,engine,companion, andmmprojare preserved. - Canonical key order:
id, repo_id, filename, family, params_billions, active_params_b, n_layers, n_kv_heads, head_dim, context_train, quant, file_size_bytes, sha256, gated, quality_tier, tags, benchmarks, shards, output_type, pooling, mmproj, engine, companion, prompt_preset. Unknown keys are appended alphabetically. Keepscripts/catalog-lib.mjs,scripts/update-catalog.py, and the Rust struct in sync when adding fields. - Validation parity: engine whitelist, GGUF-only architecture requirements, embedding pooling rules, companion/mmproj sha256 checks,
quality_tier1–5, and non-emptyprompt_presetare enforced incatalog::parse_catalog_json,scripts/update-catalog.py::validate_entry, andscripts/build-catalog.mjs::validateCatalog— update all three together.
- Catalog contract: OCR entries use
engine: llama_cpp,output_type: multimodal, and must include a companion multimodal projector (mmproj). They remain standardllama.cppmodels served byllama-server— no new engine is required (Track A). - Trained
prompt_presetis load-bearing: Dedicated OCR VLMs (PaddleOCR-VL, GLM-OCR, DeepSeek-OCR, Dots.OCR, HunyuanOCR, LightOnOCR, Qianfan-OCR) require specific prompt formats (e.g.OCR:for PaddleOCR-VL,OCRfor GLM-OCR/DeepSeek-OCR). Omitting or changing the prompt causes hallucinations or degraded output. The catalog entry'sprompt_presetsupplies the verified prompt string. - Sidecar architecture support: Supported architectures are integrated into upstream
llama.cpp(PaddleOCR-VL #18825, GLM-OCR #19677, DeepSeek-OCR #17400, Dots.OCR #17575, HunyuanOCR #21395). The sidecar pin must cover these architecture PRs. GOT-OCR2.0 is excluded due to lack of upstream llama.cpp support. - Per-architecture caveats:
- PaddleOCR-VL: upstream notes potential degraded performance at low quantization; catalog entry carries quality tier caveat.
- DeepSeek-OCR: larger memory footprint; projector and context must fit host budget.
- Dots.OCR / GLM-OCR: highly sensitive to prompt syntax; always default to catalog
prompt_preset.
- CLI interface:
modelsift ocr <model-id> <image-path> [--json] [--prompt <text>]auto-loads the model with an ephemeral serve config if not currently serving, encodes image to a base64 data URI, executes chat completion via the OpenAI-compatible proxy, and outputs transcribed text (or JSON envelope). - Playground UX:
QuickTestBardisplays the "Extract Text (OCR)" preset chip whenever the active model has vision capability and theocrtag.ChatViewsurfaces an inline warning banner if the OCR preset is activated before an image is attached.