Skip to content

Latest commit

 

History

History
230 lines (179 loc) · 31.4 KB

AGENTS.md

File metadata and controls

230 lines (179 loc) · 31.4 KB

ModelSift — Development Guidelines & Lessons Learned

This document records architectural conventions, project lessons learned, and guardrails to prevent regressions in future sessions.


1. Tauri 2 & Sidecar Management

  • externalBin Build Guardrail:
    • Tauri 2 requires src-tauri/binaries/llama-server-<target-triple> to physically exist before tauri_build::build() executes in src-tauri/build.rs.
    • Solution: src-tauri/build.rs auto-generates an executable placeholder stub if the binary has not been fetched yet.
    • Provisioning: Real binaries are downloaded via npm run sidecar:fetch or scripts/fetch-sidecar.sh.
  • The Pin Must Cover Every Architecture We Let Users Import:
    • Ollama imports can carry architectures newer than the pin. lfm3 landed upstream on 2026-10-27 (first release b11935); the previous b10645 pin failed with unknown model architecture: 'lfm3' even after a successful import. Current pin: b12032.
    • Newer builds renamed --no-mmap to -lm none (--load-mode); server_manager must pass the new spelling. Check llama-server --help after any pin bump.
  • Never Hardcode Soname Versions in the Fetch Script:
    • Library versions rotate with every pin (libggml 0.22.0 → 0.28.0, libllama-server-impl gained/lost soname links). scripts/fetch-sidecar.sh derives libfoo.so.N / libfoo.N.dylib links from the staged payload and records sidecars/binaries/.pinned-tag; a tag bump clears the previous libs before staging so stale links cannot shadow the new payload.
    • The executable itself must also be synced into target/{debug,release} (and <target>/binaries/<triple>): resolve_sidecar_path() prefers those dev paths, and a stale 88-byte build.rs stub there reproduces the "Please run ./scripts/fetch-sidecar.sh" error even when the real payload is installed.
  • Multi-Platform Target Detection:
    • macOS targets: x86_64-apple-darwin (Intel) or aarch64-apple-darwin (Apple Silicon).
    • Linux targets: x86_64-unknown-linux-gnu.
    • Always inspect uname -s and uname -m when writing download or launch scripts.
    • On Linux, llama-server dynamic dependencies (libggml.so, libllama.so, libmtmd.so, libllama-server-impl.so) must be placed in src-tauri/binaries/ with LD_LIBRARY_PATH configured on child spawn.
    • AppImage / linuxdeploy Guardrail: When bundling .AppImage, linuxdeploy traces ELF dynamic dependencies. Ensure LD_LIBRARY_PATH="$PWD/src-tauri/binaries:$LD_LIBRARY_PATH" is set so linuxdeploy discovers internal .so files and avoids Could not find dependency: libllama-server-impl.so.
  • macOS Dynamic Library Bundling & Dev Guardrail:
    • Tauri 2 only copies the single binary in externalBin into Contents/MacOS/llama-server (or target/debug/llama-server).
    • Because llama-server is dynamically linked against .dylib files (libllama-server-impl.dylib, libllama.dylib, libggml.dylib, libllama-common.dylib, libmtmd.dylib, etc.) resolving via @rpath / @loader_path, all .dylib files, metal shaders, and their version symlinks must be synchronized into Contents/MacOS/ via scripts/post-bundle.sh and into target/debug/ via src-tauri/build.rs and server_manager::ensure_sidecar_dependencies.
    • macOS Floor Is 15.0 — Do Not Lower It Without Rebuilding the Sidecar:
    • The old b10645 prebuilt declared minos 15.0 (SDK 15.5) and linked Accelerate ILP64 symbols (_cblas_sgemm$NEWLAPACK$ILP64, present only in macOS 13.3+) plus newer libc++/libSystem, so tauri.conf.json macOS.minimumSystemVersion is "15.0" (enforced by the OS via LSMinimumSystemVersion). The current b12032 prebuilt declares minos 13.3, so 15.0 can be revisited, but never lower it without re-checking LC_BUILD_VERSION and the NEWLAPACK$ILP64 symbols on both arch assets.
    • Supporting an older floor again would require either a natively rebuilt sidecar (-DGGML_BLAS=OFF -DCMAKE_OSX_DEPLOYMENT_TARGET=<floor>, plus a nm -m | grep NEWLAPACK gate in scripts/fetch-sidecar.sh) or a dual-variant bundle — all previously prototyped and deliberately removed in favor of the 15.0 floor.

2. Cargo Workspace Dependencies

  • Subcrate Dependency Declaration:
    • Declaring a crate in root [workspace.dependencies] (e.g. chrono, tokio-util) makes the version available to the workspace, but does NOT automatically expose it to member crates.
    • Every member crate (e.g., src-tauri/Cargo.toml, crates/gateway/Cargo.toml) that imports a crate must explicitly declare:
      [dependencies]
      chrono.workspace = true
      tokio-util.workspace = true

3. Reverse Proxy & Port Strategy (ADR-3a)

  • Strict Port 13370 Binding:
    • External developer tools (Cursor, Continue, Cline, Aider, OpenAI SDKs) hardcode http://127.0.0.1:13370/v1.
    • The gateway must strictly bind to port 13370. Never loop or auto-increment to 13371 on collision.
    • On port collision, return AppError::ServerPortBind and surface an actionable error dialog.
  • Internal Sidecar Port:
    • llama-server sidecar binds to an ephemeral OS-assigned free port (e.g., 18080+). External clients never communicate directly with the sidecar.

4. Memory-Fit Math & GGUF Metadata Integrity

  • Grouped-Query Attention (GQA):
    • Always use n_kv_heads (e.g., 8 for Llama 3.1 8B), never total attention heads (n_heads = 32), when calculating KV cache bytes. Using n_heads inflates KV cache size by 4×.
  • Gemma 2 head_dim:
    • Gemma 2 decouples head_dim (explicitly 256) from hidden_size / n_heads. Always respect explicit head_dim.
  • Binary Header Verification:
    • crates/downloader/src/gguf.rs reads GGUF binary headers post-download to verify n_layers, n_kv_heads, and head_dim against catalog records.
    • Parser sanity limits must stay above real vocab sizes: tokenizer.ggml.tokens arrays routinely hit 128k/256k entries. The old 100k array cap rejected valid files (Ollama's LFM3-1.7B blob failed to import with a misleading "not a valid GGUF binary" error); the cap is now 1M. Never lower it below 1M without checking real tokenizer sizes.
    • Per-layer scalar arrays: some architectures store per-layer scalars as GGUF arrays (LFM3 interleaved sliding-window attention stores {arch}.attention.head_count_kv per layer). Numeric header lookups use GgufValue::as_u32_max, which resolves arrays to their max element (conservative for KV-cache math) instead of dropping them.
    • Ollama blobs: Ollama model layers are raw GGUF named sha256-<digest>; the manifest's layer from field carries the original filename and is used as a quantization fallback when the GGUF has no general.file_type.

5. Git & Storage Guardrails

  • Zero Binaries in Git:
    • Never commit .so, .dylib, .dll, .exe, llama-server*, or .tar.gz archives.
    • Never commit .gguf, .bin, or .part model weights.
    • All binaries and weights are ignored via .gitignore and provisioned on demand.

6. Binary Size & Packaging Guardrails

  • Release Profile Optimizations:
    • Workspace root Cargo.toml must declare [profile.release] with lto = "thin", codegen-units = 1, panic = "abort", and strip = true. This cuts the core binary size from 25 MB down to 13 MB.
  • Tauri Packaging Deduplication:
    • In src-tauri/tauri.conf.json, "resources": ["binaries/*"] provisions dynamic libraries into the application resource path (/usr/lib/ModelSift/ for current packages; /usr/lib/llm-advisor/ remains a runtime fallback for pre-rename installs).
    • Do NOT add "files": { "/usr/lib/ModelSift": "binaries" } in linux.deb or linux.rpm. Doing so duplicates the 37 MB .so tree into a redundant 62 MB nested folder where version symlinks are dereferenced into full duplicate files.
  • Sidecar Auxiliary Library Pruning:
    • Upstream llama.cpp releases contain implementation shared libraries for unrelated CLI tools (libllama-bench-impl.so, libllama-cli-impl.so, libllama-perplexity-impl.so, etc.).
    • scripts/fetch-sidecar.sh filters these out during download, preserving only llama-server and its dynamic runtime dependencies.
    • Never filter libggml-rpc*: the macOS x64 llama-server prebuilt links @rpath/libggml-rpc.0.dylib, so dropping it fails every Intel launch with Library not loaded and can trigger a multi-hour native-fallback job. Keep the --version verification bounded by the script's watchdog as well.

7. Linux Package Dependency Metadata (deb/rpm)

  • Tauri CLI auto-injects linked runtime deps — do not duplicate them manually:
    • When bundling on Linux, tauri-cli appends dependencies for the libraries it links (see src/interface/rust.rs in the published tauri-cli crate):
      • deb: libwebkit2gtk-4.1-0, libgtk-3-0, plus libayatana-appindicator3-1 when the tray-icon feature is enabled.
      • rpm: the portable soname form libwebkit2gtk-4.1.so.0()(64bit), libgtk-3.so.0()(64bit), libayatana-appindicator3.so.1()(64bit) (()(64bit) suffix on 64-bit targets).
  • bundle.linux.rpm.depends must contain sonames, never distro package names:
    • Fedora-only names (webkit2gtk4.1, gtk3, librsvg2, openssl) do not resolve on openSUSE, and openSUSE source-package names such as libayatana-appindicator resolve nowhere. On Fedora the latter surfaces as nothing provides libayatana-appindicator needed by llm-advisor and dnf refuses the whole transaction.
    • Use libfoo.so.N()(64bit) (with the 64-bit suffix — all release targets are 64-bit). Fedora mapping examples: libayatana-appindicator3.so.1()(64bit) → libayatana-appindicator-gtk3, libcrypto.so.3()(64bit) → openssl-libs.
    • v0.2.3 shipped the broken libayatana-appindicator requires; install.sh detects that metadata on failure and prints AppImage / force-install remedies until users upgrade.
  • bundle.linux.deb.depends keeps Debian/Ubuntu binary package names (auto-injected duplicates are harmless).

8. Minimum System Version Floors

  • Linux floor = release build host: release.yml builds on ubuntu-24.04, so the binaries require glibc >= 2.39 (verified with objdump -T: the app binary imports GLIBC_2.39; the pinned llama.cpp sidecar only needs 2.34, so the app governs).
    • Measure per release: objdump -T target/release/llm-advisor | grep -oE 'GLIBC_[0-9]+\.[0-9]+' | sort -Vu | tail -1.
    • install.sh enforces it via MIN_GLIBC; release.yml fails the release when the built app or sidecar libs require a newer glibc than install.sh advertises (guards silent runner-image bumps). musl systems (Alpine) are refused by the installer.
    • Supported minimums: Ubuntu 24.04+, Debian 13+ (glibc 2.41), Fedora 40+ (2.39), RHEL/Rocky/Alma 10+ (2.39; EPEL for the tray dependency), openSUSE Leap 16+ (2.40)/Tumbleweed.
  • macOS floor = 15.0, declared in src-tauri/tauri.conf.json (bundle.macOS.minimumSystemVersion) and dictated by the upstream llama.cpp prebuilt (minos 15.0, Accelerate ILP64 symbols — see section 1). install.sh enforces the same value via MIN_MACOS; release.yml verifies both stay in sync.
  • Bumping the runner image or the sidecar pin can raise these floors — update MIN_GLIBC/MIN_MACOS in install.sh, the README note, and this section in the same commit.

9. Embedding Model Serving

  • Catalog contract: embedding entries must set output_type: "embedding" and an explicit pooling (mean, cls, last, or rank). catalog::parse_catalog_json rejects invalid pooling values and pooling on non-embedding entries; scripts/update-catalog.py and scripts/build-catalog.mjs enforce the same rules.
  • Sidecar flags: embedding instances launch with --embeddings, plus --pooling <value> when the catalog declares one. They must not receive -ctk/-ctv (there is no KV cache). Text instances keep the existing KV flags.
  • Context clamp: -c for an embedding model is the maximum input length, so src-tauri/src/lib.rs clamps it to entry.context_train with a floor of 256 tokens (e.g. MiniLM stays at 512).
  • Capability-aware routing: ServerManager::route_model(requested, ModelCapability) is the single routing entry point. Chat (/v1/chat/completions, /v1/completions) and embeddings (/v1/embeddings) must never cross capabilities. Explicitly naming the wrong type returns WrongCapability → HTTP 400 invalid_request_error; an empty capability pool returns NoMatchingModel → HTTP 503.
  • Runtime metadata: RunningProcess/RunningInstanceInfo, GET /v1/models, and ServerState::Serving.instances expose output_type so the UI can badge instances and the gateway can keep capabilities separate.
  • Fit math: the fit engine still returns zero KV bytes for embedding models. BERT-family GGUFs often omit head_count_kv/key_length; the downloader skips those header checks rather than failing verification.
  • Nomic v2 MoE: nomic-embed-text-v2-moe-q8_0 has a real 512-token context limit and is documented as such in the README.

10. Cross-Platform Window Chrome

  • Frameless by design: src-tauri/tauri.conf.json sets decorations: false on every OS. src/components/layout/TitleBar.tsx is the only title bar; do not reintroduce OS-specific native chrome (the old GTK header-bar CSS override and its gtk crate were removed as obsolete).
  • Drag region: the title bar uses data-tauri-drag-region="deep" (Tauri 2.11+) so the entire bar drags, while buttons/inputs opt out automatically. Double-click toggling comes from Tauri's injected script (core:window:allow-internal-toggle-maximize, included in core:default).
  • Window permissions in src-tauri/capabilities/default.json must keep allow-minimize, allow-toggle-maximize, allow-close, allow-start-dragging, and allow-start-resize-dragging; allow-is-maximized also comes from core:default.
  • Resize border: decorations: false removes the native resize frame, so WindowResizeHandles.tsx provides invisible 8-direction hit areas backed by startResizeDragging. Keep them mounted at the app shell (hidden when maximized); without them edge resizing breaks on Linux/X11.
  • Trade-off: macOS loses native traffic lights and system window-menu behaviors (move/align) in exchange for identical chrome across OSes. This is intentional.

11. Branding & Identifiers

  • Display brand is ModelSift (one word, capital S), tagline The local model advisor. The UI, docs, installer banners, tray menu, release name, and package productName use this spelling. Do not reintroduce "LLM Advisor" in user-facing copy.
  • Deliberately retained technical identifiers (do not rename without a migration plan):
    • Tauri identifier: dev.yoel3imari.llm-advisor — controls the app-data directory, single-instance identity, and updater continuity.
    • Cargo crate / binary: llm-advisor; npm package: llm-advisor. Release assets and install.sh APP_BIN still use it.
    • Data directories: ~/.local/share/dev.yoel3imari.llm-advisor/, ~/Library/Application Support/dev.yoel3imari.llm-advisor/, %APPDATA%\dev.yoel3imari.llm-advisor\.
    • localStorage keys: llm-advisor-sidebar-collapsed, llm-advisor.chat.sessions.v1, llm_advisor_theme, and the demo's llm_advisor_* keys.
  • GitHub repository: renamed to yoel3imari/modelsift. GitHub redirects the old yoel3imari/llm-advisor slug indefinitely, unless a new repo is created with the old name. Never reuse llm-advisor for a different repository; old installers, updater endpoints, catalog sync, and raw install URLs depend on that redirect.
  • Packaging paths: productName is now ModelSift, so Tauri uses /usr/lib/ModelSift for Linux resources, ModelSift.desktop/ModelSift.app, and kebab-cases the deb/rpm package name to model-sift; Windows installs under ModelSift. Every sidecar lookup and uninstall path keeps the old llm-advisor locations as fallbacks so upgrades from 0.2.x keep working. scripts/post-bundle.sh and both uninstall scripts must continue to handle both names. The deb declares replaces/conflicts and the rpm declares obsoletes for llm-advisor so the package manager replaces the pre-rename package instead of failing on the shared /usr/bin/llm-advisor path.
  • Updater note: changing productName changes the macOS .app directory name, the Windows install/registry identity, and the Linux package name. Keep the identifier stable and verify an in-place upgrade from the last LLM Advisor release on every target before shipping the rename; users may otherwise end up with both builds installed.

12. Headless CLI, Admin API & Runtime Ownership

  • Shared core: crates/app_core owns settings, library, downloads, inference lifecycle, gateway lifecycle, catalog sync, and factory reset. src-tauri is a thin adapter (AppState derefs to AppCore) and crates/cli is the second consumer. New orchestration logic belongs in app_core, not in Tauri commands or the CLI.
  • Frontend contract: the Tauri event bridge must keep emitting the existing names and payloads: download-complete (entry_id, filename, size_bytes), download-failed (entry_id, reason), catalog:updated (count, etag), and settings-changed (the settings object). The tray tooltip is updated from CoreEvent::GatewayStarted.
  • Admin API: routes live under /__modelsift on the same listener and are disabled when no AdminBackend is installed (404). Authentication is a bearer token stored in <data>/admin.token (0600, 32 random bytes hex-encoded) and compared in constant time. Keep it loopback-only; never log the token.
  • Ownership: <data>/runtime.lock is an exclusive fs2 advisory lock held for the lifetime of the gateway; <data>/runtime.json publishes owner, pid, gateway_port, version, and started_at. stop_gateway() removes runtime.json only while the process still holds the lock, so a desktop app that failed to acquire the gateway never deletes a CLI daemon's state. Only RuntimeOwner::Cli daemons accept POST /__modelsift/shutdown; desktop-owned gateways return HTTP 409.
  • Ports: external clients still use the strict 13370 default; do not auto-increment on collision. --port 0 exists for tests and binds an ephemeral port, which must be read back from runtime.json (the gateway returns the actual bound port).
  • CLI: crates/cli builds a single modelsift binary. A second [[bin]] name = "llm-advisor" collides with Tauri's mainBinaryName, so the compatibility alias is an install-time symlink created by install.sh --with-cli only when llm-advisor is not already the desktop binary. Exit codes are part of the contract: 0 success, 1 general error, 3 not running, 4 port/ownership conflict, 5 model error.
  • Detached mode: modelsift serve --detach spawns itself in a new process group, redirects stdout/stderr to <data>/logs/modelsift.log, and waits for the admin API (and --model, when given) before returning. The child must clean up runtime.json on every exit path, including failed initial model loads and SIGTERM.
  • Release plumbing: .github/workflows/release.yml publishes raw CLI binaries named modelsift-<target-triple> (.exe on Windows). install.sh --with-cli resolves that asset and installs it to ~/.local/bin (or --cli-dir). Keep the asset name and installer lookup in sync.

13. Runtime Abstraction, Jobs & Workspaces

  • Two execution models, one core:
    • Long-lived chat and embedding servers stay in server_manager (ServerManager::route_model remains the capability-aware entry point). Do not move them into the job queue.
    • One-shot engine work (verification today; transcription, image generation, speech next) goes through crates/runtime::JobQueue. run_process spawns the engine binary, streams stdout/stderr into the job log, parses progress, and kills the process group on cancellation.
  • Engine detection: crates/runtime/src/engine.rs defines EngineKind / EngineCapability / EngineRegistry::detect. Detection looks for bundled sidecars and well-known binary names. Missing engines must remain visible with an actionable hint; never hide a capability because its binary is absent.
  • Durable queue: JobQueue persists records to <data>/jobs.json. Statuses serialize lowercase (queued, running, succeeded, failed, cancelled); kinds are model_verify, engine_run, and hardware_benchmark. Concurrency is bounded, history is pruned, and jobs left running by a crash are recovered as failed on startup. JobQueue::submit receives an Arc<JobContext> and returns artifacts.
  • Paths: job logs live in <data>/logs/jobs/<job-id>.log, artifacts in <data>/artifacts/<job-id>/. Never write job output into the model library or the workspace tree directly; register it through the workspace store when the job belongs to a workspace.
  • First real job: AppCore::start_library_verify re-hashes installed models, parses GGUF headers, and writes verify-report.json as an artifact. It is exposed as start_library_verify (Tauri), POST /__modelsift/jobs with {"kind":"model_verify"} (admin), and modelsift jobs verify [--wait] (CLI).
  • Hardware benchmark: JobKind::HardwareBenchmark runs three fixed prompts (latency, throughput, long-context prefill) per selected chat model through the running sidecar, measuring tok/s, TTFT, prefill speed, RSS and memory headroom. Results are written to <data>/benchmarks/<job-id>.json, mirrored to <data>/benchmarks/latest.json, and published as the benchmark-report.json job artifact. The composite score is 0.5*speed + 0.3*latency + 0.2*headroom (120 tok/s and 100 ms TTFT are full marks; headroom is a percentage of VRAM or RAM) with S/A/B/C/D/F grades at 90/80/70/60/50. Exposed as start_benchmark / get_latest_benchmark_report (Tauri), POST /__modelsift/benchmark and GET /__modelsift/benchmark/latest (admin), and modelsift benchmark [--model <id>] [--wait|--latest] (CLI).
  • Workspaces: crates/workspace::WorkspaceStore owns workspaces, sessions, artifacts and presets under <data>/workspaces/, plus <data>/workspace-presets.json. IDs and artifact filenames are validated for path traversal; keep those guards on every new entry point. WorkspaceKind is lowercase serde (general, notebook, writing, tts, image, video).
  • Cleanup: CleanupPolicy supports max_age_days, max_total_bytes, and delete_orphan_files (default true). gc() only walks registered workspace directories and reports workspaces_scanned, artifacts_deleted, orphan_files_deleted, and reclaimed_bytes. Factory reset clears jobs and workspaces through AppCore.
  • App core ownership: AppCore holds Arc<JobQueue>, Arc<WorkspaceStore>, and Arc<EngineRegistry> and implements gateway::AdminBackend. Tauri commands are thin adapters; new orchestration belongs in app_core so the CLI inherits it. WorkspaceStore::add_artifact exists but is not yet wired into AppCore; future engine jobs with a workspace_id must register their artifacts there.
  • Frontend contract: the Tauri app emits job-updated with the full JobRecord payload. The Workspaces view (src/views/WorkspacesView.tsx) polls while work is active and also listens for the event. The Benchmark view (src/views/BenchmarkView.tsx) polls get_job for the active benchmark, then loads get_latest_benchmark_report when it succeeds. Keep src/ipc/mock.ts in sync with new commands so npm run dev/Vitest remain functional; mockFactoryReset() must clear jobs, workspaces, sessions, presets and the latest benchmark report.
  • Admin routes (all under /__modelsift, bearer-token guarded): GET /engines, GET|POST /jobs, GET /jobs/{id}, GET /jobs/{id}/logs, POST /jobs/{id}/cancel, POST /benchmark, GET /benchmark/latest, GET|POST /workspaces, POST /workspaces/gc, GET|PATCH|DELETE /workspaces/{id}, GET|POST /workspaces/{id}/sessions, DELETE /workspaces/{id}/sessions/{sid}, GET /workspaces/{id}/artifacts, DELETE /workspaces/{id}/artifacts/{aid}?delete_file=, GET|POST /presets, DELETE /presets/{id}. Collection routes return { "engines": [...] }, { "jobs": [...] }, { "workspaces": [...] }, etc.; single-resource routes return the object directly.
  • CLI surface: modelsift jobs list|show|logs|verify|cancel, modelsift workspaces list|show|create|rm|gc, modelsift benchmark [--model <id>] [--context <n>] [--max-tokens <n>] [--wait|--latest], plus modelsift engines. jobs verify --wait and benchmark --wait poll the job and exit 5 on failure. Keep text output and --json envelopes ({"ok":true,"data":...}) stable.
  • Tests: cargo test -p runtime -p workspace -p app_core -p gateway -p modelsift-cli covers the queue, store, admin routes and CLI lifecycle; npm test covers the Workspaces view. Runtime process tests are Unix-gated (sh -c).

14. Non-LLM Catalog Entries (Image / Audio Models)

  • engine field: CatalogEntry.engine (domain::ModelEngine, serde snake_case, default llama_cpp) discriminates asset loaders: llama_cpp (GGUF), stable_diffusion (safetensors), whisper_cpp (legacy GGML .bin), piper (.onnx). Missing engine always means llama_cpp — keep the #[serde(default)] for backward compat with old catalogs and library records.
  • Relaxed architecture fields: n_layers / n_kv_heads / head_dim / context_train (and params_billions in the scripts) are only required for llama_cpp entries; image/audio entries may carry zeros. Relaxations live in catalog::parse_catalog_json, scripts/update-catalog.py::validate_entry, and scripts/build-catalog.mjs::validateCatalog — update all three together. quality_tier stays 1–5 everywhere.
  • GGUF-only checks: header cross-checks in the downloader and verify_model_file must be gated on entry.uses_gguf_header() (engine == llama_cpp). Whisper .bin files are GGML (lmgg magic), not GGUF; .onnx/.safetensors have no header at all. Those assets verify on size + sha256 alone.
  • Download naming: single-file destinations are {id}.{real extension from entry.filename} via downloader::asset_extension — never hardcode .gguf again. Existing GGUF installs keep their names.
  • Companion files: CatalogEntry.companion (CompanionFile) downloads a config next to the weights under its real basename (Piper requires <voice>.onnx.json beside <voice>.onnx). verify_model_file folds a missing/corrupt companion into the verify result.
  • Routing: ModelCapability::Chat matches only Text | Multimodal outputs; image/audio models are never chat-routable. AppCore::load_model rejects non-llama_cpp engines with an actionable "downloadable only" error — these engines have no run pipeline yet (no JobKind::EngineRun producers).
  • Frontend: src/types/domain.ts mirrors engine/companion as optional (older payloads default to llama_cpp). Tables render — for zero params/context, and the output-type filter already includes Image/Audio.
  • Catalog content: entries added so far — 10 whisper models (ggerganov/whisper.cpp, GGML .bin, metadata parsed from the GGML header), 7 Piper voices (rhasspy/piper-voices, .onnx + .onnx.json companion; configs are non-LFS so their tree API returns git oids, not sha256 — download and hash them to catalogue), 4 Stable Diffusion checkpoints (stabilityai/* + stable-diffusion-v1-5/*, safetensors; sizes/sha256 from HF LFS oids). Catalog total is 133 before the OCR entries.

15. Catalog Source Layout (Two Layers)

  • Source of truth is per-entry: crates/catalog/entries/<id>.json holds exactly one CatalogEntry object. The filename must equal <id>.json. Never hand-edit crates/catalog/catalog.json — it is generated.
  • Distribution artifact: crates/catalog/catalog.json is regenerated by node scripts/build-catalog.mjs (sorted by id, canonical key order shared with domain::CatalogEntry). It stays committed because it is embedded via include_str! and served for remote etag sync.
  • Commands:
    • node scripts/split-catalog.mjs [--force] — one-time migration from the monolith; refuses to clobber an existing entries directory unless forced.
    • node scripts/build-catalog.mjs — validate entries and regenerate catalog.json.
    • node scripts/build-catalog.mjs --verify-only — validate only, no writes.
    • node scripts/build-catalog.mjs --check — CI guard; fails when catalog.json is stale. Wired into .github/workflows/ci.yml.
    • python3 scripts/update-catalog.py --online [--only <id>] [--force] — refresh HF metadata/benchmarks into entry files, then regenerate catalog.json. Entries verified within --max-age-days (default 7) are skipped; the cache lives at crates/catalog/.catalog-cache.json and is gitignored.
  • Curated fields are never clobbered: update-catalog.py merges HF size/sha256/architecture/benchmarks into existing entries; tags, quality_tier, prompt_preset, pooling, engine, companion, and mmproj are preserved.
  • Canonical key order: id, repo_id, filename, family, params_billions, active_params_b, n_layers, n_kv_heads, head_dim, context_train, quant, file_size_bytes, sha256, gated, quality_tier, tags, benchmarks, shards, output_type, pooling, mmproj, engine, companion, prompt_preset. Unknown keys are appended alphabetically. Keep scripts/catalog-lib.mjs, scripts/update-catalog.py, and the Rust struct in sync when adding fields.
  • Validation parity: engine whitelist, GGUF-only architecture requirements, embedding pooling rules, companion/mmproj sha256 checks, quality_tier 1–5, and non-empty prompt_preset are enforced in catalog::parse_catalog_json, scripts/update-catalog.py::validate_entry, and scripts/build-catalog.mjs::validateCatalog — update all three together.

16. OCR via llama.cpp Multimodal VLMs

  • Catalog contract: OCR entries use engine: llama_cpp, output_type: multimodal, and must include a companion multimodal projector (mmproj). They remain standard llama.cpp models served by llama-server — no new engine is required (Track A).
  • Trained prompt_preset is load-bearing: Dedicated OCR VLMs (PaddleOCR-VL, GLM-OCR, DeepSeek-OCR, Dots.OCR, HunyuanOCR, LightOnOCR, Qianfan-OCR) require specific prompt formats (e.g. OCR: for PaddleOCR-VL, OCR for GLM-OCR/DeepSeek-OCR). Omitting or changing the prompt causes hallucinations or degraded output. The catalog entry's prompt_preset supplies the verified prompt string.
  • Sidecar architecture support: Supported architectures are integrated into upstream llama.cpp (PaddleOCR-VL #18825, GLM-OCR #19677, DeepSeek-OCR #17400, Dots.OCR #17575, HunyuanOCR #21395). The sidecar pin must cover these architecture PRs. GOT-OCR2.0 is excluded due to lack of upstream llama.cpp support.
  • Per-architecture caveats:
    • PaddleOCR-VL: upstream notes potential degraded performance at low quantization; catalog entry carries quality tier caveat.
    • DeepSeek-OCR: larger memory footprint; projector and context must fit host budget.
    • Dots.OCR / GLM-OCR: highly sensitive to prompt syntax; always default to catalog prompt_preset.
  • CLI interface: modelsift ocr <model-id> <image-path> [--json] [--prompt <text>] auto-loads the model with an ephemeral serve config if not currently serving, encodes image to a base64 data URI, executes chat completion via the OpenAI-compatible proxy, and outputs transcribed text (or JSON envelope).
  • Playground UX: QuickTestBar displays the "Extract Text (OCR)" preset chip whenever the active model has vision capability and the ocr tag. ChatView surfaces an inline warning banner if the OCR preset is activated before an image is attached.