Research: local-model selection, fine-tuning & optimization for grandplan's tasks

[wimaan3]

Context

grandplan drives several distinct local-LLM tasks through Ollama on the 16 GB RAM, no-GPU target — capture/organize, structural placement (LlmPlacer), contextual reconcile, entity/relation extraction, and edit/update detection — all emitting structured JSON. Model choice today is a single hand-picked default (gemma4:e4b for capture; qwen2.5:14b earmarked for the future KB agent) with no measured basis for which local model is actually best per task, whether a fine-tuned/optimized small model beats a larger general one, or what the real quality vs latency vs RAM trade-offs are.

docs/research/REFRESH-2026.md surveyed the 2026 small-model field (Gemma 3 4B, Phi-4-mini 3.8B, Qwen3, the Q4_K_M-on-CPU consensus) but stopped at what exists — not what is measured best for our specific JSON tasks. This is a research/evaluation spike to put model selection on a measured footing and scope whether fine-tuning/optimization is worth it. Must stay offline-reproducible and within the 16 GB no-GPU constraint.

Threads

1. Selection — define the task set + a per-task quality metric, build a candidate matrix (model × quant), and rank on quality / tok-s / peak RAM on a CPU-only box. Extend the existing offline eval harness (scripts/eval_retrieval.py, scripts/bench_reconcile.py, ADR-0011 organize-quality-vs-cost) to the organize / placement / reconcile outputs, not just retrieval.
2. Fine-tuning — assess whether a small base (Gemma 3 4B / Phi-4-mini / Qwen3) fine-tuned via LoRA/QLoRA on grandplan's specific structured-JSON tasks beats a larger general model. Cover: training-data generation (distill from a stronger local model + reuse the gold/eval sets), base-model license compatibility (Apache/MIT/Gemma terms), and GGUF export back into Ollama. Keep the whole loop offline + reproducible.
3. Optimization — quantization sweep (Q4_K_M vs Q5/Q6/Q8), grammar-constrained / structured output, num_ctx tuning, prompt compression, keep-alive / warm-start (cold-start currently causes the placement/reconcile timeout→heuristic fallback on first run), and — if it fits the budget — speculative decoding; all measured against the same harness.

Acceptance criteria

• A documented per-task metric + reproducible eval harness covering organize / placement / reconcile (extends the retrieval eval).
• A candidate model × quant matrix with measured quality / tok-s / peak RAM on a 16 GB no-GPU box, and a recommended default per task.
• A fine-tuning feasibility writeup: data-generation plan, LoRA/QLoRA recipe, base-model license check, expected lift vs cost — ending in a go/no-go.
• An optimization findings writeup (quant + structured-output + ctx/prompt/warm-start) naming the winning settings.
• Findings land as an ADR and/or a docs/research/ note; update the model-strategy defaults if the data warrants.