chore(ai): explicit model policy — router/chat/program/summarize + preview-vs-stable

[JeffOtano]

Context

`convex/ai/providers.ts:23-93` ships one generic primary/fallback pair per provider. Two problems:

1. No tier-by-task policy. Current: everything hits Sonnet 4.6 / Gemini 3 Flash / GPT-5.4 primary. World-class split: cheap for routing/classification/summarization, mid for chat, premium for planning/programming.

2. Preview models in production. `primaryModel: "gemini-3-flash-preview"` — we ship a preview model as the default to our Gemini BYOK majority. One silent API change from Google breaks the whole product.

Files

• `convex/ai/providers.ts:23-93` — current config
• `convex/ai/coach.ts:192-249` — `buildCoachAgents` / `buildCoachAgentsForProvider` consumers
• New (probably): `docs/ai/model-policy.md` or an ADR

Acceptance

• Write a short policy doc: which model runs which task tier across all 4 providers
• Tiers: `router` (cheapest — Haiku 4.5 / Gemini Flash-lite / GPT-5.4-nano / OpenRouter auto), `chat` (mid — Sonnet 4.6 / Gemini Pro / GPT-5.4-mini), `programming` (premium — Opus 4.6 / Gemini Pro + high effort / GPT-5.4), `summarize` (router tier)
• Preview-vs-stable rule: `*-preview` models MUST NOT be the default primary. Can only be opt-in via user setting or feature flag
• Move `gemini-3-flash-preview` off the default path; replace with stable `gemini-3-flash` (or latest stable)
• Wire `prepareStep` to swap models by tier (pairs with perf(ai): adaptive thinking via prepareStep (effort=high on program_week) #205, perf(ai): prepareStep dynamic tool subset per agent step #210)
• Document per-tier cost in the policy doc for future reference

References

• OpenAI Compare Models
• Google Gemini Models
• Anthropic pricing tiers
• LLM routing: 85% cost reduction case study

Related: #190 (intent routing is one consumer of this policy), #205 (effort per step), #210 (prepareStep tool subset)

[e6o]

Good call on splitting model policy by task type. We see this pattern a lot — teams sending everything to GPT-4/Sonnet when classification and routing tasks work fine on much cheaper models.

The tier approach you're describing (cheap for routing/classification, mid for chat, premium for programming) is exactly right. A rough guide based on what we've seen:

• Classification/routing/summarization: GPT-4o-mini or Gemini Flash — ~$0.15/1M input tokens. Works great for these tasks.
• Chat/conversational: GPT-4.1 or Sonnet — good balance of quality and cost.
• Planning/coding/reasoning: Premium models only when needed.

The other thing to watch: the gemini-3-flash-preview issue you flagged is real. Preview models can change behavior without notice, which breaks production at the worst time. Pin to stable versions and test previews in staging only.

We built InferCut to automate this kind of tier-based routing — it sits as a proxy and routes each request to the most cost-effective model that meets your quality bar. Zero risk: if it can't route cheaper, calls pass through unchanged. Might be useful as a complement to the manual model policy you're building here.