Productionize AI orchestration, safety, usage metering, and evaluations

[Vinosaamaa]

Parent

#107 (identified by #238)

What to build

Make the AI Study Assistant safe, measurable, and reliable with a real provider. The runtime must use a production model only when configured, stream grounded answers with citations, resist untrusted instructions, meter actual token/cost usage, and route uncertain or unsafe requests to human support.

Acceptance criteria

• Production provider/model configuration is validated at startup; incompatible defaults such as an OpenAI provider with an Ollama model name fail fast.
• The orchestration loop has bounded steps/time/tokens, structured tool contracts, cancellation, retries only where safe, and deterministic audit records.
• Responses cite only retrieved, currently authorized evidence; unsupported claims are refused or labeled uncertain and can create a Support Question/TA handoff.
• Input/output safety covers prompt injection, harmful content, sensitive data, and abuse without sending unauthorized Course content to third parties.
• Usage persists provider, model, prompt/completion/cached tokens, latency, outcome, and estimated cost; quotas enforce measured usage rather than invocation count alone.
• Instructors get truthful usage/error/quality dashboards without access to private learner content beyond policy.
• A versioned evaluation suite measures groundedness, citation correctness, refusal, scope isolation, handoff quality, latency, and cost against release thresholds.
• Provider outage, timeout, quota exhaustion, and safety rejection produce useful UI states and never fabricate an answer.

Blocked by

• Replace demo embeddings with scoped production vector retrieval #247

[Vinosaamaa]

Owner direction for the launch program, recorded during the current implementation:

• Make product, design and technical decisions autonomously; the existing design documents may be revised when evidence supports a better product.
• Use free hosting and free resources for the initial release. No paid tier, automatic recharge or billable provider usage is authorized.
• No hosting or provider infrastructure exists yet. Account provisioning and deployment remain part of the launch work.
• Keep AI provider/model choice configurable, including OpenAI, Anthropic, xAI/Grok, compatible endpoints and a no-provider/local option.
• Investigate subscription-backed access only through each provider's supported product integration. Do not describe chat subscriptions as API credits or collect consumer session cookies.
• Retain issue-linked implementation, detailed design and implementation records, system review, exact-head verification and Engineering receipts.

#242 remains the active integration slice. Hosting feasibility is owned by #243; provider abstraction and usage modes are owned by #248. These directions supersede the old design's assumption that deployment must preserve all separate runtime processes.

[Vinosaamaa]

Frontend integration requirement from timeout review: the current parser resolves EOF without requiring an authoritative complete event and ignores the new status/error events. The hook discards partial text in finally but can remain in streaming state with no recovery. Partial answers are not persisted. Integrate the catalog and {code,status,message} errors, reject EOF before complete, clear draft/citations, and offer Source only recovery for GENERATION_ALREADY_ATTEMPTED without a second provider call. Keep returned persisted-answer audit metadata authoritative on replay.