fix(cache): serve published analytics while refresh runs

[salmonumbrella]

Archive queries read the last published analytics cache while a refresh builds in the background. Responses report the generation, publication time, staleness, pending additions, and build state.

• CLI queries wait when a cache build is required, then return rows or an error. --fresh includes writes committed before the request; if a build is already running, callers receive the queued follow-up job's ID. HTTP and MCP callers can poll that job and repeat the query.
• Automatic refreshes honor auto_build_cache, min_rebuild_interval, and retry delays. Deleted rows can remain visible until the replacement is published. Manual sync accepts --build-cache and --no-build-cache and leaves full cache checks to the background worker. Explicit cache flags for sync and sync-full require daemon API schema 2.31.0.
• Adds advanced MCP query_sql through a dedicated DuckDB instance confined to archive analytics files. MCP advertises it only for supported SQLite daemons. Restricted SQL setup and queries use the request context. Owner CLI and HTTP SQL retain their existing capabilities.

API schema 2.31.0 and generated clients cover freshness metadata, accepted builds, job status, and restricted archive SQL. The daemon retains the last 100 completed jobs plus active work; restarting clears that history. A completed check can reuse an unchanged publication.

Includes test timing fixes for Windows lifecycle checks and the timing lint rules.

Refs #912

[roborev-ci]

roborev: Combined Review (4688ce0)

Verdict: Changes require fixes for 2 findings.

Medium

• cmd/msgvault/cmd/serve.go:1975: The generic CLI proxy schedules a cache refresh for every listed sync command, including sync-circleback --probe and sync-notion-meetings --probe, even though probe modes only validate access and perform no archive writes. Detect probe mode before registering the deferred refresh, or have the subprocess report whether it actually wrote archive data before queuing a build.

  Reported by: codex

• cmd/msgvault/cmd/cache_jobs.go:97: Cache jobs run in an untracked goroutine, so daemon shutdown cancels the context without waiting for execute/cmd.Run to finish before closing the archive and returning. A build may still be terminating during cleanup or restart, leaving locks or staged cache files in an indeterminate state. Track active cache jobs with a wait group and wait for them after cancellation, bounded by the shutdown deadline, before closing the store.

  Reported by: codex

---

Reviewers: 2 done | Synthesis: codex, 6s | Total: 11m10s

[roborev-ci]

roborev: Combined Review (bd30f40)

Verdict: Changes require fixes for 2 findings.

Medium

• cmd/msgvault/cmd/cache_jobs.go:85: A cache build request arriving while another build is active is coalesced into the current job and discarded. A manual sync can commit new messages during the build, allowing it to publish an outdated snapshot without queuing a follow-up build. Track a pending follow-up refresh when writes occur during an active build, or recheck staleness after completion and enqueue another build before marking the refresh settled.

  Reported by: codex

• cmd/msgvault/cmd/cache_jobs.go:44: verifyWhenDue records lastVerification before the asynchronous scheduled build runs but never clears it when the build fails. For conversation-only drift, ordinary queries can suppress automatic verification for the full interval while serving stale data. Reset or otherwise invalidate lastVerification when a scheduled verification job finishes unsuccessfully, preferably with bounded retry/backoff.

  Reported by: codex

---

Reviewers: 2 done | Synthesis: codex, 6s | Total: 9m48s

[roborev-ci]

roborev: Combined Review (b838c0e)

Verdict: Changes require fixes for 3 findings.

High

• internal/mcp/handlers.go:161: The read-only query_sql MCP tool allows SELECT statements using DuckDB file-reading functions such as read_blob, read_text, or read_csv_auto, enabling authenticated clients to read arbitrary host files. Remove arbitrary SQL exposure or enforce an allowlist of archive analytics views and reject external file/table functions; alternatively use a restricted DuckDB connection with external access disabled.

  Reported by: codex (security)

Medium

• internal/query/duckdb.go:303: DuckDBEngine.QuerySQL reads the publication marker and executes Parquet-backed SQL without the shared cache read lock or view refresh, allowing concurrent publication to produce mixed, removed, or stale cache state. Acquire e.acquireCacheRead(ctx) for the full QuerySQL operation so the committed generation is validated and optional columns/views are refreshed before execution.

  Reported by: codex

• cmd/msgvault/cmd/serve.go:1085: Failed scheduled automatic builds set verificationRetryAt, but the stale-cache query path accepts scheduled automatic builds without checking that backoff, potentially launching a subprocess on every query after a persistent failure. Apply the retry cooldown before accepting scheduled automatic builds in the staleness path, or centralize scheduled-job acceptance so all automatic retry paths honor verificationRetryAt.

  Reported by: codex

---

Reviewers: 2 done | Synthesis: codex, 7s | Total: 10m47s

[wesm]

What trust level do you intend consumers of the new query_sql MCP tool to have?

The project treats CLI and HTTP API users as privileged, so file-reading SQL alone does not establish a vulnerability under that threat model. Extending that capability to MCP does raise a design question about consumers with different levels of trust.

A synthetic test against this PR confirmed that an MCP-only key can read an outside-archive file through read_text or read_blob, with MCP writes disabled. The same key was denied direct access to daemon SQL; MCP used its own daemon credentials. The MCP guide currently says it cannot read Google credentials.

Would you prefer to keep SQL as an explicitly trusted advanced capability, make it opt-in or restricted for less-trusted MCP consumers, or move it to a separate PR? This seems worth an explicit decision before changing the tool or treating the review's severity as settled.

[salmonumbrella]

@wesm you’re right. The goal of #912 and this PR is for archive queries to return promptly from a published cache generation, report freshness, and let callers request a background refresh. query_sql gives MCP clients that query path; it is not meant to give an MCP-only read-only key access to arbitrary host files.

I’d keep the MCP SQL tool, but restrict its DuckDB connection to archive analytics data, with external file access disabled. CLI and HTTP can retain their existing privileged behavior. If we cannot make that boundary reliable in this PR, I’d move the MCP tool to a follow-up rather than ship the file access. Your synthetic test makes this an actionable MCP trust-boundary issue.