Skip to content

feat(setup): add provider defaults and lane status - #753

Open
salmonumbrella wants to merge 7 commits into
kenn-io:mainfrom
salmonumbrella:feat/issue-634-smart-defaults
Open

feat(setup): add provider defaults and lane status#753
salmonumbrella wants to merge 7 commits into
kenn-io:mainfrom
salmonumbrella:feat/issue-634-smart-defaults

Conversation

@salmonumbrella

Copy link
Copy Markdown
Contributor

What changed

  • Add msgvault setup providers: reads VOYAGE_API_KEY, MISTRAL_API_KEY, and OPENAI_API_KEY (and probes a local Ollama server at [chat].server when no hosted key is present), prints a plan, asks once per hosted provider, and writes recommended values for every lane that is still unset through the ETag-guarded config editor:
    • Voyage: [vector] on with api_format = "voyage-contextual", voyage-context-4, dimension 1024, run_after_sync plus a 15-minute cron; [vector.people] on; [vector.multimodal] on once the probe manifest exists at <home>/voyage-capabilities.json (enabling it earlier makes the daemon refuse every vector lane, so setup writes only the schedule and prints the probe command until then).
    • Mistral: [attachments.documents] on with the recorded postures (standard / default-opt-out unless --document-retention / --document-training say otherwise); document vectors on when a text lane exists. Uploads stay manual behind the probe manifest and documents consent-mistral --yes.
    • OpenAI: the text lane on text-embedding-3-small only when no Voyage key is present, and the people sweep onboarded as profile openai on gpt-5.6-luna at medium reasoning through the existing person provider add, check, consent, and use gates.
    • No hosted key: the documented local Ollama text lane and a loopback openai_chat sweep profile on [chat].model, when the server is reachable.
  • Add msgvault setup status: one report of every lane (text search, semantic people search, visual attachments, documents, document vectors, people sweep, activity projection, media policy) with provider, model, recorded consent, schedule, the reason a lane is off, the command that turns it on, and the MCP tools live under that configuration. --json on both commands.
  • Document api_format, [vector.people], [vector.multimodal], and [activity] in the configuration reference, add a Recommended Configuration page with the file setup writes, and describe both commands in the CLI reference and setup guide.

Why

Every lane from the #534 sequence shipped correctly as opt-in with its own table, key variable, consent step, and build command. A fresh install had to assemble six tables by hand and nothing reported what was on, what was off, and why. Setup now chooses values from the keys the operator has; it adds no lane, no provider, and no consent semantics, and a configured lane keeps its model because switching the embedding policy invalidates the index.

Usage

export VOYAGE_API_KEY=... MISTRAL_API_KEY=... OPENAI_API_KEY=...
msgvault setup providers --dry-run   # plan and provider disclosures, nothing written
msgvault setup providers             # one y/N per provider, then config.toml is written
msgvault setup status                # what is on, what is off, and the next command

Refs #634. Refs #589 #599 #611 #616 #620 #628.

🤖 Generated with Claude Code

@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (0bb6944)

Verdict: Address one High-severity and two Medium-severity issues before merging.

High

  • Unauthenticated remote Ollama fallback can exfiltrate archive datacmd/msgvault/cmd/setup_providers.go:443-454 (specifically 443-450)
    When no hosted key is present, setup may select any reachable Ollama server, including a non-loopback endpoint, for text embeddings without authentication or disclosure. Archived message text could therefore be sent to a remote server. Require a loopback endpoint or explicit authenticated remote configuration and disclosure.

Medium

  • Document text is omitted from the OpenAI consent disclosurecmd/msgvault/cmd/setup_providers.go:689-691
    When documents are enabled, document_text is added to the people-sweep inference profile, but the disclosure mentions only conversation and meeting text. Include document text in the disclosure.

  • Embedding-only Ollama model may be selected for chatcmd/msgvault/cmd/setup_providers.go:627-639
    If only nomic-embed-text is available and the configured chat model is unavailable, setup selects the first model for people-sweep chat, causing the chat check to fail and potentially leaving partial configuration. Require an available configured chat model or filter out embedding-only models with an actionable pending state.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 14m48s

@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (b67fa33)

Verdict: Needs changes — two medium-severity issues were identified.

Medium

  • cmd/msgvault/cmd/setup_providers.go:862-875 — Configuration changes are persisted before people-provider onboarding completes. If onboarding, consent, checking, or selection fails, setup returns an error while leaving other lanes enabled and configured, causing later runs to treat them as already configured. Stage onboarding before publishing configuration or roll back all setup edits on failure.

  • cmd/msgvault/cmd/setup_providers.go:296-309, 607-610 — The people-sweep plan adds document_text before consent decisions. If the Mistral gate is declined, the document lane is skipped but document_text remains in the OpenAI inference profile and disclosure. Recompute dependent plans after gate decisions and remove document sources when the document lane is declined.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 10m44s

@wesm wesm self-assigned this Sep 5, 2026
@wesm

wesm commented Sep 5, 2026

Copy link
Copy Markdown
Member

looking

salmonumbrella and others added 2 commits September 4, 2026 19:34
Configure unset retrieval and people lanes from available providers with
explicit consent and recommended defaults. Report which lanes are active
and the steps needed to finish setup.

- feat(setup): keep provider onboarding output consistent with the pass
- fix(setup): keep the local fallback local and disclose every source

Refs kenn-io#634

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Generated with Codex
A failed people-provider check, consent, or selection left other lanes
configured, so a retry treated the incomplete setup as finished. Restore
all setup config edits on failure, including a newly created file, using
the published file identity to preserve concurrent edits.

Recompute dependent lanes after a provider is declined. Declining Mistral
must also disable document vectors and remove document_text from the
people-sweep profile and its subsequent disclosure.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@wesm
wesm force-pushed the feat/issue-634-smart-defaults branch from b67fa33 to 92f2710 Compare September 5, 2026 00:36
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (92f2710)

Verdict: Medium-severity issues remain around backend readiness, credential reporting, and sensitive-data consent.

Medium

  • cmd/msgvault/cmd/setup_providers.go:209-214, 444 — Setup selects pgvector from the database DSN without verifying that the binary includes pgvector support, potentially producing a configuration the daemon cannot start with. Check backend availability or leave the lane pending with rebuild guidance.

  • cmd/msgvault/cmd/setup_lanes.go:298-312, 340-346, 400-409 — Lane status reports visual, document, and people-sweep lanes as enabled without checking credential environment variables. Missing provider keys therefore produce no readiness warning. Report missing credentials for hosted lanes.

  • cmd/msgvault/cmd/setup_providers.go:638, 723-726setup providers automatically enables and consents to the people-sweep profile with allowSensitive: true, while the disclosure does not clearly state that sensitive archive excerpts may be sent and is suppressed in JSON mode. Require a separate explicit sensitive-content opt-in or disclose the policy in all modes; keep allow_sensitive false without that opt-in.


Reviewers: 2 done | Synthesis: codex, 9s | Total: 13m44s

Keep the people sweep pending until --allow-sensitive explicitly permits
sensitive archive excerpts and personal inferences. Describe that policy
in both human and JSON plans; --yes alone must not authorize it.

Leave vector lanes pending when the binary lacks the database backend,
and report missing credential environment variables for hosted lanes so
the setup report does not imply that unusable providers are ready.

Fix the new documentation links to use the /docs/ URL prefix required by
the published site and its link checker.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (e8eb885)

Summary verdict: 3 medium-severity issues require attention.

Medium

  • cmd/msgvault/cmd/setup_lanes.go:384-394: Document semantic search requires separate document_embedding and query_embedding consents, but setup only recommends msgvault documents vectors consent --yes. Track both consent purposes, disclose query-text egress, and include --purpose queries --yes.

  • cmd/msgvault/cmd/setup_providers.go:524-528, 582-586: Enabling a disabled lane overwrites existing retention and training postures with command defaults. Preserve non-empty postures and apply defaults only when unset; require an explicit override flag.

  • cmd/msgvault/cmd/setup_lanes.go:199-212, cmd/msgvault/cmd/setup_providers.go:404-424: Provider detection uses unsafe URL substring matching, allowing attacker-controlled endpoints to be classified as OpenAI or local. Parse URLs and compare exact trusted hostnames; treat other non-loopback endpoints as custom hosted providers requiring explicit handling.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 32m10s

Document search sends both archive text and query text to an embedding
provider. Report their separate consent records so users can see which
permission is still missing and how to grant it.

Enabling a lane must not replace saved retention or training assertions
with command defaults. Preserve each assertion unless its own flag is
explicitly supplied.

Recognize providers by exact URL hosts, not substrings. Leave dependent
lanes on custom hosted endpoints for explicit configuration instead of
extending the data sent under another provider's disclosure.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (e03ba4f)

Verdict: Medium-severity issues remain in provider readiness reporting, daemon onboarding, and schedule preservation.

Medium

  • cmd/msgvault/cmd/setup_lanes.go:279-291 — Lane status can report text search, people search, and document vectors as on when required provider credentials are unavailable. Mark lanes pending and propagate dependency readiness.

  • cmd/msgvault/cmd/setup_providers.go:1008; cmd/msgvault/cmd/person_provider_setup.go:1001-1029 — People-provider onboarding through an existing daemon does not forward the configured credential environment variable, causing checks to fail when the daemon lacks that environment. Forward only the required credential using the controlled environment mechanism.

  • cmd/msgvault/cmd/setup_providers.go:383-393 — Explicit empty or false schedule settings are treated as unset and overwritten with defaults. Detect explicitly present TOML keys and preserve opt-outs. The same applies to multimodal schedules.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 16m14s

Missing embedding credentials make text search unavailable to its people
and document consumers too. Report those lanes as pending instead of on.
Preserve explicit empty cron and false after-sync settings when applying
text and visual defaults so setup does not undo scheduling opt-outs.

A running daemon may lack the caller's provider key during onboarding.
Forward only that key for a check pinned to the saved profile fingerprint,
and preserve the caller context on the generated command. Keep ordinary
checks on daemon-owned credentials and reject keys for other profiles.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (f06f780)

Verdict: Two medium-severity readiness and configuration issues were identified.

Medium

  • cmd/msgvault/cmd/setup_lanes.go:310-321 — Consent-gated lanes remain marked on when required consent is missing or unknown, despite runtime gates preventing operation. Set the lane state to pending in this case.

  • cmd/msgvault/cmd/setup_providers.go:489-500 — Local Ollama setup changes the embedding endpoint and format but leaves an existing api_key_env set, potentially causing a disabled profile to remain permanently pending due to stale credentials. Clear api_key_env for unauthenticated local Ollama, or skip automatic selection when it was explicitly configured.


Reviewers: 2 done | Synthesis: codex, 9s | Total: 29m7s

Enabled configuration does not make a consent-gated lane ready. Report
missing or unreadable consent as pending, and keep document vectors
pending until both document and query consent are active.

Local Ollama selection uses an unauthenticated endpoint. Clear the old
embedding credential reference so an unused hosted key cannot leave the
new local lane pending.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (29dffae)

Verdict: Two medium-severity issues require attention; no high or critical findings were reported.

Medium

  • cmd/msgvault/cmd/setup_lanes.go:279-289 — Configured vector lanes are reported as on without verifying that the selected SQLite/pgvector backend is compiled into the binary. This can cause daemon startup failure and contradict documented pending/rebuild behavior. Check backend availability and provide rebuild guidance when unavailable.

  • cmd/msgvault/cmd/setup_providers.go:631 — Document-vector follow-ups only include consent commands, which do not build the vector index. Add msgvault documents vectors build after both consent commands so semantic search is usable.


Reviewers: 2 done | Synthesis: codex, 9s | Total: 18m29s

Configured vector lanes cannot run when the binary lacks their database
backend. Use the same compiled-backend check for setup and status, and
carry the pending state to dependent lanes with rebuild guidance.

Consent records authorize document-vector work but do not create an
index. Include the build command after both consent steps so the setup
follow-ups finish the indexing work.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (ed3ac6e)

Verdict: Two medium-severity issues require attention; no critical or high-severity findings were reported.

Medium

  • cmd/msgvault/cmd/setup_lanes.go:214-218 — Visual consent is reported as active whenever any active/building generation is consented, without matching the current multimodal configuration or capability-manifest policy. After configuration changes, runtime consent is rejected while setup status still reports the lane as on. Require generation and consent-policy fingerprints to match; otherwise report pending with the rebuild/consent command.

  • cmd/msgvault/cmd/setup_providers.go:874-877, 939-943 — The literal unknown posture is treated as an explicit assertion. Enabling documents with the documented unknown defaults creates an enabled lane that documents consent-mistral refuses, while later setup runs treat it as already enabled and do not apply posture flags. Treat unknown as unset or keep the lane pending until explicit postures are supplied, and allow posture flags to repair an enabled unknown configuration.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 29m20s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants