Skip to content

Let config.toml choose how much of each session the archive keeps - #1564

Open
naveenspark wants to merge 1 commit into
kenn-io:mainfrom
naveenspark:feature/usage-only-storage
Open

Let config.toml choose how much of each session the archive keeps#1564
naveenspark wants to merge 1 commit into
kenn-io:mainfrom
naveenspark:feature/usage-only-storage

Conversation

@naveenspark

@naveenspark naveenspark commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Adds archive_content to config.toml so an archive can keep less than everything.

archive_content = "transcripts"   # "full" (default), "transcripts", or "usage"

What you get

  • transcripts keeps the conversation and drops tool inputs and outputs. Search, transcript viewing, tool analytics, and the session tree keep working. On a 43 GB archive, tool payloads were 33 GB.
  • usage keeps only what token and cost reports need. About 41k sessions fit in 2.5 GB in the contributor's canary.
  • AGENTSVIEW_ARCHIVE_CONTENT covers a reporting archive in its own data directory.

Applying it to an existing archive

Set the key, restart the daemon, run agentsview sync --full. Back up first: dropped content cannot be recovered without the source files.

What transcripts is not

A storage policy, not a redaction guarantee. It removes the tool payload tables and structured inputs, drops rows and summaries that parsers mark as tool output, and rewrites the tool summaries parsers inline into message text. Text a provider wrote into a message body in its own format stays as written. usage stores no text at all.

Costs

  • Every archive re-parses once on upgrade (data version 98).
  • Under transcripts, checks that read tool payloads see nothing, and archived sessions from RooCode, Kilo Legacy, gptme, OpenHands, and Aider parsed by earlier releases lose some text on rebuild, since those parsers mixed tool output into ordinary rows.
  • Under usage, insights, recall, eval trajectories, and pin notes are refused or dropped.

@roborev-ci

roborev-ci Bot commented Sep 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (39eb413)

Verdict: Changes require fixes before approval; usage-only redaction is incomplete and can misclassify automated sessions.

High

  • cmd/agentsview/sync.go:516-553, internal/remotesync/import.go:183-196, internal/server/huma_routes_sync.go:97-107: Remote SSH/HTTP imports and lazily created server sync engines do not receive UsageOnly. With AGENTSVIEW_USAGE_ONLY=1, they may persist full transcripts, including private prompts, tool payloads, results, and content-derived titles. Propagate UsageOnly through all remote transport/import structures and on-demand engine configurations, and test that remote imports contain no transcript content.

Medium

  • internal/sync/usage_only.go:24-27: Database writes recompute is_automated from stored text. Because usage-only storage blanks message content and FirstMessage, automated sessions are reclassified as interactive, producing incorrect automated-scope usage reports. Preserve the parser-derived classification or bypass text-based recomputation for usage-only writes.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 10m6s

@roborev-ci

roborev-ci Bot commented Sep 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (2751bd1)

Verdict: Usage-only enforcement is incomplete across ingestion paths, and automation classification can be corrupted.

High

  • internal/remotesync/import.go:183-195
    Remote SSH/HTTP imports construct EngineConfig without propagating UsageOnly, allowing raw transcripts and tool payloads into usage-only archives. Propagate UsageOnly through remote sync/import configurations, including prepared HTTP imports.

Medium

  • internal/sync/engine.go:918-921, internal/sync/usage_only.go:5-14
    Usage-only redaction removes text required by updateSessionAutomationFromMessagesTx, causing automated one-shot sessions to be reset as non-automated. Preserve the parsed automation classification or skip automation recomputation for usage-only writes.

  • cmd/agentsview/import.go:34-54, internal/importer/importer.go:216-254, internal/server/huma_routes_sync.go:215-224, internal/artifact/import_session.go:305-312, internal/server/upload.go:238-242
    Direct, on-demand, artifact, and upload import paths bypass usage-only redaction and can persist prompts, tool payloads/results, and session titles. Apply consistent redaction across all ingestion paths or enforce it at a centralized storage boundary.


Reviewers: 2 done | Synthesis: codex, 9s | Total: 11m33s

@roborev-ci

roborev-ci Bot commented Sep 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (cc62663)

Verdict: High-severity privacy issue and medium-severity automation misclassification remain in usage-only mode.

High

  • Remote imports bypass usage-only modeinternal/remotesync/import.go:188-196
    Remote SSH/HTTP imports construct sync.EngineConfig without propagating UsageOnly, causing full transcripts, tool payloads, titles, and secret findings to be persisted despite usage-only mode. Propagate UsageOnly through all remote importer, CLI, server, and prepared-import configurations.

Medium

  • Automated sessions are misclassified when content is strippedinternal/sync/usage_only.go:9-13
    Clearing FirstMessage and message content causes database write paths to recompute is_automated as false, affecting usage filters and one-shot counts. Preserve the parser’s automation classification or skip content-based recomputation during usage-only writes.

Reviewers: 2 done | Synthesis: codex, 8s | Total: 9m45s

@roborev-ci

roborev-ci Bot commented Sep 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (25fca2a)

Verdict: Two medium-severity issues remain in usage-only storage and resync handling.

Medium

  • internal/sync/engine.go:850-924, 2959-2963NewEngine ignores an already-enabled database usage-only policy when cfg.UsageOnly is false. Full resync can therefore open a regular replacement database and persist transcript/tool content. Derive usage-only mode from both configuration and database.UsageOnlyStorageEnabled(), and use it for engine flags and replacement DB selection.

  • internal/db/usage_only_storage.go:42-45, 96-111 — Automation classification only promotes sessions. After a second user turn, is_automated remains true even though IsAutomatedTranscript requires false when user_message_count > 1, producing incorrect usage breakdowns. Demote sessions once the stored user-message count exceeds one and add a regression test.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 18m10s

@roborev-ci

roborev-ci Bot commented Sep 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (2254e21)

Verdict: High-severity usage-only data leaks remain possible during resync replacement handling.

High

  • internal/sync/engine.go:851-925: NewEngine derives e.usageOnly only from cfg.UsageOnly, ignoring an already usage-only database opened via db.OpenUsageOnly() or EnableUsageOnlyStorage(). Resync replacements may therefore use db.Open and admit full transcript content. Derive and consistently use an effective usage-only setting from both sources.

  • internal/sync/engine.go:3009 and 3383: Usage-only resyncs copy raw session, message, tool-call, tool-result, and secret-finding fields through CopyTrashedDataFrom and CopyOrphanedDataFromExcluding, bypassing usage-only projection. Make these copy operations usage-only aware or compact copied rows before swapping in the replacement database.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 12m49s

@roborev-ci

roborev-ci Bot commented Sep 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (cb97315)

Verdict: Usage-only filtering is bypassed during resync preservation, allowing transcript and tool payloads to re-enter the archive.

High

  • internal/sync/engine.go:2960-2964; internal/db/orphaned.go:1883-2035
    Usage-only resync preservation uses raw orphan/trash-copy and related SQL paths, bypassing the usage-only projection. This can copy transcript content, thinking text, tool inputs/results, events, secret findings, and content-bearing insight/recall fields into a usage-only archive. Make preservation helpers policy-aware and apply the same storage projection before insertion.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 12m49s

@roborev-ci

roborev-ci Bot commented Sep 2, 2026

Copy link
Copy Markdown

roborev: Combined Review (b74946b)

Verdict: Two medium-severity issues require attention; both affect usage-only archive privacy and enforcement.

Medium

  • internal/sync/engine.go:2971-2975, 3020, 3394-3498 — Usage-only resync copy paths use raw ATTACH SQL and bypass redaction, potentially restoring message text, tool payloads, findings, titles, insight text, and recall snippets from trashed, orphaned, or auxiliary records. Apply usage-only projections to every copy path or restrict copies to approved metadata and usage fields.

  • internal/sync/engine.go:852-854, 923-926NewEngine derives usage-only behavior solely from cfg.UsageOnly, ignoring databases already configured for usage-only storage. This can allow resync with a normal replacement database and violate the database’s monotonic storage policy. Derive the effective mode from both configuration and database state.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 19m46s

@wesm

wesm commented Sep 2, 2026

Copy link
Copy Markdown
Member

looking

@wesm wesm self-assigned this Sep 2, 2026
@wesm wesm changed the title Add usage-only archive storage for reporting Add archive_content policy for transcript-only and usage-only archives Sep 3, 2026
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (b59259a)

Verdict: High-risk issue identified, plus one Medium-severity policy-boundary issue.

High

  • internal/sync/engine.go:18359-18389 — The usage-only OpenCode archive guard falls back to (ordinal, role), but parsed OpenCode messages do not populate SourceUUID. A later row with the same ordinal can replace a missing historical assistant row, allowing nondecreasing token counts to pass and historical usage data to be deleted. Preserve the OpenCode message ID as SourceUUID, or fail closed when stable identities cannot be matched.

Medium

  • internal/sync/engine.go:3343, internal/sync/engine.go:3466 — Usage-only full resyncs copy cached insights and recall entries from the old archive. These tables contain content-bearing fields, violating the documented usage-only archive boundary. Skip or project these auxiliary tables during usage-only rebuilds, including when switching an existing full archive to usage-only.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 16m36s

@wesm
wesm force-pushed the feature/usage-only-storage branch from b59259a to eef1ebe Compare September 3, 2026 02:09
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (eef1ebe)

Verdict: Changes are not safe for transcript-only or usage-only archive policies because plaintext and derived sensitive data can remain accessible.

High

  • internal/db/archive_content.go:128-153, internal/db/archive_content.go:319-334 — Transcript-only projection clears tool-call payloads and result events but leaves Message.Content intact. Standalone RoleTool messages from providers such as Omnigent and Devin may contain tool output, including secrets, which remain stored and searchable. Clear content and content lengths for standalone tool-result messages during write-time projection and resync-copy SQL.

Medium

  • internal/db/archive_content.go:309-313 — Transcript-only orphan/trash copies retain copied signal columns and secret findings derived from discarded inputs or results. Existing version markers prevent recomputation. Recompute derived data from projected messages, or clear/invalidate content-derived signals and findings during transcript-only copies.

  • internal/db/archive_content.go:337-386 — Usage-only projection does not handle pinned_messages.note. Resync paths preserve plaintext notes, and PinMessage stores new notes unchanged, allowing transcript text or secrets to remain accessible through the pins API. Clear or reject pin notes under ArchiveContentUsage, including during resync pin copies.


Reviewers: 2 done | Synthesis: codex, 9s | Total: 13m9s

@wesm
wesm force-pushed the feature/usage-only-storage branch from eef1ebe to 19ff7f5 Compare September 3, 2026 02:47
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (19ff7f5)

Verdict: Two medium-severity issues require attention; no security vulnerabilities were identified.

Medium

  • internal/server/huma_routes_sync.go:205-225archive_content is applied only at daemon startup. agentsview sync --full requests sent to a running daemon use cached configuration, so policy changes silently fail to reproject existing content. Reload or pass the archive policy with resync requests, or require and document a daemon restart.

  • internal/db/archive_content.go:144-148 — Transcript projection clears InputJSON but retains FilePath, which may expose paths derived from discarded tool inputs. Clear FilePath or explicitly classify and document it as retained metadata.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 26m14s

@wesm
wesm force-pushed the feature/usage-only-storage branch from 19ff7f5 to 95f293b Compare September 3, 2026 03:14
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (95f293b)

Verdict: One medium-severity issue found; transcript-only storage may retain tool output in non-tool-role messages.

Medium

  • internal/db/archive_content.go:137-141, 334-342 — Transcript projection only clears content for Role == "tool". RooCode may emit tool results as RoleUser or RoleSystem, and the ATTACH-copy path has the same gap. Mark all tool-result carrier messages consistently and clear their content and length while preserving genuine transcript text.

Reviewers: 2 done | Synthesis: codex, 6s | Total: 14m5s

@wesm
wesm force-pushed the feature/usage-only-storage branch from 95f293b to e1dab18 Compare September 3, 2026 11:36
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (e1dab18)

Verdict: Medium-severity issue found; transcript-only archiving can retain Kilo command output.

  • Mediuminternal/db/archive_content.go:164-170 (also 167-170): The transcripts projection fails to classify Kilo/Kilo Legacy’s unpaired command_output fallback rows as tool output. These are stored as user-role messages with nonempty content and no SourceSubtypeToolResult, so command output—including sensitive file contents or secrets—may be persisted. The same predicate affects copied-session/orphan resync paths.

    Fix: Mark fallback rows with SourceSubtypeToolResult and handle equivalent legacy rows in the copied-session SQL projection during migration/resync.


Reviewers: 2 done | Synthesis: codex, 9s | Total: 23m26s

@wesm
wesm force-pushed the feature/usage-only-storage branch from e1dab18 to 5a7ac56 Compare September 3, 2026 12:23
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (5a7ac56)

Verdict: High-severity gaps remain in transcript-only sanitization, with additional medium-severity bypasses in tool-output and derived-content storage.

High

  • internal/db/archive_content.go:170-180, internal/parser/openhands.go:375-386
    Unpaired OpenHands observations are stored as ordinary user messages without SourceSubtypeToolResult or ToolResults, so transcript-only archives retain their tool-output text. Mark the fallback as SourceSubtypeToolResult and add coverage.

Medium

  • internal/db/archive_content.go:175-176, internal/db/archive_content.go:362-365
    Transcript-only projection fails to remove gptme tool output represented as assistant content because role:"tool" records are mapped to RoleAssistant. Mark these records with SourceSubtypeToolResult and handle them in both normal and legacy resync-copy projections.

  • internal/db/insights.go:94-111, internal/db/recall.go:174-201, internal/db/recall_extract.go:1376-1423
    Usage-only policy can be bypassed by direct insight and recall extraction APIs, which may persist transcript-derived prompts, content, snippets, and evidence. Recall/import placeholder sessions also bypass sessionForStorage. Enforce the policy at these write boundaries and route placeholder inserts through the storage projection.


Reviewers: 2 done | Synthesis: codex, 9s | Total: 13m28s

@wesm wesm changed the title Add archive_content policy for transcript-only and usage-only archives Let config.toml choose how much of each session the archive keeps Sep 3, 2026
@wesm
wesm force-pushed the feature/usage-only-storage branch from 5a7ac56 to 806900c Compare September 3, 2026 13:07
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (806900c)

Verdict: High-severity privacy-boundary gaps remain in legacy tool-output copying and several direct-write paths.

High

  • internal/db/archive_content.go:382-386
    Orphaned/trashed session copies scrub only rows marked role = 'tool' or source_subtype = 'tool_result'. Legacy gptme, OpenHands, Kilo, and RooCode tool output stored as other roles can survive into narrowed archives, including secrets. Apply provider-aware legacy classification or fail closed for unclassified result-like content.

Medium

  • internal/db/archive_content.go:393-410
    Transcript-policy copies reset only quality_signal_version; stale tool-failure, outcome, and quality counters derived from discarded payloads remain visible. Clear all transcript-derived signal fields or recompute them before exposure.

  • internal/db/archive_content.go:49-61, internal/db/sessions.go:1500
    Usage-only sessionForStorage removes titles and secret fields but preserves session-level derived signals. Direct UpsertSession calls can therefore persist transcript-derived signal data in usage-only archives. Clear all derived signal columns in the projection or enforce this atomically on upsert.

  • internal/db/recall_eval_ingest.go:70, internal/db/insights.go:203, internal/db/recall.go:218
    Raw eval ingestion and ATTACH-based insight/recall copies bypass usage-only rejection enforced by higher-level APIs, allowing transcript-derived text into usage archives. Enforce the policy at each bulk-write boundary or centralize writes through guarded methods.

  • internal/db/recall_extract.go:1495
    CommitExtractedUnit directly calls insertExtractedRecallEntriesTx, bypassing the archive-policy check. Recall titles, bodies, triggers, and evidence can be stored in usage-only archives. Enforce the policy at this boundary or in the shared insertion path, and ensure rejected work does not advance extraction state.


Reviewers: 2 done | Synthesis: codex, 11s | Total: 16m40s

@wesm
wesm force-pushed the feature/usage-only-storage branch from 806900c to a823321 Compare September 3, 2026 14:00
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (a823321)

Verdict: One medium-severity issue found; otherwise, no concrete security vulnerabilities identified.

  • Mediuminternal/db/archive_content.go:159-197; internal/parser/aider.go:317-322: Aider tool-channel output is stored as ordinary assistant text, causing transcript-only archives to retain command/tool output. Mark these messages as SourceSubtypeToolResult or classify them explicitly during projection.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 32m47s

@wesm
wesm force-pushed the feature/usage-only-storage branch from a823321 to 5dd57ee Compare September 3, 2026 15:07
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (5dd57ee)

Verdict: High-severity tool-input leakage remains, with additional medium-severity consistency and stale-data risks.

High

  • internal/db/archive_content.go:153-197, 169, 400-404 — Transcript-only storage clears structured tool inputs but preserves tool inputs embedded in assistant Message.Content. Sensitive commands and arguments remain searchable and may be retained during orphan resync copies or PostgreSQL export. Strip tool-use renderings while preserving ordinary assistant text, and apply the same projection to SQL copy paths.

Medium

  • internal/db/artifact_import_session.go:75-83 — Transcript-only artifact imports preserve manifest-derived tool-failure and quality signals for content that is no longer stored. Recompute signals from projected messages, or clear and mark them stale.

  • internal/sync/engine.go:18472-18481 — OpenCode usage-only protection compares only retained tool-call counts, so a replacement call with the same count can silently lose subagent relationships and usage linkage. Compare stable tool-use and subagent identifiers.

  • internal/db/recall.go:509-547SupersedeRecallEntry bypasses the usage-only archive guard and can insert transcript-derived titles, bodies, and evidence. Enforce the derived-text policy in the supersede or shared insertion path.

  • internal/sync/engine.go:3340 and internal/vector/export.go:179-195 — Policy resync replaces sessions.db but leaves stale full-content data in vectors.db, allowing pg push --full to export content removed by the archive policy. Invalidate or rebuild the vector mirror before allowing export.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 39m7s

@wesm
wesm force-pushed the feature/usage-only-storage branch from 5dd57ee to 1ee3d47 Compare September 3, 2026 17:05
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (1ee3d47)

Verdict: Medium-severity issues remain in transcript-only archive redaction.

Medium

  • Provider-specific tool renderings can leak sensitive inputsinternal/db/archive_content.go:202 (invoked at :167). Transcript-only redaction handles generic and Cortex renderings but misses Codex, Gemini, and Antigravity formats, allowing commands and tool arguments to remain stored. The same issue affects orphaned-session copies during resync. Share canonical provider redaction logic with storage projection and add coverage for custom renderers.

  • Unpaired Codex subagent notifications retain full outputinternal/parser/codex.go:807. Fallback messages are stored as ordinary user messages without SourceSubtypeToolResult or ToolResults, so transcript-only filtering preserves their subagent output. Mark fallback messages as tool output and handle legacy copied rows during resync.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 24m39s

@wesm
wesm force-pushed the feature/usage-only-storage branch from 1ee3d47 to e9752c0 Compare September 3, 2026 21:00
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (e9752c0)

Verdict: High-severity archive redaction and vector-retention issues remain.

Critical

None.

High

  • Copilot tool inputs may remain in transcript archives.
    Locations: internal/db/archive_content.go:221-231, internal/parser/content.go:494-514
    VS Code/Visual Studio Copilot parsers embed commands or invocation messages in assistant text, but their parsed tool calls lack matching renderings in the redaction candidates. Clearing tool_calls.input_json therefore leaves sensitive commands, tokens, URLs, and arguments in messages.content.
    Fix: Preserve the exact rendering or add provider-specific redaction candidates; fail closed by removing unmatched tool-call bodies.

  • Narrowing the archive policy does not scrub existing vector data.
    Locations: internal/sync/engine.go:2969, cmd/agentsview/main.go:275-277
    Rebuilding the SQLite archive with a narrower policy does not invalidate or fully refresh vectors.db. Existing plaintext snippets may remain searchable and can be exported to PostgreSQL, allowing recovery of data excluded by the new archive policy.
    Fix: Include the archive policy in vector-mirror identity, force a full refresh that removes stale rows and vectors, and block vector search/export until completion.

Medium

  • Redaction may modify assistant prose instead of the tool invocation.
    Location: internal/db/archive_content.go:196-204
    strings.Replace(..., 1) replaces the first matching rendering. If assistant prose quotes the same text before the tool block, the prose is altered while the sensitive invocation remains.
    Fix: Track rendering offsets during parsing or associate replacements with the tool-call segment.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 25m52s

@wesm
wesm force-pushed the feature/usage-only-storage branch from e9752c0 to 97c7622 Compare September 3, 2026 22:24
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (97c7622)

Verdict: The archive-content policy has multiple text-retention gaps that can expose transcript data or user queries.

High

  • internal/db/archive_content.go:215-234 and internal/parser/content.go:488-524 — Transcript-only redaction can leave provider-specific tool commands, summaries, or patterns in Message.Content after structured inputs are cleared. This affects OpenHands and may also affect Kimi, Grok, and other custom renderers. Credential-bearing commands could remain accessible through transcript APIs. Record exact parser renderings or fail closed on unmatched tool text, with ingestion and resync coverage.

Medium

  • internal/db/archive_content.go:79-91 — Usage-only projection clears SessionName but leaves PreserveSessionName enabled, allowing Codex upserts at sessions.go:1561-1568 to restore content-derived session titles. Clear PreserveSessionName or make preservation policy-aware.

  • internal/db/recall_query_events.go:52-96RecordRecallQueryEvent remains writable under archive_content = "usage" and persists arbitrary user queries in query_text, violating the usage-only no-text guarantee. Reject or redact these writes.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 26m9s

@wesm
wesm force-pushed the feature/usage-only-storage branch from 97c7622 to 8f31fd7 Compare September 4, 2026 00:35
@roborev-ci

roborev-ci Bot commented Sep 4, 2026

Copy link
Copy Markdown

roborev: Combined Review (8f31fd7)

Verdict: One high-severity redaction gap remains for Grok tool-call summaries.

  • High — internal/db/archive_content.go:216-235: Transcript redaction does not recognize Grok parser-generated summaries such as [backend web_search] search: <query> or [backend code_interpreter] <code>. Because these tool calls lack Rendering metadata and match none of the candidate renderers, sensitive tool arguments may remain in stored assistant message text.
    • Fix: Add Grok rendering metadata and a matching redaction candidate, or explicitly redact Grok backend summaries before persistence. Add behavioral tests for search queries and code.

Reviewers: 2 done | Synthesis: codex, 6s | Total: 13m29s

@wesm
wesm force-pushed the feature/usage-only-storage branch from 8f31fd7 to e0c3854 Compare September 4, 2026 01:49
@roborev-ci

roborev-ci Bot commented Sep 4, 2026

Copy link
Copy Markdown

roborev: Combined Review (e0c3854)

Verdict: The archive redaction boundary is incomplete; provider-specific renderings and pin notes can leak transcript content in usage/transcript archives.

Critical

None reported.

High

  • internal/parser/openhands.go:327-356 — OpenHands action commands are embedded in message text, but tool calls lack rendering metadata. Transcript projections cannot reconstruct and redact formats such as [Bash]\n$ cat /secret, leaving tool arguments in archive_content = "transcripts" archives. Record the exact rendering and support it in both direct writes and orphan-copy projection.

Medium

  • internal/db/archive_content.go:608-654 — Orphan/trash copying loses the parser-only Rendering field and does not recognize Grok’s custom [backend ...] format. Grok queries, URLs, or code may remain after input_json is cleared. Persist renderer metadata or add Grok-specific reconstruction with a fail-closed fallback.

  • internal/db/orphaned.go:1204 (pin copy at 1254-1258) — Usage-only resync copies pinned_messages.note verbatim. Pin notes may contain transcript excerpts or secrets, bypassing the usage archive policy. Reject or discard notes in usage mode and copy pins with note = NULL or omit them in every resync path.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 24m3s

@wesm
wesm force-pushed the feature/usage-only-storage branch from e0c3854 to 6a53660 Compare September 4, 2026 02:21
@roborev-ci

roborev-ci Bot commented Sep 4, 2026

Copy link
Copy Markdown

roborev: Combined Review (6a53660)

Verdict: One medium-severity rendering issue found; no critical or high-severity issues.

Medium

  • internal/parser/vscode_copilot.go:536, used by internal/parser/visualstudio_copilot.go:1362: Visual Studio Copilot passes a copied ParsedToolCall slice to formatVSCodeCopilotToolCalls, leaving the returned call’s Rendering empty. In transcript-only mode, formats such as [Bash: run_in_terminal]\n$ ... are not recognized, so the raw command remains in message content. Return the mutated slice from the Visual Studio trace path or assign the generated rendering back to the original call.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 42m35s

@wesm
wesm force-pushed the feature/usage-only-storage branch from 6a53660 to 47ece0d Compare September 4, 2026 03:35
@roborev-ci

roborev-ci Bot commented Sep 4, 2026

Copy link
Copy Markdown

roborev: Combined Review (47ece0d)

Verdict: Two medium-severity issues remain in transcript redaction during resync and copied-message handling.

Medium

  • internal/db/archive_content.go:228 — Source-less resyncs cannot use Rendering, and fallback candidates omit provider-specific formats. Shorter header matches may redact only headers while leaving command bodies or patterns in archives. Require the longest complete match and include provider-specific rendering candidates.

  • internal/db/archive_content.go:619 — Copied tool-call redaction only processes assistant messages with has_tool_use = 1, unlike normal projection, which handles any message containing tool calls. Legacy or inconsistent rows may retain inline tool content after payload columns are cleared. Select copied messages based on joined tool-call rows instead.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 39m8s

@wesm
wesm force-pushed the feature/usage-only-storage branch from 47ece0d to 61e9f52 Compare September 4, 2026 04:35
@roborev-ci

roborev-ci Bot commented Sep 4, 2026

Copy link
Copy Markdown

roborev: Combined Review (61e9f52)

Verdict: Medium-severity issues remain in usage-only rewrites and transcript redaction.

Medium

  • Session titles can persist in usage-only archives
    internal/db/sessions.go:1391-1395, internal/db/sessions.go:2844-2863
    Usage-only projections clear titles in memory, but upserts omit display_name, and incremental updates do not clear title fields. Existing titles may therefore survive rewrites. Clear display_name, first_message, and session_name in usage-only upsert and incremental write paths while preserving full-mode rename behavior.

  • Pinned-message notes can persist in usage-only rewrites
    internal/db/messages.go:1357-1381, internal/db/messages.go:1825-1837
    Message replacement restores existing pin notes unchanged, allowing free-text notes to survive usage-only rewrites. Clear pin notes during usage-only replacement, including diff and batch paths.

  • Transcript-only resync may retain OpenHands tool commands
    internal/db/archive_content.go:615-658; internal/parser/content.go:491-524
    OpenHands terminal renderings such as [Bash]\n$ <command> are not recognized for redaction, so secrets may remain in messages.content after input_json is cleared. Add OpenHands-specific rendering/provenance handling or fail closed when content cannot be recognized, and add a resync test containing a secret.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 28m48s

@wesm
wesm force-pushed the feature/usage-only-storage branch from 61e9f52 to e4eb810 Compare September 4, 2026 12:33
@roborev-ci

roborev-ci Bot commented Sep 4, 2026

Copy link
Copy Markdown

roborev: Combined Review (e4eb810)

Verdict: High-severity transcript redaction flaw and two medium-severity usage-only cleanup gaps require attention.

High

  • internal/db/archive_content.go:204-207 — Transcript redaction replaces only the last matching tool rendering. Repeated assistant text can leave parser-inserted renderings intact, potentially exposing tool arguments in transcript-only storage. Replace all parser-generated occurrences or track rendering offsets during parsing.

Medium

  • internal/db/project_identity.go:785-845; internal/db/sessions.go:2372-2390 — Project-identity upserts and RefreshSessionName can bypass usage-only cleanup, leaving text-bearing metadata such as names, first messages, and pinned notes after switching from full storage. Use a shared cleanup helper for every usage-only session write.

  • internal/db/sessions.go:1516-1519UpsertSession clears secret-related fields only in its in-memory projection; its SQL does not update those columns or call settleUsageOnlySignalsTx. Existing signal values and secret findings may persist in usage-only archives. Apply the cleanup during existing-session usage-policy writes.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 29m31s

Operators who only want token and cost reports, and operators whose
archive is dominated by tool output, both need to store less than the
full transcript. `archive_content` in config.toml selects one of three
policies: "full" (default), "transcripts", or "usage".
`AGENTSVIEW_ARCHIVE_CONTENT` sets the policy when the config file does
not, for a dedicated reporting archive in its own data directory.

The transcripts policy keeps message and thinking text, titles, and tool
call metadata while dropping tool inputs and results, which on a large
archive are most of the file. The usage policy keeps only the session and
message rows that token and cost reports need, classifies automation
while the source text is still available, guards partial OpenCode
sources against erasing retained usage, and settles transcript-derived
signal and secret state so startup backfills do not revisit rows.

The database handle is the single authority for the policy. Open
variants and the sync engine only tighten it. Every write path projects
sessions and messages before rows are written, and resync copies
archived and trashed sessions through the same projection in SQL because
those copies bypass the write path. Signals and secret findings are
computed from the projected messages so a recompute from stored rows
reproduces them. A usage rebuild also leaves cached insights and recall
entries behind, since both hold transcript-derived text.

OpenCode messages now record their storage message ID as source_uuid so
archive guards match rows by identity instead of ordinal. The data
version moves to 98 so existing rows are re-parsed to carry it.

An existing archive is not rewritten when the policy changes; a full
resync applies it to every session, including ones whose source files
are gone.

Squashed commits:

- Add usage-only archive storage
- stabilize vector shutdown cancellation test
- preserve usage-only mode in no-sync daemon
- perf(storage): compact usage-only archives
- fix(storage): preserve usage-only reporting semantics
- fix(storage): preserve subagent usage links
- fix(sync): preserve compact usage history
- fix(storage): settle compact signal state
- fix(storage): satisfy compact call lint
- fix(storage): guard optional message sanitization
- Merge remote-tracking branch 'upstream/main' into feature/usage-only-storage
- feat(storage): choose archive content policy in config.toml
- fix(storage): match OpenCode rows by message ID and skip derived text

Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
Generated with Claude Code
Co-authored-by: Claude <noreply@anthropic.com>
@wesm
wesm force-pushed the feature/usage-only-storage branch from e4eb810 to 4b525fe Compare September 4, 2026 14:59
@roborev-ci

roborev-ci Bot commented Sep 4, 2026

Copy link
Copy Markdown

roborev: Combined Review (4b525fe)

Verdict: Medium-severity issue found in transcript-only archive sanitization.

  • Mediuminternal/db/archive_content.go:217-224
    Sanitization can modify Message.Content without updating ParsedToolCall.Rendering, causing the exact rendering match to fail. The function may then clear InputJSON while retaining sensitive inline tool arguments in transcript archives, search results, or APIs. Sanitize Rendering consistently or fail closed by removing unmatched inline renderings.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 34m29s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants