Skip to content

fix(memory-store): supersession + per-fact provenance end-to-end - #36

Merged
Peefy merged 1 commit into
mainfrom
fix/supersession-and-provenance
Sep 11, 2026
Merged

fix(memory-store): supersession + per-fact provenance end-to-end#36
Peefy merged 1 commit into
mainfrom
fix/supersession-and-provenance

Conversation

@Peefy

@Peefy Peefy commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Summary

The schema already supported soft-deprecation (deprecated_at, deprecation_reason, superseded_by_summary_id plus a partial WHERE deprecated_at IS NULL index), and deprecateMessages was implemented on RawMessageStorageManager, but the search layer, CLI, and synthesis prompt never saw them. A user repro (three versions of a decision, two deprecated) saw all three tie at similarity 1.000 and the lex tiebreaker picked arbitrarily; per-fact provenance (metadata.source) was visible in list --json but stripped from search --json and from the LLM synthesis context. There was also no CLI deprecate subcommand, so users had to drop into JS to deprecate anything.

Changes

  • Plumb includeDeprecated end-to-end — adds the field to SearchInput / UnifiedMemorySearchInput, threads it through runSemanticSearchForEmbedding and runLexicalSearchForKeywords (both the dep functions and the SQLite fallbacks), and adds --include-deprecated to opencontext search (default off, preserves current-truth).
  • New opencontext deprecate CLI subcommand — mirrors add's shape; flags --user (required), --id (repeatable, required), --reason, --superseded-by, --json, --help. Calls getRawMessageManager().deprecateMessages(...).
  • Surface per-fact provenanceSearchEvidence.metadata now carries the underlying hit's metadata. CLI --context-only prints a source: <uri-or-em-dash> line per evidence item. buildSynthesisPrompt line format is now [n] (id, source=<uri-or-tier-or-'unknown'>, score=N.NNNN) snippet, with metadata.source taking precedence over the channel-level source.
  • Fix vec0 KNN deprecation-blindness — wraps searchChunksWithVectorTable in a 3-attempt widen-and-retry loop (cap 4096 to respect sqlite-vec's k value in knn query too large limit) so limit is preserved even when most top-K candidates are deprecated. Adds a cross-reference comment on the parent-level mirror in searchMessagesWithVectorTable so the two vec0 KNN paths stay in sync.

Verification

User's repro (clean SQLite-backed memory store):

=== step 1: ingest three versions ===
REST    = 31acd868-b328-4169-818a-f822ccd92001
GraphQL = ad80216b-3af6-4494-a4de-35d889947dd9
tRPC    = 16ef97bd-0969-4f16-9b11-9d9ce1d6fa10

=== step 2: deprecate the older two via the new deprecate CLI ===
deprecated 1 message: 31acd868-...
deprecated 1 message: ad80216b-...

=== step 3: idempotency check (re-deprecate REST) ===
{"ok":true,"exit":0,"count":0,"ids":["31acd868-..."],"deprecatedAt":...}

=== step 4: top-1 current-truth (default hides deprecated) — expect tRPC only ===
  count = 1
  - memory:16ef97bd sim=1.0000 src=meeting://2026-08-15 | Public API: tRPC
  evidence[0].metadata.source = "meeting://2026-08-15"

=== step 5: --include-deprecated audit view — expect all 3 ===
  count = 3
  - memory:16ef97bd sim=1.0000 src=meeting://2026-08-15 | Public API: tRPC
  - memory:31acd868 sim=1.0000 src=meeting://2026-03-01 | Public API: REST
  - memory:ad80216b sim=1.0000 src=meeting://2026-06-01 | Public API: GraphQL

=== step 6: --context-only shows per-fact source ===
# user=u1 query="Public API"
# count=1 (no synthesis call made)
[1.000] raw @ 2026-08-15T00:00:00.000Z
    id: 16ef97bd-...
    source: meeting://2026-08-15
    Public API: tRPC

Test plan

  • pnpm --filter @melandlabs/opencontext test — 109/109 pass (added 3 tests: default includeDeprecated is omitted, --include-deprecated forwards true, --context-only surfaces per-fact metadata.source)
  • pnpm --filter @melandlabs/memory-store test — 203/203 search tests pass
  • pnpm --filter @melandlabs/sqlite test — 23/23 pass
  • pnpm --filter @melandlabs/indexeddb test — 10/10 pass
  • User's repro: add × 3 → deprecate × 2 → search --limit 1 returns tRPC only; --include-deprecated returns all 3; idempotent re-deprecate reports count=0

The two pre-existing test failures on main (applicability-propagation.test.ts:292 readonly/mutable cast, and cli-shared.test.ts @melandlabs/rag/chroma-vector-store resolution) are unrelated to this change — both confirmed pre-existing via git stash round-trip.

🤖 Generated with Claude Code

The schema already supported soft-deprecation (`deprecated_at`,
`deprecation_reason`, `superseded_by_summary_id` plus a partial
`WHERE deprecated_at IS NULL` index), and `deprecateMessages` was
implemented on `RawMessageStorageManager`, but the search layer, CLI,
and synthesis prompt never saw them. A user repro (three versions of a
decision, two deprecated) saw all three tie at similarity 1.000 and the
lex tiebreaker picked arbitrarily; per-fact provenance
(`metadata.source`) was visible in `list --json` but stripped from
`search --json` and from the LLM synthesis context. There was also no
CLI `deprecate` subcommand, so users had to drop into JS.

Plumbs `includeDeprecated` through `SearchInput` →
`UnifiedMemorySearchInput` → the ANN/lexical deps and the SQLite
fallbacks. Adds `--include-deprecated` to `opencontext search` (default
off, preserves current-truth). Adds a new `opencontext deprecate`
subcommand that mirrors `add`'s shape and surfaces
`--reason` / `--superseded-by`. Surfaces per-fact `metadata.source`
through `SearchEvidence` → CLI `--context-only` output → the
`buildSynthesisPrompt` line format. Wraps `searchChunksWithVectorTable`
in a widen-and-retry loop so vec0 KNN preserves `limit` even when most
top-K candidates are deprecated; mirrors that pattern on
`searchMessagesWithVectorTable` (which already had a similar loop).

Verified via the user's repro (REST → GraphQL → tRPC): top-1 search
returns tRPC only, `--include-deprecated` returns all three, evidence
envelope carries `metadata.source = "meeting://2026-08-15"`, and
`deprecate` is idempotent. Existing tests pass (the only pre-existing
failures on `main` — `applicability-propagation.test.ts:292` and
`cli-shared.test.ts` rag resolution — are unrelated to this change).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@Peefy
Peefy force-pushed the fix/supersession-and-provenance branch from a464cb6 to d70e54f Compare September 11, 2026 04:27
@Peefy
Peefy merged commit 7afc98f into main Sep 11, 2026
13 checks passed
@Peefy
Peefy deleted the fix/supersession-and-provenance branch September 11, 2026 04:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant