Add memory supersede: atomic consolidate of multiple memories into one

[doctatortot]

The problem

During heavy use of omem I keep running into this workflow gap:

1. I store a long context as multiple small memories because of embedding context limits — e.g. seagate-zombie-luks part 1/3, part 2/3, part 3/3 (all-minilm-512 had 512-token ctx, forced ~1.2KB chunking).
2. Later I switch to a bigger embedder (nomic-embed-text, 2048 ctx) and re-save the same content as a single consolidated memory.
3. The three part-X-of-N fragments are still in the index. They show up in search, dilute results, and split attention.

The Memory schema already has superseded_by: Option<String> and a state field, but there's no API to set them — and no search filter that respects them.

This isn't hypothetical: I hit it three separate times in one session (seagate recovery, asd config, central-syslog) and ended up with 8+ orphaned fragments cluttering recall.

What I'd like to add

An atomic supersede operation that:

1. Creates a new memory (the consolidated one)
2. Marks N old memories as superseded by it (state = "superseded", superseded_by = <new_id>, superseded_at = now())
3. Default search/list filters exclude superseded entries
4. memory_get(id) still returns superseded memories (history preserved)
5. Power users can opt back in via include_superseded: bool on search/list

Design questions for you before I code

Endpoint shape — A1 or A2?

A1. POST /v1/memories grows an optional replaces: ["id1", "id2", ...] field. One round-trip, atomic by design.

POST /v1/memories
{
  "content": "consolidated content...",
  "tags": ["seagate"],
  "replaces": ["part1_id", "part2_id", "part3_id"]
}

A2. Separate POST /v1/memories/:id/supersede (or batch variant). More orthogonal but two round-trips and a window where the new exists but olds still show up.

My lean is A1 — matches how the workflow actually plays out, and atomicity comes free.

State + timestamp

• state = "superseded" (new enum variant alongside active)
• superseded_by = <new_id> (existing field)
• superseded_at = now() (new — cheap, useful for audit)

Idempotency / chains

Superseding an already-superseded memory: my call is reject with 409. Force the caller to either re-target the chain head or explicitly opt-in via a force_chain: bool flag. Quiet chain extension feels like a future-bug machine.

Filtering by default

• memory_search / memory_list: filter state == "active" by default. Add include_superseded: bool (default false) to opt in.
• memory_get(id): returns the memory regardless of state. The link is the link; you asked for it specifically.

Tag/source inheritance

My lean: pristine. The new memory's tags/source are whatever the caller passed; the replaces field is the only link to history. Caller can copy tags forward if they want.

Cross-tenant

Reject with 403. Obviously nonsensical, but worth being explicit in the validation.

Plugin tool surface

• memory_store grows replaces: [...] to mirror the API
• New memory_supersede(superseded_by, ids) tool for the rare bare-op case (no new memory, just retire some)

What this would touch

• domain/memory.rs — add superseded_at, possibly a MemoryState enum if not already
• store/lancedb.rs — vector_search + fts_search get a include_superseded: bool filter; new method supersede_batch(new_id, ids)
• api/handlers/memory.rs (or wherever POST /v1/memories lives) — accept replaces field, wrap creation + supersede in a single Lance transaction
• retrieve/pipeline.rs — plumb the filter through SearchRequest
• Plugin TS: memory_store schema gets replaces, new memory_supersede tool
• Tests at each layer

Happy to do all of it. Wanted to surface the design first so we agree on shape before I sink time into a 250-line PR you'd send back for restructure.

Which direction on A1 vs A2? And anything I'm missing in the questions above?

[yhyyz]

Great design writeup @doctatortot — we're on board with this feature. Here's our feedback:

Decision: A1

POST /v1/memories with an optional replaces field is the right call. Reasons:

1. Matches the actual workflow — users always want "replace these old fragments with this new consolidated one", not "supersede without replacement"
2. Internal precedent — the reconciler's handle_supersede (reconciler.rs:231) already does exactly this pattern: create new → mark old
3. Simpler for clients — one tool call from an LLM agent, no multi-step orchestration
4. No inconsistency window — avoids the gap where new and old memories are both visible in search

For the rare "retire without replacement" case, PUT /v1/memories/:id { "state": "superseded" } already works (we just landed memory_type/state update support via your PR #10).

Notes on your other design points

Point	Our take
Superseded state variant	✅ Agree — cleaner than reusing Archived
superseded_at field	Suggest reusing invalidated_at — it already exists on the Memory struct and the reconciler already sets it when superseding. Adding a second timestamp for the same event increases maintenance cost with no clear benefit.
409 on already-superseded	✅ Agree — prevents silent chain corruption
Default search/list excludes superseded	✅ Agree — extend existing state != 'deleted' filter to state NOT IN ('deleted', 'superseded')
memory_get still returns superseded	✅ Agree — history preserved
Tag inheritance: none	✅ Agree — caller controls new memory's tags explicitly
Cross-tenant: 403	Not needed actually — tenant isolation is physical (each tenant has its own LanceDB store). If a replaces ID doesn't exist in the caller's store, it's simply 404. No cross-tenant risk by design.
Plugin tools	✅ Agree — replaces as optional array on memory_store

One additional point

For replaces IDs that don't exist (typo, already deleted, etc.): suggest 400 with a clear error listing which IDs were not found, rather than silently ignoring them. This prevents users from thinking they superseded something when they actually didn't.

---

Go ahead and implement when ready — looking forward to the PR! 🚀