Skip to content

fix(memory): improve child retrieval and document BEAM v2 - #29

Open
hrw991009 wants to merge 5 commits into
mainfrom
fix/core-memory-child-retrieval-reranking
Open

fix(memory): improve child retrieval and document BEAM v2#29
hrw991009 wants to merge 5 commits into
mainfrom
fix/core-memory-child-retrieval-reranking

Conversation

@hrw991009

Copy link
Copy Markdown
Collaborator

Closes #27

Summary

This PR improves OpenContext's core raw-message retrieval strategy and documents the resulting BEAM 128k evaluation.

The changes are divided into two main parts:

  1. Core retrieval and context optimization
  2. BEAM v1/v2 diagnostic reports

Retrieval optimizations

  • Preserve each complete RawMessage as the parent record.
  • Split oversized messages into independently embedded search children using a 400-token target and 80-token overlap.
  • Restore either the complete short parent or a continuous matched-child window for long messages.
  • Generate semantic and lexical candidates independently and merge them with RRF.
  • Overfetch four times the requested limit, deduplicate by parent message, rerank the candidate window, and apply Top-K only after reranking.
  • Add an optional local Transformers.js cross-encoder reranker.
  • Expose backend, channel, candidate, reranker, and semantic-degradation diagnostics.
  • Use the same child indexing and retrieval semantics across SQLite, Chroma, LanceDB, and Milvus adapters.
  • Keep the BEAM adapter limited to one upstream turn per RawMessage without benchmark-specific chunking or ranking rules.
  • Preserve the official BEAM Answerer, Judge, prompt, and nugget scoring behavior.

Reports

  • Translate the original BEAM 128k v1 diagnostic report into English.
  • Add a new v2 report covering:
    • parent and child index construction;
    • retrieval and reranking evidence;
    • token and context reduction;
    • category and failure-stage analysis;
    • comparison with the v1 baseline;
    • remaining limitations and recommended follow-up work.

BEAM 128k result

Metric v1 v2 Change
Nugget Mean 0.5305 0.6120 +0.0816
Pass Rate 56.75% 66.75% +10.00 pp
Passed 227/400 267/400 +40
Total recorded tokens 31,456,385 4,006,056 -87.27%
Mean Answerer prompt tokens 74,191 6,213 -91.62%
Mean context characters 343,439 25,958 -92.44%
MRR 0.2958 0.4783 +0.1825

The v2 run completed all 400 questions with zero final execution errors. Semantic and lexical candidates were visible for all questions, and the local reranker changed the candidate order for all 400 retrievals.

Remaining issues

  • Multi-evidence retrieval remains the main bottleneck: 57 retrieval misses and 38 partial retrievals remain.
  • Only 41.41% of retrieval-applicable questions received every required source.
  • Event ordering, summarization, and abstention remain weaker categories.
  • The current pointwise reranker improves relevance order but does not optimize evidence-set coverage.
  • Local reranking adds approximately 2.85 seconds per question.
  • The complete BEAM run covered the SQLite path only; Chroma, LanceDB, and Milvus were not evaluated against live external services.
  • The v2 result was produced from an uncommitted working tree, so a future formal baseline should record the dirty diff or run from a clean commit.

Validation

  • @melandlabs/ai-rag: 10 tests passed
  • @melandlabs/memory-store: 176 tests passed
  • @melandlabs/shared: 73 tests passed
  • @melandlabs/sqlite: 23 tests passed
  • @melandlabs/rag: 30 tests passed
  • @melandlabs/indexeddb: 10 tests passed
  • @melandlabs/okf: 140 tests passed
  • BEAM: 6 tests passed
  • TypeScript typecheck passed for shared, sqlite, rag, indexeddb, okf, memory-store, opencontext, and BEAM
  • git diff --check passed

No paid benchmark or external backend service is exercised by the test suite.

@hrw991009
hrw991009 requested review from Peefy September 3, 2026 03:11
@Peefy

Peefy commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

PR conflict. Could u help fix it?

Propagate trusted applicability through hybrid retrieval and stabilize empty-stdin Codex preflight on macOS.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[benchmark][BEAM] Address retrieval and context construction issues exposed by the 128k run

2 participants