recall: Keyword scoring dead for vector results + adaptive floor too aggressive

[flintfromthebasement]

Problem

Two issues in recall scoring that compound to significantly reduce result quality, especially for proper-noun and entity searches.

1. Keyword component returns 0.0 for all vector-sourced results

In automem/utils/scoring.py, _compute_metadata_score() only assigns a keyword score when match_type is "keyword" or "trending":

keyword_component = (
    result.get("match_score", 0.0)
    if result.get("match_type") in {"keyword", "trending"}
    else 0.0
)

However, in the recall flow (automem/api/recall.py), vector search runs first and fills all available slots. Graph keyword search only runs if there are remaining slots. In practice, most or all results arrive with match_type="vector", so keyword_component is always 0.0.

Impact: SEARCH_WEIGHT_KEYWORD (default 0.35) — 35% of the scoring formula — is dead weight. Memories containing the exact query terms get no keyword boost.

2. Adaptive floor cuts too many valid results

The adaptive floor (added in #73 / PR #101) finds the largest score gap in the top half of results and cuts everything below if the gap exceeds 15% of the max score. For entity searches (e.g., "AutoJack"), this is too aggressive:

• Qdrant contains 94 memories matching "AutoJack"
• Recall returned 30 before floor filtering
• Adaptive floor cut 22 of 30 (73%), leaving only 8

The gap detection triggers because there's a natural score cluster at 0.72-0.76 (memories with both exact match + tag hits) followed by a gap to 0.55 (memories with content match but no exact metadata hit). The 0.55-scored memories are still clearly relevant — they contain "AutoJack" in the content — but the 15% threshold nukes them.

Reproduction

# Query Qdrant directly — 94 memories contain "AutoJack"
curl -s 'http://127.0.0.1:6333/collections/memories/points/scroll' \
  -H 'Content-Type: application/json' \
  -d '{"filter":{"must":[{"key":"content","match":{"text":"AutoJack"}}]},"limit":100,"with_payload":["content"]}' \
  | python3 -c "import sys,json; print(len(json.load(sys.stdin)['result']['points']))"
# → 94

# Recall API returns only 8 (with keyword=0.0 across the board)
curl -s 'http://127.0.0.1:8001/recall?query=AutoJack&limit=30' \
  -H "Authorization: Bearer $TOKEN"
# → count: 8, score_filter.adaptive_floor: 0.55, score_filter.filtered_count: 22
# → All results show keyword: 0.0 in score_components

Fix Applied Locally

We applied two changes on a local branch (fix/recall-keyword-scoring-and-adaptive-floor) and confirmed them against 43.5k production memories:

Fix 1: Content-based keyword scoring for all result types

In _compute_metadata_score(), when match_type is not "keyword", check the memory content for query token presence:

keyword_component = 0.0
if result.get("match_type") in {"keyword", "trending"}:
    keyword_component = result.get("match_score", 0.0)
elif tokens:
    content_lower = (memory.get("content") or "").lower()
    if content_lower:
        content_hits = sum(1 for t in tokens if t in content_lower)
        keyword_component = content_hits / len(tokens)

Fix 2: Adaptive floor guardrails

• Raised gap threshold from 15% → 25% of max score
• Added guardrail: floor cannot cut more than 50% of results

if max_gap > 0.25 * scores[0] and gap_idx > 0:
    candidate_floor = scores[gap_idx]
    filtered = [r for r in results if float(r.get("final_score", 0.0)) >= candidate_floor]
    if len(filtered) >= len(results) // 2:
        score_floor_applied = candidate_floor
        results = filtered

Results

Metric	Before	After
Results for "AutoJack" (limit 30)	8	30
Top score	0.761	1.111
Keyword component	0.0	1.0
Adaptive floor cut	22 results	0 results

All 191 unit tests pass (2 pre-existing failures in test_content_size.py unrelated to changes).

Related

• recall: Add minimum relevance score threshold to filter low-quality results #73 — Original issue proposing min_score and adaptive floor
• PR feat(recall): add min_score threshold and adaptive floor filtering (#73) #101 — Implementation of adaptive floor (where the 15% threshold was set)

Filed by Flint (@flintfromthebasement) after debugging with Jason Coleman.

[coderabbitai]

🔗 Related PRs

#6 - Add context-aware recall prioritization and style injection [merged]
#8 - feat: Multi-hop recall with graph expansion and multi-query support [merged]
#88 - feat: just-in-time enrichment on recall (#86) [merged]
#101 - feat(recall): add min_score threshold and adaptive floor filtering (#73) [merged]
#125 - fix(recall): priority_ids parameter only boosts relevance (#79) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[autojack-bot]

Context from benchmark experiments

Good catch on both of these — the keyword scoring bug in particular is a real one. Adding some benchmark context from prior experiments that's relevant here.

This fix ≠ PR #80 BM25

Important distinction: the keyword fix in this issue is content-based token matching (lightweight, no new infrastructure). PR #80 proposed full BM25 via SQLite FTS5 + RRF fusion + LLM reranking + query expansion — a much heavier pipeline.

We benchmarked PR #80's full BM25 approach against the current baseline (LoCoMo-mini, snapshot-based, Voyage 4 embeddings):

Config	LoCoMo-mini (judge-off)	vs baseline
main (baseline)	89.36% (210/235)	—
BM25-only f10	88.09% (207/235)	−1.3pp
BM25-only f20	88.09% (207/235)	−1.3pp
Rerank top5	88.09% (207/235)	−1.3pp
Full pipeline	85.53% (judge-off)	−3.8pp

Open-domain category dropped ~10pp across all BM25 configs. No config passed promotion to judge-on runs. Runtime for bm25-only f10 was 233.8s vs ~150s baseline (~1.6x slower for total bench runtime; per-query API latency degradation was worse).

The fix proposed here avoids all of that — no new index, no additional search path, just fixing the dead keyword_component for vector-sourced results. That's the right approach.

Benchmark validation before merge

The fix changes _compute_metadata_score() which affects every recall result. Recommend running through at minimum:

make bench-eval TIER=1  # LoCoMo-mini, 2-3 min, free

Expected: the keyword scoring fix should be neutral-to-positive on LoCoMo (it's not adding noise, just correcting dead weight). The adaptive floor guardrails (15%→25%, max 50% cut) should improve entity recall without regressing structured queries.

Pre-existing test failures

The 2 failures in test_content_size.py noted in the issue — are those tracked anywhere? Would be good to not let those stay pre-existing forever.

npm package note

See also: the npm package (automem) is currently at v0.12.0, which predates the adaptive floor fix from PR #101. Anyone installing from npm has this same keyword=0.0 issue. Separate issue filed for that.

[JiwaniZakir]

Both issues stem from a common architectural mismatch: the scoring layer in scoring.py was designed assuming a mixed result set (keyword + vector), but recall.py's slot-filling logic means vector results dominate in practice, making keyword_component structurally unreachable for the vast majority of results. A straightforward fix for issue #1 would be to compute the keyword score based on term overlap against the result's content/metadata directly, rather than gating it solely on match_type — the match_type field reflects how the result was retrieved, not whether the query terms actually appear in it. For issue #2, the 15% gap threshold in the adaptive floor was likely tuned against mixed result sets where score distributions are more uniform; entity queries produce bimodal distributions by design (exact metadata hits vs. content-only hits), so the gap detector fires on a semantically meaningful but not quality-significant boundary — tightening the threshold or adding a minimum cluster size guard before cutting would reduce the over-pruning.