Skip to content

Search bar: GitHub-style code search (⌘K), +3.6 nDCG@10 on CoSQA - #126

Merged
charan-rathore merged 3 commits into
mainfrom
feat/search-bar
Nov 18, 2026
Merged

charan-rathore merged 3 commits into
mainfrom
feat/search-bar

Conversation

@charan-rathore

Copy link
Copy Markdown
Owner

GitHub-style code search over everything IntelliRAG has indexed. Builds on #125 (merge that first; this PR then shows only the search commit).

What you get

  • ⌘K / Ctrl+K palette: instant ranked passages, matching lines highlighted, ↑↓/Enter navigation, Tab completion, "did you mean".
  • Operators: path: lang: kind: sym: "exact phrase" -exclude.
  • GET /api/search?q=…&corpus=… and ?suggest=… for tools and agents.
  • It learns: questions people confirmed a passage answers (Learn from use: experience memory and a self-improvement loop (SIM) #125's experience log) become query→code associations.

Measured on CoSQA (313 dev queries, 6,267 Python functions, tuned on train only)

nDCG@10 R@5
plain BM25 42.86 52.08
search bar 46.42 59.42

Synonym lexicon and RM3 were tried and lost, so they're off. Dense retrievers still score higher on CoSQA-style queries; eval/code-search/README.md says so and explains which numbers are comparable. CI re-runs the benchmark from microsoft/CodeXGLUE and fails if the gain drops below 2 points.

Tests: 12 new unit tests. The full rag suite fails the same 21 environment-only tests as before.


Generated by Claude Code

Experience memory (experience.ts): rated answers and source picks are filed
against the passages they came from; later questions worded like them find
those passages (BM25 fusion over confirmed wordings, a bounded case-based
lift, and support when a very similar question was confirmed). Wordings only
count for questions they resemble (no hubs), reliability counts only ratings
on similar questions, and links go stale when the passage changes. Absent
names and a turn's own terms are still checked against the source text.
New Evidence-panel button: "This passage answers my question"; the MCP
feedback tool files to a per-repo memory on disk.

SIM loop (scripts/sim-optimize.ts, after arXiv 2611.18856): root causes found
per failing case, every attempt recorded with rationale, predicted and
observed fixes and provenance; the proposer reads the memory. Run 1 kept
soloMargin 1.35 -> 1.5 and boostFloor 0.25 -> 0.2 (held-out 52.9 -> 58.8%);
both are now defaults. The conversation eval, which the loop never saw,
rose 71.8 -> 76.9% (follow-ups 77.3 -> 86.4%); repo-support unchanged.
The CRSI conditions on the distractor target accepted no edits: an honest
null, written up.

Learning eval: originals 71.4 -> 81.0% (clean), held-out wordings +2.4 to
+4.8 points, refusals unchanged, 20% wrong ratings cost nothing. CI gates it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
- code-search.ts: identifier-aware BM25F with name/docstring fields, operators
  (path: lang: kind: sym: "phrase" -word), did-you-mean (OSA edit distance),
  completions, highlighted snippets, learned query->code associations
- search.server.ts + /api/search + server fns; associations learned from the
  experience log's confirmed passages
- Ctrl/Cmd+K palette with keyboard navigation and operator help
- CoSQA bench: 46.42 nDCG@10 vs 42.86 plain BM25 (313 dev queries), CI check
- unit tests for tokenizing, operators, filters, associations, spelling, snippets

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
intellirag-live-own-track Ready Ready Preview Nov 18, 2026 10:24pm UTC

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T22:14:40.275444Z fd1f9b3 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fd1f9b3188

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +271 to +273
let q = this.queryTerms([...p.words, ...p.phrases]);
if (!q.size) return [];
let ranked = this.scoreAll(q);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Evaluate filter-only searches against the indexed documents

Queries such as sym:resumeQueue or path:src/queue always return no results because parsing removes the operators from words, leaving an empty term map that returns before passesFilters runs. This also affects quoted searches containing only discarded code keywords, such as "return". These are valid searches for the advertised operators, including the examples offered by the palette itself. When there are constraints but no searchable tokens, filter the indexed documents rather than unconditionally returning an empty result.

Useful? React with 👍 / 👎.

Comment thread mcp/src/server.ts
case "ask": { const r = ask(pick(a.corpus), String(a.question), a.session); return { text: render(r), data: r }; }
case "search": { const r = search(pick(a.corpus), String(a.query), Math.min(20, Math.max(1, Number(a.limit) || 8))); return { text: r.map((p) => `[${p.ref}] ${p.score} ${p.file}${p.heading ? ` › ${p.heading}` : ""}`).join("\n") || "No candidates.", data: r }; }
case "feedback": { const ok = feedback(String(a.session), String(a.ref), Boolean(a.helpful)); return { text: ok ? "Noted." : "That passage isn't in this session's recent answers.", data: { ok } }; }
case "feedback": { const ok = feedback(String(a.session), String(a.ref), Boolean(a.helpful), current ?? undefined); return { text: ok ? "Noted. Questions worded like this one will find that passage." : "Ask something in this session first.", data: { ok } }; }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Persist MCP feedback against the answer's corpus

When multiple corpora are indexed and ask explicitly selects one other than the latest indexed corpus, feedback is saved to current instead of the corpus that supplied the answer. I reproduced this through the built MCP server: asking corpus A after indexing B and marking A's passage helpful created an experience entry under B's ID. Consequently A does not learn, and B can inherit unrelated feedback when passage IDs and text coincide. Retain the originating corpus with the session turn and use it when recording feedback.

Useful? React with 👍 / 👎.

Comment on lines +104 to +105
const others = ids.flatMap((id) => (para.cases[id] as string[]).map((q, i) => ({ id, q, i })).filter((x) => x.i !== fold).map(({ id, q }) => ({ id, q })));
const log = train(s, policy, [...originals, ...others]);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exclude held-out wordings from optimizer training

During optimization, rsi calls evaluate with folds 0 and 1, but this filter includes fold 2 in others for both calls. train then retrieves and labels those supposedly held-out questions using the gold evidence, incorporating their wording and ratings into the experience log used to select harness changes. Thus the reported held-out split influences optimization even though its scores are computed separately, undermining the claimed unseen-wording evaluation. Reserve fold 2 outside both training and scoring during harness selection, then regenerate the reported results.

Useful? React with 👍 / 👎.

Comment on lines +42 to +45
const key = `${rows.length}:${rows[0]?.chunk.id ?? ""}:${rows.at(-1)?.chunk.id ?? ""}:${rows.reduce((n, r) => n + r.chunk.text.length, 0)}:${pairs.length}`;
const cacheKey = scope.kind === "all" ? "*" : scope.corpusId;
const hit = CACHE.get(cacheKey);
if (hit && hit.key === key) return hit;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include all source revisions in the search cache identity

If an interior document is edited without changing the total chunk count or text length, and the first and last chunks belong to unchanged documents, this key remains identical even though replaceChunks generated new IDs and text for the edited document. The cache then returns both the old index and old rows, so deleted content remains searchable while replacement content cannot be found until another change invalidates the cache or the process restarts. I reproduced this with mocked storage rows: replacing an interior freeze passage with an equal-length thawed passage still returned the removed passage for freeze and no result for thawed. Include every chunk's identity/content revision in the cache key.

Useful? React with 👍 / 👎.

Comment on lines +283 to +284
fileExperience(state, { corpusId: input.corpusId, question: input.question, passages: [{ chunkId: input.chunkId, text: input.text }], vote: 1 });
await persist(state);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Invalidate cached answers when recording a source pick

When an answer contains the wrong or incomplete evidence and the user marks a different candidate as answering the question, this function updates experience but leaves the existing answer cache intact. Repeating that question with the same settings reaches findCachedAnswer in query.server.ts and returns the old answer before retrieval reads the updated experience; cache entries with a null outcome are eligible for reuse. Thus the explicit source correction cannot improve that question's answer unless the user separately marks the answer as a dead end or the cache is otherwise invalidated. Invalidate the affected cached answer when accepting a source pick, or incorporate an experience revision into answer-cache identity.

Useful? React with 👍 / 👎.

@github-actions

Copy link
Copy Markdown

Semantic search bench, nfcorpus (4a845f4)

nfcorpus (3633 docs, 323 test queries)

arm nDCG@10 MRR@10 Recall@100 nDCG@10 vs BM25 (95% CI)
bm25 0.320 0.530 0.241 -
dense 0.340 0.525 0.311 +0.021 [-0.000, 0.042]
denseBinary 0.341 0.526 0.310 +0.021 [-0.000, 0.042]
hybrid 0.360 0.565 0.318 +0.040 [0.027, 0.054]
hybridRerank 0.361 0.584 0.318 +0.041 [0.027, 0.056]

Per query (ms, p50/p95): BM25 0.39/4.64; embed query 4.5/8.4; dense exact 2.52/4.38; dense binary+rescore 1.51/1.70; fuse 0.11; cross-encoder top 20 600/629.
Index: 5.3 MB float, 0.17 MB binary; binary top-10 agreement with exact 99.2%; documents embedded in 260 s.

Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-nfcorpus.json

@github-actions

Copy link
Copy Markdown

Semantic search bench, scifact (4a845f4)

scifact (5183 docs, 300 test queries)

arm nDCG@10 MRR@10 Recall@100 nDCG@10 vs BM25 (95% CI)
bm25 0.672 0.642 0.886 -
dense 0.706 0.674 0.945 +0.034 [-0.000, 0.069]
denseBinary 0.706 0.674 0.937 +0.034 [-0.000, 0.069]
hybrid 0.703 0.667 0.968 +0.031 [0.007, 0.054]
hybridRerank 0.684 0.650 0.968 +0.012 [-0.020, 0.044]

Per query (ms, p50/p95): BM25 4.68/11.01; embed query 3.7/5.0; dense exact 1.66/2.24; dense binary+rescore 0.89/1.20; fuse 0.09; cross-encoder top 20 350/364.
Index: 7.6 MB float, 0.24 MB binary; binary top-10 agreement with exact 99.7%; documents embedded in 210 s.

Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-scifact.json

Pooling ablation: mean (SciFact)

scifact (5183 docs, 300 test queries)

arm nDCG@10 MRR@10 Recall@100 nDCG@10 vs BM25 (95% CI)
bm25 0.672 0.642 0.886 -
dense 0.704 0.673 0.943 +0.032 [-0.004, 0.067]
denseBinary 0.704 0.673 0.950 +0.032 [-0.004, 0.067]
hybrid 0.704 0.671 0.965 +0.032 [0.009, 0.054]
hybridRerank 0.690 0.655 0.965 +0.018 [-0.015, 0.049]

Per query (ms, p50/p95): BM25 4.72/10.44; embed query 3.9/5.3; dense exact 1.66/2.24; dense binary+rescore 0.87/1.12; fuse 0.09; cross-encoder top 20 348/360.
Index: 7.6 MB float, 0.24 MB binary; binary top-10 agreement with exact 99.6%; documents embedded in 218 s.

Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-mean.json

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
@charan-rathore
charan-rathore merged commit cbaa8a5 into main Nov 18, 2026
5 checks passed
@github-actions

Copy link
Copy Markdown

Semantic search bench, fiqa (4a845f4)

fiqa (57638 docs, 648 test queries)

arm nDCG@10 MRR@10 Recall@100 nDCG@10 vs BM25 (95% CI)
bm25 0.249 0.302 0.538 -
dense 0.397 0.479 0.691 +0.148 [0.125, 0.172]
denseBinary 0.398 0.480 0.677 +0.148 [0.126, 0.173]
hybrid 0.360 0.432 0.683 +0.111 [0.096, 0.126]
hybridRerank 0.376 0.444 0.683 +0.127 [0.108, 0.147]

Per query (ms, p50/p95): BM25 35.29/83.79; embed query 5.3/9.6; dense exact 36.09/38.70; dense binary+rescore 3.66/5.56; fuse 0.11; cross-encoder top 20 546/590.
Index: 84.4 MB float, 2.64 MB binary; binary top-10 agreement with exact 98.6%; documents embedded in 1865 s.

Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-fiqa.json

This branch was successfully deployed

1 active deployment
Preview — 4bdb1fe6 Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants