Repository navigation
Search bar: GitHub-style code search (⌘K), +3.6 nDCG@10 on CoSQA - #126
Conversation
Experience memory (experience.ts): rated answers and source picks are filed against the passages they came from; later questions worded like them find those passages (BM25 fusion over confirmed wordings, a bounded case-based lift, and support when a very similar question was confirmed). Wordings only count for questions they resemble (no hubs), reliability counts only ratings on similar questions, and links go stale when the passage changes. Absent names and a turn's own terms are still checked against the source text. New Evidence-panel button: "This passage answers my question"; the MCP feedback tool files to a per-repo memory on disk. SIM loop (scripts/sim-optimize.ts, after arXiv 2611.18856): root causes found per failing case, every attempt recorded with rationale, predicted and observed fixes and provenance; the proposer reads the memory. Run 1 kept soloMargin 1.35 -> 1.5 and boostFloor 0.25 -> 0.2 (held-out 52.9 -> 58.8%); both are now defaults. The conversation eval, which the loop never saw, rose 71.8 -> 76.9% (follow-ups 77.3 -> 86.4%); repo-support unchanged. The CRSI conditions on the distractor target accepted no edits: an honest null, written up. Learning eval: originals 71.4 -> 81.0% (clean), held-out wordings +2.4 to +4.8 points, refusals unchanged, 20% wrong ratings cost nothing. CI gates it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
- code-search.ts: identifier-aware BM25F with name/docstring fields, operators (path: lang: kind: sym: "phrase" -word), did-you-mean (OSA edit distance), completions, highlighted snippets, learned query->code associations - search.server.ts + /api/search + server fns; associations learned from the experience log's confirmed passages - Ctrl/Cmd+K palette with keyboard navigation and operator help - CoSQA bench: 46.42 nDCG@10 vs 42.86 plain BM25 (313 dev queries), CI check - unit tests for tokenizing, operators, filters, associations, spelling, snippets Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd1f9b3188
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let q = this.queryTerms([...p.words, ...p.phrases]); | ||
| if (!q.size) return []; | ||
| let ranked = this.scoreAll(q); |
There was a problem hiding this comment.
Evaluate filter-only searches against the indexed documents
Queries such as sym:resumeQueue or path:src/queue always return no results because parsing removes the operators from words, leaving an empty term map that returns before passesFilters runs. This also affects quoted searches containing only discarded code keywords, such as "return". These are valid searches for the advertised operators, including the examples offered by the palette itself. When there are constraints but no searchable tokens, filter the indexed documents rather than unconditionally returning an empty result.
Useful? React with 👍 / 👎.
| case "ask": { const r = ask(pick(a.corpus), String(a.question), a.session); return { text: render(r), data: r }; } | ||
| case "search": { const r = search(pick(a.corpus), String(a.query), Math.min(20, Math.max(1, Number(a.limit) || 8))); return { text: r.map((p) => `[${p.ref}] ${p.score} ${p.file}${p.heading ? ` › ${p.heading}` : ""}`).join("\n") || "No candidates.", data: r }; } | ||
| case "feedback": { const ok = feedback(String(a.session), String(a.ref), Boolean(a.helpful)); return { text: ok ? "Noted." : "That passage isn't in this session's recent answers.", data: { ok } }; } | ||
| case "feedback": { const ok = feedback(String(a.session), String(a.ref), Boolean(a.helpful), current ?? undefined); return { text: ok ? "Noted. Questions worded like this one will find that passage." : "Ask something in this session first.", data: { ok } }; } |
There was a problem hiding this comment.
Persist MCP feedback against the answer's corpus
When multiple corpora are indexed and ask explicitly selects one other than the latest indexed corpus, feedback is saved to current instead of the corpus that supplied the answer. I reproduced this through the built MCP server: asking corpus A after indexing B and marking A's passage helpful created an experience entry under B's ID. Consequently A does not learn, and B can inherit unrelated feedback when passage IDs and text coincide. Retain the originating corpus with the session turn and use it when recording feedback.
Useful? React with 👍 / 👎.
| const others = ids.flatMap((id) => (para.cases[id] as string[]).map((q, i) => ({ id, q, i })).filter((x) => x.i !== fold).map(({ id, q }) => ({ id, q }))); | ||
| const log = train(s, policy, [...originals, ...others]); |
There was a problem hiding this comment.
Exclude held-out wordings from optimizer training
During optimization, rsi calls evaluate with folds 0 and 1, but this filter includes fold 2 in others for both calls. train then retrieves and labels those supposedly held-out questions using the gold evidence, incorporating their wording and ratings into the experience log used to select harness changes. Thus the reported held-out split influences optimization even though its scores are computed separately, undermining the claimed unseen-wording evaluation. Reserve fold 2 outside both training and scoring during harness selection, then regenerate the reported results.
Useful? React with 👍 / 👎.
| const key = `${rows.length}:${rows[0]?.chunk.id ?? ""}:${rows.at(-1)?.chunk.id ?? ""}:${rows.reduce((n, r) => n + r.chunk.text.length, 0)}:${pairs.length}`; | ||
| const cacheKey = scope.kind === "all" ? "*" : scope.corpusId; | ||
| const hit = CACHE.get(cacheKey); | ||
| if (hit && hit.key === key) return hit; |
There was a problem hiding this comment.
Include all source revisions in the search cache identity
If an interior document is edited without changing the total chunk count or text length, and the first and last chunks belong to unchanged documents, this key remains identical even though replaceChunks generated new IDs and text for the edited document. The cache then returns both the old index and old rows, so deleted content remains searchable while replacement content cannot be found until another change invalidates the cache or the process restarts. I reproduced this with mocked storage rows: replacing an interior freeze passage with an equal-length thawed passage still returned the removed passage for freeze and no result for thawed. Include every chunk's identity/content revision in the cache key.
Useful? React with 👍 / 👎.
| fileExperience(state, { corpusId: input.corpusId, question: input.question, passages: [{ chunkId: input.chunkId, text: input.text }], vote: 1 }); | ||
| await persist(state); |
There was a problem hiding this comment.
Invalidate cached answers when recording a source pick
When an answer contains the wrong or incomplete evidence and the user marks a different candidate as answering the question, this function updates experience but leaves the existing answer cache intact. Repeating that question with the same settings reaches findCachedAnswer in query.server.ts and returns the old answer before retrieval reads the updated experience; cache entries with a null outcome are eligible for reuse. Thus the explicit source correction cannot improve that question's answer unless the user separately marks the answer as a dead end or the cache is otherwise invalidated. Invalidate the affected cached answer when accepting a source pick, or incorporate an experience revision into answer-cache identity.
Useful? React with 👍 / 👎.
Semantic search bench, nfcorpus (4a845f4)nfcorpus (3633 docs, 323 test queries)
Per query (ms, p50/p95): BM25 0.39/4.64; embed query 4.5/8.4; dense exact 2.52/4.38; dense binary+rescore 1.51/1.70; fuse 0.11; cross-encoder top 20 600/629. Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-nfcorpus.json |
Semantic search bench, scifact (4a845f4)scifact (5183 docs, 300 test queries)
Per query (ms, p50/p95): BM25 4.68/11.01; embed query 3.7/5.0; dense exact 1.66/2.24; dense binary+rescore 0.89/1.20; fuse 0.09; cross-encoder top 20 350/364. Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-scifact.json Pooling ablation: mean (SciFact)scifact (5183 docs, 300 test queries)
Per query (ms, p50/p95): BM25 4.72/10.44; embed query 3.9/5.3; dense exact 1.66/2.24; dense binary+rescore 0.87/1.12; fuse 0.09; cross-encoder top 20 348/360. Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-mean.json |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
Semantic search bench, fiqa (4a845f4)fiqa (57638 docs, 648 test queries)
Per query (ms, p50/p95): BM25 35.29/83.79; embed query 5.3/9.6; dense exact 36.09/38.70; dense binary+rescore 3.66/5.56; fuse 0.11; cross-encoder top 20 546/590. Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-fiqa.json |
GitHub-style code search over everything IntelliRAG has indexed. Builds on #125 (merge that first; this PR then shows only the search commit).
What you get
path:lang:kind:sym:"exact phrase"-exclude.GET /api/search?q=…&corpus=…and?suggest=…for tools and agents.Measured on CoSQA (313 dev queries, 6,267 Python functions, tuned on train only)
Synonym lexicon and RM3 were tried and lost, so they're off. Dense retrievers still score higher on CoSQA-style queries; eval/code-search/README.md says so and explains which numbers are comparable. CI re-runs the benchmark from microsoft/CodeXGLUE and fails if the gain drops below 2 points.
Tests: 12 new unit tests. The full rag suite fails the same 21 environment-only tests as before.
Generated by Claude Code