Repository navigation
Sleep cycle, code links (+10 pts on real GitHub issues), answers that never cut off - #138
Conversation
A budget stop used to fail the answer or leave half a sentence on screen. Now the model continues from where it stopped (up to two more turns), and if it still can't finish, the text is cut back to its last whole sentence, list item or paragraph, open code blocks are closed, and a one-line note says it was shortened. Output budget raised from 1536 to 4096 tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
- sleep.ts: wake memory (rated questions) is replayed offline; people's vocabulary is consolidated into a separate lexicon; dreams (identifier words, section context, lexicon swaps, model-written when configured) are tried on a copy of memory and kept only if they help without losing any confirmed question or bringing back a dead end; a SIM-style journal budgets strategies by what has worked and pauses ones that keep failing. (Behrouz et al. 'Language Models Need Sleep'; Yu et al. SIM, 2611.18856) - case-memory.ts: resolved questions lend their files to similar new ones (BugLocator) with a FixCache prior; sleep re-weighs it by replaying its own history and adopts weights only past a margin. - dev-entities.ts + retrieve-core: code a question names (traceback paths, module.Class, calls) links to the files defining it; those passages join the candidates with a bounded lift. Search bar uses the same links. - Dreams never overwrite or crowd out a person's ratings; a dead end retires similar dreams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
Shows confirmed questions, dead ends and dreams kept; the last sleep's replay (nothing forgotten, or what was rolled back); each dream strategy's kept-of-tried with the reason for every rejection; whether resolved questions are in use and why; and the latest ratings. Bars grow in, the moon breathes while it sleeps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
170 SWE-bench Lite issues (pytest, sphinx, scikit-learn, django), issue text as written, each repo indexed at its earliest issue's commit. Code links tuned on sympy only. Per-issue chart links every issue that moved to its fixing PR. Case memory is reported as a negative result: sleep's guard reduced its harm but it does not transfer across issues. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5dedce252d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const piece = await opts.turn(soFar); | ||
| return { text: joinPieces(soFar, piece), continued: i, shortened: false }; |
There was a problem hiding this comment.
Preserve leading whitespace on successful continuation turns
When an output-budget stop occurs immediately after a word, a successful continuation can lose the whitespace separating it from the next word. Both provider adapters return text through validateCompletion(), which calls .trim(), while budget-stop partials retain their original whitespace. Consequently, a partial ending in running followed by a successful continuation containing jobs finish. becomes runningjobs finish. here; code continuations can likewise become invalid. Preserve the continuation's raw boundary whitespace before joining rather than trimming each successful piece independently.
Useful? React with 👍 / 👎.
| const fp = fingerprint(scoped); | ||
| const hit = byScope.get(scopeKey); | ||
| if (hit && hit.fp === fp) return hit.defs; | ||
| const defs = new DefinitionIndex(scoped.filter((r) => r.chunk.chunk_kind === "code").map((r) => ({ path: r.slug, text: r.chunk.text }))); |
There was a problem hiding this comment.
Index actual file paths for retrieval entity links
For repositories imported through GitHub ingestion, r.slug is a slugified document identifier, not a file path: ingestion supplies ${target.repo}-${file.path} as the slug hint, and store.server.ts normalizes it with slugify(). Registering that identifier as DefinitionIndex's path means its suffix checks cannot resolve quoted paths such as src/widget.py or dotted modules such as package.widget, so those new links never reach candidate injection or reranking in production retrieval. Use chunk.filepath for path matching and map linked paths back to document slugs where needed; the benchmark hides this mismatch by setting slug: path.
Useful? React with 👍 / 👎.
| state.experience = input.experience; | ||
| state.sleep = { ...(state.sleep ?? {}), [corpusId]: input.journal }; | ||
| state.cases = { ...(state.cases ?? {}), [corpusId]: input.cases }; | ||
| state.sinceSleep = { ...(state.sinceSleep ?? {}), [corpusId]: 0 }; |
There was a problem hiding this comment.
Avoid overwriting ratings recorded during a sleep cycle
If another request records feedback while a sleep cycle is awaiting rankFiles() or the subsequent graph refresh, this assignment replaces the latest experience log with the older snapshot captured by sleepCorpus(). Because that snapshot contains all corpora, even feedback for an unrelated corpus can be silently erased; concurrent sleeps can also overwrite each other's dreams. Replacing the target corpus's case state and resetting its rating counter similarly discards intervening updates. Serialize sleep with feedback writes or apply a version-checked merge against the current state instead of saving the stale snapshot wholesale.
Useful? React with 👍 / 👎.
| let after = confirmed.filter((p) => found(p, mem)).length; | ||
| let rolledBack = false; | ||
| if (after < before) { |
There was a problem hiding this comment.
Validate individual probes in the final sleep replay
When there are more than 40 rated probes, the per-dream guard skips probes without a shared token, relying on this final replay to catch other effects. However, comparing only the number of confirmed passages found allows a newly recovered question to compensate for losing a previously working one, and rejected passages are never checked here at all. Such nonlocal effects can occur when adding a dream evicts an older dream through the per-passage cap. A 41-probe reproduction retains a dream with before: 39, after: 39 despite losing one previously found passage and reviving a dead end. Compare every probe against its baseline and roll back on either prohibited transition instead of checking only the aggregate positive count.
Useful? React with 👍 / 👎.
| const latest = cases.at(-1)!.at; | ||
| for (const c of cases) { | ||
| const w = Math.pow(0.5, (latest - c.at) / halfLife); | ||
| for (const f of c.files) prior.set(f, (prior.get(f) ?? 0) + w); |
There was a problem hiding this comment.
Use consistent units for case-prior decay
In the application, fileExperience() records case timestamps with Date.now(), so this subtraction measures milliseconds, but the default half-life is just 30. As a result, cases recorded one second apart differ in prior weight by roughly ten billion, and a case one minute older underflows to zero. If sleep enables a nonzero prior weight, the purported historical fix-count prior therefore behaves almost entirely as a boost for the latest rating, unlike the ordinal timestamps used by the tests and benchmark. Use case-order age consistently or express the half-life in the same documented time units as production timestamps.
Useful? React with 👍 / 👎.
| let after = confirmed.filter((p) => found(p, mem)).length; | ||
| let rolledBack = false; | ||
| if (after < before) { |
There was a problem hiding this comment.
Validate individual probes in the final sleep replay
When there are more than 40 rated probes, the per-dream checks skip questions that share no terms with the new dream, but the final replay only compares aggregate positive counts and never rechecks dead ends. This lets a newly recovered positive compensate for losing a previously working question, and lets a skipped dead end return without triggering rollback. Such nonlocal effects are possible because recording a dream can evict an older, differently worded dream through the per-passage cap in experience.ts. The final guard should compare each probe against its baseline, enforcing both the positive and negative conditions used in the per-dream loop rather than merely checking after < before.
Useful? React with 👍 / 👎.
| const latest = cases.at(-1)!.at; | ||
| for (const c of cases) { | ||
| const w = Math.pow(0.5, (latest - c.at) / halfLife); | ||
| for (const f of c.files) prior.set(f, (prior.get(f) ?? 0) + w); |
There was a problem hiding this comment.
Use consistent units for case-prior decay
For production feedback, fileExperience() records each case with at: Date.now(), so the timestamp differences here are milliseconds, but the default half-life is just 30. Cases therefore lose half their prior every 30 milliseconds, and a case one minute older than the latest rating underflows to zero. In contrast, addCase()'s default and the benchmark use sequential case numbers, where the same constant retains approximately 30 cases of history. This makes any accepted prior weight behave almost exclusively as a boost for the latest rating in production rather than a decayed fix-count history. Normalize timestamps to the intended age unit or consistently use case sequence numbers.
Useful? React with 👍 / 👎.
| const caseState = corpusId ? state.cases?.[corpusId] : undefined; | ||
| const key = `${caseState?.cases.length ?? 0}:${JSON.stringify(caseState?.weights ?? null)}:${rows.length}:${rows[0]?.chunk.id ?? ""}:${rows.at(-1)?.chunk.id ?? ""}:${rows.reduce((n, r) => n + r.chunk.text.length, 0)}:${pairs.length}`; | ||
| const cacheKey = scope.kind === "all" ? "*" : scope.corpusId; | ||
| const hit = CACHE.get(cacheKey); | ||
| if (hit && hit.key === key) return hit; |
There was a problem hiding this comment.
Invalidate cached search state when capped case history changes
Once a corpus reaches the 2,000-case limit imposed by addCase(), every subsequent rating replaces part of the case history without changing cases.length. If the weights and positive-experience count also remain unchanged, this key stays identical and build() returns a cached Built holding the old case-state object, so new ratings no longer affect case-based search ranking. This is reproducible with a capped history: appending one case leaves the cached result order unchanged while rebuilding the same search module reverses it. Include a case-history revision or content fingerprint in the cache key rather than relying on its length.
Useful? React with 👍 / 👎.
Semantic search bench, nfcorpus (b0e535a)nfcorpus (3633 docs, 323 test queries)
Per query (ms, p50/p95): BM25 0.35/5.15; embed query 7.8/12.5; dense exact 2.89/5.92; dense binary+rescore 1.74/3.13; fuse 0.15; cross-encoder top 20 920/953. Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-nfcorpus.json |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
Semantic search bench, scifact (b0e535a)scifact (5183 docs, 300 test queries)
Per query (ms, p50/p95): BM25 4.68/9.88; embed query 5.7/7.8; dense exact 2.42/3.34; dense binary+rescore 1.29/1.68; fuse 0.08; cross-encoder top 20 462/515. Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-scifact.json Pooling ablation: mean (SciFact)scifact (5183 docs, 300 test queries)
Per query (ms, p50/p95): BM25 4.78/9.94; embed query 5.8/8.2; dense exact 2.42/3.88; dense binary+rescore 1.29/1.69; fuse 0.09; cross-encoder top 20 470/538. Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-mean.json |
Semantic search bench, nfcorpus (4141fda)nfcorpus (3633 docs, 323 test queries)
Per query (ms, p50/p95): BM25 0.29/3.03; embed query 2.9/3.6; dense exact 1.16/1.68; dense binary+rescore 0.79/1.08; fuse 0.07; cross-encoder top 20 343/386. Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-nfcorpus.json |
Semantic search bench, scifact (4141fda)scifact (5183 docs, 300 test queries)
Per query (ms, p50/p95): BM25 4.77/10.07; embed query 5.7/7.8; dense exact 2.44/2.70; dense binary+rescore 1.28/1.61; fuse 0.08; cross-encoder top 20 458/512. Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-scifact.json Pooling ablation: mean (SciFact)scifact (5183 docs, 300 test queries)
Per query (ms, p50/p95): BM25 4.81/10.09; embed query 5.7/8.0; dense exact 2.43/3.83; dense binary+rescore 1.28/1.66; fuse 0.09; cross-encoder top 20 458/509. Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-mean.json |
Semantic search bench, fiqa (b0e535a)fiqa (57638 docs, 648 test queries)
Per query (ms, p50/p95): BM25 36.84/89.80; embed query 8.5/15.6; dense exact 42.18/47.23; dense binary+rescore 4.22/6.28; fuse 0.11; cross-encoder top 20 895/932. Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-fiqa.json |
Semantic search bench, fiqa (4141fda)fiqa (57638 docs, 648 test queries)
Per query (ms, p50/p95): BM25 25.52/60.95; embed query 5.7/7.6; dense exact 36.92/46.70; dense binary+rescore 4.32/4.48; fuse 0.11; cross-encoder top 20 945/1042. Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-fiqa.json |
Three things, each measured.
1. Code links: real GitHub issues. On 170 SWE-bench Lite issues from django, scikit-learn, sphinx and pytest, given the issue text as written, the file the fix changed now comes first 45.3% of the time, up from 35.3%. It is in the top 5 65.9% of the time, up from 60.6%. Each repo is indexed at its earliest issue's commit, and the lift was tuned on sympy only. See eval/github-issues for the chart, which links every issue that moved to its fixing PR.
2. A sleep cycle, following "Language Models Need Sleep" (Behrouz, Hashemi, Mirrokni) and SIM (arXiv 2611.18856):
Honest results: one sleep doesn't move the 14-question rewording test, and case memory doesn't transfer across GitHub issues. Sleep's guard cut that damage from −6.5 to −1.8 points, so case memory stays off unless replay clears a 0.05 MRR margin. Both are documented as negative results.
3. Answers never stop mid-sentence. A budget stop now continues from where it stopped, up to two more turns. If it still can't finish, the answer is trimmed to its last whole sentence with a note. Output budget goes from 1536 to 4096 tokens.
Tests:
Generated by Claude Code