Skip to content

Sleep cycle, code links (+10 pts on real GitHub issues), answers that never cut off - #138

Merged
charan-rathore merged 5 commits into
mainfrom
feat/sleep-cycle
Nov 19, 2026
Merged

charan-rathore merged 5 commits into
mainfrom
feat/sleep-cycle

Conversation

@charan-rathore

Copy link
Copy Markdown
Owner

Three things, each measured.

1. Code links: real GitHub issues. On 170 SWE-bench Lite issues from django, scikit-learn, sphinx and pytest, given the issue text as written, the file the fix changed now comes first 45.3% of the time, up from 35.3%. It is in the top 5 65.9% of the time, up from 60.6%. Each repo is indexed at its earliest issue's commit, and the lift was tuned on sympy only. See eval/github-issues for the chart, which links every issue that moved to its fixing PR.

2. A sleep cycle, following "Language Models Need Sleep" (Behrouz, Hashemi, Mirrokni) and SIM (arXiv 2611.18856):

  • Replays confirmed questions.
  • Consolidates people's vocabulary into separate memory.
  • Dreams practice questions and keeps one only if it helps without losing any confirmed question or reviving a dead end; a final full replay rolls back the whole sleep if anything slipped through.
  • Keeps a journal that budgets strategies by what has worked.
  • Runs after every 8 ratings, or from the new Memory panel.

Honest results: one sleep doesn't move the 14-question rewording test, and case memory doesn't transfer across GitHub issues. Sleep's guard cut that damage from −6.5 to −1.8 points, so case memory stays off unless replay clears a 0.05 MRR margin. Both are documented as negative results.

3. Answers never stop mid-sentence. A budget stop now continues from where it stopped, up to two more turns. If it still can't finish, the answer is trimmed to its last whole sentence with a note. Output budget goes from 1536 to 4096 tokens.

Tests:

  • New: sleep (dreams, forgetting guard, rollback, journal, dead ends retire dreams), case memory, entity linking, answer completion, and updated generation-validation.
  • The full rag suite fails the same 21 environment-only tests as main.
  • Learning and conversation evals still pass their checks, and the MCP smoke test passes.

Generated by Claude Code

A budget stop used to fail the answer or leave half a sentence on screen.
Now the model continues from where it stopped (up to two more turns), and
if it still can't finish, the text is cut back to its last whole sentence,
list item or paragraph, open code blocks are closed, and a one-line note
says it was shortened. Output budget raised from 1536 to 4096 tokens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
- sleep.ts: wake memory (rated questions) is replayed offline; people's
  vocabulary is consolidated into a separate lexicon; dreams (identifier
  words, section context, lexicon swaps, model-written when configured) are
  tried on a copy of memory and kept only if they help without losing any
  confirmed question or bringing back a dead end; a SIM-style journal
  budgets strategies by what has worked and pauses ones that keep failing.
  (Behrouz et al. 'Language Models Need Sleep'; Yu et al. SIM, 2611.18856)
- case-memory.ts: resolved questions lend their files to similar new ones
  (BugLocator) with a FixCache prior; sleep re-weighs it by replaying its own
  history and adopts weights only past a margin.
- dev-entities.ts + retrieve-core: code a question names (traceback paths,
  module.Class, calls) links to the files defining it; those passages join
  the candidates with a bounded lift. Search bar uses the same links.
- Dreams never overwrite or crowd out a person's ratings; a dead end retires
  similar dreams.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
Shows confirmed questions, dead ends and dreams kept; the last sleep's
replay (nothing forgotten, or what was rolled back); each dream strategy's
kept-of-tried with the reason for every rejection; whether resolved
questions are in use and why; and the latest ratings. Bars grow in, the
moon breathes while it sleeps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
170 SWE-bench Lite issues (pytest, sphinx, scikit-learn, django), issue
text as written, each repo indexed at its earliest issue's commit. Code
links tuned on sympy only. Per-issue chart links every issue that moved to
its fixing PR. Case memory is reported as a negative result: sleep's guard
reduced its harm but it does not transfer across issues.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
@vercel

vercel Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
intellirag-live-own-track Ready Ready Preview Nov 19, 2026 7:55am UTC

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T07:44:52.210486Z 5dedce2 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5dedce252d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +52 to +53
const piece = await opts.turn(soFar);
return { text: joinPieces(soFar, piece), continued: i, shortened: false };

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve leading whitespace on successful continuation turns

When an output-budget stop occurs immediately after a word, a successful continuation can lose the whitespace separating it from the next word. Both provider adapters return text through validateCompletion(), which calls .trim(), while budget-stop partials retain their original whitespace. Consequently, a partial ending in running followed by a successful continuation containing jobs finish. becomes runningjobs finish. here; code continuations can likewise become invalid. Preserve the continuation's raw boundary whitespace before joining rather than trimming each successful piece independently.

Useful? React with 👍 / 👎.

const fp = fingerprint(scoped);
const hit = byScope.get(scopeKey);
if (hit && hit.fp === fp) return hit.defs;
const defs = new DefinitionIndex(scoped.filter((r) => r.chunk.chunk_kind === "code").map((r) => ({ path: r.slug, text: r.chunk.text })));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Index actual file paths for retrieval entity links

For repositories imported through GitHub ingestion, r.slug is a slugified document identifier, not a file path: ingestion supplies ${target.repo}-${file.path} as the slug hint, and store.server.ts normalizes it with slugify(). Registering that identifier as DefinitionIndex's path means its suffix checks cannot resolve quoted paths such as src/widget.py or dotted modules such as package.widget, so those new links never reach candidate injection or reranking in production retrieval. Use chunk.filepath for path matching and map linked paths back to document slugs where needed; the benchmark hides this mismatch by setting slug: path.

Useful? React with 👍 / 👎.

Comment on lines +294 to +297
state.experience = input.experience;
state.sleep = { ...(state.sleep ?? {}), [corpusId]: input.journal };
state.cases = { ...(state.cases ?? {}), [corpusId]: input.cases };
state.sinceSleep = { ...(state.sinceSleep ?? {}), [corpusId]: 0 };

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid overwriting ratings recorded during a sleep cycle

If another request records feedback while a sleep cycle is awaiting rankFiles() or the subsequent graph refresh, this assignment replaces the latest experience log with the older snapshot captured by sleepCorpus(). Because that snapshot contains all corpora, even feedback for an unrelated corpus can be silently erased; concurrent sleeps can also overwrite each other's dreams. Replacing the target corpus's case state and resetting its rating counter similarly discards intervening updates. Serialize sleep with feedback writes or apply a version-checked merge against the current state instead of saving the stale snapshot wholesale.

Useful? React with 👍 / 👎.

Comment thread web/src/lib/rag/sleep.ts
Comment on lines +335 to +337
let after = confirmed.filter((p) => found(p, mem)).length;
let rolledBack = false;
if (after < before) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate individual probes in the final sleep replay

When there are more than 40 rated probes, the per-dream guard skips probes without a shared token, relying on this final replay to catch other effects. However, comparing only the number of confirmed passages found allows a newly recovered question to compensate for losing a previously working one, and rejected passages are never checked here at all. Such nonlocal effects can occur when adding a dream evicts an older dream through the per-passage cap. A 41-probe reproduction retains a dream with before: 39, after: 39 despite losing one previously found passage and reviving a dead end. Compare every probe against its baseline and roll back on either prohibited transition instead of checking only the aggregate positive count.

Useful? React with 👍 / 👎.

Comment on lines +82 to +85
const latest = cases.at(-1)!.at;
for (const c of cases) {
const w = Math.pow(0.5, (latest - c.at) / halfLife);
for (const f of c.files) prior.set(f, (prior.get(f) ?? 0) + w);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use consistent units for case-prior decay

In the application, fileExperience() records case timestamps with Date.now(), so this subtraction measures milliseconds, but the default half-life is just 30. As a result, cases recorded one second apart differ in prior weight by roughly ten billion, and a case one minute older underflows to zero. If sleep enables a nonzero prior weight, the purported historical fix-count prior therefore behaves almost entirely as a boost for the latest rating, unlike the ordinal timestamps used by the tests and benchmark. Use case-order age consistently or express the half-life in the same documented time units as production timestamps.

Useful? React with 👍 / 👎.

Comment thread web/src/lib/rag/sleep.ts
Comment on lines +335 to +337
let after = confirmed.filter((p) => found(p, mem)).length;
let rolledBack = false;
if (after < before) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate individual probes in the final sleep replay

When there are more than 40 rated probes, the per-dream checks skip questions that share no terms with the new dream, but the final replay only compares aggregate positive counts and never rechecks dead ends. This lets a newly recovered positive compensate for losing a previously working question, and lets a skipped dead end return without triggering rollback. Such nonlocal effects are possible because recording a dream can evict an older, differently worded dream through the per-passage cap in experience.ts. The final guard should compare each probe against its baseline, enforcing both the positive and negative conditions used in the per-dream loop rather than merely checking after < before.

Useful? React with 👍 / 👎.

Comment on lines +82 to +85
const latest = cases.at(-1)!.at;
for (const c of cases) {
const w = Math.pow(0.5, (latest - c.at) / halfLife);
for (const f of c.files) prior.set(f, (prior.get(f) ?? 0) + w);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use consistent units for case-prior decay

For production feedback, fileExperience() records each case with at: Date.now(), so the timestamp differences here are milliseconds, but the default half-life is just 30. Cases therefore lose half their prior every 30 milliseconds, and a case one minute older than the latest rating underflows to zero. In contrast, addCase()'s default and the benchmark use sequential case numbers, where the same constant retains approximately 30 cases of history. This makes any accepted prior weight behave almost exclusively as a boost for the latest rating in production rather than a decayed fix-count history. Normalize timestamps to the intended age unit or consistently use case sequence numbers.

Useful? React with 👍 / 👎.

Comment on lines +69 to 73
const caseState = corpusId ? state.cases?.[corpusId] : undefined;
const key = `${caseState?.cases.length ?? 0}:${JSON.stringify(caseState?.weights ?? null)}:${rows.length}:${rows[0]?.chunk.id ?? ""}:${rows.at(-1)?.chunk.id ?? ""}:${rows.reduce((n, r) => n + r.chunk.text.length, 0)}:${pairs.length}`;
const cacheKey = scope.kind === "all" ? "*" : scope.corpusId;
const hit = CACHE.get(cacheKey);
if (hit && hit.key === key) return hit;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Invalidate cached search state when capped case history changes

Once a corpus reaches the 2,000-case limit imposed by addCase(), every subsequent rating replaces part of the case history without changing cases.length. If the weights and positive-experience count also remain unchanged, this key stays identical and build() returns a cached Built holding the old case-state object, so new ratings no longer affect case-based search ranking. This is reproducible with a capped history: appending one case leaves the cached result order unchanged while rebuilding the same search module reverses it. Include a case-history revision or content fingerprint in the cache key rather than relying on its length.

Useful? React with 👍 / 👎.

@github-actions

Copy link
Copy Markdown

Semantic search bench, nfcorpus (b0e535a)

nfcorpus (3633 docs, 323 test queries)

arm nDCG@10 MRR@10 Recall@100 nDCG@10 vs BM25 (95% CI)
bm25 0.320 0.530 0.241 -
dense 0.344 0.528 0.309 +0.024 [0.003, 0.045]
denseBinary 0.344 0.529 0.307 +0.024 [0.003, 0.045]
hybrid 0.361 0.568 0.315 +0.041 [0.028, 0.055]
hybridRerank 0.361 0.588 0.315 +0.042 [0.028, 0.057]

Per query (ms, p50/p95): BM25 0.35/5.15; embed query 7.8/12.5; dense exact 2.89/5.92; dense binary+rescore 1.74/3.13; fuse 0.15; cross-encoder top 20 920/953.
Index: 5.3 MB float, 0.17 MB binary; binary top-10 agreement with exact 98.9%; documents embedded in 385 s.

Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-nfcorpus.json

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D3gm7sJERkZTxCR1BAcsgW
@github-actions

Copy link
Copy Markdown

Semantic search bench, scifact (b0e535a)

scifact (5183 docs, 300 test queries)

arm nDCG@10 MRR@10 Recall@100 nDCG@10 vs BM25 (95% CI)
bm25 0.672 0.642 0.886 -
dense 0.706 0.674 0.945 +0.034 [-0.000, 0.069]
denseBinary 0.706 0.674 0.937 +0.034 [-0.000, 0.069]
hybrid 0.703 0.667 0.968 +0.031 [0.007, 0.054]
hybridRerank 0.684 0.650 0.968 +0.012 [-0.020, 0.044]

Per query (ms, p50/p95): BM25 4.68/9.88; embed query 5.7/7.8; dense exact 2.42/3.34; dense binary+rescore 1.29/1.68; fuse 0.08; cross-encoder top 20 462/515.
Index: 7.6 MB float, 0.24 MB binary; binary top-10 agreement with exact 99.7%; documents embedded in 332 s.

Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-scifact.json

Pooling ablation: mean (SciFact)

scifact (5183 docs, 300 test queries)

arm nDCG@10 MRR@10 Recall@100 nDCG@10 vs BM25 (95% CI)
bm25 0.672 0.642 0.886 -
dense 0.704 0.673 0.943 +0.032 [-0.004, 0.067]
denseBinary 0.704 0.673 0.950 +0.032 [-0.004, 0.067]
hybrid 0.704 0.671 0.965 +0.032 [0.009, 0.054]
hybridRerank 0.690 0.655 0.965 +0.018 [-0.015, 0.049]

Per query (ms, p50/p95): BM25 4.78/9.94; embed query 5.8/8.2; dense exact 2.42/3.88; dense binary+rescore 1.29/1.69; fuse 0.09; cross-encoder top 20 470/538.
Index: 7.6 MB float, 0.24 MB binary; binary top-10 agreement with exact 99.6%; documents embedded in 297 s.

Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-mean.json

@github-actions

Copy link
Copy Markdown

Semantic search bench, nfcorpus (4141fda)

nfcorpus (3633 docs, 323 test queries)

arm nDCG@10 MRR@10 Recall@100 nDCG@10 vs BM25 (95% CI)
bm25 0.320 0.530 0.241 -
dense 0.343 0.528 0.309 +0.024 [0.003, 0.045]
denseBinary 0.344 0.529 0.307 +0.024 [0.003, 0.045]
hybrid 0.361 0.568 0.315 +0.041 [0.028, 0.055]
hybridRerank 0.361 0.587 0.315 +0.041 [0.027, 0.056]

Per query (ms, p50/p95): BM25 0.29/3.03; embed query 2.9/3.6; dense exact 1.16/1.68; dense binary+rescore 0.79/1.08; fuse 0.07; cross-encoder top 20 343/386.
Index: 5.3 MB float, 0.17 MB binary; binary top-10 agreement with exact 98.9%; documents embedded in cache.

Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-nfcorpus.json

@github-actions

Copy link
Copy Markdown

Semantic search bench, scifact (4141fda)

scifact (5183 docs, 300 test queries)

arm nDCG@10 MRR@10 Recall@100 nDCG@10 vs BM25 (95% CI)
bm25 0.672 0.642 0.886 -
dense 0.706 0.674 0.945 +0.034 [-0.000, 0.069]
denseBinary 0.706 0.674 0.937 +0.034 [-0.000, 0.069]
hybrid 0.703 0.667 0.968 +0.031 [0.007, 0.054]
hybridRerank 0.684 0.650 0.968 +0.012 [-0.020, 0.044]

Per query (ms, p50/p95): BM25 4.77/10.07; embed query 5.7/7.8; dense exact 2.44/2.70; dense binary+rescore 1.28/1.61; fuse 0.08; cross-encoder top 20 458/512.
Index: 7.6 MB float, 0.24 MB binary; binary top-10 agreement with exact 99.7%; documents embedded in 284 s.

Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-scifact.json

Pooling ablation: mean (SciFact)

scifact (5183 docs, 300 test queries)

arm nDCG@10 MRR@10 Recall@100 nDCG@10 vs BM25 (95% CI)
bm25 0.672 0.642 0.886 -
dense 0.704 0.673 0.943 +0.032 [-0.004, 0.067]
denseBinary 0.704 0.673 0.950 +0.032 [-0.004, 0.067]
hybrid 0.704 0.671 0.965 +0.032 [0.009, 0.054]
hybridRerank 0.690 0.655 0.965 +0.018 [-0.015, 0.049]

Per query (ms, p50/p95): BM25 4.81/10.09; embed query 5.7/8.0; dense exact 2.43/3.83; dense binary+rescore 1.28/1.66; fuse 0.09; cross-encoder top 20 458/509.
Index: 7.6 MB float, 0.24 MB binary; binary top-10 agreement with exact 99.6%; documents embedded in 302 s.

Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-mean.json

@charan-rathore
charan-rathore merged commit c2137e2 into main Nov 19, 2026
8 checks passed
@github-actions

Copy link
Copy Markdown

Semantic search bench, fiqa (b0e535a)

fiqa (57638 docs, 648 test queries)

arm nDCG@10 MRR@10 Recall@100 nDCG@10 vs BM25 (95% CI)
bm25 0.249 0.302 0.538 -
dense 0.399 0.480 0.693 +0.150 [0.126, 0.175]
denseBinary 0.399 0.480 0.679 +0.150 [0.127, 0.175]
hybrid 0.359 0.434 0.682 +0.110 [0.095, 0.125]
hybridRerank 0.376 0.445 0.682 +0.127 [0.108, 0.147]

Per query (ms, p50/p95): BM25 36.84/89.80; embed query 8.5/15.6; dense exact 42.18/47.23; dense binary+rescore 4.22/6.28; fuse 0.11; cross-encoder top 20 895/932.
Index: 84.4 MB float, 2.64 MB binary; binary top-10 agreement with exact 98.5%; documents embedded in 2968 s.

Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-fiqa.json

@github-actions

Copy link
Copy Markdown

Semantic search bench, fiqa (4141fda)

fiqa (57638 docs, 648 test queries)

arm nDCG@10 MRR@10 Recall@100 nDCG@10 vs BM25 (95% CI)
bm25 0.249 0.302 0.538 -
dense 0.397 0.479 0.691 +0.148 [0.125, 0.172]
denseBinary 0.398 0.480 0.677 +0.148 [0.126, 0.173]
hybrid 0.360 0.432 0.683 +0.111 [0.096, 0.126]
hybridRerank 0.376 0.444 0.683 +0.127 [0.108, 0.147]

Per query (ms, p50/p95): BM25 25.52/60.95; embed query 5.7/7.6; dense exact 36.92/46.70; dense binary+rescore 4.32/4.48; fuse 0.11; cross-encoder top 20 945/1042.
Index: 84.4 MB float, 2.64 MB binary; binary top-10 agreement with exact 98.6%; documents embedded in 2926 s.

Saved /home/runner/work/IntelliRAG/IntelliRAG/web/semantic-bench-fiqa.json

This branch was successfully deployed

1 active deployment
Preview — bdd965ab Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants