Skip to content

About

RAG platform built from scratch - async pipelines, 5 chunking strategies, RAGAS evaluation, Celery workers, PostgreSQL lifecycle management

Topics

Resources

Stars

5 stars

Watchers

0 watching

Forks

Repository files navigation

IntelliRAG

Retrieval that decides what it can prove before a model speaks.

Ask a question about a pinned source. IntelliRAG finds the passages, checks whether they actually support an answer, packs them into a fixed token budget, and tells you which parts of your question the context never mentions. If the evidence isn't there, it refuses instead of improvising.

Try the live console · Interactive architecture · Browser walkthrough · Measured evaluation · Code graph

How IntelliRAG works. Before any question (offline): bring sources, cut into passages, index every word, optionally add meaning vectors, store; a sleep cycle replays rated questions. On every question (online): ask, read the code it names, search two ways, rank with memory, evidence gate, answer or refuse. After the answer: your rating is filed and improves the next answer.

In plain words

  • Offline, once per source: IntelliRAG reads your repository or documents, cuts them into passages and builds an index. Nothing here waits on a question.
  • Online, per question: it finds the passages that match, ranks them, and lets only the ones that clear an evidence gate be cited. That's about 5 ms of search before any model runs. With no passage strong enough, it says the source doesn't cover it.
  • Afterwards: marking an answer useful or wrong, or picking the passage that answered you, makes the next similar question better. A background "sleep" builds on those ratings, and keeps a change only if no answer that already works breaks.

Watch the 57-second walkthrough: a question, its sources, and the proof

The problem

Retrieval-augmented systems rarely fail by answering from nothing. They fail by answering with sources attached. In a 2026 penalty-aware audit, 98% and 100% of two commercial RAG products' wrong answers came with cited sources or grounding chunks (arXiv:2608.26385). Practitioners describe the same failure: the retriever returns something topically close, and the model fills the gap convincingly.

Most of these errors happen when the retrieved context is relevant but insufficient (Sufficient Context, ICLR 2025). So IntelliRAG treats "is this enough to answer?" as a first-class, inspectable decision, not something left to the prompt.

What it does differently

Stage Decision Where
Retrieve BM25, plus dense vectors only when compatible stored embeddings exist, fused with RRF retrieve-core.ts
Rerank A calibrated mix of idf recall, phrase and title overlap. It is not a cross-encoder and not a confidence score. ranking.ts
Gate A passage needs a support floor and distinctive overlap; a decisive rank-1 margin keeps only that source. If nothing passes, the answer is a refusal with no citations. ranking.ts, evidence.ts
Pack 1,400-token budget. Passages are verbatim slices of the stored chunk, code fences are never split, and a chunk that doesn't fit is trimmed to its most relevant contiguous paragraphs. ranking.ts, coverage-pack.ts
Measure Lists the question terms (function words and quantities removed) that appear in no packed passage, shown under every answer coverage-pack.ts, context-coverage.tsx
Answer With a server or visitor key: temperature 0 and a cite-or-refuse prompt. Without one: cited extracts, labelled as extracts. query.server.ts
Cache Keyed by question, corpus and a policy hash covering retrieval mode, evidence policy, model and graph edits. Corrections invalidate it. graphify/persist.server.ts, query.server.ts

The pipeline in detail

How a question becomes a cited answer or a refusal: 1 ask, 2 query, 3 rank, 4 check support (refuse if nothing is supported), 5 pack within 1,400 tokens, 6 measure coverage, 7 answer, 8 answer with trace; A ingest a pinned GitHub source, B optional model, C a frozen eval replays the same functions

Follow the numbers. 1–2: you ask. 3–7: the evidence is decided before any model runs. 8: you get the answer, its citations, and the question terms nothing in the context covers. A–C happen offline: ingest at a pinned commit, an optional model, and a frozen evaluation that calls the same functions the server does.

The web pipeline, stage by stage: index on import, answer on request, evidence gate, optional second opinion, inspect and give feedback

Click the pipeline for the interactive version, where every stage links to its source file.

The coverage line reads like this on a question the source can't settle:

5/9 question terms in context · not found: server processes durable

That is the honest summary of "does this README establish a shared, durable rate limit across three servers?": it doesn't.

Measured, including what lost

One pinned README (sindresorhus/p-queue at 180ab9e), 15 answerable questions with exact evidence anchors, 5 that must be refused, and the same 1,400-token budget for every arm. No model runs in these numbers; they measure whether the required evidence reached the reader.

Arm Clean corpus + distractor documents Refused 5/5
Plain BM25 73.3% 63.3% —
IntelliRAG, rank-order packing (default) 73.3% 83.3% ✓
IntelliRAG, coverage-aware packing (opt-in) 76.7% 73.3% ✓

Evidence recall on the clean corpus and with distractors, and every question with distractors including the two neither method answers

The default ties BM25 on the clean corpus and beats it by 20 points once noise documents are mixed in. The coverage packer wins one extra question on the clean corpus, but the gain's bootstrap interval is [−10, +20] points, and it loses 10 points under distractors. So it stays opt-in. Every run, including the losing ones, is in eval/repo-support/runs/, and the evaluation write-up covers method, history and limits.

Stage time per query on a local CPU: keyword retrieval about 0.1 ms, reranking about 4 ms, context assembly about 0.4 ms

Retrieval itself is fast: the median query spends about 0.1 ms on keyword retrieval, 4 ms reranking and 0.4 ms packing on a laptop CPU. Network, database, embeddings and the model are not included. The September before/after chart shows how a deduplication bug cost half the evidence (36.7%) and what fixing it recovered.

Tested on real GitHub issues

170 real GitHub issues, before and after

Paste a bug report and IntelliRAG ranks the repository's files. I tested it on 170 real issues from django, scikit-learn, sphinx and pytest (SWE-bench Lite), using the reporter's own text. The file the fix changed now comes first 45.3% of the time, up from 35.3%, and is in the top 5 65.9% of the time, up from 60.6%. No model is involved. The gain comes from reading the code an issue names (traceback paths, module.Class, function calls) and linking it to the files that define it. Settings were tuned on different issues (sympy). Method, per-repository numbers and a negative result: eval/github-issues.

It sleeps

The design follows Google's "Language Models Need Sleep" (Behrouz, Hashemi, Mirrokni) and Self-Improvement Memory (Yu et al., arXiv 2611.18856). IntelliRAG has no weights to train, so the sleep acts on its memory:

Phase What happens Guard
Awake Every rated answer and every picked source is filed against the passages it came from. The source text is never edited.
Replay During sleep, the questions people confirmed are asked again. This is the metric.
Consolidate It learns which words people use for which words the sources use. Kept in separate memory; the source's term statistics never change.
Dream It writes practice questions for passages nobody has asked about: words hidden in identifiers, section context, people's vocabulary, or a model's questions when one is configured. Each dream is tried on a copy of memory. It is kept only if its passage is now found, no confirmed question is lost, and no dead end comes back. A final full replay rolls back the whole sleep if anything slipped through.
Journal Each strategy's kept-of-tried count and the reason for every rejection are recorded. The next sleep budgets toward what worked and pauses a strategy that failed twice running.

It sleeps by itself after every 8 ratings, or when you press Sleep now in the Memory panel. The panel shows what was kept and why everything else wasn't.

Measured honestly. On the 14-question rewording test, one sleep neither helped nor hurt (54.8% before and after on held-out wordings). On the GitHub issues, sleep's guard cut the damage of a memory that didn't transfer from −6.5 to −1.8 points, but didn't turn it into a gain. The gains so far come from ratings and picks (held-out rewordings 50.0% → 54.8%, original questions 71.4% → 81.0%) and from the code links above. With a model configured, the model-written dream strategy is where the paper expects the bigger effect. That hasn't been measured here yet.

Answers never stop mid-sentence. If the model runs out of output budget, it continues from where it stopped. If it still can't finish, the answer is cut back to its last whole sentence or list item, with a one-line note.

It gets better the more it's used

Held-out rewordings and original questions, per learning arm, on the clean corpus and with distractors

Three loops improve IntelliRAG as people use it. Each is measured on held-out data, and each can only add evidence: none of them can make an unsupported question look answered.

Loop What it learns Measured
Conversation (session-intent.ts) A follow-up is searched with the earlier questions on its topic Follow-ups with complete evidence: 63.6% → 86.4%, refusals 1/3 → 2/3 (eval)
Experience (experience.ts) A rated answer, or a passage picked as the one that answers a question, is filed against that passage. Later questions worded like it find it. Originals asked again: 71.4% → 81.0%; new wordings +2.4 to +4.8 points; 20% wrong ratings cost nothing (eval)
Self-improvement memory (sim-optimize.ts, after arXiv 2611.18856) The retrieval policy is edited round by round. Each attempt is recorded with its root cause, rationale, predicted fixes and observed result. Held-out wording 52.9% → 58.8%. Its accepted packing edit also lifted the follow-up eval it never saw by 9 points (write-up, including the null result)

Experience is the "file good answers back" step of Karpathy's LLM-wiki pattern, done per passage: the source is never edited, and links go stale when the passage they learned about changes. Agents get the same memory through the MCP server's feedback tool. It is kept per repository, so a repo keeps learning across sessions.

Search like GitHub code search

Press ⌘K (Ctrl+K) to search every indexed source instantly. It splits identifiers (parseJSONResponse matches "parse json response"), weights function names and docstrings, and takes operators: path:src/queue, lang:ts, sym:resume, kind:prose, "exact words", -word. Results show the matching lines highlighted, with "did you mean" for typos and completions from your sources. The same engine answers GET /api/search?q=…&corpus=….

On CoSQA (313 real web queries, 6,267 Python functions) it scores 46.4 nDCG@10 against 42.9 for plain BM25, with no model. Details and an honest comparison with dense retrievers: eval/code-search.

Research behind the design

Paper What we took Status here
Sufficient Context (Joren et al., ICLR 2025) Errors concentrate where context is relevant but insufficient, so expose sufficiency before trusting fluency Lexical proxy shipped on every answer; an LLM autorater is not used
AdaGReS (2025) Redundancy-penalised greedy selection under a token budget with an adaptive penalty Implemented in coverage-pack.ts; opt-in after measurement
Lin & Bilmes (ACL 2011) Cost-benefit greedy plus a best-single-unit check for knapsack-constrained submodular selection Implemented; the guarantee holds only without the redundancy term
Recall Is Not Enough (2026) Score what the packed context contains, not the candidate list The eval scores the text actually packed, after trimming
Lost in the Middle (TACL 2024) Position matters; keep the strongest evidence first The top supported chunk is the whole-chunk anchor

Use it from your coding agent

IntelliRAG also runs as an MCP server with no dependencies, so Claude Code, Codex, Hermes or any MCP client can answer from a repository's docs with citations, or refuse:

claude mcp add intellirag -- npx -y github:charan-rathore/IntelliRAG                    # Claude Code
codex mcp add intellirag -- npx -y github:charan-rathore/IntelliRAG                     # Codex
hermes mcp add intellirag --command npx --args -y github:charan-rathore/IntelliRAG     # Hermes Agent
> index_folder                        (the repo you're in, or index_github owner/repo at a pinned commit)
> ask  "How do I cancel a queued task with an AbortSignal?"   session: fix-1234
  3 supported passages from sindresorhus/p-queue@180ab9e …
> ask  "What if it is already running?"                        session: fix-1234
  Reading: Read as a follow-up: searched together with the previous question (about abortsignal, cancel).
  [readme.md#14] readme.md › options › signal
  If the operation is already running, the signal will need to be handled by the operation itself.
> ask  "What production database password does it specify?"   session: fix-1234
  REFUSED. Nothing in this corpus passed the support gate for this question.

It exposes seven tools: index_folder, index_github, ask, search, feedback, new_topic and sources. They call the same retrieval, support gate, packing and conversation reading as the console (web/src/lib/rag), bundled into mcp/dist and checked over stdio in CI. The agent skill is skills/intellirag/SKILL.md, written to Hermes Agent's authoring standard. The Hermes catalog entry is integrations/hermes/manifest.yaml.

Run it

git clone https://github.com/charan-rathore/IntelliRAG.git
cd IntelliRAG/web
npm ci
npm run dev            # http://localhost:8080, pinned source works without a key
npm run typecheck && npm run build
npx tsx --test src/lib/rag/*.test.ts                 # retrieval, gate, packing, persistence
npx tsx scripts/repo-support-eval.ts                 # the measured table above, no network
npx tsx scripts/repo-support-eval.ts --distractors   # the noise stress test

docs/WEB.md covers the Vercel setup, security posture and release gate. Durable imports, persistent semantic search and provider generation need private configuration; they aren't a frontend toggle.

Repository map

flowchart LR
  subgraph web["web/ · the served app"]
    UI["src/components<br/>console, inspector, coverage"] --> Q["lib/rag/query.server.ts<br/>stream: sources → tokens → done"]
    Q --> R["retrieve-core.ts<br/>BM25 · dense · RRF"]
    R --> K["ranking.ts<br/>rerank · gate · pack"]
    K --> C["coverage-pack.ts<br/>budgeted selection · coverage"]
    R --> E["evidence.ts<br/>classify support"]
    Q --> S["store.server.ts<br/>Postgres / PGlite · cache"]
    I["ingest.server.ts<br/>chunking · GitHub · SSRF guard"] --> S
  end
  EV["web/scripts/repo-support-eval.ts"] -.same functions.-> R
  PY["rag-platform/<br/>Python reference + benchmarks"]
Loading
Folder What's in it
web/ The app you use: TanStack Start UI and the retrieval pipeline in web/src/lib/rag/
mcp/ The same pipeline as an MCP server for Claude Code, Codex and Hermes
eval/ Frozen benchmarks: repository support, rewordings, conversations, code search, real GitHub issues
docs/ Diagrams, deployment (WEB.md), roadmap, audit trail, code graph
rag-platform/ Python reference implementation and benchmarks (not the served runtime)
skills/, integrations/ Agent skill and editor integrations
scripts/ Code graph and diagram generators

docs/codegraph/ is a generated dependency graph of the whole repository (files, imports, hubs and import cycles), rebuilt with node scripts/codegraph.mjs. Start there to find the blast radius of a change. The Python rag-platform/ is a separate benchmark and reference implementation; it is not the serving runtime.

Limits

  • The evaluation is one README with project-authored annotations. Both splits have been inspected, so the results are exploratory, not a blind holdout.
  • The public host has no durable vector index or server model key. It shows keyword retrieval and cited extracts honestly; it doesn't demonstrate production semantic RAG.
  • The coverage check is lexical. A paraphrase can answer a "missing" term, and a matched term can still be used in an unrelated sense.
  • The graph view shows lexical document links, not verified entity extraction.

The target is verified task completion per unit of compute. A change ships when it beats the frozen baseline at the same budget and its failures are published.

License

MIT

About

RAG platform built from scratch - async pipelines, 5 chunking strategies, RAGAS evaluation, Celery workers, PostgreSQL lifecycle management

Topics

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages