Skip to content

fix(rag): derive readiness from complete document embeddings - #148

Merged
charan-rathore merged 1 commit into
mainfrom
fix/embedding-readiness-transitions
Nov 22, 2026
Merged

charan-rathore merged 1 commit into
mainfrom
fix/embedding-readiness-transitions

Conversation

@charan-rathore

Copy link
Copy Markdown
Owner

Readiness could become stale after replacing some vectors, and a batch covering several documents could assign its first model to all of them. Direct chunk replacement also left old readiness behind.

Recompute each touched document from its complete chunk set. Only fully embedded, model-consistent documents are ready. Lock parent documents in sorted order before batched vector updates, and invalidate the parent before direct chunk replacement. The in-memory store follows the same transitions.

Validation on main plus this change:

  • Native npm test: 776 passing, including 505 RAG/motion/database cases, 227 script checks, 32 app/auth cases and 12 generation checks.
  • Typecheck and production build pass. Build had no DATABASE_URL and skipped external migrations.
  • Hardening, persistence and the separate in-memory regression pass. Conversation and learning checks pass with and without distractors.
  • PostgreSQL 16.15: five forced schedules verify real lock waits and final states for partial batches, conflicting models, both embedding/replacement orders, stale chunk IDs and opposite document input orders.
  • Paired local PGlite measurements: four-chunk median 2.638 to 2.543 ms; 27-chunk median 11.099 to 3.964 ms. These are synthetic local measurements, not hosted latency.

No schema, retrieval ranking, vector-dimension policy or provider change. The PostgreSQL evidence is bounded concurrency coverage, not hosted configuration parity, soak testing or ambiguous-commit recovery. Readiness continues to mean complete non-null, consistent-model coverage, not universal stored-vector validity.

@vercel

vercel Bot commented Nov 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
intellirag-live-own-track Ready Ready Preview Nov 22, 2026 9:41pm UTC

@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-22T21:39:46.023709Z
We couldn’t complete this review. Comment @codex review to try again.
43a3114 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@charan-rathore
charan-rathore merged commit b6384fc into main Nov 22, 2026
5 checks passed

This branch was successfully deployed

1 active deployment
Preview — 43a31141 Deployed Nov 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant