Personalized RAG Tutor AI

Repository: github.com/sahilleth/RAG-tutor

Personalized RAG Tutor AI is a full-stack tutoring application that lets users upload PDF learning material, build a retrieval index from that material, and chat with an AI tutor that answers using the uploaded content as grounding context.

The project is designed around a production-style Retrieval-Augmented Generation pipeline:

• PDFs are uploaded through the frontend and stored in AWS S3 (synchronous upload — no background race before Train).
• The backend ingests PDFs from S3 using PyMuPDF, with optional OCR for scanned pages.
• Text is extracted page by page, cleaned, semantically chunked, and stored in Pinecone.
• Each chunk is indexed with both dense embeddings and BM25 sparse token weights.
• Already-indexed files are skipped on re-train (idempotent ingestion via manifest + Pinecone metadata).
• Chat requests use hybrid retrieval, reranking, MMR diversification, and relative score filtering before generating tutor responses.
• Answers stream over SSE; sources show page, snippet, and relevance score in the UI.
• Long-running ingestion runs in a Celery worker through Redis so the API does not timeout.

Tech Stack

Frontend

• Next.js 14
• React 18
• TypeScript
• Tailwind CSS
• GSAP animations
• react-markdown for formatted tutor replies (no raw ** / ### in chat)
• IndexedDB chat persistence with automatic reset when materials change
• Upload progress bar (XHR), API detail error messages, and mobile-friendly chat layout

Backend

• FastAPI
• Python 3.11
• Gunicorn + Uvicorn worker
• LangChain
• LangChain Experimental SemanticChunker
• PyMuPDF / fitz for PDF loading and text extraction
• Celery for background ingestion jobs
• Redis as Celery broker, result backend, and answer cache
• slowapi for rate limiting on /ask and upload endpoints
• Retry helpers for Hugging Face embeddings and Pinecone upserts
• Structured logging (no print() in workers)

Retrieval and Storage

• AWS S3 for uploaded PDF storage
• Hugging Face Inference Providers for dense embeddings
• ibm-granite/granite-embedding-97m-multilingual-r2
• Pinecone for dense + sparse hybrid search (per-user namespaces)
• bm25s for lightweight BM25 sparse encoding (persisted per namespace)
• Google Gemini for tutor generation (pin model in .env, e.g. gemini-2.0-flash)
• Cross-encoder reranking via sentence-transformers (optional Cohere rerank API)
• Optional LangSmith tracing for retrieval + generation debugging

Project Flow

flowchart TD
    A["User uploads PDFs"] --> B["FastAPI upload endpoint"]
    B --> C["Compress and validate PDF with fitz"]
    C --> D["Store PDF in AWS S3 synchronously"]
    D --> E["User clicks Train Tutor"]
    E --> F["FastAPI submits Celery task"]
    F --> G["Redis broker queues task"]
    G --> H["Celery worker runs ingestion"]
    H --> I["Download PDFs from S3"]
    I --> J["Extract text page by page with fitz"]
    J --> K["Page-level semantic chunking"]
    K --> L["Dense embedding model"]
    K --> M["BM25 sparse encoder"]
    L --> N["Single hybrid Pinecone upsert payload"]
    M --> N
    N --> O["Pinecone hybrid index"]
    P["User asks question"] --> Q["FastAPI tutor endpoint"]
    Q --> R["Hybrid dense + BM25 retrieval"]
    R --> S["Rerank + filter + Gemini answer"]
    S --> T["Frontend streams response + sources"]

Loading

Ingestion Flow

The ingestion service lives in:

rag-tutor-ai-backend/app/services/ingest.py

The high-level ingestion process is:

1. List available PDFs from S3 (scoped by namespace prefix when X-RAG-Namespace is set).
2. Check indexed-file manifest and Pinecone metadata to avoid reprocessing already indexed files.
3. Download new PDFs from S3.
4. Open each PDF using fitz.open(stream=pdf_bytes, filetype="pdf").
5. Extract text one page at a time. Scanned pages can fall back to OCR when OCR_ENABLED=true (requires Tesseract + pytesseract on the host).
6. Clean text by normalizing whitespace, fixing hyphenation, and removing simple page labels.
7. Create one LangChain Document per PDF page with metadata:

{
    "source_file": "...",
    "source_name": "...",
    "page": 0
}

8. Run page-level semantic chunking (or fixed-size chunks when FAST_INGEST_MODE=true).
9. Log the top generated chunks for debugging.
10. Upload chunks to Pinecone in batches of 100 (with retry on transient HF / Pinecone failures).

PDF parsing is run concurrently with a ThreadPoolExecutor, while the expensive chunking and embedding work happens inside the Celery worker in production.

Upload endpoint behavior

Uploads are handled synchronously in POST /api/v1/ingest/upload-multiple-pdf so S3 finishes before the user can Train. Limits are enforced per file, per request total, and by max file count (MAX_UPLOAD_FILE_MB, MAX_UPLOAD_TOTAL_MB, MAX_FILES_PER_REQUEST). Pass replace_library=true to delete existing PDFs and vectors before uploading new ones.

Page-Level Semantic Chunking

The project uses page-level semantic chunking instead of splitting the full book as one giant text blob.

That means each PDF page becomes its own starting document before semantic splitting. This keeps chunks tied to the correct page and makes citations/debugging much easier.

Current chunking strategy:

• Process pages in batches of 10.
• Skip semantic chunking for short pages under 600 characters.
• Store short pages as a single chunk with chunking_strategy = "short_page_skip".
• For medium pages under 1000 characters, use RecursiveCharacterTextSplitter with overlap (recursive_overlap).
• For longer pages, use LangChain SemanticChunker.
• Use the vector embedding model to detect semantic breakpoints.
• Use percentile breakpoint thresholding with threshold amount 70.
• Use min_chunk_size = 450.
• Default chunk size 1000 with overlap 120.

Each generated chunk keeps page-aware metadata:

{
    "source_file": "...",
    "source_name": "...",
    "page": 12,
    "chunking_strategy": "langchain_page_semantic",
    "page_chunk_index": 0,
    "page_chunk_count": 3,
    "page_context": "..."
}

Why this matters:

• Chunks stay grounded to the original PDF page.
• Retrieval results can show page-level references.
• Short front-matter pages avoid unnecessary embedding calls.
• Long pages are split by meaning, not only by character count.
• Large PDFs avoid request timeout because chunking runs in Celery.

Set FAST_INGEST_MODE=true in .env to skip semantic chunking and use fixed-size recursive splits only (faster ingest, lower embedding cost).

Hybrid Retrieval

The project uses a single Pinecone index for hybrid retrieval.

Each stored chunk contains:

• Dense embedding vector from Hugging Face.
• Sparse BM25 vector from bm25s.
• Metadata, including source file, page, chunk indexes, and original text.

The BM25 implementation is in:

rag-tutor-ai-backend/app/db/bm25.py

The sparse encoder:

1. Tokenizes chunk text with bm25s.
2. Builds BM25 token weights.
3. Hashes each token into a stable Pinecone sparse index.
4. Maps the result into Pinecone's required sparse payload:

{
    "indices": [123, 456],
    "values": [0.82, 0.41]
}

During upsert, each chunk becomes one hybrid Pinecone vector:

{
    "id": "...",
    "values": dense_vector,
    "sparse_values": {
        "indices": [...],
        "values": [...]
    },
    "metadata": {
        "text": "...",
        "retrieval_strategy": "hybrid_dense_bm25_v2",
        "source_file": "...",
        "page": 0
    }
}

At query time:

• The question is embedded into a dense vector.
• The question is encoded into a BM25 sparse vector.
• HYBRID_ALPHA controls the weighting.

HYBRID_ALPHA=0.5

Higher values favor dense semantic similarity. Lower values favor BM25 keyword matching.

Optional source_file metadata filter scopes retrieval to a single PDF (active document selector in chat).

Retrieval, Reranking, and Generation

After hybrid search, the pipeline in rag-tutor-ai-backend/app/rag/ applies:

1. Multi-query retrieval — optional query expansion, HyDE, and history-aware rewriting via Gemini. Set ENABLE_HYDE=false and ENABLE_QUERY_EXPANSION=false in dev to avoid burning free-tier quota (each disabled feature saves 1–2 Gemini calls per question).
2. Merge — deduplicate chunks across query variants, keep best score.
3. MMR diversification — prefer diverse pages; penalize weak matches from other PDFs.
4. Reranking — retrieve top RERANK_TOP_N (default 15), rerank, keep top RETRIEVAL_K (default 5).
   • RERANKER_BACKEND=cross_encoder — sentence-transformers (cross-encoder/ms-marco-MiniLM-L-6-v2)
   • RERANKER_BACKEND=keyword — lightweight keyword overlap fallback
   • RERANKER_BACKEND=cohere — Cohere rerank API
   • Cross-encoder scores are kept separate from hybrid retrieval scores; chunks below RERANK_MIN_SCORE are dropped.
5. Relative score filtering — RETRIEVAL_MIN_RELATIVE_SCORE drops weak cross-PDF matches (e.g. wrong PDF appearing in sources).
6. Page context enrichment — parent page text attached for better generation context.
7. Citation grounding — parse Sources: p. N from the answer and validate against retrieved chunk pages; strip unverified references.
8. Gemini generation — chat history sent with each request; answers cached in Redis (ENABLE_ANSWER_CACHE, short TTL).
9. Streaming — POST /api/v1/tutor/ask/stream returns SSE tokens, then final sources/citations payload.

Debug retrieval without generation: POST /api/v1/tutor/retrieve-debug.

Chat and studio UX

• Train Tutor is the primary CTA when no embeddings exist (not a dead “Chat” button).
• Active document dropdown limits chat to one PDF; default searches all indexed PDFs.
• Replace library checkbox on upload clears old S3 files and Pinecone vectors first.
• Sources used panel expands to show page number, snippet, and score.
• Chat auto-clears when materials change (upload, train, or delete) via IndexedDB fingerprint sync.
• Friendly error text when Gemini returns 429 quota errors.

Production System Design With Redis and Celery

PDF ingestion can take too long for a normal web request, especially for large books with hundreds of pages. To avoid request timeouts, ingestion is moved out of the FastAPI request process and into a Celery worker.

Production architecture:

flowchart LR
    A["FastAPI Web Service"] --> B["Redis Broker"]
    B --> C["Celery Worker"]
    C --> D["S3 PDF Download"]
    C --> E["Semantic Chunking"]
    C --> F["Dense + BM25 Encoding"]
    C --> G["Pinecone Upsert"]
    C --> H["Redis Result Backend"]
    I["Frontend"] --> A
    I --> J["Poll Task Status"]
    J --> A
    A --> H

Loading

The request flow is:

1. Frontend calls:

POST /api/v1/ingest/run-ingestion?reset_db=false

2. FastAPI submits a Celery task (with namespace from X-RAG-Namespace) and immediately returns:

{
  "status": "queued",
  "task_id": "...",
  "message": "Ingestion task queued."
}

3. Celery worker receives the task from Redis.
4. Worker performs PDF ingestion, semantic chunking, embeddings, BM25 encoding, and Pinecone upsert.
5. Frontend polls:

GET /api/v1/ingest/ingestion-tasks/{task_id}

6. The UI shows task states:

PENDING -> STARTED -> SUCCESS
PENDING -> STARTED -> FAILURE

While STARTED, task meta reports per-PDF progress (extracting, chunking, embedding) so the studio status line updates beyond bare PENDING / SUCCESS.

This keeps the web API responsive while ingestion runs in the background.

render.yaml defines three Render services: the FastAPI web app, a dedicated Celery worker (rag-tutor-celery), and Redis (Key Value). Without the worker service, ingestion tasks queue but never run in production.

Per-User Namespaces

For multi-tenant isolation, each client can send an X-RAG-Namespace header. This scopes:

• Pinecone vectors to a namespace
• S3 object prefix (materials/{namespace}/...)
• BM25 corpus files on disk (data/namespaces/{namespace}/)
• Indexed-file manifest per namespace

The frontend stores a namespace in localStorage after the first upload. Requests without a namespace use PINECONE_DEFAULT_NAMESPACE (empty string keeps backward compatibility with a shared index).

Local Setup

Clone the repository:

git clone https://github.com/sahilleth/RAG-tutor.git
cd RAG-tutor

Backend

cd rag-tutor-ai-backend
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env
# Fill in GOOGLE_API_KEY, HUGGINGFACEHUB_API_TOKEN, PINECONE_*, AWS_*

Create a Pinecone index with dimension 384 and metric dotproduct.

Start Redis, then run three processes:

# Terminal 1 — API
source .venv/bin/activate
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN 2>/dev/null
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

# Terminal 2 — Celery worker
source .venv/bin/activate
celery -A app.core.celery_app.celery_app worker --loglevel=info

# Terminal 3 — Frontend
cd ../rag-tutor-ai-frontend
npm install
npm run dev

Open http://localhost:3000.

If shell AWS_* variables are exported, they override .env and can cause S3 authentication errors. Unset them before starting the backend.

Frontend environment

# rag-tutor-ai-frontend/.env.local
NEXT_PUBLIC_API_BASE_URL=http://localhost:8000

Health, Rate Limits, and Observability

GET /health returns composite status for Redis, Pinecone, S3, and Celery worker availability, plus an alerts array when any dependency is degraded (e.g. celery_worker_unavailable, pinecone_unavailable).

Rate limits apply to /api/v1/tutor/ask, /ask/stream, and upload endpoints (ASK_RATE_LIMIT, UPLOAD_RATE_LIMIT).

Optional features:

• Answer cache — Redis-backed cache for identical questions (ENABLE_ANSWER_CACHE, ANSWER_CACHE_TTL_SECONDS)
• LangSmith — LANGCHAIN_TRACING_V2=true, LANGCHAIN_API_KEY, LANGCHAIN_PROJECT
• Eval script — python scripts/eval_rag.py --questions data/eval_questions.example.json (also run in CI when Pinecone secrets are configured)

Testing

cd rag-tutor-ai-backend
source .venv/bin/activate
PYTHONPATH=. pytest tests/test_rag.py -q

Unit tests cover merge/dedup, MMR, relative score filtering, rerank thresholding, citation grounding, namespace sanitization, and BM25 round-trip.

License

MIT