This is not an introduction to RAGs. Knowledge of what embeddings or re-ranking are is assumed. I also tried to focus this guide to a few core aspects: the evals (evaluations), the analysis (how to measure changes made to the RAG) and the security features.
The main differentiators of this RAG are:
-
Modular pipeline: RAGBench is a pipeline with plug and play components. You can replace chunker, embedding model, re-ranker. Retrieval itself is not pluggable: both the keyword (BM25) and vector legs are Postgres indexes over the same table.
-
Laptop mode uses Docker Compose with OpenAI for inference and judging. It is appropriate for public datasets, not for demonstrating confidential-corpus privacy.
-
AWS private mode keeps the Compose application on the demo EC2 instance and runs inference plus judging on a separate private L40S vLLM instance in the same VPC. It has no laptop-to-AWS route or public model endpoint.
-
PII masking: When 3rd party models are used, Personally Identifiable Information (PII) can be masked to preserve the privacy of the info.
-
Evals: An evaluation stack is included to generate metrics on how well the current pipeline behaves.
-
Analysis: and finally, a dashboarrd uses the metrics generated by the evals to compare the effects of changing modules and settings in the pipeline.
This is currently a research project to explore different retrieval methods and the evals to measure them. It is not aimed at production. Specifically, it is missing the following features that would be needed in a deployment.
-
Auth: User authentication and authorisation are missing. The current RAG is wide open.
-
Live document updates: Re-indexing as documents are added, removed or modified. Deletion is handled (the foreign key cascade clears chunks and their embeddings), but there is no watch-and-reingest path for documents that change on disk.
-
multi tenancy
-
Observability: logs and alerts on both the infrastructure of the RAG and its internal behaviour, including the generation speed on load (tokens per minute).
-
Continuous Integration
-
Backups and disaster recovery
-
High Availability
-
Load balancing and multi concurrent user support.
-
Extensive test suites. From more unit tests to regression tests to E2E tests to detect any drifting of the codebase
I would also look at using a commonly used framework like Haystack as a starting point for a production system.
This codebase was almost entirely written using Claude Code and Codex. The groundwork was created using a manually written detailed PRD document, derived from my initial RAG reserach.
My workflow keeps evolving but as of writing, it follows the Planner, Implementer & Verifier pattern for new features:
- Planner: Start with a clear context and add a description of the end goal for a feature. Be careful not to tell the model how to do its work. Focus on constraints instead.
- Implementer: Use a strong model (Opus 5 High) and tell it to delegate the grunt work to sub agents. Alternatively, for more complex tasks I also use a loop technique (/goal). Looping consumes an inordinate amount of tokens so I often try to avoid it.
- Verifier: Ask a second strong model from a different provider (eg: OpenAI), typically GPT 5.6 Sol High, to perform a review of the changes made by Opus against the PRD.
I would like to eventually introduce a graduated review process where a model reviews PRs, scores them based on their risk and impact, and only notifies the developer for high impact tickets that need a human in the loop.
Before the first query, the task worker prepares each uploaded document:
- Docling parses its structure.
- The text is split into chunks.
- Each chunk is embedded and the vector is stored in PostgreSQL for vector search.
- The same PostgreSQL row holds the chunk text for BM25 search.
- If contextual retrieval is enabled, an LLM-generated description is added to each chunk before embedding.
Contextual retrieval can help chunks that make little sense alone, but it adds one LLM call per chunk. Changing the embedding model, chunking, or contextual retrieval requires re-ingestion.
| Component | Port | Role |
|---|---|---|
| Web app | 8000 | Chat, upload, settings, and analytics |
| RAG server | 8001 | Retrieval, generation, sessions, and metrics |
| Task worker | — | Background document processing |
| Eval service | 8002 | Evaluation runs and saved results |
| PostgreSQL | — | Documents, chunks, embeddings, chat history, tasks, and both search indexes |
| TEI | 80 (in-compose) | Self-hosted embedding inference, always on |
The eval service calls the same RAG server endpoint as the application. An end-to-end eval therefore measures the code path used by real queries.
| Stage | Common settings | Typical trade-off |
|---|---|---|
| Generation | Model and prompts | Quality, latency, cost, privacy |
| Embedding | Model | Retrieval quality; requires re-ingestion |
| Retrieval | Hybrid search, top_k, RRF |
Recall versus query time |
| Reranking | Enabled, model, top_n |
Ranking quality versus latency |
| Ingestion | Chunk size, overlap, contextual retrieval | Recall, precision, ingestion cost |
| Evaluation | Judge, metrics, scoring weights | Coverage, cost, and judge bias |
The settings used for the stages above depend heavily on the type of documents being embedded. For instance, legal documents would be expected to have multiple instances of the words "law" or "legal". Applying generic chunking and embedding rules to these would result in too many documents bunched up close together in the vector space, which would make it hard to find the most suitable documents for a query.
- Run an evaluation and save the baseline.
- Change one setting.
- Run the same evaluation again.
- Compare paired results and their uncertainty.
- Keep the change only if the improvement is credible and worth its cost.
Next: 2. Get RAGBench running.
Implementation detail: docs/internal/architecture.md.