The Bard's Guide To RAG

This repository is the companion code for The Bard's Guide to RAG by Tim Burns.

The article uses Shakespeare as a readable way to explain Retrieval-Augmented Generation (RAG). Its central idea is that search is not only about matching words. A reader can recognize jealousy, ambition, love, betrayal, or political corruption even when those exact words are missing. Embeddings help close that gap by representing passages as vectors, so related ideas can be found by meaning instead of exact phrasing.

The project turns that idea into a hands-on learning path: download the Shakespeare corpus, chunk the plays in several ways, create embeddings with Amazon Bedrock, index the results in OpenSearch, compare retrieval strategies, and finally use retrieved passages as evidence for generated answers.

Table Of Contents

• Article
• Project Map
• Other READMEs
• How To Start

Article

• Medium article: The Bard's Guide to RAG
• Subtitle: What 37 plays, four chunking strategies, and Retrieval-Augmented Generation (RAG) taught me about AI pipelines
• Topics: Shakespeare, retrieval-augmented generation, OpenSearch, Amazon Bedrock, and chunking strategies

In short: the article explains why ordinary keyword search struggles with literary meaning, then uses Shakespeare to show how chunking, embeddings, vector search, BM25, hybrid search, and evidence-grounded prompting fit together inside a small RAG pipeline.

Project Map

notebooks/

The main learning path. These notebooks are meant to be read and run in order. They start with downloading the Shakespeare corpus, then move through chunking, embedding, OpenSearch indexing, vector search, BM25 search, hybrid retrieval, prompting with evidence, and RAG evaluation.

Start here when you want to follow the essay as executable chapters.

infra/

Terraform and AWS helper code for the tutorial environment. The infrastructure creates a simple S3 source bucket, an OpenSearch search backend, and IAM access for notebook-driven ingestion and querying.

This folder also includes aws_cost_report.py, a small reporting script used by the Makefile to summarize tagged AWS costs.

infra/modules/

Small Terraform modules used by the Shakespeare stack:

• opensearch-serverless/ creates the default OpenSearch Serverless VECTORSEARCH collection.
• opensearch-provisioned/ provides an alternative provisioned OpenSearch domain for readers who want to compare the two service models.
• shakespeare-source-bucket/ creates the S3 bucket that stores raw corpus files and processed notebook outputs.

infra/stacks/shakespeare/

The runnable Terraform stack for the project. It wires the modules together, defines providers and variables, and exposes outputs consumed by the notebooks.

Use this folder when deploying or destroying the AWS resources.

src/python/util/

Small utility scripts that support the tutorial. Today this contains aoss_ping.py, which signs a request with AWS credentials and checks whether the OpenSearch Serverless collection is reachable.

Project Root

Root-level files provide the project entry points:

• Makefile creates the virtual environment, installs dependencies, writes notebook config from Terraform outputs, runs Python formatting and linting, checks OpenSearch connectivity, lists S3 files, and reports AWS costs.
• requirements.txt lists the Python packages used by the notebooks and helper scripts.
• AGENTS.md records the project philosophy: this is a teaching artifact, so code should stay small, readable, and directly tied to the essay.

Other READMEs

• Notebook Roadmap
• Shakespeare Terraform Stack

How To Start

1. Read the Medium article for the conceptual path.
2. Deploy the Terraform stack in infra/stacks/shakespeare/.
3. Run make config from the project root.
4. Open notebooks/01_download_corpus.ipynb.
5. Work through the notebooks in order.

The intended pace is essay-first: read the explanation, run the smallest useful piece of code, inspect the result, and then move to the next idea.