Context_Engineering4LF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "html"
    }
   },
   "source": [
    "# Context Engineering: LLMs for LF\n",
    "\n",
    "## Objective\n",
    "Generate Lingua Franca (LF) code from natural language prompt.\n",
    "\n",
    "## Context\n",
    "Large Language Models (LLMs) are a powerful tool for generating code. However, LLMs are often trained on standard programming languages like C/C++, Java, Python, or JavaScript. This makes it difficult to generate code in Domain-specific languages like Lingua Franca.\n",
    "\n",
    "This project shows how to curate context to guide the LLM.  We demonstrate the technique by showing how to generate code that uses the Lingua Franca coordination language together with C and Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. LF code generation with manually engineered context\n",
    "\n",
    "### 1.1 Environment Setup:\n",
    "- Virtual environment installation.\n",
    "- Necessary libraries installation.\n",
    "- Necessary imports.\n",
    "- OPEN_AI_API_KEY loading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "# Installing Required Packages\n",
    "!python -m pip install --upgrade pip\n",
    "!python -m pip install python-dotenv openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing Libraries\n",
    "import os\n",
    "from dotenv import load_dotenv\n",
    "from openai import OpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `.env` file is used to store sensitive information that you don't want to hard-code into your source code.\n",
    "\n",
    "\n",
    "For this project `OPENAI_API_KEY` is stored in `.env` file as:\n",
    "\n",
    "\n",
    "`OPENAI_API_KEY='your_openai_api_key'`\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading OPENAI_API_KEY from the .env file\n",
    "load_dotenv()\n",
    "os.environ['OPENAI_API_KEY'] = os.environ.get('OPENAI_API_KEY')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 OpenAI models\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "html"
    }
   },
   "source": [
    "| Model | context length | Input | Output | Training Data|\n",
    "| :--- | :--- | :--- | :--- | :--- | \n",
    "| GPT-4o |  128k | \\$5  | \\$15  | Up to Oct 2023|\n",
    "| GPT-4 Turbo |  128k  |  \\$10 | \\$30  |Up to Dec 2023|\n",
    "| GPT-3.5 Turbo |  16k  |  \\$0.50 |  \\$1.50| Up to Sep 2021\n",
    "\n",
    "\n",
    "\\* prices per 1 million tokens.\n",
    "\\* 100 tokens ~= 75 words.\n",
    "\n",
    "The foolowing C++ line (10 words):\n",
    "\n",
    "`for (int i = 0; i < g[x].size(); i++) {⤶`\n",
    "\n",
    "is composed of 23 tokens:\n",
    "\n",
    "~~~~\n",
    "for \n",
    "(\n",
    "i\n",
    "nt \n",
    "i\n",
    " \n",
    "= \n",
    "0; \n",
    "i\n",
    " \n",
    "<\n",
    " \n",
    "g\n",
    "[\n",
    "x\n",
    "]\n",
    ".s\n",
    "ize(\n",
    "); \n",
    "i\n",
    "+\n",
    "+) \n",
    "{⤶\n",
    "~~~~\n",
    "\n",
    "- A 50 lines C++ code ~= 600 tokens. Which costs 0.3 cents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Prompting OpenAI models\n",
    "#### 1.3.1 Without context\n",
    "We start by prompting OpenAI LLMs: `gpt-3.5` and `gpt-4o` without any additional context.\n",
    "\n",
    "The generated code will be stored under `LFGPT/without_context`.\n",
    "\n",
    "We start by creating the target directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "# Creating the target directory\n",
    "!mkdir -p LFGPT/without_context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Introduce a `run_iterator` variable to prevent overwriting the generated code during multiple executions.\n",
    "\n",
    "Define the prompt for a DropSensor reactor.\n",
    "\n",
    "![DropSensor sample](img/DropSensor.png)\n",
    "\n",
    "- Triggers the accelerometer with period of 250 ms.\n",
    "- If the absolute values of the accelerometer readings (`x`, `y`, `z`) are all less than the defined `threshold`, then `print(x, y, z)`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "# Defining the prompt\n",
    "prompt = f\"\"\" \n",
    "  Provide a Lingua Franca code for 'target C'. \n",
    "  Import the LF 'Accelerometer' reactor from library.\n",
    "  Create a main reactor named `DropSensor` that:\n",
    "  - has a parameter named `threshold`, with a default value of '0.3'.\n",
    "  - instantiates an accelerometer.\n",
    "  - declares a timer with a period of 250 ms.\n",
    "  - Has a reaction that triggers the accelerometer with period of 250 ms by setting the accelerometer trigger to true.\n",
    "  - has another reaction which fires whenever any of the accelerometer readings (`x`, `y`, or `z`) change. Inside this reaction: It checks whether the absolute values of the accelerometer readings (`x`, `y`, `z`) are all less than the defined `threshold`. If the condition is true, it prints the values of the accelerometer outputs to the console.  \n",
    "  \"\"\"\n",
    "# File name suffix\n",
    "file_name = \"LFGPT/without_context/DropSensor\"\n",
    "\n",
    "# Defining run iterartor\n",
    "run_iterator = 0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initializes an OpenAI client for interacting with the OpenAI API.\n",
    "client = OpenAI()\n",
    "\n",
    "\n",
    "MODELS = [\"gpt-4o\", \"gpt-3.5-turbo\"] \n",
    "run_iterator += 1\n",
    "\n",
    "for MODEL in MODELS:\n",
    "  fout = open(f'{file_name}_{MODEL}_{run_iterator}.lf', 'w')\n",
    "  \n",
    "  completion = client.chat.completions.create(\n",
    "    model=MODEL,\n",
    "    messages=[\n",
    "      {\"role\": \"system\", \"content\": prompt}\n",
    "    ]\n",
    "  )\n",
    "\n",
    "  print(completion.choices[0].message.content, file=fout)\n",
    "  fout.close()\n",
    "\n",
    "  print(f'{file_name}_{MODEL}_{run_iterator}.lf')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.2 With manual context\n",
    "We start by prompting OpenAI LLMs: `gpt-3.5` and `gpt-4o` with handcrafted context.\n",
    "\n",
    "The generated code will be stored under `LFGPT/with_context`.\n",
    "\n",
    "We start by creating the target directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "# Creating the target directory\n",
    "!mkdir -p LFGPT/with_context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "# Defining the context\n",
    "context = r\"\"\"\n",
    "  A reactor is a software component that reacts to input events, timer events, and internal events. It has private state variables that are not visible to any other reactor. Its reactions can consist of altering its own state, sending messages to other reactors, or affecting the environment through some kind of actuation or side effect (e.g., printing a message, as in the above HelloWorld example).\n",
    "  The general structure of a reactor definition is as follows:\n",
    "  [main or federated] reactor <class-name> [(parameters)] {\n",
    "    input <name>: <type>\n",
    "    output <name>: <type>\n",
    "    state <name>: <type> [= <value>]\n",
    "    timer <name>([<offset>[, <period>]])\n",
    "    logical action <name>[: <type>]\n",
    "    physical action <name>[: <type>]\n",
    "    reaction [<name>] (triggers) [<uses>] [-> <effects>] [{= ... body ...=}]\n",
    "    <instance-name> = new <class-name>([<parameter-assignments>])\n",
    "    <port-name> [, ...] -> <port-name> [, ...] [after <delay>]\n",
    "}\n",
    "  \"\"\"\n",
    "\n",
    "\n",
    "# Defining run iterartor\n",
    "run_iterator = 0\n",
    "\n",
    "\n",
    "# File name suffix\n",
    "file_name = \"LFGPT/with_context/DropSensor\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initializes an OpenAI client for interacting with the OpenAI API.\n",
    "client = OpenAI()\n",
    "\n",
    "\n",
    "MODELS = [\"gpt-4o\", \"gpt-3.5-turbo\"] \n",
    "run_iterator += 1\n",
    "\n",
    "for MODEL in MODELS:\n",
    "  fout = open(f'{file_name}_{MODEL}_{run_iterator}.lf', 'w')\n",
    "  \n",
    "  completion = client.chat.completions.create(\n",
    "    model=MODEL,\n",
    "    # Adding context to the prompt\n",
    "    messages=[\n",
    "      {\"role\": \"system\", \"content\": context + prompt}\n",
    "    ]\n",
    "  )\n",
    "\n",
    "  print(completion.choices[0].message.content, file=fout)\n",
    "  fout.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. LF code generation with automatically Retrieved context\n",
    "Now we will use Retrieval-Augmented Generation (RAG). RAG combines retrieval-based methods with generative models to improve the quality and relevance of generated code.\n",
    "\n",
    "![Retrieval-Augmented Generation (RAG)](img/RAG.png)\n",
    "\n",
    "### 2.1 Retreival Model\n",
    "### Embedding\n",
    "A vector embedding, also called embedding, is a numerical representation of the semantics, or meaning of a text. \n",
    "\n",
    "\n",
    "\n",
    "<img src=\"img/embedding.png\" alt=\"Embedding sample\" style=\"width: 1000px;\"/>\n",
    "\n",
    "\n",
    "Two pieces of text with similar meanings will have mathematically similar embeddings, even if the actual text is quite different.\n",
    "\n",
    "We will use LlamaIndex [https://www.llamaindex.ai/] to generate the embedding of each file in our codebase [https://github.com/lf-lang/playground-lingua-franca]. \n",
    "\n",
    "#### How it works?\n",
    "- LlamaIndex splits files as `nodes`, \n",
    "- creates the embedding of each `nodes` using a LLM,\n",
    "- and stores it in a `VectorStoreIndex`.\n",
    "- We then request the `VectorStoreIndex` to retrieve the `nodes` that are most similar to the user prompt.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: llama-index in ./.venv/lib/python3.10/site-packages (0.10.11)\n",
      "Requirement already satisfied: llama-index-agent-openai<0.2.0,>=0.1.4 in ./.venv/lib/python3.10/site-packages (from llama-index) (0.1.5)\n",
      "Requirement already satisfied: llama-index-cli<0.2.0,>=0.1.2 in ./.venv/lib/python3.10/site-packages (from llama-index) (0.1.4)\n",
      "Requirement already satisfied: llama-index-core<0.11.0,>=0.10.11.post1 in ./.venv/lib/python3.10/site-packages (from llama-index) (0.10.11.post1)\n",
      "Requirement already satisfied: llama-index-embeddings-openai<0.2.0,>=0.1.5 in ./.venv/lib/python3.10/site-packages (from llama-index) (0.1.6)\n",
      "Requirement already satisfied: llama-index-indices-managed-llama-cloud<0.2.0,>=0.1.2 in ./.venv/lib/python3.10/site-packages (from llama-index) (0.1.3)\n",
      "Requirement already satisfied: llama-index-legacy<0.10.0,>=0.9.48 in ./.venv/lib/python3.10/site-packages (from llama-index) (0.9.48)\n",
      "Requirement already satisfied: llama-index-llms-openai<0.2.0,>=0.1.5 in ./.venv/lib/python3.10/site-packages (from llama-index) (0.1.6)\n",
      "Requirement already satisfied: llama-index-multi-modal-llms-openai<0.2.0,>=0.1.3 in ./.venv/lib/python3.10/site-packages (from llama-index) (0.1.4)\n",
      "Requirement already satisfied: llama-index-program-openai<0.2.0,>=0.1.3 in ./.venv/lib/python3.10/site-packages (from llama-index) (0.1.4)\n",
      "Requirement already satisfied: llama-index-question-gen-openai<0.2.0,>=0.1.2 in ./.venv/lib/python3.10/site-packages (from llama-index) (0.1.3)\n",
      "Requirement already satisfied: llama-index-readers-file<0.2.0,>=0.1.4 in ./.venv/lib/python3.10/site-packages (from llama-index) (0.1.5)\n",
      "Requirement already satisfied: llama-index-readers-llama-parse<0.2.0,>=0.1.2 in ./.venv/lib/python3.10/site-packages (from llama-index) (0.1.3)\n",
      "Requirement already satisfied: llama-index-vector-stores-chroma<0.2.0,>=0.1.1 in ./.venv/lib/python3.10/site-packages (from llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.1.3)\n",
      "Requirement already satisfied: PyYAML>=6.0.1 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (6.0.1)\n",
      "Requirement already satisfied: SQLAlchemy>=1.4.49 in ./.venv/lib/python3.10/site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (2.0.27)\n",
      "Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (3.9.3)\n",
      "Requirement already satisfied: dataclasses-json in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (0.6.4)\n",
      "Requirement already satisfied: deprecated>=1.2.9.3 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.2.14)\n",
      "Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.0.8)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (2024.2.0)\n",
      "Requirement already satisfied: httpx in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (0.27.0)\n",
      "Requirement already satisfied: llamaindex-py-client<0.2.0,>=0.1.13 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (0.1.13)\n",
      "Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.6.0)\n",
      "Requirement already satisfied: networkx>=3.0 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (3.2.1)\n",
      "Requirement already satisfied: nltk<4.0.0,>=3.8.1 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (3.8.1)\n",
      "Requirement already satisfied: numpy in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.26.4)\n",
      "Requirement already satisfied: openai>=1.1.0 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.12.0)\n",
      "Requirement already satisfied: pandas in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (2.2.0)\n",
      "Requirement already satisfied: pillow>=9.0.0 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (10.2.0)\n",
      "Requirement already satisfied: requests>=2.31.0 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (2.31.0)\n",
      "Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (8.2.3)\n",
      "Requirement already satisfied: tiktoken>=0.3.3 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (0.6.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.66.1 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (4.66.2)\n",
      "Requirement already satisfied: typing-extensions>=4.5.0 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (4.9.0)\n",
      "Requirement already satisfied: typing-inspect>=0.8.0 in ./.venv/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (0.9.0)\n",
      "Requirement already satisfied: beautifulsoup4<5.0.0,>=4.12.3 in ./.venv/lib/python3.10/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (4.12.3)\n",
      "Requirement already satisfied: bs4<0.0.3,>=0.0.2 in ./.venv/lib/python3.10/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (0.0.2)\n",
      "Requirement already satisfied: pymupdf<2.0.0,>=1.23.21 in ./.venv/lib/python3.10/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (1.23.25)\n",
      "Requirement already satisfied: pypdf<5.0.0,>=4.0.1 in ./.venv/lib/python3.10/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (4.0.2)\n",
      "Requirement already satisfied: llama-parse<0.4.0,>=0.3.3 in ./.venv/lib/python3.10/site-packages (from llama-index-readers-llama-parse<0.2.0,>=0.1.2->llama-index) (0.3.4)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in ./.venv/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.3.1)\n",
      "Requirement already satisfied: attrs>=17.3.0 in ./.venv/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (23.2.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in ./.venv/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.4.1)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in ./.venv/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (6.0.5)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in ./.venv/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.9.4)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0 in ./.venv/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (4.0.3)\n",
      "Requirement already satisfied: soupsieve>1.2 in ./.venv/lib/python3.10/site-packages (from beautifulsoup4<5.0.0,>=4.12.3->llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (2.5)\n",
      "Requirement already satisfied: wrapt<2,>=1.10 in ./.venv/lib/python3.10/site-packages (from deprecated>=1.2.9.3->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.16.0)\n",
      "Requirement already satisfied: chromadb<0.5.0,>=0.4.22 in ./.venv/lib/python3.10/site-packages (from llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.4.22)\n",
      "Requirement already satisfied: onnxruntime<2.0.0,>=1.17.0 in ./.venv/lib/python3.10/site-packages (from llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.17.0)\n",
      "Requirement already satisfied: tokenizers<0.16.0,>=0.15.1 in ./.venv/lib/python3.10/site-packages (from llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.15.2)\n",
      "Requirement already satisfied: pydantic>=1.10 in ./.venv/lib/python3.10/site-packages (from llamaindex-py-client<0.2.0,>=0.1.13->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (2.6.1)\n",
      "Requirement already satisfied: anyio in ./.venv/lib/python3.10/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (4.3.0)\n",
      "Requirement already satisfied: certifi in ./.venv/lib/python3.10/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (2024.2.2)\n",
      "Requirement already satisfied: httpcore==1.* in ./.venv/lib/python3.10/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.0.4)\n",
      "Requirement already satisfied: idna in ./.venv/lib/python3.10/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (3.6)\n",
      "Requirement already satisfied: sniffio in ./.venv/lib/python3.10/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.3.0)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in ./.venv/lib/python3.10/site-packages (from httpcore==1.*->httpx->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (0.14.0)\n",
      "Requirement already satisfied: click in ./.venv/lib/python3.10/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (8.1.7)\n",
      "Requirement already satisfied: joblib in ./.venv/lib/python3.10/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.3.2)\n",
      "Requirement already satisfied: regex>=2021.8.3 in ./.venv/lib/python3.10/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (2023.12.25)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in ./.venv/lib/python3.10/site-packages (from openai>=1.1.0->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.9.0)\n",
      "Requirement already satisfied: PyMuPDFb==1.23.22 in ./.venv/lib/python3.10/site-packages (from pymupdf<2.0.0,>=1.23.21->llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (1.23.22)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in ./.venv/lib/python3.10/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in ./.venv/lib/python3.10/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (2.2.1)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in ./.venv/lib/python3.10/site-packages (from SQLAlchemy>=1.4.49->SQLAlchemy[asyncio]>=1.4.49->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (3.0.3)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in ./.venv/lib/python3.10/site-packages (from typing-inspect>=0.8.0->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.0.0)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in ./.venv/lib/python3.10/site-packages (from dataclasses-json->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (3.20.2)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in ./.venv/lib/python3.10/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in ./.venv/lib/python3.10/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (2024.1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in ./.venv/lib/python3.10/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (2024.1)\n",
      "Requirement already satisfied: exceptiongroup>=1.0.2 in ./.venv/lib/python3.10/site-packages (from anyio->httpx->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.2.0)\n",
      "Requirement already satisfied: build>=1.0.3 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.0.3)\n",
      "Requirement already satisfied: chroma-hnswlib==0.7.3 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.7.3)\n",
      "Requirement already satisfied: fastapi>=0.95.2 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.109.2)\n",
      "Requirement already satisfied: uvicorn>=0.18.3 in ./.venv/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.27.1)\n",
      "Requirement already satisfied: posthog>=2.4.0 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (3.4.2)\n",
      "Requirement already satisfied: pulsar-client>=3.1.0 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (3.4.0)\n",
      "Requirement already satisfied: opentelemetry-api>=1.2.0 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.22.0)\n",
      "Requirement already satisfied: opentelemetry-exporter-otlp-proto-grpc>=1.2.0 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.22.0)\n",
      "Requirement already satisfied: opentelemetry-instrumentation-fastapi>=0.41b0 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.43b0)\n",
      "Requirement already satisfied: opentelemetry-sdk>=1.2.0 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.22.0)\n",
      "Requirement already satisfied: pypika>=0.48.9 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.48.9)\n",
      "Requirement already satisfied: overrides>=7.3.1 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (7.7.0)\n",
      "Requirement already satisfied: importlib-resources in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (6.1.1)\n",
      "Requirement already satisfied: grpcio>=1.58.0 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.62.0)\n",
      "Requirement already satisfied: bcrypt>=4.0.1 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (4.1.2)\n",
      "Requirement already satisfied: typer>=0.9.0 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.9.0)\n",
      "Requirement already satisfied: kubernetes>=28.1.0 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (29.0.0)\n",
      "Requirement already satisfied: mmh3>=4.0.1 in ./.venv/lib/python3.10/site-packages (from chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (4.1.0)\n",
      "Requirement already satisfied: packaging>=17.0 in ./.venv/lib/python3.10/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (23.2)\n",
      "Requirement already satisfied: coloredlogs in ./.venv/lib/python3.10/site-packages (from onnxruntime<2.0.0,>=1.17.0->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (15.0.1)\n",
      "Requirement already satisfied: flatbuffers in ./.venv/lib/python3.10/site-packages (from onnxruntime<2.0.0,>=1.17.0->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (23.5.26)\n",
      "Requirement already satisfied: protobuf in ./.venv/lib/python3.10/site-packages (from onnxruntime<2.0.0,>=1.17.0->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (4.25.3)\n",
      "Requirement already satisfied: sympy in ./.venv/lib/python3.10/site-packages (from onnxruntime<2.0.0,>=1.17.0->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.12)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in ./.venv/lib/python3.10/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.13->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.16.2 in ./.venv/lib/python3.10/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.13->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (2.16.2)\n",
      "Requirement already satisfied: six>=1.5 in ./.venv/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->llama-index-core<0.11.0,>=0.10.11.post1->llama-index) (1.16.0)\n",
      "Requirement already satisfied: huggingface_hub<1.0,>=0.16.4 in ./.venv/lib/python3.10/site-packages (from tokenizers<0.16.0,>=0.15.1->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.20.3)\n",
      "Requirement already satisfied: pyproject_hooks in ./.venv/lib/python3.10/site-packages (from build>=1.0.3->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.0.0)\n",
      "Requirement already satisfied: tomli>=1.1.0 in ./.venv/lib/python3.10/site-packages (from build>=1.0.3->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (2.0.1)\n",
      "Requirement already satisfied: starlette<0.37.0,>=0.36.3 in ./.venv/lib/python3.10/site-packages (from fastapi>=0.95.2->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.36.3)\n",
      "Requirement already satisfied: filelock in ./.venv/lib/python3.10/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers<0.16.0,>=0.15.1->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (3.13.1)\n",
      "Requirement already satisfied: google-auth>=1.0.1 in ./.venv/lib/python3.10/site-packages (from kubernetes>=28.1.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (2.28.1)\n",
      "Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in ./.venv/lib/python3.10/site-packages (from kubernetes>=28.1.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.7.0)\n",
      "Requirement already satisfied: requests-oauthlib in ./.venv/lib/python3.10/site-packages (from kubernetes>=28.1.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.3.1)\n",
      "Requirement already satisfied: oauthlib>=3.2.2 in ./.venv/lib/python3.10/site-packages (from kubernetes>=28.1.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (3.2.2)\n",
      "Requirement already satisfied: importlib-metadata<7.0,>=6.0 in ./.venv/lib/python3.10/site-packages (from opentelemetry-api>=1.2.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (6.11.0)\n",
      "Requirement already satisfied: backoff<3.0.0,>=1.10.0 in ./.venv/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (2.2.1)\n",
      "Requirement already satisfied: googleapis-common-protos~=1.52 in ./.venv/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.62.0)\n",
      "Requirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.22.0 in ./.venv/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.22.0)\n",
      "Requirement already satisfied: opentelemetry-proto==1.22.0 in ./.venv/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.22.0)\n",
      "Requirement already satisfied: opentelemetry-instrumentation-asgi==0.43b0 in ./.venv/lib/python3.10/site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.43b0)\n",
      "Requirement already satisfied: opentelemetry-instrumentation==0.43b0 in ./.venv/lib/python3.10/site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.43b0)\n",
      "Requirement already satisfied: opentelemetry-semantic-conventions==0.43b0 in ./.venv/lib/python3.10/site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.43b0)\n",
      "Requirement already satisfied: opentelemetry-util-http==0.43b0 in ./.venv/lib/python3.10/site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.43b0)\n",
      "Requirement already satisfied: setuptools>=16.0 in ./.venv/lib/python3.10/site-packages (from opentelemetry-instrumentation==0.43b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (69.1.0)\n",
      "Requirement already satisfied: asgiref~=3.0 in ./.venv/lib/python3.10/site-packages (from opentelemetry-instrumentation-asgi==0.43b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (3.7.2)\n",
      "Requirement already satisfied: monotonic>=1.5 in ./.venv/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.6)\n",
      "Requirement already satisfied: httptools>=0.5.0 in ./.venv/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.6.1)\n",
      "Requirement already satisfied: python-dotenv>=0.13 in ./.venv/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.0.1)\n",
      "Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in ./.venv/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.19.0)\n",
      "Requirement already satisfied: watchfiles>=0.13 in ./.venv/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.21.0)\n",
      "Requirement already satisfied: websockets>=10.4 in ./.venv/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (12.0)\n",
      "Requirement already satisfied: humanfriendly>=9.1 in ./.venv/lib/python3.10/site-packages (from coloredlogs->onnxruntime<2.0.0,>=1.17.0->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (10.0)\n",
      "Requirement already satisfied: mpmath>=0.19 in ./.venv/lib/python3.10/site-packages (from sympy->onnxruntime<2.0.0,>=1.17.0->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (1.3.0)\n",
      "Requirement already satisfied: cachetools<6.0,>=2.0.0 in ./.venv/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (5.3.2)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in ./.venv/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.3.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in ./.venv/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (4.9)\n",
      "Requirement already satisfied: zipp>=0.5 in ./.venv/lib/python3.10/site-packages (from importlib-metadata<7.0,>=6.0->opentelemetry-api>=1.2.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (3.17.0)\n",
      "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in ./.venv/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes>=28.1.0->chromadb<0.5.0,>=0.4.22->llama-index-vector-stores-chroma<0.2.0,>=0.1.1->llama-index-cli<0.2.0,>=0.1.2->llama-index) (0.5.1)\n"
     ]
    }
   ],
   "source": [
    "# Install llama-index\n",
    "!python -m pip install llama-index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing packages\n",
    "import os\n",
    "from llama_index.core import (\n",
    "    load_index_from_storage,\n",
    "    SimpleDirectoryReader,\n",
    "    StorageContext,\n",
    "    VectorStoreIndex\n",
    "    \n",
    ")\n",
    "\n",
    "# Setting codebase directory\n",
    "# contains examples directory from https://github.com/lf-lang/playground-lingua-franca\n",
    "DIR = \"./python_codebase\"\n",
    "\n",
    "# Setting storage directory\n",
    "PERSIST_DIR = \"./py_codebase\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating indexes from scratch ...\n"
     ]
    }
   ],
   "source": [
    "# If `VectorStoreIndex` exists load it from disk.\n",
    "# Else, Build it from code_base and save it to disk, for future use.\n",
    "\n",
    "def load_storage_context():\n",
    "    # if 'PERSIST_DIR' exists, load indexes from persisted data\n",
    "    # else load create indexes from code base dierctory, 'data'.\n",
    "    if (os.path.exists(PERSIST_DIR)):\n",
    "        # load the existing index\n",
    "        print(\"Loading persisted indexes ...\")\n",
    "        # rebuild storage context\n",
    "        storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)\n",
    "        return storage_context      \n",
    "    else:\n",
    "        print(\"Creating indexes from scratch ...\")\n",
    "        # load documents from directory\n",
    "        documents = SimpleDirectoryReader(DIR, recursive=True).load_data()\n",
    "\n",
    "        # build index with embedding model \"gpt-4o\"\n",
    "        index = VectorStoreIndex.from_documents(documents, model=\"gpt-4o\")\n",
    "\n",
    "        # save indexes on disk\n",
    "        index.storage_context.persist(persist_dir=PERSIST_DIR) \n",
    "        return index.storage_context\n",
    "\n",
    "# Loading `VectorStoreIndex` (storage context) from disk\n",
    "storage_context = load_storage_context()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Retrieval-Augmented Generation\n",
    "\n",
    "From here we only focus on `gpt-4o` model.\n",
    "\n",
    "The generated code will be stored under `LFGPT/rag`.\n",
    "\n",
    "We start by creating the target directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "# Creating the target directory\n",
    "!mkdir -p LFGPT/rag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAI\n",
    "\n",
    "# Define LLM to use for Code geneartion\n",
    "MODEL = \"gpt-4o\"\n",
    "\n",
    "# Define the Top K retrived documents (LF files)\n",
    "TOP_K = 10\n",
    "\n",
    "# Instatntiate the LLM\n",
    "llm = OpenAI(temperature=1, model=MODEL)\n",
    "\n",
    "# Instantiate a ServiceContext using the OpenAI_API_key\n",
    "index = load_index_from_storage(storage_context, llm=llm)        \n",
    " \n",
    "# Configure retriever\n",
    "query_engine = index.as_query_engine(similarity_top_k=TOP_K)\n",
    "\n",
    "\n",
    "\n",
    "# Defining run iterartor\n",
    "run_iterator = 0\n",
    "\n",
    "# File name suffix\n",
    "file_name = \"LFGPT/rag/DropSensor\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the prompt\n",
    "prompt = f\"\"\" \n",
    "  Provide a Lingua Franca code for 'target C'. \n",
    "  Import the LF 'Accelerometer' reactor from library.\n",
    "  Create a main reactor named `DropSensor` that:\n",
    "  - has a parameter named `threshold`, with a default value of '0.3'.\n",
    "  - instantiates an accelerometer.\n",
    "  - declares a timer with a period of 250 ms.\n",
    "  - Has a reaction that triggers the accelerometer with period of 250 ms by setting the accelerometer trigger to true.\n",
    "  - has another reaction which fires whenever any of the accelerometer readings (`x`, `y`, or `z`) change. Inside this reaction: It checks whether the absolute values of the accelerometer readings (`x`, `y`, `z`) are all less than the defined `threshold`. If the condition is true, it prints the values of the accelerometer outputs to the console.  \n",
    "  \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def rag():    \n",
    "    response = query_engine.query(prompt)\n",
    "    files = '/**\\n'\n",
    "    for node in response.source_nodes:\n",
    "        files += f'* {node.metadata[\"file_path\"]}\\n'\n",
    "    files += '*/\\n'\n",
    "\n",
    "    fout = open(f'{file_name}_{run_iterator}_K-{TOP_K}.lf', 'w')\n",
    "\n",
    "    print(files+response.response, file=fout)\n",
    "    fout.close()\n",
    "\n",
    "    print(f'{file_name}_{run_iterator}_K-{TOP_K}.lf')\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LFGPT/rag/DropSensor_1_K-10.lf\n"
     ]
    }
   ],
   "source": [
    "run_iterator += 1\n",
    "rag()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python target with index composed of Python only files\n",
    "\n",
    "Redefine the prompt with Python as target language `target Python`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the prompt\n",
    "prompt = f\"\"\" \n",
    "  Provide a Lingua Franca code for 'target Python'. \n",
    "  Import the LF 'Accelerometer' reactor from library.\n",
    "  Create a main reactor named `DropSensor` that:\n",
    "  - has a parameter named `threshold`, with a default value of '0.3'.\n",
    "  - instantiates an accelerometer.\n",
    "  - declares a timer with a period of 250 ms.\n",
    "  - Has a reaction that triggers the accelerometer with period of 250 ms by setting the accelerometer trigger to true.\n",
    "  - has another reaction which fires whenever any of the accelerometer readings (`x`, `y`, or `z`) change. Inside this reaction: It checks whether the absolute values of the accelerometer readings (`x`, `y`, `z`) are all less than the defined `threshold`. If the condition is true, it prints the values of the accelerometer outputs to the console.  \n",
    "  \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading persisted indexes ...\n"
     ]
    }
   ],
   "source": [
    "# Setting codebase directory\n",
    "# contains Python examples from :\n",
    "# 1. https://github.com/lf-lang/playground-lingua-franca\n",
    "# 2. https://github.com/lf-lang/lf-lang.github.io\n",
    "DIR = \"./python_codebase\"\n",
    "\n",
    "# Setting storage directory\n",
    "PERSIST_DIR = \"./index_python_codebase\"\n",
    "\n",
    "# Loading `VectorStoreIndex` (storage context) from disk\n",
    "storage_context = load_storage_context()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate a ServiceContext using the OpenAI_API_key\n",
    "index = load_index_from_storage(storage_context, llm=llm)        \n",
    " \n",
    "# Configure retriever\n",
    "query_engine = index.as_query_engine(similarity_top_k=TOP_K)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LFGPT/rag/DropSensor_3_K-10.lf\n"
     ]
    }
   ],
   "source": [
    "run_iterator += 1\n",
    "rag()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}