initial commit

Author: antonis19

.gitignore

@@ -0,0 +1,15 @@
+node_modules
+.DS_Store
+.env
+.cache/*
+.vscode/*
+.ipynb_checkpoints/*
+package-lock.json
+node_modules/*
+py39/*
+venv/*
+vectorstore/*
+__pycache__/*
+html/*
+dom/*
+OAI_CONFIG_LIST

README.md

@@ -0,0 +1,109 @@
+# AutoBrowse
+AutoBrowse is an autonomous AI agent that can browse the web.
+You simply give AutoBrowse a task and it will complete it by interacting with the web browser as if it were a human.
+
+Some examples of tasks you can give it:
+
+- Go to booking.com and find a hotel for 2 people in Madrid for 2 nights starting 2 November for under 200 EUR per night.
+
+- Sign up to ryanair.com with email: <[email protected]> and password C0mplexPassword!.
+
+- Go to Craigslist and search for Nintendo DS. Click on the first result.
+
+## How to run
+
+1. Create a Python 3.9 environment
+
+    ```bash
+    conda create --name py39 python=3.9
+    ```
+
+2. Activate the environment
+
+    ```bash
+    conda activate py39
+    ```
+
+3. Install dependencies
+
+    ```bash
+    pip install -r requirements.txt
+    ```
+
+4. Start the browser environment
+
+    Check the README under `browser-console/` for instructions on how to run it.
+
+
+5. Create a file `OAI_CONFIG_LIST` with the following content and put in your OpenAI API key:
+
+    ```json
+    [
+        {
+            "model": "gpt-4",
+            "api_key": "<your-api-key>"
+        },
+        {
+            "model": "gpt-3.5-turbo",
+            "api_key": "<your-api-key>"
+        },
+        {
+            "model": "gpt-3.5-turbo-16k",
+            "api_key": "<your-api-key>"
+        }
+    ]
+    ```
+
+
+
+
+5. Run AutoBrowse 
+
+    ```bash
+    python autobrowse.py
+    ```
+
+    You will then be prompted give a task to AutoBrowse.
+
+
+You can make modifications agent configurations by modifying the `agent_config.py` file. You can edit the system prompts, change the OpenAI models used etc.
+
+
+## How it works
+AutoBrowse uses [autogen](https://github.com/microsoft/autogen) agents and a browser console to plan and execute the task.
+
+The design consists of 3 agents:
+
+1. An HTML assistant that answers questions about the HTML of the current page open in the browser.
+
+2. A code generator agent that generates puppeteer.js code to interact with the browser (i.e. navigate to a new page, click on a button, fill in form elements)
+
+3. A planner agent that coordinates the use of the two agents above to fulfill the high-level task description provided by the user.
+
+The agents interact with the browser through a websocket connection to a sandboxed browser environment that has an endpoint to accept puppeteer.js code to execute, as well as and endpoint to return the rendered HTML of the current open page.
+
+### HTML Assistant
+Since HTML documents can be quite long and can exceed the token limit of OpenAI the following approach is taken to answer queries about the HTML:
+
+- The HTML returned from the browser environment is stripped down and simplified to reduce its size. This done by keeping only the most important attributes like id, name, type, and class . Moreover, `script`, `style`, `noscript`, `img`, `svg`, `link`, `meta`  tags are removed altogether.
+
+- The processed HTML is chunked into 15,000 token (as counted by OpenAI) so that they can easily fit in the 16K context window of `gpt-3.5-turbo-16k`. 
+
+- Using RAG with OpenAI embeddings, the most relevant chunk is provided as context to the question, and `gpt-3.5-turbo` can then answer the question about the HTML.
+
+### Code Generator
+The code generator uses `gpt-4` to generate puppeteer.js code to interact with the browser. A user proxy agent attached to the code generator sends this code to the browser environment to be executed and reports back the result, so that the code generator can amend the code if there are any errors. Because the code generation needs to be as accurate as possible the more expensive `gpt-4` model is used in favor of the cheaper `gpt-3.5-turbo`.
+
+### Planner
+
+The planner receives the task description from the user and tries to complete it by invoking the HTML Assistant and Code Generator as necessary. The planner, in addition to its own thinking, has the ability to invoke two functions:
+
+
+1. `ask_html_assistant()` to ask the HTML assistant a question about the current HTML (e.g. extract the HTML for the sign-up form), and 
+
+2. `ask_code_generator()` to ask the code generator to produce puppeteer.js code to send to the browser. The planner may also add HTML retrieved from the HTML assistant to provide more context to the code generator.
+
+
+
+
+

agent_config.py

@@ -0,0 +1,49 @@
+
+config = {
+    "html_assistant": {
+        "model": "gpt-3.5-turbo-16k",
+        "system_message": """You are a helpful AI Assistant. You will answer questions about HTML code. Respond only with HTML code from the HTML that is provided to you.
+            (i.e. find the answer only in the HTML that you are given, don't make up imaginary HTML) """,
+    },
+    "code_generator": {
+        "model": "gpt-4",
+        "system_message": """You are a Javascript engineer. You generate puppeteer.js javascript code to fulfill
+            a given task that has to do with web browsing. The output of this agent should only be code (inside codeblocks). You may also be
+            asked to correct code.  You should assume that the puppeteer environment has already been initialized with the following code:
+            const browser = await puppeteer.launch({ headless: false });
+            const page = await browser.newPage();
+            Whenever you go to a new website you should use the  { waitUntil: 'networkidle0' } option to make sure the page is fully loaded.
+            Before typing text into an input field you should first click on it to make sure it is focused.
+            If you get a 'Node is either not clickable or not an Element' error, you are probably trying to click on the wrong element, so if there is no other element
+            that you can click, you should reply with NOT_CLICKABLE <element_name> .
+            You may also be provided the execution result of the code. If you see success:true in the execution result, you should reply with TERMINATE .
+            """
+    },
+    "code_generator_user_proxy": {
+        "max_consecutive_auto_reply" : 1,
+    },
+
+    "planner": {
+        "model": "gpt-4",
+        "system_message": """You are a planner. You generate a plan to fulfill a web browsing task. This is done through the use of 2 other AI assistant agents. You can propose the usage of two functions : 1. ask_html_assistant (to ask questions about the current
+            page in the browser - the result will be HTML code) Keep in mind that this agent is not able to make any modifications to the page, only respond to questions about it. 2. ask_code_generator (to generate and execute puppeteer.js code in the browser) .
+            The code_generator does not have the HTML context, so you may need to provide it with the HTML from the html_assistant.
+            PLEASE MAKE SURE TO ASK THE HTML ASSISTANT FOR RELEVANT CONTEXT AND PROVIDE THE RETRIEVED HTML AS CONTEXT TO THE CODE GENERATOR!!!! If the context is not needed then pass an empty string as context_html.
+            So you might want to first ask the html_assistant a question about the HTML content, and then use the result of that question as context input to the code_generator. 
+            When the function call to ask_code_generator comes back with TERMINATE, that means the code has been generated and executed successfully.
+            IMPORTANT:  You should not write out the entire plan right away and instead focus on the next step of the plan.
+            If any step of the plan does not work out (e.g. code execution fails), DO NOT proceed to the next step without first trying to fix the current step.
+            Also try to prompt the other agents to do things as granularly as possible (i.e. don't try to do too many things in one function call, like trying to extracting 2 html selectors at the same time from the html, or trying to ask the code generator to do many actions in one function call).
+            Please ensure to remove any cookie notices and other popups.
+            When you suggest a function call do not produce any ```json code blocks.     Make sure you suggest function calls correctly so that autogen can parse them. Explicitly say it when you are suggesting a function call.
+            Don't provide HTML selectors to the code generator but rather actual HTML fragments.
+            Please ensure to accept any cookie notices and remove any other popups.
+            When the plan has been successfully completed reply with FINISHED.
+            Take a deep breath and work on this problem step-by-step.
+            """,
+    },
+
+    "planner_user_proxy": {
+        "max_consecutive_auto_reply": 35,
+    },
+}