Initial Commit

.gitignore

@@ -0,0 +1,166 @@
+.DS_Store
+*.txt
+/experiments
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
+.pdm.toml
+.pdm-python
+.pdm-build/
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/

README.md

@@ -0,0 +1,37 @@
+# Atari-GPT
+
+This is the official codebase for [Atari-GPT](https://arxiv.org/pdf/2408.15950), a new benchmark for Large Language Models (LLMs) on Atari games. To see more results see our project [webpage](https://sites.google.com/view/atari-gpt/).
+
+## Set up
+
+In order to run the code you need to have an API key for the respective model. To reproduce results from the paper you will need all 3 API keys.
+
+    For Google you can get an API key [here](https://ai.google.dev). Once you have this API key put it in a file called GOOGLE_API_KEY.txt.
+
+
+    For Anthropic you can get an API key [here](https://www.anthropic.com/api). Once you have this API key put it in a file called ANTHROPIC_API_KEY.txt.
+
+    For OpenAI you can get an API key [here](https://openai.com/api/). Once you have this API key put it in a file called OPENAI_API_KEY.txt.
+
+
+## Installation
+
+To run the code you will need to have Anaconda and run the following commands:
+
+<br>&ensp;&ensp;&ensp;&ensp;`conda env create --file=environment.yaml`
+<br>&ensp;&ensp;&ensp;&ensp;`conda activate atari_gpt`
+<br>&ensp;&ensp;&ensp;&ensp;`python full_evaluation.py`
+
+## Citing Atari-GPT 
+
+```
+@misc{waytowich2024atarigptinvestigatingcapabilitiesmultimodal,
+      title={Atari-GPT: Investigating the Capabilities of Multimodal Large Language Models as Low-Level Policies for Atari Games}, 
+      author={Nicholas R. Waytowich and Devin White and MD Sunbeam and Vinicius G. Goecks},
+      year={2024},
+      eprint={2408.15950},
+      archivePrefix={arXiv},
+      primaryClass={cs.AI},
+      url={https://arxiv.org/abs/2408.15950}, 
+}
+```

environment.yaml

@@ -0,0 +1,92 @@
+name: atari_gpt
+channels:
+  - defaults
+  - conda-forge
+dependencies:
+  - ca-certificates=2024.7.2=hca03da5_0
+  - libcxx=14.0.6=h848a8c0_0
+  - libffi=3.4.4=hca03da5_1
+  - ncurses=6.4=h313beb8_0
+  - openssl=3.0.14=h80987f9_0
+  - pip=24.0=py39hca03da5_0
+  - python=3.9.19=hb885b13_1
+  - readline=8.2=h1a28f6b_0
+  - setuptools=72.1.0=py39hca03da5_0
+  - sqlite=3.45.3=h80987f9_0
+  - tk=8.6.14=h6ba3021_0
+  - tzdata=2024a=h04d1e81_0
+  - wheel=0.43.0=py39hca03da5_0
+  - xz=5.4.6=h80987f9_1
+  - zlib=1.2.13=h18a0788_1
+  - pip:
+      - ale-py==0.8.1
+      - annotated-types==0.6.0
+      - anthropic==0.25.7
+      - anyio==4.3.0
+      - autorom==0.4.2
+      - autorom-accept-rom-license==0.6.1
+      - cachetools==5.3.3
+      - certifi==2024.2.2
+      - charset-normalizer==3.3.2
+      - click==8.1.7
+      - cloudpickle==3.0.0
+      - contourpy==1.2.1
+      - cycler==0.12.1
+      - decorator==4.4.2
+      - distro==1.9.0
+      - exceptiongroup==1.2.1
+      - farama-notifications==0.0.4
+      - filelock==3.14.0
+      - fonttools==4.53.1
+      - fsspec==2024.3.1
+      - google-ai-generativelanguage==0.6.2
+      - google-api-core==2.19.0
+      - google-api-python-client==2.127.0
+      - google-auth==2.29.0
+      - google-auth-httplib2==0.2.0
+      - google-generativeai==0.5.2
+      - googleapis-common-protos==1.63.0
+      - groq==0.5.0
+      - grpcio==1.63.0
+      - grpcio-status==1.62.2
+      - gymnasium==0.29.1
+      - h11==0.14.0
+      - httpcore==1.0.5
+      - httplib2==0.22.0
+      - httpx==0.27.0
+      - huggingface-hub==0.22.2
+      - idna==3.7
+      - imageio==2.34.2
+      - imageio-ffmpeg==0.5.1
+      - importlib-metadata==8.2.0
+      - importlib-resources==6.4.0
+      - kiwisolver==1.4.5
+      - matplotlib==3.9.1.post1
+      - moviepy==1.0.3
+      - numpy==1.26.4
+      - openai==1.25.0
+      - opencv-python==4.9.0.80
+      - packaging==24.0
+      - pillow==10.4.0
+      - proglog==0.1.10
+      - proto-plus==1.23.0
+      - protobuf==4.25.3
+      - pyasn1==0.6.0
+      - pyasn1-modules==0.4.0
+      - pydantic==2.7.1
+      - pydantic-core==2.18.2
+      - pygame==2.6.0
+      - pyparsing==3.1.2
+      - python-dateutil==2.9.0.post0
+      - pyyaml==6.0.1
+      - requests==2.31.0
+      - rsa==4.9
+      - shimmy==0.2.1
+      - six==1.16.0
+      - sniffio==1.3.1
+      - tokenizers==0.19.1
+      - tqdm==4.66.2
+      - typing-extensions==4.11.0
+      - uritemplate==4.1.1
+      - urllib3==2.2.1
+      - zipp==3.19.2

envs.json

@@ -0,0 +1,12 @@
+{
+    "ALE/Alien-v5": "You are a game playing assistant and will be provided an image. This will be of the game Alien, your goal is to provide me with what you believe to be the best action I could take to beat the game. Think about all possible actions and why each action is or is not the best action to take. The potential actions I can take are '0' NOOP, '1' FIRE, '2' UP, '3' RIGHT, '4' LEFT, and '5' DOWN, '6' UPRIGHT, '7' UPLEFT, '8' DOWNRIGHT, and '9' DOWNLEFT,’10’: UPFIRE, ’11’: RIGHTFIRE, ’12’: LEFTFIRE, ’13’: DOWNFIRE, ’14’: UPRIGHTFIRE, ’15’: UPLEFTFIRE, ’16’: DOWNRIGHTFIRE, ’17’: DOWNLEFTFIRE. Provide output as a json structured as {reasoning: reasoning for actions and why to choose an action, action: The environment action which would provide the best next state}. The action key should only have the action I should take for the current frame as a number.",
+    "BreakoutDeterministic-v4": "You are a game playing assistant and will be provided an image. This will be of the game Breakout, your goal is to provide me with what you believe to be the best action I could take to beat the game. Think about all possible actions and why each action is or is not the best action to take. The potential actions I can take are ‘0’: NOOP ‘1’: FIRE ‘2’: RIGHT ‘3’: LEFT. Provide output as a json structured as {reasoning: reasoning for actions and why to choose an action, action: The environment action which would provide the best next state}. The action key should only have the action I should take for the current frame as a number.",
+    "ALE/Frogger-v5": "You are a game playing assistant and will be provided an image. This will be of the game Frogger, your goal is to provide me with what you believe to be the best action I could take to beat the game. Think about all possible actions and why each action is or is not the best action to take. The potential actions I can take are ‘0’: NOOP ‘1’: UP ‘2’: RIGHT ‘3’: LEFT ‘4’: DOWN. Provide output as a json structured as {reasoning: reasoning for actions and why to choose an action, action: The environment action which would provide the best next state}. The action key should only have the action I should take for the current frame as a number.", 
+    "MsPacmanDeterministic-v4": "You are a game playing assistant and will be provided an image. This will be of the game Ms. Pacman, your goal is to provide me with what you believe to be the best action I could take to beat the game. Think about all possible actions and why each action is or is not the best action to take. The potential actions I can take are ‘0’: NOOP ‘1’: UP ‘2’: RIGHT ‘3’: LEFT ‘4’: DOWN ‘5’: UPRIGHT ‘6’: UPLEFT ‘7’: DOWNRIGHT ‘8’: DOWNLEFT. Provide output as a json structured as {reasoning: reasoning for actions and why to choose an action, action: The environment action which would provide the best next state}. The action key should only have the action I should take for the current frame as a number.", 
+    "PongDeterministic-v4": "You are a game playing assistant and will be provided an image. This will be of the game Pong, your goal is to provide me with what you believe to be the best action I could take to beat the game. Think about all possible actions and why each action is or is not the best action to take. The potential actions I can take are ‘0’: NOOP ‘1’: FIRE ‘2’: RIGHT ‘3’: LEFT ‘4’: RIGHTFIRE ‘5’: LEFTFIRE. Provide output as a json structured as {reasoning: reasoning for actions and why to choose an action, action: The environment action which would provide the best next state}. The action key should only have the action I should take for the current frame as a number.",
+    "SeaquestDeterministic-v4": "You are a game playing assistant and will be provided an image. This will be of the game Seaquest, your goal is to provide me with what you believe to be the best action I could take to beat the game. Think about all possible actions and why each action is or is not the best action to take. The potential actions I can take are '0' NOOP, '1' FIRE, '2' UP, '3' RIGHT, '4' LEFT, and '5' DOWN, '6' UPRIGHT, '7' UPLEFT, '8' DOWNRIGHT, and '9' DOWNLEFT,’10’: UPFIRE, ’11’: RIGHTFIRE, ’12’: LEFTFIRE, ’13’: DOWNFIRE, ’14’: UPRIGHTFIRE, ’15’: UPLEFTFIRE, ’16’: DOWNRIGHTFIRE, ’17’: DOWNLEFTFIRE DOWN. Provide output as a json structured as {reasoning: reasoning for actions and why to choose an action, action: The environment action which would provide the best next state}. The action key should only have the action I should take for the current frame as a number.", 
+    "SpaceInvadersDeterministic-v4": "You are a game playing assistant and will be provided an image. This will be of the game Space Invaders, your goal is to provide me with what you believe to be the best action I could take to beat the game. Think about all possible actions and why each action is or is not the best action to take. The potential actions I can take are ‘0’ NOOP ‘1’ FIRE ‘2’ RIGHT ‘3’ LEFT ‘4’ RIGHTFIRE ‘5’ LEFTFIRE. Provide output as a json structured as {reasoning: reasoning for actions and why to choose an action, action: The environment action which would provide the best next state}. The action key should only have the action I should take for the current frame as a number."
+}
+
+
+

full_evaluation.py

@@ -0,0 +1,14 @@
+import json
+from run_experiments import run
+with open('envs.json') as file:
+    environment_list = json.load(file)
+
+models = ['rand', 'gpt4', 'gpt4o', 'gemini', 'claude']
+
+for model in models:
+    environments = list(environment_list.keys())
+    for game in environments:
+        print('Running test for: ', game)
+        print('\n\n')
+        results = run(game, environment_list[game], model)
+