[RFC] Kepler.gl AI Assistant Module

[lixun910]

[RFC] Kepler.gl AI Assistant Module

Author: Xun Li Dec 17 2024

Google Doc: https://docs.google.com/document/d/1SK16P_rQRvuWMjMdesEICHQ2RHwvF-7vd9kzNEGP17Y/edit?usp=sharing

The AI Assistant is a module that adds an AI chatbot to Kepler.gl. This module aims to integrates Kepler.gl with AI-powered capabilities, enabling it to interact with multiple AI models seamlessly.

Overview

The AI Assistant Module facilitates communication, processes prompts, and optionally calls back into the main application to deliver results. This setup supports advanced functionalities like tool-based function calling for enhanced interactivity.

The system is designed to enable Kepler.gl, a React-based single-page application, to integrate an AI Assistant Module for performing tasks with large language models (LLMs) like OpenAI GPT models, Google Gemini models, and Ollama models.

An example of how the AI Assistant Module can be used to update a basemap in Kepler.gl:

This flow enables a seamless interaction where a user can update a basemap in Kepler.gl through a simple AI-driven prompt, showcasing the integration of LLMs with application actions and rendering.

The AI Assistant Module (Github: https://github.com/lixun910/ai-assistant) is a open-source project that helps adding AI capabilities to your React SPA applications while keeping your data secure and private.

Key features:

• Built-in chat UI.
• Built-in "Screenshot to Ask" feature.
• Built-in "Talk to Ask" feature.
• Support OpenAI, Google Gemini, Ollama models.
• Support function calling with type hint.
• Support custom message UI for function calling.
• Addons:
  • query your dataset using duckdb
  • map your data
  • analyze and visualize your data
    ...

Proposal

From the previous PRs, we have a working prototype of the AI Assistant Module with the following features:

#2735
#2777
#2819
#2828

1. Show dataset/layer/variable info.
2. Change the basemap style.
3. Load data from url.
4. Create a map layer using variable.
5. Filter the data of a variable.
6. Create a histogram.

The next step is to integrate more kepler.gl features into the AI Assistant Module.

P0

Make the LLM guiding the user through multiple steps to:

• create map layer based on the dataset metadata (variable names and types)
• edit layer properties based on the layer attributes
• blend and rearrange layers

Support color palette

• choose color palette for the layer
• create custom color palette

Support custom map styles

• choose map layers (water, buildings, roads, etc.)
• create custom map style from url

Support map settings

• update map legend for the layer
• set split map (guide user to choose maps to split on left and right)

Support time playback

• identify the time variable from the dataset metadata
• set time playback for the layer