GLASS

This repo implements software that allows the user to connect to added devices by pointing a camera towards them and then to control connected devices with hand gestures. We have optimized this software to run on three PIs two of which are on a pair of glasses that have a camera and a display.

Enviroment Setup

Note: if python does not default to python3 on your system, then substitute python3 for python in the following commands.

Navigate to the root directory:

cd path/to/image_vec_db

Use the venv module to create a new virtual environment:

python -m venv .venv

Activate the virtual environment:

# On macOS/Linux.
source .venv/bin/activate

# On Windows.
.venv\Scripts\activate

Upgrade pip:

python -m pip install --upgrade pip

# Also upgrade these for TensorFlow or PyTorch
python -m pip install --upgrade pip wheel setuptools

Use pip to install all the packages listed in your requirements.txt:

pip install -r requirements.txt

When you are done working in the virtual environment, you can deactivate it.

deactivate

Notes:

• For specific PyTorch installs see get-started.
• To use the vector database device detection feature, the sqlite-vec package is required. This may require compiling the package on some devices. See installing sqlite-vec and updated sqlite. pysqlite3 will be used if availble but will default to sqlite3. If compiling sqlite-vec from source, set an enviroment variable named SQLITE_VEC_PATH to the full path of vec0.so.
• Pi 5 additional setup is explained in the Pi 5 setup docs.

Model Setup

To download the required image ebedding model and gesture recognitoion model see models

Config

The most up to date and working config will be used by default. Develoupers can add a new config using the config scripts. New component implementations must be added manually, see adding component implementations

Saved Configs

• Configs generated with config scripts will be saved as config/full/full_config.json.
• The deploy.json config should be used on the PI controllers.
• The single_machine_default.json config can be used to run the controllers on a computer.
• The no_ml.json config can be used to run the controllers on a computer without device detection and gesture recognition.
• The default.json config is used by default.

Adding Devices

The user is required to add devices using the website or GUI. See the add device instrunctions for details. Device info is added to the device_info directory

Note:

• An added device will only be detected if a centroid has been added for it.
• To connect to a device without adding a centroid, you must specify the device in your config with a SimpleDeviceDetectionInterface component.

Launch

There is one CLI launch script that works for all three devices. Each device should be setup to automatically run the script and launch its respective controller.

Controllers:

• The central controller runs on the PI 5.
• The glass display controller runs on the PI Zero 2W connected to the display.
• The glass output controller runs on the PI Zero 2W connected to the camera and buttons.

CLI Commands:

The CLI interface allows you to run any of the three launches. The device_name arg specifies which of the three device controllers to launch. The config_path arg allows develoupers to launch the program with a different config during develoupment. The most up to date and working config will be used by default. See saved configs for useful alternative configs. When specifing the config_path use the relative path from your working directory. For example config/full/full_config.json if exicuting from the repo root. The ctr_ip arg allows you to optionally set the ip address (e.g., 127.0.0.1) of the central controller (PI 5). If not provided, the value from the config file will be used.

Notes:

• Run the setup commands first
• Download the models if needed

Script:

# PowerShell
python src/app/cli.py  -h

usage: cli.py [-h] --device_name
              {central_controller,glass_display_controller,glass_output_controller}
              [--config_path CONFIG_PATH] [--ctr_ip CTR_IP]

Run the application with specified device and configuration.

options:
  -h, --help            show this help message and exit
  --device_name {central_controller,glass_display_controller,glass_output_controller}
                        The name of the device to run.
  --config_path CONFIG_PATH
                        The relative path to the configuration file.
                        Default is config/full/default.json.
  --ctr_ip CTR_IP       The IP address of the central controller
                        (e.g., 127.0.0.1). If not provided, the value
                        from the config file will be used.

Examples:

cd path/to/image_vec_db

# Terminal 1: Launch Central Controller (PowerShell)
python src/app/cli.py --device_name central_controller --config_path config/full/default.json --ctr_ip 127.0.0.1

# Terminal 2: Launch Glass Display Controller (PowerShell)
python src/app/cli.py --device_name glass_display_controller --config_path config/full/default.json --ctr_ip 127.0.0.1

# Terminal 3: Launch Glass Output Controller (PowerShell)
python src/app/cli.py --device_name glass_output_controller --config_path config/full/default.json --ctr_ip 127.0.0.1

Notes:

• The default.json config will be used by default if the --config_path flag is not inlcuded.
• The 127.0.0.1 ip address will be used by default if the --ctr_ip flag is not inlcuded.

Structure

image_vec_db/
  config/
  device_info/
  model/
  smart-devices-dataset/
    v0/
    metadata.csv
  src/
    app/
      __init__.py
      config.py
      cli.py
    core/
    database/
  tests/
  requirements.txt
  README.md

Examples

You don't need a smart device or our smart glasses to try out the software! We have a separate examples repo with virtual devices to test with. Try out the virtual lamp or virtual mouse. The examples repo include example device info files that can be loaded into the add device GUI using the 'Load JSON' button. Alternitivly the device info may be entered manually. To learn more about adding devices see adding devices. To use this software on your local computer we recommend launching with the following config: config\full\single_machine_default.json.

Notes:

• The add device gui can generate device info JSONs for user entered devices.
• More example device info files are included in device_info\gesture_to_command_map\examples and device_info\connection_info\examples.
• The provided device info files can be loaded using the 'Load JSON' button in the add device GUI.