This repository contains the official implementation of STARS, a SpatioTemporal Autoencoded Representation for Social-interaction. The codebase supports converting pedestrian motion datasets into graph formats, training homogeneous and heterogeneous GNN-VAEs, and running downstream linear probes for social classification tasks.
Run this command block from the root of the repository to download all required datasets, build the Docker environment, and execute the test harness for all experiments:
- Download real datasets and pretrained checkpoints
./download_data.sh
- Build the Docker container environment
./container build
-
Generate the required heterogeneous datasets
3a. The training dataset (all scenes)
./container cmd 'cd src && python dataloaders/socialnav_to_hdf5_v3.py convert --train_dirs ../data/dataset_split/train/ssub ../data/dataset_split/val/ssub --no_synth --output ../data/social_nav_v3.h5'3b. The human oracle dataset for evaluation (validation scenes only)
./container cmd "cd src && python dataloaders/socialnav_to_hdf5_v3.py convert --train_dir ../data/dataset_split/val/ssub --no_synth --output ../data/socialnav_human_oracle_v3.h5" -
Run all experiments and validate results
./container cmd "python src/test_paper_results.py --run-all --fast"
In
--fastmode, thetest_paper_results.pyscript will run only 1 seed per pipeline instead of 10.
To run the experiments and test harness using the real datasets and the pretrained heterogeneous model, run the download script from the repository root:
./download_data.shTo release this repository and allow others to download the real datasets and pretrained checkpoints, upload the following files to a public file hosting service (such as Google Drive, Hugging Face, or Zenodo) and update the File IDs/URLs in download_data.sh:
combined_v1.h5(~563MB): Preprocessed unified SEAN+SNSB heterogeneous dataset. Required for model training and probe evaluation.social_nav_eval_combined.h5(~4.2MB): SNSB evaluation dataset. Required for probe validation.split_indices_final.pt(~18KB): Final scene splits. Required for splits parity.best_model_optuna.pt(~16MB): Pretrained heterogeneous GNN-VAE model checkpoint (best model from the Optuna search). Required to run the test harness / probe evaluations without retraining from scratch.dataset_splits.zip(~500MB): Archive of the 10 data splits (s1..s10) needed to run the Random Forest baseline sweeps.
Recommendation: Upload these files to a public Google Drive folder. You can obtain the direct download file IDs for each file and paste them into the corresponding variables in download_data.sh (COMBINED_H5_ID, SNS_EVAL_H5_ID, etc.). Alternatively, host them on Hugging Face Hub or Zenodo and modify the curl commands to fetch from those URLs.
All dependencies are managed via Docker.
# Build the container
./container build
# Enter the shell
./container shellAlternatively, you can activate your local python environment and install requirements:
cd src
# source /path/to/your/venv/bin/activateA test harness is provided to verify that the generated metrics match the quantitative results (Table 1 F1-scores) from the paper.
First, navigate to the source directory (or run within the container shell):
cd srcYou can run all experiments (both heterogeneous and homogeneous pipelines) and automatically compare the results against the paper's ground truth with a single command:
# Run a fast, smaller test sweep (1 seed) for all experiments to verify pipeline execution
python test_paper_results.py --run-all --fast
# Run the full test sweep for all experiments (10 seeds, matches the paper setup)
python test_paper_results.py --run-allIf you only want to test a single pipeline:
# Run a fast, smaller sweep for the heterogeneous pipeline
python test_paper_results.py --run-hetero --fast
# Run the full heterogeneous pipeline sweep
python test_paper_results.py --run-hetero
# Run a fast, smaller sweep for the homogeneous pipeline
python test_paper_results.py --run-homo --fast
# Run the full homogeneous pipeline sweep
python test_paper_results.py --run-homoIf you have already run the pipeline and have the output CSV files, you can validate them directly without retraining:
python test_paper_results.py --csv-sean path/to/summary_sean.csv --csv-sns path/to/summary_snsb.csvExperiments are divided into three main pipelines: Heterogeneous, Homogeneous, and Downstream Probing. We provide orchestrated shell scripts in src/hetero_scripts for reproducibility.
The usual execution order is:
run_train_vae.sh(Self-supervised pre-training)run_label_eff.sh(Downstream linear probe sweeping)run_summarize.sh(Results summarization)visualize_probes_latent.py(Generate t-SNE and PCA figures for the SNS latent space)
All commands below should be run from the src/ directory. GPU selection is managed via CUDA_VISIBLE_DEVICES. The wrapper scripts schedule at most one training/eval process per GPU at any time.
To train 10 independent unified SEAN+SNSB Heterogeneous VAEs on the fixed data split:
CUDA_VISIBLE_DEVICES=0,1,2,3 ./hetero_scripts/run_train_vae.sh \
--pipeline combined \
--out-root checkpoints/repro_hetero_optuned128Run the label-efficiency sweep (10 VAEs x 10 runs x 6 fractions):
CUDA_VISIBLE_DEVICES=0,1,2,3 ./hetero_scripts/run_label_eff.sh \
--pipeline combined \
--vae-root checkpoints/repro_hetero_optuned128 \
--out-prefix label_eff_results_repro_combinedSummarize the results:
./hetero_scripts/run_summarize.sh \
--pipeline combined \
--in-prefix label_eff_results_repro_combined \
--merged label_eff_results_repro_combined.csvExpected merged size: 1201 lines (1 header + 10 VAEs * 6 fractions * 10 runs * 2 tasks).
To train 10 independent SEAN-only heterogeneous VAEs on the fixed data split:
CUDA_VISIBLE_DEVICES=0,1,2,3 ./hetero_scripts/run_train_vae.sh \
--pipeline sean \
--out-root checkpoints/repro_sean_optuned128Run the label-efficiency sweep (10 VAEs x 10 runs x 6 fractions):
CUDA_VISIBLE_DEVICES=0,1,2,3 ./hetero_scripts/run_label_eff.sh \
--pipeline sean \
--vae-root checkpoints/repro_sean_optuned128 \
--out-prefix label_eff_results_repro_seanSummarize the results:
./hetero_scripts/run_summarize.sh \
--pipeline sean \
--in-prefix label_eff_results_repro_sean \
--merged label_eff_results_repro_sean.csvExpected merged size: 601 lines (1 header + 10 VAEs * 6 fractions * 10 runs).
The homogeneous graph model uses config-driven training scripts. To train the homogeneous model:
python train_hgnn_unified_v2.py --config params/config.yamlAdd --ssub to run in SSUB-only mode (no valid_mask, no node features).
train_hgnn_unified_v2.py reads only the homognn_params section of params/config.yaml; all other top-level sections (vae, linear_probe, optuna_hgnn, etc.) are ignored by this script. The key fields to set under homognn_params:
h5_path— the input graph dataset (HDF5) to train on.experiments_root/run_name— output location for this run. The script createsexperiments_root/<timestamped run_name>/and copies the resolved config into it; use a uniquerun_nameso checkpoints aren't overwritten.device— the GPU to train on (e.g.cuda:0), orcpu. One run uses a single GPU.seed— vary this to produce the independent seeds for a multi-seed sweep.- Architecture keys (
hidden_dim,latent_dim,num_mp_layers,edge_transformer_layers,num_heads,dropout) — define the encoder. If you later probe this checkpoint, the probe config's matching keys must be identical or weight loading will fail.
All label-efficiency ablations keep the pretrained VAE encoder weights frozen and modify either the probe input or prediction baseline via the --extra argument in run_label_eff.sh.
- Random encoder:
CUDA_VISIBLE_DEVICES=0,1,2,3 ./hetero_scripts/run_label_eff.sh \ --pipeline combined --vae-root checkpoints/repro_hetero_optuned128 \ --out-prefix label_eff_results_repro_random_encoder --extra "--mode random_encoder" - Weighted random baseline:
CUDA_VISIBLE_DEVICES=0,1,2,3 ./hetero_scripts/run_label_eff.sh \ --pipeline combined --vae-root checkpoints/repro_hetero_optuned128 \ --out-prefix label_eff_results_repro_wrb --extra "--mode weighted_random_baseline" - Robot-only latent representation:
CUDA_VISIBLE_DEVICES=0,1,2,3 ./hetero_scripts/run_label_eff.sh \ --pipeline combined --vae-root checkpoints/repro_hetero_optuned128 \ --out-prefix label_eff_results_repro_robot_only --extra "--ablation robot_only" - Bystander-only latent representation:
CUDA_VISIBLE_DEVICES=0,1,2,3 ./hetero_scripts/run_label_eff.sh \ --pipeline combined --vae-root checkpoints/repro_hetero_optuned128 \ --out-prefix label_eff_results_repro_bystander_only --extra "--ablation bystander_only"
- Raw robot 592D features (probes normalized input features directly):
CUDA_VISIBLE_DEVICES=0,1,2,3 ./hetero_scripts/run_label_eff.sh \ --pipeline sean --vae-root checkpoints/repro_sean_optuned128 \ --out-prefix label_eff_results_repro_sean_raw_robot --extra "--ablation raw_robot"
To generate t-SNE and PCA visualization plots for the SNS dataset on the test split:
CUDA_VISIBLE_DEVICES=0 python visualize_probes_latent.py \
--checkpoint checkpoints/repro_hetero_optuned128/run0/best_model.pt \
--split testTo run a parallelized Optuna search to tune model hyperparameters:
CUDA_VISIBLE_DEVICES=0,1,2,3 ./hetero_scripts/run_optuna.sh \
--pipeline combined \
--study repro_unified_256x128 \
--storage sqlite:///optuna_repro_unified_256x128.db \
--n-trials 10Note: --n-trials is per-GPU, making the total trials equal to n_trials * visible_GPUs.
--pipeline:combined(SEAN+SNSB) |sean(SEAN-only)--n-vaes: Number of VAE checkpoints to train (default: 10)--out-root: Directory where checkpoints are saved--hidden-dim/--latent-dim: Dimensions of hidden layers & latent space (default: 256 / 128)--epochs/--lr/--batch-size/--dropout: Training hyperparameters--extra: Verbatim string arguments passed to train script
--pipeline:combined|sean--vae-root: Directory containing the pretrained VAE runs--out-prefix: Prefix for the output CSV results files--no-flip-surprise: Disables surprise target flipping (SEAN-only pipeline only)--extra: Parameters passed directly to probe script (e.g.--mode random_encoderor--ablation robot_only)
--pipeline:combined|sean--in-prefix: Prefix of input CSV files to merge--merged: Filename of the output merged raw CSV file--task:sean|snsb|both
summarize_label_efficiency.py aggregates probe outputs across runs and computes mean
- Combined Model (SEAN + SNSB):
- SEAN:
f1_macro,auc_macro, plus per-label F1/AUC/accuracy. - SNSB:
f1_macro3(mean F1 of follow/avoid/not_consider classes),f1_macro(5-class sklearn macro),acc,pa(prediction accuracy), plus per-class metrics.
- SEAN:
- SEAN-only Model:
f1_macro,auc_macro,acc_macro, plus per-label F1/AUC/accuracy.
Below is a detailed map of the core files implementing the heterogeneous GNN-VAE architecture:
src/models/:- unified_gnn_vae_v1.py: The combined Heterogeneous GNN-VAE (SEAN+SNSB) model. Implements the
HeteroMPLayerwhich dynamically refines edge embeddings. - gnn_vae_v4.py: The SEAN-only heterogeneous GNN-VAE model.
- unified_gnn_vae_v1.py: The combined Heterogeneous GNN-VAE (SEAN+SNSB) model. Implements the
src/dataloaders/:- unified_gnn_vae_datasets_v1.py: Combined dataloader wrappers.
- unified_graph_dataset_v1.py: Combined SEAN graph dataset.
- sean_graph_dataset_v4.py: SEAN-only graph dataset.
- socialnav_hetero_graph_dataset_v1.py: SNSB heterogeneous graph dataset.
src/probes/: Linear probes for the downstream classification tasks.src/train_unified_gnn_vae_v3.py: Trainer script for the combined model.src/train_gnn_vae_v4.py: Trainer script for the SEAN-only model.src/eval_label_efficiency.py: Downstream label-efficiency sweep script for the combined model.src/eval_sean_label_efficiency_v4.py: Downstream label-efficiency sweep script for the SEAN-only model.src/summarize_label_efficiency.py: Summarizes LP sweep metrics.src/optuna_unified_v1.py: Parallelized Optuna tuning for the combined model.src/hetero_scripts/: Orchestration shell scripts to automate runs.