Generalized Measures of Anticipation and Responsivity in Online Language Processing

Experiments and analysis

The experiments for our paper can be reproduced from the repository root. Create a Python environment and install the pinned dependencies first:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt

The complete reproduction pipeline has four stages.

1. Compute the exact and sampling-based language model estimates:

   OUT_ROOT=data/estimates bash run_grid.sh

   run_grid.sh invokes src/compute_exact_metrics.py once for each language model and src/compute_sampling_metrics.py for each sampling configuration. The default grid uses GPT-2 small and GPT-Neo 125M, sample sizes from 4 to 512, continuation lengths of 5, 10, and 15 tokens, temperature 1.0, and a CUDA device. It therefore runs 50 configurations in total. The estimates, runtime measurements, and bootstrap summary statistics are written below data/estimates; logs are written below logs/batch.

   The grid settings can be changed with environment variables. For example, the following command runs both models with two sample sizes, two continuation lengths, temperature 0.8, and top-k sampling with $k=50$:

   N_SAMPLES="128 512" N_TOKENS="5 10" TEMPERATURE=0.8 TOP_K=50 \
     OUT_ROOT=data/estimates bash run_grid.sh

   To run on a CPU, set DEVICE=cpu. The script also supports SLURM arrays and uses SLURM_ARRAY_TASK_ID and SLURM_ARRAY_TASK_COUNT to divide the configurations between tasks.

2. Join the estimates to the psycholinguistic measurements in the Aligned dataset:

   python3 src/apply_estimates_to_dataset.py

   This script joins the estimate directories created in the preceding stage to the psycholinguistic measurements in the Aligned dataset. The input and output paths are defined at the top of the script. The resulting dataset includes the current word estimates, the previous word estimates used for spillover analyses, and the baseline predictors.

3. Open analysis/emnlp2024/crossvalidation.ipynb and run all cells. The notebook reads data/psyling/aligned_with_estimates_v2.csv and writes the regression results to analysis/emnlp2024/crossvalidation_results.

4. Open analysis/emnlp2024/analysis.ipynb and run all cells. This notebook reads the estimate summaries and runtimes together with the cross-validation results, performs the statistical analyses, and generates the paper figures.

Data

The Aligned dataset in the data folder is taken from the prediction-resource repository. Please cite the correct associated papers:

• de Varda et al., 2023 for the dataset of aligned cloze probabilities, predictability ratings, EEG and reading time data.
• Frank et al., 2013 for the eye-tracking and self-paced reading time data.
• Frank et al., 2015 for the dataset of event-related brain potentials.

BibTeX

@inproceedings{giulianelli-etal-2024-generalized,
    title={Generalized Measures of Anticipation and Responsivity in Online Language Processing},
    author={Giulianelli, Mario and Opedal, Andreas and Cotterell, Ryan},
    booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
    year = {2024},
    publisher = {Association for Computational Linguistics},
    address = {Miami, Florida, USA},
    url={https://arxiv.org/pdf/2409.10728}
}