Syn-STARTS: Synthesized START Triage Scenario Generation Framework

This repository contains the official implementation, datasets, and analysis scripts for our paper: "Syn-STARTS: Synthesized START Triage Scenario Generation Framework for Scalable LLM Evaluation."

Repository Structure

.
├── analysis/              # Scripts for result analysis and figure generation  
├── datasets/              # Experimental datasets (expert-authored and Syn-STARTS)  
├── generate_dataset/      # Scripts for corpus construction  
├── results/               # Raw results and logs from evaluations  
├── run_task/              # Scripts to run the triage tag prediction task  
└── src/                   # Core modules for generation and validation 

Installation

Please ensure Docker is correctly installed in your environment.

1. Set up User and Group IDs for Docker:

printf "UID=$(id -u)\nGID=$(id -g)\n" > .env

2. Build the Docker Image:

docker compose build

3. Configure API Keys (for closed models): To run experiments on closed-source models, add your API keys to the .env file:

OPENAI_API_KEY="your_openai_key"
ANTHROPIC_API_KEY="your_anthropic_key"

Usage: Replicating the Experiments

1. Corpus Construction

The foundational 2,000-case corpus is generated using the scripts in generate_dataset/.

docker compose run --rm experiment poetry run python3 generate_dataset/generate_dataset.py \
  --exp_id "dataset_final" \ # name of directory to save corpus
  --model_id "meta-llama/Llama-3.1-70B-Instruct" \
  --total_scenarios 2000 \

• generate_dataset/workflow.py: Manages the iterative generate-and-validate process to build the corpus with the target tag distribution.
• src/generate_dataset/scenario_generator.py: Calls the generator LLM (Llama-3.1-70B-Instruct) using prompts defined in prompt_templates/ to produce a single Syn-STARTS case as a JSON object.
• src/generate_dataset/triage_validator.py: Implements the three-step validation pipeline (START Consistency, Medical Plausibility, Narrative Consistency) to verify each generated case.

(Note: The pre-generated 2,000-case corpus is already provided in datasets/Syn-STARTS_datasets/all_scenarios/sampled_data.csv)

The Syn-SRARTS datasets for evaluation tasks, detailed in the [## Datasets] section below, were then sampled from this corpus.

2. Running Evaluations

The triage tag prediction task is executed using the scripts in run_task/.

• For open-source models:

# Example: Run Mistral 7B on the TRIAGE-adult dataset
docker compose run --rm experiment poetry run python3 run_tasks/run_tasks.py \
  --model_id "mistralai/Mistral-7B-Instruct-v0.2" \
  --questions_csv "datasets/TRIAGE/triage_questions_adult.csv" \
  --output_dir "results/

• For closed-source models:

# Example: Run GPT-3.5-turbo on a Syn-STARTS dataset
docker compose run --rm experiment poetry run python3 run_tasks/run_tasks_closed_models.py \
  --model_name "gpt-3.5-turbo" \
  --questions_csv "datasets/TRIAGE/triage_questions_adult.csv" \
  --output_dir "results/"

3. Analysis

Scripts in the analysis/ directory are used to process the raw results from results/ and generate the data and figures presented in the paper, which are in turn saved to their respective subdirectories within the analysis/ folder (e.g., analysis/figure4/, analysis/figure5/).

# Example: Run GPT-3.5-turbo on a Syn-STARTS dataset
docker compose run --rm experiment poetry run python3 {analysis_script_path} \

Datasets

All datasets used in our experiments are located in the datasets/ directory.

• datasets/TRIAGE/: The expert-authored benchmark from TRIAGE.

  • triage_questions_adult.csv: The "TRIAGE-adult" dataset (n=54) used as our gold standard.

• datasets/Syn-STARTS_datasets/: Our synthetic datasets.

• all_scenarios/sampled_data.csv: The full 2,000-case validated corpus.

• turing-test_20/: The 20 case pairs used for the blinded expert review.

• batch/: The ten replicate datasets for each experimental configuration. - TRIAGEa_{0..9}.csv: "TRIAGE-adult" tag distribution (n=54; {18, 11, 22, 3}). - syn_{SCALE}_{0..9}.csv: Uniform tag distribution, where SCALE is one of {12, 20, 56, 100, 200}.

Citation

If you use this repository, please cite our paper: