judge

LLM-as-Judge

Universal scorer for every records JSONL produced by any of the per-method folders under ../method/. Every method writes the same schema, so results are directly comparable.

Files

Path	Purpose
run_judge.py	Thin CLI wrapper around streamarena.judge.run_judge.
scripts/run_judge.sh	Shell wrapper with env-var defaults.

Environment variables

Variable	Purpose
JUDGE_URL	full generateContent URL, may contain {model}
JUDGE_API_KEY	judge API key
JUDGE_MODEL	judge model id (default gemini-3.1-pro)
CONCURRENCY	thread pool size (default 16)

How to run

export JUDGE_URL="https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent"
export JUDGE_API_KEY="your-key"

bash scripts/run_judge.sh ../method/offline/results/offline_qwen_qwen3.5.jsonl

Or directly:

python run_judge.py \
  --input ../method/offline/results/offline_qwen_qwen3.5.jsonl \
  --judge-url    "$JUDGE_URL" \
  --judge-api-key "$JUDGE_API_KEY" \
  --judge-model  gemini-3.1-pro

Auto-judge from a method run

Every per-method launcher supports AUTO_JUDGE=1, which chains this scorer right after the evaluation finishes. Example:

export SERPER_API_KEY=...
export JUDGE_URL="https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent"
export JUDGE_API_KEY=...

AUTO_JUDGE=1 \
BASE_URL=http://localhost:12347/v1 MODEL=qwen3.5 \
VIDEO_DIR=./data/StreamArena/videos \
bash ../method/offline/scripts/run_qwen.sh

What it does

1. Loads the records JSONL and skips rows without a question field.
2. Sends question / reference_answer / model_answer to the judge with a strict "Correct / Incorrect" prompt.
3. For Pro rows also enforces a timing window (ref_sec - 0.5 s .. ref_sec + 2.0 s) and uses a relaxed prompt.
4. Writes <input>.judge.jsonl with a slim per-row summary plus judge_score / judge_raw / optional judge_error.
5. Prints overall accuracy, per-qtype breakdown, per-video breakdown, and a failure-mode histogram.