Skip to content

datasets.md

Latest commit

 

History

History
153 lines (125 loc) · 5.66 KB

datasets.md

File metadata and controls

153 lines (125 loc) · 5.66 KB
Raw

Datasets

The ten public benchmarks reported in the paper, with their upstream source, the on-disk path the loader expects, and the exact command to evaluate them. Download everything with python scripts/download_data.py --datasets all.

All commands assume a vLLM server is up and the env vars from vllm_setup.md are exported.

C1 — Financial Numerical Reasoning

FinQA — finqa

  • Source: czyssrs/FinQA (EMNLP 2021)
  • Path: data/raw/finqa/test.json (1,147 test)
python -m mocafin.evaluate --dataset finqa \
  --data_path data/raw/finqa/test.json \
  --output results/finqa.jsonl --workers 16

DocMath-Simplong / Complong — dm_simplong / dm_complong

  • Source: yale-nlp/DocMath-R (ACL 2024)
  • Path: data/raw/docmath/data/{simplong,complong}_testmini-00000-of-00001.parquet (100 / 300 testmini)
python -m mocafin.evaluate --dataset dm_simplong \
  --data_path data/raw/docmath/data/simplong_testmini-00000-of-00001.parquet \
  --output results/dm_simplong.jsonl --workers 16

python -m mocafin.evaluate --dataset dm_complong \
  --data_path data/raw/docmath/data/complong_testmini-00000-of-00001.parquet \
  --output results/dm_complong.jsonl --workers 8

FinanceMath — financemath

  • Source: yale-nlp/FinanceMath (ACL 2024, gated)
  • Path: data/raw/financemath/validation.json (200 validation)
python -m mocafin.evaluate --dataset financemath \
  --data_path data/raw/financemath/validation.json \
  --output results/financemath.jsonl --workers 16

C2 — General Tabular Reasoning

HiTab — hitab

  • Source: microsoft/HiTab (ACL 2022)
  • Path: data/raw/hitab/repo/data/test_samples.jsonl (1,584 test; tables under tables/raw/)
python -m mocafin.evaluate --dataset hitab \
  --data_path data/raw/hitab/repo/data/test_samples.jsonl \
  --output results/hitab.jsonl --workers 16

MultiHiertt — multihiertt

python -m mocafin.evaluate --dataset multihiertt \
  --data_path data/raw/multihiertt/dev.json \
  --output results/multihiertt.jsonl --workers 16

TabMWP — tabmwp

  • Source: lupantech/PromptPG (ICLR 2023)
  • Path: data/raw/tabmwp/repo/data/tabmwp/problems_test1k.json (1,000 test subset)
python -m mocafin.evaluate --dataset tabmwp \
  --data_path data/raw/tabmwp/repo/data/tabmwp/problems_test1k.json \
  --output results/tabmwp.jsonl --workers 16

WikiTableQuestions — wikitq

python -m mocafin.evaluate --dataset wikitq \
  --data_path data/raw/wikitq/repo/data/pristine-unseen-tables.tsv \
  --output results/wikitq.jsonl --workers 16

C3 — Domain Knowledge QA

ESGenius — esgenius

  • Source: ANGEL-NTU/ESGenius (EMNLP 2025)
  • Path: data/raw/esgenius/ESGenius_1136q.json (1,136 MCQ)
python -m mocafin.evaluate --dataset esgenius \
  --data_path data/raw/esgenius/ESGenius_1136q.json \
  --output results/esgenius.jsonl --workers 16

C4 — Multimodal Chart Reasoning

FinChart-Bench — finchart_bench (TF / MC / QA)

  • Source: Tizzzzy/FinChart-Bench (2025)
  • Path: data/raw/finchart_bench/ — the loader reads whichever of TF_data.json, MC_data.json, QA_data.json are present in the directory you point at, and tags each sample with a subtask field. Point at a subtask-only directory to run one track.
  • Requires the vision pass (--vision_transcribe question_conditioned) and a multimodal vLLM server (the same Qwen3.6-27B server works).
# open QA subtask (the headline C4 result)
python -m mocafin.evaluate --dataset finchart_bench \
  --data_path data/raw/finchart_bench_qa_only \
  --vision_transcribe question_conditioned \
  --output results/finchart_qa.jsonl --workers 4

# True/False and Multiple-Choice subtasks
python -m mocafin.evaluate --dataset finchart_bench \
  --data_path data/raw/finchart_bench_tf_only \
  --vision_transcribe question_conditioned \
  --output results/finchart_tf.jsonl --workers 4

python -m mocafin.evaluate --dataset finchart_bench \
  --data_path data/raw/finchart_bench_mc_only \
  --vision_transcribe question_conditioned \
  --output results/finchart_mc.jsonl --workers 4

Per-subtask accuracy can be broken out from the subtask field in the JSONL:

import json, collections
ok, tot = collections.Counter(), collections.Counter()
for line in open("results/finchart_qa.jsonl"):
    r = json.loads(line); s = r.get("subtask", "qa")
    tot[s] += 1; ok[s] += int(r.get("is_correct", False))
for k in tot: print(k, f"{ok[k]/tot[k]*100:.2f}%")

Output

Every run writes:

  • results/<name>.jsonl — one record per example (answer, generated code, market snapshot, trader books, correctness).
  • results/<name>_metrics.json — summary: answer_accuracy, code_execution_rate, code_answer_consistency, abstention_rate, mean_market_confidence.

answer_accuracy is the headline exact-match / execution-accuracy metric used in the paper. Runs are resumable: re-running the same command continues from the existing JSONL.