Skip to content

Commit

Permalink
add docvqa eval scripts
Browse files Browse the repository at this point in the history
  • Loading branch information
vealocia committed Sep 11, 2023
1 parent b553964 commit 157074d
Showing 1 changed file with 44 additions and 0 deletions.
44 changes: 44 additions & 0 deletions eval_mm/EVALUATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,50 @@ python -m torch.distributed.launch --use-env \
</details>
### [DocVQA](https://www.docvqa.org/datasets)
<details>
<summary>Data Preparation</summary>
```bash
mkdir -p data/docvqa && cd data/docvqa

# download images and annotations from https://www.docvqa.org/datasets

# download converted files
# train
wget https://ofasys-wlcb.oss-cn-wulanchabu.aliyuncs.com/Qwen-VL/evaluation/docvqa/train.jsonl
# val
wget https://ofasys-wlcb.oss-cn-wulanchabu.aliyuncs.com/Qwen-VL/evaluation/docvqa/val.jsonl
# test
wget https://ofasys-wlcb.oss-cn-wulanchabu.aliyuncs.com/Qwen-VL/evaluation/docvqa/test.jsonl
cd ../..
```
</details>
<details>
<summary>Evaluation</summary>
```bash
# evaluate vqa score on docvqa val split
ds="docvqa_val"
checkpoint=/PATH/TO/CHECKPOINT
python -m torch.distributed.launch --use-env \
--nproc_per_node ${NPROC_PER_NODE:-8} \
--nnodes ${WORLD_SIZE:-1} \
--node_rank ${RANK:-0} \
--master_addr ${MASTER_ADDR:-127.0.0.1} \
--master_port ${MASTER_PORT:-12345} \
evaluate_vqa.py \
--checkpoint $checkpoint \
--dataset $ds \
--batch-size 8 \
--num-workers 2
```
</details>
### [ChartQA](https://aclanthology.org/2022.findings-acl.177/)
<details>
Expand Down

0 comments on commit 157074d

Please sign in to comment.