clarification on how MLQA eval is done in mt5

[dorost1234]

Hi

MLQA only have eval/test sets, could you please assist me and clarify on which dataset you have trained the model for the zero-shot experience? The result reported in the paper, are they computed on the eval set or the test set?

thank you.

[hanss0n]

It says so in the paper. The English zero-shot evaluation on MLQA was done using SQuAD and the multilingual one was conducted using SQuAD + translations of SQuAD in the target languages, i.e. Arabic, Spanish, English, German, Hindi, Vietnamese and simplified Chinese. The MLQA set used is most likely the test one since its intended use is benchmarking.