Releasing evaluation log probabilites

[tholiao]

Hi, thanks for open-sourcing model code! Could you release the log probabilities for evaluation tasks (i.e., the model probabilities for valid answers for each prompt on each question for all evaluated datasets)? This data would allow for for fine-grained evaluation of models and comparing against other LLMs.

cf. facebookresearch/metaseq#25

[HaokunLiu]

Oops, I don't think our code saved the individual probabilities. But I will keep it in mind when designing future project codebases. If you need them, would you mind modifying the validation_epoch_end function in src.models.EncoderDecoder.py to save them to a file and then running fine-tuning experiments from your side?