About eval in mathverse_testmini_vision and mathverse_testmini

[xiaoshideta]

Hello，I use the test.sh， like：
accelerate launch --num_processes=1 --main_process_port 23456 -m lmms_eval
--model llava_onevision_llada
--gen_kwargs='{"temperature":0,"cfg":0,"remasking":"low_confidence","gen_length":64,"block_length":64,"gen_steps":32,"think_mode":"think","threshold": 1, "prefix_refresh_interval": 32}'
--model_args pretrained=/root/autodl-tmp/model/Llada-V,conv_template=llava_llada,model_name=llava_llada,use_fast_dllm=true
--tasks mathverse_testmini_vision
--batch_size 1
--log_samples
--log_samples_suffix mathverse_testmini_vision
--output_path exp/llava_v_eval/LLaDA-V

but I meet a problem， ERROR | lmms_eval.tasks.mathverse.mathverse_evals:get_chat_response:138 - HTTPSConnectionPool(host='api.openai.com', port=443): Max retries exceeded with url: /v1/chat/completions (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0e716a8880>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

I want to know that dataset eval must need the openai api？

[xiaoshideta]

accelerate launch --num_processes=1 --main_process_port 23456 -m lmms_eval
--model llava_onevision_llada
--gen_kwargs='{"temperature":0,"cfg":0,"remasking":"low_confidence","gen_length":64,"block_length":64,"gen_steps":32,"think_mode":"think","threshold": 1, "prefix_refresh_interval": 32}'
--model_args pretrained=/root/autodl-tmp/model/Llada-V,conv_template=llava_llada,model_name=llava_llada,use_fast_dllm=true
--tasks mathverse_testmini_vision
--batch_size 1
--log_samples
--log_samples_suffix mathverse_testmini_vision
--output_path exp/llava_v_eval/LLaDA-V
it generates six files，mathverse_testmini_vision_dominant_scores.json shows that average accuracy: 31.73？

[xiaoshideta]

your report is 28.5？I want to know where is different

[yyyouy]

你好，我用的是 test.sh，比如：accelerate launch --num_processes=1 --main_process_port 23456 -m lmms_eval --model llava_onevision_llada --gen_kwargs='{“temperature”：0，“cfg”：0，“remasking”：“low_confidence”，“gen_length”：64，“block_length”：64，“gen_steps”：32，“think_mode”：“think”，“threshold”：1，“prefix_refresh_interval”： 32}' --model_args pretrained=/root/autodl-tmp/model/Llada-V，conv_template=llava_llada，model_name=llava_llada，use_fast_dllm=true --tasks mathverse_testmini_vision --batch_size 1 --log_samples --log_samples_suffix mathverse_testmini_vision --output_path exp/llava_v_eval/LLaDA-V

但我遇到了问题，错误 |lmms_eval.tasks.mathverse.mathverse_evals：get_chat_response：138 - HTTPSConnectionPool（host='api.openai.com'， port=443）：最大重试次数超过，URL： /v1/chat/completions （由 NewConnectionError（'<urllib3.connection.HTTPSConnection 对象 0x7f0e716a8880>： 未能建立新连接：[Errno 101] Network is inreachable'））

我想知道数据集评估必须需要OpenAI API？

yes, openai api is needed, because it needs gpt model to evaluate the response from llada v

[yyyouy]

你的报告是28.5？我想知道哪里不同

Our results is 28.5 and the scripts you used is using Fast-dllm, and their evaluation results is 28.6 (see Fast dllm paper). If you have question, you can try the eval/scripts/evaluate.sh scripts to further evaluate our model.

[xiaoshideta]

你的报告是28.5？我想知道哪里不同

Our results is 28.5 and the scripts you used is using Fast-dllm, and their evaluation results is 28.6 (see Fast dllm paper). If you have question, you can try the eval/scripts/evaluate.sh scripts to further evaluate our model.

We have addressed the related issues and the system now operates normally. Nevertheless, the results remain inconsistent with 28.5, and further diagnosis is ongoing.