How to evaluate the distilled model

[HuuYuLong]

Thanks for the amazing work!

I’m having trouble testing VMEvalKit with the stage1 checkpoint, due to all "layer_types" in the saved config being "full_attention".

Could you share any scripts or specific settings for this? Thanks!

[Hongyuan-Tao]

Thank you for your feedback! You can first try replacing the corresponding files in the model weight path with config.json, configuration_infinitevl.py and modeling_infinitevl.py from InfiniteVL. We will also fix this issue in a timely manner.

[Hongyuan-Tao]

To evaluate the InfiniteVL model using VLMEvalkit, you only need to make two quick modifications to the codebase:

Step 1: Add the model configuration
In vlmeval/config.py, locate "qwen2vl_series" and add the InfiniteVL entry:
'InfiniteVL': partial(Qwen2VLChat, model_path='YOUR_CKPT_PATH', min_pixels=128*28*28, max_pixels=2304*28*28),

Step 2: Update the model loading logic
In vlmeval/vlm/qwen2_vl/model.py, replace the existing model initialization code with the following snippet:

from transformers import AutoModelForCausalLM
self.model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True
).eval().cuda()

If you encounter any issues, feel free to discuss them with us here!