Request for updated evaluation scripts to reproduce Qwen3.5-397B-A17B metrics

[Pengxiang-zhao]

Could you please update the test code for Qwen3.5-397B-A17B? I would like to reproduce the benchmark metrics shown in the latest results.

[qykong]

Hi, for Qwen3.5 model family, we employed the this general implementation https://github.com/Tongyi-MAI/MobileWorld/blob/main/src/mobile_world/agents/implementations/general_e2e_agent.py

[simplelifetime]

Having the same problem. Will the general implementation hurts the performance? Qwen3.5 may have a special-designed action space and prompt for evaluation?

[qykong]

That's a valid concern. It's possible that a specialized action space and prompt design could improve performance for Qwen3.5 models. Once they release their official evaluation guidelines or a dedicated implementation, we will re-evaluate and update the results accordingly. Thanks for raising this!