[Usage]: vision encoder speedup?

[yanyongyu]

Your current environment

pytorch 2.7.1, vLLM 0.10.1.1, H20 * 8

How would you like to use vllm

I'm currently using Qwen2.5-VL 32B to do some multi-modal inferencing. From the result of torch profiler, i see the vision encoder takes a "long" time in the prefill stage. I'd like to know if there is any way to increase the performance of the vision encoder (like compilation etc.). Thanks!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

You can try enabling batch level DP: https://docs.vllm.ai/en/latest/configuration/optimization.html#data-parallelism-dp

[yanyongyu]

Hi @DarkLight1337, thanks for your reply. In our case, we need to reduce the TTFT instead of throughput (batch size is very small). DP doesn't seem to help?

[DarkLight1337]

To reduce TTFT, you can try API server scale-out (also mentioned in the same doc page). Note however that it currently doesn't work for the default Dockerfile entrypoint (see #25309) so you have to change the entrypoint to vllm serve and add --model before passing the model name

[yanyongyu]

It seems "API server scale-out" helps to reduce the multi-modal data pre-processing time 🤔 . Actually, we wrap the vllm v1 engine and do the pre-processing outside of vllm. So, I'm looking for ways to reduce the vision encoder embedding time on the GPU side.

[DarkLight1337]

I see. Actually based on #22742, using batch-level DP can also reduce TTFT so I suggest you try that first

[yanyongyu]

@DarkLight1337 Thanks. DP for ViT looks interesting. I will have a look.

[yanyongyu]

By the way, will vLLM consider doing some operations like compiling the ViT in the future?

[DarkLight1337]

Yes, please try out #23207 and report your results there

[ywang96]

@yanyongyu Assuming you're running TP=8, I definitely recommend running batch-level DP ViT.

You can also try out a few features that'll improve the system performance in general, such as --async-scheduling and --mm-processor-cache-type shm