GPU-parallel inference

[vadimcn]

I have a 16-GPU machine available, and a model that fits on a single GPU. What is the best way to optimize vLLM performance in this situation?

I've tried tensor parallel, however it seems that it requires both the number of attention heads and the vocabulary size to be divisible by the number of GPUs, and in my case the GCD of these is 4 😞