[Usage]: Problem with concurrency in encoder-based embedder serving with V1 Engine

[gabinguo]

Your current environment

The output of python collect_env.py

Collecting environment information...
uv is set
==============================
        System Info
==============================
OS                           : Rocky Linux 9.6 (Blue Onyx) (x86_64)
GCC version                  : (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5)
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.34

==============================
       PyTorch Info
==============================
PyTorch version              : 2.8.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.5 (main, Aug 14 2024, 05:08:31) [Clang 18.1.8 ] (64-bit runtime)
Python platform              : Linux-5.10.0-31-amd64-x86_64-with-glibc2.34

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.93
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : GPU 0: NVIDIA GeForce RTX 3080 Ti
Nvidia driver version        : 570.86.10
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        40 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               2
On-line CPU(s) list:                  0,1
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
CPU family:                           6
Model:                                79
Thread(s) per core:                   1
Core(s) per socket:                   1
Socket(s):                            2
Stepping:                             1
BogoMIPS:                             6399.92
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat umip md_clear arch_capabilities
Virtualization:                       VT-x
L1d cache:                            64 KiB (2 instances)
L1i cache:                            64 KiB (2 instances)
L2 cache:                             8 MiB (2 instances)
L3 cache:                             32 MiB (2 instances)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0,1
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Mitigation; PTE Inversion; VMX flush not necessary, SMT disabled
Vulnerability Mds:                    Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:               Mitigation; PTI
Vulnerability Mmio stale data:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Mitigation; Clear CPU buffers; SMT Host state unknown

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.3.1.post1
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.14.1
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-ml-py==13.580.82
[pip3] nvidia-nccl-cu12==2.27.3
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pynvml==13.0.1
[pip3] pyzmq==27.1.0
[pip3] torch==2.8.0
[pip3] torchaudio==2.8.0
[pip3] torchvision==0.23.0
[pip3] transformers==4.56.2
[pip3] triton==3.4.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.10.2
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-1     0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
NCCL_VERSION=2.25.1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=12.8.1
VLLM_CONFIGURE_LOGGING=1
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/lib64
VLLM_USE_V1=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

How would you like to use vllm

Dear vLLM team : )

I would like to ask a question regarding the embedding model serving, specifically BAAI/bge-large-en-v1.5 model.

I served it with cli command:

vllm serve BAAI/bge-large-en-v1.5 --max-model-len 512 --gpu-memory-utilization 0.9 --max-num-batched-tokens 819200 --max-num-seqs 1600 --dtype float32 --override-pooler-config {pooling_type: MEAN, normalize: true} --enforce-eager

• --enforce-eager I recently discovered that in vLLM 0.10.2, with V1, I have to put this flag to make the engine run for this model. Found it here

• More importantly, when I perform a k6 load-test on the vllm embedding server, I found the server handle the incoming requests one after another without multiple requests being processed at the same time. I tried tuning the --max-num-seqs or --max-num-batched-tokens, it didn't help. The only thing helps but not by a lot is the --api-server-count argument. I would like to ask what is the correct configurations to handle X incoming requests at the same time for encoder based model like bge series?

For your information, in the k6 load-test, each virtual user will send a group of 10 requests, while each request has a batch of 32 sentences in the payload.

For X VUs, average request time, i.e. response time for embedding 32 sentences in a request:

• 1VU, avg req 373.67ms
• 3VU, avg req 450.71ms
• 5VU, avg req 507.25ms
• 8VU, avg req 519.45ms
• 10VU, avg req 584.41ms
• 15VU, avg req 761.26ms
• 20VU, avg req 945.33ms
• 25VU, avg req 1.15s
• 30VU, avg req 1.37s
• 40VU, avg req 1.79s
• 50VU, avg req 2.22s

I would expect it stays constant the speed for few concurrent requests and start to struggle when we put more requests.

Thanks in advance for your help : )

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

cc @maxdebayser

[ayushsatyam146]

@DarkLight1337 can I work on this? If possible please assign this to me, Thanks!

[DarkLight1337]

The enforce eager issue should be resolved already by #24872 on latest main and should be in the upcoming v0.11.0 release

[DarkLight1337]

I suspect changing batch size doesn't help much because chunked prefill is not available for MEAN pooling

[DarkLight1337]

@DarkLight1337 can I work on this? If possible please assign this to me, Thanks!

Feel free to help investigate while waiting for a response from @maxdebayser

[noooop]

https://github.com/noooop/snippet/blob/main/benchmarks/embed/v1-TokensPrompt.py

python -u -m benchmarks.embed.v1-TokensPrompt BAAI/bge-large-en-v1.5

• X-axis: Throughput (token/s)
• Y-axis: Latency, Time needed for one step (ms) <- logarithmic scale
• The curve lower right is better ↘

Offline API (llm.embed()) speed is reasonable.

---

The OpenAI API server may have performance bottlenecks.

[DarkLight1337]

Can you try the flag VLLM_ENABLE_V1_MULTIPROCESSING=0 to see if IPC is the issue?

[DarkLight1337]

Not sure whether setting this is better than increasing --api-server-count though (note: this arg currently only works for vllm serve, see #25309)

[maxdebayser]

If you print the number of requests being processed in the GPU model runner's execute_model model it is more than one, but it's true that there are big bottlenecks in the entrypoint layer. Especially if you send a request with multiple prompts, it can't take advantage of multiple API servers. Currently I'm also looking into these performance issues, but I think it's a good thing if we have more people investigating this. And please share your results in the v1-pooling channel in the vllm slack.

[gabinguo]

Thanks so much for the replies ❤️! It helped a lot in understanding the problem. Besides, do we have a workaround solution for this concurrency bottleneck?

[maxdebayser]

Not yet :/

[noooop]

Will #26414 help?