(EngineCore_DP0 pid=1132) DEBUG 04-10 10:19:20 [compilation/decorators.py:194] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.llama.LlamaModel'>: ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds']
(EngineCore_DP1 pid=1423) DEBUG 04-10 10:19:20 [compilation/decorators.py:194] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.llama.LlamaModel'>: ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds']
(EngineCore_DP3 pid=1158) DEBUG 04-10 10:19:22 [distributed/parallel_state.py:1161] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.42.1.232:45821 backend=nccl
(EngineCore_DP3 pid=1158) DEBUG 04-10 10:19:22 [distributed/parallel_state.py:1193] Adjusting world_size=4 rank=3 distributed_init_method=tcp://0.0.0.0:40309 for DP
(EngineCore_DP3 pid=1158) INFO 04-10 10:19:22 [distributed/parallel_state.py:1200] world_size=4 rank=3 local_rank=0 distributed_init_method=tcp://0.0.0.0:40309 backend=nccl
(EngineCore_DP0 pid=1132) DEBUG 04-10 10:19:22 [distributed/parallel_state.py:1161] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.42.1.232:45097 backend=nccl
(EngineCore_DP0 pid=1132) DEBUG 04-10 10:19:22 [distributed/parallel_state.py:1193] Adjusting world_size=4 rank=0 distributed_init_method=tcp://0.0.0.0:40309 for DP
(EngineCore_DP0 pid=1132) INFO 04-10 10:19:22 [distributed/parallel_state.py:1200] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://0.0.0.0:40309 backend=nccl
(EngineCore_DP1 pid=1423) DEBUG 04-10 10:19:22 [distributed/parallel_state.py:1161] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.42.1.232:38833 backend=nccl
(EngineCore_DP1 pid=1423) DEBUG 04-10 10:19:22 [distributed/parallel_state.py:1193] Adjusting world_size=4 rank=1 distributed_init_method=tcp://0.0.0.0:40309 for DP
(EngineCore_DP1 pid=1423) INFO 04-10 10:19:22 [distributed/parallel_state.py:1200] world_size=4 rank=1 local_rank=0 distributed_init_method=tcp://0.0.0.0:40309 backend=nccl
(EngineCore_DP2 pid=1145) DEBUG 04-10 10:19:22 [distributed/parallel_state.py:1161] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.42.1.232:52089 backend=nccl
(EngineCore_DP2 pid=1145) DEBUG 04-10 10:19:22 [distributed/parallel_state.py:1193] Adjusting world_size=4 rank=2 distributed_init_method=tcp://0.0.0.0:40309 for DP
(EngineCore_DP2 pid=1145) INFO 04-10 10:19:22 [distributed/parallel_state.py:1200] world_size=4 rank=2 local_rank=0 distributed_init_method=tcp://0.0.0.0:40309 backend=nccl
[Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
(EngineCore_DP0 pid=1132) DEBUG 04-10 10:19:22 [distributed/parallel_state.py:1244] Detected 1 nodes in the distributed environment
(EngineCore_DP1 pid=1423) DEBUG 04-10 10:19:22 [distributed/parallel_state.py:1244] Detected 1 nodes in the distributed environment
(EngineCore_DP2 pid=1145) DEBUG 04-10 10:19:22 [distributed/parallel_state.py:1244] Detected 1 nodes in the distributed environment
(EngineCore_DP3 pid=1158) DEBUG 04-10 10:19:22 [distributed/parallel_state.py:1244] Detected 1 nodes in the distributed environment
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
(EngineCore_DP0 pid=1132) DEBUG 04-10 10:19:22 [utils/nccl.py:34] Found nccl from library libnccl.so.2
(EngineCore_DP1 pid=1423) DEBUG 04-10 10:19:22 [utils/nccl.py:34] Found nccl from library libnccl.so.2
(EngineCore_DP0 pid=1132) INFO 04-10 10:19:22 [distributed/device_communicators/pynccl.py:111] vLLM is using nccl==2.27.5
(EngineCore_DP3 pid=1158) DEBUG 04-10 10:19:22 [utils/nccl.py:34] Found nccl from library libnccl.so.2
(EngineCore_DP2 pid=1145) DEBUG 04-10 10:19:22 [utils/nccl.py:34] Found nccl from library libnccl.so.2
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] EngineCore failed to start.
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] Traceback (most recent call last):
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 995, in run_engine_core
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1406, in __init__
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] super().__init__(
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in __init__
(EngineCore_DP3 pid=1158) Process EngineCore_DP3:
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] super().__init__(
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 132, in __init__
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.model_executor = executor_class(vllm_config)
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self._init_executor()
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.driver_worker.init_device()
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.worker.init_device() # type: ignore
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 216, in init_device
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] init_worker_distributed_environment(
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 915, in init_worker_distributed_environment
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ensure_model_parallel_initialized(
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1435, in ensure_model_parallel_initialized
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] initialize_model_parallel(
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1387, in initialize_model_parallel
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] _DP = init_model_parallel_group(
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1062, in init_model_parallel_group
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] return GroupCoordinator(
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 364, in __init__
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.device_communicator = device_comm_cls(
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 58, in __init__
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.pynccl_comm = PyNcclCommunicator(
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 146, in __init__
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.all_reduce(data)
(EngineCore_DP3 pid=1158) Traceback (most recent call last):
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 172, in all_reduce
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.nccl.ncclAllReduce(
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 422, in ncclAllReduce
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.NCCL_CHECK(
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 366, in NCCL_CHECK
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] raise RuntimeError(f"NCCL error: {error_str}")
(EngineCore_DP3 pid=1158) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
(EngineCore_DP3 pid=1158) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP3 pid=1158) self.run()
(EngineCore_DP3 pid=1158) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP3 pid=1158) self._target(*self._args, **self._kwargs)
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1012, in run_engine_core
(EngineCore_DP3 pid=1158) raise e
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 995, in run_engine_core
(EngineCore_DP3 pid=1158) engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP3 pid=1158) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1406, in __init__
(EngineCore_DP3 pid=1158) super().__init__(
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in __init__
(EngineCore_DP3 pid=1158) super().__init__(
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 132, in __init__
(EngineCore_DP3 pid=1158) self.model_executor = executor_class(vllm_config)
(EngineCore_DP3 pid=1158) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP3 pid=1158) self._init_executor()
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP3 pid=1158) self.driver_worker.init_device()
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP3 pid=1158) self.worker.init_device() # type: ignore
(EngineCore_DP3 pid=1158) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 216, in init_device
(EngineCore_DP3 pid=1158) init_worker_distributed_environment(
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 915, in init_worker_distributed_environment
(EngineCore_DP3 pid=1158) ensure_model_parallel_initialized(
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1435, in ensure_model_parallel_initialized
(EngineCore_DP3 pid=1158) initialize_model_parallel(
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1387, in initialize_model_parallel
(EngineCore_DP3 pid=1158) _DP = init_model_parallel_group(
(EngineCore_DP3 pid=1158) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1062, in init_model_parallel_group
(EngineCore_DP3 pid=1158) return GroupCoordinator(
(EngineCore_DP3 pid=1158) ^^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 364, in __init__
(EngineCore_DP3 pid=1158) self.device_communicator = device_comm_cls(
(EngineCore_DP3 pid=1158) ^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 58, in __init__
(EngineCore_DP3 pid=1158) self.pynccl_comm = PyNcclCommunicator(
(EngineCore_DP3 pid=1158) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 146, in __init__
(EngineCore_DP3 pid=1158) self.all_reduce(data)
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 172, in all_reduce
(EngineCore_DP3 pid=1158) self.nccl.ncclAllReduce(
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 422, in ncclAllReduce
(EngineCore_DP3 pid=1158) self.NCCL_CHECK(
(EngineCore_DP3 pid=1158) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 366, in NCCL_CHECK
(EngineCore_DP3 pid=1158) raise RuntimeError(f"NCCL error: {error_str}")
(EngineCore_DP3 pid=1158) RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] EngineCore failed to start.
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] Traceback (most recent call last):
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 995, in run_engine_core
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1406, in __init__
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] super().__init__(
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in __init__
(EngineCore_DP0 pid=1132) Process EngineCore_DP0:
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] super().__init__(
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 132, in __init__
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self._init_executor()
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.driver_worker.init_device()
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.worker.init_device() # type: ignore
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 216, in init_device
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] init_worker_distributed_environment(
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 915, in init_worker_distributed_environment
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ensure_model_parallel_initialized(
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1435, in ensure_model_parallel_initialized
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] initialize_model_parallel(
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1387, in initialize_model_parallel
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] _DP = init_model_parallel_group(
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1062, in init_model_parallel_group
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] return GroupCoordinator(
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 364, in __init__
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.device_communicator = device_comm_cls(
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 58, in __init__
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.pynccl_comm = PyNcclCommunicator(
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 146, in __init__
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.all_reduce(data)
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 172, in all_reduce
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.nccl.ncclAllReduce(
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 422, in ncclAllReduce
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.NCCL_CHECK(
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 366, in NCCL_CHECK
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] raise RuntimeError(f"NCCL error: {error_str}")
(EngineCore_DP0 pid=1132) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
(EngineCore_DP0 pid=1132) Traceback (most recent call last):
(EngineCore_DP0 pid=1132) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=1132) self.run()
(EngineCore_DP0 pid=1132) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=1132) self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1012, in run_engine_core
(EngineCore_DP0 pid=1132) raise e
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 995, in run_engine_core
(EngineCore_DP0 pid=1132) engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=1132) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1406, in __init__
(EngineCore_DP0 pid=1132) super().__init__(
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in __init__
(EngineCore_DP0 pid=1132) super().__init__(
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 132, in __init__
(EngineCore_DP0 pid=1132) self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=1132) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=1132) self._init_executor()
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=1132) self.driver_worker.init_device()
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=1132) self.worker.init_device() # type: ignore
(EngineCore_DP0 pid=1132) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 216, in init_device
(EngineCore_DP0 pid=1132) init_worker_distributed_environment(
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 915, in init_worker_distributed_environment
(EngineCore_DP0 pid=1132) ensure_model_parallel_initialized(
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1435, in ensure_model_parallel_initialized
(EngineCore_DP0 pid=1132) initialize_model_parallel(
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1387, in initialize_model_parallel
(EngineCore_DP0 pid=1132) _DP = init_model_parallel_group(
(EngineCore_DP0 pid=1132) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1062, in init_model_parallel_group
(EngineCore_DP0 pid=1132) return GroupCoordinator(
(EngineCore_DP0 pid=1132) ^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 364, in __init__
(EngineCore_DP0 pid=1132) self.device_communicator = device_comm_cls(
(EngineCore_DP0 pid=1132) ^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 58, in __init__
(EngineCore_DP0 pid=1132) self.pynccl_comm = PyNcclCommunicator(
(EngineCore_DP0 pid=1132) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 146, in __init__
(EngineCore_DP0 pid=1132) self.all_reduce(data)
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 172, in all_reduce
(EngineCore_DP0 pid=1132) self.nccl.ncclAllReduce(
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 422, in ncclAllReduce
(EngineCore_DP0 pid=1132) self.NCCL_CHECK(
(EngineCore_DP0 pid=1132) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 366, in NCCL_CHECK
(EngineCore_DP0 pid=1132) raise RuntimeError(f"NCCL error: {error_str}")
(EngineCore_DP0 pid=1132) RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] EngineCore failed to start.
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] Traceback (most recent call last):
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 995, in run_engine_core
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1406, in __init__
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] super().__init__(
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in __init__
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] super().__init__(
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 132, in __init__
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.model_executor = executor_class(vllm_config)
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self._init_executor()
(EngineCore_DP2 pid=1145) Process EngineCore_DP2:
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.driver_worker.init_device()
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.worker.init_device() # type: ignore
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 216, in init_device
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] init_worker_distributed_environment(
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 915, in init_worker_distributed_environment
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ensure_model_parallel_initialized(
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1435, in ensure_model_parallel_initialized
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] initialize_model_parallel(
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1387, in initialize_model_parallel
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] _DP = init_model_parallel_group(
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1062, in init_model_parallel_group
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] return GroupCoordinator(
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 364, in __init__
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.device_communicator = device_comm_cls(
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 58, in __init__
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.pynccl_comm = PyNcclCommunicator(
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 146, in __init__
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.all_reduce(data)
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 172, in all_reduce
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.nccl.ncclAllReduce(
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 422, in ncclAllReduce
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.NCCL_CHECK(
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 366, in NCCL_CHECK
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] raise RuntimeError(f"NCCL error: {error_str}")
(EngineCore_DP2 pid=1145) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
(EngineCore_DP2 pid=1145) Traceback (most recent call last):
(EngineCore_DP2 pid=1145) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP2 pid=1145) self.run()
(EngineCore_DP2 pid=1145) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP2 pid=1145) self._target(*self._args, **self._kwargs)
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1012, in run_engine_core
(EngineCore_DP2 pid=1145) raise e
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 995, in run_engine_core
(EngineCore_DP2 pid=1145) engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP2 pid=1145) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1406, in __init__
(EngineCore_DP2 pid=1145) super().__init__(
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in __init__
(EngineCore_DP2 pid=1145) super().__init__(
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 132, in __init__
(EngineCore_DP2 pid=1145) self.model_executor = executor_class(vllm_config)
(EngineCore_DP2 pid=1145) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP2 pid=1145) self._init_executor()
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP2 pid=1145) self.driver_worker.init_device()
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP2 pid=1145) self.worker.init_device() # type: ignore
(EngineCore_DP2 pid=1145) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 216, in init_device
(EngineCore_DP2 pid=1145) init_worker_distributed_environment(
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 915, in init_worker_distributed_environment
(EngineCore_DP2 pid=1145) ensure_model_parallel_initialized(
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1435, in ensure_model_parallel_initialized
(EngineCore_DP2 pid=1145) initialize_model_parallel(
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1387, in initialize_model_parallel
(EngineCore_DP2 pid=1145) _DP = init_model_parallel_group(
(EngineCore_DP2 pid=1145) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1062, in init_model_parallel_group
(EngineCore_DP2 pid=1145) return GroupCoordinator(
(EngineCore_DP2 pid=1145) ^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 364, in __init__
(EngineCore_DP2 pid=1145) self.device_communicator = device_comm_cls(
(EngineCore_DP2 pid=1145) ^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 58, in __init__
(EngineCore_DP2 pid=1145) self.pynccl_comm = PyNcclCommunicator(
(EngineCore_DP2 pid=1145) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 146, in __init__
(EngineCore_DP2 pid=1145) self.all_reduce(data)
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 172, in all_reduce
(EngineCore_DP2 pid=1145) self.nccl.ncclAllReduce(
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 422, in ncclAllReduce
(EngineCore_DP2 pid=1145) self.NCCL_CHECK(
(EngineCore_DP2 pid=1145) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 366, in NCCL_CHECK
(EngineCore_DP2 pid=1145) raise RuntimeError(f"NCCL error: {error_str}")
(EngineCore_DP2 pid=1145) RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] EngineCore failed to start.
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] Traceback (most recent call last):
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 995, in run_engine_core
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) Process EngineCore_DP1:
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1406, in __init__
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] super().__init__(
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in __init__
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] super().__init__(
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 132, in __init__
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.model_executor = executor_class(vllm_config)
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self._init_executor()
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.driver_worker.init_device()
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.worker.init_device() # type: ignore
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 216, in init_device
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] init_worker_distributed_environment(
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 915, in init_worker_distributed_environment
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ensure_model_parallel_initialized(
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1435, in ensure_model_parallel_initialized
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] initialize_model_parallel(
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1387, in initialize_model_parallel
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] _DP = init_model_parallel_group(
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1062, in init_model_parallel_group
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] return GroupCoordinator(
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 364, in __init__
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.device_communicator = device_comm_cls(
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 58, in __init__
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.pynccl_comm = PyNcclCommunicator(
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 146, in __init__
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.all_reduce(data)
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 172, in all_reduce
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.nccl.ncclAllReduce(
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 422, in ncclAllReduce
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] self.NCCL_CHECK(
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 366, in NCCL_CHECK
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] raise RuntimeError(f"NCCL error: {error_str}")
(EngineCore_DP1 pid=1423) ERROR 04-10 10:19:23 [v1/engine/core.py:1008] RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
(EngineCore_DP1 pid=1423) Traceback (most recent call last):
(EngineCore_DP1 pid=1423) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP1 pid=1423) self.run()
(EngineCore_DP1 pid=1423) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP1 pid=1423) self._target(*self._args, **self._kwargs)
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1012, in run_engine_core
(EngineCore_DP1 pid=1423) raise e
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 995, in run_engine_core
(EngineCore_DP1 pid=1423) engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP1 pid=1423) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1406, in __init__
(EngineCore_DP1 pid=1423) super().__init__(
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in __init__
(EngineCore_DP1 pid=1423) super().__init__(
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 132, in __init__
(EngineCore_DP1 pid=1423) self.model_executor = executor_class(vllm_config)
(EngineCore_DP1 pid=1423) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP1 pid=1423) self._init_executor()
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP1 pid=1423) self.driver_worker.init_device()
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP1 pid=1423) self.worker.init_device() # type: ignore
(EngineCore_DP1 pid=1423) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 216, in init_device
(EngineCore_DP1 pid=1423) init_worker_distributed_environment(
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 915, in init_worker_distributed_environment
(EngineCore_DP1 pid=1423) ensure_model_parallel_initialized(
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1435, in ensure_model_parallel_initialized
(EngineCore_DP1 pid=1423) initialize_model_parallel(
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1387, in initialize_model_parallel
(EngineCore_DP1 pid=1423) _DP = init_model_parallel_group(
(EngineCore_DP1 pid=1423) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1062, in init_model_parallel_group
(EngineCore_DP1 pid=1423) return GroupCoordinator(
(EngineCore_DP1 pid=1423) ^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 364, in __init__
(EngineCore_DP1 pid=1423) self.device_communicator = device_comm_cls(
(EngineCore_DP1 pid=1423) ^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 58, in __init__
(EngineCore_DP1 pid=1423) self.pynccl_comm = PyNcclCommunicator(
(EngineCore_DP1 pid=1423) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 146, in __init__
(EngineCore_DP1 pid=1423) self.all_reduce(data)
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 172, in all_reduce
(EngineCore_DP1 pid=1423) self.nccl.ncclAllReduce(
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 422, in ncclAllReduce
(EngineCore_DP1 pid=1423) self.NCCL_CHECK(
(EngineCore_DP1 pid=1423) File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 366, in NCCL_CHECK
(EngineCore_DP1 pid=1423) raise RuntimeError(f"NCCL error: {error_str}")
(EngineCore_DP1 pid=1423) RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
[rank3]:[W410 10:19:23.767739036 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[rank0]:[W410 10:19:23.844798484 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[rank1]:[W410 10:19:23.863087788 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[rank2]:[W410 10:19:23.900860958 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
