Can I specify cuda 11.8 when I build and install vLLM from source?

[crazybullet28]

I would like to run the Phi-2 model on cuda 11.8.
I tried the current v0.2.4 version, but it would have error 'AttributeError: 'PhiConfig' object has no attribute 'layer_norm_epsilon'. I searched issues and found that recently there are some updates for Phi-2 to fix this error. The update is currently only available from source.

[chenxu2048]

Follow the instructions in the build from source section of the documentation, and check that the vllm version ends with -cu118.

Note that nvcc must be version 11.8, and PyTorch must also be built with CUDA 11.8. Check it with:

nvcc -V
python -c 'print(__import__("torch").version.cuda)'

[crazybullet28]

Hi @chenxu2048 , thanks for your reply. I'm still a little bit confusing. Do you mean that when I run the command pip install -e ., I should pass some arguments such as -cu118? As in the instructions, it only shows install the vLLM released version with cu118, but even the latest release version v2.7.0 still having the problem I've mentioned above.
And according to the ticket #2428 , there are some updates to fix the Phi-2 issue have been merged into main branch. That's why I want to install from the source, and specify Cuda 11.8.
Do you have any suggestions? I've already ensured that my nvcc and PyTorch version are build with Cuda 11.8.

[chenxu2048]

Do you mean that when I run the command pip install -e ., I should pass some arguments such as -cu118?

CUDA version is detected in setup.py script.

def get_nvcc_cuda_version(cuda_dir: str) -> Version:
    """Get the CUDA version from nvcc.

    Adapted from https://github.com/NVIDIA/apex/blob/8b7a1ff183741dd8f9b87e7bafd04cfde99cea28/setup.py
    """
    nvcc_output = subprocess.check_output([cuda_dir + "/bin/nvcc", "-V"],
                                          universal_newlines=True)
    output = nvcc_output.split()
    release_idx = output.index("release") + 1
    nvcc_cuda_version = parse(output[release_idx].split(",")[0])
    return nvcc_cuda_version

As in the instructions, it only shows install the vLLM released version with cu118.

setup.py reads the version in vllm/__init__.py and adds a suffix like +cu118, when CUDA is not 12.1. Command pip install -e . generates a version like v0.2.4+cu118 or v0.2.7+cu118, but the code is different with that in pip or github release.

def get_vllm_version() -> str:
    version = find_version(get_path("vllm", "__init__.py"))

    if _is_hip():
        # Get the HIP version
        hipcc_version = get_hipcc_rocm_version()
        if hipcc_version != MAIN_CUDA_VERSION:
            rocm_version_str = hipcc_version.replace(".", "")[:3]
            version += f"+rocm{rocm_version_str}"
    elif _is_neuron():
        # Get the Neuron version
        neuron_version = str(neuronxcc_version)
        if neuron_version != MAIN_CUDA_VERSION:
            neuron_version_str = neuron_version.replace(".", "")[:3]
            version += f"+neuron{neuron_version_str}"
    else:
        cuda_version = str(nvcc_cuda_version)
        if cuda_version != MAIN_CUDA_VERSION:
            cuda_version_str = cuda_version.replace(".", "")[:3]
            version += f"+cu{cuda_version_str}"

    return version

When you run pip without -e, vLLM would be copied into site-packages folder. If you want to modify source code after installation, run pip with -e/--editable. Here is the documentation of -e.