MI100 Users: This fork adds support for AMD Instinct MI100 (gfx908) GPUs. For full setup instructions -- BIOS, drivers, ROCm 7.12, native build, and launch config -- see MI100_SETUP.md. For performance benchmarks and optimization results, see BENCH.md.

---

Easy, fast, and cheap LLM serving for everyone

| Documentation | Blog | Paper | Twitter/X | User Forum | Developer Slack |

---

What This Fork Adds

This is a fork of vllm-project/vllm (v0.18.0) with patches for MI100 (gfx908):

• FP8 emulation kernel (MI100FP8ScaledMMLinearKernel) -- dequantizes FP8 to FP16 and uses rocBLAS, since MI100 lacks native FP8 hardware
• MI100 platform detection -- on_mi100(), _ON_MI100, gfx908 added to _ON_GFX9 family
• Pixtral chunked attention -- memory-efficient attention for vision transformer
• Custom all-reduce for gfx908 -- quickreduce XGMI-aware all-reduce now enabled for MI100 multi-GPU TP
• C++ paged attention for gfx908 -- hand-optimized HIP paged attention kernel with MFMA, BF16 uses FP16 MFMA fallback
• INT4 quantization -- AWQ (Triton) and GPTQ (Exllama) confirmed working on gfx908 with --dtype float16

Quick Start

git clone https://github.com/larkinwc/vllm-gfx908.git
cd vllm-gfx908 && git checkout mi100-fixes

# See MI100_SETUP.md for full instructions

Upstream vLLM

Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.

vLLM is fast with:

• State-of-the-art serving throughput
• Efficient management of attention key and value memory with PagedAttention
• Continuous batching of incoming requests, chunked prefill, prefix caching
• Fast and flexible model execution with piecewise and full CUDA/HIP graphs
• Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and more
• Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton
• Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL
• Speculative decoding including n-gram, suffix, EAGLE, DFlash
• Automatic kernel generation and graph-level transformations using torch.compile
• Disaggregated prefill, decode, and encode

vLLM is flexible and easy to use with:

• Seamless integration with popular Hugging Face models
• High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more
• Tensor, pipeline, data, expert, and context parallelism for distributed inference
• Streaming outputs
• Generation of structured outputs using xgrammar or guidance
• Tool calling and reasoning parsers
• OpenAI-compatible API server, plus Anthropic Messages API and gRPC support
• Efficient multi-LoRA support for dense and MoE layers
• Support for NVIDIA GPUs, AMD GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, and more.

vLLM seamlessly supports 200+ model architectures on Hugging Face, including:

• Decoder-only LLMs (e.g., Llama, Qwen, Gemma)
• Mixture-of-Expert LLMs (e.g., Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS)
• Hybrid attention and state-space models (e.g., Mamba, Qwen3.5)
• Multi-modal models (e.g., LLaVA, Qwen-VL, Pixtral)
• Embedding and retrieval models (e.g., E5-Mistral, GTE, ColBERT)
• Reward and classification models (e.g., Qwen-Math)

Find the full list of supported models here.

Getting Started

Install vLLM with uv (recommended) or pip:

uv pip install vllm

Or build from source for development.

Visit our documentation to learn more.

For the original project, see github.com/vllm-project/vllm.