[Frontend][RFC] Rust front-end integration by njhill · Pull Request #40848 · vllm-project/vllm

njhill · 2026-04-24T23:53:16Z

See corresponding RFC for introducing a rust-based alternative front-end process in vLLM #40846.

For now we have staged the poc implementation in https://github.com/Inferact/vllm-frontend-rs. This PR contains the logic to integrate it into vLLM.

You can try it out by building this branch:

# Full install
uv pip install .

# Or use pre-compiled wheel
VLLM_USE_PRECOMPILED=1 uv pip install --editable .
./build_rust.sh

# Run it
VLLM_USE_RUST_FRONTEND=1 vllm serve ...

Co-authored with @BugenZhao

Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Bugen Zhao <i@bugenzhao.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>

mergify · 2026-04-24T23:53:57Z

Documentation preview: https://vllm--40848.org.readthedocs.build/en/40848/

gemini-code-assist

Code Review

This pull request introduces a Rust-based frontend to vLLM, adding a git submodule and updating the build system to support native extensions via setuptools-rust. It includes a new process manager for the Rust binary and environment variables for configuration. Feedback identifies a future-dated toolchain version, fragile environment variable parsing, and an inconsistent path to the Rust manifest file in the setup configuration.

gemini-code-assist · 2026-04-24T23:57:50Z

@@ -0,0 +1,2 @@
+[toolchain]
+channel = "nightly-2026-03-10"


The specified Rust toolchain date nightly-2026-03-10 appears to be in the future. This will cause build failures as the toolchain cannot be found. Please update it to a valid, existing nightly version.

BTW, why is the nightly toolchain required?

It's because we're heavily using coroutines which are still unstable (in grammar) features yet for async stream.

I'm concerned about the use of nightlies and experimental coroutine_trait language feature. It comes with a risk and will inflict pain on downstream rebuilds. I would prefer to have vLLM use a stable MSRV instead.

Regarding coroutine-style async streams, I think there are several alternatives that work under stable toolchain, at the cost of a bit performance overhead, which should be acceptable.

I would agree that there would be much less concern if we can find an alternative to get rid of unstable features and switch to a stable toolchain.

Thank you for considering to use Rust stable feature! It would help our downstream testing and rebuilds a lot.

Updates: I've replaced all (actually only a few) unstable language features used by the Rust frontend to stable alternative libraries / desugared syntax, so it now builds with stable toolchain!

refactor: replace futures-async-stream with stable alternative asynk-strim Inferact/vllm-frontend-rs#153

chore: switch to stable toolchain Inferact/vllm-frontend-rs#154

gemini-code-assist · 2026-04-24T23:57:50Z

+    When enabled, resolves VLLM_RUST_FRONTEND_PATH ("auto" by default)
+    to the actual binary path.
+    """
+    use_rust = bool(int(os.environ.get("VLLM_USE_RUST_FRONTEND", "0")))


The boolean conversion bool(int(os.environ.get("VLLM_USE_RUST_FRONTEND", "0"))) is fragile. It will raise a ValueError if the environment variable is set to common boolean strings like "true" or "false". Consider using a more robust check similar to how VLLM_USE_PRECOMPILED_RUST is handled at line 572.

Suggested change

use_rust = bool(int(os.environ.get("VLLM_USE_RUST_FRONTEND", "0")))

use_rust = os.environ.get("VLLM_USE_RUST_FRONTEND", "").strip().lower() in ("1", "true")

gemini-code-assist · 2026-04-24T23:57:51Z

+rust_extensions = [
+    RustExtension(
+        target="vllm.vllm-rs",
+        path="rust/src/cmd/Cargo.toml",


The path to the Cargo.toml manifest for the RustExtension is inconsistent with the build script. build_rust.sh uses the manifest at the root of the submodule (rust/Cargo.toml), while this points to a sub-directory. This may lead to build failures or missing workspace dependencies if the submodule is structured as a workspace.

Suggested change

path="rust/src/cmd/Cargo.toml",

path="rust/Cargo.toml",

IIRC we cannot pass a virtual (workspace) manifest here, so this comment seems invalid.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Signed-off-by: Nick Hill <nickhill123@gmail.com>

mgoin · 2026-04-25T10:11:34Z

@@ -0,0 +1,3 @@
+[submodule "rust"]
+	path = rust


Nit: what about "rsrc" to match "csrc"? Or just putting it as a subdir to avoid adding another top level dir, like "csrc/rust/"

Yeah we can decide on the best name. I considered this but it seemed to be non-standard. But now I realize csrc isn't really standard either so maybe you're right and rsrc would be better. I'm not keen on putting it under csrc.

Personally I don't find it a common convention to use rsrc for the directory name of rust sources, also putting it under csrc will be more confusing. I would still incline to rust for this.

Signed-off-by: Nick Hill <nickhill123@gmail.com>

chaunceyjiang · 2026-04-27T06:06:56Z

Should it be moved under the vLLM organization? Would it be donated from the Inferact organization to the vLLM organization? Also, the vllm-frontend-rs repository does not have a formal LICENSE file.

@chaunceyjiang yes we are just staging it in the Inferact org initially, where we had been experimenting.

It seems like the general preference (including mine) is to have the code live within a subdir of this main vllm repo (i.e. the submodule in this PR will contain the actual code that's currently in the vllm-frontend-rs repo). In which case there's no need to have a new repo under the vllm org anyhow.

Good point re the license, will add that asap.

Signed-off-by: Nick Hill <nickhill123@gmail.com>

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

Signed-off-by: Nick Hill <nickhill123@gmail.com>

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

yewentao256

Thanks for the work!
Do we have e2e benchmark showing how much perf we can get from this refactor?

Ideally for a large dense model and a large moe model

LucasWilkinson · 2026-04-29T18:18:09Z

+# checkout does not recurse submodules, and the Dockerfile only sees what's in
+# the build context, so initialize the submodule here before building.
+git submodule sync --recursive
+git submodule update --init --recursive


until this point we've tried to avoid using git submodules as the UX can be a bit rough; we've preferred to use use cmakes FetchContent. I think it would be worth looking into that I think. At the very least i guess its probably worth discussing the use of git submodules given this would be the first (im personally Im generally ok with it but I know there's varying opinions in the community)

cc @tlrmchlsmth

I do have pretty strong preferences against using submodules. At NeuralMagic in the DeepSparse days we found stale submodules to be a very annoying footgun.

If we're moving the code to in-tree it seems like we could avoid this. Or use FetchContent like Lucas suggests

@LucasWilkinson I also agree that we should avoid using a submodule. This was only done here to stage things initially so that we could keep/show the integration scaffolding in this PR and still allow folks to pull it and build/try it.

We still need to have some final decision on where the rust code should live, which I think so far is leaning towards living in a subdir of the main vllm repo since it's tightly coupled and considered primarily an "internal" component (see discussion related to this in #40846)

One thing that @BugenZhao is a concerned about is how we retain the commit history though if we do move the code here, since there's already a fair amount of them in https://github.com/Inferact/vllm-frontend-rs. There's quite a lot of code now and it is useful to be able to see the provenance of different parts of it.

If we do want to try to keep it, the commits can be recreated in this repo with updated paths, but it would mean having a whole bunch of new commits on main in one go, or possibly doing a merge commit (rather than squash-merge) which we haven't used in the past. Would welcome any opinions/thoughts about this.

# Conflicts: # docker/Dockerfile Signed-off-by: Bugen Zhao <i@bugenzhao.com>

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

[Frontend] Rust frontend integration

a0c27b9

Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Bugen Zhao <i@bugenzhao.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>

njhill mentioned this pull request Apr 24, 2026

[RFC]: Rust front-end #40846

Open

mergify Bot added documentation Improvements or additions to documentation ci/build frontend nvidia labels Apr 24, 2026

github-project-automation Bot added this to NVIDIA Apr 24, 2026

mergify Bot added cpu Related to CPU backends v1 labels Apr 24, 2026

gemini-code-assist Bot reviewed Apr 24, 2026

View reviewed changes

njhill marked this pull request as ready for review April 25, 2026 00:08

njhill requested review from DarkLight1337, bigPYJ1151, hmellor, mgoin, russellb and xuechendi as code owners April 25, 2026 00:08

claude Bot reviewed Apr 25, 2026

View reviewed changes

njhill added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 25, 2026

use https for submodule

0fbf081

Signed-off-by: Nick Hill <nickhill123@gmail.com>

mgoin reviewed Apr 25, 2026

View reviewed changes

njhill added 3 commits April 25, 2026 16:09

add rocm

5b28f17

Signed-off-by: Nick Hill <nickhill123@gmail.com>

fix shutdown test

23fba08

Signed-off-by: Nick Hill <nickhill123@gmail.com>

Merge remote-tracking branch 'origin/main' into rust_frontend

7dc64e0

njhill requested review from aarnphm, chaunceyjiang, gshtras and tjtanaa as code owners April 26, 2026 15:55

mergify Bot added the rocm Related to AMD ROCm label Apr 26, 2026

github-project-automation Bot moved this to Todo in AMD Apr 26, 2026

github-project-automation Bot added this to AMD Apr 26, 2026

chaunceyjiang reviewed Apr 27, 2026

View reviewed changes

njhill added 2 commits April 27, 2026 20:34

fix rocm build

dc31ff2

Signed-off-by: Nick Hill <nickhill123@gmail.com>

update submodule to latest

ea83543

Signed-off-by: Nick Hill <nickhill123@gmail.com>

njhill removed the ready ONLY add when PR is ready to merge/full CI is needed label Apr 28, 2026

try fix shutdown test

b1e87b8

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

BugenZhao requested review from NickLucche and robertgshaw2-redhat as code owners April 28, 2026 06:48

njhill added ready ONLY add when PR is ready to merge/full CI is needed and removed ready ONLY add when PR is ready to merge/full CI is needed labels Apr 28, 2026

njhill added 3 commits April 28, 2026 15:05

bump shutdown timeout some more

e688944

Signed-off-by: Nick Hill <nickhill123@gmail.com>

rocm build fd limit fix

ffe6b3f

Signed-off-by: Nick Hill <nickhill123@gmail.com>

Merge remote-tracking branch 'origin/main' into rust_frontend

903d79e

njhill added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 29, 2026

BugenZhao added 2 commits April 29, 2026 15:58

Skip flaky Rust frontend shutdown timeout test

fbe08cb

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

Merge branch 'main' into rust_frontend

09a074b

yewentao256 reviewed Apr 29, 2026

View reviewed changes

LucasWilkinson reviewed Apr 29, 2026

View reviewed changes

BugenZhao added 2 commits May 5, 2026 13:36

Merge remote-tracking branch 'upstream/main' into rust_frontend

7f227c5

# Conflicts: # docker/Dockerfile Signed-off-by: Bugen Zhao <i@bugenzhao.com>

Bump Rust frontend submodule

ac9e1a1

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

	use_rust = bool(int(os.environ.get("VLLM_USE_RUST_FRONTEND", "0")))
	use_rust = os.environ.get("VLLM_USE_RUST_FRONTEND", "").strip().lower() in ("1", "true")

Uh oh!

Conversation

njhill commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented Apr 24, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yewentao256 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

njhill commented Apr 24, 2026 •

edited

Loading

yewentao256 left a comment •

edited

Loading