Skip to content

Conversation

DarkLight1337
Copy link
Member

@DarkLight1337 DarkLight1337 commented Sep 20, 2025

Purpose

I found that API server headless mode and scale-out is silently ignored using the Dockerfile's entrypoint (python -m vllm.entrypoints.openai.api_server) because the logic for applying API server scale-out is only run inside the CLI file. So, I have moved the relevant logic to the main api_server.py to make them available to both entrypoints.

cc @njhill

Test Plan

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the API server startup logic to fix an issue where scaling out the API server did not work with the Dockerfile entrypoint. The core logic for handling different server modes (single, multi-process, headless) has been moved from the CLI-specific file vllm/entrypoints/cli/serve.py to the more general vllm/entrypoints/openai/api_server.py. This allows both the CLI entrypoint and the direct module entrypoint (python -m ...) to use the same startup logic. The changes are well-structured, and the refactoring is clean. The related test and comments have also been updated accordingly. The fix correctly addresses the described problem.

@DarkLight1337 DarkLight1337 changed the title [Bugfix] Enable API server scale-out using Dockerfile entrypoint [Bugfix] Enable API server headless and scale-out using Dockerfile entrypoint Sep 20, 2025
@DarkLight1337 DarkLight1337 changed the title [Bugfix] Enable API server headless and scale-out using Dockerfile entrypoint [Bugfix] Enable API server headless mode and scale-out using Dockerfile entrypoint Sep 20, 2025
@DarkLight1337
Copy link
Member Author

Closing as superseded by #25967

@DarkLight1337 DarkLight1337 deleted the python-m-api-scaleout branch October 1, 2025 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
frontend ready ONLY add when PR is ready to merge/full CI is needed v1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant