Fix: Resolve critical bugs in vLLM Online and Offline inference #209
Conversation
Fix: Resolved critical bugs in vLLM Online and Offline inference. Online Inference: Updated installation instructions in the requirements and README. The previous setup referenced an outdated prebuilt vLLM version. The new stable release changed the installation method, which is now correctly documented. Offline Inference: Fixed a breaking change in llm.generate(<parameters>) due to deprecation in vLLM ≥ 0.10.2. Replaced the old input handling with the new TokensPrompt interface (from vllm.inputs import TokensPrompt) to ensure compatibility with the latest vLLM API.
|
@dkundel-openai Could you please look into this, it took quite a time and research to solve this bug. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
| outputs = llm.generate( | ||
| prompt_token_ids=[prefill_ids], # batch of size 1 | ||
| [TokensPrompt(prompt_token_ids=prefill_ids)], | ||
| sampling_params=sampling, |
There was a problem hiding this comment.
The offline inference example now calls llm.generate([TokensPrompt(prompt_token_ids=prefill_ids)], …) but the snippet still only imports LLM and SamplingParams. Because TokensPrompt is not imported (e.g., from vllm.inputs import TokensPrompt), anyone copying this updated code will hit a NameError before reaching generation. Consider adding the import alongside the other vLLM imports so the example runs as advertised.
Useful? React with 👍 / 👎.
hrithiksagar
changed the title
1 participant
Fix: Resolve critical bugs in vLLM Online and Offline inference
Online Inference:
Updated the installation instructions in
requirementsandREADME. The previous setup referenced an outdated prebuilt vLLM version. The new stable release changed the installation method, which is now correctly documented.Offline Inference:
Addressed a breaking change in
llm.generate(<parameters>)due to deprecation in vLLM ≥ 0.10.2.Replaced the old input handling with the new
TokensPromptinterface (from vllm.inputs import TokensPrompt) to ensure compatibility with the latest vLLM API.