[Feat.] Refactor llm_inference/run.py to use ParallelInferenceManager with batch inference#59
Merged
Merged
Conversation
yl231
commented
Jan 16, 2026
yl231
commented
Contributor
- Replace sequential processing with parallel batch inference infrastructure
- Maintain compatibility with origin/main: same CLI interface and functionality
- Add --num-runs parameter support (default: 1) to match batch_inference.py
- Use ParallelInferenceManager for efficient parallel processing with workers
- Add README_PARALLEL.md documentation
- Replace sequential processing with parallel batch inference infrastructure - Maintain compatibility with origin/main: same CLI interface and functionality - Add --num-runs parameter support (default: 1) to match batch_inference.py - Use ParallelInferenceManager for efficient parallel processing with workers - Fix mypy type annotations: Tuple return types and Optional parameters - Add README_PARALLEL.md documentation
Contributor
Author
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a significant and well-executed refactoring of the LLM inference process, replacing the previous sequential logic with a robust parallel batch inference system. The new ParallelInferenceManager and batch_inference.py script are excellent additions that will greatly improve performance and maintainability. The changes are well-structured, and the new documentation is very helpful. My review includes suggestions to enhance cross-platform compatibility, simplify some of the concurrency logic for better clarity, and improve error handling and efficiency in a few areas.
- Replace sequential processing with parallel batch inference infrastructure - Maintain compatibility with origin/main: same CLI interface and functionality - Add --num-runs parameter support (default: 1) to match batch_inference.py - Use ParallelInferenceManager for efficient parallel processing with workers - Fix mypy type annotations: Tuple return types and Optional parameters - Add README_PARALLEL.md documentation Code quality improvements: - Replace fcntl with cross-platform filelock for Windows compatibility - Use threading.local() to cache ModelInference instances per thread - Remove redundant periodic cache consolidation (save_single_result handles it) - Remove os.chdir() in batch_inference.py, use absolute paths instead - Add logging for exceptions when converting model names (no silent failures) - Document clear_failed_entries as utility function (not auto-called)
- Call clear_failed_entries automatically before processing each model - Clear failed entries before processing so they can be retried - Update function documentation to reflect automatic usage - Log number of failed entries cleared for visibility
jiarong0907
approved these changes
Jan 16, 2026
jiarong0907
left a comment
jiarong0907
left a comment
Contributor
There was a problem hiding this comment.
lgtm. One comment: You can directly call it README since it appears in a different folder.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.