prefix caching for MM - #1039
Open
nikheal2 wants to merge 7 commits into
Open
Conversation
Signed-off-by: nikheal2 <suryawanshin74@gmail.com> changing logger from debug to info Signed-off-by: nikheal2 <suryawanshin74@gmail.com> changing logger to vllm logger Signed-off-by: nikheal2 <suryawanshin74@gmail.com> refcatoring cache count logic Signed-off-by: nikheal2 <suryawanshin74@gmail.com> fixing metrics Signed-off-by: nikheal2 <suryawanshin74@gmail.com> adding test cases Signed-off-by: nikheal2 <suryawanshin74@gmail.com> adding debug log Signed-off-by: nikheal2 <suryawanshin74@gmail.com> adding debug log Signed-off-by: nikheal2 <suryawanshin74@gmail.com> fix image identifier Signed-off-by: nikheal2 <suryawanshin74@gmail.com> fix logging Signed-off-by: nikheal2 <suryawanshin74@gmail.com>
|
👋 Hi! Thank you for contributing. We also recommend installing prek and configuring it to check your code before every local commit. |
Signed-off-by: nikheal2 <suryawanshin74@gmail.com>
gkumbhat
reviewed
Jul 29, 2026
| # by the inline fallback encode path (TP=1 / async encoder off / warmup). | ||
| # The async encoder subprocess has its own cache. Per-worker and | ||
| # deterministic across ranks (identical request stream), so no coordination. | ||
| self.mm_encoder_cache = MMEncoderCache( |
Collaborator
There was a problem hiding this comment.
This should only be used for inline, so should we put this initialization conditional on when this is going to be used ?
Signed-off-by: nikheal2 <suryawanshin74@gmail.com>
Signed-off-by: nikheal2 <suryawanshin74@gmail.com>
Signed-off-by: nikheal2 <suryawanshin74@gmail.com>
nikheal2
marked this pull request as ready for review
August 4, 2026 18:58
nikheal2
requested review from
prashantgupta24,
rafvasq,
sducouedic,
tdoublep and
yannicks1
as code owners
August 4, 2026 18:58
gkumbhat
reviewed
Aug 4, 2026
gkumbhat
reviewed
Aug 4, 2026
gkumbhat
reviewed
Aug 4, 2026
Signed-off-by: nikheal2 <suryawanshin74@gmail.com>
Signed-off-by: nikheal2 <suryawanshin74@gmail.com>
gkumbhat
approved these changes
Aug 6, 2026
gkumbhat
left a comment
gkumbhat
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. Thanks for adding this feature.. Will wait on some test results and will merge after that.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR enables caching vision encoder outputs keyed by image has (mm_hash)
Description
Related Issues
Test Plan
Checklist
bash format.sh)Signed-off-by:line (DCO compliance)