Skip to content

Conversation

bwasti
Copy link
Contributor

@bwasti bwasti commented Sep 30, 2025

Following #25769, This change adds additional unit tests for the batch invariant kernel-overrides.

Purpose

We want more rigorous testing across kernels to ensure batch invariance. This PR lays out a test set to that end.

Test Plan

VLLM_KERNEL_OVERRIDE_BATCH_INVARIANT=1 pytest -s -v tests/v1/generation/batch_invariance/test_multi_gpu_ops.py

Test Result

Pass


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@bwasti bwasti requested a review from mgoin as a code owner September 30, 2025 18:33
@mergify mergify bot added the v1 label Sep 30, 2025
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request significantly enhances the testing for batch invariance, particularly in multi-GPU scenarios, which is crucial for ensuring deterministic and correct model outputs. The introduction of test_multi_gpu_ops.py with its "needle-in-a-haystack" methodology provides a robust framework for verifying the behavior of parallel operators. The updates to existing tests to include randomized batch sizes and prompt lengths are also a welcome improvement. The necessary changes to the FlashInfer backend to support this batch-invariant mode appear correct and are well-contained. I have a couple of suggestions to improve code clarity and correctness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

1 participant