[VL] Add timestamp to shuffle test#12503
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates Velox shuffle writer tests to include a Timestamp column (with nanoseconds set to 0 to match Spark’s microsecond precision) and tightens shuffle writer correctness by enforcing that all input RowVectors written through a writer instance have an equivalent RowType.
Changes:
- Add
facebook::velox::Timestampcolumns to the shared shuffle test input vectors and introduce shared “has-null” variants in the test base. - Reuse shared “has-null” inputs in CPU shuffle writer tests instead of constructing ad-hoc vectors in the test body.
- Add
RowTypeequivalence checks in shuffle writers (and centralizerowType_inVeloxShuffleWriter) to fail fast on schema mismatches across writes.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| cpp/velox/tests/VeloxShuffleWriterTestBase.h | Adds timestamp test data and shared “has-null” input row vectors to be reused across tests. |
| cpp/velox/tests/VeloxShuffleWriterTest.cc | Switches CPU tests to reuse shared “has-null” inputs from the test base. |
| cpp/velox/tests/VeloxGpuShuffleWriterTest.cc | Adds timestamp column to locally constructed GPU test inputs (but currently still builds schema-mismatched vectors). |
| cpp/velox/shuffle/VeloxSortShuffleWriter.h | Removes per-writer rowType_ member in favor of the base class field. |
| cpp/velox/shuffle/VeloxSortShuffleWriter.cc | Initializes rowType_ via rv->rowType() and enforces row-type equivalence across writes. |
| cpp/velox/shuffle/VeloxShuffleWriter.h | Adds shared rowType_ storage to the base shuffle writer class. |
| cpp/velox/shuffle/VeloxRssSortShuffleWriter.h | Removes duplicate rowType_ member (now inherited from base). |
| cpp/velox/shuffle/VeloxRssSortShuffleWriter.cc | Adds row-type equivalence checks on subsequent RowVector inputs. |
| cpp/velox/shuffle/VeloxHashShuffleWriter.cc | Stores the writer’s rowType_ and enforces equivalence for subsequent inputs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
zhztheplayer
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Timestamp type is missing in the shuffle test. This patch adds timestamp type in the test data. Note that because spark only supports microsecond precision, the
nanosecondfield should be set to 0, otherwise the sort shuffle tests will fail.Also add type check for the shuffle inputs.
Verified
velox_gpu_shuffle_writer_testlocally.