Skip to content

[VL] Fix overflow of pageNumber in VeloxSortShuffleWriter #11101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
FelixYBW merged 2 commits into from
Nov 19, 2025
Merged

[VL] Fix overflow of pageNumber in VeloxSortShuffleWriter #11101

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1fe33eb
[VL] Fix overflow of pageNumber in VeloxSortShuffleWriter
zhli1142015 Nov 17, 2025
40a9147
address comments
zhli1142015 Nov 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion cpp/velox/shuffle/VeloxSortShuffleWriter.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ constexpr uint32_t kMaskLower27Bits = (1 << 27) - 1;
constexpr uint64_t kMaskLower40Bits = (1UL << 40) - 1;
constexpr uint32_t kPartitionIdStartByteIndex = 5;
constexpr uint32_t kPartitionIdEndByteIndex = 7;
constexpr uint32_t kMaxPageNumber = (1 << 13) - 1; // 13-bit max = 8191

uint64_t toCompactRowId(uint32_t partitionId, uint32_t pageNumber, uint32_t offsetInPage) {
// |63 partitionId(24) |39 inputIndex(13) |26 rowIndex(27) |
Expand Down Expand Up @@ -216,7 +217,7 @@ void VeloxSortShuffleWriter::insertRows(
}

arrow::Status VeloxSortShuffleWriter::maybeSpill(uint32_t nextRows) {
if ((uint64_t)offset_ + nextRows > std::numeric_limits<uint32_t>::max()) {
if ((uint64_t)offset_ + nextRows > std::numeric_limits<uint32_t>::max() || pageNumber_ >= kMaxPageNumber) {
RETURN_NOT_OK(evictAllPartitions());
}
return arrow::Status::OK();
Expand Down
4 changes: 3 additions & 1 deletion cpp/velox/shuffle/VeloxSortShuffleWriter.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,9 @@ class VeloxSortShuffleWriter final : public VeloxShuffleWriter {
std::list<facebook::velox::BufferPtr> pages_;
std::vector<char*> pageAddresses_;
char* currentPage_;
// 13-bit: max 8192 pages
uint32_t pageNumber_;
// 27-bit: max 128MB page size
uint32_t pageCursor_;
// For debug.
uint32_t currenPageSize_;
Expand All @@ -116,7 +118,7 @@ class VeloxSortShuffleWriter final : public VeloxShuffleWriter {

// Row ID -> Partition ID
// subscript: The index of row in the current input RowVector
// value: Partition ID
// value: Partition ID (24-bit: max 16M partitions)
// Updated for each input RowVector.
std::vector<uint32_t> row2Partition_;

Expand Down