GH-48176: [C++][Parquet] Fix arrow-ipc-message-internal-test & arrow-acero-hash-join-node-test failures #48166
Conversation
kou
changed the title
|
|
kou
changed the title
| // Check if the scalar is a BooleanScalar before casting | ||
| if (mask.scalar()->type->id() == Type::BOOL) { |
There was a problem hiding this comment.
Why do we need this check? This check didn't exist in the existing code.
There was a problem hiding this comment.
@kou,
Please find the root cause of this issue on s390x platform below:
The test was failing with a std::bad_cast exception in the ProbeBatch_ResidualFilter function in hash_join.cc at line 309. The issue occurred when the code tried to cast a Datum's scalar to a BooleanScalar using:
const auto& mask_scalar = mask.scalar_as();
The problem was that the test uses literal(NullScalar()) as one of the filter expressions, which creates a NullScalar (of type NullType), not a BooleanScalar (of type BoolType). When the code tried to cast this to a BooleanScalar, it failed with std::bad_cast on s390x architecture
The Fix
I modified the ProbeBatch_ResidualFilter function to check the scalar's type before attempting the cast
| // On Big-endian systems, 4 bytes are appended to indicate it as a BE system. Hence the | ||
| // total size is 4 bytes more than the LE systems. |
There was a problem hiding this comment.
It seems that https://arrow.apache.org/docs/format/Columnar.html doesn't mention the 4 bytes for endianness. Could you share a document URL that mentions the 4 bytes?
There was a problem hiding this comment.
@kou..
I guess, I need to modify the comment here.. The root cause for this testcase failure is as below:
The test was failing because it contained a hardcoded expected byte array that was generated on macOS+ARM+LLVM, but in our case, the test was running on a Linux+s390x+GCC platform. FlatBuffer serialization can produce slightly different output across different platforms and toolchains, which caused the buffer size mismatch (expected 232 bytes vs actual 228 bytes).
…arrow-acero-hash-join-node-test failures
Vishwanatha-HD
force-pushed
the
fixParqIssues1
branch
from
a37f5a3 to
a4565ef
Compare
I have fixed the lint failures. Thanks.. |
Vishwanatha-HD
left a comment
Vishwanatha-HD
left a comment
There was a problem hiding this comment.
I have taken care of the review comments and made necessary changes to the code. Pls review my changes. Thanks..
github-actions
bot
added
awaiting committer review
2 participants
Rationale for this change
This PR is intended to enable Parquet DB support on Big-endian (s390x) systems. The fix in this PR fixes two test failures
i.e. "arrow-ipc-message-internal-test" & "arrow-acero-hash-join-node-test" testcase failures.
As part of "arrow-acero-hash-join-node-test", the failing testcase was "HashJoin.FineGrainedResidualFilter"..
As part of "arrow-ipc-message-internal-test", the failing testcase was "TestMessageInternal.TestByteIdentical"..
What changes are included in this PR?
The fix includes changes to "hash_join.cc" and "message_internal_test.cc" files to address the above testcase failures.
Are these changes tested?
Yes. The changes are tested on s390x arch to make sure things are working fine. The fix is also tested on x86 arch, to make sure there is no new regression introduced.
Are there any user-facing changes?
No