Draft: [GLUTEN-12474][VL] Re-add partition-column sort for row-based fallback write with dynamic partitions.#12523
Open
littlelittlewhite09 wants to merge 2 commits into
Open
Conversation
…k write with dynamic partitions.
Problem:
When writing to a dynamically-partitioned table via the row-based fallback
WriteFilesExec (e.g. ORC/Hive tables where the native Velox write is not
applicable), the job fails with FileAlreadyExistsException.
Root cause:
- With planned write enabled, `V1Writes` drops the write's required sort at
logical optimization time when the child already provides the ordering
(e.g. a `SortAggregate` keyed by the partition column outputs rows already
ordered by that column). So the physical `WriteFilesExec.requiredChildOrdering`
is `Nil`.
- Gluten then replaces the `SortAggregate` with an unordered native hash
aggregate, and `EliminateLocalSort` eagerly removes the now-redundant local
sort feeding it. The ordering the write implicitly depended on is destroyed.
- When the write falls back to the row-based `WriteFilesExec`, its
`DynamicPartitionDataSingleWriter` requires rows sorted by the partition
columns. Unsorted rows make it re-create an already-closed partition file
and fail with `FileAlreadyExistsException`.
Fix:
Add the `EnsureRowBasedWriteFilesOrdering` post-transform rule that re-adds the
local sort on the dynamic partition (and bucket) columns for the row-based
fallback write only. Native Velox writes do not need the ordering and are
skipped (via the `ColumnarWriteFilesExec.NoopLeaf` check). The required ordering
and the "already satisfied" check reuse `V1WritesUtils` so behavior stays
identical to vanilla `FileFormatWriter`. When the write's child is an offloaded
transformer, a native `SortExecTransformer` is inserted (validated, falling back
to row-based `SortExec`) so the sort runs columnar in Velox. The rule runs
before `InsertTransitions`.
littlelittlewhite09
changed the title
|
Run Gluten Clickhouse CI on x86 |
Author
|
It seems that this pr is not compatiable with spark3.3 |
|
Run Gluten Clickhouse CI on x86 |
littlelittlewhite09
force-pushed
the
fix-writeSort
branch
from
July 15, 2026 14:32
734c105 to
d4a00e8
Compare
|
Run Gluten Clickhouse CI on x86 |
…m to fix Spark 3.3 build. EnsureRowBasedWriteFilesOrdering imported V1WritesUtils directly, but that class only exists since Spark 3.4 (planned write, SPARK-37287). Since the rule lives in gluten-substrait, which is compiled against every supported Spark profile, building with -Pspark-3.3 failed to resolve the import.
littlelittlewhite09
force-pushed
the
fix-writeSort
branch
from
July 15, 2026 14:36
d4a00e8 to
7b6877b
Compare
|
Run Gluten Clickhouse CI on x86 |
Member
|
@littlelittlewhite09, thanks for the PR. Please check if the test failure is related. You may need to rebase the code if some errors are unrelated. Thanks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem:
When writing to a dynamically-partitioned table via the row-based fallback WriteFilesExec (e.g. ORC/Hive tables where the native Velox write is not applicable), the job fails with FileAlreadyExistsException.
Root cause:
V1Writesdrops the write's required sort at logical optimization time when the child already provides the ordering (e.g. aSortAggregatekeyed by the partition column outputs rows already ordered by that column). So the physicalWriteFilesExec.requiredChildOrderingisNil.SortAggregatewith an unordered native hash aggregate, andEliminateLocalSorteagerly removes the now-redundant local sort feeding it. The ordering the write implicitly depended on is destroyed.WriteFilesExec, itsDynamicPartitionDataSingleWriterrequires rows sorted by the partition columns. Unsorted rows make it re-create an already-closed partition file and fail withFileAlreadyExistsException.What changes are proposed in this pull request?
fix #12474
How was this patch tested?
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8