Skip to content

[GLUTEN-11283][VL] Respect spark default zstd compression level#11284

Merged
philo-he merged 2 commits into
apache:mainfrom
wecharyu:GLUTEN-11283
Dec 16, 2025
Merged

[GLUTEN-11283][VL] Respect spark default zstd compression level#11284
philo-he merged 2 commits into
apache:mainfrom
wecharyu:GLUTEN-11283

Conversation

@wecharyu

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Spark default zstd compression leve is 3:
https://github.com/apache/parquet-java/blob/7ec32846663782846615e835bd06859c75b8f6b0/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/ZstandardCodec.java#L52

But velox default compression level is 1:
https://github.com/facebookincubator/velox/blob/a0bd1dd248c7def13dd17724e1569fd990515401/velox/dwio/parquet/writer/arrow/util/CompressionInternal.h#L76

So the output parquet size could be larger in gluten if user does not configure the parquet.compression.codec.zstd.level.

Fix #11283

How was this patch tested?

Test written file size locally.

@github-actions github-actions Bot added the VELOX label Dec 11, 2025
@philo-he

Copy link
Copy Markdown
Member

@wecharyu, I note Spark has the following config with zatd level set to 1 by default. Is it related? Do you know why Spark doesn't respect it?

https://github.com/apache/spark/blob/43f7936d7b3a4701e3d0fdb44663006cbe0db70b/core/src/main/scala/org/apache/spark/internal/config/package.scala#L2137

@wecharyu

Copy link
Copy Markdown
Contributor Author

@philo-he The IO_COMPRESSION_ZSTD_LEVEL in package org.apache.spark.io is only used for internal data like eventlog writer, more discussion in SPARK-39743.

spark.hadoop.parquet.compression.codec.zstd.level can be used to set zstd compression level for table writer now.

@philo-he philo-he left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for the clarification. Looks good. cc @FelixYBW

@philo-he
philo-he merged commit 855843f into apache:main Dec 16, 2025
104 of 105 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[VL] Respect spark default zstd compression level

3 participants