Skip to content

[MINOR] Respect Spark bloom filter config#11561

Merged
jinchengchenghh merged 1 commit into
apache:mainfrom
jinchengchenghh:bloom_config
Mar 10, 2026
Merged

[MINOR] Respect Spark bloom filter config#11561
jinchengchenghh merged 1 commit into
apache:mainfrom
jinchengchenghh:bloom_config

Conversation

@jinchengchenghh

@jinchengchenghh jinchengchenghh commented Feb 4, 2026

Copy link
Copy Markdown
Contributor

Respect Spark four configs, if not exists, Velox will use its default config value.

spark.sql.optimizer.runtime.bloomFilter.expectedNumItems
spark.sql.optimizer.runtime.bloomFilter.maxNumItems
spark.sql.optimizer.runtime.bloomFilter.numBits
spark.sql.optimizer.runtime.bloomFilter.maxNumBits

@github-actions github-actions Bot added CORE works for Gluten Core VELOX labels Feb 4, 2026
@github-actions

github-actions Bot commented Feb 4, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@FelixYBW

Copy link
Copy Markdown
Contributor

We may remove the gluten config if there are existed ones for Spark.

@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

4 similar comments
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions github-actions Bot added the DOCS label Feb 24, 2026
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@FelixYBW

Copy link
Copy Markdown
Contributor

how about config:

spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold
spark.sql.optimizer.runtime.bloomFilter.creationSideThreshold
spark.sql.optimizer.runtime.bloomFilter.maxNumItems

@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@jinchengchenghh

jinchengchenghh commented Feb 25, 2026

Copy link
Copy Markdown
Contributor Author

spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold
spark.sql.optimizer.runtime.bloomFilter.creationSideThreshold
These above two configs decide if insert bloom filter for the join, which take effect in Spark java code.

spark.sql.optimizer.runtime.bloomFilter.maxNumItems
velox does not have corresponding config, I will check if we can respect the config in Velox.

@jinchengchenghh

Copy link
Copy Markdown
Contributor Author

After spark3.4 add this PR, the num bits algorithm is changed, apache/spark#37697, and it consider the max num items config, I will update Velox function to respect this behavior

@jinchengchenghh

Copy link
Copy Markdown
Contributor Author

Before that PR, max num items is only used in checkInputDataTypes(Spark analysis) to check the config expectedItems is reasonable

@FelixYBW

Copy link
Copy Markdown
Contributor

spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold spark.sql.optimizer.runtime.bloomFilter.creationSideThreshold These above two configs decide if insert bloom filter for the join, which take effect in Spark java code.

So they are transparent to Gluten, right?

@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions

github-actions Bot commented Mar 5, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

github-actions Bot commented Mar 9, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

github-actions Bot commented Mar 9, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions

github-actions Bot commented Mar 9, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@jinchengchenghh
jinchengchenghh merged commit a3c973f into apache:main Mar 10, 2026
66 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core DOCS VELOX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants