[MINOR] Respect Spark bloom filter config#11561
Conversation
|
Run Gluten Clickhouse CI on x86 |
|
We may remove the gluten config if there are existed ones for Spark. |
84dec02 to
9da8e07
Compare
|
Run Gluten Clickhouse CI on x86 |
4 similar comments
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
how about config: |
|
Run Gluten Clickhouse CI on x86 |
|
spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold spark.sql.optimizer.runtime.bloomFilter.maxNumItems |
|
After spark3.4 add this PR, the num bits algorithm is changed, apache/spark#37697, and it consider the max num items config, I will update Velox function to respect this behavior |
|
Before that PR, max num items is only used in checkInputDataTypes(Spark analysis) to check the config expectedItems is reasonable |
So they are transparent to Gluten, right? |
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
6def26c to
2fc6dce
Compare
|
Run Gluten Clickhouse CI on x86 |
2fc6dce to
8ab7fb5
Compare
8ab7fb5 to
c61eba6
Compare
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
Respect Spark four configs, if not exists, Velox will use its default config value.