feat: flip pushdown_filters default to true

DataFusion 55 defaults pushdown_filters to true so filter pushdown
into the Parquet scan is on out of the box. The new
pushdown_filter_mode=auto (heuristic) prevents narrow-projection
regressions that motivated the previous conservative default.

Updates:
- config.rs: default false -> true
- upgrade guide: add section on the flip + how to opt out
- configs.md / information_schema.slt: reflect new default
- parquet_filter_pushdown.slt: cleanup section now RESETs instead
  of assuming the pre-flip default of false

Author: zhuqi-lucas

datafusion/common/src/config.rs

@@ -1201,7 +1201,7 @@ config_namespace! {
 
         /// (reading) If true, filter expressions are be applied during the parquet decoding operation to
         /// reduce the number of rows decoded. This optimization is sometimes called "late materialization".
-        pub pushdown_filters: bool, default = false
+        pub pushdown_filters: bool, default = true
 
         /// (reading) When `pushdown_filters` is enabled, determines how DataFusion
         /// pushes each filter into the Parquet scan. Options are `auto` (the default)

datafusion/sqllogictest/test_files/information_schema.slt

@@ -261,7 +261,7 @@ datafusion.execution.parquet.maximum_parallel_row_group_writers 1
 datafusion.execution.parquet.metadata_size_hint 524288
 datafusion.execution.parquet.pruning true
 datafusion.execution.parquet.pushdown_filter_mode auto
-datafusion.execution.parquet.pushdown_filters false
+datafusion.execution.parquet.pushdown_filters true
 datafusion.execution.parquet.reorder_filters false
 datafusion.execution.parquet.schema_force_view_types true
 datafusion.execution.parquet.skip_arrow_metadata false
@@ -421,7 +421,7 @@ datafusion.execution.parquet.maximum_parallel_row_group_writers 1 (writing) By d
 datafusion.execution.parquet.metadata_size_hint 524288 (reading) If specified, the parquet reader will try and fetch the last `size_hint` bytes of the parquet file optimistically. If not specified, two reads are required: One read to fetch the 8-byte parquet footer and another to fetch the metadata length encoded in the footer Default setting to 512 KiB, which should be sufficient for most parquet files, it can reduce one I/O operation per parquet file. If the metadata is larger than the hint, two reads will still be performed.
 datafusion.execution.parquet.pruning true (reading) If true, the parquet reader attempts to skip entire row groups based on the predicate in the query and the metadata (min/max values) stored in the parquet file
 datafusion.execution.parquet.pushdown_filter_mode auto (reading) When `pushdown_filters` is enabled, determines how DataFusion pushes each filter into the Parquet scan. Options are `auto` (the default)  `always`, and `heurstic` (plan time heuristic).
-datafusion.execution.parquet.pushdown_filters false (reading) If true, filter expressions are be applied during the parquet decoding operation to reduce the number of rows decoded. This optimization is sometimes called "late materialization".
+datafusion.execution.parquet.pushdown_filters true (reading) If true, filter expressions are be applied during the parquet decoding operation to reduce the number of rows decoded. This optimization is sometimes called "late materialization".
 datafusion.execution.parquet.reorder_filters false (reading) If true, filter expressions evaluated during the parquet decoding operation will be reordered heuristically to minimize the cost of evaluation. If false, the filters are applied in the same order as written in the query
 datafusion.execution.parquet.schema_force_view_types true (reading) If true, parquet reader will read columns of `Utf8/Utf8Large` with `Utf8View`, and `Binary/BinaryLarge` with `BinaryView`.
 datafusion.execution.parquet.skip_arrow_metadata false (writing) Skip encoding the embedded arrow metadata in the KV_meta This is analogous to the `ArrowWriterOptions::with_skip_arrow_metadata`. Refer to <https://docs.rs/parquet/53.3.0/parquet/arrow/arrow_writer/struct.ArrowWriterOptions.html#method.with_skip_arrow_metadata>

datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt

@@ -939,7 +939,7 @@ A 78
 
 # Cleanup
 statement ok
-set datafusion.execution.parquet.pushdown_filters = false;
+RESET datafusion.execution.parquet.pushdown_filters;
 
 statement ok
 set datafusion.execution.parquet.reorder_filters = false;

docs/source/library-user-guide/upgrading/55.0.0.md

@@ -275,12 +275,37 @@ if tracking.contains_dynamic_filter() {
 }
 ```
 
+### Parquet `pushdown_filters` default is now `true`
+
+`datafusion.execution.parquet.pushdown_filters` now defaults to `true`. In
+DataFusion 54 and earlier the default was `false`, so filters were only pushed
+into the Parquet scan when the user (or a downstream framework) explicitly
+enabled the option. Together with the new `pushdown_filter_mode` option
+described below, enabling pushdown by default is a net win on the ClickBench
+benchmark suite.
+
+To restore the pre-55 behavior, set `pushdown_filters = false` (either per
+session or in `SessionConfig`):
+
+```sql
+SET datafusion.execution.parquet.pushdown_filters = false;
+```
+
 ### New `pushdown_filter_mode` option for Parquet filter pushdown
 
 DataFusion 55 adds a new option,
 `datafusion.execution.parquet.pushdown_filter_mode`, that controls how filters
 are pushed into Parquet scans when `pushdown_filters` is enabled. It accepts:
 
+- `auto` (default) — currently equivalent to `heuristic`; may evolve as more
+  sophisticated placement strategies are implemented
+  (see [issue #22883](https://github.com/apache/datafusion/issues/22883)).
+- `always` — unconditionally push every filter into the scan (the DataFusion 54
+  behavior when `pushdown_filters = true`).
+- `heuristic` — apply a plan-time heuristic that declines pushdown for
+  narrow-projection queries where the `RowFilter` overhead tends to outweigh
+  the decode it would save (for example `SELECT col1 FROM t WHERE col1 <> ''`).
+
 Because the default is `auto`, enabling `pushdown_filters` no longer guarantees
 that **every** filter is pushed into the scan as was the case in DataFusion 54
 and earlier. To get the old behavior, also set `pushdown_filter_mode = always`:

docs/source/user-guide/configs.md

@@ -84,7 +84,7 @@ The following configuration settings are available:
 | datafusion.execution.parquet.pruning                                    | true                      | (reading) If true, the parquet reader attempts to skip entire row groups based on the predicate in the query and the metadata (min/max values) stored in the parquet file                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | datafusion.execution.parquet.skip_metadata                              | true                      | (reading) If true, the parquet reader skip the optional embedded metadata that may be in the file Schema. This setting can help avoid schema conflicts when querying multiple parquet files with schemas containing compatible types but different metadata                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | datafusion.execution.parquet.metadata_size_hint                         | 524288                    | (reading) If specified, the parquet reader will try and fetch the last `size_hint` bytes of the parquet file optimistically. If not specified, two reads are required: One read to fetch the 8-byte parquet footer and another to fetch the metadata length encoded in the footer Default setting to 512 KiB, which should be sufficient for most parquet files, it can reduce one I/O operation per parquet file. If the metadata is larger than the hint, two reads will still be performed.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
-| datafusion.execution.parquet.pushdown_filters                           | false                     | (reading) If true, filter expressions are be applied during the parquet decoding operation to reduce the number of rows decoded. This optimization is sometimes called "late materialization".                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+| datafusion.execution.parquet.pushdown_filters                           | true                      | (reading) If true, filter expressions are be applied during the parquet decoding operation to reduce the number of rows decoded. This optimization is sometimes called "late materialization".                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
 | datafusion.execution.parquet.pushdown_filter_mode                       | auto                      | (reading) When `pushdown_filters` is enabled, determines how DataFusion pushes each filter into the Parquet scan. Options are `auto` (the default) `always`, and `heurstic` (plan time heuristic).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
 | datafusion.execution.parquet.reorder_filters                            | false                     | (reading) If true, filter expressions evaluated during the parquet decoding operation will be reordered heuristically to minimize the cost of evaluation. If false, the filters are applied in the same order as written in the query                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
 | datafusion.execution.parquet.force_filter_selections                    | false                     | (reading) Force the use of RowSelections for filter results, when pushdown_filters is enabled. If false, the reader will automatically choose between a RowSelection and a Bitmap based on the number and pattern of selected rows.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |