Commit 445851a
### Rationale for this change
When every value in a `FLOAT`, `DOUBLE`, or `FLOAT16` column is the same infinity, the Parquet writer stored a *finite* min/max statistic instead of the infinity actually present in the column:
- all `+Inf` → `min` written as `FLT_MAX` / `DBL_MAX` (should be `+Inf`)
- all `-Inf` → `max` written as `-FLT_MAX` / `-DBL_MAX` (should be `-Inf`)
The running min/max were seeded with `std::numeric_limits<T>::max()` / `::lowest()` — the largest/smallest *finite* values. These are not identity elements for min/max once the data contains infinities: `Min(FLT_MAX, +Inf)` keeps `FLT_MAX`, so an all-`+Inf` column never displaces the seed (and symmetrically for `-Inf`). The wrong value is then written with `is_min_value_exact = true`, so a reader is told the exact minimum of an all-`+Inf` column is `FLT_MAX` — a value not even present in the column. Range-based row-group filtering stays correct (the bound is only loosened in the safe direction), but engines that read min/max statistics as values (statistics-based `MIN`/`MAX` evaluation, metadata introspection) return wrong results.
See GH-50182 for a full reproduction in both PyArrow and C++.
### What changes are included in this PR?
- `CompareHelper::DefaultMin()` / `DefaultMax()` now seed floating-point accumulation with `+Inf` / `-Inf` (the true identities) instead of the largest finite values; integral types are unchanged.
- The same change is applied to the `Float16` comparator, via new `Float16Constants::positive_infinity()` / `negative_infinity()`.
- The "no valid value was seen" detection in `CleanStatistic` / `CleanFloat16Statistic` (which relied on the old finite seeds) is updated to match the new `±Inf` seeds. An all-NaN or empty column still leaves `min = +Inf, max = -Inf` (`min > max`, which real data can never produce), so it stays unambiguously detectable and continues to emit no statistics.
### Are these changes tested?
Yes. A new typed test `TestFloatStatistics.Infinities` (covering `FLOAT`, `DOUBLE`, and `FLOAT16`) asserts that all-`+Inf`, all-`-Inf`, and mixed `±Inf` columns produce the exact infinity as min/max (the existing `AssertMinMaxAre` helper also checks `is_min_value_exact`/`is_max_value_exact`). The existing all-NaN tests continue to pass, exercising the updated empty/all-NaN detection.
### Are there any user-facing changes?
Parquet files written for all-infinity floating-point columns now record the correct `±Inf` min/max statistics instead of the largest finite value. No public API signatures change.
Two minor behavior changes on existing public low-level APIs are worth noting (the normal writer accumulation path is unaffected, since a column with ≥1 non-NaN value always yields `min ≤ max`):
- `TypedComparator<FloatType/DoubleType/Float16LogicalType>::GetMinMax()` now returns `(+Inf, -Inf)` for an all-NaN (or empty) batch, where it previously returned the largest/smallest finite value as a sentinel.
- `TypedStatistics::SetMinMax()` now treats any inverted (`min > max`) input as "no valid statistics", whereas it previously only special-cased that one exact finite sentinel pair.
**This PR contains a "Critical Fix".** It fixes a bug that produced incorrect data: the writer emitted min/max statistics that are wrong yet flagged exact (`is_min_value_exact = true`), which can yield incorrect results for readers that evaluate `MIN`/`MAX` directly from Parquet statistics.
* GitHub Issue: #50182
Lead-authored-by: Chungmin Lee <chungminlee@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
1 parent b69ba75 commit 445851a
2 files changed
Lines changed: 97 additions & 12 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
61 | | - | |
62 | | - | |
63 | 61 | | |
64 | 62 | | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
65 | 69 | | |
66 | 70 | | |
67 | 71 | | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | 72 | | |
72 | 73 | | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
73 | 78 | | |
74 | 79 | | |
75 | 80 | | |
| |||
79 | 84 | | |
80 | 85 | | |
81 | 86 | | |
82 | | - | |
83 | | - | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
84 | 105 | | |
85 | 106 | | |
86 | 107 | | |
| |||
300 | 321 | | |
301 | 322 | | |
302 | 323 | | |
303 | | - | |
304 | | - | |
| 324 | + | |
| 325 | + | |
305 | 326 | | |
306 | 327 | | |
307 | 328 | | |
| |||
330 | 351 | | |
331 | 352 | | |
332 | 353 | | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
333 | 365 | | |
334 | 366 | | |
335 | 367 | | |
| |||
357 | 389 | | |
358 | 390 | | |
359 | 391 | | |
360 | | - | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
361 | 395 | | |
362 | 396 | | |
363 | 397 | | |
| |||
384 | 418 | | |
385 | 419 | | |
386 | 420 | | |
387 | | - | |
388 | | - | |
| 421 | + | |
389 | 422 | | |
390 | 423 | | |
391 | 424 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1485 | 1485 | | |
1486 | 1486 | | |
1487 | 1487 | | |
| 1488 | + | |
1488 | 1489 | | |
1489 | 1490 | | |
1490 | 1491 | | |
| |||
1559 | 1560 | | |
1560 | 1561 | | |
1561 | 1562 | | |
| 1563 | + | |
| 1564 | + | |
| 1565 | + | |
| 1566 | + | |
| 1567 | + | |
| 1568 | + | |
| 1569 | + | |
| 1570 | + | |
| 1571 | + | |
| 1572 | + | |
| 1573 | + | |
| 1574 | + | |
| 1575 | + | |
| 1576 | + | |
| 1577 | + | |
| 1578 | + | |
| 1579 | + | |
| 1580 | + | |
| 1581 | + | |
| 1582 | + | |
| 1583 | + | |
| 1584 | + | |
| 1585 | + | |
| 1586 | + | |
1562 | 1587 | | |
1563 | 1588 | | |
1564 | 1589 | | |
| |||
1611 | 1636 | | |
1612 | 1637 | | |
1613 | 1638 | | |
| 1639 | + | |
| 1640 | + | |
| 1641 | + | |
| 1642 | + | |
| 1643 | + | |
| 1644 | + | |
| 1645 | + | |
| 1646 | + | |
| 1647 | + | |
| 1648 | + | |
| 1649 | + | |
| 1650 | + | |
| 1651 | + | |
| 1652 | + | |
| 1653 | + | |
| 1654 | + | |
| 1655 | + | |
| 1656 | + | |
| 1657 | + | |
| 1658 | + | |
| 1659 | + | |
| 1660 | + | |
| 1661 | + | |
| 1662 | + | |
| 1663 | + | |
| 1664 | + | |
1614 | 1665 | | |
1615 | 1666 | | |
1616 | 1667 | | |
1617 | 1668 | | |
1618 | 1669 | | |
1619 | 1670 | | |
| 1671 | + | |
1620 | 1672 | | |
1621 | 1673 | | |
1622 | 1674 | | |
| |||
0 commit comments