Summary
COUNT(x) FILTER (WHERE <pred>) and COUNT(*) FILTER (WHERE <pred>) over a Zarr
table return the grid cardinality (total row count), silently ignoring the
per-aggregate FILTER predicate. The result is wrong and plausible-looking.
Reproduction
Against a stock fixture (data/synthetic_v3.zarr, 700 rows, temperature in
[-50, 60) so nothing is > 100):
CREATE EXTERNAL TABLE t STORED AS ZARR LOCATION 'data/synthetic_v3.zarr';
SELECT COUNT(*) FILTER (WHERE temperature > 100) FROM t;
-- returns 700 (WRONG — correct answer is 0)
SELECT SUM(CASE WHEN temperature > 100 THEN 1 ELSE 0 END) FROM t;
-- returns 0 (correct)
Root cause
CountStatisticsRule (src/optimizer/count_optimization.rs) folds COUNT to a
constant from statistics. It correctly declines when there is a plan-level WHERE
(table_scan.filters), but an aggregate FILTER modifier is not a WHERE
clause — it lives on the AggregateFunction expression, so the scan still has no
filters and the fold fires.
extract_count_info(aggr_expr) recognises the count() and returns
(is_count_star, column_name) without inspecting the expression's filter
field, so the modifier is dropped and the count is folded to num_rows
(or num_rows - null_count).
This is the same class of bug as #(the recent) "don't drop the argument expression
in aggregate pushdown" (commit e7c871b): a modifier silently dropped during
aggregate recognition.
Suggested fix
In extract_count_info (or its caller loop), decline the optimization when the
aggregate carries a filter — return None / skip so DataFusion evaluates the
count normally. The parallel path in optimizer::cardinality (the CardinalityRule
count fold, exercised via the CLI) should get the same guard; both reproduce.
Workaround
Use SUM(CASE WHEN <pred> THEN 1 ELSE 0 END) — a SUM over a computed column reads
values and is not folded.
Discovery
Surfaced while validating the write-path Phase 2 sink; zarr-python confirmed the
written store was byte-correct and per-cell filtered reads were correct, isolating
the fault to the COUNT+FILTER fold.
Summary
COUNT(x) FILTER (WHERE <pred>)andCOUNT(*) FILTER (WHERE <pred>)over a Zarrtable return the grid cardinality (total row count), silently ignoring the
per-aggregate
FILTERpredicate. The result is wrong and plausible-looking.Reproduction
Against a stock fixture (
data/synthetic_v3.zarr, 700 rows,temperaturein[-50, 60)so nothing is> 100):Root cause
CountStatisticsRule(src/optimizer/count_optimization.rs) foldsCOUNTto aconstant from statistics. It correctly declines when there is a plan-level
WHERE(
table_scan.filters), but an aggregateFILTERmodifier is not aWHEREclause — it lives on the
AggregateFunctionexpression, so the scan still has nofilters and the fold fires.
extract_count_info(aggr_expr)recognises thecount()and returns(is_count_star, column_name)without inspecting the expression'sfilterfield, so the modifier is dropped and the count is folded to
num_rows(or
num_rows - null_count).This is the same class of bug as #(the recent) "don't drop the argument expression
in aggregate pushdown" (commit
e7c871b): a modifier silently dropped duringaggregate recognition.
Suggested fix
In
extract_count_info(or its caller loop), decline the optimization when theaggregate carries a
filter— returnNone/ skip so DataFusion evaluates thecount normally. The parallel path in
optimizer::cardinality(theCardinalityRulecount fold, exercised via the CLI) should get the same guard; both reproduce.
Workaround
Use
SUM(CASE WHEN <pred> THEN 1 ELSE 0 END)— a SUM over a computed column readsvalues and is not folded.
Discovery
Surfaced while validating the write-path Phase 2 sink; zarr-python confirmed the
written store was byte-correct and per-cell filtered reads were correct, isolating
the fault to the COUNT+FILTER fold.