Skip to content

Commit 56ecc8d

Browse files
committed
Address Copilot review on #23128: reject negative FixedSizeList size, drop per-compare allocation in equal_to
Four Copilot inline comments handled together: * `group_column_supported_type` and `make_group_column` both now reject `FixedSizeList` with a negative `list_size`. Arrow defines that size as non-negative, but the enum lets callers construct a negative value programmatically; without this guard the `i32 as usize` cast inside the builder would wrap and risk panics / OOM. Guards are duplicated on both sides so direct callers of `make_group_column` (and the `supported_schema` planner gate) both fail safely. Consistency fuzz already covers the `make_group_column` side via the existing unsupported_cases path; an explicit dedicated test below covers both surfaces. * `FixedSizeListGroupValueBuilder::new` asserts `list_len >= 0` so any direct construction that bypasses the factory fails fast with a clear message instead of silently producing a broken builder. `list_len_usize` now uses `usize::try_from(...).expect(...)` instead of `as usize`; with the constructor's assertion the conversion is provably infallible, but going through `try_from` means a future invariant break shows up as an explicit panic rather than a silent wrap. * `equal_to` no longer calls `list_array.value(rhs_row)`, which constructs a sliced child `ArrayRef` on every comparison. The hot path now borrows `list_array.values()` and uses `value_offset(rhs_row)` to compute the child base index, mirroring the existing approach in `append_val`. Zero allocations per equality check. New test: `negative_list_size_is_rejected_by_allow_list_and_dispatcher` locks in the negative-size guard at both surfaces. 13/13 FSL tests pass, 128/128 aggregates pass, clippy clean.
1 parent ecdd425 commit 56ecc8d

2 files changed

Lines changed: 78 additions & 5 deletions

File tree

datafusion/physical-plan/src/aggregates/group_values/multi_group_by/fixed_size_list.rs

Lines changed: 58 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,15 @@ where
6060
"FixedSizeListGroupValueBuilder built with non-FixedSizeList type {other:?}"
6161
),
6262
};
63+
// `group_column_supported_type` and `make_group_column` both reject
64+
// negative `list_len` upstream so this branch should be unreachable
65+
// for any schema that survives the allow-list. Assert defensively
66+
// to fail fast (and with a clear message) if a direct caller ever
67+
// bypasses the factory with an invalid Arrow type.
68+
assert!(
69+
list_len >= 0,
70+
"FixedSizeListGroupValueBuilder requires non-negative list size, got {list_len}"
71+
);
6372
let child = PrimitiveGroupValueBuilder::<T, true>::new(field.data_type().clone());
6473
Self {
6574
field,
@@ -70,8 +79,14 @@ where
7079
}
7180
}
7281

82+
/// Lossless widening to `usize`. The constructor asserts
83+
/// `list_len >= 0`, so the conversion is well-defined. We still go
84+
/// through `try_from` rather than `as usize` so any future
85+
/// invariant break surfaces as a panic with a clear message rather
86+
/// than a silent wrap.
7387
fn list_len_usize(&self) -> usize {
74-
self.list_len as usize
88+
usize::try_from(self.list_len)
89+
.expect("list_len validated >= 0 in `new`; conversion to usize cannot fail")
7590
}
7691
}
7792

@@ -91,11 +106,16 @@ where
91106
.as_any()
92107
.downcast_ref::<FixedSizeListArray>()
93108
.expect("FixedSizeListGroupValueBuilder called with non-FixedSizeList array");
94-
let rhs_sublist: ArrayRef = list_array.value(rhs_row);
109+
// Use the borrowed child array + `value_offset` (same approach as
110+
// `append_val` below) instead of `list_array.value(rhs_row)`,
111+
// which would allocate a fresh sliced `ArrayRef` on every
112+
// equality check inside grouping's hot path.
113+
let child_array = list_array.values();
95114
let list_len = self.list_len_usize();
96115
let lhs_base = lhs_row * list_len;
116+
let rhs_base = list_array.value_offset(rhs_row) as usize;
97117
for j in 0..list_len {
98-
if !self.child.equal_to(lhs_base + j, &rhs_sublist, j) {
118+
if !self.child.equal_to(lhs_base + j, child_array, rhs_base + j) {
99119
return false;
100120
}
101121
}
@@ -417,6 +437,41 @@ mod tests {
417437
assert_eq!(values.values(), &[1, 2, 3, 4, 5, 6]);
418438
}
419439

440+
/// `FixedSizeList` carries its size as `i32`; the enum lets callers
441+
/// construct a negative size programmatically even though Arrow
442+
/// considers it invalid. Both the supported-type allow-list and the
443+
/// dispatcher must reject such a type so the cast inside the
444+
/// builder cannot wrap to a huge `usize` and trigger panics / OOM.
445+
#[test]
446+
fn negative_list_size_is_rejected_by_allow_list_and_dispatcher() {
447+
use crate::aggregates::group_values::multi_group_by::{
448+
group_column_supported_type, make_group_column,
449+
};
450+
use arrow::datatypes::Field;
451+
452+
let invalid = DataType::FixedSizeList(
453+
Arc::new(Field::new("item", DataType::Int32, true)),
454+
-1,
455+
);
456+
457+
assert!(
458+
!group_column_supported_type(&invalid),
459+
"allow-list must reject FixedSizeList(_, -1)"
460+
);
461+
462+
let field = Field::new("col", invalid, true);
463+
let result = make_group_column(&field);
464+
let err = match result {
465+
Ok(_) => panic!("dispatcher must reject FixedSizeList with negative size"),
466+
Err(e) => e,
467+
};
468+
let msg = err.to_string();
469+
assert!(
470+
msg.contains("negative size") && msg.contains("-1"),
471+
"error should mention the invalid negative size, got: {msg}"
472+
);
473+
}
474+
420475
#[test]
421476
fn take_n_returns_prefix_and_shifts_remainder() {
422477
let input = fsl_array(&[

datafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -926,7 +926,15 @@ macro_rules! instantiate_primitive {
926926
fn group_column_supported_type(data_type: &DataType) -> bool {
927927
// FixedSizeList<primitive> is the only nested type currently supported.
928928
// List / LargeList / Struct will follow in subsequent PRs of #22715.
929-
if let DataType::FixedSizeList(child_field, _) = data_type {
929+
if let DataType::FixedSizeList(child_field, list_size) = data_type {
930+
// Arrow defines `FixedSizeList` size as a non-negative `i32`, but
931+
// the enum still lets callers construct a negative size. Reject
932+
// here so the schema is routed to the `GroupValuesRows` fallback
933+
// instead of risking a `negative_size as usize` wrap inside the
934+
// builder. Keep this guard in lockstep with `make_group_column`.
935+
if *list_size < 0 {
936+
return false;
937+
}
930938
return matches!(
931939
child_field.data_type(),
932940
DataType::Int8
@@ -1087,7 +1095,17 @@ fn make_group_column(field: &Field) -> Result<Box<dyn GroupColumn>> {
10871095
v.push(Box::new(BooleanGroupValueBuilder::<false>::new()));
10881096
}
10891097
}
1090-
DataType::FixedSizeList(ref child_field, _) => {
1098+
DataType::FixedSizeList(ref child_field, list_size) => {
1099+
// Defense in depth against an invalid Arrow type that the
1100+
// allow-list might have missed: negative `list_size` would
1101+
// wrap when cast to `usize` and trigger panics / OOM in the
1102+
// builder. Reject explicitly here as well so direct callers
1103+
// of `make_group_column` fail safely.
1104+
if list_size < 0 {
1105+
return not_impl_err!(
1106+
"FixedSizeList with negative size {list_size} not supported in GroupValuesColumn"
1107+
);
1108+
}
10911109
// `group_column_supported_type` already restricts the child to
10921110
// the primitive subset supported here. Any unsupported child
10931111
// type returned `false` upstream and was routed to the

0 commit comments

Comments
 (0)