Skip to content

Commit 2fbc758

Browse files
authored
perf: avoid intermediate slice allocation in Spark slice function (#23481)
## Which issue does this PR close? - N/A (small performance improvement) ## Rationale for this change The Spark `slice` function's `calculate_start_end` helper reads the length of each list element in a per-row hot loop via `values.value(row).len()`. `GenericListArray::value(row)` materializes a new `ArrayRef` for the sublist on every iteration purely to call `.len()` on it. The length is already available from the offset buffer, so this allocation is pure overhead. Replacing it with `values.value_length(row)` reads the length directly from the offsets with no allocation. Benchmarked with the existing `datafusion/spark/benches/slice.rs` over 1M rows: | case | before | after | improvement | |------|--------|-------|-------------| | List(Int64), array args | 54.0 ms | 40.0 ms | ~26% faster | | List(Int64), scalar args | 88.4 ms | 69.8 ms | ~21% faster | ## What changes are included in this PR? A single-line change in `datafusion/spark/src/function/array/slice.rs` replacing `values.value(row).len() as i64` with `values.value_length(row) as i64`. ## Are these changes tested? Covered by existing unit tests in `datafusion/spark/src/function/array/slice.rs` and slt coverage for the Spark `slice` function; behavior is unchanged. The performance impact was measured with the existing `slice` criterion benchmark. ## Are there any user-facing changes? No.
1 parent a29c58f commit 2fbc758

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

  • datafusion/spark/src/function/array

datafusion/spark/src/function/array/slice.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -157,7 +157,7 @@ fn calculate_start_end(args: &[ArrayRef]) -> Result<(ArrayRef, ArrayRef)> {
157157
}
158158
let start = start.value(row);
159159
let length = length.value(row);
160-
let value_length = values.value(row).len() as i64;
160+
let value_length = values.value_length(row) as i64;
161161

162162
if start == 0 {
163163
return exec_err!("Start index must not be zero");

0 commit comments

Comments
 (0)