Commit 2fbc758
authored
perf: avoid intermediate slice allocation in Spark slice function (#23481)
## Which issue does this PR close?
- N/A (small performance improvement)
## Rationale for this change
The Spark `slice` function's `calculate_start_end` helper reads the
length of each list element in a per-row hot loop via
`values.value(row).len()`. `GenericListArray::value(row)` materializes a
new `ArrayRef` for the sublist on every iteration purely to call
`.len()` on it. The length is already available from the offset buffer,
so this allocation is pure overhead.
Replacing it with `values.value_length(row)` reads the length directly
from the offsets with no allocation.
Benchmarked with the existing `datafusion/spark/benches/slice.rs` over
1M rows:
| case | before | after | improvement |
|------|--------|-------|-------------|
| List(Int64), array args | 54.0 ms | 40.0 ms | ~26% faster |
| List(Int64), scalar args | 88.4 ms | 69.8 ms | ~21% faster |
## What changes are included in this PR?
A single-line change in `datafusion/spark/src/function/array/slice.rs`
replacing `values.value(row).len() as i64` with
`values.value_length(row) as i64`.
## Are these changes tested?
Covered by existing unit tests in
`datafusion/spark/src/function/array/slice.rs` and slt coverage for the
Spark `slice` function; behavior is unchanged. The performance impact
was measured with the existing `slice` criterion benchmark.
## Are there any user-facing changes?
No.1 parent a29c58f commit 2fbc758
1 file changed
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
157 | 157 | | |
158 | 158 | | |
159 | 159 | | |
160 | | - | |
| 160 | + | |
161 | 161 | | |
162 | 162 | | |
163 | 163 | | |
| |||
0 commit comments