feat: add BuildHasher variants for hash_utils

[xudong963]

Which issue does this PR close?

• Closes Add BuildHasher variants for hash_utils #21428.

Rationale for this change

This PR adds BuildHasher-based variants for hash_utils so callers can compute row hashes with a caller-provided hash builder instead of always using DataFusion's default RandomState.

The main constraint is performance: with_hashes is a hot path, especially for string, dictionary, and nested array hashing. A previous version in #21429 caused measurable regressions in the default RandomState path, for example large_utf8: single, no nulls regressed from roughly 26.7us to 36.3us, and large_utf8: multiple, no nulls from roughly 112us to 127us.

This version keeps the default path performance-oriented by avoiding a fully generic BuildHasher rewrite of the existing hot loops.

What changes are included in this PR?

This PR adds:

• with_hashes_with_hasher
• create_hashes_with_hasher
• custom-hasher implementations for primitive, string, binary, byte-view, dictionary, and nested arrays
• tests covering custom hashers, multi-column hashing, and dictionary equivalence

The implementation intentionally uses a hybrid design:

• Default RandomState leaf hot paths remain specialized.
• Custom BuildHasher leaf paths live separately in hash_utils/build_hasher.rs.
• Nested/structural logic is shared through an internal child-hashing adapter, so struct/list/map/union/run/dictionary behavior does not need to be broadly duplicated.

The trade-off is that there is still some duplication for primitive/string/binary leaf loops. That duplication is intentional: those are the hottest loops, and keeping them separate prevents the existing RandomState path from becoming generic over BuildHasher or being perturbed by the custom-hasher implementation.

Are these changes tested?

Yes.

[xudong963]

run benchmark with_hashes

[adriangbot]

🤖 Criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4310621161-1803-2z8k9 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing xudong963/hash-buildhasher (0823c8d) to fd093fb (merge-base) diff
BENCH_NAME=with_hashes
BENCH_COMMAND=cargo bench --features=parquet --bench with_hashes
BENCH_FILTER=
Results will be posted here when complete

---

File an issue against this benchmark runner

[adriangbot]

🤖 Criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        main                                   xudong963_hash-buildhasher
-----                                        ----                                   --------------------------
dense_union: multiple, no nulls              1.00    234.0±0.28µs        ? ?/sec    1.00    234.2±0.37µs        ? ?/sec
dense_union: single, no nulls                1.00     78.3±0.12µs        ? ?/sec    1.00     78.3±0.10µs        ? ?/sec
dictionary_utf8_int32: multiple, no nulls    1.09     16.1±1.75µs        ? ?/sec    1.00     14.8±0.13µs        ? ?/sec
dictionary_utf8_int32: multiple, nulls       1.00     32.2±0.07µs        ? ?/sec    1.00     32.2±0.04µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.00      4.6±0.00µs        ? ?/sec    1.00      4.6±0.01µs        ? ?/sec
dictionary_utf8_int32: single, nulls         1.00     10.9±0.01µs        ? ?/sec    1.02     11.0±0.01µs        ? ?/sec
int64: multiple, no nulls                    1.00     16.9±0.01µs        ? ?/sec    1.00     17.0±0.08µs        ? ?/sec
int64: multiple, nulls                       1.00     26.8±0.02µs        ? ?/sec    1.07     28.7±0.03µs        ? ?/sec
int64: single, no nulls                      1.00      4.6±0.01µs        ? ?/sec    1.00      4.6±0.00µs        ? ?/sec
int64: single, nulls                         1.00      8.7±0.01µs        ? ?/sec    1.00      8.8±0.01µs        ? ?/sec
large_utf8: multiple, no nulls               1.03    111.2±1.46µs        ? ?/sec    1.00    108.2±0.49µs        ? ?/sec
large_utf8: multiple, nulls                  1.00    134.2±0.60µs        ? ?/sec    1.00    134.6±0.47µs        ? ?/sec
large_utf8: single, no nulls                 1.00     33.9±0.10µs        ? ?/sec    1.01     34.2±0.32µs        ? ?/sec
large_utf8: single, nulls                    1.04     46.1±0.13µs        ? ?/sec    1.00     44.5±0.71µs        ? ?/sec
list_array: multiple, no nulls               1.01    147.0±0.39µs        ? ?/sec    1.00    146.2±0.34µs        ? ?/sec
list_array: multiple, nulls                  1.01    177.4±0.46µs        ? ?/sec    1.00    176.1±0.63µs        ? ?/sec
list_array: single, no nulls                 1.00     49.3±0.15µs        ? ?/sec    1.00     49.2±0.14µs        ? ?/sec
list_array: single, nulls                    1.01     59.4±0.14µs        ? ?/sec    1.00     58.7±0.15µs        ? ?/sec
list_array_sliced: 1/10 of 81920 rows        1.00     49.2±0.21µs        ? ?/sec    1.00     49.2±0.40µs        ? ?/sec
list_array_sliced: 1/2 of 16384 rows         1.00     49.5±0.11µs        ? ?/sec    1.00     49.5±0.12µs        ? ?/sec
list_array_sliced: 1/5 of 40960 rows         1.00     49.6±0.20µs        ? ?/sec    1.00     49.7±0.08µs        ? ?/sec
map_array: multiple, no nulls                1.00    242.6±0.72µs        ? ?/sec    1.00    242.7±0.61µs        ? ?/sec
map_array: multiple, nulls                   1.00    247.8±1.20µs        ? ?/sec    1.00    247.9±0.77µs        ? ?/sec
map_array: single, no nulls                  1.00     81.1±0.23µs        ? ?/sec    1.01     81.7±0.20µs        ? ?/sec
map_array: single, nulls                     1.01     83.3±0.51µs        ? ?/sec    1.00     82.8±0.28µs        ? ?/sec
map_array_sliced: 1/10 of 81920 rows         1.00     81.4±0.25µs        ? ?/sec    1.00     81.7±0.33µs        ? ?/sec
map_array_sliced: 1/2 of 16384 rows          1.01     81.3±0.18µs        ? ?/sec    1.00     80.9±0.22µs        ? ?/sec
map_array_sliced: 1/5 of 40960 rows          1.00     81.0±0.28µs        ? ?/sec    1.01     81.7±0.44µs        ? ?/sec
run_array_int32: multiple, no nulls          1.00      5.0±0.01µs        ? ?/sec    1.00      5.0±0.01µs        ? ?/sec
run_array_int32: multiple, nulls             1.01      6.7±0.01µs        ? ?/sec    1.00      6.7±0.01µs        ? ?/sec
run_array_int32: single, no nulls            1.00   1831.9±4.47ns        ? ?/sec    1.00   1825.9±2.24ns        ? ?/sec
run_array_int32: single, nulls               1.00      2.4±0.01µs        ? ?/sec    1.01      2.4±0.00µs        ? ?/sec
sparse_union: multiple, no nulls             1.00    256.0±0.46µs        ? ?/sec    1.00    255.3±0.69µs        ? ?/sec
sparse_union: single, no nulls               1.00     85.4±0.23µs        ? ?/sec    1.00     85.4±0.22µs        ? ?/sec
sparse_union_sliced: 1/10 of 81920 rows      1.01     86.0±0.22µs        ? ?/sec    1.00     85.3±0.17µs        ? ?/sec
sparse_union_sliced: 1/2 of 16384 rows       1.00     85.5±0.26µs        ? ?/sec    1.00     85.5±0.23µs        ? ?/sec
sparse_union_sliced: 1/5 of 40960 rows       1.00     85.6±0.21µs        ? ?/sec    1.00     85.4±0.25µs        ? ?/sec
struct_array: multiple, no nulls             1.01    156.2±1.48µs        ? ?/sec    1.00    154.6±0.64µs        ? ?/sec
struct_array: multiple, nulls                1.00    170.1±0.87µs        ? ?/sec    1.00    170.7±1.04µs        ? ?/sec
struct_array: single, no nulls               1.01     52.8±0.15µs        ? ?/sec    1.00     52.2±0.32µs        ? ?/sec
struct_array: single, nulls                  1.00     57.8±0.15µs        ? ?/sec    1.00     57.7±0.17µs        ? ?/sec
utf8: multiple, no nulls                     1.01    103.3±0.41µs        ? ?/sec    1.00    102.3±0.38µs        ? ?/sec
utf8: multiple, nulls                        1.00    144.1±2.47µs        ? ?/sec    1.00    143.8±2.55µs        ? ?/sec
utf8: single, no nulls                       1.00     27.3±0.12µs        ? ?/sec    1.03     28.1±0.13µs        ? ?/sec
utf8: single, nulls                          1.00     46.6±0.55µs        ? ?/sec    1.01     47.2±0.19µs        ? ?/sec
utf8_view (small): multiple, no nulls        1.00     19.0±0.02µs        ? ?/sec    1.02     19.3±0.16µs        ? ?/sec
utf8_view (small): multiple, nulls           1.02     29.9±0.07µs        ? ?/sec    1.00     29.2±0.05µs        ? ?/sec
utf8_view (small): single, no nulls          1.00      5.1±0.01µs        ? ?/sec    1.00      5.1±0.03µs        ? ?/sec
utf8_view (small): single, nulls             1.01      9.7±0.01µs        ? ?/sec    1.00      9.7±0.01µs        ? ?/sec
utf8_view: multiple, no nulls                1.00    106.0±0.35µs        ? ?/sec    1.00    105.9±0.40µs        ? ?/sec
utf8_view: multiple, nulls                   1.01    116.8±0.19µs        ? ?/sec    1.00    115.1±0.44µs        ? ?/sec
utf8_view: single, no nulls                  1.00     34.8±0.10µs        ? ?/sec    1.00     34.7±0.12µs        ? ?/sec
utf8_view: single, nulls                     1.00     34.1±0.06µs        ? ?/sec    1.08     36.8±0.15µs        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	525.1s
Peak memory	4.0 GiB
Avg memory	4.0 GiB
CPU user	662.1s
CPU sys	1.3s
Peak spill	0 B

branch

Metric	Value
Wall time	525.1s
Peak memory	4.0 GiB
Avg memory	4.0 GiB
CPU user	661.7s
CPU sys	0.7s
Peak spill	0 B

---

File an issue against this benchmark runner

[xudong963]

@Dandandan, this PR resolves the regression for the #21429. It would be sweet to get your thoughts!

[github-actions]

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

[xudong963]

run benchmark with_hashes

[adriangbot]

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4968099420-1046-zktcl 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing xudong963/hash-buildhasher (ddc3035) to 95cda37 (merge-base) diff using: with_hashes
Results will be posted here when complete

---

File an issue against this benchmark runner

[adriangbot]

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   xudong963_hash-buildhasher
-----                                        ----                                   --------------------------
dense_union: multiple, no nulls              1.01    280.8±0.77µs        ? ?/sec    1.00    276.9±0.56µs        ? ?/sec
dense_union: single, no nulls                1.01     93.9±0.23µs        ? ?/sec    1.00     92.7±0.28µs        ? ?/sec
dictionary_utf8_int32: multiple, no nulls    1.13     16.5±0.17µs        ? ?/sec    1.00     14.6±0.01µs        ? ?/sec
dictionary_utf8_int32: multiple, nulls       1.00     32.0±0.02µs        ? ?/sec    1.00     32.1±0.02µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.00      4.6±0.00µs        ? ?/sec    1.02      4.7±0.06µs        ? ?/sec
dictionary_utf8_int32: single, nulls         1.00     10.8±0.01µs        ? ?/sec    1.01     11.0±0.01µs        ? ?/sec
int64: multiple, no nulls                    1.00     16.9±0.01µs        ? ?/sec    1.00     16.9±0.04µs        ? ?/sec
int64: multiple, nulls                       1.00     26.8±0.02µs        ? ?/sec    1.00     26.7±0.01µs        ? ?/sec
int64: single, no nulls                      1.00      4.6±0.06µs        ? ?/sec    1.00      4.6±0.00µs        ? ?/sec
int64: single, nulls                         1.00      8.7±0.00µs        ? ?/sec    1.00      8.8±0.01µs        ? ?/sec
large_utf8: multiple, no nulls               1.00    103.6±0.45µs        ? ?/sec    1.00    103.4±0.46µs        ? ?/sec
large_utf8: multiple, nulls                  1.00    157.8±0.42µs        ? ?/sec    1.00    158.2±0.51µs        ? ?/sec
large_utf8: single, no nulls                 1.01     26.7±0.29µs        ? ?/sec    1.00     26.4±0.34µs        ? ?/sec
large_utf8: single, nulls                    1.00     45.4±0.49µs        ? ?/sec    1.00     45.2±0.39µs        ? ?/sec
list_array: multiple, no nulls               1.01    146.3±0.20µs        ? ?/sec    1.00    145.2±0.05µs        ? ?/sec
list_array: multiple, nulls                  1.00    176.3±0.17µs        ? ?/sec    1.00    176.4±0.16µs        ? ?/sec
list_array: single, no nulls                 1.00     48.9±0.04µs        ? ?/sec    1.02     50.2±0.66µs        ? ?/sec
list_array: single, nulls                    1.00     59.1±0.04µs        ? ?/sec    1.00     58.9±0.03µs        ? ?/sec
list_array_sliced: 1/10 of 81920 rows        1.01     49.0±0.05µs        ? ?/sec    1.00     48.7±0.08µs        ? ?/sec
list_array_sliced: 1/2 of 16384 rows         1.00     49.2±0.06µs        ? ?/sec    1.00     49.2±0.02µs        ? ?/sec
list_array_sliced: 1/5 of 40960 rows         1.00     49.2±0.04µs        ? ?/sec    1.00     49.0±0.09µs        ? ?/sec
map_array: multiple, no nulls                1.00    240.5±0.16µs        ? ?/sec    1.00    241.6±0.28µs        ? ?/sec
map_array: multiple, nulls                   1.00    246.6±0.48µs        ? ?/sec    1.00    246.5±0.08µs        ? ?/sec
map_array: single, no nulls                  1.00     80.4±0.07µs        ? ?/sec    1.00     80.8±0.04µs        ? ?/sec
map_array: single, nulls                     1.00     82.6±0.16µs        ? ?/sec    1.00     82.4±0.04µs        ? ?/sec
map_array_sliced: 1/10 of 81920 rows         1.00     80.4±0.06µs        ? ?/sec    1.01     81.0±0.15µs        ? ?/sec
map_array_sliced: 1/2 of 16384 rows          1.00     80.3±0.08µs        ? ?/sec    1.00     80.4±0.05µs        ? ?/sec
map_array_sliced: 1/5 of 40960 rows          1.00     80.3±0.05µs        ? ?/sec    1.01     80.9±0.14µs        ? ?/sec
run_array_int32: multiple, no nulls          1.00      5.0±0.01µs        ? ?/sec    1.00      5.0±0.01µs        ? ?/sec
run_array_int32: multiple, nulls             1.00      6.8±0.01µs        ? ?/sec    1.00      6.8±0.01µs        ? ?/sec
run_array_int32: single, no nulls            1.00   1820.8±4.17ns        ? ?/sec    1.00   1829.6±4.72ns        ? ?/sec
run_array_int32: single, nulls               1.00      2.5±0.01µs        ? ?/sec    1.00      2.5±0.01µs        ? ?/sec
sparse_union: multiple, no nulls             1.07    373.9±4.49µs        ? ?/sec    1.00    351.0±2.67µs        ? ?/sec
sparse_union: single, no nulls               1.07    124.8±1.56µs        ? ?/sec    1.00    116.4±0.85µs        ? ?/sec
sparse_union_sliced: 1/10 of 81920 rows      1.07    124.8±1.67µs        ? ?/sec    1.00    116.4±0.78µs        ? ?/sec
sparse_union_sliced: 1/2 of 16384 rows       1.07    124.9±1.60µs        ? ?/sec    1.00    116.5±0.87µs        ? ?/sec
sparse_union_sliced: 1/5 of 40960 rows       1.08    125.9±1.45µs        ? ?/sec    1.00    116.7±0.96µs        ? ?/sec
struct_array: multiple, no nulls             1.00    177.7±1.03µs        ? ?/sec    1.00    177.1±1.21µs        ? ?/sec
struct_array: multiple, nulls                1.01    195.6±0.70µs        ? ?/sec    1.00    194.3±1.10µs        ? ?/sec
struct_array: single, no nulls               1.01     60.1±0.15µs        ? ?/sec    1.00     59.5±0.48µs        ? ?/sec
struct_array: single, nulls                  1.01     65.9±0.09µs        ? ?/sec    1.00     65.2±0.21µs        ? ?/sec
utf8: multiple, no nulls                     1.01    111.5±0.26µs        ? ?/sec    1.00    109.9±1.51µs        ? ?/sec
utf8: multiple, nulls                        1.00    136.3±0.38µs        ? ?/sec    1.00    136.6±0.56µs        ? ?/sec
utf8: single, no nulls                       1.00     33.1±0.16µs        ? ?/sec    1.04     34.4±0.07µs        ? ?/sec
utf8: single, nulls                          1.00     36.7±0.18µs        ? ?/sec    1.02     37.5±0.24µs        ? ?/sec
utf8_view (small): multiple, no nulls        1.00     19.0±0.01µs        ? ?/sec    1.00     19.0±0.01µs        ? ?/sec
utf8_view (small): multiple, nulls           1.01     29.3±0.01µs        ? ?/sec    1.00     29.2±0.06µs        ? ?/sec
utf8_view (small): single, no nulls          1.00      5.0±0.00µs        ? ?/sec    1.00      5.0±0.00µs        ? ?/sec
utf8_view (small): single, nulls             1.01      9.8±0.02µs        ? ?/sec    1.00      9.7±0.01µs        ? ?/sec
utf8_view: multiple, no nulls                1.00    107.7±0.27µs        ? ?/sec    1.05    113.3±0.36µs        ? ?/sec
utf8_view: multiple, nulls                   1.00    119.6±0.40µs        ? ?/sec    1.00    119.1±0.24µs        ? ?/sec
utf8_view: single, no nulls                  1.00     31.5±0.08µs        ? ?/sec    1.10     34.5±0.07µs        ? ?/sec
utf8_view: single, nulls                     1.00     34.1±0.08µs        ? ?/sec    1.00     34.2±0.08µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	595.1s
Peak memory	40.6 MiB
Avg memory	31.8 MiB
CPU user	652.7s
CPU sys	0.6s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	590.1s
Peak memory	39.9 MiB
Avg memory	31.8 MiB
CPU user	653.9s
CPU sys	0.6s
Peak spill	0 B

---

File an issue against this benchmark runner

[zhuqi-lucas]

Thanks @xudong963 for the work!
Design looks reasonable — hybrid split makes the perf trade-off explicit and benchmarks confirm no regression on the default path.

Three things:

• Test coverage: only Int32 / Float64 / (Int32+Utf8 multi-col) exercise the custom-hasher path. Would be good to add at least Dictionary, Struct, and Utf8View with long strings (to hit the HAS_BUFFERS=true const-generic branch).

• Parity test: since the leaf loops are intentionally duplicated for perf, a test that runs both paths with a shared hash builder (e.g. RandomState wrapped as BuildHasher) and asserts equal output would catch drift cheaply.

• Nit: hash_null_with_hasher names its parameter mul_col; the original hash_null uses multi_col.

Question: what's the near-term consumer of create_hashes_with_hasher? #21428 has the motivation but no concrete caller — helpful to know if there's a follow-up PR that consumes this.

[xudong963]

@zhuqi-lucas Thanks! Current HEAD already has custom-hasher coverage for Dictionary and checks equivalence with the underlying Utf8 values. I’ll add coverage for Struct and long Utf8View values, including the external-buffer path.

For parity, I can add scoped single-column leaf tests using the same RandomState. Full bit-for-bit parity is not currently a valid invariant: the optimized HashState path seeds primitive/view rehashing from the previous column hash, while the generic BuildHasher path uses combine_hashes. Multi-column inputs and multi-field structs can therefore differ even with the same builder. I’ll document that distinction; if full parity is expected, we should align on the API design first.

Both null helpers currently use mul_col; I’ll rename both to multi_col.

The near-term consumer is from Massive's latest stateful streaming system that hashes Arrow group-key columns with a fixed-seed XXH3 builder for stable key-group and checkpoint routing. (The project is still private). I guess if @AdamGS has some other use cases.

[zhuqi-lucas]

@zhuqi-lucas Thanks! Current HEAD already has custom-hasher coverage for Dictionary and checks equivalence with the underlying Utf8 values. I’ll add coverage for Struct and long Utf8View values, including the external-buffer path.

For parity, I can add scoped single-column leaf tests using the same RandomState. Full bit-for-bit parity is not currently a valid invariant: the optimized HashState path seeds primitive/view rehashing from the previous column hash, while the generic BuildHasher path uses combine_hashes. Multi-column inputs and multi-field structs can therefore differ even with the same builder. I’ll document that distinction; if full parity is expected, we should align on the API design first.

Both null helpers currently use mul_col; I’ll rename both to multi_col.

The near-term consumer is from Massive's latest stateful streaming system that hashes Arrow group-key columns with a fixed-seed XXH3 builder for stable key-group and checkpoint routing. (The project is still private). I guess if @AdamGS has some other use cases.

Got it, thanks for this info @xudong963 !

[xudong963]

run benchmark with_hashes

[adriangbot]

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4988050748-1092-5rqd7 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing xudong963/hash-buildhasher (a1cbec8) to 18121a6 (merge-base) diff using: with_hashes
Results will be posted here when complete

---

File an issue against this benchmark runner

[adriangbot]

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   xudong963_hash-buildhasher
-----                                        ----                                   --------------------------
dense_union: multiple, no nulls              1.01    279.5±0.89µs        ? ?/sec    1.00    277.9±1.00µs        ? ?/sec
dense_union: single, no nulls                1.00     93.4±0.28µs        ? ?/sec    1.00     93.0±0.20µs        ? ?/sec
dictionary_utf8_int32: multiple, no nulls    1.00     14.7±0.07µs        ? ?/sec    1.00     14.7±0.02µs        ? ?/sec
dictionary_utf8_int32: multiple, nulls       1.00     32.2±0.01µs        ? ?/sec    1.01     32.4±0.02µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.00      4.6±0.01µs        ? ?/sec    1.01      4.6±0.06µs        ? ?/sec
dictionary_utf8_int32: single, nulls         1.01     11.0±0.01µs        ? ?/sec    1.00     10.9±0.01µs        ? ?/sec
int64: multiple, no nulls                    1.00     16.9±0.01µs        ? ?/sec    1.00     16.9±0.01µs        ? ?/sec
int64: multiple, nulls                       1.01     26.9±0.01µs        ? ?/sec    1.00     26.8±0.01µs        ? ?/sec
int64: single, no nulls                      1.00      4.6±0.01µs        ? ?/sec    1.00      4.6±0.00µs        ? ?/sec
int64: single, nulls                         1.01      8.8±0.01µs        ? ?/sec    1.00      8.7±0.01µs        ? ?/sec
large_utf8: multiple, no nulls               1.00    112.8±0.38µs        ? ?/sec    1.00    112.9±0.54µs        ? ?/sec
large_utf8: multiple, nulls                  1.00    132.5±0.55µs        ? ?/sec    1.00    132.6±0.52µs        ? ?/sec
large_utf8: single, no nulls                 1.02     35.4±0.14µs        ? ?/sec    1.00     34.8±0.12µs        ? ?/sec
large_utf8: single, nulls                    1.01     36.4±0.16µs        ? ?/sec    1.00     36.1±0.46µs        ? ?/sec
list_array: multiple, no nulls               1.00    145.6±0.18µs        ? ?/sec    1.01    146.5±0.11µs        ? ?/sec
list_array: multiple, nulls                  1.00    176.2±0.09µs        ? ?/sec    1.00    176.6±0.34µs        ? ?/sec
list_array: single, no nulls                 1.00     48.8±0.03µs        ? ?/sec    1.01     49.1±0.03µs        ? ?/sec
list_array: single, nulls                    1.00     58.9±0.07µs        ? ?/sec    1.00     58.9±0.12µs        ? ?/sec
list_array_sliced: 1/10 of 81920 rows        1.01     49.1±0.11µs        ? ?/sec    1.00     48.7±0.03µs        ? ?/sec
list_array_sliced: 1/2 of 16384 rows         1.01     49.6±0.21µs        ? ?/sec    1.00     49.3±0.05µs        ? ?/sec
list_array_sliced: 1/5 of 40960 rows         1.01     49.6±0.10µs        ? ?/sec    1.00     49.0±0.03µs        ? ?/sec
map_array: multiple, no nulls                1.00    241.8±0.11µs        ? ?/sec    1.00    241.3±0.29µs        ? ?/sec
map_array: multiple, nulls                   1.00    248.3±0.11µs        ? ?/sec    1.00    247.5±0.28µs        ? ?/sec
map_array: single, no nulls                  1.00     81.0±0.05µs        ? ?/sec    1.00     81.1±0.21µs        ? ?/sec
map_array: single, nulls                     1.00     83.2±0.04µs        ? ?/sec    1.00     83.1±0.06µs        ? ?/sec
map_array_sliced: 1/10 of 81920 rows         1.00     81.0±0.14µs        ? ?/sec    1.00     80.9±0.07µs        ? ?/sec
map_array_sliced: 1/2 of 16384 rows          1.00     81.1±0.22µs        ? ?/sec    1.00     81.0±0.06µs        ? ?/sec
map_array_sliced: 1/5 of 40960 rows          1.00     80.8±0.06µs        ? ?/sec    1.00     80.9±0.06µs        ? ?/sec
run_array_int32: multiple, no nulls          1.00      5.0±0.01µs        ? ?/sec    1.01      5.0±0.01µs        ? ?/sec
run_array_int32: multiple, nulls             1.00      6.7±0.01µs        ? ?/sec    1.00      6.7±0.01µs        ? ?/sec
run_array_int32: single, no nulls            1.00   1830.7±3.61ns        ? ?/sec    1.04   1895.8±2.38ns        ? ?/sec
run_array_int32: single, nulls               1.00      2.4±0.00µs        ? ?/sec    1.00      2.4±0.00µs        ? ?/sec
sparse_union: multiple, no nulls             1.00    349.6±2.99µs        ? ?/sec    1.00    350.0±2.56µs        ? ?/sec
sparse_union: single, no nulls               1.00    116.5±0.97µs        ? ?/sec    1.00    116.5±0.94µs        ? ?/sec
sparse_union_sliced: 1/10 of 81920 rows      1.00    116.8±0.74µs        ? ?/sec    1.00    116.7±1.08µs        ? ?/sec
sparse_union_sliced: 1/2 of 16384 rows       1.00    116.7±0.81µs        ? ?/sec    1.00    117.1±0.99µs        ? ?/sec
sparse_union_sliced: 1/5 of 40960 rows       1.00    116.9±0.92µs        ? ?/sec    1.00    116.8±0.89µs        ? ?/sec
struct_array: multiple, no nulls             1.00    181.0±0.51µs        ? ?/sec    1.01    182.6±0.41µs        ? ?/sec
struct_array: multiple, nulls                1.00    195.9±0.51µs        ? ?/sec    1.00    195.2±0.45µs        ? ?/sec
struct_array: single, no nulls               1.00     60.6±0.09µs        ? ?/sec    1.00     60.5±0.24µs        ? ?/sec
struct_array: single, nulls                  1.01     65.6±0.10µs        ? ?/sec    1.00     64.8±0.26µs        ? ?/sec
utf8: multiple, no nulls                     1.00    116.7±0.38µs        ? ?/sec    1.00    116.9±0.43µs        ? ?/sec
utf8: multiple, nulls                        1.00    142.0±0.87µs        ? ?/sec    1.00    142.5±1.83µs        ? ?/sec
utf8: single, no nulls                       1.00     26.6±0.13µs        ? ?/sec    1.02     27.2±0.15µs        ? ?/sec
utf8: single, nulls                          1.01     47.8±0.21µs        ? ?/sec    1.00     47.1±0.12µs        ? ?/sec
utf8_view (small): multiple, no nulls        1.00     19.0±0.01µs        ? ?/sec    1.00     19.0±0.01µs        ? ?/sec
utf8_view (small): multiple, nulls           1.00     29.3±0.02µs        ? ?/sec    1.02     29.9±0.03µs        ? ?/sec
utf8_view (small): single, no nulls          1.00      5.0±0.00µs        ? ?/sec    1.00      5.0±0.00µs        ? ?/sec
utf8_view (small): single, nulls             1.01      9.7±0.04µs        ? ?/sec    1.00      9.6±0.01µs        ? ?/sec
utf8_view: multiple, no nulls                1.00    105.7±0.77µs        ? ?/sec    1.00    105.5±0.37µs        ? ?/sec
utf8_view: multiple, nulls                   1.03    114.9±0.21µs        ? ?/sec    1.00    112.1±0.25µs        ? ?/sec
utf8_view: single, no nulls                  1.00     31.5±0.06µs        ? ?/sec    1.00     31.5±0.11µs        ? ?/sec
utf8_view: single, nulls                     1.08     36.9±0.09µs        ? ?/sec    1.00     34.1±0.07µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	595.1s
Peak memory	37.8 MiB
Avg memory	28.9 MiB
CPU user	648.8s
CPU sys	0.6s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	590.1s
Peak memory	40.9 MiB
Avg memory	32.2 MiB
CPU user	650.1s
CPU sys	0.6s
Peak spill	0 B

---

File an issue against this benchmark runner