Skip to content

Add benchmarks for different execution plans and Arrow IPC serde #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yashmayya wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Add benchmarks for different execution plans and Arrow IPC serde #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
c7f6c86
Add benchmarks for different execution plans and Arrow IPC serde
yashmayya Jan 21, 2026
579d8c4
Update filter condition; use single-threaded Tokio runtime
yashmayya Jan 22, 2026
dece129
Move serde/deser benchmarks to their own file
gortiz Jan 23, 2026
a17dfed
Add count_group_by_bench
yashmayya Jan 23, 2026
3222eb2
Try to use zero copy deser (#2)
gortiz Jan 26, 2026
8abdad0
use zero copy read in count_group_by_bench.rs
gortiz Jan 27, 2026
4c1e95c
change the filter benchmark to filter out 50% of the rows
gortiz Jan 27, 2026
f478fa3
Add a benchmark for distinct count
gortiz Jan 27, 2026
e3d02b7
Add hash_join_bench
yashmayya Jan 28, 2026
eb09232
Improve distinct_group_by_bench.rs
gortiz Jan 28, 2026
09c7a3b
Merge branch 'serde-and-physical-plan-exec-benchmarks' of github.com:…
gortiz Jan 28, 2026
fe6753d
Reduce num_batches to 1
gortiz Jan 28, 2026
1a6f0dd
New benchmarks
gortiz Jan 29, 2026
5a11da7
change columns to lower case
gortiz Jan 30, 2026
3846e08
Fix filter_jni_benchmark
gortiz Jan 30, 2026
d399216
Add different batche sizes for serde and deser
gortiz Jan 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 147 additions & 11 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

38 changes: 38 additions & 0 deletions datafusion/physical-plan/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,10 +73,12 @@ pin-project-lite = "^0.2.7"
tokio = { workspace = true }

[dev-dependencies]
arrow = { workspace = true, features = ["ffi"] }
criterion = { workspace = true, features = ["async_futures"] }
datafusion-functions-aggregate = { workspace = true }
datafusion-functions-window = { workspace = true }
insta = { workspace = true }
jni = { version = "0.21.1", features = ["invocation"] }
rand = { workspace = true }
rstest = { workspace = true }
rstest_reuse = "0.7.0"
Expand All @@ -102,3 +104,39 @@ name = "sort_preserving_merge"
harness = false
name = "aggregate_vectorized"
required-features = ["test_utils"]

[[bench]]
harness = false
name = "filter_bench"

[[bench]]
harness = false
name = "sort_bench"

[[bench]]
harness = false
name = "deser"

[[bench]]
harness = false
name = "serde"

[[bench]]
harness = false
name = "count_group_by_bench"

[[bench]]
harness = false
name = "hash_join_bench"

[[bench]]
harness = false
name = "hash_join_by_type"

[[bench]]
harness = false
name = "distinct_group_by_bench"

[[bench]]
harness = false
name = "filter_jni_benchmark"
1 change: 1 addition & 0 deletions datafusion/physical-plan/benches/Untitled
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
create_schema
Loading