You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Use octet_length for ClickBench Q27/Q28 byte-length semantics (#23475)
### Which issue does this PR close?
* Closes#23086
### Rationale for this change
ClickBench Q27 and Q28 are intended to aggregate the **byte length** of
string values, while DataFusion intentionally defines `length` as an
alias for `character_length`, which counts UTF-8 characters rather than
bytes. This change updates the in-repo ClickBench SQL to use
`octet_length(...)` explicitly, aligning these benchmark queries with
ClickBench's intended semantics without changing DataFusion's SQL
behavior.
### What changes are included in this PR?
* Updated ClickBench Q27 to use `AVG(octet_length("URL"))` instead of
`AVG(length("URL"))` in:
* `benchmarks/queries/clickbench/queries/q27.sql`
* `benchmarks/sql_benchmarks/clickbench/benchmarks/q27.benchmark`
* Updated ClickBench Q28 to use `AVG(octet_length("Referer"))` instead
of `AVG(length("Referer"))` in:
* `benchmarks/queries/clickbench/queries/q28.sql`
* `benchmarks/sql_benchmarks/clickbench/benchmarks/q28.benchmark`
* Updated the corresponding SQLLogicTest (`clickbench.slt`) queries and
expected `EXPLAIN` output so the logical and physical plans reference
`octet_length(...)` rather than `character_length(...)` or
`length(...)`.
* Added brief comments in the benchmark SQL noting that `length(...)` in
DataFusion counts characters and that `octet_length(...)` is used to
preserve ClickBench byte-length semantics.
### Are these changes tested?
The patch updates the expected SQLLogicTest
(`datafusion/sqllogictest/test_files/clickbench.slt`) output for Q27 and
Q28 to reflect the new `octet_length(...)` plans.
No test execution or benchmark results are included in this patch.
Performance was **not measured** as part of this change.
### Are there any user-facing changes?
There are no user-facing API or SQL semantic changes. This change only
updates the in-repo ClickBench benchmark queries and their corresponding
SQLLogicTest expectations to use explicit byte-length semantics.
### LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed.
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
-- Must set for ClickBench hits_partitioned dataset. See https://github.com/apache/datafusion/issues/16591
2
2
-- set datafusion.execution.parquet.binary_as_string = true
3
3
4
-
SELECT"CounterID", AVG(length("URL")) AS l, COUNT(*) AS c FROM hits WHERE"URL"<>''GROUP BY"CounterID"HAVINGCOUNT(*) >100000ORDER BY l DESCLIMIT25;
4
+
-- DataFusion length(...) counts characters; use octet_length(...) for ClickBench byte-length semantics.
5
+
SELECT"CounterID", AVG(octet_length("URL")) AS l, COUNT(*) AS c FROM hits WHERE"URL"<>''GROUP BY"CounterID"HAVINGCOUNT(*) >100000ORDER BY l DESCLIMIT25;
-- Must set for ClickBench hits_partitioned dataset. See https://github.com/apache/datafusion/issues/16591
2
2
-- set datafusion.execution.parquet.binary_as_string = true
3
3
4
-
SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE"Referer"<>''GROUP BY k HAVINGCOUNT(*) >100000ORDER BY l DESCLIMIT25;
4
+
-- DataFusion length(...) counts characters; use octet_length(...) for ClickBench byte-length semantics.
5
+
SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(octet_length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE"Referer"<>''GROUP BY k HAVINGCOUNT(*) >100000ORDER BY l DESCLIMIT25;
Copy file name to clipboardExpand all lines: benchmarks/sql_benchmarks/clickbench/benchmarks/q27.benchmark
+2-1Lines changed: 2 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -11,6 +11,7 @@ SELECT COUNT(*) > 0 from hits;
11
11
true
12
12
13
13
run
14
-
SELECT "CounterID", AVG(length("URL")) AS l, COUNT(*) AS c FROM hits WHERE "URL" <> '' GROUP BY "CounterID" HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
14
+
-- DataFusion length(...) counts characters; use octet_length(...) for ClickBench byte-length semantics.
15
+
SELECT "CounterID", AVG(octet_length("URL")) AS l, COUNT(*) AS c FROM hits WHERE "URL" <> '' GROUP BY "CounterID" HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
15
16
16
17
result sql_benchmarks/clickbench/results/${CLICKBENCH_TYPE:-single}/q27.csv
Copy file name to clipboardExpand all lines: benchmarks/sql_benchmarks/clickbench/benchmarks/q28.benchmark
+2-1Lines changed: 2 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -11,6 +11,7 @@ SELECT COUNT(*) > 0 from hits;
11
11
true
12
12
13
13
run
14
-
SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
14
+
-- DataFusion length(...) counts characters; use octet_length(...) for ClickBench byte-length semantics.
15
+
SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(octet_length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
15
16
16
17
result sql_benchmarks/clickbench/results/${CLICKBENCH_TYPE:-single}/q28.csv
Copy file name to clipboardExpand all lines: datafusion/sqllogictest/test_files/clickbench.slt
+16-16Lines changed: 16 additions & 16 deletions
Original file line number
Diff line number
Diff line change
@@ -728,58 +728,58 @@ SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "EventTime",
728
728
729
729
## Q27
730
730
query TT
731
-
EXPLAIN SELECT "CounterID", AVG(length("URL")) AS l, COUNT(*) AS c FROM hits WHERE "URL" <> '' GROUP BY "CounterID" HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
731
+
EXPLAIN SELECT "CounterID", AVG(octet_length("URL")) AS l, COUNT(*) AS c FROM hits WHERE "URL" <> '' GROUP BY "CounterID" HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
732
732
----
733
733
logical_plan
734
734
01)Sort: l DESC NULLS FIRST, fetch=25
735
-
02)--Projection: hits.CounterID, avg(length(hits.URL)) AS l, count(Int64(1)) AS count(*) AS c
735
+
02)--Projection: hits.CounterID, avg(octet_length(hits.URL)) AS l, count(Int64(1)) AS count(*) AS c
736
736
03)----Filter: count(Int64(1)) > Int64(100000)
737
-
04)------Aggregate: groupBy=[[hits.CounterID]], aggr=[[avg(CAST(character_length(hits.URL) AS length(hits.URL) AS Float64)), count(Int64(1))]]
737
+
04)------Aggregate: groupBy=[[hits.CounterID]], aggr=[[avg(CAST(octet_length(hits.URL) AS Float64)), count(Int64(1))]]
10)------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[CounterID, URL], file_type=parquet, predicate=URL@13 != , pruning_predicate=URL_null_count@2 != row_count@3 AND (URL_min@0 != OR != URL_max@1), required_guarantees=[URL not in ()]
752
752
753
753
query IRI
754
-
SELECT "CounterID", AVG(length("URL")) AS l, COUNT(*) AS c FROM hits WHERE "URL" <> '' GROUP BY "CounterID" HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
754
+
SELECT "CounterID", AVG(octet_length("URL")) AS l, COUNT(*) AS c FROM hits WHERE "URL" <> '' GROUP BY "CounterID" HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
755
755
----
756
756
757
757
## Q28
758
758
query TT
759
-
EXPLAIN SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
759
+
EXPLAIN SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(octet_length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
760
760
----
761
761
logical_plan
762
762
01)Sort: l DESC NULLS FIRST, fetch=25
763
-
02)--Projection: regexp_replace(hits.Referer,Utf8("^https?://(?:www\.)?([^/]+)/.*$"),Utf8("\1")) AS k, avg(length(hits.Referer)) AS l, count(Int64(1)) AS count(*) AS c, min(hits.Referer)
763
+
02)--Projection: regexp_replace(hits.Referer,Utf8("^https?://(?:www\.)?([^/]+)/.*$"),Utf8("\1")) AS k, avg(octet_length(hits.Referer)) AS l, count(Int64(1)) AS count(*) AS c, min(hits.Referer)
764
764
03)----Filter: count(Int64(1)) > Int64(100000)
765
-
04)------Aggregate: groupBy=[[regexp_replace(hits.Referer, Utf8View("^https?://(?:www\.)?([^/]+)/.*$"), Utf8View("\1")) AS regexp_replace(hits.Referer,Utf8("^https?://(?:www\.)?([^/]+)/.*$"),Utf8("\1"))]], aggr=[[avg(CAST(character_length(hits.Referer) AS length(hits.Referer) AS Float64)), count(Int64(1)), min(hits.Referer)]]
765
+
04)------Aggregate: groupBy=[[regexp_replace(hits.Referer, Utf8View("^https?://(?:www\.)?([^/]+)/.*$"), Utf8View("\1")) AS regexp_replace(hits.Referer,Utf8("^https?://(?:www\.)?([^/]+)/.*$"),Utf8("\1"))]], aggr=[[avg(CAST(octet_length(hits.Referer) AS Float64)), count(Int64(1)), min(hits.Referer)]]
02)--ProjectionExec: expr=[regexp_replace(hits.Referer,Utf8("^https?://(?:www\.)?([^/]+)/.*$"),Utf8("\1"))@0 as k, avg(length(hits.Referer))@1 as l, count(Int64(1))@2 as c, min(hits.Referer)@3 as min(hits.Referer)]
02)--ProjectionExec: expr=[regexp_replace(hits.Referer,Utf8("^https?://(?:www\.)?([^/]+)/.*$"),Utf8("\1"))@0 as k, avg(octet_length(hits.Referer))@1 as l, count(Int64(1))@2 as c, min(hits.Referer)@3 as min(hits.Referer)]
10)------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[Referer], file_type=parquet, predicate=Referer@14 != , pruning_predicate=Referer_null_count@2 != row_count@3 AND (Referer_min@0 != OR != Referer_max@1), required_guarantees=[Referer not in ()]
780
780
781
781
query TRIT
782
-
SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
782
+
SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(octet_length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
0 commit comments