Skip to content

[VL] Support function ToPrettyString#11566

Merged
jinchengchenghh merged 2 commits into
apache:mainfrom
jinchengchenghh:to_pretty
Feb 24, 2026
Merged

[VL] Support function ToPrettyString#11566
jinchengchenghh merged 2 commits into
apache:mainfrom
jinchengchenghh:to_pretty

Conversation

@jinchengchenghh

@jinchengchenghh jinchengchenghh commented Feb 5, 2026

Copy link
Copy Markdown
Contributor

The expression is available since Spark 3.5. Prior versions use CAST for binary output.
Spark 4.0 introduces the optional config spark.sql.binaryOutputStyle.
This PR supports only the default behavior (HEX_DISCRETE), which is used when the config is unset.

@github-actions github-actions Bot added CORE works for Gluten Core BUILD VELOX labels Feb 5, 2026
@github-actions

github-actions Bot commented Feb 5, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@rui-mo rui-mo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you resolve the below failure?

Error:  /__w/incubator-gluten/incubator-gluten/gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionMappings.scala:110: error: not found: type ToPrettyString
Error:      Sig[ToPrettyString](TO_PRETTY_STRING),

@github-actions

github-actions Bot commented Feb 5, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@jinchengchenghh

Copy link
Copy Markdown
Contributor Author

The version spark 3.5 works well, but spark4.0 and Spark 4.1 introduces BINARY_OUTPUT_STYLE, we only support the default HEX_DISCRETE as spark 3.5, so I will update the test after spark 4.0

val BINARY_OUTPUT_STYLE = buildConf("spark.sql.binaryOutputStyle")
    .doc("The output style used display binary data. Valid values are 'UTF-8', " +
      "'BASIC', 'BASE64', 'HEX', and 'HEX_DISCRETE'.")
    .version("4.0.0")
    .enumConf(BinaryOutputStyle)
    .createOptional

@github-actions

github-actions Bot commented Feb 5, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

2 similar comments
@github-actions

github-actions Bot commented Feb 6, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@jinchengchenghh

Copy link
Copy Markdown
Contributor Author

Could you help approve this PR? Thanks! @rui-mo

@rui-mo rui-mo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@ArnavBalyan

Copy link
Copy Markdown
Member

Thanks

@rui-mo rui-mo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to add a test to ensure this function could be offloaded by verifying the plan.

@jinchengchenghh

Copy link
Copy Markdown
Contributor Author

I try to add a unit test

  test("to_pretty_string") {
    val df = spark.sql("SELECT isnull(1)")
    df.show()
    print("plan is " + df.queryExecution.executedPlan.toString())
    checkGlutenPlan[ProjectExecTransformer](df)
  }

+-----------+
|(1 IS NULL)|
+-----------+
| false|
+-----------+

plan is VeloxColumnarToRow
+- ^(1) ProjectExecTransformer [isnull(1) AS (1 IS NULL)#146]
+- ^(1) InputIteratorTransformer[]
+- RowToVeloxColumnar
+- *(1) Scan OneRowRelation[]

But the plan does not contain the to_pretty_string, so it is hard to verify if it is really offload to native unless we can check the native plan.

But ToPrettyString is really offloaded since the unit test getRows::binary failed with this PR

@jinchengchenghh
jinchengchenghh merged commit d168c49 into apache:main Feb 24, 2026
105 of 107 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLICKHOUSE CORE works for Gluten Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants