[GLUTEN-10636][VL]Use backend validation to find all unsupported expression by jiangjiangtian · Pull Request #10637 · apache/gluten

jiangjiangtian · 2025-09-05T09:59:58Z

What changes are proposed in this pull request?

Now PartialProject can't be applied to the expressions that are unsupported by native backend, so maybe we can use native validation to find all unsupported expressions and apply PartialProject to them.

This PR uses the following method to find all unsupported expressions: Traverse the expression trees in post-order. For each tree node, if this expression is not offload-able, then we replace it by an Alias.
For example, if the expression is func4(func3(func2(func1()))), then we first determine whether func1() is offload-able. Second, we determine whether func2() is offload-able and so on.

After this PR, expressions like map_from_arrays that hasn't been supported by velox can be calculated using PartialProjectExec.
This PR also adds a config named spark.gluten.sql.columnar.partial.validation to control whether to use native validation to find all unsupported expressions.

How was this patch tested?

unit test.

Related issue: #10636

github-actions · 2025-09-05T10:00:14Z

#10636

github-actions · 2025-09-05T10:00:28Z

Run Gluten Clickhouse CI on x86

jiangjiangtian · 2025-09-05T10:03:11Z

@jinchengchenghh @WangGuangxin @zhztheplayer Do you think it is a reasonable way to extend PartialProject to all unsupported expressions. Please take a look, thanks!

github-actions · 2025-09-05T10:11:25Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-09-05T11:48:23Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-09-05T12:06:21Z

Run Gluten Clickhouse CI on x86

zhztheplayer · 2025-09-05T12:33:43Z

I agree we should prefer PartialProject over vanilla Spark project in almost all cases where projects fall back.

jiangjiangtian · 2025-09-08T02:33:20Z

I agree we should prefer PartialProject over vanilla Spark project in almost all cases where projects fall back.

I think so. In this PR, I use validation to find the expressions that backend doesn't support, do you think it is a reasonable way?

github-actions · 2025-09-08T10:43:16Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-09-12T08:34:16Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-09-12T08:47:46Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-09-12T09:02:44Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-09-15T02:37:59Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-10-21T09:59:07Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-10-21T10:04:26Z

Run Gluten Clickhouse CI on x86

zhztheplayer · 2025-10-21T15:15:05Z

Kindly ping @jinchengchenghh @WangGuangxin

jinchengchenghh

Thanks for your enhancement.
This looks like a bit complicated, could we reuse replaceWithExpressionTransformer to validate?

jinchengchenghh · 2025-10-21T16:09:13Z

-case class ColumnarPartialProjectExec(projectList: Seq[NamedExpression], child: SparkPlan)(
+case class ColumnarPartialProjectExec(
+    projectList: Seq[Expression],
+    child: SparkPlan,
    replacedAlias: Seq[Alias])


Why you move replacedAlias to argument list?

Because after this PR, SubQuery may be replaced. If replacedAlias is not put into the first argument list, then prepareSubqueries can't find the subquery in replacedAlias. So I put it in the first argument list.

The Alias is unique, with this change, the exprId is different, and the alias origins from projectList, the subquery should be found from it.

The Alias is unique, with this change, the exprId is different

Sorry, I can't understand this. Do you mean that the Alias's exprId is different from the original exprId?

the alias origins from projectList, the subquery should be found from it.

I get it. Thanks!

github-actions · 2025-10-24T09:51:23Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-10-27T07:10:19Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-10-28T02:21:04Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-10-28T06:36:59Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-10-28T09:33:03Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-10-29T02:58:39Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-10-29T08:53:49Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-10-30T02:28:55Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-10-30T06:41:39Z

Run Gluten Clickhouse CI on x86

FelixYBW · 2025-10-30T07:46:12Z

good enhancement!

github-actions · 2025-10-31T11:08:14Z

Run Gluten Clickhouse CI on x86

jiangjiangtian · 2025-11-03T03:31:07Z

@jinchengchenghh Please take a look again. It seems that the CI failure has nothing to do with this PR. Thanks!

jinchengchenghh

Thanks for your enhancement!

github-actions Bot added CORE works for Gluten Core VELOX DOCS labels Sep 5, 2025

jiangjiangtian force-pushed the find_unsupported_using_validation branch from 4997a7d to ac5a794 Compare September 5, 2025 10:10

jiangjiangtian force-pushed the find_unsupported_using_validation branch from ac5a794 to 75518fa Compare September 5, 2025 11:47

jiangjiangtian force-pushed the find_unsupported_using_validation branch from 75518fa to e859240 Compare September 5, 2025 12:05

jiangjiangtian force-pushed the find_unsupported_using_validation branch from e859240 to a16cb17 Compare September 8, 2025 09:33

jiangjiangtian force-pushed the find_unsupported_using_validation branch from a16cb17 to e5f4e2d Compare September 12, 2025 08:33

github-actions Bot removed the DOCS label Sep 12, 2025

jiangjiangtian force-pushed the find_unsupported_using_validation branch from e5f4e2d to 9d7475e Compare September 12, 2025 08:47

jiangjiangtian force-pushed the find_unsupported_using_validation branch from 9468a3c to d5dccbc Compare September 15, 2025 02:37

jiangjiangtian force-pushed the find_unsupported_using_validation branch from d5dccbc to 4f58ad3 Compare October 21, 2025 09:58

jiangjiangtian force-pushed the find_unsupported_using_validation branch from 4f58ad3 to fc3d30e Compare October 21, 2025 10:03

jinchengchenghh reviewed Oct 21, 2025

View reviewed changes

jiangjiangtian force-pushed the find_unsupported_using_validation branch from ea63e28 to fde57c7 Compare October 24, 2025 09:50

jiangjiangtian force-pushed the find_unsupported_using_validation branch from ed6e63c to 78af02c Compare October 28, 2025 06:36

jiangjiangtian force-pushed the find_unsupported_using_validation branch from 78af02c to fa50849 Compare October 28, 2025 09:32

jiangjiangtian force-pushed the find_unsupported_using_validation branch from a0ef2c0 to 84919be Compare October 30, 2025 06:41

jiangjiangtian mentioned this pull request Oct 31, 2025

[GLUTEN-10966][VL][FOLLOWUP] Fix UnsupportedOperationException when the result type of expressions in GenerateArrowProjection is array #10986

Merged

蒋添 and others added 10 commits October 31, 2025 19:07

Use backend to find all unsupported expression

f5edad5

fix

83288b2

add enterValidation and finishValidation

216c619

optimize

a69b117

fix unit tests

1dfa35c

fix

a8ce3f0

fix

99f37e7

fix

5866a94

fix

5a6c53b

fix

53ec176

jiangjiangtian force-pushed the find_unsupported_using_validation branch from 84919be to 53ec176 Compare October 31, 2025 11:07

jinchengchenghh approved these changes Nov 3, 2025

View reviewed changes

jinchengchenghh merged commit f4b7f25 into apache:main Nov 11, 2025
101 of 102 checks passed

Uh oh!

Conversation

jiangjiangtian commented Sep 5, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Uh oh!

github-actions Bot commented Sep 5, 2025

Uh oh!

github-actions Bot commented Sep 5, 2025

Uh oh!

jiangjiangtian commented Sep 5, 2025

Uh oh!

github-actions Bot commented Sep 5, 2025

Uh oh!

github-actions Bot commented Sep 5, 2025

Uh oh!

github-actions Bot commented Sep 5, 2025

Uh oh!

zhztheplayer commented Sep 5, 2025

Uh oh!

jiangjiangtian commented Sep 8, 2025

Uh oh!

github-actions Bot commented Sep 8, 2025

Uh oh!

github-actions Bot commented Sep 12, 2025

Uh oh!

github-actions Bot commented Sep 12, 2025

Uh oh!

github-actions Bot commented Sep 12, 2025

Uh oh!

github-actions Bot commented Sep 15, 2025

Uh oh!

github-actions Bot commented Oct 21, 2025

Uh oh!

github-actions Bot commented Oct 21, 2025

Uh oh!

zhztheplayer commented Oct 21, 2025

Uh oh!

jinchengchenghh left a comment

Choose a reason for hiding this comment

Uh oh!

jinchengchenghh Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

jiangjiangtian Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

jinchengchenghh Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

jiangjiangtian Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 24, 2025

Uh oh!

github-actions Bot commented Oct 27, 2025

Uh oh!

github-actions Bot commented Oct 28, 2025

Uh oh!

github-actions Bot commented Oct 28, 2025

Uh oh!

github-actions Bot commented Oct 28, 2025

Uh oh!

github-actions Bot commented Oct 29, 2025

Uh oh!

github-actions Bot commented Oct 29, 2025

Uh oh!

github-actions Bot commented Oct 30, 2025

Uh oh!

github-actions Bot commented Oct 30, 2025

Uh oh!

FelixYBW commented Oct 30, 2025

Uh oh!

github-actions Bot commented Oct 31, 2025

Uh oh!

jiangjiangtian commented Nov 3, 2025

Uh oh!

jinchengchenghh left a comment

jiangjiangtian commented Sep 5, 2025 •

edited by github-actions Bot

Loading

jiangjiangtian Oct 23, 2025 •

edited

Loading