[VL] Fix GlutenRemoveRedundantSortsSuite in Spark 4.x

[zml1206]

No description provided.

[Copilot]

Pull request overview

This PR updates the Spark 4.x (Spark 4.0 and 4.1) Gluten unit tests for redundant sort removal so they validate Gluten’s columnar sort operator (SortExecTransformer) and re-enables the suite in Velox backend test settings.

Changes:

• Rewrote GlutenRemoveRedundantSortsSuite (Spark 4.0/4.1) to assert redundant-sort behavior by counting SortExecTransformer nodes.
• Added SparkConf override to disable forced SHJ for the suite to stabilize join-related sort expectations.
• Enabled GlutenRemoveRedundantSortsSuite in Spark 4.0/4.1 VeloxTestSettings, including all Gluten-prefixed tests.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/GlutenRemoveRedundantSortsSuite.scala	Replaces Spark SortExec-based assumptions with SortExecTransformer-based assertions for Spark 4.1.
gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Re-enables the redundant-sorts suite for Spark 4.1 Velox UTs and includes Gluten tests.
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/execution/GlutenRemoveRedundantSortsSuite.scala	Same test rewrite as Spark 4.1, adapted for Spark 4.0 module.
gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Re-enables the redundant-sorts suite for Spark 4.0 Velox UTs and includes Gluten tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zml1206]

@zhouyuan @philo-he Should we disable tests that only verify optimization rules and are unrelated to Gluten? Examples include GlutenPlannerSuite, GlutenProjectedOrderingAndPartitioningSuite, GlutenRemoveRedundantProjectsSuite, GlutenRandomDataGeneratorSuite, GlutenSetCommandSuite, GlutenWholeStageCodegenSparkSubmitSuite, GlutenWholeStageCodegenSuite, GlutenExplainSuite, GlutenPythonDataSourceSuite, GlutenPythonUDFSuite, GlutenPythonUDTFSuite, and GlutenRowQueueSuite.

[zml1206]

@zhouyuan @philo-he Should we disable tests that only verify optimization rules and are unrelated to Gluten? Examples include GlutenPlannerSuite, GlutenProjectedOrderingAndPartitioningSuite, GlutenRemoveRedundantProjectsSuite, GlutenRandomDataGeneratorSuite, GlutenSetCommandSuite, GlutenWholeStageCodegenSparkSubmitSuite, GlutenWholeStageCodegenSuite, GlutenExplainSuite, GlutenPythonDataSourceSuite, GlutenPythonUDFSuite, GlutenPythonUDTFSuite, and GlutenRowQueueSuite.

One more thing: Spark 3.x does not enable these tests; I wonder if that is the reason.

[philo-he]

Looks good. Thanks for the continued efforts.

[philo-he]

@zhouyuan @philo-he Should we disable tests that only verify optimization rules and are unrelated to Gluten? Examples include GlutenPlannerSuite, GlutenProjectedOrderingAndPartitioningSuite, GlutenRemoveRedundantProjectsSuite, GlutenRandomDataGeneratorSuite, GlutenSetCommandSuite, GlutenWholeStageCodegenSparkSubmitSuite, GlutenWholeStageCodegenSuite, GlutenExplainSuite, GlutenPythonDataSourceSuite, GlutenPythonUDFSuite, GlutenPythonUDTFSuite, and GlutenRowQueueSuite.

One more thing: Spark 3.x does not enable these tests; I wonder if that is the reason.

As discussed offline, if Gluten is still involved, we could try enabling the tests. By introducing and enabling them in Gluten, our goal is to check whether any issues arise when integrated with Gluten, which is slightly different from the original test purpose. Thank you!