[SPARK-51439][SQL] Support SQL UDF with DEFAULT argument #50408

wengh · 2025-03-26T17:59:46Z

Continuing @allisonwang-db's work on #50373 and #49471

What changes were proposed in this pull request?

This PR adds support for DEFAULT arguments in SQL UDF. Examples:

CREATE FUNCTION foo1d1(a INT DEFAULT 10) RETURNS INT RETURN a;
SELECT foo1d1();   -- 10
SELECT foo1d1(20); -- 20

CREATE FUNCTION foo1d6(a INT, b INT DEFAULT 7) RETURNS TABLE(a INT, b INT) RETURN SELECT a, b;
SELECT * FROM foo1d6(5);    -- 5, 7
SELECT * FROM foo1d6(5, 2); -- 5, 2

See sql-udf.sql for more valid and invalid examples.

Why are the changes needed?

To support default arguments in SQL UDFs.

Does this PR introduce any user-facing change?

Yes. Now SQL UDFs support DEFAULT arguments.

A side effect of the grammar change is that some invalid function parameter definitions are now no longer rejected by the grammar, but instead rejected by the parser logic.

Examples:

-- multiple COMMENT or multiple NOT NULL
CREATE TEMPORARY FUNCTION foo(a INT COMMENT 'hello' COMMENT 'world') RETURNS INT RETURN a;

-- before:
[PARSE_SYNTAX_ERROR] Syntax error at or near 'COMMENT'. SQLSTATE: 42601
== SQL (line 2, position 1) ==
CREATE TEMPORARY FUNCTION foo(a INT COMMENT 'hello' COMMENT 'world') RETURNS INT RETURN a;
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-- after:
[CREATE_TABLE_COLUMN_DESCRIPTOR_DUPLICATE] CREATE TABLE column a specifies descriptor "COMMENT" more than once, which is invalid. SQLSTATE: 42710
== SQL (line 1, position 1) ==
CREATE TEMPORARY FUNCTION foo(a INT COMMENT 'hello' COMMENT 'world')...
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-- GENERATED ALWAYS AS
CREATE TEMPORARY FUNCTION foo(a INT GENERATED ALWAYS AS (1)) RETURNS INT RETURN a;

-- before:
[PARSE_SYNTAX_ERROR] Syntax error at or near 'GENERATED'. SQLSTATE: 42601
== SQL (line 2, position 1) ==
CREATE TEMPORARY FUNCTION foo(a INT GENERATED ALWAYS AS (1)) RETURNS INT RETURN a;
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-- after:
[INVALID_SQL_SYNTAX.CREATE_FUNC_WITH_GENERATED_COLUMNS_AS_PARAMETERS] Invalid SQL syntax: CREATE FUNCTION with generated columns as parameters is not allowed. SQLSTATE: 42000
== SQL (line 2, position 1) ==
CREATE TEMPORARY FUNCTION foo(a INT GENERATED ALWAYS AS (1)) RETURNS INT RETURN a;
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This doesn't change the behavior of existing valid SQL.

How was this patch tested?

End-to-end regression tests in sql-udf.sql and simple tests in SQLFunctionSuite.

Was this patch authored or co-authored using generative AI tooling?

No

wengh · 2025-03-27T15:55:05Z

@cloud-fan @allisonwang-db Please take a look at this follow up of [SPARK-50763][SQL] Add Analyzer rule for resolving SQL table functions!

allisonwang-db

Looks great!

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserInterface.scala

sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

wengh

explain refactor

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala

sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/UserDefinedFunction.scala

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/parsers.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ColumnDefinition.scala

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala

sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

This reverts commit c9330e9.

sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AbstractSqlParser.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala

cloud-fan

LGTM if CI passes

allisonwang-db · 2025-04-15T01:25:23Z

sql/api/src/main/scala/org/apache/spark/sql/types/StructType.scala

@@ -521,6 +521,7 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
 */
 @Stable
 object StructType extends AbstractDataType {
+  private[sql] val SQL_FUNCTION_DEFAULT_METADATA_KEY = "spark.sql.function.default"


Let's just keep 'default'?

allisonwang-db

Looks good!

allisonwang-db · 2025-04-15T01:27:02Z

sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala

-  private def getDDLDefault = getCurrentDefaultValue()
-    .map(" DEFAULT " + _)
-    .getOrElse("")
+  private def getDDLDefault =


shall we keep the original ordering? getCurrentDefaultValue().orElse(getParameterDefault())?

the ordering shouldn't matter because they are not set at the same time

allisonwang-db · 2025-04-15T01:30:11Z

sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala

   */
-  private[sql] def getDefault(): Option[String] = {


Let's keep the original function name if possible? I think the docstring itself makes it clear it's for function parameter. It would be good to reduce cosmetic changes given the PR is already large.

allisonwang-db · 2025-04-15T01:32:17Z

sql/core/src/test/scala/org/apache/spark/sql/execution/SQLFunctionSuite.scala

@@ -74,4 +74,17 @@ class SQLFunctionSuite extends QueryTest with SharedSparkSession {
          |""".stripMargin), Seq(Row(2), Row(4)))
    }
  }
+
+  test("SQL scalar function with default value") {


can we add another test with SQL table functions?

This test is for convenience when debugging. The SQL tests already check that it works with table functions.

cloud-fan · 2025-04-23T15:53:55Z

@wengh can you re-trigger the Github Action jobs? The link failure seems flaky.

wengh · 2025-04-23T22:35:25Z

@cloud-fan thanks for the reminder. tests are passing now.

cloud-fan · 2025-04-24T00:00:51Z

thanks, merging to master/4.0 (to make the SQL UDF feature complete)!

Continuing allisonwang-db's work on #50373 and #49471 This PR adds support for DEFAULT arguments in SQL UDF. Examples: ```sql CREATE FUNCTION foo1d1(a INT DEFAULT 10) RETURNS INT RETURN a; SELECT foo1d1(); -- 10 SELECT foo1d1(20); -- 20 CREATE FUNCTION foo1d6(a INT, b INT DEFAULT 7) RETURNS TABLE(a INT, b INT) RETURN SELECT a, b; SELECT * FROM foo1d6(5); -- 5, 7 SELECT * FROM foo1d6(5, 2); -- 5, 2 ``` See sql-udf.sql for more valid and invalid examples. To support default arguments in SQL UDFs. Yes. Now SQL UDFs support DEFAULT arguments. A side effect of the grammar change is that some invalid function parameter definitions are now no longer rejected by the grammar, but instead rejected by the parser logic. Examples: ```sql -- multiple COMMENT or multiple NOT NULL CREATE TEMPORARY FUNCTION foo(a INT COMMENT 'hello' COMMENT 'world') RETURNS INT RETURN a; -- before: [PARSE_SYNTAX_ERROR] Syntax error at or near 'COMMENT'. SQLSTATE: 42601 == SQL (line 2, position 1) == CREATE TEMPORARY FUNCTION foo(a INT COMMENT 'hello' COMMENT 'world') RETURNS INT RETURN a; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- after: [CREATE_TABLE_COLUMN_DESCRIPTOR_DUPLICATE] CREATE TABLE column a specifies descriptor "COMMENT" more than once, which is invalid. SQLSTATE: 42710 == SQL (line 1, position 1) == CREATE TEMPORARY FUNCTION foo(a INT COMMENT 'hello' COMMENT 'world')... ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``` ```sql -- GENERATED ALWAYS AS CREATE TEMPORARY FUNCTION foo(a INT GENERATED ALWAYS AS (1)) RETURNS INT RETURN a; -- before: [PARSE_SYNTAX_ERROR] Syntax error at or near 'GENERATED'. SQLSTATE: 42601 == SQL (line 2, position 1) == CREATE TEMPORARY FUNCTION foo(a INT GENERATED ALWAYS AS (1)) RETURNS INT RETURN a; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- after: [INVALID_SQL_SYNTAX.CREATE_FUNC_WITH_GENERATED_COLUMNS_AS_PARAMETERS] Invalid SQL syntax: CREATE FUNCTION with generated columns as parameters is not allowed. SQLSTATE: 42000 == SQL (line 2, position 1) == CREATE TEMPORARY FUNCTION foo(a INT GENERATED ALWAYS AS (1)) RETURNS INT RETURN a; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``` This doesn't change the behavior of existing valid SQL. End-to-end regression tests in `sql-udf.sql` and simple tests in `SQLFunctionSuite`. No Closes #50408 from wengh/sql-udf-default. Lead-authored-by: Haoyu Weng <[email protected]> Co-authored-by: Allison Wang <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

Continuing allisonwang-db's work on apache#50373 and apache#49471 ### What changes were proposed in this pull request? This PR adds support for DEFAULT arguments in SQL UDF. Examples: ```sql CREATE FUNCTION foo1d1(a INT DEFAULT 10) RETURNS INT RETURN a; SELECT foo1d1(); -- 10 SELECT foo1d1(20); -- 20 CREATE FUNCTION foo1d6(a INT, b INT DEFAULT 7) RETURNS TABLE(a INT, b INT) RETURN SELECT a, b; SELECT * FROM foo1d6(5); -- 5, 7 SELECT * FROM foo1d6(5, 2); -- 5, 2 ``` See sql-udf.sql for more valid and invalid examples. ### Why are the changes needed? To support default arguments in SQL UDFs. ### Does this PR introduce _any_ user-facing change? Yes. Now SQL UDFs support DEFAULT arguments. A side effect of the grammar change is that some invalid function parameter definitions are now no longer rejected by the grammar, but instead rejected by the parser logic. Examples: ```sql -- multiple COMMENT or multiple NOT NULL CREATE TEMPORARY FUNCTION foo(a INT COMMENT 'hello' COMMENT 'world') RETURNS INT RETURN a; -- before: [PARSE_SYNTAX_ERROR] Syntax error at or near 'COMMENT'. SQLSTATE: 42601 == SQL (line 2, position 1) == CREATE TEMPORARY FUNCTION foo(a INT COMMENT 'hello' COMMENT 'world') RETURNS INT RETURN a; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- after: [CREATE_TABLE_COLUMN_DESCRIPTOR_DUPLICATE] CREATE TABLE column a specifies descriptor "COMMENT" more than once, which is invalid. SQLSTATE: 42710 == SQL (line 1, position 1) == CREATE TEMPORARY FUNCTION foo(a INT COMMENT 'hello' COMMENT 'world')... ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``` ```sql -- GENERATED ALWAYS AS CREATE TEMPORARY FUNCTION foo(a INT GENERATED ALWAYS AS (1)) RETURNS INT RETURN a; -- before: [PARSE_SYNTAX_ERROR] Syntax error at or near 'GENERATED'. SQLSTATE: 42601 == SQL (line 2, position 1) == CREATE TEMPORARY FUNCTION foo(a INT GENERATED ALWAYS AS (1)) RETURNS INT RETURN a; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- after: [INVALID_SQL_SYNTAX.CREATE_FUNC_WITH_GENERATED_COLUMNS_AS_PARAMETERS] Invalid SQL syntax: CREATE FUNCTION with generated columns as parameters is not allowed. SQLSTATE: 42000 == SQL (line 2, position 1) == CREATE TEMPORARY FUNCTION foo(a INT GENERATED ALWAYS AS (1)) RETURNS INT RETURN a; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``` This doesn't change the behavior of existing valid SQL. ### How was this patch tested? End-to-end regression tests in `sql-udf.sql` and simple tests in `SQLFunctionSuite`. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#50408 from wengh/sql-udf-default. Lead-authored-by: Haoyu Weng <[email protected]> Co-authored-by: Allison Wang <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

github-actions bot added SQL CONNECT labels Mar 26, 2025

wengh changed the title ~~[WIP][SPARK-51439] Support SQL UDF with DEFAULT argument~~ [WIP][SPARK-51439][SQL] Support SQL UDF with DEFAULT argument Mar 26, 2025

wengh marked this pull request as ready for review March 27, 2025 15:05

wengh changed the title ~~[WIP][SPARK-51439][SQL] Support SQL UDF with DEFAULT argument~~ [SPARK-51439][SQL] Support SQL UDF with DEFAULT argument Mar 27, 2025

wengh force-pushed the sql-udf-default branch from ecfb9ce to aee7338 Compare March 27, 2025 16:01

allisonwang-db approved these changes Mar 28, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserInterface.scala Outdated Show resolved Hide resolved

wengh marked this pull request as draft March 28, 2025 00:31

allisonwang-db reviewed Mar 28, 2025

View reviewed changes

sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4 Outdated Show resolved Hide resolved

cloud-fan reviewed Mar 28, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala Outdated Show resolved Hide resolved

wengh force-pushed the sql-udf-default branch from 8f51e83 to 8c73a14 Compare March 28, 2025 16:48

wengh requested review from allisonwang-db and cloud-fan March 28, 2025 17:13

wengh marked this pull request as ready for review March 28, 2025 17:13

wengh commented Mar 28, 2025

View reviewed changes

cloud-fan reviewed Mar 30, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala Outdated Show resolved Hide resolved

zhengruifeng reviewed Mar 31, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/UserDefinedFunction.scala Show resolved Hide resolved

wengh requested review from cloud-fan and zhengruifeng April 1, 2025 00:22

cloud-fan reviewed Apr 1, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Apr 1, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Apr 1, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/parsers.scala Outdated Show resolved Hide resolved

wengh requested a review from cloud-fan April 2, 2025 23:03

cloud-fan reviewed Apr 5, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ColumnDefinition.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Apr 5, 2025

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Apr 8, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala Show resolved Hide resolved

cloud-fan reviewed Apr 8, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala Outdated Show resolved Hide resolved

cloud-fan approved these changes Apr 8, 2025

View reviewed changes

fix

82bd38e

wengh added 6 commits April 8, 2025 14:48

reuse parseColDefinition

645e3e7

remove unnecessary metadata

db81e02

rename getDefault

d856283

assert no generation expression

a963268

Revert "rename getDefault"

0b30c23

This reverts commit c9330e9.

refactor

e4a6883

wengh force-pushed the sql-udf-default branch from 275f637 to e4a6883 Compare April 8, 2025 21:51

reject column constraints

c7b2e0d