-
Notifications
You must be signed in to change notification settings - Fork 247
Open
Labels
enhancementNew feature or requestNew feature or request
Description
What is the problem the feature request solves?
Some data types seem to be missing in the current implementation of try_*.
I might be a bit late to the party 😅 but I didn’t find an open issue tracking this.
Comes from here #2073
Describe the potential solution
Support for more data types, similar to how it was done in Sail.
Here are some example tests from Spark showing expected behavior:
>>> df = spark.sql("""
... SELECT
... try_add(DATE '2015-09-30', 1) as d1,
... try_add(DATE '2000-01-01', 366) as d2,
... try_add(DATE '2021-01-01', 1) as d3,
... try_add(NULL, 100) as d4
... """)
>>> df.show()
+----------+----------+----------+----+
| d1| d2| d3| d4|
+----------+----------+----------+----+
|2015-10-01|2001-01-01|2021-01-02|NULL|
+----------+----------+----------+----+
>>> df = spark.sql("""
... SELECT
... try_add(DATE '2015-01-31', INTERVAL 1 MONTH) as d1,
... try_add(DATE '2020-02-29', INTERVAL 12 MONTH) as d2,
... try_add(NULL, INTERVAL 3 MONTH) as d3
... """)
>>> df.show()
+----------+----------+----+
| d1| d2| d3|
+----------+----------+----+
|2015-02-28|2021-02-28|NULL|
+----------+----------+----+
>>> df = spark.sql("""
... SELECT
... try_add(DATE '2000-07-31', INTERVAL -1 MONTH) as d1,
... try_add(DATE '2021-01-31', INTERVAL -1 MONTH) as d2
... """)
>>> df.show()
+----------+----------+
| d1| d2|
+----------+----------+
|2000-06-30|2020-12-31|
+----------+----------+
>>> df = spark.sql("""
... SELECT
... try_add(INTERVAL '1' YEAR, INTERVAL '2' YEAR) as result
... """)
>>> spark.conf.set("spark.sql.session.timeZone", "UTC")
>>> df = spark.sql("""
... SELECT
... try_add(TIMESTAMP '2021-01-01 00:00:00', INTERVAL 1 DAY) as result
... """)
>>> df.show(truncate=False)
+-------------------+
|result |
+-------------------+
|2021-01-02 00:00:00|
+-------------------+Additional context
Implementation in Sail for reference:
lakehq/sail#635
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request