Skip to content

Commit 3ca6450

Browse files
authored
Adding CONVERT_TZ and DATETIME functions to SQL and PPL (opensearch-project#848)
* Added `CONVERT_TZ` to the PPL lexer/parser and the SQL lexer/parser. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added `convert_tz` to the BuiltinFunctionName.java. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added convert_tz to DateTimeFunction.java register, private FunctionResolver convert_tz and ExprValue exprConvert_TZ. It implements the functionality for converting between time zones. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added IT PPL and SQL tests for various conditions including time zones that are outside the existing range (consistent with MySQL standard). Added implementation for convert_tz consistent with MySQL implementation. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added DATETIME to OpenSearchPPLParser.g4 and OpenSearchSQLParser.g4. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Rebase merge conflict resolution. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added ppl doctest for convert_tz Signed-off-by: MitchellGale-BitQuill <[email protected]> * Completed implementation for datetime and convert_Tz. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Removed SQL test from PPL IT test Signed-off-by: MitchellGale-BitQuill <[email protected]> * Removed redundant convert to string. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Fixed doctests and reverted changes to adddate function in DateTimeFunctionTest.java. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Fixed doctest. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Fixed doctest. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added additional integration tests for PPL and SQL tests for convert_tz. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added null for timezones outside of basic range for DATETIME. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added test cases for null with the datetime function. Made DateTime function call conevrt_tz function. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Seperated out test from DateTimeFunction.java. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Updated tests. Fixed exception to be less general. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Removed changes. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Fixed rel timezone issue. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added support for non valid datetime to return null for convert_tz. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Made exception more verbose. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Removed unneeded format changes in DateTimeFunction.java. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added more doctests. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Removed formatting changes. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Reverting sql/ppl DateTimeFunctionsIT.java. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Reverted changes to DateTimeFunctionTest.java Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added more information about invalid date for convert_tz Signed-off-by: MitchellGale-BitQuill <[email protected]> * Converted "from Field" and "To Field" to use "Fieldn" where n is the field number. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added date validation. Added test cases in IT to cover cases. Added test in ConvertTZTest.java. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Fixed date validation function. Broke up some unit tests. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Fixed formatting. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added DateTime tests, broke up functions. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added space in DateTimeTest.java. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Tidied up code and tests. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Fixed local date time rst test. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Removed nested try/catch exceptions. Signed-off-by: MitchellGale-BitQuill <[email protected]> * removed exprConvertTZ function call from within try catch statement. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Reverted change from parse localdate Signed-off-by: MitchellGale-BitQuill <[email protected]> * Removed extra casting around fromTz variable. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Updated wording for functions.rst Signed-off-by: MitchellGale-BitQuill <[email protected]> * Updated wording for datetime.rst to describe null for conert_tz Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added more test cases for functions.rst Signed-off-by: MitchellGale-BitQuill <[email protected]> * Fixed doctests. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Removed extra import. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Renamed isValidTimeZone function to isValidMySqlTimeZoneId. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Has ExprDatetimeValue doing work for exprConvertTZ call. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Deleted fromTZ Signed-off-by: MitchellGale-BitQuill <[email protected]> * Moved fixed variables to top of class. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added Null to support of exprDateTime. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added missing expr functions for makedate/time Signed-off-by: MitchellGale-BitQuill <[email protected]> * cleaning up after rebase merge. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Rebase merge conflict resolution. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Fixed missing DATETIME in SQL Parser. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Fixed IT test. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Addressed PR comments Signed-off-by: MitchellGale-BitQuill <[email protected]> * Added missing variables after rebase Signed-off-by: MitchellGale-BitQuill <[email protected]> * Fixed checkstyle errors after rebase. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Moved formatter for date time over. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Changed function resolved to default Signed-off-by: MitchellGale-BitQuill <[email protected]> * Removed unneeded code Signed-off-by: MitchellGale-BitQuill <[email protected]> Signed-off-by: MitchellGale-BitQuill <[email protected]>
1 parent 10e44ee commit 3ca6450

File tree

20 files changed

+1743
-39
lines changed

20 files changed

+1743
-39
lines changed

core/src/main/java/org/opensearch/sql/data/model/ExprDatetimeValue.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ public class ExprDatetimeValue extends AbstractExprValue {
3333

3434
static {
3535
FORMATTER_VARIABLE_NANOS = new DateTimeFormatterBuilder()
36-
.appendPattern("yyyy-MM-dd HH:mm:ss")
36+
.appendPattern("uuuu-MM-dd HH:mm:ss[xxx]")
3737
.appendFraction(
3838
ChronoField.NANO_OF_SECOND,
3939
MIN_FRACTION_SECONDS,

core/src/main/java/org/opensearch/sql/expression/DSL.java

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -271,10 +271,18 @@ public FunctionExpression adddate(Expression... expressions) {
271271
return function(BuiltinFunctionName.ADDDATE, expressions);
272272
}
273273

274+
public FunctionExpression convert_tz(Expression... expressions) {
275+
return function(BuiltinFunctionName.CONVERT_TZ, expressions);
276+
}
277+
274278
public FunctionExpression date(Expression... expressions) {
275279
return function(BuiltinFunctionName.DATE, expressions);
276280
}
277281

282+
public FunctionExpression datetime(Expression... expressions) {
283+
return function(BuiltinFunctionName.DATETIME, expressions);
284+
}
285+
278286
public FunctionExpression date_add(Expression... expressions) {
279287
return function(BuiltinFunctionName.DATE_ADD, expressions);
280288
}

core/src/main/java/org/opensearch/sql/expression/datetime/DateTimeFunction.java

Lines changed: 130 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,20 +22,24 @@
2222
import static org.opensearch.sql.utils.DateTimeFormatters.DATE_FORMATTER_SHORT_YEAR;
2323
import static org.opensearch.sql.utils.DateTimeFormatters.DATE_TIME_FORMATTER_LONG_YEAR;
2424
import static org.opensearch.sql.utils.DateTimeFormatters.DATE_TIME_FORMATTER_SHORT_YEAR;
25+
import static org.opensearch.sql.utils.DateTimeFormatters.DATE_TIME_FORMATTER_STRICT_WITH_TZ;
2526

2627
import java.math.BigDecimal;
2728
import java.math.RoundingMode;
2829
import java.text.DecimalFormat;
30+
import java.time.DateTimeException;
2931
import java.time.Instant;
3032
import java.time.LocalDate;
3133
import java.time.LocalDateTime;
3234
import java.time.LocalTime;
3335
import java.time.ZoneId;
3436
import java.time.ZoneOffset;
37+
import java.time.ZonedDateTime;
3538
import java.time.format.DateTimeFormatter;
3639
import java.time.format.DateTimeParseException;
3740
import java.time.format.TextStyle;
3841
import java.util.Locale;
42+
import java.util.TimeZone;
3943
import java.util.concurrent.TimeUnit;
4044
import javax.annotation.Nullable;
4145
import lombok.experimental.UtilityClass;
@@ -50,11 +54,13 @@
5054
import org.opensearch.sql.data.model.ExprTimestampValue;
5155
import org.opensearch.sql.data.model.ExprValue;
5256
import org.opensearch.sql.data.type.ExprCoreType;
57+
import org.opensearch.sql.exception.ExpressionEvaluationException;
5358
import org.opensearch.sql.expression.function.BuiltinFunctionName;
5459
import org.opensearch.sql.expression.function.BuiltinFunctionRepository;
5560
import org.opensearch.sql.expression.function.DefaultFunctionResolver;
5661
import org.opensearch.sql.expression.function.FunctionName;
5762
import org.opensearch.sql.expression.function.FunctionResolver;
63+
import org.opensearch.sql.utils.DateTimeUtils;
5864

5965
/**
6066
* The definition of date and time functions.
@@ -63,7 +69,6 @@
6369
*/
6470
@UtilityClass
6571
public class DateTimeFunction {
66-
6772
// The number of days from year zero to year 1970.
6873
private static final Long DAYS_0000_TO_1970 = (146097 * 5L) - (30L * 365L + 7L);
6974

@@ -78,7 +83,9 @@ public class DateTimeFunction {
7883
*/
7984
public void register(BuiltinFunctionRepository repository) {
8085
repository.register(adddate());
86+
repository.register(convert_tz());
8187
repository.register(date());
88+
repository.register(datetime());
8289
repository.register(date_add());
8390
repository.register(date_sub());
8491
repository.register(day());
@@ -214,6 +221,21 @@ private DefaultFunctionResolver adddate() {
214221
return add_date(BuiltinFunctionName.ADDDATE.getName());
215222
}
216223

224+
/**
225+
* Converts date/time from a specified timezone to another specified timezone.
226+
* The supported signatures:
227+
* (DATETIME, STRING, STRING) -> DATETIME
228+
* (STRING, STRING, STRING) -> DATETIME
229+
*/
230+
private DefaultFunctionResolver convert_tz() {
231+
return define(BuiltinFunctionName.CONVERT_TZ.getName(),
232+
impl(nullMissingHandling(DateTimeFunction::exprConvertTZ),
233+
DATETIME, DATETIME, STRING, STRING),
234+
impl(nullMissingHandling(DateTimeFunction::exprConvertTZ),
235+
DATETIME, STRING, STRING, STRING)
236+
);
237+
}
238+
217239
/**
218240
* Extracts the date part of a date and time value.
219241
* Also to construct a date type. The supported signatures:
@@ -227,6 +249,21 @@ private DefaultFunctionResolver date() {
227249
impl(nullMissingHandling(DateTimeFunction::exprDate), DATE, TIMESTAMP));
228250
}
229251

252+
/**
253+
* Specify a datetime with time zone field and a time zone to convert to.
254+
* Returns a local date time.
255+
* (STRING, STRING) -> DATETIME
256+
* (STRING) -> DATETIME
257+
*/
258+
private FunctionResolver datetime() {
259+
return define(BuiltinFunctionName.DATETIME.getName(),
260+
impl(nullMissingHandling(DateTimeFunction::exprDateTime),
261+
DATETIME, STRING, STRING),
262+
impl(nullMissingHandling(DateTimeFunction::exprDateTimeNoTimezone),
263+
DATETIME, STRING)
264+
);
265+
}
266+
230267
private DefaultFunctionResolver date_add() {
231268
return add_date(BuiltinFunctionName.DATE_ADD.getName());
232269
}
@@ -570,6 +607,42 @@ private ExprValue exprAddDateDays(ExprValue date, ExprValue days) {
570607
: exprValue);
571608
}
572609

610+
/**
611+
* CONVERT_TZ function implementation for ExprValue.
612+
* Returns null for time zones outside of +13:00 and -12:00.
613+
*
614+
* @param startingDateTime ExprValue of DateTime that is being converted from
615+
* @param fromTz ExprValue of time zone, representing the time to convert from.
616+
* @param toTz ExprValue of time zone, representing the time to convert to.
617+
* @return DateTime that has been converted to the to_tz timezone.
618+
*/
619+
private ExprValue exprConvertTZ(ExprValue startingDateTime, ExprValue fromTz, ExprValue toTz) {
620+
if (startingDateTime.type() == ExprCoreType.STRING) {
621+
startingDateTime = exprDateTimeNoTimezone(startingDateTime);
622+
}
623+
try {
624+
ZoneId convertedFromTz = ZoneId.of(fromTz.stringValue());
625+
ZoneId convertedToTz = ZoneId.of(toTz.stringValue());
626+
627+
// isValidMySqlTimeZoneId checks if the timezone is within the range accepted by
628+
// MySQL standard.
629+
if (!DateTimeUtils.isValidMySqlTimeZoneId(convertedFromTz)
630+
|| !DateTimeUtils.isValidMySqlTimeZoneId(convertedToTz)) {
631+
return ExprNullValue.of();
632+
}
633+
ZonedDateTime zonedDateTime =
634+
startingDateTime.datetimeValue().atZone(convertedFromTz);
635+
return new ExprDatetimeValue(
636+
zonedDateTime.withZoneSameInstant(convertedToTz).toLocalDateTime());
637+
638+
639+
// Catches exception for invalid timezones.
640+
// ex. "+0:00" is an invalid timezone and would result in this exception being thrown.
641+
} catch (ExpressionEvaluationException | DateTimeException e) {
642+
return ExprNullValue.of();
643+
}
644+
}
645+
573646
/**
574647
* Date implementation for ExprValue.
575648
*
@@ -584,6 +657,62 @@ private ExprValue exprDate(ExprValue exprValue) {
584657
}
585658
}
586659

660+
/**
661+
* DateTime implementation for ExprValue.
662+
*
663+
* @param dateTime ExprValue of String type.
664+
* @param timeZone ExprValue of String type (or null).
665+
* @return ExprValue of date type.
666+
*/
667+
private ExprValue exprDateTime(ExprValue dateTime, ExprValue timeZone) {
668+
String defaultTimeZone = TimeZone.getDefault().getID();
669+
670+
671+
try {
672+
LocalDateTime ldtFormatted =
673+
LocalDateTime.parse(dateTime.stringValue(), DATE_TIME_FORMATTER_STRICT_WITH_TZ);
674+
if (timeZone.isNull()) {
675+
return new ExprDatetimeValue(ldtFormatted);
676+
}
677+
678+
// Used if datetime field is invalid format.
679+
} catch (DateTimeParseException e) {
680+
return ExprNullValue.of();
681+
}
682+
683+
ExprValue convertTZResult;
684+
ExprDatetimeValue ldt;
685+
String toTz;
686+
687+
try {
688+
ZonedDateTime zdtWithZoneOffset =
689+
ZonedDateTime.parse(dateTime.stringValue(), DATE_TIME_FORMATTER_STRICT_WITH_TZ);
690+
ZoneId fromTZ = zdtWithZoneOffset.getZone();
691+
692+
ldt = new ExprDatetimeValue(zdtWithZoneOffset.toLocalDateTime());
693+
toTz = String.valueOf(fromTZ);
694+
} catch (DateTimeParseException e) {
695+
ldt = new ExprDatetimeValue(dateTime.stringValue());
696+
toTz = defaultTimeZone;
697+
}
698+
convertTZResult = exprConvertTZ(
699+
ldt,
700+
new ExprStringValue(toTz),
701+
timeZone);
702+
703+
return convertTZResult;
704+
}
705+
706+
/**
707+
* DateTime implementation for ExprValue without a timezone to convert to.
708+
*
709+
* @param dateTime ExprValue of String type.
710+
* @return ExprValue of date type.
711+
*/
712+
private ExprValue exprDateTimeNoTimezone(ExprValue dateTime) {
713+
return exprDateTime(dateTime, ExprNullValue.of());
714+
}
715+
587716
/**
588717
* Name of the Weekday implementation for ExprValue.
589718
*

core/src/main/java/org/opensearch/sql/expression/function/BuiltinFunctionName.java

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,9 @@ public enum BuiltinFunctionName {
5858
* Date and Time Functions.
5959
*/
6060
ADDDATE(FunctionName.of("adddate")),
61+
CONVERT_TZ(FunctionName.of("convert_tz")),
6162
DATE(FunctionName.of("date")),
63+
DATETIME(FunctionName.of("datetime")),
6264
DATE_ADD(FunctionName.of("date_add")),
6365
DATE_SUB(FunctionName.of("date_sub")),
6466
DAY(FunctionName.of("day")),

core/src/main/java/org/opensearch/sql/utils/DateTimeFormatters.java

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,4 +162,10 @@ public class DateTimeFormatters {
162162
.appendPattern("MMddHHmmss")
163163
.toFormatter()
164164
.withResolverStyle(ResolverStyle.STRICT);
165+
166+
public static final DateTimeFormatter DATE_TIME_FORMATTER_STRICT_WITH_TZ =
167+
new DateTimeFormatterBuilder()
168+
.appendPattern("uuuu-MM-dd HH:mm:ss[xxx]")
169+
.toFormatter()
170+
.withResolverStyle(ResolverStyle.STRICT);
165171
}

core/src/main/java/org/opensearch/sql/utils/DateTimeUtils.java

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
package org.opensearch.sql.utils;
77

88
import java.time.Instant;
9+
import java.time.LocalDateTime;
910
import java.time.ZoneId;
1011
import java.time.ZonedDateTime;
1112
import lombok.experimental.UtilityClass;
@@ -84,4 +85,34 @@ public static long roundYear(long utcMillis, int interval) {
8485
return initDateTime.plusYears(yearToAdd).toInstant().toEpochMilli();
8586
}
8687

88+
89+
/**
90+
* isValidMySqlTimeZoneId for timezones which match timezone the range set by MySQL.
91+
*
92+
* @param zone ZoneId of ZoneId type.
93+
* @return Boolean.
94+
*/
95+
public Boolean isValidMySqlTimeZoneId(ZoneId zone) {
96+
String timeZoneMax = "+14:00";
97+
String timeZoneMin = "-13:59";
98+
String timeZoneZero = "+00:00";
99+
100+
ZoneId maxTz = ZoneId.of(timeZoneMax);
101+
ZoneId minTz = ZoneId.of(timeZoneMin);
102+
ZoneId defaultTz = ZoneId.of(timeZoneZero);
103+
104+
ZonedDateTime defaultDateTime = LocalDateTime.of(2000, 1, 2, 12, 0).atZone(defaultTz);
105+
106+
ZonedDateTime maxTzValidator =
107+
defaultDateTime.withZoneSameInstant(maxTz).withZoneSameLocal(defaultTz);
108+
ZonedDateTime minTzValidator =
109+
defaultDateTime.withZoneSameInstant(minTz).withZoneSameLocal(defaultTz);
110+
ZonedDateTime passedTzValidator =
111+
defaultDateTime.withZoneSameInstant(zone).withZoneSameLocal(defaultTz);
112+
113+
return (passedTzValidator.isBefore(maxTzValidator)
114+
|| passedTzValidator.isEqual(maxTzValidator))
115+
&& (passedTzValidator.isAfter(minTzValidator)
116+
|| passedTzValidator.isEqual(minTzValidator));
117+
}
87118
}

0 commit comments

Comments
 (0)