Skip to content

Commit

Permalink
Add functions DATE_ADD, DATE_SUB, TIMESTAMPADD, TIMESTAMPDIFF, CURREN…
Browse files Browse the repository at this point in the history 
…T_TIMEZONE and UTC_TIMESTAMP (opensearch-project#839)

* Add functions DATE_ADD, DATE_SUB, TIMESTAMPADD, TIMESTAMPDIFF, CURRENT_TIMEZONE and UTC_TIMESTAMP

Signed-off-by: Lantao Jin <[email protected]>

* remove tokens and refactor

Signed-off-by: Lantao Jin <[email protected]>

* address comment'

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
  • Loading branch information
@LantaoJin
LantaoJin authored Oct 31, 2024
1 parent c590d29 commit 99aab5a
Show file tree
Hide file tree
Showing 11 changed files with 572 additions and 60 deletions.
135 changes: 133 additions & 2 deletions docs/ppl-lang/functions/ppl-datetime.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Argument type: DATE, LONG

(DATE, LONG) -> DATE

Antonyms: `SUBDATE`_
Antonyms: `SUBDATE`

Example:

Expand Down Expand Up @@ -795,7 +795,7 @@ Argument type: DATE/TIMESTAMP, LONG

(DATE, LONG) -> DATE

Antonyms: `ADDDATE`_
Antonyms: `ADDDATE`

Example:

Expand Down Expand Up @@ -982,3 +982,134 @@ Example:
+----------------------------+


### `DATE_ADD`

**Description:**

Usage: date_add(date, INTERVAL expr unit) adds the interval expr to date.

Argument type: DATE, INTERVAL

Return type: DATE

Antonyms: `DATE_SUB`

Example::

os> source=people | eval `'2020-08-26' + 1d` = DATE_ADD(DATE('2020-08-26'), INTERVAL 1 DAY) | fields `'2020-08-26' + 1d`
fetched rows / total rows = 1/1
+---------------------+
| '2020-08-26' + 1d |
|---------------------+
| 2020-08-27 |
+---------------------+


### `DATE_SUB`

**Description:**

Usage: date_sub(date, INTERVAL expr unit) subtracts the interval expr from date.

Argument type: DATE, INTERVAL

Return type: DATE

Antonyms: `DATE_ADD`

Example::

os> source=people | eval `'2008-01-02' - 31d` = DATE_SUB(DATE('2008-01-02'), INTERVAL 31 DAY) | fields `'2008-01-02' - 31d`
fetched rows / total rows = 1/1
+---------------------+
| '2008-01-02' - 31d |
|---------------------+
| 2007-12-02 |
+---------------------+


### `TIMESTAMPADD`

**Description:**

Usage: Returns a TIMESTAMP value based on a passed in DATE/TIMESTAMP/STRING argument and an INTERVAL and INTEGER argument which determine the amount of time to be added.
If the third argument is a STRING, it must be formatted as a valid TIMESTAMP.
If the third argument is a DATE, it will be automatically converted to a TIMESTAMP.

Argument type: INTERVAL, INTEGER, DATE/TIMESTAMP/STRING

INTERVAL must be one of the following tokens: [SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, QUARTER, YEAR]

Examples::

os> source=people | eval `TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00')` = TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00') | eval `TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00')` = TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00') | fields `TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00')`, `TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00')`
fetched rows / total rows = 1/1
+----------------------------------------------+--------------------------------------------------+
| TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00') | TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00') |
|----------------------------------------------+--------------------------------------------------|
| 2000-01-18 00:00:00 | 1999-10-01 00:00:00 |
+----------------------------------------------+--------------------------------------------------+


### `TIMESTAMPDIFF`

**Description:**

Usage: TIMESTAMPDIFF(interval, start, end) returns the difference between the start and end date/times in interval units.
Arguments will be automatically converted to a ]TIMESTAMP when appropriate.
Any argument that is a STRING must be formatted as a valid TIMESTAMP.

Argument type: INTERVAL, DATE/TIMESTAMP/STRING, DATE/TIMESTAMP/STRING

INTERVAL must be one of the following tokens: [SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, QUARTER, YEAR]

Examples::

os> source=people | eval `TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00')` = TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00') | eval `TIMESTAMPDIFF(SECOND, timestamp('1997-01-01 00:00:23'), timestamp('1997-01-01 00:00:00'))` = TIMESTAMPDIFF(SECOND, timestamp('1997-01-01 00:00:23'), timestamp('1997-01-01 00:00:00')) | fields `TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00')`, `TIMESTAMPDIFF(SECOND, timestamp('1997-01-01 00:00:23'), timestamp('1997-01-01 00:00:00'))`
fetched rows / total rows = 1/1
+-------------------------------------------------------------------+-------------------------------------------------------------------------------------------+
| TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00') | TIMESTAMPDIFF(SECOND, timestamp('1997-01-01 00:00:23'), timestamp('1997-01-01 00:00:00')) |
|-------------------------------------------------------------------+-------------------------------------------------------------------------------------------|
| 4 | -23 |
+-------------------------------------------------------------------+-------------------------------------------------------------------------------------------+


### `UTC_TIMESTAMP`

**Description:**

Returns the current UTC timestamp as a value in 'YYYY-MM-DD hh:mm:ss'.

Return type: TIMESTAMP

Specification: UTC_TIMESTAMP() -> TIMESTAMP

Example::

> source=people | eval `UTC_TIMESTAMP()` = UTC_TIMESTAMP() | fields `UTC_TIMESTAMP()`
fetched rows / total rows = 1/1
+---------------------+
| UTC_TIMESTAMP() |
|---------------------|
| 2022-10-03 17:54:28 |
+---------------------+


### `CURRENT_TIMEZONE`

**Description:**

Returns the current local timezone.

Return type: STRING

Example::

> source=people | eval `CURRENT_TIMEZONE()` = CURRENT_TIMEZONE() | fields `CURRENT_TIMEZONE()`
fetched rows / total rows = 1/1
+------------------------+
| CURRENT_TIMEZONE() |
|------------------------|
| America/Chicago |
+------------------------+

154 changes: 111 additions & 43 deletions ...on/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLBuiltInDateTimeFunctionITSuite.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,117 @@ class FlintSparkPPLBuiltInDateTimeFunctionITSuite
comparePlans(logicalPlan, expectedPlan, checkAnalysis = false)
}

test("test DATE_ADD") {
val frame1 = sql(s"""
| source = $testTable | eval `'2020-08-26' + 2d` = DATE_ADD(DATE('2020-08-26'), INTERVAL 2 DAY)
| | fields `'2020-08-26' + 2d` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2020-08-28"))), frame1)

val frame2 = sql(s"""
| source = $testTable | eval `'2020-08-26' - 2d` = DATE_ADD(DATE('2020-08-26'), INTERVAL -2 DAY)
| | fields `'2020-08-26' - 2d` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2020-08-24"))), frame2)

val frame3 = sql(s"""
| source = $testTable | eval `'2020-08-26' + 2m` = DATE_ADD(DATE('2020-08-26'), INTERVAL 2 MONTH)
| | fields `'2020-08-26' + 2m` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2020-10-26"))), frame3)

val frame4 = sql(s"""
| source = $testTable | eval `'2020-08-26' + 2y` = DATE_ADD(DATE('2020-08-26'), INTERVAL 2 YEAR)
| | fields `'2020-08-26' + 2y` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2022-08-26"))), frame4)

val ex = intercept[AnalysisException](sql(s"""
| source = $testTable | eval `'2020-08-26 01:01:01' + 2h` = DATE_ADD(TIMESTAMP('2020-08-26 01:01:01'), INTERVAL 2 HOUR)
| | fields `'2020-08-26 01:01:01' + 2h` | head 1
| """.stripMargin))
assert(ex.getMessage.contains("""Parameter 1 requires the "DATE" type"""))
}

test("test DATE_SUB") {
val frame1 = sql(s"""
| source = $testTable | eval `'2020-08-26' - 2d` = DATE_SUB(DATE('2020-08-26'), INTERVAL 2 DAY)
| | fields `'2020-08-26' - 2d` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2020-08-24"))), frame1)

val frame2 = sql(s"""
| source = $testTable | eval `'2020-08-26' + 2d` = DATE_SUB(DATE('2020-08-26'), INTERVAL -2 DAY)
| | fields `'2020-08-26' + 2d` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2020-08-28"))), frame2)

val frame3 = sql(s"""
| source = $testTable | eval `'2020-08-26' - 2m` = DATE_SUB(DATE('2020-08-26'), INTERVAL 12 MONTH)
| | fields `'2020-08-26' - 2m` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2019-08-26"))), frame3)

val frame4 = sql(s"""
| source = $testTable | eval `'2020-08-26' - 2y` = DATE_SUB(DATE('2020-08-26'), INTERVAL 2 YEAR)
| | fields `'2020-08-26' - 2y` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2018-08-26"))), frame4)

val ex = intercept[AnalysisException](sql(s"""
| source = $testTable | eval `'2020-08-26 01:01:01' - 2h` = DATE_SUB(TIMESTAMP('2020-08-26 01:01:01'), INTERVAL 2 HOUR)
| | fields `'2020-08-26 01:01:01' - 2h` | head 1
| """.stripMargin))
assert(ex.getMessage.contains("""Parameter 1 requires the "DATE" type"""))
}

test("test TIMESTAMPADD") {
val frame = sql(s"""
| source = $testTable
| | eval `TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00')` = TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00')
| | eval `TIMESTAMPADD(DAY, 17, TIMESTAMP('2000-01-01 00:00:00'))` = TIMESTAMPADD(DAY, 17, TIMESTAMP('2000-01-01 00:00:00'))
| | eval `TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00')` = TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00')
| | fields `TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00')`, `TIMESTAMPADD(DAY, 17, TIMESTAMP('2000-01-01 00:00:00'))`, `TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00')`
| | head 1
| """.stripMargin)
assertSameRows(
Seq(
Row(
Timestamp.valueOf("2000-01-18 00:00:00"),
Timestamp.valueOf("2000-01-18 00:00:00"),
Timestamp.valueOf("1999-10-01 00:00:00"))),
frame)
}

test("test TIMESTAMPDIFF") {
val frame = sql(s"""
| source = $testTable
| | eval `TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00')` = TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00')
| | eval `TIMESTAMPDIFF(SECOND, TIMESTAMP('2000-01-01 00:00:23'), TIMESTAMP('2000-01-01 00:00:00'))` = TIMESTAMPDIFF(SECOND, TIMESTAMP('2000-01-01 00:00:23'), TIMESTAMP('2000-01-01 00:00:00'))
| | fields `TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00')`, `TIMESTAMPDIFF(SECOND, TIMESTAMP('2000-01-01 00:00:23'), TIMESTAMP('2000-01-01 00:00:00'))`
| | head 1
| """.stripMargin)
assertSameRows(Seq(Row(4, -23)), frame)
}

test("test CURRENT_TIMEZONE") {
val frame = sql(s"""
| source = $testTable
| | eval `CURRENT_TIMEZONE` = CURRENT_TIMEZONE()
| | fields `CURRENT_TIMEZONE`
| """.stripMargin)
assert(frame.collect().length > 0)
}

test("test UTC_TIMESTAMP") {
val frame = sql(s"""
| source = $testTable
| | eval `UTC_TIMESTAMP` = UTC_TIMESTAMP()
| | fields `UTC_TIMESTAMP`
| """.stripMargin)
assert(frame.collect().length > 0)
}

test("test hour, minute, second, HOUR_OF_DAY, MINUTE_OF_HOUR") {
val frame = sql(s"""
| source = $testTable
Expand Down Expand Up @@ -284,24 +395,6 @@ class FlintSparkPPLBuiltInDateTimeFunctionITSuite
assert(ex.getMessage.contains("ADDTIME is not a builtin function of PPL"))
}

test("test DATE_ADD is not supported") {
val ex = intercept[UnsupportedOperationException](sql(s"""
| source = $testTable
| | eval `DATE_ADD` = DATE_ADD()
| | fields DATE_ADD | head 1
| """.stripMargin))
assert(ex.getMessage.contains("DATE_ADD is not a builtin function of PPL"))
}

test("test DATE_SUB is not supported") {
val ex = intercept[UnsupportedOperationException](sql(s"""
| source = $testTable
| | eval `DATE_SUB` = DATE_SUB()
| | fields DATE_SUB | head 1
| """.stripMargin))
assert(ex.getMessage.contains("DATE_SUB is not a builtin function of PPL"))
}

test("test DATETIME is not supported") {
val ex = intercept[UnsupportedOperationException](sql(s"""
| source = $testTable
Expand Down Expand Up @@ -445,22 +538,6 @@ class FlintSparkPPLBuiltInDateTimeFunctionITSuite
assert(ex.getMessage.contains("TIMEDIFF is not a builtin function of PPL"))
}

test("test TIMESTAMPADD is not supported") {
intercept[Exception](sql(s"""
| source = $testTable
| | eval `TIMESTAMPADD` = TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00')
| | fields TIMESTAMPADD | head 1
| """.stripMargin))
}

test("test TIMESTAMPDIFF is not supported") {
intercept[Exception](sql(s"""
| source = $testTable
| | eval `TIMESTAMPDIFF_1` = TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00')
| | fields TIMESTAMPDIFF_1 | head 1
| """.stripMargin))
}

test("test TO_DAYS is not supported") {
val ex = intercept[UnsupportedOperationException](sql(s"""
| source = $testTable
Expand Down Expand Up @@ -497,15 +574,6 @@ class FlintSparkPPLBuiltInDateTimeFunctionITSuite
assert(ex.getMessage.contains("UTC_TIME is not a builtin function of PPL"))
}

test("test UTC_TIMESTAMP is not supported") {
val ex = intercept[UnsupportedOperationException](sql(s"""
| source = $testTable
| | eval `UTC_TIMESTAMP` = UTC_TIMESTAMP()
| | fields UTC_TIMESTAMP | head 1
| """.stripMargin))
assert(ex.getMessage.contains("UTC_TIMESTAMP is not a builtin function of PPL"))
}

test("test YEARWEEK is not supported") {
val ex = intercept[UnsupportedOperationException](sql(s"""
| source = $testTable
Expand Down
1 change: 1 addition & 0 deletions ppl-spark-integration/src/main/antlr4/OpenSearchPPLLexer.g4
View file
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,7 @@ CURDATE: 'CURDATE';
CURRENT_DATE: 'CURRENT_DATE';
CURRENT_TIME: 'CURRENT_TIME';
CURRENT_TIMESTAMP: 'CURRENT_TIMESTAMP';
CURRENT_TIMEZONE: 'CURRENT_TIMEZONE';
CURTIME: 'CURTIME';
DATE: 'DATE';
DATEDIFF: 'DATEDIFF';
Expand Down
3 changes: 3 additions & 0 deletions ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4
View file
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,7 @@ valueExpression
| primaryExpression # valueExpressionDefault
| positionFunction # positionFunctionCall
| caseFunction # caseExpr
| timestampFunction # timestampFunctionCall
| LT_PRTHS valueExpression RT_PRTHS # parentheticValueExpr
| LT_SQR_PRTHS subSearch RT_SQR_PRTHS # scalarSubqueryExpr
;
Expand Down Expand Up @@ -677,6 +678,7 @@ dateTimeFunctionName
| CURRENT_DATE
| CURRENT_TIME
| CURRENT_TIMESTAMP
| CURRENT_TIMEZONE
| CURTIME
| DATE
| DATEDIFF
Expand Down Expand Up @@ -893,6 +895,7 @@ literalValue
| decimalLiteral
| booleanLiteral
| datetimeLiteral //#datetime
| intervalLiteral
;

intervalLiteral
Expand Down
5 changes: 0 additions & 5 deletions ppl-spark-integration/src/main/java/org/opensearch/sql/ast/expression/Interval.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,6 @@ public class Interval extends UnresolvedExpression {
private final UnresolvedExpression value;
private final IntervalUnit unit;

public Interval(UnresolvedExpression value, String unit) {
this.value = value;
this.unit = IntervalUnit.of(unit);
}

@Override
public List<UnresolvedExpression> getChild() {
return Collections.singletonList(value);
Expand Down
1 change: 1 addition & 0 deletions ppl-spark-integration/src/main/java/org/opensearch/sql/ast/expression/IntervalUnit.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ public enum IntervalUnit {
UNKNOWN,

MICROSECOND,
MILLISECOND,
SECOND,
MINUTE,
HOUR,
Expand Down
Loading

0 comments on commit 99aab5a

Please sign in to comment.