Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functions DATE_ADD, DATE_SUB, TIMESTAMPADD, TIMESTAMPDIFF, CURRENT_TIMEZONE and UTC_TIMESTAMP #839

Merged
merged 4 commits into from
Oct 31, 2024
Merged

Add functions DATE_ADD, DATE_SUB, TIMESTAMPADD, TIMESTAMPDIFF, CURRENT_TIMEZONE and UTC_TIMESTAMP #839

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
85ac8d9
Add functions DATE_ADD, DATE_SUB, TIMESTAMPADD, TIMESTAMPDIFF, CURREN…
LantaoJin Oct 30, 2024
c162259
remove tokens and refactor
LantaoJin Oct 30, 2024
119ace7
Merge remote-tracking branch 'upstream/main' into issues/838
LantaoJin Oct 31, 2024
c52bfd9
address comment'
LantaoJin Oct 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 133 additions & 2 deletions docs/ppl-lang/functions/ppl-datetime.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Argument type: DATE, LONG

(DATE, LONG) -> DATE

Antonyms: `SUBDATE`_
Antonyms: `SUBDATE`

Example:

Expand Down Expand Up @@ -795,7 +795,7 @@ Argument type: DATE/TIMESTAMP, LONG

(DATE, LONG) -> DATE

Antonyms: `ADDDATE`_
Antonyms: `ADDDATE`

Example:

Expand Down Expand Up @@ -982,3 +982,134 @@ Example:
+----------------------------+


### `DATE_ADD`

**Description:**

Usage: date_add(date, INTERVAL expr unit) adds the interval expr to date.

Argument type: DATE, INTERVAL

Return type: DATE

Antonyms: `DATE_SUB`

Example::

os> source=people | eval `'2020-08-26' + 1d` = DATE_ADD(DATE('2020-08-26'), INTERVAL 1 DAY) | fields `'2020-08-26' + 1d`
fetched rows / total rows = 1/1
+---------------------+
| '2020-08-26' + 1d |
|---------------------+
| 2020-08-27 |
+---------------------+


### `DATE_SUB`

**Description:**

Usage: date_sub(date, INTERVAL expr unit) subtracts the interval expr from date.

Argument type: DATE, INTERVAL

Return type: DATE

Antonyms: `DATE_ADD`

Example::

os> source=people | eval `'2008-01-02' - 31d` = DATE_SUB(DATE('2008-01-02'), INTERVAL 31 DAY) | fields `'2008-01-02' - 31d`
fetched rows / total rows = 1/1
+---------------------+
| '2008-01-02' - 31d |
|---------------------+
| 2007-12-02 |
+---------------------+


### `TIMESTAMPADD`

**Description:**

Usage: Returns a TIMESTAMP value based on a passed in DATE/TIMESTAMP/STRING argument and an INTERVAL and INTEGER argument which determine the amount of time to be added.
If the third argument is a STRING, it must be formatted as a valid TIMESTAMP.
If the third argument is a DATE, it will be automatically converted to a TIMESTAMP.

Argument type: INTERVAL, INTEGER, DATE/TIMESTAMP/STRING

INTERVAL must be one of the following tokens: [SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, QUARTER, YEAR]

Examples::

os> source=people | eval `TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00')` = TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00') | eval `TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00')` = TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00') | fields `TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00')`, `TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00')`
fetched rows / total rows = 1/1
+----------------------------------------------+--------------------------------------------------+
| TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00') | TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00') |
|----------------------------------------------+--------------------------------------------------|
| 2000-01-18 00:00:00 | 1999-10-01 00:00:00 |
+----------------------------------------------+--------------------------------------------------+


### `TIMESTAMPDIFF`

**Description:**

Usage: TIMESTAMPDIFF(interval, start, end) returns the difference between the start and end date/times in interval units.
Arguments will be automatically converted to a ]TIMESTAMP when appropriate.
Any argument that is a STRING must be formatted as a valid TIMESTAMP.

Argument type: INTERVAL, DATE/TIMESTAMP/STRING, DATE/TIMESTAMP/STRING

INTERVAL must be one of the following tokens: [SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, QUARTER, YEAR]

Examples::

os> source=people | eval `TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00')` = TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00') | eval `TIMESTAMPDIFF(SECOND, timestamp('1997-01-01 00:00:23'), timestamp('1997-01-01 00:00:00'))` = TIMESTAMPDIFF(SECOND, timestamp('1997-01-01 00:00:23'), timestamp('1997-01-01 00:00:00')) | fields `TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00')`, `TIMESTAMPDIFF(SECOND, timestamp('1997-01-01 00:00:23'), timestamp('1997-01-01 00:00:00'))`
fetched rows / total rows = 1/1
+-------------------------------------------------------------------+-------------------------------------------------------------------------------------------+
| TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00') | TIMESTAMPDIFF(SECOND, timestamp('1997-01-01 00:00:23'), timestamp('1997-01-01 00:00:00')) |
|-------------------------------------------------------------------+-------------------------------------------------------------------------------------------|
| 4 | -23 |
+-------------------------------------------------------------------+-------------------------------------------------------------------------------------------+


### `UTC_TIMESTAMP`

**Description:**

Returns the current UTC timestamp as a value in 'YYYY-MM-DD hh:mm:ss'.

Return type: TIMESTAMP

Specification: UTC_TIMESTAMP() -> TIMESTAMP

Example::

> source=people | eval `UTC_TIMESTAMP()` = UTC_TIMESTAMP() | fields `UTC_TIMESTAMP()`
fetched rows / total rows = 1/1
+---------------------+
| UTC_TIMESTAMP() |
|---------------------|
| 2022-10-03 17:54:28 |
+---------------------+


### `CURRENT_TIMEZONE`

**Description:**

Returns the current local timezone.

Return type: STRING

Example::

> source=people | eval `CURRENT_TIMEZONE()` = CURRENT_TIMEZONE() | fields `CURRENT_TIMEZONE()`
fetched rows / total rows = 1/1
+------------------------+
| CURRENT_TIMEZONE() |
|------------------------|
| America/Chicago |
+------------------------+

154 changes: 111 additions & 43 deletions ...on/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLBuiltInDateTimeFunctionITSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,117 @@ class FlintSparkPPLBuiltInDateTimeFunctionITSuite
comparePlans(logicalPlan, expectedPlan, checkAnalysis = false)
}

test("test DATE_ADD") {
val frame1 = sql(s"""
| source = $testTable | eval `'2020-08-26' + 2d` = DATE_ADD(DATE('2020-08-26'), INTERVAL 2 DAY)
| | fields `'2020-08-26' + 2d` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2020-08-28"))), frame1)

val frame2 = sql(s"""
| source = $testTable | eval `'2020-08-26' - 2d` = DATE_ADD(DATE('2020-08-26'), INTERVAL -2 DAY)
| | fields `'2020-08-26' - 2d` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2020-08-24"))), frame2)

val frame3 = sql(s"""
| source = $testTable | eval `'2020-08-26' + 2m` = DATE_ADD(DATE('2020-08-26'), INTERVAL 2 MONTH)
| | fields `'2020-08-26' + 2m` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2020-10-26"))), frame3)

val frame4 = sql(s"""
| source = $testTable | eval `'2020-08-26' + 2y` = DATE_ADD(DATE('2020-08-26'), INTERVAL 2 YEAR)
| | fields `'2020-08-26' + 2y` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2022-08-26"))), frame4)

val ex = intercept[AnalysisException](sql(s"""
| source = $testTable | eval `'2020-08-26 01:01:01' + 2h` = DATE_ADD(TIMESTAMP('2020-08-26 01:01:01'), INTERVAL 2 HOUR)
| | fields `'2020-08-26 01:01:01' + 2h` | head 1
| """.stripMargin))
assert(ex.getMessage.contains("""Parameter 1 requires the "DATE" type"""))
}

test("test DATE_SUB") {
val frame1 = sql(s"""
| source = $testTable | eval `'2020-08-26' - 2d` = DATE_SUB(DATE('2020-08-26'), INTERVAL 2 DAY)
| | fields `'2020-08-26' - 2d` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2020-08-24"))), frame1)

val frame2 = sql(s"""
| source = $testTable | eval `'2020-08-26' + 2d` = DATE_SUB(DATE('2020-08-26'), INTERVAL -2 DAY)
| | fields `'2020-08-26' + 2d` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2020-08-28"))), frame2)

val frame3 = sql(s"""
| source = $testTable | eval `'2020-08-26' - 2m` = DATE_SUB(DATE('2020-08-26'), INTERVAL 12 MONTH)
| | fields `'2020-08-26' - 2m` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2019-08-26"))), frame3)

val frame4 = sql(s"""
| source = $testTable | eval `'2020-08-26' - 2y` = DATE_SUB(DATE('2020-08-26'), INTERVAL 2 YEAR)
| | fields `'2020-08-26' - 2y` | head 1
| """.stripMargin)
assertSameRows(Seq(Row(Date.valueOf("2018-08-26"))), frame4)

val ex = intercept[AnalysisException](sql(s"""
| source = $testTable | eval `'2020-08-26 01:01:01' - 2h` = DATE_SUB(TIMESTAMP('2020-08-26 01:01:01'), INTERVAL 2 HOUR)
| | fields `'2020-08-26 01:01:01' - 2h` | head 1
| """.stripMargin))
assert(ex.getMessage.contains("""Parameter 1 requires the "DATE" type"""))
}

test("test TIMESTAMPADD") {
val frame = sql(s"""
| source = $testTable
| | eval `TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00')` = TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00')
| | eval `TIMESTAMPADD(DAY, 17, TIMESTAMP('2000-01-01 00:00:00'))` = TIMESTAMPADD(DAY, 17, TIMESTAMP('2000-01-01 00:00:00'))
| | eval `TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00')` = TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00')
| | fields `TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00')`, `TIMESTAMPADD(DAY, 17, TIMESTAMP('2000-01-01 00:00:00'))`, `TIMESTAMPADD(QUARTER, -1, '2000-01-01 00:00:00')`
| | head 1
| """.stripMargin)
assertSameRows(
Seq(
Row(
Timestamp.valueOf("2000-01-18 00:00:00"),
Timestamp.valueOf("2000-01-18 00:00:00"),
Timestamp.valueOf("1999-10-01 00:00:00"))),
frame)
}

test("test TIMESTAMPDIFF") {
val frame = sql(s"""
| source = $testTable
| | eval `TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00')` = TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00')
| | eval `TIMESTAMPDIFF(SECOND, TIMESTAMP('2000-01-01 00:00:23'), TIMESTAMP('2000-01-01 00:00:00'))` = TIMESTAMPDIFF(SECOND, TIMESTAMP('2000-01-01 00:00:23'), TIMESTAMP('2000-01-01 00:00:00'))
| | fields `TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00')`, `TIMESTAMPDIFF(SECOND, TIMESTAMP('2000-01-01 00:00:23'), TIMESTAMP('2000-01-01 00:00:00'))`
| | head 1
| """.stripMargin)
assertSameRows(Seq(Row(4, -23)), frame)
}

test("test CURRENT_TIMEZONE") {
val frame = sql(s"""
| source = $testTable
| | eval `CURRENT_TIMEZONE` = CURRENT_TIMEZONE()
| | fields `CURRENT_TIMEZONE`
| """.stripMargin)
assert(frame.collect().length > 0)
}

test("test UTC_TIMESTAMP") {
val frame = sql(s"""
| source = $testTable
| | eval `UTC_TIMESTAMP` = UTC_TIMESTAMP()
| | fields `UTC_TIMESTAMP`
| """.stripMargin)
assert(frame.collect().length > 0)
}

test("test hour, minute, second, HOUR_OF_DAY, MINUTE_OF_HOUR") {
val frame = sql(s"""
| source = $testTable
Expand Down Expand Up @@ -284,24 +395,6 @@ class FlintSparkPPLBuiltInDateTimeFunctionITSuite
assert(ex.getMessage.contains("ADDTIME is not a builtin function of PPL"))
}

test("test DATE_ADD is not supported") {
val ex = intercept[UnsupportedOperationException](sql(s"""
| source = $testTable
| | eval `DATE_ADD` = DATE_ADD()
| | fields DATE_ADD | head 1
| """.stripMargin))
assert(ex.getMessage.contains("DATE_ADD is not a builtin function of PPL"))
}

test("test DATE_SUB is not supported") {
val ex = intercept[UnsupportedOperationException](sql(s"""
| source = $testTable
| | eval `DATE_SUB` = DATE_SUB()
| | fields DATE_SUB | head 1
| """.stripMargin))
assert(ex.getMessage.contains("DATE_SUB is not a builtin function of PPL"))
}

test("test DATETIME is not supported") {
val ex = intercept[UnsupportedOperationException](sql(s"""
| source = $testTable
Expand Down Expand Up @@ -445,22 +538,6 @@ class FlintSparkPPLBuiltInDateTimeFunctionITSuite
assert(ex.getMessage.contains("TIMEDIFF is not a builtin function of PPL"))
}

test("test TIMESTAMPADD is not supported") {
intercept[Exception](sql(s"""
| source = $testTable
| | eval `TIMESTAMPADD` = TIMESTAMPADD(DAY, 17, '2000-01-01 00:00:00')
| | fields TIMESTAMPADD | head 1
| """.stripMargin))
}

test("test TIMESTAMPDIFF is not supported") {
intercept[Exception](sql(s"""
| source = $testTable
| | eval `TIMESTAMPDIFF_1` = TIMESTAMPDIFF(YEAR, '1997-01-01 00:00:00', '2001-03-06 00:00:00')
| | fields TIMESTAMPDIFF_1 | head 1
| """.stripMargin))
}

test("test TO_DAYS is not supported") {
val ex = intercept[UnsupportedOperationException](sql(s"""
| source = $testTable
Expand Down Expand Up @@ -497,15 +574,6 @@ class FlintSparkPPLBuiltInDateTimeFunctionITSuite
assert(ex.getMessage.contains("UTC_TIME is not a builtin function of PPL"))
}

test("test UTC_TIMESTAMP is not supported") {
val ex = intercept[UnsupportedOperationException](sql(s"""
| source = $testTable
| | eval `UTC_TIMESTAMP` = UTC_TIMESTAMP()
| | fields UTC_TIMESTAMP | head 1
| """.stripMargin))
assert(ex.getMessage.contains("UTC_TIMESTAMP is not a builtin function of PPL"))
}

test("test YEARWEEK is not supported") {
val ex = intercept[UnsupportedOperationException](sql(s"""
| source = $testTable
Expand Down
1 change: 1 addition & 0 deletions ppl-spark-integration/src/main/antlr4/OpenSearchPPLLexer.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,7 @@ CURDATE: 'CURDATE';
CURRENT_DATE: 'CURRENT_DATE';
CURRENT_TIME: 'CURRENT_TIME';
CURRENT_TIMESTAMP: 'CURRENT_TIMESTAMP';
CURRENT_TIMEZONE: 'CURRENT_TIMEZONE';
CURTIME: 'CURTIME';
DATE: 'DATE';
DATEDIFF: 'DATEDIFF';
Expand Down
3 changes: 3 additions & 0 deletions ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,7 @@ valueExpression
| primaryExpression # valueExpressionDefault
| positionFunction # positionFunctionCall
| caseFunction # caseExpr
| timestampFunction # timestampFunctionCall
| LT_PRTHS valueExpression RT_PRTHS # parentheticValueExpr
| LT_SQR_PRTHS subSearch RT_SQR_PRTHS # scalarSubqueryExpr
;
Expand Down Expand Up @@ -672,6 +673,7 @@ dateTimeFunctionName
| CURRENT_DATE
| CURRENT_TIME
| CURRENT_TIMESTAMP
| CURRENT_TIMEZONE
| CURTIME
| DATE
| DATEDIFF
Expand Down Expand Up @@ -888,6 +890,7 @@ literalValue
| decimalLiteral
| booleanLiteral
| datetimeLiteral //#datetime
| intervalLiteral
;

intervalLiteral
Expand Down
5 changes: 0 additions & 5 deletions ppl-spark-integration/src/main/java/org/opensearch/sql/ast/expression/Interval.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,6 @@ public class Interval extends UnresolvedExpression {
private final UnresolvedExpression value;
private final IntervalUnit unit;

public Interval(UnresolvedExpression value, String unit) {
this.value = value;
this.unit = IntervalUnit.of(unit);
}

@Override
public List<UnresolvedExpression> getChild() {
return Collections.singletonList(value);
Expand Down
1 change: 1 addition & 0 deletions ppl-spark-integration/src/main/java/org/opensearch/sql/ast/expression/IntervalUnit.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ public enum IntervalUnit {
UNKNOWN,

MICROSECOND,
MILLISECOND,
SECOND,
MINUTE,
HOUR,
Expand Down
Loading
Loading