From 439b12a3fef9c41c06311fa01f7fdd0097fd707f Mon Sep 17 00:00:00 2001 From: YANGDB Date: Wed, 7 Aug 2024 13:48:31 -0700 Subject: [PATCH 1/3] update correlation documentation with experimental comment and warning --- docs/PPL-Correlation-command.md | 4 ++++ ppl-spark-integration/README.md | 32 +++++++------------------------- 2 files changed, 11 insertions(+), 25 deletions(-) diff --git a/docs/PPL-Correlation-command.md b/docs/PPL-Correlation-command.md index f7ef3e266..1b6083234 100644 --- a/docs/PPL-Correlation-command.md +++ b/docs/PPL-Correlation-command.md @@ -1,5 +1,7 @@ ## PPL Correlation Command +> This is an experimental command - it may be removed in future versions + ## Overview In the past year OpenSearch Observability & security teams have been busy with many aspects of improving data monitoring and visibility. @@ -262,6 +264,8 @@ The new correlation command is actually a ‘hidden’ join command therefore th Catalyst engine will optimize this query according to the most efficient join ordering. +> This is an experimental command - it may be removed in future versions + * * * ## Appendix diff --git a/ppl-spark-integration/README.md b/ppl-spark-integration/README.md index 1538f43be..36c883373 100644 --- a/ppl-spark-integration/README.md +++ b/ppl-spark-integration/README.md @@ -221,8 +221,8 @@ Next tasks ahead will resolve this: This section describes the next steps planned for enabling additional commands and gamer translation. -#### Supported -The next samples of PPL queries are currently supported: +#### Example PPL Queries +See the next samples of PPL queries : **Fields** - `source = table` @@ -272,31 +272,13 @@ Limitation: Overriding existing field is unsupported, following queries throw ex - `source = table | stats sum(productsAmount) by span(transactionDate, 1d) as age_date | sort age_date` - `source = table | stats sum(productsAmount) by span(transactionDate, 1w) as age_date, productId` -> For additional details, review [FlintSparkPPLTimeWindowITSuite.scala](../integ-test/src/test/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLTimeWindowITSuite.scala) - -#### Supported Commands: - - `search` - [See details](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/cmd/search.rst) - - `where` - [See details](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/cmd/where.rst) - - `fields` - [See details](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/cmd/fields.rst) - - `eval` - [See details](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/cmd/eval.rst) - - `head` - [See details](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/cmd/head.rst) - - `stats` - [See details](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/cmd/stats.rst) (supports AVG, COUNT, DISTINCT_COUNT, MAX, MIN and SUM aggregation functions) - - `sort` - [See details](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/cmd/sort.rst) - - `correlation` - [See details](../docs/PPL-Correlation-command.md) - -> For additional details, review [Integration Tests](../integ-test/src/test/scala/org/opensearch/flint/spark/) - --- -#### Planned Support +For additional details, view [PPL Commands](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/index.rst) +For additional details on Spark PPL available support campaign, see [PPL Project](https://github.com/orgs/opensearch-project/projects/214/views/2) - - support the `explain` command to return the explained PPL query logical plan and expected execution plan +#### Experimental Commands: + - `correlation` - [See details](../docs/PPL-Correlation-command.md) - - attend [sort](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/cmd/sort.rst) partially supported, missing capability to sort by alias field (span like or aggregation) - - attend `alias` - partially supported, missing capability to sort by / group-by alias field name +> This is an experimental command - it may be removed in future versions - - add [conditions](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/functions/condition.rst) support - - add [top](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/cmd/top.rst) support - - add [cast](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/functions/conversion.rst) support - - add [math](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/functions/math.rst) support - - add [deduplicate](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/cmd/dedup.rst) support \ No newline at end of file From bdfb0ec67b9a73589fb5c3175b9a9b92f334af52 Mon Sep 17 00:00:00 2001 From: YANGDB Date: Wed, 7 Aug 2024 13:58:37 -0700 Subject: [PATCH 2/3] update correlation documentation with experimental comment and warning Signed-off-by: YANGDB --- ppl-spark-integration/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/ppl-spark-integration/README.md b/ppl-spark-integration/README.md index 36c883373..479f41723 100644 --- a/ppl-spark-integration/README.md +++ b/ppl-spark-integration/README.md @@ -274,8 +274,10 @@ Limitation: Overriding existing field is unsupported, following queries throw ex --- -For additional details, view [PPL Commands](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/index.rst) -For additional details on Spark PPL available support campaign, see [PPL Project](https://github.com/orgs/opensearch-project/projects/214/views/2) +For additional details on PPL commands - view [PPL Commands](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/index.rst) + +For additional details on Spark PPL commands project, see [PPL Project](https://github.com/orgs/opensearch-project/projects/214/views/2) +For additional details on Spark PPL commands support campaign, see [PPL Commands Campaign](https://github.com/opensearch-project/opensearch-spark/issues/408) #### Experimental Commands: - `correlation` - [See details](../docs/PPL-Correlation-command.md) From 2735e2d3ff4bb8dc9e144388a4334cc608a143d0 Mon Sep 17 00:00:00 2001 From: YANGDB Date: Wed, 7 Aug 2024 13:58:57 -0700 Subject: [PATCH 3/3] update correlation documentation with experimental comment and warning Signed-off-by: YANGDB --- ppl-spark-integration/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ppl-spark-integration/README.md b/ppl-spark-integration/README.md index 479f41723..2a17fe19f 100644 --- a/ppl-spark-integration/README.md +++ b/ppl-spark-integration/README.md @@ -274,7 +274,7 @@ Limitation: Overriding existing field is unsupported, following queries throw ex --- -For additional details on PPL commands - view [PPL Commands](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/index.rst) +For additional details on PPL commands - view [PPL Commands Docs](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/index.rst) For additional details on Spark PPL commands project, see [PPL Project](https://github.com/orgs/opensearch-project/projects/214/views/2) For additional details on Spark PPL commands support campaign, see [PPL Commands Campaign](https://github.com/opensearch-project/opensearch-spark/issues/408)