Add planning document for PPL revisions

README.md

@@ -17,6 +17,8 @@ Please refer to the [Flint Index Reference Manual](./docs/index.md) for more inf
 
 * For additional details on Spark PPL Architecture, see [PPL Architecture](docs/ppl-lang/PPL-on-Spark.md)
 
+* For the PPL planned release content [release-plan](docs/ppl-lang/planning/release-plan.md)
+
 * For additional details on Spark PPL commands project, see [PPL Project](https://github.com/orgs/opensearch-project/projects/214/views/2)
 
 ## Prerequisites
@@ -92,7 +94,7 @@ If you discover a potential security issue in this project we ask that you notif
 
 ## License
 
-See the [LICENSE](./LICENSE.txt) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
+See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
 
 ## Copyright
 

docs/ppl-lang/PPL-Example-Commands.md

@@ -12,7 +12,7 @@
 - `explain simple | describe table`
 
 #### **Fields**
-[See additional command details](ppl-fields-command.md)
+[See additional command details](commands/ppl-fields-command.md)
 - `source = table`
 - `source = table | fields a,b,c`
 - `source = table | fields + a,b,c`
@@ -70,7 +70,7 @@ _- **Limitation: new field added by eval command with a function cannot be dropp
 
 
 #### **Eval**:
-[See additional command details](ppl-eval-command.md)
+[See additional command details](commands/ppl-eval-command.md)
 
 Assumptions: `a`, `b`, `c` are existing fields in `table`
 - `source = table | eval f = 1 | fields a,b,c,f`
@@ -128,7 +128,7 @@ source = table |  where ispresent(a) |
 - `source = table | eval a = signum(a) | where a < 0`
 
 #### **Aggregations**
-[See additional command details](ppl-stats-command.md)
+[See additional command details](commands/ppl-stats-command.md)
 
 - `source = table | stats avg(a) `
 - `source = table | where a < 50 | stats avg(c) `
@@ -155,7 +155,7 @@ source = table |  where ispresent(a) |
 
 #### **Dedup**
 
-[See additional command details](ppl-dedup-command.md)
+[See additional command details](commands/ppl-dedup-command.md)
 
 - `source = table | dedup a | fields a,b,c`
 - `source = table | dedup a,b | fields a,b,c`
@@ -172,31 +172,31 @@ source = table |  where ispresent(a) |
 - `source = table | dedup 1 a consecutive=true| fields a,b,c` (Consecutive deduplication is unsupported)
 
 #### **Rare**
-[See additional command details](ppl-rare-command.md)
+[See additional command details](commands/ppl-rare-command.md)
 
 - `source=accounts | rare gender`
 - `source=accounts | rare age by gender`
 
 #### **Top**
-[See additional command details](ppl-top-command.md)
+[See additional command details](commands/ppl-top-command.md)
 
 - `source=accounts | top gender`
 - `source=accounts | top 1 gender`
 - `source=accounts | top 1 age by gender`
 
 #### **Parse**
-[See additional command details](ppl-parse-command.md)
+[See additional command details](commands/ppl-parse-command.md)
 
 - `source=accounts | parse email '.+@(?<host>.+)' | fields email, host `
 - `source=accounts | parse email '.+@(?<host>.+)' | top 1 host `
 - `source=accounts | parse email '.+@(?<host>.+)' | stats count() by host`
 - `source=accounts | parse email '.+@(?<host>.+)' | eval eval_result=1 | fields host, eval_result`
 - `source=accounts | parse email '.+@(?<host>.+)' | where age > 45 | sort - age | fields age, email, host`
 - `source=accounts | parse address '(?<streetNumber>\d+) (?<street>.+)' | where streetNumber > 500 | sort num(streetNumber) | fields streetNumber, street`
-- Limitation: [see limitations](ppl-parse-command.md#limitations)
+- Limitation: [see limitations](commands/ppl-parse-command.md#limitations)
 
 #### **Grok**
-[See additional command details](ppl-grok-command.md)
+[See additional command details](commands/ppl-grok-command.md)
 
 - `source=accounts | grok email '.+@%{HOSTNAME:host}' | top 1 host`
 - `source=accounts | grok email '.+@%{HOSTNAME:host}' | stats count() by host`
@@ -207,26 +207,26 @@ source = table |  where ispresent(a) |
 
 - **Limitation: Overriding existing field is unsupported:**_
 - `source=accounts | grok address '%{NUMBER} %{GREEDYDATA:address}' | fields address`
-- [see limitations](ppl-parse-command.md#limitations)
+- [see limitations](commands/ppl-parse-command.md#limitations)
 
 #### **Patterns**
-[See additional command details](ppl-patterns-command.md)
+[See additional command details](commands/ppl-patterns-command.md)
 
 - `source=accounts | patterns email | fields email, patterns_field `
 - `source=accounts | patterns email | where age > 45 | sort - age | fields email, patterns_field`
 - `source=apache | patterns new_field='no_numbers' pattern='[0-9]' message | fields message, no_numbers`
 - `source=apache | patterns new_field='no_numbers' pattern='[0-9]' message | stats count() by no_numbers`
-- Limitation: [see limitations](ppl-parse-command.md#limitations)
+- Limitation: [see limitations](commands/ppl-parse-command.md#limitations)
 
 #### **Rename**
-[See additional command details](ppl-rename-command.md)
+[See additional command details](commands/ppl-rename-command.md)
 
 - `source=accounts | rename email as user_email | fields id, user_email`
 - `source=accounts | rename id as user_id, email as user_email | fields user_id, user_email`
 
 
 #### **Join**
-[See additional command details](ppl-join-command.md)
+[See additional command details](commands/ppl-join-command.md)
 
 - `source = table1 | inner join left = l right = r on l.a = r.a table2 | fields l.a, r.a, b, c`
 - `source = table1 | left join left = l right = r on l.a = r.a table2 | fields l.a, r.a, b, c`
@@ -240,7 +240,7 @@ _- **Limitation: sub-searches is unsupported in join right side now**_
 
 
 #### **Lookup**
-[See additional command details](ppl-lookup-command.md)
+[See additional command details](commands/ppl-lookup-command.md)
 
 - `source = table1 | lookup table2 id`
 - `source = table1 | lookup table2 id, name`
@@ -255,7 +255,7 @@ _- **Limitation: "REPLACE" or "APPEND" clause must contain "AS"**_
 
 
 #### **InSubquery**
-[See additional command details](ppl-subquery-command.md)
+[See additional command details](commands/ppl-subquery-command.md)
 
 - `source = outer | where a in [ source = inner | fields b ]`
 - `source = outer | where (a) in [ source = inner | fields b ]`
@@ -353,7 +353,7 @@ source = supplier
 
 ---
 #### Experimental Commands:
-[See additional command details](ppl-correlation-command.md)
+[See additional command details](commands/ppl-correlation-command.md)
 
 ```sql
 - `source alb_logs, traces, metrics | where ip="10.0.0.1" AND cloud.provider="aws"| correlate exact on (ip, port) scope(@timestamp, 2018-07-02T22:23:00, 1 D)`

docs/ppl-lang/README.md

@@ -15,7 +15,7 @@ source=accounts
 ```
 
 For additional examples see the next [documentation](PPL-Example-Commands.md). 
-
+For the PPL planned release content [release-plan](planning/release-plan.md)
 ---
 ### Commands Specifications
 
@@ -24,45 +24,45 @@ For additional examples see the next [documentation](PPL-Example-Commands.md).
 
     - [`explain command `](PPL-Example-Commands.md/#explain)
 
-    - [`dedup command `](ppl-dedup-command.md)
+    - [`dedup command `](commands/ppl-dedup-command.md)
 
     - [`describe command`](PPL-Example-Commands.md/#describe)
 
     - [`fillnull command`](ppl-fillnull-command.md)
 
-    - [`eval command`](ppl-eval-command.md)
+    - [`eval command`](commands/ppl-eval-command.md)
 
-    - [`fields command`](ppl-fields-command.md)
+    - [`fields command`](commands/ppl-fields-command.md)
 
-    - [`grok command`](ppl-grok-command.md)
+    - [`grok command`](commands/ppl-grok-command.md)
 
-    - [`parse command`](ppl-parse-command.md)
+    - [`parse command`](commands/ppl-parse-command.md)
 
-    - [`patterns command`](ppl-patterns-command.md)
+    - [`patterns command`](commands/ppl-patterns-command.md)
 
-    - [`rename command`](ppl-rename-command.md)
+    - [`rename command`](commands/ppl-rename-command.md)
 
-    - [`search command`](ppl-search-command.md)
+    - [`search command`](commands/ppl-search-command.md)
 
-    - [`sort command`](ppl-sort-command.md)
+    - [`sort command`](commands/ppl-sort-command.md)
 
-    - [`stats command`](ppl-stats-command.md)
+    - [`stats command`](commands/ppl-stats-command.md)
 
-    - [`where command`](ppl-where-command.md)
+    - [`where command`](commands/ppl-where-command.md)
 
-    - [`head command`](ppl-head-command.md)
+    - [`head command`](commands/ppl-head-command.md)
 
-    - [`rare command`](ppl-rare-command.md)
+    - [`rare command`](commands/ppl-rare-command.md)
 
-    - [`top command`](ppl-top-command.md)
+    - [`top command`](commands/ppl-top-command.md)
 
-    - [`join commands`](ppl-join-command.md)
+    - [`join commands`](commands/ppl-join-command.md)
 
-    - [`lookup commands`](ppl-lookup-command.md)
+    - [`lookup commands`](commands/ppl-lookup-command.md)
 
-    - [`subquery commands`](ppl-subquery-command.md)
+    - [`subquery commands`](commands/ppl-subquery-command.md)
 
-    - [`correlation commands`](ppl-correlation-command.md)
+    - [`correlation commands`](commands/ppl-correlation-command.md)
 
 
 * **Functions**

docs/ppl-lang/ppl-dedup-command.md → docs/ppl-lang/commands/ppl-dedup-command.md

@@ -4,11 +4,6 @@
 
 - [Description](#description)
 - [Syntax](#syntax)
-- [Examples](#examples)
-    - [Example 1: Dedup by one field](#example-1-dedup-by-one-field)
-    - [Example 2: Keep 2 duplicates documents](#example-2-keep-2-duplicates-documents)
-    - [Example 3: Keep or Ignore the empty field by default](#example-3-keep-or-ignore-the-empty-field-by-default)
-    - [Example 4: Dedup in consecutive document](#example-4-dedup-in-consecutive-document)
 - [Limitation](#limitation)
 
 ## Description

docs/ppl-lang/ppl-join-command.md → docs/ppl-lang/commands/ppl-join-command.md

@@ -162,7 +162,7 @@ SEARCH source=customer
 | SORT - custdist, - c_count
 ```
 
-### Comparison with [Correlation](ppl-correlation-command)
+### Comparison with [Correlation](ppl-correlation-command.md)
 
 A primary difference between `correlate` and `join` is that both sides of `correlate` are tables, but both sides of `join` are subqueries. 
 For example:

docs/ppl-lang/ppl-lookup-command.md → docs/ppl-lang/commands/ppl-lookup-command.md

@@ -3,7 +3,7 @@
 ## Overview
 Lookup command enriches your search data by adding or replacing data from a lookup index (dimension table).
 You can extend fields of an index with values from a dimension table, append or replace values when lookup condition is matched.
-As an alternative of [Join command](ppl-join-command), lookup command is more suitable for enriching the source data with a static dataset.
+As an alternative of [Join command](ppl-join-command.md), lookup command is more suitable for enriching the source data with a static dataset.
 
 
 ### Syntax of Lookup Command