From 5241a57479b6823450e5cbae0957a6059e9aa7c3 Mon Sep 17 00:00:00 2001
From: YANGDB <yang.db.dev@gmail.com>
Date: Mon, 7 Oct 2024 17:00:02 -0700
Subject: [PATCH 1/5] add release-plan.md doc

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
---
 docs/ppl-lang/planning/release-plan.md | 185 +++++++++++++++++++++++++
 1 file changed, 185 insertions(+)
 create mode 100644 docs/ppl-lang/planning/release-plan.md

diff --git a/docs/ppl-lang/planning/release-plan.md b/docs/ppl-lang/planning/release-plan.md
new file mode 100644
index 000000000..c016ac8d0
--- /dev/null
+++ b/docs/ppl-lang/planning/release-plan.md
@@ -0,0 +1,185 @@
+# PPL Release Plan 
+The next doc present a release plan summarizing the commands / features planned for the upcoming releases.
+
+Each release will have a list of all the next parts:
+ - commands / functions
+ - limitations
+ - comments
+
+Each command / function would have the following properties:
+ - `status`: the maturity of the feature (experimental, stable, deprecated)
+ - `since`: the version it was introduced 
+
+---
+## Version 0.4
+
+### Commands
+| command           | status       | since |
+|-------------------|--------------|-------|
+| fields            | stable       | 0.4   |
+| search            | stable       | 0.4   |
+| sort              | stable       | 0.4   |
+| stats             | stable       | 0.4   |
+| stats + span      | stable       | 0.4   |
+| stats + span + by | stable       | 0.4   |
+| where             | stable       | 0.4   |
+| head              | stable       | 0.4   |
+| correlation       | experimental | 0.4   |
+
+### Functions
+
+| function           | status  | since |
+|--------------------|---------|-------|
+| logical conditions | stable  | 0.4   |
+
+
+---
+## Version 0.5
+
+### Commands
+| command           | status       | since |
+|-------------------|--------------|-------|
+| fields            | stable       | 0.4   |
+| search            | stable       | 0.4   |
+| sort              | stable       | 0.4   |
+| stats             | stable       | 0.4   |
+| stats + span      | stable       | 0.4   |
+| stats + span + by | stable       | 0.4   |
+| where             | stable       | 0.4   |
+| head              | stable       | 0.4   |
+| correlation       | experimental | 0.4   |
+| explain           | experimental | 0.5   |
+| eval              | experimental | 0.5   |
+| dedup             | stable       | 0.5   |
+| describe          | stable       | 0.5   |
+| nested-fields     | experimental | 0.5   |
+| grok              | experimental | 0.5   |
+| parse             | experimental | 0.5   |
+| patterns          | experimental | 0.5   |
+| rename            | experimental | 0.5   |
+| rare              | experimental | 0.5   |
+| top               | experimental | 0.5   |
+
+
+### Functions
+
+| function           | status       | since |
+|--------------------|--------------|-------|
+| logical conditions | stable       | 0.4   |
+| case               | experimental | 0.5   |
+| if                 | experimental | 0.5   | 
+| filed (- / +)      | experimental | 0.5   | 
+
+### Limitations
+- `fields - list` shows incorrect results for spark version 3.3 - see [issue](https://github.com/opensearch-project/opensearch-spark/pull/732)
+- `eval` with comma separated expression needs spark version >= 3.4
+- `dedup` command with `allowedDuplication > 1` feature needs spark version >= 3.4
+
+---
+## Version 0.6
+
+### Commands
+| command               | status       | since |
+|-----------------------|--------------|-------|
+| fields                | stable       | 0.4   |
+| search                | stable       | 0.4   |
+| sort                  | stable       | 0.4   |
+| stats                 | stable       | 0.4   |
+| stats + span          | stable       | 0.4   |
+| stats + span + by     | stable       | 0.4   |
+| where                 | stable       | 0.4   |
+| head                  | stable       | 0.4   |
+| correlation           | experimental | 0.4   |
+| explain               | experimental | 0.5   |
+| eval                  | experimental | 0.5   |
+| dedup                 | stable       | 0.5   |
+| describe              | stable       | 0.5   |
+| nested-fields         | experimental | 0.5   |
+| grok                  | experimental | 0.5   |
+| parse                 | experimental | 0.5   |
+| patterns              | experimental | 0.5   |
+| rename                | experimental | 0.5   |
+| rare                  | experimental | 0.5   |
+| top                   | experimental | 0.5   |
+| join                  | experimental | 0.6   |
+| lookup                | experimental | 0.6   |
+| inSubQuery (subQuery) | experimental | 0.6   |
+| help                  | experimental | 0.6   |
+| fieldsummary          | experimental | 0.6   |
+
+
+### Functions
+
+| function           | status       | since |
+|--------------------|--------------|-------|
+| logical conditions | stable       | 0.4   |
+| case               | experimental | 0.5   |
+| if                 | experimental | 0.5   |
+| isEmpty            | experimental | 0.5   |
+| isPresent          | experimental | 0.5   |
+| filed (- / +)      | experimental | 0.5   | 
+| isBlank            | experimental | 0.6   |
+| coalesce           | experimental | 0.6   |
+
+---
+## Version 0.7
+
+### Commands
+| command               | status       | since |
+|-----------------------|--------------|-------|
+| fields                | stable       | 0.4   |
+| search                | stable       | 0.4   |
+| sort                  | stable       | 0.4   |
+| stats                 | stable       | 0.4   |
+| stats + span          | stable       | 0.4   |
+| stats + span + by     | stable       | 0.4   |
+| where                 | stable       | 0.4   |
+| head                  | stable       | 0.4   |
+| correlation           | experimental | 0.4   |
+| explain               | experimental | 0.5   |
+| eval                  | experimental | 0.5   |
+| dedup                 | stable       | 0.5   |
+| describe              | stable       | 0.5   |
+| nested-fields         | experimental | 0.5   |
+| grok                  | experimental | 0.5   |
+| parse                 | experimental | 0.5   |
+| patterns              | experimental | 0.5   |
+| rename                | experimental | 0.5   |
+| rare                  | experimental | 0.5   |
+| top                   | experimental | 0.5   |
+| join                  | experimental | 0.6   |
+| lookup                | experimental | 0.6   |
+| inSubQuery (subQuery) | experimental | 0.6   |
+| help                  | experimental | 0.6   |
+| fieldsummary          | experimental | 0.6   |
+| input                 | experimental | 0.7   |
+| file  (as source)     | experimental | 0.7   |
+| trendline             | experimental | 0.7   |
+| fillnull              | experimental | 0.7   |
+| flatten               | experimental | 0.7   |
+| expand_field          | experimental | 0.7   |
+| expand_field          | experimental | 0.7   |
+| CIDIR (IP)            | experimental | 0.7   |
+| IP Location           | experimental | 0.7   |
+
+### Functions
+
+| function           | status       | since |
+|--------------------|--------------|-------|
+| logical conditions | stable       | 0.4   |
+| case               | experimental | 0.5   |
+| if                 | experimental | 0.5   |
+| isEmpty            | experimental | 0.5   |
+| isPresent          | experimental | 0.5   |
+| filed (- / +)      | experimental | 0.5   | 
+| isBlank            | experimental | 0.6   |
+| coalesce           | experimental | 0.6   |
+| between            | experimental | 0.7   |
+
+
+---
+## Version 0.7
+
+### Commands
+
+### Functions

From baa5f3d5c3d0ded0234822f3efe97836328058ad Mon Sep 17 00:00:00 2001
From: YANGDB <yang.db.dev@gmail.com>
Date: Mon, 7 Oct 2024 17:14:57 -0700
Subject: [PATCH 2/5] add release-plan.md doc add reference to the plan

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
---
 README.md                                     |  4 +-
 docs/ppl-lang/PPL-Example-Commands.md         | 34 ++++++++---------
 docs/ppl-lang/README.md                       | 38 +++++++++----------
 .../{ => commands}/ppl-correlation-command.md |  0
 .../{ => commands}/ppl-dedup-command.md       |  5 ---
 .../{ => commands}/ppl-eval-command.md        |  0
 .../{ => commands}/ppl-fields-command.md      |  0
 .../{ => commands}/ppl-grok-command.md        |  0
 .../{ => commands}/ppl-head-command.md        |  0
 .../{ => commands}/ppl-join-command.md        |  2 +-
 .../{ => commands}/ppl-lookup-command.md      |  2 +-
 .../{ => commands}/ppl-parse-command.md       |  0
 .../{ => commands}/ppl-patterns-command.md    |  0
 .../{ => commands}/ppl-rare-command.md        |  0
 .../{ => commands}/ppl-rename-command.md      |  0
 .../{ => commands}/ppl-search-command.md      |  0
 .../{ => commands}/ppl-sort-command.md        |  0
 .../{ => commands}/ppl-stats-command.md       |  0
 .../{ => commands}/ppl-subquery-command.md    |  0
 .../{ => commands}/ppl-top-command.md         |  0
 .../{ => commands}/ppl-where-command.md       |  0
 21 files changed, 41 insertions(+), 44 deletions(-)
 rename docs/ppl-lang/{ => commands}/ppl-correlation-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-dedup-command.md (93%)
 rename docs/ppl-lang/{ => commands}/ppl-eval-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-fields-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-grok-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-head-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-join-command.md (99%)
 rename docs/ppl-lang/{ => commands}/ppl-lookup-command.md (95%)
 rename docs/ppl-lang/{ => commands}/ppl-parse-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-patterns-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-rare-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-rename-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-search-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-sort-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-stats-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-subquery-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-top-command.md (100%)
 rename docs/ppl-lang/{ => commands}/ppl-where-command.md (100%)

diff --git a/README.md b/README.md
index 4c470e98b..42aa28fb1 100644
--- a/README.md
+++ b/README.md
@@ -17,6 +17,8 @@ Please refer to the [Flint Index Reference Manual](./docs/index.md) for more inf
 
 * For additional details on Spark PPL Architecture, see [PPL Architecture](docs/ppl-lang/PPL-on-Spark.md)
 
+* For the PPL planned release content [release-plan](docs/ppl-lang/planning/release-plan.md)
+
 * For additional details on Spark PPL commands project, see [PPL Project](https://github.com/orgs/opensearch-project/projects/214/views/2)
 
 ## Prerequisites
@@ -92,7 +94,7 @@ If you discover a potential security issue in this project we ask that you notif
 
 ## License
 
-See the [LICENSE](./LICENSE.txt) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
+See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
 
 ## Copyright
 
diff --git a/docs/ppl-lang/PPL-Example-Commands.md b/docs/ppl-lang/PPL-Example-Commands.md
index f68865f75..97ea6dfe4 100644
--- a/docs/ppl-lang/PPL-Example-Commands.md
+++ b/docs/ppl-lang/PPL-Example-Commands.md
@@ -12,7 +12,7 @@
 - `explain simple | describe table`
 
 #### **Fields**
-[See additional command details](ppl-fields-command.md)
+[See additional command details](commands/ppl-fields-command.md)
 - `source = table`
 - `source = table | fields a,b,c`
 - `source = table | fields + a,b,c`
@@ -69,7 +69,7 @@ _- **Limitation: new field added by eval command with a function cannot be dropp
 
 
 #### **Eval**:
-[See additional command details](ppl-eval-command.md)
+[See additional command details](commands/ppl-eval-command.md)
 
 Assumptions: `a`, `b`, `c` are existing fields in `table`
 - `source = table | eval f = 1 | fields a,b,c,f`
@@ -118,7 +118,7 @@ source = table |  where ispresent(a) |
 - `source = table | eval a = signum(a) | where a < 0`
 
 #### **Aggregations**
-[See additional command details](ppl-stats-command.md)
+[See additional command details](commands/ppl-stats-command.md)
 
 - `source = table | stats avg(a) `
 - `source = table | where a < 50 | stats avg(c) `
@@ -145,7 +145,7 @@ source = table |  where ispresent(a) |
 
 #### **Dedup**
 
-[See additional command details](ppl-dedup-command.md)
+[See additional command details](commands/ppl-dedup-command.md)
 
 - `source = table | dedup a | fields a,b,c`
 - `source = table | dedup a,b | fields a,b,c`
@@ -162,20 +162,20 @@ source = table |  where ispresent(a) |
 - `source = table | dedup 1 a consecutive=true| fields a,b,c` (Consecutive deduplication is unsupported)
 
 #### **Rare**
-[See additional command details](ppl-rare-command.md)
+[See additional command details](commands/ppl-rare-command.md)
 
 - `source=accounts | rare gender`
 - `source=accounts | rare age by gender`
 
 #### **Top**
-[See additional command details](ppl-top-command.md)
+[See additional command details](commands/ppl-top-command.md)
 
 - `source=accounts | top gender`
 - `source=accounts | top 1 gender`
 - `source=accounts | top 1 age by gender`
 
 #### **Parse**
-[See additional command details](ppl-parse-command.md)
+[See additional command details](commands/ppl-parse-command.md)
 
 - `source=accounts | parse email '.+@(?<host>.+)' | fields email, host `
 - `source=accounts | parse email '.+@(?<host>.+)' | top 1 host `
@@ -183,10 +183,10 @@ source = table |  where ispresent(a) |
 - `source=accounts | parse email '.+@(?<host>.+)' | eval eval_result=1 | fields host, eval_result`
 - `source=accounts | parse email '.+@(?<host>.+)' | where age > 45 | sort - age | fields age, email, host`
 - `source=accounts | parse address '(?<streetNumber>\d+) (?<street>.+)' | where streetNumber > 500 | sort num(streetNumber) | fields streetNumber, street`
-- Limitation: [see limitations](ppl-parse-command.md#limitations)
+- Limitation: [see limitations](commands/ppl-parse-command.md#limitations)
 
 #### **Grok**
-[See additional command details](ppl-grok-command.md)
+[See additional command details](commands/ppl-grok-command.md)
 
 - `source=accounts | grok email '.+@%{HOSTNAME:host}' | top 1 host`
 - `source=accounts | grok email '.+@%{HOSTNAME:host}' | stats count() by host`
@@ -197,26 +197,26 @@ source = table |  where ispresent(a) |
 
 - **Limitation: Overriding existing field is unsupported:**_
 - `source=accounts | grok address '%{NUMBER} %{GREEDYDATA:address}' | fields address`
-- [see limitations](ppl-parse-command.md#limitations)
+- [see limitations](commands/ppl-parse-command.md#limitations)
 
 #### **Patterns**
-[See additional command details](ppl-patterns-command.md)
+[See additional command details](commands/ppl-patterns-command.md)
 
 - `source=accounts | patterns email | fields email, patterns_field `
 - `source=accounts | patterns email | where age > 45 | sort - age | fields email, patterns_field`
 - `source=apache | patterns new_field='no_numbers' pattern='[0-9]' message | fields message, no_numbers`
 - `source=apache | patterns new_field='no_numbers' pattern='[0-9]' message | stats count() by no_numbers`
-- Limitation: [see limitations](ppl-parse-command.md#limitations)
+- Limitation: [see limitations](commands/ppl-parse-command.md#limitations)
 
 #### **Rename**
-[See additional command details](ppl-rename-command.md)
+[See additional command details](commands/ppl-rename-command.md)
 
 - `source=accounts | rename email as user_email | fields id, user_email`
 - `source=accounts | rename id as user_id, email as user_email | fields user_id, user_email`
 
 
 #### **Join**
-[See additional command details](ppl-join-command.md)
+[See additional command details](commands/ppl-join-command.md)
 
 - `source = table1 | inner join left = l right = r on l.a = r.a table2 | fields l.a, r.a, b, c`
 - `source = table1 | left join left = l right = r on l.a = r.a table2 | fields l.a, r.a, b, c`
@@ -230,7 +230,7 @@ _- **Limitation: sub-searches is unsupported in join right side now**_
 
 
 #### **Lookup**
-[See additional command details](ppl-lookup-command.md)
+[See additional command details](commands/ppl-lookup-command.md)
 
 - `source = table1 | lookup table2 id`
 - `source = table1 | lookup table2 id, name`
@@ -245,7 +245,7 @@ _- **Limitation: "REPLACE" or "APPEND" clause must contain "AS"**_
 
 
 #### **InSubquery**
-[See additional command details](ppl-subquery-command.md)
+[See additional command details](commands/ppl-subquery-command.md)
 
 - `source = outer | where a in [ source = inner | fields b ]`
 - `source = outer | where (a) in [ source = inner | fields b ]`
@@ -343,7 +343,7 @@ source = supplier
 
 ---
 #### Experimental Commands:
-[See additional command details](ppl-correlation-command.md)
+[See additional command details](commands/ppl-correlation-command.md)
 
 ```sql
 - `source alb_logs, traces, metrics | where ip="10.0.0.1" AND cloud.provider="aws"| correlate exact on (ip, port) scope(@timestamp, 2018-07-02T22:23:00, 1 D)`
diff --git a/docs/ppl-lang/README.md b/docs/ppl-lang/README.md
index 16ff636f7..4496ea3f2 100644
--- a/docs/ppl-lang/README.md
+++ b/docs/ppl-lang/README.md
@@ -15,7 +15,7 @@ source=accounts
 ```
 
 For additional examples see the next [documentation](PPL-Example-Commands.md). 
-
+For the PPL planned release content [release-plan](planning/release-plan.md)
 ---
 ### Commands Specifications
 
@@ -24,43 +24,43 @@ For additional examples see the next [documentation](PPL-Example-Commands.md).
 
     - [`explain command `](PPL-Example-Commands.md/#explain)
   
-    - [`dedup command `](ppl-dedup-command.md)
+    - [`dedup command `](commands/ppl-dedup-command.md)
 
     - [`describe command`](PPL-Example-Commands.md/#describe)
 
-    - [`eval command`](ppl-eval-command.md)
+    - [`eval command`](commands/ppl-eval-command.md)
 
-    - [`fields command`](ppl-fields-command.md)
+    - [`fields command`](commands/ppl-fields-command.md)
 
-    - [`grok command`](ppl-grok-command.md)
+    - [`grok command`](commands/ppl-grok-command.md)
 
-    - [`parse command`](ppl-parse-command.md)
+    - [`parse command`](commands/ppl-parse-command.md)
 
-    - [`patterns command`](ppl-patterns-command.md)
+    - [`patterns command`](commands/ppl-patterns-command.md)
 
-    - [`rename command`](ppl-rename-command.md)
+    - [`rename command`](commands/ppl-rename-command.md)
 
-    - [`search command`](ppl-search-command.md)
+    - [`search command`](commands/ppl-search-command.md)
 
-    - [`sort command`](ppl-sort-command.md)
+    - [`sort command`](commands/ppl-sort-command.md)
 
-    - [`stats command`](ppl-stats-command.md)
+    - [`stats command`](commands/ppl-stats-command.md)
 
-    - [`where command`](ppl-where-command.md)
+    - [`where command`](commands/ppl-where-command.md)
 
-    - [`head command`](ppl-head-command.md)
+    - [`head command`](commands/ppl-head-command.md)
 
-    - [`rare command`](ppl-rare-command.md)
+    - [`rare command`](commands/ppl-rare-command.md)
 
-    - [`top command`](ppl-top-command.md)
+    - [`top command`](commands/ppl-top-command.md)
 
-    - [`join commands`](ppl-join-command.md)
+    - [`join commands`](commands/ppl-join-command.md)
   
-    - [`lookup commands`](ppl-lookup-command.md)
+    - [`lookup commands`](commands/ppl-lookup-command.md)
   
-    - [`subquery commands`](ppl-subquery-command.md)
+    - [`subquery commands`](commands/ppl-subquery-command.md)
   
-    - [`correlation commands`](ppl-correlation-command.md)
+    - [`correlation commands`](commands/ppl-correlation-command.md)
 
 
 * **Functions**
diff --git a/docs/ppl-lang/ppl-correlation-command.md b/docs/ppl-lang/commands/ppl-correlation-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-correlation-command.md
rename to docs/ppl-lang/commands/ppl-correlation-command.md
diff --git a/docs/ppl-lang/ppl-dedup-command.md b/docs/ppl-lang/commands/ppl-dedup-command.md
similarity index 93%
rename from docs/ppl-lang/ppl-dedup-command.md
rename to docs/ppl-lang/commands/ppl-dedup-command.md
index 28fe7f4a4..89e1a019f 100644
--- a/docs/ppl-lang/ppl-dedup-command.md
+++ b/docs/ppl-lang/commands/ppl-dedup-command.md
@@ -4,11 +4,6 @@
 
 - [Description](#description)
 - [Syntax](#syntax)
-- [Examples](#examples)
-    - [Example 1: Dedup by one field](#example-1-dedup-by-one-field)
-    - [Example 2: Keep 2 duplicates documents](#example-2-keep-2-duplicates-documents)
-    - [Example 3: Keep or Ignore the empty field by default](#example-3-keep-or-ignore-the-empty-field-by-default)
-    - [Example 4: Dedup in consecutive document](#example-4-dedup-in-consecutive-document)
 - [Limitation](#limitation)
 
 ## Description
diff --git a/docs/ppl-lang/ppl-eval-command.md b/docs/ppl-lang/commands/ppl-eval-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-eval-command.md
rename to docs/ppl-lang/commands/ppl-eval-command.md
diff --git a/docs/ppl-lang/ppl-fields-command.md b/docs/ppl-lang/commands/ppl-fields-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-fields-command.md
rename to docs/ppl-lang/commands/ppl-fields-command.md
diff --git a/docs/ppl-lang/ppl-grok-command.md b/docs/ppl-lang/commands/ppl-grok-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-grok-command.md
rename to docs/ppl-lang/commands/ppl-grok-command.md
diff --git a/docs/ppl-lang/ppl-head-command.md b/docs/ppl-lang/commands/ppl-head-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-head-command.md
rename to docs/ppl-lang/commands/ppl-head-command.md
diff --git a/docs/ppl-lang/ppl-join-command.md b/docs/ppl-lang/commands/ppl-join-command.md
similarity index 99%
rename from docs/ppl-lang/ppl-join-command.md
rename to docs/ppl-lang/commands/ppl-join-command.md
index 525373f7c..3c6400c5d 100644
--- a/docs/ppl-lang/ppl-join-command.md
+++ b/docs/ppl-lang/commands/ppl-join-command.md
@@ -162,7 +162,7 @@ SEARCH source=customer
 | SORT - custdist, - c_count
 ```
 
-### Comparison with [Correlation](ppl-correlation-command)
+### Comparison with [Correlation](ppl-correlation-command.md)
 
 A primary difference between `correlate` and `join` is that both sides of `correlate` are tables, but both sides of `join` are subqueries. 
 For example:
diff --git a/docs/ppl-lang/ppl-lookup-command.md b/docs/ppl-lang/commands/ppl-lookup-command.md
similarity index 95%
rename from docs/ppl-lang/ppl-lookup-command.md
rename to docs/ppl-lang/commands/ppl-lookup-command.md
index 1b8350533..13e57a006 100644
--- a/docs/ppl-lang/ppl-lookup-command.md
+++ b/docs/ppl-lang/commands/ppl-lookup-command.md
@@ -3,7 +3,7 @@
 ## Overview
 Lookup command enriches your search data by adding or replacing data from a lookup index (dimension table).
 You can extend fields of an index with values from a dimension table, append or replace values when lookup condition is matched.
-As an alternative of [Join command](ppl-join-command), lookup command is more suitable for enriching the source data with a static dataset.
+As an alternative of [Join command](ppl-join-command.md), lookup command is more suitable for enriching the source data with a static dataset.
 
 
 ### Syntax of Lookup Command
diff --git a/docs/ppl-lang/ppl-parse-command.md b/docs/ppl-lang/commands/ppl-parse-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-parse-command.md
rename to docs/ppl-lang/commands/ppl-parse-command.md
diff --git a/docs/ppl-lang/ppl-patterns-command.md b/docs/ppl-lang/commands/ppl-patterns-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-patterns-command.md
rename to docs/ppl-lang/commands/ppl-patterns-command.md
diff --git a/docs/ppl-lang/ppl-rare-command.md b/docs/ppl-lang/commands/ppl-rare-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-rare-command.md
rename to docs/ppl-lang/commands/ppl-rare-command.md
diff --git a/docs/ppl-lang/ppl-rename-command.md b/docs/ppl-lang/commands/ppl-rename-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-rename-command.md
rename to docs/ppl-lang/commands/ppl-rename-command.md
diff --git a/docs/ppl-lang/ppl-search-command.md b/docs/ppl-lang/commands/ppl-search-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-search-command.md
rename to docs/ppl-lang/commands/ppl-search-command.md
diff --git a/docs/ppl-lang/ppl-sort-command.md b/docs/ppl-lang/commands/ppl-sort-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-sort-command.md
rename to docs/ppl-lang/commands/ppl-sort-command.md
diff --git a/docs/ppl-lang/ppl-stats-command.md b/docs/ppl-lang/commands/ppl-stats-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-stats-command.md
rename to docs/ppl-lang/commands/ppl-stats-command.md
diff --git a/docs/ppl-lang/ppl-subquery-command.md b/docs/ppl-lang/commands/ppl-subquery-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-subquery-command.md
rename to docs/ppl-lang/commands/ppl-subquery-command.md
diff --git a/docs/ppl-lang/ppl-top-command.md b/docs/ppl-lang/commands/ppl-top-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-top-command.md
rename to docs/ppl-lang/commands/ppl-top-command.md
diff --git a/docs/ppl-lang/ppl-where-command.md b/docs/ppl-lang/commands/ppl-where-command.md
similarity index 100%
rename from docs/ppl-lang/ppl-where-command.md
rename to docs/ppl-lang/commands/ppl-where-command.md

From ba2a06223ed20d1759ddef921ebb7b7763b72c35 Mon Sep 17 00:00:00 2001
From: YANGDB <yang.db.dev@gmail.com>
Date: Tue, 8 Oct 2024 17:36:47 -0700
Subject: [PATCH 3/5] update release with existing release artifacts

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
---
 docs/ppl-lang/planning/release-plan.md | 32 +++++++++++++++++---------
 1 file changed, 21 insertions(+), 11 deletions(-)

diff --git a/docs/ppl-lang/planning/release-plan.md b/docs/ppl-lang/planning/release-plan.md
index c016ac8d0..365b64101 100644
--- a/docs/ppl-lang/planning/release-plan.md
+++ b/docs/ppl-lang/planning/release-plan.md
@@ -3,15 +3,27 @@ The next doc present a release plan summarizing the commands / features planned
 
 Each release will have a list of all the next parts:
  - commands / functions
- - limitations
+ - limitations:
+   - deprecated
+   - breaking-changes
  - comments
+ - 
 
 Each command / function would have the following properties:
  - `status`: the maturity of the feature (experimental, stable, deprecated)
  - `since`: the version it was introduced 
 
+#### SemVer Support in PPL Releases:
+Each release will follow the SemVer versioning system:
+
+- `MAJOR` version when there are incompatible API changes or breaking changes to existing commands and functionality.
+- `MINOR` version when adding new features in a backward-compatible manner, including new commands or enhancements to the existing system.
+- `PATCH` version when making backward-compatible bug fixes, performance improvements, or minor updates.
+
 ---
-## Version 0.4
+## [Version 0.4.1](https://github.com/opensearch-project/opensearch-spark/releases/tag/v0.4.1)
+
+**Status**: latest
 
 ### Commands
 | command           | status       | since |
@@ -34,7 +46,9 @@ Each command / function would have the following properties:
 
 
 ---
-## Version 0.5
+## [Version 0.5.0](https://github.com/opensearch-project/opensearch-spark/releases/tag/v0.5.0)
+
+**Status**: pre-release 
 
 ### Commands
 | command           | status       | since |
@@ -76,7 +90,8 @@ Each command / function would have the following properties:
 - `dedup` command with `allowedDuplication > 1` feature needs spark version >= 3.4
 
 ---
-## Version 0.6
+## Version 0.6.0 (planned)
+**Status**: planned
 
 ### Commands
 | command               | status       | since |
@@ -122,7 +137,8 @@ Each command / function would have the following properties:
 | coalesce           | experimental | 0.6   |
 
 ---
-## Version 0.7
+## Version 0.7.0 (planned)
+**Status**: planned
 
 ### Commands
 | command               | status       | since |
@@ -177,9 +193,3 @@ Each command / function would have the following properties:
 | between            | experimental | 0.7   |
 
 
----
-## Version 0.7
-
-### Commands
-
-### Functions

From 7173e40f2fe15b86441b8dd5898f8e43011dd065 Mon Sep 17 00:00:00 2001
From: YANGDB <yang.db.dev@gmail.com>
Date: Fri, 11 Oct 2024 11:30:33 -0700
Subject: [PATCH 4/5] update doc for each command with  - spark  - status  -
 introduced in

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
---
 .../commands/ppl-correlation-command.md       |  20 +-
 docs/ppl-lang/commands/ppl-dedup-command.md   |  20 +-
 docs/ppl-lang/commands/ppl-eval-command.md    |  18 +-
 docs/ppl-lang/commands/ppl-fields-command.md  |  16 ++
 docs/ppl-lang/commands/ppl-grok-command.md    |  17 +-
 docs/ppl-lang/commands/ppl-head-command.md    |  15 ++
 docs/ppl-lang/commands/ppl-join-command.md    |  15 ++
 docs/ppl-lang/commands/ppl-lookup-command.md  |  16 ++
 docs/ppl-lang/commands/ppl-parse-command.md   |  14 +
 .../ppl-lang/commands/ppl-patterns-command.md |  15 ++
 docs/ppl-lang/commands/ppl-rare-command.md    |  15 ++
 docs/ppl-lang/commands/ppl-rename-command.md  |  21 +-
 docs/ppl-lang/commands/ppl-search-command.md  |  15 ++
 docs/ppl-lang/commands/ppl-sort-command.md    |  15 ++
 docs/ppl-lang/commands/ppl-stats-command.md   |  16 ++
 .../ppl-lang/commands/ppl-subquery-command.md |  15 ++
 docs/ppl-lang/commands/ppl-top-command.md     |  15 ++
 docs/ppl-lang/commands/ppl-where-command.md   |  15 ++
 docs/ppl-lang/planning/release-plan.md        | 248 +++++++-----------
 19 files changed, 386 insertions(+), 155 deletions(-)

diff --git a/docs/ppl-lang/commands/ppl-correlation-command.md b/docs/ppl-lang/commands/ppl-correlation-command.md
index 2e8507a14..bdebf1b85 100644
--- a/docs/ppl-lang/commands/ppl-correlation-command.md
+++ b/docs/ppl-lang/commands/ppl-correlation-command.md
@@ -1,7 +1,19 @@
-## PPL Correlation Command
-
-> This is an experimental command - it may be removed in future versions
-
+## PPL `correlation` Command
+
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:yellow;">Experimental</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.4.0</th>
+  </tr>
+</table>
 
 ## Overview
 
diff --git a/docs/ppl-lang/commands/ppl-dedup-command.md b/docs/ppl-lang/commands/ppl-dedup-command.md
index 89e1a019f..18b2f8c15 100644
--- a/docs/ppl-lang/commands/ppl-dedup-command.md
+++ b/docs/ppl-lang/commands/ppl-dedup-command.md
@@ -1,4 +1,20 @@
-# PPL dedup command
+# PPL `dedup` command
+
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.5.1+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:yellow;">Experimental</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.5.0</th>
+  </tr>
+</table>
+
 
 ## Table of contents
 
@@ -121,7 +137,7 @@ PPL query:
 
 ### Limitation:
 
-**Spark Support** ( >= 3.4)
+**Spark Support** ( 3.5.1 +)
 
 To translate `dedup` command with `allowedDuplication > 1`, such as `| dedup 2 a,b` to Spark plan, the solution is translating to a plan with Window function (e.g row_number) and a new column `row_number_col` as Filter.
  
diff --git a/docs/ppl-lang/commands/ppl-eval-command.md b/docs/ppl-lang/commands/ppl-eval-command.md
index cd0898c1b..69a957bcb 100644
--- a/docs/ppl-lang/commands/ppl-eval-command.md
+++ b/docs/ppl-lang/commands/ppl-eval-command.md
@@ -1,5 +1,21 @@
 # PPL `eval` command
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.5.1+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:yellow;">Experimental</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.5.0</th>
+  </tr>
+</table>
+
+
 ## Description
  The ``eval`` command evaluate the expression and append the result to the search result.
 
@@ -106,7 +122,7 @@ eval status_category =
 ```
 
 ### Limitation:
- - `eval` with comma separated expression needs spark version >= 3.4
+**Spark Support** ( 3.5.1 +)
 
  - Overriding existing field is unsupported, following queries throw exceptions with "Reference 'a' is ambiguous"
 
diff --git a/docs/ppl-lang/commands/ppl-fields-command.md b/docs/ppl-lang/commands/ppl-fields-command.md
index e37fc644f..ec54942bd 100644
--- a/docs/ppl-lang/commands/ppl-fields-command.md
+++ b/docs/ppl-lang/commands/ppl-fields-command.md
@@ -1,5 +1,21 @@
 ## PPL `fields` command
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:greenyellow;">Stable</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.4.0</th>
+  </tr>
+</table>
+
+
 **Description**
 Using ``field`` command to keep or remove fields from the search result.
 
diff --git a/docs/ppl-lang/commands/ppl-grok-command.md b/docs/ppl-lang/commands/ppl-grok-command.md
index 06028109b..3bb2b2a67 100644
--- a/docs/ppl-lang/commands/ppl-grok-command.md
+++ b/docs/ppl-lang/commands/ppl-grok-command.md
@@ -1,4 +1,19 @@
-## PPL Correlation Command
+## PPL `grok` Command
+
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:yellow;">Experimental</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.5.0</th>
+  </tr>
+</table>
 
 
 ### Description
diff --git a/docs/ppl-lang/commands/ppl-head-command.md b/docs/ppl-lang/commands/ppl-head-command.md
index e4172b1c6..7c1be26ca 100644
--- a/docs/ppl-lang/commands/ppl-head-command.md
+++ b/docs/ppl-lang/commands/ppl-head-command.md
@@ -1,5 +1,20 @@
 ## PPL `head` Command
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:greenyellow;">Stable</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.4.0</th>
+  </tr>
+</table>
+
 **Description**
 The ``head`` command returns the first N number of specified results after an optional offset in search order.
 
diff --git a/docs/ppl-lang/commands/ppl-join-command.md b/docs/ppl-lang/commands/ppl-join-command.md
index 3c6400c5d..bf1435540 100644
--- a/docs/ppl-lang/commands/ppl-join-command.md
+++ b/docs/ppl-lang/commands/ppl-join-command.md
@@ -1,5 +1,20 @@
 ## PPL Join Command
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:yellow;">Experimental</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.6.0</th>
+  </tr>
+</table>
+
 ## Overview
 
 [Trace analytics](https://opensearch.org/docs/latest/observability-plugin/trace/ta-dashboards/) considered using SQL/PPL for its queries, but some graphs rely on joining two indices (span index and service map index) together which is not supported by SQL/PPL. Trace analytics was implemented with DSL + javascript, would be good if `join` being added to SQL could support this use case.
diff --git a/docs/ppl-lang/commands/ppl-lookup-command.md b/docs/ppl-lang/commands/ppl-lookup-command.md
index 13e57a006..79cdf7a67 100644
--- a/docs/ppl-lang/commands/ppl-lookup-command.md
+++ b/docs/ppl-lang/commands/ppl-lookup-command.md
@@ -1,5 +1,21 @@
 ## PPL Lookup Command
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.5.1+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:yellow;">Experimental</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.6.0</th>
+  </tr>
+</table>
+
+
 ## Overview
 Lookup command enriches your search data by adding or replacing data from a lookup index (dimension table).
 You can extend fields of an index with values from a dimension table, append or replace values when lookup condition is matched.
diff --git a/docs/ppl-lang/commands/ppl-parse-command.md b/docs/ppl-lang/commands/ppl-parse-command.md
index 10be21cc0..803a065cd 100644
--- a/docs/ppl-lang/commands/ppl-parse-command.md
+++ b/docs/ppl-lang/commands/ppl-parse-command.md
@@ -1,5 +1,19 @@
 ## PPL Parse Command
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:yellow;">Experimental</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.5.0</th>
+  </tr>
+</table>
 
 ### Description
 The ``parse`` command parses a text field with a regular expression and appends the result to the search result.
diff --git a/docs/ppl-lang/commands/ppl-patterns-command.md b/docs/ppl-lang/commands/ppl-patterns-command.md
index c20e01944..bb267dc59 100644
--- a/docs/ppl-lang/commands/ppl-patterns-command.md
+++ b/docs/ppl-lang/commands/ppl-patterns-command.md
@@ -1,5 +1,20 @@
 ## PPL `patterns` command 
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:yellow;">Experimental</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.5.0</th>
+  </tr>
+</table>
+
 ### Description
  The ``patterns`` command extracts log patterns from a text field and appends the results to the search result. Grouping logs by their patterns makes it easier to aggregate stats from large volumes of log data for analysis and troubleshooting.
 
diff --git a/docs/ppl-lang/commands/ppl-rare-command.md b/docs/ppl-lang/commands/ppl-rare-command.md
index 5645382f8..c97e93eb5 100644
--- a/docs/ppl-lang/commands/ppl-rare-command.md
+++ b/docs/ppl-lang/commands/ppl-rare-command.md
@@ -1,5 +1,20 @@
 ## PPL rare Command
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:yellow;">Experimental</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.5.0</th>
+  </tr>
+</table>
+
 **Description**
 Using ``rare`` command to find the least common tuple of values of all fields in the field list.
 
diff --git a/docs/ppl-lang/commands/ppl-rename-command.md b/docs/ppl-lang/commands/ppl-rename-command.md
index 8a3e4e3b5..18acc6b5f 100644
--- a/docs/ppl-lang/commands/ppl-rename-command.md
+++ b/docs/ppl-lang/commands/ppl-rename-command.md
@@ -1,5 +1,22 @@
 ## PPL `rename` command
 
+
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.5.1+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:yellow;">Experimental</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.5.0</th>
+  </tr>
+</table>
+
+
 ### Description
 Using ``rename`` command to rename one or more fields in the search result.
 
@@ -47,7 +64,9 @@ PPL query:
     +------+---------+
 
 ### Limitation:
-- `rename` command needs spark version >= 3.4
+
+**Spark Support** ( 3.5.1 +)
+
 
 - Overriding existing field is unsupported:
 
diff --git a/docs/ppl-lang/commands/ppl-search-command.md b/docs/ppl-lang/commands/ppl-search-command.md
index f81d9d907..fbc26916a 100644
--- a/docs/ppl-lang/commands/ppl-search-command.md
+++ b/docs/ppl-lang/commands/ppl-search-command.md
@@ -1,5 +1,20 @@
 ## PPL `search` command
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:greenyellow;">Stable</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.4.0</th>
+  </tr>
+</table>
+
 ### Description
 Using ``search`` command to retrieve document from the index. ``search`` command could be only used as the first command in the PPL query.
 
diff --git a/docs/ppl-lang/commands/ppl-sort-command.md b/docs/ppl-lang/commands/ppl-sort-command.md
index c3bf304d7..1f8087f56 100644
--- a/docs/ppl-lang/commands/ppl-sort-command.md
+++ b/docs/ppl-lang/commands/ppl-sort-command.md
@@ -1,5 +1,20 @@
 ## PPL `sort`command
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:greenyellow;">Stable</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.4.0</th>
+  </tr>
+</table>
+
 ### Description
 Using ``sort`` command to sorts all the search result by the specified fields.
 
diff --git a/docs/ppl-lang/commands/ppl-stats-command.md b/docs/ppl-lang/commands/ppl-stats-command.md
index 552f83e46..63473dfc0 100644
--- a/docs/ppl-lang/commands/ppl-stats-command.md
+++ b/docs/ppl-lang/commands/ppl-stats-command.md
@@ -1,5 +1,21 @@
 ## PPL `stats` command
 
+
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:greenyellow;">Stable</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.4.0</th>
+  </tr>
+</table>
+
 ### Description
 Using ``stats`` command to calculate the aggregation from search result.
  
diff --git a/docs/ppl-lang/commands/ppl-subquery-command.md b/docs/ppl-lang/commands/ppl-subquery-command.md
index 1762306d2..ff3710c43 100644
--- a/docs/ppl-lang/commands/ppl-subquery-command.md
+++ b/docs/ppl-lang/commands/ppl-subquery-command.md
@@ -1,5 +1,20 @@
 ## PPL SubQuery Commands:
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:yellow;">Experimental</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.6.0</th>
+  </tr>
+</table>
+
 **Syntax**
 The subquery command should be implemented using a clean, logical syntax that integrates with existing PPL structure.
 
diff --git a/docs/ppl-lang/commands/ppl-top-command.md b/docs/ppl-lang/commands/ppl-top-command.md
index 4ba56f692..e635c5c53 100644
--- a/docs/ppl-lang/commands/ppl-top-command.md
+++ b/docs/ppl-lang/commands/ppl-top-command.md
@@ -1,5 +1,20 @@
 ## PPL top Command
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:greenyellow;">Stable</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.4.0</th>
+  </tr>
+</table>
+
 **Description**
 Using ``top`` command to find the most common tuple of values of all fields in the field list.
 
diff --git a/docs/ppl-lang/commands/ppl-where-command.md b/docs/ppl-lang/commands/ppl-where-command.md
index 94ddc1f5c..75c283766 100644
--- a/docs/ppl-lang/commands/ppl-where-command.md
+++ b/docs/ppl-lang/commands/ppl-where-command.md
@@ -1,5 +1,20 @@
 ## PPL where Command
 
+<table>
+  <tr>
+    <th style="color:gainsboro;">Spark</th>
+    <th style="color:greenyellow;">3.0.0+ </th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Status</th>
+    <th style="color:greenyellow;">Stable</th>
+  </tr>
+  <tr>
+    <th style="color:gainsboro;">Introduced In</th>
+    <th style="color:lightgreen;">0.4.0</th>
+  </tr>
+</table>
+
 ### Description
 The ``where`` command bool-expression to filter the search result. The ``where`` command only return the result when bool-expression evaluated to true.
 
diff --git a/docs/ppl-lang/planning/release-plan.md b/docs/ppl-lang/planning/release-plan.md
index 365b64101..2f83ee670 100644
--- a/docs/ppl-lang/planning/release-plan.md
+++ b/docs/ppl-lang/planning/release-plan.md
@@ -7,7 +7,15 @@ Each release will have a list of all the next parts:
    - deprecated
    - breaking-changes
  - comments
- - 
+
+### Command metadata categories
+
+- `experimental`: Commands that are new, not fully tested, or still in the final stages of development. These features may change or be removed in future releases without warning.
+
+- `stable`: Commands that are fully tested, documented, and considered reliable for general use. These features are expected to remain consistent across releases, with no breaking changes unless explicitly stated.
+
+- `deprecated`: Commands that are planned for removal in future releases. While they may still work, their usage is discouraged, and alternative stable commands should be used when available.
+
 
 Each command / function would have the following properties:
  - `status`: the maturity of the feature (experimental, stable, deprecated)
@@ -23,173 +31,121 @@ Each release will follow the SemVer versioning system:
 ---
 ## [Version 0.4.1](https://github.com/opensearch-project/opensearch-spark/releases/tag/v0.4.1)
 
-**Status**: latest
+**Status**: stable
 
 ### Commands
-| command           | status       | since |
-|-------------------|--------------|-------|
-| fields            | stable       | 0.4   |
-| search            | stable       | 0.4   |
-| sort              | stable       | 0.4   |
-| stats             | stable       | 0.4   |
-| stats + span      | stable       | 0.4   |
-| stats + span + by | stable       | 0.4   |
-| where             | stable       | 0.4   |
-| head              | stable       | 0.4   |
-| correlation       | experimental | 0.4   |
+| command           | status       | since | doc |
+|-------------------|--------------|-------|-----|
+| fields            | stable       | 0.4   |[ppl-fields-command.md](../commands/ppl-fields-command.md)|
+| search            | stable       | 0.4   |[ppl-search-command.md](../commands/ppl-search-command.md)|
+| sort              | stable       | 0.4   |[ppl-sort-command.md](../commands/ppl-sort-command.md)|
+| stats             | stable       | 0.4   |[ppl-stats-command.md](../commands/ppl-stats-command.md)|
+| stats + span      | stable       | 0.4   |[ppl-stats-command.md](../commands/ppl-stats-command.md)|
+| stats + span + by | stable       | 0.4   |[ppl-stats-command.md](../commands/ppl-stats-command.md)|
+| where             | stable       | 0.4   |[ppl-where-command.md](../commands/ppl-where-command.md)|
+| head              | stable       | 0.4   |[ppl-head-command.md](../commands/ppl-head-command.md)|
+| correlation       | experimental | 0.4   |[ppl-correlation-command.md](../commands/ppl-correlation-command.md)|
 
 ### Functions
 
-| function           | status  | since |
-|--------------------|---------|-------|
-| logical conditions | stable  | 0.4   |
+| function           | status  | since | doc |
+|--------------------|---------|-------|-----|
+| logical conditions | stable  | 0.4   |[ppl-expressions.md](../functions/ppl-expressions.md)|
 
 
 ---
 ## [Version 0.5.0](https://github.com/opensearch-project/opensearch-spark/releases/tag/v0.5.0)
 
-**Status**: pre-release 
+**Status**: latest
 
 ### Commands
-| command           | status       | since |
-|-------------------|--------------|-------|
-| fields            | stable       | 0.4   |
-| search            | stable       | 0.4   |
-| sort              | stable       | 0.4   |
-| stats             | stable       | 0.4   |
-| stats + span      | stable       | 0.4   |
-| stats + span + by | stable       | 0.4   |
-| where             | stable       | 0.4   |
-| head              | stable       | 0.4   |
-| correlation       | experimental | 0.4   |
-| explain           | experimental | 0.5   |
-| eval              | experimental | 0.5   |
-| dedup             | stable       | 0.5   |
-| describe          | stable       | 0.5   |
-| nested-fields     | experimental | 0.5   |
-| grok              | experimental | 0.5   |
-| parse             | experimental | 0.5   |
-| patterns          | experimental | 0.5   |
-| rename            | experimental | 0.5   |
-| rare              | experimental | 0.5   |
-| top               | experimental | 0.5   |
+| command           | status       | since | doc |
+|-------------------|--------------|-------|-----|
+| fields            | stable       | 0.4   |[ppl-fields-command.md](../commands/ppl-fields-command.md)|
+| search            | stable       | 0.4   |[ppl-search-command.md](../commands/ppl-search-command.md)|
+| sort              | stable       | 0.4   |[ppl-sort-command.md](../commands/ppl-sort-command.md)|
+| stats             | stable       | 0.4   |[ppl-stats-command.md](../commands/ppl-stats-command.md)|
+| stats + span      | stable       | 0.4   |[ppl-stats-command.md](../commands/ppl-stats-command.md)|
+| stats + span + by | stable       | 0.4   |[ppl-stats-command.md](../commands/ppl-stats-command.md)|
+| where             | stable       | 0.4   |[ppl-where-command.md](../commands/ppl-where-command.md)|
+| head              | stable       | 0.4   |[ppl-head-command.md](../commands/ppl-head-command.md)|
+| correlation       | experimental | 0.4   |[ppl-correlation-command.md](../commands/ppl-correlation-command.md)|
+| explain           | experimental | 0.5   ||
+| eval              | experimental | 0.5   |[ppl-eval-command.md](../commands/ppl-eval-command.md)|
+| dedup             | experimental | 0.5   |[ppl-dedup-command.md](../commands/ppl-dedup-command.md)|
+| describe          | experimental | 0.5   ||
+| nested-fields     | experimental | 0.5   |[ppl-subquery-command.md](../commands/ppl-subquery-command.md)|
+| grok              | experimental | 0.5   |[ppl-grok-command.md](../commands/ppl-grok-command.md)|
+| parse             | experimental | 0.5   |[ppl-parse-command.md](../commands/ppl-parse-command.md)|
+| patterns          | experimental | 0.5   |[ppl-patterns-command.md](../commands/ppl-patterns-command.md)|
+| rename            | experimental | 0.5   |[ppl-rename-command.md](../commands/ppl-rename-command.md)|
+| rare              | experimental | 0.5   |[ppl-rare-command.md](../commands/ppl-rare-command.md)|
+| top               | experimental | 0.5   |[ppl-top-command.md](../commands/ppl-top-command.md)|
 
 
 ### Functions
 
-| function           | status       | since |
-|--------------------|--------------|-------|
-| logical conditions | stable       | 0.4   |
-| case               | experimental | 0.5   |
-| if                 | experimental | 0.5   | 
-| filed (- / +)      | experimental | 0.5   | 
+| function           | status       | since | doc |
+|--------------------|--------------|-------|-----|
+| logical conditions | stable       | 0.4   |[ppl-expressions.md](../functions/ppl-expressions.md)|
+| case               | experimental | 0.5   ||
+| if                 | experimental | 0.5   || 
+| filed (- / +)      | experimental | 0.5   ||
 
 ### Limitations
-- `fields - list` shows incorrect results for spark version 3.3 - see [issue](https://github.com/opensearch-project/opensearch-spark/pull/732)
-- `eval` with comma separated expression needs spark version >= 3.4
-- `dedup` command with `allowedDuplication > 1` feature needs spark version >= 3.4
+- `fields - list` shows incorrect results for spark version 3.5.1 - see [issue](https://github.com/opensearch-project/opensearch-spark/pull/732)
+- `eval` with comma separated expression needs spark version >= 3.5.1
+- `rename` needs spark version >= 3.5.1
+- `dedup` command with `allowedDuplication > 1` feature needs spark version >= 3.5.1
 
 ---
-## Version 0.6.0 (planned)
+## Version 0.6.0 (pre-release)
 **Status**: planned
 
 ### Commands
-| command               | status       | since |
-|-----------------------|--------------|-------|
-| fields                | stable       | 0.4   |
-| search                | stable       | 0.4   |
-| sort                  | stable       | 0.4   |
-| stats                 | stable       | 0.4   |
-| stats + span          | stable       | 0.4   |
-| stats + span + by     | stable       | 0.4   |
-| where                 | stable       | 0.4   |
-| head                  | stable       | 0.4   |
-| correlation           | experimental | 0.4   |
-| explain               | experimental | 0.5   |
-| eval                  | experimental | 0.5   |
-| dedup                 | stable       | 0.5   |
-| describe              | stable       | 0.5   |
-| nested-fields         | experimental | 0.5   |
-| grok                  | experimental | 0.5   |
-| parse                 | experimental | 0.5   |
-| patterns              | experimental | 0.5   |
-| rename                | experimental | 0.5   |
-| rare                  | experimental | 0.5   |
-| top                   | experimental | 0.5   |
-| join                  | experimental | 0.6   |
-| lookup                | experimental | 0.6   |
-| inSubQuery (subQuery) | experimental | 0.6   |
-| help                  | experimental | 0.6   |
-| fieldsummary          | experimental | 0.6   |
-
-
+| command               | status       | since | doc                                                                  |
+|-----------------------|--------------|-------|----------------------------------------------------------------------|
+| fields            | stable       | 0.4   | [ppl-fields-command.md](../commands/ppl-fields-command.md)           |
+| search            | stable       | 0.4   | [ppl-search-command.md](../commands/ppl-search-command.md)           |
+| sort              | stable       | 0.4   | [ppl-sort-command.md](../commands/ppl-sort-command.md)               |
+| stats             | stable       | 0.4   | [ppl-stats-command.md](../commands/ppl-stats-command.md)             |
+| stats + span      | stable       | 0.4   | [ppl-stats-command.md](../commands/ppl-stats-command.md)             |
+| stats + span + by | stable       | 0.4   | [ppl-stats-command.md](../commands/ppl-stats-command.md)             |
+| where             | stable       | 0.4   | [ppl-where-command.md](../commands/ppl-where-command.md)             |
+| head              | stable       | 0.4   | [ppl-head-command.md](../commands/ppl-head-command.md)               |
+| correlation       | experimental | 0.4   | [ppl-correlation-command.md](../commands/ppl-correlation-command.md) |
+| explain           | experimental | 0.5   |                                                                      |
+| eval              | experimental | 0.5   | [ppl-eval-command.md](../commands/ppl-eval-command.md)               |
+| dedup             | experimental | 0.5   | [ppl-dedup-command.md](../commands/ppl-dedup-command.md)             |
+| describe          | experimental | 0.5   |                                                                      |
+| nested-fields     | experimental | 0.5   | [ppl-subquery-command.md](../commands/ppl-subquery-command.md)       |
+| grok              | experimental | 0.5   | [ppl-grok-command.md](../commands/ppl-grok-command.md)               |
+| parse             | experimental | 0.5   | [ppl-parse-command.md](../commands/ppl-parse-command.md)             |
+| patterns          | experimental | 0.5   | [ppl-patterns-command.md](../commands/ppl-patterns-command.md)       |
+| rename            | experimental | 0.5   | [ppl-rename-command.md](../commands/ppl-rename-command.md)           |
+| rare              | experimental | 0.5   | [ppl-rare-command.md](../commands/ppl-rare-command.md)               |
+| top               | experimental | 0.5   | [ppl-top-command.md](../commands/ppl-top-command.md)                 |
+| join                  | experimental | 0.6   | [ppl-join-command.md](../commands/ppl-join-command.md)               |
+| lookup                | experimental | 0.6   | [ppl-lookup-command.md](../commands/ppl-lookup-command.md)           |
+| inSubQuery (subQuery) | experimental | 0.6   | [ppl-subquery-command.md](../commands/ppl-subquery-command.md)       |
+| help                  | experimental | 0.6   |                                                                      |
+| fieldsummary          | experimental | 0.6   | WIP                                                                  |
+| trendline             | experimental | 0.6   | WIP                                                                  |
+| fillnull              | experimental | 0.6   | WIP                                                                  |
+| flatten               | experimental | 0.6   | WIP                                                                  |
+| expand_field          | experimental | 0.6   | WIP                                                                  |
+| CIDIR (IP)            | experimental | 0.6   | WIP                                                                  |
+| IP Location           | experimental | 0.6   | WIP                                                                  |
 ### Functions
 
-| function           | status       | since |
-|--------------------|--------------|-------|
-| logical conditions | stable       | 0.4   |
-| case               | experimental | 0.5   |
-| if                 | experimental | 0.5   |
-| isEmpty            | experimental | 0.5   |
-| isPresent          | experimental | 0.5   |
-| filed (- / +)      | experimental | 0.5   | 
-| isBlank            | experimental | 0.6   |
-| coalesce           | experimental | 0.6   |
-
----
-## Version 0.7.0 (planned)
-**Status**: planned
-
-### Commands
-| command               | status       | since |
-|-----------------------|--------------|-------|
-| fields                | stable       | 0.4   |
-| search                | stable       | 0.4   |
-| sort                  | stable       | 0.4   |
-| stats                 | stable       | 0.4   |
-| stats + span          | stable       | 0.4   |
-| stats + span + by     | stable       | 0.4   |
-| where                 | stable       | 0.4   |
-| head                  | stable       | 0.4   |
-| correlation           | experimental | 0.4   |
-| explain               | experimental | 0.5   |
-| eval                  | experimental | 0.5   |
-| dedup                 | stable       | 0.5   |
-| describe              | stable       | 0.5   |
-| nested-fields         | experimental | 0.5   |
-| grok                  | experimental | 0.5   |
-| parse                 | experimental | 0.5   |
-| patterns              | experimental | 0.5   |
-| rename                | experimental | 0.5   |
-| rare                  | experimental | 0.5   |
-| top                   | experimental | 0.5   |
-| join                  | experimental | 0.6   |
-| lookup                | experimental | 0.6   |
-| inSubQuery (subQuery) | experimental | 0.6   |
-| help                  | experimental | 0.6   |
-| fieldsummary          | experimental | 0.6   |
-| input                 | experimental | 0.7   |
-| file  (as source)     | experimental | 0.7   |
-| trendline             | experimental | 0.7   |
-| fillnull              | experimental | 0.7   |
-| flatten               | experimental | 0.7   |
-| expand_field          | experimental | 0.7   |
-| expand_field          | experimental | 0.7   |
-| CIDIR (IP)            | experimental | 0.7   |
-| IP Location           | experimental | 0.7   |
-
-### Functions
-
-| function           | status       | since |
-|--------------------|--------------|-------|
-| logical conditions | stable       | 0.4   |
-| case               | experimental | 0.5   |
-| if                 | experimental | 0.5   |
-| isEmpty            | experimental | 0.5   |
-| isPresent          | experimental | 0.5   |
-| filed (- / +)      | experimental | 0.5   | 
-| isBlank            | experimental | 0.6   |
-| coalesce           | experimental | 0.6   |
-| between            | experimental | 0.7   |
-
-
+| function           | status       | since | doc |
+|--------------------|--------------|-------|----|
+| logical conditions | stable       | 0.4   |[ppl-expressions.md](../functions/ppl-expressions.md)|
+| case               | experimental | 0.5   ||
+| if                 | experimental | 0.5   ||
+| isEmpty            | experimental | 0.5   ||
+| isPresent          | experimental | 0.5   ||
+| filed (- / +)      | experimental | 0.5   ||
+| isBlank            | experimental | 0.6   ||
+| coalesce           | experimental | 0.6   ||
+| between            | experimental | 0.7   ||

From 45d88e3dd7bebb28f1352da24cb86d4ecc18d45f Mon Sep 17 00:00:00 2001
From: YANGDB <yang.db.dev@gmail.com>
Date: Fri, 11 Oct 2024 15:55:06 -0700
Subject: [PATCH 5/5] update doc for each command with  - spark  - status  -
 introduced in

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
---
 .../{release-plan.md => release-doc.md}       | 27 +++++++++++++++++++
 1 file changed, 27 insertions(+)
 rename docs/ppl-lang/planning/{release-plan.md => release-doc.md} (86%)

diff --git a/docs/ppl-lang/planning/release-plan.md b/docs/ppl-lang/planning/release-doc.md
similarity index 86%
rename from docs/ppl-lang/planning/release-plan.md
rename to docs/ppl-lang/planning/release-doc.md
index 2f83ee670..2b4108e12 100644
--- a/docs/ppl-lang/planning/release-plan.md
+++ b/docs/ppl-lang/planning/release-doc.md
@@ -28,6 +28,33 @@ Each release will follow the SemVer versioning system:
 - `MINOR` version when adding new features in a backward-compatible manner, including new commands or enhancements to the existing system.
 - `PATCH` version when making backward-compatible bug fixes, performance improvements, or minor updates.
 
+
+### Release Process in OpenSearch - Spark
+The following process defines the release strategy that OpenSearch Spark will follow:
+
+#### Alignment with OpenSearch Project:
+OpenSearch Spark will align its release cadence with the OpenSearch project, ensuring consistency in versions and support between the two projects.
+
+#### Branching Strategy:
+A new release tag and branch will be created for each major release cycle, following the completion of the previous release.
+
+#### Code Freeze and Pre-release Artifact:
+
+- Code Freeze: Prior to the official release, there will be a designated code freeze period. During this time, only critical bug fixes will be permitted, and no new features will be added - designated to the pre-release.
+- Pre-release Artifact: A pre-release artifact will be generated and made available for testing. This artifact will be thoroughly tested, and the associated documentation will be updated to reflect the changes.
+- Testing and Validation: Rigorous testing (including regression, integration, and performance tests) will be conducted on the pre-release artifact
+
+#### Release Preparation:
+
+- Final updates to documentation, including version notes, will be made.
+- Release artifacts will be built, and the final release branch will be created.
+- The new release will be announced and made available to the community via the OpenSearch release channels.
+
+#### Post-release:
+
+Post-release monitoring will be conducted to ensure no critical issues are found after the release.
+Any identified issues will be fixed in PATCH versions, which will be released as necessary.
+
 ---
 ## [Version 0.4.1](https://github.com/opensearch-project/opensearch-spark/releases/tag/v0.4.1)