Merge remote-tracking branch 'upstream/main' into issues/806

docs/ppl-lang/PPL-Example-Commands.md

@@ -58,8 +58,16 @@ _- **Limitation: new field added by eval command with a function cannot be dropp
 - `source = table | where a not in (1, 2, 3) | fields a,b,c`
 - `source = table | where a between 1 and 4` - Note: This returns a >= 1 and a <= 4, i.e. [1, 4]
 - `source = table | where b not between '2024-09-10' and '2025-09-10'` - Note: This returns b >= '2024-09-10' and b <= '2025-09-10'
-- `source = table | where cidrmatch(ip, '192.169.1.0/24')`
+- `source = table | where cidrmatch(ip, '192.169.1.0/24')` 
 - `source = table | where cidrmatch(ipv6, '2003:db8::/32')`
+- `source = table | trendline sma(2, temperature) as temp_trend`
+
+#### **IP related queries**
+[See additional command details](functions/ppl-ip.md)
+
+- `source = table | where cidrmatch(ip, '192.169.1.0/24')`
+- `source = table | where isV6 = false and isValid = true and cidrmatch(ipAddress, '192.168.1.0/24')`
+- `source = table | where isV6 = true | eval inRange = case(cidrmatch(ipAddress, '2003:db8::/32'), 'in' else 'out') | fields ip, inRange`
 
 ```sql
  source = table | eval status_category =
@@ -122,6 +130,15 @@ Assumptions: `a`, `b`, `c`, `d`, `e` are existing fields in `table`
 - `source = table | fillnull using a = 101, b = 102`
 - `source = table | fillnull using a = concat(b, c), d = 2 * pi() * e`
 
+### Flatten
+[See additional command details](ppl-flatten-command.md)
+Assumptions: `bridges`, `coor` are existing fields in `table`, and the field's types are `struct<?,?>` or `array<struct<?,?>>`  
+- `source = table | flatten bridges`
+- `source = table | flatten coor`
+- `source = table | flatten bridges | flatten coor`
+- `source = table | fields bridges | flatten bridges`
+- `source = table | fields country, bridges | flatten bridges | fields country, length | stats avg(length) as avg by country`
+
 ```sql
 source = table | eval e = eval status_category =
         case(a >= 200 AND a < 300, 'Success',
@@ -289,7 +306,11 @@ source = table |  where ispresent(a) |
 - `source = table1 | left semi join left = l right = r on l.a = r.a table2`
 - `source = table1 | left anti join left = l right = r on l.a = r.a table2`
 - `source = table1 | join left = l right = r [ source = table2 | where d > 10 | head 5 ]`
-
+- `source = table1 | inner join on table1.a = table2.a table2 | fields table1.a, table2.a, table1.b, table1.c` (directly refer table name)
+- `source = table1 | inner join on a = c table2 | fields a, b, c, d` (ignore side aliases as long as no ambiguous)
+- `source = table1 as t1 | join left = l right = r on l.a = r.a table2 as t2 | fields l.a, r.a` (side alias overrides table alias)
+- `source = table1 as t1 | join left = l right = r on l.a = r.a table2 as t2 | fields t1.a, t2.a` (error, side alias overrides table alias)
+- `source = table1 | join left = l right = r on l.a = r.a [ source = table2 ] as s | fields l.a, s.a` (error, side alias overrides subquery alias)
 
 #### **Lookup**
 [See additional command details](ppl-lookup-command.md)

docs/ppl-lang/README.md

@@ -31,6 +31,8 @@ For additional examples see the next [documentation](PPL-Example-Commands.md).
     - [`describe command`](PPL-Example-Commands.md/#describe)
 
     - [`fillnull command`](ppl-fillnull-command.md)
+
+    - [`flatten command`](ppl-flatten-command.md)
 
     - [`eval command`](ppl-eval-command.md)
 
@@ -67,7 +69,8 @@ For additional examples see the next [documentation](PPL-Example-Commands.md).
     - [`subquery commands`](ppl-subquery-command.md)
 
     - [`correlation commands`](ppl-correlation-command.md)
-
+
+    - [`trendline commands`](ppl-trendline-command.md)
 
 * **Functions**
 
@@ -88,6 +91,8 @@ For additional examples see the next [documentation](PPL-Example-Commands.md).
     - [`Cryptographic Functions`](functions/ppl-cryptographic.md)
 
     - [`IP Address Functions`](functions/ppl-ip.md)
+
+    - [`Lambda Functions`](functions/ppl-lambda.md)
 
 ---
 ### PPL On Spark
@@ -106,4 +111,4 @@ See samples of [PPL queries](PPL-Example-Commands.md)
 
 ---
 ### PPL Project Roadmap
-[PPL Github Project Roadmap](https://github.com/orgs/opensearch-project/projects/214)
+[PPL Github Project Roadmap](https://github.com/orgs/opensearch-project/projects/214)

docs/ppl-lang/functions/ppl-json.md

@@ -4,11 +4,11 @@
 
 **Description**
 
-`json(value)` Evaluates whether a value can be parsed as JSON. Returns the json string if valid, null otherwise.
+`json(value)` Evaluates whether a string can be parsed as JSON format. Returns the string value if valid, null otherwise.
 
-**Argument type:** STRING/JSON_ARRAY/JSON_OBJECT
+**Argument type:** STRING
 
-**Return type:** STRING
+**Return type:** STRING/NULL
 
 A STRING expression of a valid JSON object format.
 
@@ -47,15 +47,15 @@ A StructType expression of a valid JSON object.
 
 Example:
 
-    os> source=people | eval result = json(json_object('key', 123.45)) | fields result
+    os> source=people | eval result = json_object('key', 123.45) | fields result
     fetched rows / total rows = 1/1
     +------------------+
     | result           |
     +------------------+
     | {"key":123.45}   |
     +------------------+
 
-    os> source=people | eval result = json(json_object('outer', json_object('inner', 123.45))) | fields result
+    os> source=people | eval result = json_object('outer', json_object('inner', 123.45)) | fields result
     fetched rows / total rows = 1/1
     +------------------------------+
     | result                       |
@@ -81,29 +81,58 @@ Example:
 
     os> source=people | eval `json_array` = json_array(1, 2, 0, -1, 1.1, -0.11)
     fetched rows / total rows = 1/1
-    +----------------------------+
-    | json_array                 |
-    +----------------------------+
-    | 1.0,2.0,0.0,-1.0,1.1,-0.11 |
-    +----------------------------+
+    +------------------------------+
+    | json_array                   |
+    +------------------------------+
+    | [1.0,2.0,0.0,-1.0,1.1,-0.11] |
+    +------------------------------+
 
-    os> source=people | eval `json_array_object` = json(json_object("array", json_array(1, 2, 0, -1, 1.1, -0.11)))
+    os> source=people | eval `json_array_object` = json_object("array", json_array(1, 2, 0, -1, 1.1, -0.11))
     fetched rows / total rows = 1/1
     +----------------------------------------+
     | json_array_object                      |
     +----------------------------------------+
     | {"array":[1.0,2.0,0.0,-1.0,1.1,-0.11]} |
     +----------------------------------------+
 
+### `TO_JSON_STRING`
+
+**Description**
+
+`to_json_string(jsonObject)` Returns a JSON string with a given json object value.
+
+**Argument type:** JSON_OBJECT (Spark StructType/ArrayType)
+
+**Return type:** STRING
+
+Example:
+
+    os> source=people | eval `json_string` = to_json_string(json_array(1, 2, 0, -1, 1.1, -0.11)) | fields json_string
+    fetched rows / total rows = 1/1
+    +--------------------------------+
+    | json_string                    |
+    +--------------------------------+
+    | [1.0,2.0,0.0,-1.0,1.1,-0.11]   |
+    +--------------------------------+
+
+    os> source=people | eval `json_string` = to_json_string(json_object('key', 123.45)) | fields json_string
+    fetched rows / total rows = 1/1
+    +-----------------+
+    | json_string     |
+    +-----------------+
+    | {'key', 123.45} |
+    +-----------------+
+
+
 ### `JSON_ARRAY_LENGTH`
 
 **Description**
 
-`json_array_length(jsonArray)` Returns the number of elements in the outermost JSON array.
+`json_array_length(jsonArrayString)` Returns the number of elements in the outermost JSON array string.
 
-**Argument type:** STRING/JSON_ARRAY
+**Argument type:** STRING
 
-A STRING expression of a valid JSON array format, or JSON_ARRAY object.
+A STRING expression of a valid JSON array format.
 
 **Return type:** INTEGER
 
@@ -119,6 +148,21 @@ Example:
     | 4         | 5         | null        |
     +-----------+-----------+-------------+
 
+
+### `ARRAY_LENGTH`
+
+**Description**
+
+`array_length(jsonArray)` Returns the number of elements in the outermost array.
+
+**Argument type:** ARRAY
+
+ARRAY or JSON_ARRAY object.
+
+**Return type:** INTEGER
+
+Example:
+
     os> source=people | eval `json_array` = json_array_length(json_array(1,2,3,4)), `empty_array` = json_array_length(json_array())
     fetched rows / total rows = 1/1
     +--------------+---------------+
@@ -127,6 +171,7 @@ Example:
     | 4            | 0             |
     +--------------+---------------+
 
+
 ### `JSON_EXTRACT`
 
 **Description**