Merge branch 'opensearch-project:main' into search-pipelines

.github/workflows/coverage.yml

@@ -0,0 +1,55 @@
+name: API Coverage
+
+on: [push, pull_request_target]
+
+env:
+  JAVA_VERSION: 11
+  OPENSEARCH_INITIAL_ADMIN_PASSWORD: BobgG7YrtsdKf9M
+
+jobs:
+  coverage:
+    permissions:
+      pull-requests: write
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Repo
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+      - name: Build and Run Docker Container
+        run: |
+          docker build coverage --tag opensearch-with-api-plugin
+          docker run -d -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e OPENSEARCH_INITIAL_ADMIN_PASSWORD="$OPENSEARCH_INITIAL_ADMIN_PASSWORD" opensearch-with-api-plugin
+          sleep 15
+      - name: Display OpenSearch Info
+        run: |
+          curl   -ks -u "admin:$OPENSEARCH_INITIAL_ADMIN_PASSWORD" https://localhost:9200/ | jq
+      - name: Dump and Compare API
+        run: |
+          curl -ks -u "admin:$OPENSEARCH_INITIAL_ADMIN_PASSWORD" https://localhost:9200/_plugins/api | jq > OpenSearch.auto.openapi.json
+          docker run --rm --mount type=bind,source=.,target=/specs openapitools/openapi-diff:latest /specs/OpenSearch.openapi.json /specs/OpenSearch.auto.openapi.json --json /specs/diff.json
+      - name: Show Diff
+        run: |
+          echo "-------- Missing APIs"
+          jq -r '.newEndpoints | group_by(.pathUrl)[] | "\(.[0].pathUrl): \([.[].method])"' diff.json
+          echo "-------- Legacy APIs"
+          jq -r '.missingEndpoints | group_by(.pathUrl)[] | "\(.[0].pathUrl): \([.[].method])"' diff.json
+      - name: Gather Coverage
+        id: coverage
+        shell: bash
+        run: |
+          current=`jq -r '.paths | keys | length' OpenSearch.openapi.json`
+          total=`jq -r '.paths | keys | length' OpenSearch.auto.openapi.json`
+          percent=$((current * 100 / total))
+          echo "API specs implemented for $current/$total ($percent%) APIs."
+          cat >>"$GITHUB_OUTPUT" <<EOL
+          current=$current
+          total=$total
+          percent=$percent
+          EOL
+      - uses: peter-evans/create-or-update-comment@v4
+        if: github.event_name == 'pull_request_target'
+        with:
+          issue-number: ${{ github.event.number }}
+          body: |
+            API specs implemented for ${{ steps.coverage.outputs.current }}/${{ steps.coverage.outputs.total }} (${{ steps.coverage.outputs.percent }}%) APIs.

CLIENT_GENERATOR_GUIDE.md

@@ -23,21 +23,22 @@ In the [OpenAPI spec](./OpenSearch.openapi.json), this grouping is denoted by `x
 - If two operations have identical HTTP methods, but different paths: use the path that best matches the path parameters provided.
 - If two operations have identical path, but different HTTP methods:
     - GET/POST: if the request body is provided then use POST, otherwise use GET
-    - PUT/POST: Either works, but PUT is preferred.
+    - PUT/POST: Either works, but PUT is preferred when an optional path parameter is provided.
 
 The psuedo-code that combines the `search` operations into a single API method is as follows:
 ```python
 def search(self, index=None, body=None):
     if index is None:
-        if body is None:
-            return self.perform_request("GET", "/_search")
-        else:
-            return self.perform_request("POST", "/_search", body=body)
+        path = "/_search"
     else:
-        if body is None:
-            return self.perform_request("GET", f"/{index}/_search")
-        else:
-            return self.perform_request("POST", f"/{index}/_search", body=body)
+        path = f"/{index}/_search"
+
+    if body is None:
+        method = "GET"
+    else:
+        method = "POST"
+
+    return self.perform_request(method, path, body=body)
 ```
 
 ## Handling Path Parameters

DEVELOPER_GUIDE.md

@@ -54,7 +54,7 @@ Popular IDEs for Smithy models include Visual Studio Code and IntelliJ, and they
 - [OpenAPI Specifications](https://plugins.jetbrains.com/plugin/14394-openapi-specifications)
 
 ## File Structure
-The OpenSearch API is composed of over 300 operations. These operations are grouped into API actions based on the functionality they provide. Each API action is later translated to an API method in each OpenSearch client. For example:
+The OpenSearch API is composed of over 300 operations. These operations are grouped into API actions based on the functionality they provide (This grouping is done through the `@xOperationGroup` Smithy trait). Each API action is later translated to an API method in each OpenSearch client. For example:
 - The `cat.health` action will turn into `client.cat.health()`
 - While the `index` action will turn into `client.index()`
 
@@ -95,6 +95,9 @@ As mentioned in the previous section, each API action is composed of multiple op
 - `GET /{index}/_search`
 - `POST /{index}/_search`
 
+To group these operations together in the `search` action, we mark them with the `@xOperationGroup` trait with the same value of `search`. Note that this trait tells the client generators that these operations serve identical purpose and should be grouped together in the same API method. The `xOperationGroup` trait also tells the generators the namespace and the name of the API method. For example, operations with `xOperationGroup` trait value of `indicies.create` will result in `client.indices.create()` method to be generated.
+
+
 ### Defining operations
 
 We name each operation using the following format `[ActionName]_[HttpVerb]_[PathParameters]`
@@ -194,16 +197,17 @@ structure Operation_Output {
 
 ## Defining Common Parameters
 
-Common parameters are defined in the root folder, `model/`, and grouped by data-type in files of `common_<data_type>.smithy` format. There are a few things to note when defining common parameters:
+Common parameters that are used across OpenSearch namespaces are defined in the root folder, `model/`, and grouped by data-type in files of `common_<data_type>.smithy` format. There are a few things to note when defining global common parameters:
 - All path parameters should be prefixed with `Path` like `PathIndex` and `PathDocumentID`.
 - Smithy doesn't support _enum_ or _list_ as path parameters. We, therefore, have to define such parameters as _string_ and use `x-data-type` vendor extension to denote their actual types (More on this in the traits section).
 - Parameters of type `time` are defined as `string` and has `@pattern("^([0-9]+)(?:d|h|m|s|ms|micros|nanos)$")` trait to denote that they are in the format of `<number><unit>`. E.g. `1d`, `2h`, `3m`, `4s`, `5ms`, `6micros`, `7nanos`. We use `x-data-type: "time"` vendor extension for this type.
 - Path parameters that are defined as strings must be accompanied by a `@pattern` trait and should be default to `^[^_][\\d\\w-*]*$` to signify that they are not allowed to start with `_` to avoid URI Conflict errors.
 - The `@documentation`, `@default`, and `@deprecation` traits can later be overridden by the operations that use these parameters.
 
+Common parameters that are used within a namespace, especially namespaces representing plugins like `security` are defined in the namespace folder (e.g. `model/security`)
+
 ## Smithy Traits
 We use Smithy traits extensively in this project to work around some limitations of Smithy and to deal with some quirks of the OpenSearch API. Here are some of the traits that you should be aware of:
-- `@vendorExtensions`: Used to add metadata that are not supported by Smithy. Check OpenAPI Vendor Extensions section for more details.
 - `@suppress(["HttpMethodSemantics.UnexpectedPayload"])`: Used in DELETE operations with request body to suppress the UnexpectedPayload error.
 - `@suppress(["HttpUriConflict"])`: Used to suppress the HttpUriConflict error that is thrown when two operations have conflicting URI. Unfortunately, this happens a lot in OpenSearch API. When in doubt, add this trait to the operation.
 - `@pattern("^(?!_|template|query|field|point|clear|usage|stats|hot|reload|painless)")`: Required for most Path parameters to avoid URI Conflict errors. This is often used in tandem with the @suppress trait above. To denote the actual pattern of the parameter, use `x-string-pattern` vendor extension.