Updated/Corrected Docs (#270)

* Updated/Corrected Docs - README.md - CLIENT_GENERATOR_GUIDE.md - DEVELOPER_GUIDE.md - ./tools/README.md Signed-off-by: Theo Truong <[email protected]> * # minor corrections Signed-off-by: Theo Truong <[email protected]> * # minor corrections Signed-off-by: Theo Truong <[email protected]> --------- Signed-off-by: Theo Truong <[email protected]>
opensearch-project · Apr 25, 2024 · b405750 · b405750
1 parent 21cdfe9
commit b405750
Show file tree

Hide file tree

Showing 4 changed files with 74 additions and 42 deletions.
diff --git a/CLIENT_GENERATOR_GUIDE.md b/CLIENT_GENERATOR_GUIDE.md
@@ -19,13 +19,13 @@ In a client, the `search` operations are grouped in to a single API method, `cli
 
 In the [published OpenAPI spec](https://github.com/opensearch-project/opensearch-api-specification/releases), this grouping is denoted by `x-operation-group` vendor extension in every operation definition. The value of this extension is the name of the API action (like `search` or `indices.get_field_mapping`). Operations with the same `x-operation-group` value are guaranteed to have the same query string parameters, response body, and request body (for PUT/POST/DELETE operations). Common path parameters are also guaranteed to be the same. The only differences between operations are the HTTP method and the path. With that in mind, below are rules on how to combine operations of different HTTP methods and path compositions.
 
-- If an operation is marked with `x-ignorable: "true"`, then ignore the operation. Such an operation has been deprecated and has been replaced by a newer one. As far as the clients are concerned, ignorable operations do not exist.
+- If an operation is marked with `x-ignorable: true`, then ignore the operation. Such an operation has been deprecated and has been superseded by a newer one. As far as the clients are concerned, ignorable operations do not exist.
 - If two operations have identical HTTP methods, but different paths: use the path that best matches the path parameters provided.
 - If two operations have identical path, but different HTTP methods:
     - GET/POST: if the request body is provided then use POST, otherwise use GET
     - PUT/POST: Either works, but PUT is preferred when an optional path parameter is provided.
 
-The psuedo-code that combines the `search` operations into a single API method is as follows:
+The pseudocode that combines the `search` operations into a single API method is as follows:
 ```python
 def search(self, index=None, body=None):
     if index is None:
@@ -41,20 +41,19 @@ def search(self, index=None, body=None):
     return self.perform_request(method, path, body=body)
 ```
 
-
 ## Overloaded Name
-You will also encounter `x-overloaded-param: "metric"` for the `node_id` path parameter of `GET /_nodes/{node_id}` operation in `nodes.info` action. This is a special case where the path parameter is overloaded to accept either a node ID or a metric name. The `client.nodes.info` method when called with either `metric` or `node_id` (but not both), will use `GET /_nodes/{node_id}` operation (even though the path parameter name is `node_id`). When called with both `metric` and `node_id`, it will use `GET /_nodes/{node_id}/{metric}` operation.
+You will also encounter `x-overloaded-param: metric` for the `node_id` path parameter of the `GET /_nodes/{node_id}` operation in `nodes.info` action. This is a special case where the path parameter is overloaded to accept either a node ID or a metric name. When the user evokes the `client.nodes.info` method with either `metric` or `node_id` (but not both), the method will use the `GET /_nodes/{node_id}` operation. When evoked with both `metric` and `node_id`, it will use the `GET /_nodes/{node_id}/{metric}` operation.
 
 ## Handling Bulk Operations
-Some operations accept a bulk of data in the request body. For example, the `bulk` action accepts a bulk of index, update, and delete operations on multiple documents. Unlike other operations where the request body is a JSON object, the request body for bulk operations is a newline-seperated JSON string. The client will automatically convert the request body into a newline-seperated JSON objects. The request body of such operations will be denoted with `x-serialize: "bulk"` vendor extension.
+Some operations accept a bulk of data in the request body. For example, the `bulk` action accepts a bulk of index, update, and delete operations on multiple documents. Unlike other operations where the request body is a **JSON object**, the request body for bulk operations is an **NDJSON** (i.e a [Newline-delimited JSON](https://github.com/ndjson/ndjson-spec)). When encountering this type of operation, the client must serialize the request body accordingly, and set the `Content-Type` header to `application/x-ndjson`.
 
 ## Parameter Validation
 As of right now, most clients only validate whether required parameters are present. The clients do not validate the values of parameters against the enum values or regex patterns. This is to reduce performance overhead for the clients as the validation is already done on the server. However, the list of enum values and regex patterns are often written into the parameter description.
 
-Some clients also check for the validity of query string parameter names to guard the users from typos. If you decide to implement this feature, make sure that it's performant. Scripting languages like Python and Ruby require the code to be loaded into memory at runtime, and constructs used for this feature can be expensive to load, as far as micro-services are concerned.
+Some clients also check for the validity of query string parameter names to guard the users from typos. If you decide to implement this feature, make sure that it's performant. Scripting languages like Python and Ruby require the code to be loaded into memory at runtime, and constructs used for this feature can be expensive to load, as far as microservices are concerned.
 
 ## Global Parameters
-All operations in the spec contain a set of parameters that are common across all operations. These parameters are denoted with `x-global: true` vendor extension. The generated clients should find a way to DRY these parameters.
+All operations in the spec contain a set of parameters that are common across all operations. These parameters are denoted with `x-global: true` vendor extension. The generated clients should find a way to DRY these parameters in type definitions and method documentation.
 
 ## Default Parameter Values
-Parameters can have default values either through schema or the `x-default` vendor extension. When both are present, `x-default` will takes precedence.
+Parameters can have default values either through schema or the `x-default` vendor extension. When both are present, `x-default` will take precedence.
diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md
@@ -3,8 +3,10 @@
   - [File Structure](#file-structure)
   - [Grouping Operations](#grouping-operations)
   - [Grouping Schemas](#grouping-schemas)
+  - [Superseded Operations](#superseded-operations)
+  - [Global Parameters](#global-parameters)
   - [OpenAPI Extensions](#openapi-extensions)
-  - [Linting](#linting)
+  - [Tools](#tools)
 
 # Developer Guide
 
@@ -20,10 +22,6 @@ The Specification is written in OpenAPI 3, so understanding the OpenAPI 3 specif
 
 To make editing the specification easier, we split the OpenAPI spec into multiple files that can be found in the [spec](spec) directory. The file structure is as follows:
 
-- The API Operations are grouped by namespaces in [spec/namespaces](spec/namespaces/) directory. Each `.yaml` file in this directory represents a namespace and holds all paths and operations of the namespace.
-- The data schemas are grouped by categories in [spec/schemas](spec/schemas/) directory. Each `.yaml` file in this directory represents a category.
-- The [spec/opensearch-openapi.yaml](spec/opensearch-openapi.yaml) file is the OpenAPI root file that ties everything together.
-
 ```
 spec
 │
@@ -46,7 +44,11 @@ spec
 └── opensearch-openapi.yaml
 ```
 
-Every `.yaml` file is a valid OpenAPI 3 document. This means that you can use any OpenAPI 3 compatible tool to view and edit the files, and IDEs with OpenAPI support will provide you with autocompletion and validation in real-time.
+- The API Operations are grouped by namespaces in [spec/namespaces/](spec/namespaces) directory. Each file in this directory represents a namespace and holds all paths and operations of the namespace.
+- The data schemas are grouped by categories in [spec/schemas/](spec/schemas) directory. Each file in this directory represents a category.
+- The [spec/opensearch-openapi.yaml](spec/opensearch-openapi.yaml) file is the OpenAPI root file that ties everything together.
+
+Every `.yaml` file is a OpenAPI 3 document. This means that you can use any OpenAPI 3 compatible tool to view and edit the files, and IDEs with OpenAPI support will also offer autocomplete and validation in realtime.
 
 ## Grouping Operations
 
@@ -65,16 +67,44 @@ For this reason, every operation *must* be accompanied by the `x-operation-group
 
 ## Grouping Schemas
 
-Schemas are grouped by categories to keep their names short and aid in client generation:
+Schemas are grouped by categories to keep their names short, and aid in client generation (where the schemas are translated into data types/classes, and divided into packages/modules). The schema file names can be in one of the following formats:
 
 - `_common` category holds the common schemas that are used across multiple namespaces and features.
 - `_common.<sub_category>` category holds the common schemas of a specific sub_category. (e.g. `_common.mapping`)
 - `<namespace>._common` category holds the common schemas of a specific namespace. (e.g. `cat._common`, `_core._common`)
 - `<namespace>.<action>` category holds the schemas of a specific sub_category of a namespace. (e.g. `cat.aliases`, `_core.search`)
 
+## Superseded Operations
+
+When an operation is superseded by another operation with **identical functionality**, that is a rename or a change in the URL, it should be listed in [_superseded_operations.yaml](./spec/_superseded_operations.yaml) file. The merger tool will automatically generate the superseded operation in the OpenAPI spec. The superseded operation will have `deprecated: true` and `x-ignorable: true` properties to indicate that it should be ignored by the client generator.
+
+For example, if the `_superseded_operations.yaml` file contains the following entry:
+```yaml
+/_opendistro/_anomaly_detection/{nodeId}/stats/{stat}:
+  superseded_by: /_plugins/_anomaly_detection/{nodeId}/stats/{stat}
+  operations:
+    - GET
+    - POST
+```
+Then, the merger tool will generate 2 superseded operations: 
+- `GET /_opendistro/_anomaly_detection/{nodeId}/stats/{stat}` 
+- `POST /_opendistro/_anomaly_detection/{nodeId}/stats/{stat}`
+
+from their respective superseding operations:
+
+- `GET /_plugins/_anomaly_detection/{nodeId}/stats/{stat}`
+- `POST /_plugins/_anomaly_detection/{nodeId}/stats/{stat}`
+
+if and only if the superseding operations exist in the spec. A warning will be printed on the console if they do not. 
+
+Note that the path parameter names do not need to match. So, if the actual superseding operations have path of `/_plugins/_anomaly_detection/{node_id}/stats/{stat_id}`, the merger tool will recognize that it is the same as `/_plugins/_anomaly_detection/{nodeId}/stats/{stat}` and generate the superseded operations accordingly with the correct path parameter names.
+
+## Global Parameters
+Certain query parameters are global, and they are accepted by every operation. These parameters are listed in the [root file](spec/opensearch-openapi.yaml) under the `parameters` section with `x-global` set to true. The merger tool will automatically add these parameters to all operations.
+
 ## OpenAPI Extensions
 
-This repository includes several penAPI Specification Extensions to fill in any metadata not directly supported OpenAPI:
+This repository includes several OpenAPI Specification Extensions to fill in any metadata not natively supported by OpenAPI:
 
 - `x-operation-group`: Used to group operations into API actions.
 - `x-version-added`: OpenSearch version when the operation/parameter was added.
@@ -87,28 +117,12 @@ This repository includes several penAPI Specification Extensions to fill in any
 
 ## Tools
 
-We authored a number of tools to merge and lint specs that live in [tools](tools/). All tools have tests (run with `npm run test`) and a linter (run with `npm run lint`).
+We authored a number of tools to merge and lint specs that live in [tools](tools). All tools have tests (run with `npm run test`) and a linter (run with `npm run lint`).
 
 ### Merger
 
-The spec merger "builds", aka combines various `.yaml` files into a complete OpenAPI spec. A [workflow](./.github/workflows/build.yml) publishes the output into [releases](https://github.com/opensearch-project/opensearch-api-specification/releases).
-
-#### Auto-generating Superseded Operations
-
-When an operation is superseded by another operation with **IDENTICAL FUNCTIONALITY**, that is a rename or a change in the URL, it should be listed in [_superseded_operations.yaml](./spec/_superseded_operations.yaml) file. The merger tool will automatically generate the superseded operation in the OpenAPI spec. The superseded operation will have `deprecated` and `x-ignorable` properties set to `true` to indicate that it should be ignored by the client generator.
-For example, if the `_superseded_operations.yaml` file contains the following entry:
-```yaml
-/_opendistro/_anomaly_detection/{nodeId}/stats/{stat}:
-  superseded_by: /_plugins/_anomaly_detection/{nodeId}/stats/{stat}
-  operations:
-    - GET
-    - POST
-```
-Then, the merger tool will generate 2 operations: `GET /_opendistro/_anomaly_detection/{nodeId}/stats/{stat}` and `POST /_opendistro/_anomaly_detection/{nodeId}/stats/{stat}` from `GET /_plugins/_anomaly_detection/{nodeId}/stats/{stat}` and `POST /_plugins/_anomaly_detection/{nodeId}/stats/{stat}` respectively, if they exist (A warning will be printed on the console if they do not). Note that the path parameter names do not need to match. So, if the actual superseding operations have path of `/_plugins/_anomaly_detection/{node_id}/stats/{stat_id}`, the merger tool will recognize that it is the same as `/_plugins/_anomaly_detection/{nodeId}/stats/{stat}` and generate the superseded operations accordingly with the correct path parameter names.
-
-#### Auto-generating global parameters
-Certain query parameters are global, and they are accepted by every operation. These parameters are listed in the [root file](spec/opensearch-openapi.yaml) under the `parameters` section with `x-global` set to true. The merger tool will automatically add these parameters to all operations.
+The spec merger "builds", aka combines all `.yaml` files in a spec folder into a complete OpenAPI spec. A [workflow](./.github/workflows/build.yml) performs this task on the [spec folder](spec) of this repo then publishes the output into [releases](https://github.com/opensearch-project/opensearch-api-specification/releases).
 
 ### Linter
 
-The spec linter that validates every `.yaml` file in the `./spec` folder to assure that they follow the guidelines we have set. Check out the [Linter README](tools/README.md#linter) for more information on how to run it locally. Make sure to run the linter before submitting a PR.
+The spec linter that validates every `.yaml` file in the `./spec` folder to assure that they follow the guidelines we have set. Check out the [Linter README](tools/README.md#spec-linter) for more information on how to run it locally. Make sure to run the linter before submitting a PR.
diff --git a/README.md b/README.md
@@ -7,7 +7,8 @@
 - [Project Resources](#project-resources)
 - [Code of Conduct](#code-of-conduct)
 - [Developer Guide](#developer-guide)
-- [OpenSearch API Specs](#opensearch-api-specs)
+- [Client Generator Guide](#client-generator-guide)
+- [Published Spec](#published-spec)
 - [Security](#security)
 - [License](#license)
 - [Copyright](#copyright)
@@ -38,10 +39,17 @@ This project has adopted the [Amazon Open Source Code of Conduct](CODE_OF_CONDUC
 
 See [DEVELOPER_GUIDE](DEVELOPER_GUIDE.md).
 
-## OpenSearch API Specs
+## Client Generator Guide
+
+See [CLIENT_GENERATOR_GUIDE](CLIENT_GENERATOR_GUIDE.md).
+
+## Published Spec
 
 OpenSearch API Specs are hosted at https://opensearch-project.github.io/opensearch-api-specification/. See [PUBLISHING_SPECS](PUBLISHING_SPECS.md) for more information.
 
+Click [here](https://github.com/opensearch-project/opensearch-api-specification/releases/download/main/opensearch-openapi.yaml) to download the latest OpenSearch OpenAPI yaml file.
+
+
 ## Security
 
 If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/) or directly via email to [email protected]. Please do **not** create a public GitHub issue.

diff --git a/tools/README.md b/tools/README.md
@@ -2,8 +2,8 @@
 
 This folder contains tools for the repo:
 
-- [Merger](./merger/): merges multiple OpenAPI files into one
-- [Linter](./linter/): validates files in the spec folder
+- [Merger](./merger): merges multiple OpenAPI files into one
+- [Linter](./linter): validates files in the spec folder
 
 ## Setup
 
@@ -20,15 +20,26 @@ The merger tool merges the multi-file OpenSearch spec into a single file for pro
 Example:
 
 ```bash
-npm run merge -- ../spec ../build/opensearch-openapi.latest.yaml
+mkdir -p ../build
+export ROOT_PATH=../spec/opensearch-openapi.yaml
+export OUTPUT_PATH=../build/opensearch-openapi.yaml
+npm run merge -- $ROOT_PATH $OUTPUT_PATH
 ```
 
-## Linter
+As a shortcut, if those parameters are not provided, the tool will use the default values:
+- `../spec/opensearch-openapi.yaml` as the root path (i.e. the root file of the repo's [spec folder](../spec))
+- `../opensearch-openapi.yaml` as the output path
+
+```bash
+npm run merge
+```
+
+## Spec Linter
 
 The linter tool validates the OpenSearch spec files in the `spec` folder:
 
 ```bash
 npm run lint:spec
 ```
 
-It will print out all the errors and warnings in the spec files. This tool in still in development, and it will be integrated into the CI/CD pipeline and run automatically with every PR.
+It will print out all the errors and warnings in the spec files.