diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index ceb549a69..a325840b7 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -25,7 +25,7 @@ jobs: run: |- mkdir -p ../build npm install - export ROOT_PATH=../spec/opensearch-openapi.yaml + export ROOT_PATH=../spec export OUTPUT_PATH=../build/opensearch-openapi.yaml npm run merge -- $ROOT_PATH $OUTPUT_PATH diff --git a/.github/workflows/coverage-gather.yml b/.github/workflows/coverage-gather.yml index 703c0c6ab..d3b378b90 100644 --- a/.github/workflows/coverage-gather.yml +++ b/.github/workflows/coverage-gather.yml @@ -19,7 +19,7 @@ jobs: run: |- mkdir -p ../build npm install - export ROOT_PATH=../spec/opensearch-openapi.yaml + export ROOT_PATH=../spec export OUTPUT_PATH=../build/opensearch-openapi.yaml npm run merge -- $ROOT_PATH $OUTPUT_PATH - name: Build and Run Docker Container diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 37274d43e..145d9c2e1 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -41,14 +41,15 @@ spec │ ├── cat.aliases.yaml │ └── ... │ -└── opensearch-openapi.yaml +├── _info.yaml +├── _global_parameters.yaml +└── _superseded_operations.yaml ``` - The API Operations are grouped by namespaces in [spec/namespaces/](spec/namespaces) directory. Each file in this directory represents a namespace and holds all paths and operations of the namespace. - The data schemas are grouped by categories in [spec/schemas/](spec/schemas) directory. Each file in this directory represents a category. -- The [spec/opensearch-openapi.yaml](spec/opensearch-openapi.yaml) file is the OpenAPI root file that ties everything together. -Every `.yaml` file is a OpenAPI 3 document. This means that you can use any OpenAPI 3 compatible tool to view and edit the files, and IDEs with OpenAPI support will also offer autocomplete and validation in realtime. +Every `.yaml` file in the namespaces and schemas folders is a OpenAPI 3 document. This means that you can use any OpenAPI 3 compatible tool to view and edit the files, and IDEs with OpenAPI support will also offer autocomplete and validation in realtime. ## Grouping Operations @@ -100,7 +101,7 @@ if and only if the superseding operations exist in the spec. A warning will be p Note that the path parameter names do not need to match. So, if the actual superseding operations have path of `/_plugins/_anomaly_detection/{node_id}/stats/{stat_id}`, the merger tool will recognize that it is the same as `/_plugins/_anomaly_detection/{nodeId}/stats/{stat}` and generate the superseded operations accordingly with the correct path parameter names. ## Global Parameters -Certain query parameters are global, and they are accepted by every operation. These parameters are listed in the [root file](spec/opensearch-openapi.yaml) under the `parameters` section with `x-global` set to true. The merger tool will automatically add these parameters to all operations. +Certain query parameters are global, and they are accepted by every operation. These parameters are listed in the [spec/_global_parameters.yaml](spec/_global_parameters.yaml). The merger tool will automatically add these parameters to all operations. ## OpenAPI Extensions @@ -112,7 +113,7 @@ This repository includes several OpenAPI Specification Extensions to fill in any - `x-version-removed`: OpenSearch version when the operation/parameter was removed. - `x-deprecation-message`: Reason for deprecation and guidance on how to prepare for the next major version. - `x-ignorable`: Denotes that the operation should be ignored by the client generator. This is used in operation groups where some operations have been replaced by newer ones, but we still keep them in the specs because the server still supports them. -- `x-global`: Denotes that the parameter is a global parameter that is included in every operation. These parameters are listed in the [root file](spec/opensearch-openapi.yaml). +- `x-global`: Denotes that the parameter is a global parameter that is included in every operation. These parameters are listed in the [spec/_global_parameters.yaml](spec/_global_parameters.yaml). - `x-default`: Contains the default value of a parameter. This is often used to override the default value specified in the schema, or to avoid accidentally changing the default value when updating a shared schema. ## Tools diff --git a/_plugins/openapi.rb b/_plugins/openapi.rb index 82bae554f..23a612b27 100644 --- a/_plugins/openapi.rb +++ b/_plugins/openapi.rb @@ -4,7 +4,7 @@ def self.generate(_site, _payload) Dir.chdir('tools') do system 'npm install' - system 'npm run merge -- ../spec/opensearch-openapi.yaml ../_site/opensearch-openapi.yaml' + system 'npm run merge -- ../spec ../_site/opensearch-openapi.yaml' end @generated = true diff --git a/spec/_global_parameters.yaml b/spec/_global_parameters.yaml new file mode 100644 index 000000000..9a2cd00e9 --- /dev/null +++ b/spec/_global_parameters.yaml @@ -0,0 +1,43 @@ +openapi: 3.1.0 +info: + title: '' + version: '' +components: + parameters: + _global::query.pretty: + name: pretty + in: query + description: Whether to pretty format the returned JSON response. + schema: + type: boolean + default: false + _global::query.human: + name: human + in: query + description: Whether to return human readable values for statistics. + schema: + type: boolean + default: true + _global::query.error_trace: + name: error_trace + in: query + description: Whether to include the stack trace of returned errors. + schema: + type: boolean + default: false + _global::query.source: + name: source + in: query + description: The URL-encoded request definition. Useful for libraries that do not accept a request body for non-POST requests. + schema: + type: string + _global::query.filter_path: + name: filter_path + in: query + description: Comma-separated list of filters used to reduce the response. + schema: + oneOf: + - type: string + - type: array + items: + type: string \ No newline at end of file diff --git a/spec/_info.yaml b/spec/_info.yaml new file mode 100644 index 000000000..9da26bfa0 --- /dev/null +++ b/spec/_info.yaml @@ -0,0 +1,3 @@ +title: OpenSearch API +description: OpenSearch API +version: 1.0.0 \ No newline at end of file diff --git a/spec/opensearch-openapi.yaml b/spec/opensearch-openapi.yaml deleted file mode 100644 index 1673d43af..000000000 --- a/spec/opensearch-openapi.yaml +++ /dev/null @@ -1,548 +0,0 @@ -openapi: 3.1.0 -info: - title: OpenSearch API - description: OpenSearch API - version: 1.0.0 -paths: - /: - $ref: 'namespaces/_core.yaml#/paths/~1' - /_bulk: - $ref: 'namespaces/_core.yaml#/paths/~1_bulk' - /_count: - $ref: 'namespaces/_core.yaml#/paths/~1_count' - /_delete_by_query/{task_id}/_rethrottle: - $ref: 'namespaces/_core.yaml#/paths/~1_delete_by_query~1{task_id}~1_rethrottle' - /_field_caps: - $ref: 'namespaces/_core.yaml#/paths/~1_field_caps' - /_mget: - $ref: 'namespaces/_core.yaml#/paths/~1_mget' - /_msearch: - $ref: 'namespaces/_core.yaml#/paths/~1_msearch' - /_msearch/template: - $ref: 'namespaces/_core.yaml#/paths/~1_msearch~1template' - /_mtermvectors: - $ref: 'namespaces/_core.yaml#/paths/~1_mtermvectors' - /_rank_eval: - $ref: 'namespaces/_core.yaml#/paths/~1_rank_eval' - /_reindex: - $ref: 'namespaces/_core.yaml#/paths/~1_reindex' - /_reindex/{task_id}/_rethrottle: - $ref: 'namespaces/_core.yaml#/paths/~1_reindex~1{task_id}~1_rethrottle' - /_render/template: - $ref: 'namespaces/_core.yaml#/paths/~1_render~1template' - /_render/template/{id}: - $ref: 'namespaces/_core.yaml#/paths/~1_render~1template~1{id}' - /_script_context: - $ref: 'namespaces/_core.yaml#/paths/~1_script_context' - /_script_language: - $ref: 'namespaces/_core.yaml#/paths/~1_script_language' - /_scripts/painless/_execute: - $ref: 'namespaces/_core.yaml#/paths/~1_scripts~1painless~1_execute' - /_scripts/{id}: - $ref: 'namespaces/_core.yaml#/paths/~1_scripts~1{id}' - /_scripts/{id}/{context}: - $ref: 'namespaces/_core.yaml#/paths/~1_scripts~1{id}~1{context}' - /_search: - $ref: 'namespaces/_core.yaml#/paths/~1_search' - /_search/point_in_time: - $ref: 'namespaces/_core.yaml#/paths/~1_search~1point_in_time' - /_search/point_in_time/_all: - $ref: 'namespaces/_core.yaml#/paths/~1_search~1point_in_time~1_all' - /_search/scroll: - $ref: 'namespaces/_core.yaml#/paths/~1_search~1scroll' - /_search/scroll/{scroll_id}: - $ref: 'namespaces/_core.yaml#/paths/~1_search~1scroll~1{scroll_id}' - /_search/template: - $ref: 'namespaces/_core.yaml#/paths/~1_search~1template' - /_search_shards: - $ref: 'namespaces/_core.yaml#/paths/~1_search_shards' - /_update_by_query/{task_id}/_rethrottle: - $ref: 'namespaces/_core.yaml#/paths/~1_update_by_query~1{task_id}~1_rethrottle' - /{index}/_bulk: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_bulk' - /{index}/_count: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_count' - /{index}/_create/{id}: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_create~1{id}' - /{index}/_delete_by_query: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_delete_by_query' - /{index}/_doc: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_doc' - /{index}/_doc/{id}: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_doc~1{id}' - /{index}/_explain/{id}: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_explain~1{id}' - /{index}/_field_caps: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_field_caps' - /{index}/_mget: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_mget' - /{index}/_msearch: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_msearch' - /{index}/_msearch/template: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_msearch~1template' - /{index}/_mtermvectors: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_mtermvectors' - /{index}/_rank_eval: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_rank_eval' - /{index}/_search: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_search' - /{index}/_search/point_in_time: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_search~1point_in_time' - /{index}/_search/template: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_search~1template' - /{index}/_search_shards: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_search_shards' - /{index}/_source/{id}: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_source~1{id}' - /{index}/_termvectors: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_termvectors' - /{index}/_termvectors/{id}: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_termvectors~1{id}' - /{index}/_update/{id}: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_update~1{id}' - /{index}/_update_by_query: - $ref: 'namespaces/_core.yaml#/paths/~1{index}~1_update_by_query' - /_alias: - $ref: 'namespaces/indices.yaml#/paths/~1_alias' - /_alias/{name}: - $ref: 'namespaces/indices.yaml#/paths/~1_alias~1{name}' - /_aliases: - $ref: 'namespaces/indices.yaml#/paths/~1_aliases' - /_analyze: - $ref: 'namespaces/indices.yaml#/paths/~1_analyze' - /_cache/clear: - $ref: 'namespaces/indices.yaml#/paths/~1_cache~1clear' - /_data_stream: - $ref: 'namespaces/indices.yaml#/paths/~1_data_stream' - /_data_stream/_stats: - $ref: 'namespaces/indices.yaml#/paths/~1_data_stream~1_stats' - /_data_stream/{name}: - $ref: 'namespaces/indices.yaml#/paths/~1_data_stream~1{name}' - /_data_stream/{name}/_stats: - $ref: 'namespaces/indices.yaml#/paths/~1_data_stream~1{name}~1_stats' - /_flush: - $ref: 'namespaces/indices.yaml#/paths/~1_flush' - /_forcemerge: - $ref: 'namespaces/indices.yaml#/paths/~1_forcemerge' - /_index_template: - $ref: 'namespaces/indices.yaml#/paths/~1_index_template' - /_index_template/_simulate: - $ref: 'namespaces/indices.yaml#/paths/~1_index_template~1_simulate' - /_index_template/_simulate/{name}: - $ref: 'namespaces/indices.yaml#/paths/~1_index_template~1_simulate~1{name}' - /_index_template/_simulate_index/{name}: - $ref: 'namespaces/indices.yaml#/paths/~1_index_template~1_simulate_index~1{name}' - /_index_template/{name}: - $ref: 'namespaces/indices.yaml#/paths/~1_index_template~1{name}' - /_mapping: - $ref: 'namespaces/indices.yaml#/paths/~1_mapping' - /_mapping/field/{fields}: - $ref: 'namespaces/indices.yaml#/paths/~1_mapping~1field~1{fields}' - /_recovery: - $ref: 'namespaces/indices.yaml#/paths/~1_recovery' - /_refresh: - $ref: 'namespaces/indices.yaml#/paths/~1_refresh' - /_resolve/index/{name}: - $ref: 'namespaces/indices.yaml#/paths/~1_resolve~1index~1{name}' - /_segments: - $ref: 'namespaces/indices.yaml#/paths/~1_segments' - /_settings: - $ref: 'namespaces/indices.yaml#/paths/~1_settings' - /_settings/{name}: - $ref: 'namespaces/indices.yaml#/paths/~1_settings~1{name}' - /_shard_stores: - $ref: 'namespaces/indices.yaml#/paths/~1_shard_stores' - /_stats: - $ref: 'namespaces/indices.yaml#/paths/~1_stats' - /_stats/{metric}: - $ref: 'namespaces/indices.yaml#/paths/~1_stats~1{metric}' - /_template: - $ref: 'namespaces/indices.yaml#/paths/~1_template' - /_template/{name}: - $ref: 'namespaces/indices.yaml#/paths/~1_template~1{name}' - /_upgrade: - $ref: 'namespaces/indices.yaml#/paths/~1_upgrade' - /_validate/query: - $ref: 'namespaces/indices.yaml#/paths/~1_validate~1query' - /{alias}/_rollover: - $ref: 'namespaces/indices.yaml#/paths/~1{alias}~1_rollover' - /{alias}/_rollover/{new_index}: - $ref: 'namespaces/indices.yaml#/paths/~1{alias}~1_rollover~1{new_index}' - /{index}: - $ref: 'namespaces/indices.yaml#/paths/~1{index}' - /{index}/_alias: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_alias' - /{index}/_alias/{name}: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_alias~1{name}' - /{index}/_aliases/{name}: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_aliases~1{name}' - /{index}/_analyze: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_analyze' - /{index}/_block/{block}: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_block~1{block}' - /{index}/_cache/clear: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_cache~1clear' - /{index}/_clone/{target}: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_clone~1{target}' - /{index}/_close: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_close' - /{index}/_flush: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_flush' - /{index}/_forcemerge: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_forcemerge' - /{index}/_mapping: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_mapping' - /{index}/_mapping/field/{fields}: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_mapping~1field~1{fields}' - /{index}/_open: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_open' - /{index}/_recovery: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_recovery' - /{index}/_refresh: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_refresh' - /{index}/_segments: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_segments' - /{index}/_settings: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_settings' - /{index}/_settings/{name}: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_settings~1{name}' - /{index}/_shard_stores: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_shard_stores' - /{index}/_shrink/{target}: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_shrink~1{target}' - /{index}/_split/{target}: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_split~1{target}' - /{index}/_stats: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_stats' - /{index}/_stats/{metric}: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_stats~1{metric}' - /{index}/_upgrade: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_upgrade' - /{index}/_validate/query: - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_validate~1query' - /_cat: - $ref: 'namespaces/cat.yaml#/paths/~1_cat' - /_cat/aliases: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1aliases' - /_cat/aliases/{name}: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1aliases~1{name}' - /_cat/allocation: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1allocation' - /_cat/allocation/{node_id}: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1allocation~1{node_id}' - /_cat/cluster_manager: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1cluster_manager' - /_cat/count: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1count' - /_cat/count/{index}: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1count~1{index}' - /_cat/fielddata: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1fielddata' - /_cat/fielddata/{fields}: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1fielddata~1{fields}' - /_cat/health: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1health' - /_cat/indices: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1indices' - /_cat/indices/{index}: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1indices~1{index}' - /_cat/master: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1master' - /_cat/nodeattrs: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1nodeattrs' - /_cat/nodes: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1nodes' - /_cat/pending_tasks: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1pending_tasks' - /_cat/pit_segments: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1pit_segments' - /_cat/pit_segments/_all: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1pit_segments~1_all' - /_cat/plugins: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1plugins' - /_cat/recovery: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1recovery' - /_cat/recovery/{index}: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1recovery~1{index}' - /_cat/repositories: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1repositories' - /_cat/segment_replication: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1segment_replication' - /_cat/segment_replication/{index}: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1segment_replication~1{index}' - /_cat/segments: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1segments' - /_cat/segments/{index}: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1segments~1{index}' - /_cat/shards: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1shards' - /_cat/shards/{index}: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1shards~1{index}' - /_cat/snapshots: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1snapshots' - /_cat/snapshots/{repository}: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1snapshots~1{repository}' - /_cat/tasks: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1tasks' - /_cat/templates: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1templates' - /_cat/templates/{name}: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1templates~1{name}' - /_cat/thread_pool: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1thread_pool' - /_cat/thread_pool/{thread_pool_patterns}: - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1thread_pool~1{thread_pool_patterns}' - /_cluster/allocation/explain: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1allocation~1explain' - /_cluster/decommission/awareness: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1decommission~1awareness' - /_cluster/decommission/awareness/{awareness_attribute_name}/_status: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1decommission~1awareness~1{awareness_attribute_name}~1_status' - /_cluster/decommission/awareness/{awareness_attribute_name}/{awareness_attribute_value}: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1decommission~1awareness~1{awareness_attribute_name}~1{awareness_attribute_value}' - /_cluster/health: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1health' - /_cluster/health/{index}: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1health~1{index}' - /_cluster/pending_tasks: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1pending_tasks' - /_cluster/reroute: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1reroute' - /_cluster/routing/awareness/weights: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1routing~1awareness~1weights' - /_cluster/routing/awareness/{attribute}/weights: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1routing~1awareness~1{attribute}~1weights' - /_cluster/settings: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1settings' - /_cluster/state: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1state' - /_cluster/state/{metric}: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1state~1{metric}' - /_cluster/state/{metric}/{index}: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1state~1{metric}~1{index}' - /_cluster/stats: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1stats' - /_cluster/stats/nodes/{node_id}: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1stats~1nodes~1{node_id}' - /_cluster/voting_config_exclusions: - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1voting_config_exclusions' - /_component_template: - $ref: 'namespaces/cluster.yaml#/paths/~1_component_template' - /_component_template/{name}: - $ref: 'namespaces/cluster.yaml#/paths/~1_component_template~1{name}' - /_remote/info: - $ref: 'namespaces/cluster.yaml#/paths/~1_remote~1info' - /_cluster/nodes/hot_threads: - $ref: 'namespaces/nodes.yaml#/paths/~1_cluster~1nodes~1hot_threads' - /_cluster/nodes/hotthreads: - $ref: 'namespaces/nodes.yaml#/paths/~1_cluster~1nodes~1hotthreads' - /_cluster/nodes/{node_id}/hot_threads: - $ref: 'namespaces/nodes.yaml#/paths/~1_cluster~1nodes~1{node_id}~1hot_threads' - /_cluster/nodes/{node_id}/hotthreads: - $ref: 'namespaces/nodes.yaml#/paths/~1_cluster~1nodes~1{node_id}~1hotthreads' - /_nodes: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes' - /_nodes/hot_threads: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1hot_threads' - /_nodes/hotthreads: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1hotthreads' - /_nodes/reload_secure_settings: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1reload_secure_settings' - /_nodes/stats: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1stats' - /_nodes/stats/{metric}: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1stats~1{metric}' - /_nodes/stats/{metric}/{index_metric}: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1stats~1{metric}~1{index_metric}' - /_nodes/usage: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1usage' - /_nodes/usage/{metric}: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1usage~1{metric}' - /_nodes/{metric}: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1{metric}' - /_nodes/{node_id}: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1{node_id}' - /_nodes/{node_id}/hot_threads: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1{node_id}~1hot_threads' - /_nodes/{node_id}/hotthreads: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1{node_id}~1hotthreads' - /_nodes/{node_id}/reload_secure_settings: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1{node_id}~1reload_secure_settings' - /_nodes/{node_id}/stats: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1{node_id}~1stats' - /_nodes/{node_id}/stats/{metric}: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1{node_id}~1stats~1{metric}' - /_nodes/{node_id}/stats/{metric}/{index_metric}: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1{node_id}~1stats~1{metric}~1{index_metric}' - /_nodes/{node_id}/usage: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1{node_id}~1usage' - /_nodes/{node_id}/usage/{metric}: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1{node_id}~1usage~1{metric}' - /_nodes/{node_id}/{metric}: - $ref: 'namespaces/nodes.yaml#/paths/~1_nodes~1{node_id}~1{metric}' - /_dangling: - $ref: 'namespaces/dangling_indices.yaml#/paths/~1_dangling' - /_dangling/{index_uuid}: - $ref: 'namespaces/dangling_indices.yaml#/paths/~1_dangling~1{index_uuid}' - /_ingest/pipeline: - $ref: 'namespaces/ingest.yaml#/paths/~1_ingest~1pipeline' - /_ingest/pipeline/_simulate: - $ref: 'namespaces/ingest.yaml#/paths/~1_ingest~1pipeline~1_simulate' - /_ingest/pipeline/{id}: - $ref: 'namespaces/ingest.yaml#/paths/~1_ingest~1pipeline~1{id}' - /_ingest/pipeline/{id}/_simulate: - $ref: 'namespaces/ingest.yaml#/paths/~1_ingest~1pipeline~1{id}~1_simulate' - /_ingest/processor/grok: - $ref: 'namespaces/ingest.yaml#/paths/~1_ingest~1processor~1grok' - /_plugins/_knn/models/_search: - $ref: 'namespaces/knn.yaml#/paths/~1_plugins~1_knn~1models~1_search' - /_plugins/_knn/models/_train: - $ref: 'namespaces/knn.yaml#/paths/~1_plugins~1_knn~1models~1_train' - /_plugins/_knn/models/{model_id}: - $ref: 'namespaces/knn.yaml#/paths/~1_plugins~1_knn~1models~1{model_id}' - /_plugins/_knn/models/{model_id}/_train: - $ref: 'namespaces/knn.yaml#/paths/~1_plugins~1_knn~1models~1{model_id}~1_train' - /_plugins/_knn/stats: - $ref: 'namespaces/knn.yaml#/paths/~1_plugins~1_knn~1stats' - /_plugins/_knn/stats/{stat}: - $ref: 'namespaces/knn.yaml#/paths/~1_plugins~1_knn~1stats~1{stat}' - /_plugins/_knn/warmup/{index}: - $ref: 'namespaces/knn.yaml#/paths/~1_plugins~1_knn~1warmup~1{index}' - /_plugins/_knn/{node_id}/stats: - $ref: 'namespaces/knn.yaml#/paths/~1_plugins~1_knn~1{node_id}~1stats' - /_plugins/_knn/{node_id}/stats/{stat}: - $ref: 'namespaces/knn.yaml#/paths/~1_plugins~1_knn~1{node_id}~1stats~1{stat}' - /_plugins/_notifications/channels: - $ref: 'namespaces/notifications.yaml#/paths/~1_plugins~1_notifications~1channels' - /_plugins/_notifications/configs: - $ref: 'namespaces/notifications.yaml#/paths/~1_plugins~1_notifications~1configs' - /_plugins/_notifications/configs/{config_id}: - $ref: 'namespaces/notifications.yaml#/paths/~1_plugins~1_notifications~1configs~1{config_id}' - /_plugins/_notifications/feature/test/{config_id}: - $ref: 'namespaces/notifications.yaml#/paths/~1_plugins~1_notifications~1feature~1test~1{config_id}' - /_plugins/_notifications/features: - $ref: 'namespaces/notifications.yaml#/paths/~1_plugins~1_notifications~1features' - /_plugins/_security/api/account: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1account' - /_plugins/_security/api/actiongroups: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1actiongroups' - /_plugins/_security/api/actiongroups/{action_group}: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1actiongroups~1{action_group}' - /_plugins/_security/api/audit: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1audit' - /_plugins/_security/api/audit/config: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1audit~1config' - /_plugins/_security/api/cache: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1cache' - /_plugins/_security/api/internalusers: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1internalusers' - /_plugins/_security/api/internalusers/{username}: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1internalusers~1{username}' - /_plugins/_security/api/nodesdn: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1nodesdn' - /_plugins/_security/api/nodesdn/{cluster_name}: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1nodesdn~1{cluster_name}' - /_plugins/_security/api/roles: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1roles' - /_plugins/_security/api/roles/: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1roles~1' - /_plugins/_security/api/roles/{role}: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1roles~1{role}' - /_plugins/_security/api/rolesmapping: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1rolesmapping' - /_plugins/_security/api/rolesmapping/{role}: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1rolesmapping~1{role}' - /_plugins/_security/api/securityconfig: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1securityconfig' - /_plugins/_security/api/securityconfig/config: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1securityconfig~1config' - /_plugins/_security/api/ssl/certs: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1ssl~1certs' - /_plugins/_security/api/ssl/http/reloadcerts: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1ssl~1http~1reloadcerts' - /_plugins/_security/api/ssl/transport/reloadcerts: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1ssl~1transport~1reloadcerts' - /_plugins/_security/api/tenants/: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1tenants~1' - /_plugins/_security/api/tenants/{tenant}: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1api~1tenants~1{tenant}' - /_plugins/_security/health: - $ref: 'namespaces/security.yaml#/paths/~1_plugins~1_security~1health' - /_remotestore/_restore: - $ref: 'namespaces/remote_store.yaml#/paths/~1_remotestore~1_restore' - /_search/pipeline: - $ref: 'namespaces/search_pipeline.yaml#/paths/~1_search~1pipeline' - /_search/pipeline/{id}: - $ref: 'namespaces/search_pipeline.yaml#/paths/~1_search~1pipeline~1{id}' - /_snapshot: - $ref: 'namespaces/snapshot.yaml#/paths/~1_snapshot' - /_snapshot/_status: - $ref: 'namespaces/snapshot.yaml#/paths/~1_snapshot~1_status' - /_snapshot/{repository}: - $ref: 'namespaces/snapshot.yaml#/paths/~1_snapshot~1{repository}' - /_snapshot/{repository}/_cleanup: - $ref: 'namespaces/snapshot.yaml#/paths/~1_snapshot~1{repository}~1_cleanup' - /_snapshot/{repository}/_status: - $ref: 'namespaces/snapshot.yaml#/paths/~1_snapshot~1{repository}~1_status' - /_snapshot/{repository}/_verify: - $ref: 'namespaces/snapshot.yaml#/paths/~1_snapshot~1{repository}~1_verify' - /_snapshot/{repository}/{snapshot}: - $ref: 'namespaces/snapshot.yaml#/paths/~1_snapshot~1{repository}~1{snapshot}' - /_snapshot/{repository}/{snapshot}/_clone/{target_snapshot}: - $ref: 'namespaces/snapshot.yaml#/paths/~1_snapshot~1{repository}~1{snapshot}~1_clone~1{target_snapshot}' - /_snapshot/{repository}/{snapshot}/_restore: - $ref: 'namespaces/snapshot.yaml#/paths/~1_snapshot~1{repository}~1{snapshot}~1_restore' - /_snapshot/{repository}/{snapshot}/_status: - $ref: 'namespaces/snapshot.yaml#/paths/~1_snapshot~1{repository}~1{snapshot}~1_status' - /_tasks: - $ref: 'namespaces/tasks.yaml#/paths/~1_tasks' - /_tasks/_cancel: - $ref: 'namespaces/tasks.yaml#/paths/~1_tasks~1_cancel' - /_tasks/{task_id}: - $ref: 'namespaces/tasks.yaml#/paths/~1_tasks~1{task_id}' - /_tasks/{task_id}/_cancel: - $ref: 'namespaces/tasks.yaml#/paths/~1_tasks~1{task_id}~1_cancel' -components: - parameters: - _global::query.pretty: - x-global: true - name: pretty - in: query - description: Whether to pretty format the returned JSON response. - schema: - type: boolean - default: false - _global::query.human: - x-global: true - name: human - in: query - description: Whether to return human readable values for statistics. - schema: - type: boolean - default: true - _global::query.error_trace: - x-global: true - name: error_trace - in: query - description: Whether to include the stack trace of returned errors. - schema: - type: boolean - default: false - _global::query.source: - x-global: true - name: source - in: query - description: The URL-encoded request definition. Useful for libraries that do not accept a request body for non-POST requests. - schema: - type: string - _global::query.filter_path: - x-global: true - name: filter_path - in: query - description: Comma-separated list of filters used to reduce the response. - schema: - oneOf: - - type: string - - type: array - items: - type: string diff --git a/tools/README.md b/tools/README.md index 008449bd8..a9bffabab 100644 --- a/tools/README.md +++ b/tools/README.md @@ -21,13 +21,13 @@ Example: ```bash mkdir -p ../build -export ROOT_PATH=../spec/opensearch-openapi.yaml +export ROOT_PATH=../spec export OUTPUT_PATH=../build/opensearch-openapi.yaml npm run merge -- $ROOT_PATH $OUTPUT_PATH ``` As a shortcut, if those parameters are not provided, the tool will use the default values: -- `../spec/opensearch-openapi.yaml` as the root path (i.e. the root file of the repo's [spec folder](../spec)) +- `../spec` as the root path (i.e. the repo's [spec folder](../spec)) - `../opensearch-openapi.yaml` as the output path ```bash diff --git a/tools/helpers.ts b/tools/helpers.ts index 4f3301930..09e6fcb6f 100644 --- a/tools/helpers.ts +++ b/tools/helpers.ts @@ -43,7 +43,11 @@ export function sort_by_keys (obj: Record, priorities: string[] = [ }) } -export function write_to_yaml (file_path: string, content: Record): void { +export function read_yaml (file_path: string): Record { + return YAML.parse(fs.readFileSync(file_path, 'utf8')) +} + +export function write_yaml (file_path: string, content: Record): void { fs.writeFileSync(file_path, quote_refs(YAML.stringify(remove_anchors(content), { lineWidth: 0, singleQuote: true }))) } diff --git a/tools/linter/PathRefsValidator.ts b/tools/linter/PathRefsValidator.ts deleted file mode 100644 index 2ebb53a68..000000000 --- a/tools/linter/PathRefsValidator.ts +++ /dev/null @@ -1,85 +0,0 @@ -import { type ValidationError } from '../types' -import type RootFile from './components/RootFile' -import type NamespacesFolder from './components/NamespacesFolder' - -export default class PathRefsValidator { - root_file: RootFile - namespaces_folder: NamespacesFolder - - referenced_paths: Record> = {} // file -> paths - available_paths: Record> = {} // file -> paths - - constructor (root_file: RootFile, namespaces_folder: NamespacesFolder) { - this.root_file = root_file - this.namespaces_folder = namespaces_folder - this.#build_referenced_paths() - this.#build_available_paths() - } - - #build_referenced_paths (): void { - for (const [path, spec] of Object.entries(this.root_file.spec().paths)) { - const ref = spec?.$ref ?? '' - const file = ref.split('#')[0] - const ref_path = this.referenced_paths[file] as Set | undefined - if (!ref_path) this.referenced_paths[file] = new Set() - this.referenced_paths[file].add(path) - } - } - - #build_available_paths (): void { - for (const file of this.namespaces_folder.files) { - this.available_paths[file.file] = new Set(Object.keys(file.spec().paths ?? {})) - } - } - - validate (): ValidationError[] { - return [ - ...this.validate_unresolved_refs(), - ...this.validate_unreferenced_paths() - ] - } - - validate_unresolved_refs (): ValidationError[] { - return Object.entries(this.referenced_paths).flatMap(([ref_file, ref_paths]) => { - const available = this.available_paths[ref_file] as Set | undefined - if (!available) { - return { - file: this.root_file.file, - location: `Paths: ${[...ref_paths].join(' , ')}`, - message: `Unresolved path reference: Namespace file ${ref_file} does not exist.` - } - } - - return Array.from(ref_paths).map((path) => { - if (!available.has(path)) { - return { - file: this.root_file.file, - location: `Path: ${path}`, - message: `Unresolved path reference: Path ${path} does not exist in namespace file ${ref_file}.` - } - } - }).filter((e) => e) as ValidationError[] - }) - } - - validate_unreferenced_paths (): ValidationError[] { - return Object.entries(this.available_paths).flatMap(([ns_file, ns_paths]) => { - const referenced = this.referenced_paths[ns_file] as Set | undefined - if (!referenced) { - return { - file: ns_file, - message: 'Unreferenced paths: No paths are referenced in the root file.' - } - } - return Array.from(ns_paths).map((path) => { - if (!referenced?.has(path)) { - return { - file: ns_file, - location: `Path: ${path}`, - message: `Unreferenced path: Path ${path} is not referenced in the root file.` - } - } - }).filter((e) => e) as ValidationError[] - }) - } -} diff --git a/tools/linter/SpecValidator.ts b/tools/linter/SpecValidator.ts index b227c4fec..a9ea98f14 100644 --- a/tools/linter/SpecValidator.ts +++ b/tools/linter/SpecValidator.ts @@ -1,38 +1,30 @@ import SchemasFolder from './components/SchemasFolder' import NamespacesFolder from './components/NamespacesFolder' -import RootFile from './components/RootFile' import { type ValidationError } from '../types' -import PathRefsValidator from './PathRefsValidator' import SchemaRefsValidator from './SchemaRefsValidator' import SupersededOperationsFile from './components/SupersededOperationsFile' export default class SpecValidator { - root_file: RootFile superseded_ops_files: SupersededOperationsFile namespaces_folder: NamespacesFolder schemas_folder: SchemasFolder - path_refs_validator: PathRefsValidator schema_refs_validator: SchemaRefsValidator constructor (root_folder: string) { - this.root_file = new RootFile(`${root_folder}/opensearch-openapi.yaml`) this.superseded_ops_files = new SupersededOperationsFile(`${root_folder}/_superseded_operations.yaml`) this.namespaces_folder = new NamespacesFolder(`${root_folder}/namespaces`) this.schemas_folder = new SchemasFolder(`${root_folder}/schemas`) - this.path_refs_validator = new PathRefsValidator(this.root_file, this.namespaces_folder) this.schema_refs_validator = new SchemaRefsValidator(this.namespaces_folder, this.schemas_folder) } validate (): ValidationError[] { const component_errors = [ - ...this.root_file.validate(), ...this.namespaces_folder.validate(), ...this.schemas_folder.validate() ] if (component_errors.length > 0) return component_errors return [ - ...this.path_refs_validator.validate(), ...this.schema_refs_validator.validate(), ...this.superseded_ops_files.validate() ] diff --git a/tools/linter/components/RootFile.ts b/tools/linter/components/RootFile.ts deleted file mode 100644 index 4ee9858d9..000000000 --- a/tools/linter/components/RootFile.ts +++ /dev/null @@ -1,32 +0,0 @@ -import { type ParameterSpec, type ValidationError } from '../../types' -import FileValidator from './base/FileValidator' - -export default class RootFile extends FileValidator { - constructor (file_path: string) { - super(file_path) - this.file = file_path.split('/').pop() ?? '' - } - - validate_file (): ValidationError[] { - return [ - this.validate_paths(), - this.validate_params() - ].flat() - } - - validate_paths (): ValidationError[] { - return Object.entries(this.spec().paths).map(([path, spec]) => { - if (spec?.$ref != null) return undefined - return this.error('Every path must be a reference object to a path in a namespace file.', `Path: ${path}`) - }).filter((e) => e) as ValidationError[] - } - - validate_params (): ValidationError[] { - const params = (this.spec().components?.parameters ?? {}) as Record - return Object.entries(params).map(([name, param]) => { - const expected = `_global::${param.in}.${param.name}` - if (name !== expected) { return this.error(`Parameters in root file must be in the format '_global::{in}.{name}'. Expected '${expected}'.`, `#/components/parameters/${name}`) } - if (param['x-global'] !== true) { return this.error('Parameters in root file must have \'x-global\' extension set to true.', `#/components/parameters/${name}`) } - }).filter((e) => e) as ValidationError[] - } -} diff --git a/tools/linter/components/SupersededOperationsFile.ts b/tools/linter/components/SupersededOperationsFile.ts index 9f9252aba..335a897ff 100644 --- a/tools/linter/components/SupersededOperationsFile.ts +++ b/tools/linter/components/SupersededOperationsFile.ts @@ -1,8 +1,7 @@ import FileValidator from './base/FileValidator' import ajv from 'ajv' -import fs from 'fs' -import YAML from 'yaml' import { type ValidationError } from '../../types' +import { read_yaml } from '../../helpers' export default class SupersededOperationsFile extends FileValidator { readonly JSON_SCHEMA_PATH = '../json_schemas/_superseded_operations.yaml' @@ -14,7 +13,7 @@ export default class SupersededOperationsFile extends FileValidator { } validate_json_schema (): ValidationError | undefined { - const schema = YAML.parse(fs.readFileSync(this.JSON_SCHEMA_PATH, 'utf8')) + const schema = read_yaml(this.JSON_SCHEMA_PATH) const validator = (new ajv()).compile(schema) if (!validator(this.spec())) { return this.error(`File content does not match JSON schema found in '${this.JSON_SCHEMA_PATH}':\n ${JSON.stringify(validator.errors, null, 2)}`) diff --git a/tools/linter/components/base/FileValidator.ts b/tools/linter/components/base/FileValidator.ts index b01d59d17..f2d777216 100644 --- a/tools/linter/components/base/FileValidator.ts +++ b/tools/linter/components/base/FileValidator.ts @@ -1,8 +1,7 @@ import ValidatorBase from './ValidatorBase' import { type ValidationError } from '../../../types' -import fs from 'fs' -import YAML from 'yaml' import { type OpenAPIV3 } from 'openapi-types' +import { read_yaml } from '../../../helpers' export default class FileValidator extends ValidatorBase { file_path: string @@ -15,7 +14,7 @@ export default class FileValidator extends ValidatorBase { spec (): OpenAPIV3.Document { if (this._spec) return this._spec - this._spec = YAML.parse(fs.readFileSync(this.file_path, 'utf8')) as OpenAPIV3.Document + this._spec = read_yaml(this.file_path) as OpenAPIV3.Document return this._spec } diff --git a/tools/merger/GlobalParamsGenerator.ts b/tools/merger/GlobalParamsGenerator.ts new file mode 100644 index 000000000..df84ff8c3 --- /dev/null +++ b/tools/merger/GlobalParamsGenerator.ts @@ -0,0 +1,36 @@ +import { type OpenAPIV3 } from 'openapi-types' +import _ from 'lodash' +import { read_yaml } from '../helpers' + +export default class GlobalParamsGenerator { + global_params: Record + + constructor (root_path: string) { + const file_path = root_path + '/_global_parameters.yaml' + const spec = read_yaml(file_path) as OpenAPIV3.Document + this.global_params = this.create_global_params(spec) + } + + generate (spec: Record): void { + spec.components.parameters = { ...this.global_params, ...spec.components.parameters } + + const global_param_refs = Object.keys(this.global_params).map(param => ({ $ref: `#/components/parameters/${param}` })) + Object.entries(spec.paths as Document).forEach(([path, path_item]) => { + Object.entries(path_item as Document).forEach(([method, operation]) => { + const params = operation.parameters ?? [] + operation.parameters = [...params, ...Object.values(global_param_refs)] + }) + }) + } + + create_global_params (spec: OpenAPIV3.Document): Record { + const params = (spec.components?.parameters ?? {}) as Record + _.entries(params).forEach(([original_key, param]) => { + const global_key = `_global::${param.in}.${param.name}` + params[global_key] = param + _.set(param, 'x-global', true) + _.unset(params, original_key) + }) + return params + } +} diff --git a/tools/merger/OpenApiMerger.ts b/tools/merger/OpenApiMerger.ts index a35f065ea..7054eac12 100644 --- a/tools/merger/OpenApiMerger.ts +++ b/tools/merger/OpenApiMerger.ts @@ -1,74 +1,55 @@ import { type OpenAPIV3 } from 'openapi-types' import fs from 'fs' import _ from 'lodash' -import yaml from 'yaml' -import { write_to_yaml } from '../helpers' +import { read_yaml, write_yaml } from '../helpers' import SupersededOpsGenerator from './SupersededOpsGenerator' +import GlobalParamsGenerator from './GlobalParamsGenerator' // Create a single-file OpenAPI spec from multiple files for OpenAPI validation and programmatic consumption export default class OpenApiMerger { - root_path: string root_folder: string spec: Record - global_param_refs: OpenAPIV3.ReferenceObject[] paths: Record> = {} // namespace -> path -> path_item_object schemas: Record> = {} // category -> schema -> schema_object - constructor (root_path: string) { - this.root_path = fs.realpathSync(root_path) - this.root_folder = this.root_path.split('/').slice(0, -1).join('/') - this.spec = yaml.parse(fs.readFileSync(this.root_path, 'utf8')) - const global_params: OpenAPIV3.ParameterObject = this.spec.components?.parameters ?? {} - this.global_param_refs = Object.keys(global_params).map(param => ({ $ref: `#/components/parameters/${param}` })) - this.spec.components = { - parameters: global_params, - requestBodies: {}, - responses: {}, - schemas: {} + constructor (root_folder: string) { + this.root_folder = fs.realpathSync(root_folder) + this.spec = { + openapi: '3.1.0', + info: read_yaml(`${this.root_folder}/_info.yaml`), + paths: {}, + components: { + parameters: {}, + requestBodies: {}, + responses: {}, + schemas: {} + } } } merge (output_path: string = ''): OpenAPIV3.Document { this.#merge_schemas() this.#merge_namespaces() - this.#apply_global_params() this.#sort_spec_keys() - this.#generate_replaced_ops() + this.#generate_global_params() + this.#generate_superseded_ops() - if (output_path !== '') write_to_yaml(output_path, this.spec) + if (output_path !== '') write_yaml(output_path, this.spec) return this.spec as OpenAPIV3.Document } - // Apply global parameters to all operations in the spec. - #apply_global_params (): void { - Object.entries(this.spec.paths as Document).forEach(([path, path_item]) => { - Object.entries(path_item as Document).forEach(([method, operation]) => { - const params = operation.parameters ?? [] - operation.parameters = [...params, ...Object.values(this.global_param_refs)] - }) - }) - } - // Merge files from /namespaces folder. #merge_namespaces (): void { const folder = `${this.root_folder}/namespaces` fs.readdirSync(folder).forEach(file => { - const spec = yaml.parse(fs.readFileSync(`${folder}/${file}`, 'utf8')) - const namespace = file.split('.yaml')[0] + const spec = read_yaml(`${folder}/${file}`) this.redirect_refs_in_namespace(spec) - this.paths[namespace] = spec.paths + this.spec.paths = { ...this.spec.paths, ...spec.paths } this.spec.components.parameters = { ...this.spec.components.parameters, ...spec.components.parameters } this.spec.components.responses = { ...this.spec.components.responses, ...spec.components.responses } this.spec.components.requestBodies = { ...this.spec.components.requestBodies, ...spec.components.requestBodies } }) - - Object.entries(this.spec.paths as Document).forEach(([path, ref_obj]) => { - const ref: string = (ref_obj as Record).$ref ?? '' - const namespace = ref.match(/namespaces\/(.*)\.yaml/)?.[1] ?? '' - if (namespace === '') throw new Error(`Invalid path reference: ${ref}`) - this.spec.paths[path] = this.paths[namespace][path] - }) } // Redirect schema references in namespace files to local references in single-file spec. @@ -85,7 +66,7 @@ export default class OpenApiMerger { #merge_schemas (): void { const folder = `${this.root_folder}/schemas` fs.readdirSync(folder).forEach(file => { - const spec = yaml.parse(fs.readFileSync(`${folder}/${file}`, 'utf8')) + const spec = read_yaml(`${folder}/${file}`) const category = file.split('.yaml')[0] this.redirect_refs_in_schema(category, spec) this.schemas[category] = spec.components.schemas as Record @@ -127,7 +108,14 @@ export default class OpenApiMerger { }) } - #generate_replaced_ops (): void { + // Generate global parameters from _global_params.yaml file. + #generate_global_params (): void { + const gen = new GlobalParamsGenerator(this.root_folder) + gen.generate(this.spec) + } + + // Generate superseded operations from _superseded_operations.yaml file. + #generate_superseded_ops (): void { const gen = new SupersededOpsGenerator(this.root_folder) gen.generate(this.spec) } diff --git a/tools/merger/OpenDistro.ts b/tools/merger/OpenDistro.ts index 126e3b5eb..e35cace6e 100644 --- a/tools/merger/OpenDistro.ts +++ b/tools/merger/OpenDistro.ts @@ -1,7 +1,5 @@ -import fs from 'fs' -import YAML from 'yaml' import { type HttpVerb, type OperationPath, type SupersededOperationMap } from '../types' -import { write_to_yaml } from '../helpers' +import { read_yaml, write_yaml } from '../helpers' // One-time script to generate _superseded_operations.yaml file for OpenDistro // Keeping this for now in case we need to update the file in the near future. Can be removed after a few months. @@ -11,9 +9,9 @@ export default class OpenDistro { output: SupersededOperationMap = {} constructor (file_path: string) { - this.input = YAML.parse(fs.readFileSync(file_path, 'utf8')) + this.input = read_yaml(file_path) this.build_output() - write_to_yaml(file_path, this.output) + write_yaml(file_path, this.output) } build_output (): void { diff --git a/tools/merger/SupersededOpsGenerator.ts b/tools/merger/SupersededOpsGenerator.ts index 826a7eab0..747204c61 100644 --- a/tools/merger/SupersededOpsGenerator.ts +++ b/tools/merger/SupersededOpsGenerator.ts @@ -1,14 +1,13 @@ import { type OperationSpec, type SupersededOperationMap } from '../types' -import YAML from 'yaml' -import fs from 'fs' import _ from 'lodash' +import { read_yaml } from '../helpers' export default class SupersededOpsGenerator { superseded_ops: SupersededOperationMap constructor (root_path: string) { const file_path = root_path + '/_superseded_operations.yaml' - this.superseded_ops = YAML.parse(fs.readFileSync(file_path, 'utf8')) + this.superseded_ops = read_yaml(file_path) as SupersededOperationMap delete this.superseded_ops.$schema } diff --git a/tools/merger/merge.ts b/tools/merger/merge.ts index b4531b4b4..a32d3059b 100644 --- a/tools/merger/merge.ts +++ b/tools/merger/merge.ts @@ -1,6 +1,6 @@ import OpenApiMerger from './OpenApiMerger' -const root_path: string = process.argv[2] || '../spec/opensearch-openapi.yaml' +const root_path: string = process.argv[2] || '../spec' const output_path: string = process.argv[3] || '../opensearch-openapi.yaml' const merger = new OpenApiMerger(root_path) merger.merge(output_path) diff --git a/tools/test/linter/PathRefsValidator.test.ts b/tools/test/linter/PathRefsValidator.test.ts deleted file mode 100644 index eaf8c2826..000000000 --- a/tools/test/linter/PathRefsValidator.test.ts +++ /dev/null @@ -1,31 +0,0 @@ -import PathRefsValidator from '../../linter/PathRefsValidator' -import RootFile from '../../linter/components/RootFile' -import NamespacesFolder from '../../linter/components/NamespacesFolder' - -test('validate()', () => { - const root_folder = './test/linter/fixtures/path_refs_validator' - const root_file = new RootFile(`${root_folder}/opensearch-openapi.yaml`) - const namespaces_folder = new NamespacesFolder(`${root_folder}/namespaces`) - const validator = new PathRefsValidator(root_file, namespaces_folder) - expect(validator.validate()).toEqual([ - { - file: 'opensearch-openapi.yaml', - location: 'Path: /{index}', - message: 'Unresolved path reference: Path /{index} does not exist in namespace file namespaces/indices.yaml.' - }, - { - file: 'opensearch-openapi.yaml', - location: 'Paths: /_cluster/health , /_cluster/{id}', - message: 'Unresolved path reference: Namespace file namespaces/cluster.yaml does not exist.' - }, - { - file: 'namespaces/indices.yaml', - location: 'Path: /{index}/_aliases', - message: 'Unreferenced path: Path /{index}/_aliases is not referenced in the root file.' - }, - { - file: 'namespaces/missing.yaml', - message: 'Unreferenced paths: No paths are referenced in the root file.' - } - ]) -}) diff --git a/tools/test/linter/RootFile.test.ts b/tools/test/linter/RootFile.test.ts deleted file mode 100644 index fe365fa89..000000000 --- a/tools/test/linter/RootFile.test.ts +++ /dev/null @@ -1,27 +0,0 @@ -import RootFile from '../../linter/components/RootFile' - -test('validate()', () => { - const validator = new RootFile('./test/linter/fixtures/root.yaml') - expect(validator.validate()).toEqual([ - { - file: 'root.yaml', - location: 'Path: /', - message: 'Every path must be a reference object to a path in a namespace file.' - }, - { - file: 'root.yaml', - location: 'Path: /{index}', - message: 'Every path must be a reference object to a path in a namespace file.' - }, - { - file: 'root.yaml', - location: '#/components/parameters/_global::query.pretty', - message: "Parameters in root file must be in the format '_global::{in}.{name}'. Expected '_global::query.beautify'." - }, - { - file: 'root.yaml', - location: '#/components/parameters/_global::query.human', - message: "Parameters in root file must have 'x-global' extension set to true." - } - ]) -}) diff --git a/tools/test/linter/SpecValidator.test.ts b/tools/test/linter/SpecValidator.test.ts index 97b40a2dd..181393f02 100644 --- a/tools/test/linter/SpecValidator.test.ts +++ b/tools/test/linter/SpecValidator.test.ts @@ -6,7 +6,6 @@ test('validate()', () => { validator.namespaces_folder.validate = jest.fn().mockReturnValue([{ file: 'namespaces/', message: 'namespace error' }]) validator.schemas_folder.validate = jest.fn().mockReturnValue([{ file: 'schemas/', message: 'schema error' }]) - validator.path_refs_validator.validate = jest.fn().mockReturnValue([{ file: 'path_refs', message: 'path refs error' }]) validator.schema_refs_validator.validate = jest.fn().mockReturnValue([{ file: 'schema_refs', message: 'schema refs error' }]) expect(validator.validate()).toEqual([ @@ -18,7 +17,6 @@ test('validate()', () => { validator.schemas_folder.validate = jest.fn().mockReturnValue([]) expect(validator.validate()).toEqual([ - { file: 'path_refs', message: 'path refs error' }, { file: 'schema_refs', message: 'schema refs error' } ]) }) diff --git a/tools/test/linter/fixtures/path_refs_validator/namespaces/cat.yaml b/tools/test/linter/fixtures/path_refs_validator/namespaces/cat.yaml deleted file mode 100644 index 983c3de46..000000000 --- a/tools/test/linter/fixtures/path_refs_validator/namespaces/cat.yaml +++ /dev/null @@ -1,3 +0,0 @@ -paths: - '/_cat/aliases': {} - '/_cat/allocation': {} \ No newline at end of file diff --git a/tools/test/linter/fixtures/path_refs_validator/namespaces/indices.yaml b/tools/test/linter/fixtures/path_refs_validator/namespaces/indices.yaml deleted file mode 100644 index e5481a420..000000000 --- a/tools/test/linter/fixtures/path_refs_validator/namespaces/indices.yaml +++ /dev/null @@ -1,3 +0,0 @@ -paths: - '/{index}/_alias/{name}' : {} - '/{index}/_aliases' : {} \ No newline at end of file diff --git a/tools/test/linter/fixtures/path_refs_validator/namespaces/missing.yaml b/tools/test/linter/fixtures/path_refs_validator/namespaces/missing.yaml deleted file mode 100644 index d64fbea98..000000000 --- a/tools/test/linter/fixtures/path_refs_validator/namespaces/missing.yaml +++ /dev/null @@ -1,3 +0,0 @@ -paths: - '/_missing': {} - '/_missing/{id}': {} \ No newline at end of file diff --git a/tools/test/linter/fixtures/path_refs_validator/opensearch-openapi.yaml b/tools/test/linter/fixtures/path_refs_validator/opensearch-openapi.yaml deleted file mode 100644 index 4ea72a813..000000000 --- a/tools/test/linter/fixtures/path_refs_validator/opensearch-openapi.yaml +++ /dev/null @@ -1,13 +0,0 @@ -paths: - '/{index}': - $ref: 'namespaces/indices.yaml#/paths/~1{index}' - '/{index}/_alias/{name}': - $ref: 'namespaces/indices.yaml#/paths/~1{index}~1_alias~1{name}' - '/_cat/aliases': - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1aliases' - '/_cat/allocation': - $ref: 'namespaces/cat.yaml#/paths/~1_cat~1allocation' - '/_cluster/health': - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1health' - '/_cluster/{id}': - $ref: 'namespaces/cluster.yaml#/paths/~1_cluster~1{id}' \ No newline at end of file diff --git a/tools/test/linter/fixtures/root.yaml b/tools/test/linter/fixtures/root.yaml deleted file mode 100644 index 8c4a08750..000000000 --- a/tools/test/linter/fixtures/root.yaml +++ /dev/null @@ -1,32 +0,0 @@ -paths: - '/': {} - '/_cat': - $ref: './namespaces/cat.yaml#/paths/_cat' - '/{index}': - get: - responses: - '200': - $ref: '#/components/responses/indices.create@200' - parameters: - - $ref: '#/components/parameters/indices.create::path.index' - - $ref: '#/components/parameters/indices.create::query.pretty' -components: - parameters: - _global::query.pretty: - x-global: true - name: beautify - in: query - schema: - type: boolean - _global::query.human: - x-global: false - name: human - in: query - schema: - type: boolean - _global::query.error_trace: - x-global: true - name: error_trace - in: query - schema: - type: boolean \ No newline at end of file diff --git a/tools/test/merger/OpenApiMerger.test.ts b/tools/test/merger/OpenApiMerger.test.ts index 0e2513dd3..22d966404 100644 --- a/tools/test/merger/OpenApiMerger.test.ts +++ b/tools/test/merger/OpenApiMerger.test.ts @@ -2,7 +2,7 @@ import OpenApiMerger from '../../merger/OpenApiMerger' import fs from 'fs' test('merge()', async () => { - const merger = new OpenApiMerger('./test/merger/fixtures/spec/opensearch-openapi.yaml') + const merger = new OpenApiMerger('./test/merger/fixtures/spec/') merger.merge('./test/merger/opensearch-openapi.yaml') expect(fs.readFileSync('./test/merger/fixtures/expected.yaml', 'utf8')) .toEqual(fs.readFileSync('./test/merger/opensearch-openapi.yaml', 'utf8')) diff --git a/tools/test/merger/fixtures/expected.yaml b/tools/test/merger/fixtures/expected.yaml index 95772eb3b..3c73f80a1 100644 --- a/tools/test/merger/fixtures/expected.yaml +++ b/tools/test/merger/fixtures/expected.yaml @@ -4,6 +4,17 @@ info: description: OpenSearch API version: 1.0.0 paths: + /{index}: + post: + parameters: + - $ref: '#/components/parameters/indices.create::path.index' + - $ref: '#/components/parameters/indices.create::query.pretty' + - $ref: '#/components/parameters/_global::query.human' + requestBody: + $ref: '#/components/requestBodies/indices.create' + responses: + '200': + $ref: '#/components/responses/indices.create@200' /adopt/{animal}/dockets/{docket}: get: operationId: adopt.0 @@ -40,13 +51,13 @@ paths: components: parameters: _global::query.human: - x-global: true name: human in: query description: Whether to return human readable values for statistics. schema: type: boolean default: true + x-global: true adopt::path.animal: name: animal in: path diff --git a/tools/test/merger/fixtures/spec/_global_parameters.yaml b/tools/test/merger/fixtures/spec/_global_parameters.yaml new file mode 100644 index 000000000..cf491eee2 --- /dev/null +++ b/tools/test/merger/fixtures/spec/_global_parameters.yaml @@ -0,0 +1,13 @@ +openapi: 3.1.0 +info: + title: '' + version: '' +components: + parameters: + human: + name: human + in: query + description: Whether to return human readable values for statistics. + schema: + type: boolean + default: true \ No newline at end of file diff --git a/tools/test/merger/fixtures/spec/_info.yaml b/tools/test/merger/fixtures/spec/_info.yaml new file mode 100644 index 000000000..9da26bfa0 --- /dev/null +++ b/tools/test/merger/fixtures/spec/_info.yaml @@ -0,0 +1,3 @@ +title: OpenSearch API +description: OpenSearch API +version: 1.0.0 \ No newline at end of file diff --git a/tools/test/merger/fixtures/spec/namespaces/ignored.yaml b/tools/test/merger/fixtures/spec/namespaces/indices.yaml similarity index 100% rename from tools/test/merger/fixtures/spec/namespaces/ignored.yaml rename to tools/test/merger/fixtures/spec/namespaces/indices.yaml diff --git a/tools/test/merger/fixtures/spec/opensearch-openapi.yaml b/tools/test/merger/fixtures/spec/opensearch-openapi.yaml deleted file mode 100644 index b9bdd719d..000000000 --- a/tools/test/merger/fixtures/spec/opensearch-openapi.yaml +++ /dev/null @@ -1,18 +0,0 @@ -openapi: 3.1.0 -info: - title: OpenSearch API - description: OpenSearch API - version: 1.0.0 -paths: - '/adopt/{animal}/dockets/{docket}': - $ref: 'namespaces/shelter.yaml#/paths/~1adopt~1{animal}~1dockets~1{docket}' -components: - parameters: - _global::query.human: - x-global: true - name: human - in: query - description: Whether to return human readable values for statistics. - schema: - type: boolean - default: true \ No newline at end of file