Added support for skipping tests using semver range. (#410)

* Added support for skipping tests using semver range. Signed-off-by: dblock <[email protected]> * OpenSearch 2.0 returns a null custom_foldername for opensearch-security. Signed-off-by: dblock <[email protected]> * Security-analytics was aded in OpenSearch 2.4.0. Signed-off-by: dblock <[email protected]> * Specify semver range for entire stories. Signed-off-by: dblock <[email protected]> * Segment replication was introduced in 2.7. Signed-off-by: dblock <[email protected]> * Clear file cache was added in OpenSearch 2.8.0. Signed-off-by: dblock <[email protected]> * The primary_only and wait_for_completion parameters were introduced in 2.13 and 2.7 respectively. Signed-off-by: dblock <[email protected]> * Added remaining version tags. Signed-off-by: dblock <[email protected]> --------- Signed-off-by: dblock <[email protected]>
opensearch-project · Jul 11, 2024 · 1a73935 · 1a73935
1 parent f8788da
commit 1a73935
Show file tree

Hide file tree

Showing 39 changed files with 289 additions and 54 deletions.
diff --git a/.github/workflows/test-spec.yml b/.github/workflows/test-spec.yml
@@ -21,14 +21,18 @@ jobs:
     strategy:
       matrix:
         entry:
-          - {version: 2.15.0, hub: 'opensearchproject'}
+          - {version: 2.0.0, admin_password: 'admin'}
+          - {version: 2.15.0}
           - {version: 2.16.0, hub: 'opensearchstaging'}
-    name: test-opensearch-spec (version=${{ matrix.entry.version }}, hub=${{ matrix.entry.hub }})
+
+    name: test-opensearch-spec (version=${{ matrix.entry.version }}, hub=${{ matrix.entry.hub || 'opensearchproject' }})
     runs-on: ubuntu-latest
+
     env:
-      OPENSEARCH_DOCKER_HUB_PROJECT: ${{ matrix.entry.hub }}
+      OPENSEARCH_DOCKER_HUB_PROJECT: ${{ matrix.entry.hub || 'opensearchproject' }}
       OPENSEARCH_VERSION: ${{ matrix.entry.version }}
-      OPENSEARCH_PASSWORD: myStrongPassword123!
+      OPENSEARCH_PASSWORD: ${{ matrix.entry.admin_password || 'myStrongPassword123!' }}
+
     steps:
       - name: Checkout Repo
         uses: actions/checkout@v4

diff --git a/.github/workflows/test-tools-integ.yml b/.github/workflows/test-tools-integ.yml
@@ -24,7 +24,7 @@ jobs:
   test:
     runs-on: ubuntu-latest
     env:
-      OPENSEARCH_VERSION: 2.12.0
+      OPENSEARCH_VERSION: 2.15.0
       OPENSEARCH_PASSWORD: myStrongPassword123!
       OPENSEARCH_URL: https://localhost:9200
     steps:
@@ -35,7 +35,7 @@ jobs:
         working-directory: .github/opensearch-cluster
         run: |
           docker-compose up -d
-          sleep 30
+          sleep 15
 
       - name: Setup Node.js
         uses: actions/setup-node@v3

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -41,6 +41,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Added `remote_store` to `TranslogStats` ([#395](https://github.com/opensearch-project/opensearch-api-specification/pull/395))
 - Added `file` to `/_cache/clear` and `/{index}/_cache/clear` ([#396](https://github.com/opensearch-project/opensearch-api-specification/pull/396))
 - Added a workflow to run tests against the next version of OpenSearch ([#409](https://github.com/opensearch-project/opensearch-api-specification/pull/409))
+- Added support for skipping tests using semver range ([#410](https://github.com/opensearch-project/opensearch-api-specification/pull/410))
 
 ### Changed
 

diff --git a/SPECIFICATION_TESTING.md b/SPECIFICATION_TESTING.md
@@ -1,9 +1,10 @@
 <!-- TOC -->
-* [Specification Testing](#specification-testing)
-  * [Running Spec Tests Locally](#running-spec-tests-locally)
-  * [Writing Spec Tests](#writing-spec-tests)
-    * [Simple Test Story](#simple-test-story)
-    * [Using Output from Previous Chapters](#using-output-from-previous-chapters)
+- [Specification Testing](#specification-testing)
+  - [Running Spec Tests Locally](#running-spec-tests-locally)
+  - [Writing Spec Tests](#writing-spec-tests)
+    - [Simple Test Story](#simple-test-story)
+    - [Using Output from Previous Chapters](#using-output-from-previous-chapters)
+    - [Managing Versions](#managing-versions)
 <!-- TOC -->
 
 # Specification Testing
@@ -133,4 +134,22 @@ Consider the following chapters in [ml/model_groups](tests/ml/model_groups.yaml)
     response:
       status: 200
 ```
-As you can see, the `output` field in the first chapter saves the `model_group_id` from the response body. This value is then used in the subsequent chapters to query and delete the model group.
+As you can see, the `output` field in the first chapter saves the `model_group_id` from the response body. This value is then used in the subsequent chapters to query and delete the model group.
+
+### Managing Versions
+
+It's common to add a feature to the next version of OpenSearch. When adding a new API in the spec, make sure to specify `x-version-added`, `x-version-deprecated` or `x-version-removed`. Finally, specify a semver range in your test stories or chapters as follows.
+
+```yaml
+- synopsis: Search with `phase_took` added in OpenSearch 2.12.
+  version: '>= 2.12'
+  path: /{index}/_search
+  parameters:
+    index: movies
+    cancel_after_time_interval: 10s
+  method: POST
+  response:
+    status: 200
+```
+
+The [integration test workflow](.github/workflows/test-spec.yml) runs a matrix of OpenSearch versions, including the next version. Please check whether the workflow needs an update when adding version-specific tests.
diff --git a/json_schemas/_info.schema.yaml b/json_schemas/_info.schema.yaml
@@ -38,6 +38,8 @@ properties:
       - name
   version:
     type: string
+  x-api-version:
+    type: string
 required:
   - title
   - version
diff --git a/json_schemas/test_story.schema.yaml b/json_schemas/test_story.schema.yaml
@@ -18,6 +18,8 @@ properties:
     type: array
     items:
       $ref: '#/definitions/Chapter'
+  version:
+    $ref: '#/definitions/Version'
 required: [description, chapters]
 additionalProperties: false
 
@@ -76,6 +78,8 @@ definitions:
         $ref: '#/definitions/RequestBody'
       output:
         $ref: '#/definitions/Output'
+      version:
+        $ref: '#/definitions/Version'
     required: [path, method]
 
   Output:
@@ -90,6 +94,11 @@ definitions:
     additionalProperties:
       type: string
 
+  Version:
+    description: |
+      The semver range to execute the story or chapter against.
+    type: string
+
   RequestBody:
     type: object
     properties:

diff --git a/spec/_info.yaml b/spec/_info.yaml
@@ -2,3 +2,4 @@ $schema: ./json_schemas/_info.schema.yaml
 
 title: OpenSearch API Specification
 version: 1.0.0
+x-api-version: 2.15.0
diff --git a/spec/namespaces/indices.yaml b/spec/namespaces/indices.yaml
@@ -2766,6 +2766,7 @@ components:
         $ref: '../schemas/_common.yaml#/components/schemas/Fields'
       style: form
     indices.clear_cache::query.file:
+      x-version-added: '2.8'
       in: query
       name: file
       description: If true, clears the unused entries from the file cache on nodes with the Search role.
@@ -2865,6 +2866,7 @@ components:
         $ref: '../schemas/_common.yaml#/components/schemas/WaitForActiveShards'
       style: form
     indices.clone::query.wait_for_completion:
+      x-version-added: '2.7'
       name: wait_for_completion
       in: query
       description: Should this request wait until the operation has completed before returning.
@@ -3515,6 +3517,7 @@ components:
         type: boolean
       style: form
     indices.forcemerge::query.primary_only:
+      x-version-added: '2.13'
       name: primary_only
       in: query
       description: Specify whether the operation should only perform on primary shards. Defaults to false.
@@ -3523,6 +3526,7 @@ components:
         default: false
         description: Specify whether the operation should only perform on primary shards. Defaults to false.
     indices.forcemerge::query.wait_for_completion:
+      x-version-added: '2.7'
       in: query
       name: wait_for_completion
       description: Should the request wait until the force merge is completed.
@@ -4098,6 +4102,7 @@ components:
         $ref: '../schemas/_common.yaml#/components/schemas/WaitForActiveShards'
       style: form
     indices.open::query.wait_for_completion:
+      x-version-added: '2.7'
       name: wait_for_completion
       in: query
       description: Should this request wait until the operation has completed before returning.
@@ -4708,6 +4713,7 @@ components:
         $ref: '../schemas/_common.yaml#/components/schemas/WaitForActiveShards'
       style: form
     indices.shrink::query.wait_for_completion:
+      x-version-added: '2.7'
       name: wait_for_completion
       in: query
       description: Should this request wait until the operation has completed before returning.
@@ -4874,6 +4880,7 @@ components:
         $ref: '../schemas/_common.yaml#/components/schemas/WaitForActiveShards'
       style: form
     indices.split::query.wait_for_completion:
+      x-version-added: '2.7'
       name: wait_for_completion
       in: query
       description: Should this request wait until the operation has completed before returning.
@@ -5036,6 +5043,7 @@ components:
         type: boolean
         description: If true, only ancient (an older Lucene major release) segments will be upgraded.
     indices.upgrade::query.wait_for_completion:
+      x-version-added: '2.7'
       name: wait_for_completion
       in: query
       description: Should this request wait until the operation has completed before returning.

diff --git a/spec/schemas/_common.yaml b/spec/schemas/_common.yaml
@@ -1273,6 +1273,7 @@ components:
         licensed:
           type: boolean
         custom_foldername:
+          nullable: true
           type: string
         opensearch_version:
           $ref: '#/components/schemas/VersionString'

diff --git a/spec/schemas/_core.search.yaml b/spec/schemas/_core.search.yaml
@@ -852,6 +852,7 @@ components:
         _shards:
           $ref: '_common.yaml#/components/schemas/ShardStatistics'
         phase_took:
+          x-version-added: '2.12.0'
           $ref: '_common.yaml#/components/schemas/PhaseTook'
         hits:
           $ref: '#/components/schemas/HitsMetadata'

diff --git a/tests/_core/search.yaml b/tests/_core/search.yaml
@@ -87,11 +87,19 @@ chapters:
                 director: Bennett Miller
                 title: Moneyball
                 year: 2011
-  - synopsis: Search with parameters.
+  - synopsis: Search with cancel_after_time_interval.
     path: /{index}/_search
     parameters:
       index: movies
       cancel_after_time_interval: 10s
+    method: POST
+    response:
+      status: 200
+  - synopsis: Search with phase_took.
+    version: '>= 2.12'
+    path: /{index}/_search
+    parameters:
+      index: movies
       phase_took: true
     method: POST
     response:

diff --git a/tests/cat/fielddata.yaml b/tests/cat/fielddata.yaml
@@ -2,12 +2,20 @@ $schema: ../../json_schemas/test_story.schema.yaml
 
 description: Test cat/fielddata endpoints.
 chapters:
-  - synopsis: Cat with a json response.
+  - synopsis: Cat with an empty response.
+    path: /_cat/fielddata
+    method: GET
+    parameters:
+      format: json
+    response:
+      status: 200
+  - synopsis: Cat with a json response (from security-analytics).
+    version: '>= 2.4'
     path: /_cat/fielddata
     method: GET
     parameters:
       format: json
     response:
       status: 200
       payload:
-        - field: log_types # from security-analytics
+        - field: log_types
diff --git a/tests/cat/pit_segments.yaml b/tests/cat/pit_segments.yaml
@@ -1,6 +1,7 @@
 $schema: ../../json_schemas/test_story.schema.yaml
 
 description: Test cat/pit_segments endpoints.
+version: '>= 2.4'
 epilogues:
   - path: /games
     method: DELETE

diff --git a/tests/cat/segment_replication.yaml b/tests/cat/segment_replication.yaml
@@ -1,6 +1,7 @@
 $schema: ../../json_schemas/test_story.schema.yaml
 
 description: Test cat/segment_replication endpoints.
+version: '>= 2.7'
 chapters:
   - synopsis: Cat with a json response.
     path: /_cat/segment_replication

diff --git a/tests/indices/cache.yaml b/tests/indices/cache.yaml
@@ -22,7 +22,6 @@ chapters:
       expand_wildcards: none
       fielddata: true
       fields: '*'
-      file: false
       index:
         - movies
       ignore_unavailable: true
@@ -32,6 +31,16 @@ chapters:
       status: 200
       payload:
         _shards: {}
+  - synopsis: Clear all cache, file=true (POST).
+    version: '>= 2.8'
+    path: /_cache/clear
+    method: POST
+    parameters:
+      file: true
+    response:
+      status: 200
+      payload:
+        _shards: {}
   - synopsis: Clear index cache (POST).
     path: /{index}/_cache/clear
     method: POST

diff --git a/tests/indices/clone.yaml b/tests/indices/clone.yaml
@@ -8,7 +8,10 @@ epilogues:
   - path: /movies
     method: DELETE
     status: [200, 404]
-  - path: /games
+  - path: /games1
+    method: DELETE
+    status: [200, 404]
+  - path: /games2
     method: DELETE
     status: [200, 404]
 chapters:
@@ -26,14 +29,23 @@ chapters:
     method: POST
     parameters:
       index: movies
-      target: games
+      target: games1
       wait_for_active_shards: 1
       cluster_manager_timeout: 10s
       timeout: 10s
-      wait_for_completion: true
     response:
       status: 200
       payload:
         shards_acknowledged: true
         acknowledged: true
-        index: games
+        index: games1
+  - synopsis: Clone an index (wait_for_completion).
+    version: '>= 2.7'
+    path: /{index}/_clone/{target}
+    method: POST
+    parameters:
+      index: movies
+      target: games2
+      wait_for_completion: true
+    response:
+      status: 200
diff --git a/tests/indices/forcemerge.yaml b/tests/indices/forcemerge.yaml
@@ -21,12 +21,27 @@ chapters:
       flush: true
       max_num_segments: 1
       only_expunge_deletes: true
-      primary_only: false
       ignore_unavailable: true
     response:
       status: 200
       payload:
         _shards: {}
+  - synopsis: Force merge all indexes (primary_only).
+    version: '>= 2.13'
+    path: /_forcemerge
+    method: POST
+    parameters:
+      primary_only: true
+    response:
+      status: 200
+  - synopsis: Force merge all indexes (wait_for_completion).
+    version: '>= 2.7'
+    path: /_forcemerge
+    method: POST
+    parameters:
+      wait_for_completion: true
+    response:
+      status: 200
   - synopsis: Force merge an index.
     path: /{index}/_forcemerge
     method: POST

diff --git a/tests/indices/open.yaml b/tests/indices/open.yaml
@@ -30,25 +30,18 @@ chapters:
       wait_for_active_shards: 0
       cluster_manager_timeout: 30s
       timeout: 30s
-      wait_for_completion: true
     response:
       status: 200
       payload:
         shards_acknowledged: true
         acknowledged: true
   - synopsis: Open a closed index (wait_for_completion=false).
+    version: '>= 2.7'
     path: /{index}/_open
     method: POST
     parameters:
       index: movies
-      allow_no_indices: true
-      expand_wildcards: all
-      ignore_unavailable: true
-      wait_for_active_shards: 0
-      cluster_manager_timeout: 30s
-      timeout: 30s
       wait_for_completion: false
-      task_execution_timeout: 1m
     response:
       status: 200
   - synopsis: Open an already opened index.
Original file line number	Diff line number	Diff line change
Expand Up		@@ -2,3 +2,4 @@ $schema: ./json_schemas/_info.schema.yaml

		title: OpenSearch API Specification
		version: 1.0.0
		x-api-version: 2.15.0