From 7bf34a41c78a51433f628b54588d094afdbdb345 Mon Sep 17 00:00:00 2001 From: "Daniel (dB.) Doubrovkine" Date: Thu, 24 Oct 2024 16:52:01 -0400 Subject: [PATCH 1/3] Documented docker container testing. (#642) Signed-off-by: dblock --- TESTING_GUIDE.md | 131 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 108 insertions(+), 23 deletions(-) diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 65312657e..ea3867c1a 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -1,27 +1,31 @@ -* [Spec Testing Guide](#spec-testing-guide) - * [Running Spec Tests](#running-spec-tests) - * [Running Spec Tests Locally](#running-spec-tests-locally) - * [Prerequisites](#prerequisites-) - * [OpenSearch Cluster](#opensearch-cluster-) - * [Run Tests](#run-tests) - * [Running Spec Tests with Amazon OpenSearch](#running-spec-tests-with-amazon-opensearch) - * [Common Errors](#common-errors) - * [401 Unauthorized](#401-unauthorized) - * [FORBIDDEN/10/cluster create-index blocked (api)](#forbidden10cluster-create-index-blocked-api) - * [FAILED Cat with a json response (from security-analytics).](#failed--cat-with-a-json-response-from-security-analytics) - * [Writing Spec Tests](#writing-spec-tests) - * [Simple Test Story](#simple-test-story) - * [Using Output from Previous Chapters](#using-output-from-previous-chapters) - * [Managing Versions](#managing-versions) - * [Managing Distributions](#managing-distributions) - * [Waiting for Tasks](#waiting-for-tasks) - * [Warnings](#warnings) - * [multiple-paths-detected](#multiple-paths-detected) - * [Suppressing Warnings](#suppressing-warnings) - * [Collecting Test Coverage](#collecting-test-coverage) - * [Coverage Summary](#coverage-summary) - * [Coverage Report](#coverage-report) +- [Spec Testing Guide](#spec-testing-guide) + - [Running Spec Tests](#running-spec-tests) + - [Running Spec Tests Locally](#running-spec-tests-locally) + - [Prerequisites](#prerequisites) + - [OpenSearch Cluster](#opensearch-cluster) + - [Run Tests](#run-tests) + - [Running Spec Tests with Amazon OpenSearch](#running-spec-tests-with-amazon-opensearch) + - [Common Errors](#common-errors) + - [401 Unauthorized](#401-unauthorized) + - [FORBIDDEN/10/cluster create-index blocked (api)](#forbidden10cluster-create-index-blocked-api) + - [FAILED Cat with a json response (from security-analytics).](#failed--cat-with-a-json-response-from-security-analytics) + - [Writing Spec Tests](#writing-spec-tests) + - [Simple Test Story](#simple-test-story) + - [Using Output from Previous Chapters](#using-output-from-previous-chapters) + - [Managing Versions](#managing-versions) + - [Managing Distributions](#managing-distributions) + - [Waiting for Tasks](#waiting-for-tasks) + - [Warnings](#warnings) + - [multiple-paths-detected](#multiple-paths-detected) + - [Suppressing Warnings](#suppressing-warnings) + - [Collecting Test Coverage](#collecting-test-coverage) + - [Coverage Summary](#coverage-summary) + - [Coverage Report](#coverage-report) + - [Integration Testing](#integration-testing) + - [Stable Releases](#stable-releases) + - [Custom Setup](#custom-setup) + - [Future Releases](#future-releases) # Spec Testing Guide @@ -375,4 +379,85 @@ The test tool can display detailed and hierarchal test coverage with `--coverage GET /_alias/{name} POST /_alias/{name} HEAD /_alias/{name} +``` + +## Integration Testing + +This project runs integration tests against multiple versions of OpenSearch in the [test-spec.yml](.github/workflows/test-spec.yml) workflow. + +### Stable Releases + +The simplest entry in the test matrix executes tests in [tests/default](tests/default) against a released version of OpenSearch. +For example, the following entries run tests gainst OpenSearch 1.3.17 and 2.17.0. + +```yaml +entry: + - version: 1.3.17 + - version: 2.17.0 +``` + +### Custom Setup + +Some tests require a custom docker image. For example, testing the notifications plugin requires a custom webhook. This can be done by creating a custom `docker-compose.yml`, such as the one in [tests/plugins/notifications](tests/plugins/notifications/docker-compose.yml). + +```yaml +webhook: + image: python:latest + volumes: + - ./server.py:/server.py + ports: + - '8080:8080' + entrypoint: python server.py +``` + +The following example in the test matrix will use the custom `docker-compose.yml` and execute all tests in [tests/plugins/notifications](tests/plugins/notifications). + +```yaml +entry: + - version: 2.17.0 + tests: plugins/notifications +``` + +### Future Releases + +Snapshot builds of OpenSearch are available on Docker Hub under [opensearchstaging/opensearch/tags](https://hub.docker.com/r/opensearchstaging/opensearch/tags). + +The following example in the test matrix will use [a snapshot build of OpenSearch 2.18](https://hub.docker.com/layers/opensearchstaging/opensearch/2.18.0/images/sha256-504a9c42bc1b13cb47b39a29db8a9d300d01b8851fb95dbb9db6770f478e45b5?context=explore) to execute the default test suite in [tests/default](tests/default/). + +```yaml +- version: 2.18.0 + hub: opensearchstaging + ref: '@sha256:4445e195c53992038891519dc3be0d273cdaad1b047943d68921168ed243e7e9' +``` + +It's important to note that snapshot builds may not contain all plugins, and may contain previous versions of a plugin if the current code failed to build. It's therefore possible that updating a SHA to test new functionality available in a more recent build causes failures with existing tests. As of today the only workaround is to try the next build that will hopefully have more/all the plugins. For a discussion about this problem see [opensearch-build#5130](https://github.com/opensearch-project/opensearch-build/issues/5130). + +Use the following command to retrieve the manifest of a given build to make it easier to identify a SHA that includes all the plugins. + +```bash +$ docker run -it --entrypoint bash opensearchstaging/opensearch:2.18.0@sha256:4445e195c53992038891519dc3be0d273cdaad1b047943d68921168ed243e7e9 -c "cat /usr/share/opensearch/manifest.yml" +``` + +```yaml +--- +schema-version: '1.1' +build: + name: OpenSearch + version: 2.18.0 + platform: linux + architecture: x64 + distribution: tar + location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.18.0/10320/linux/x64/tar/dist/opensearch/opensearch-2.18.0-linux-x64.tar.gz + id: '10320' +components: + - name: OpenSearch + repository: https://github.com/opensearch-project/OpenSearch.git + ref: 2.x + commit_id: b67f76541b78e58844b54305eb21232e78167744 + location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.18.0/10320/linux/x64/tar/builds/opensearch/dist/opensearch-min-2.18.0-linux-x64.tar.gz + - name: common-utils + repository: https://github.com/opensearch-project/common-utils.git + ref: 2.x + commit_id: 63ee9746ce04a8eace0ba6320e93bf5833f6a4a6 +... ``` \ No newline at end of file From 82e37830ef2324c46365bf5d53eb735c151e1a4a Mon Sep 17 00:00:00 2001 From: "Daniel (dB.) Doubrovkine" Date: Thu, 24 Oct 2024 16:52:46 -0400 Subject: [PATCH 2/3] Fix: model training intermittent test failure. (#644) Signed-off-by: dblock --- spec/namespaces/knn.yaml | 6 +++++- tests/default/knn/train_model.yaml | 19 ++++++++++++++++++- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/spec/namespaces/knn.yaml b/spec/namespaces/knn.yaml index c38bba5b7..77b6a34cb 100644 --- a/spec/namespaces/knn.yaml +++ b/spec/namespaces/knn.yaml @@ -283,7 +283,11 @@ components: required: true responses: knn.delete_model@200: {} - knn.get_model@200: {} + knn.get_model@200: + content: + application/json: + schema: + type: object knn.search_models@200: {} knn.stats@200: {} knn.train_model@200: diff --git a/tests/default/knn/train_model.yaml b/tests/default/knn/train_model.yaml index b45ac5c6f..cb58ed97c 100644 --- a/tests/default/knn/train_model.yaml +++ b/tests/default/knn/train_model.yaml @@ -41,6 +41,9 @@ prologues: - {index: {_index: movies, _id: '9'}} - {recommendation_vector: [9.5, 9.5, 9.5, 9.5, 9.5, 9.5, 9.5, 9.5], duration: 8.9} status: [200] + - method: POST + path: /_refresh + status: [200] epilogues: - path: /movies method: DELETE @@ -70,4 +73,18 @@ chapters: response: status: 200 output: - test_model_id: payload.model_id \ No newline at end of file + test_model_id: payload.model_id + - synopsis: Wait for the model to get trained. + warnings: + multiple-paths-detected: false + method: GET + path: /_plugins/_knn/models/{model_id} + parameters: + model_id: ${train_model.test_model_id} + retry: + count: 3 + response: + status: 200 + payload: + model_id: ${train_model.test_model_id} + state: created From 110474fbd4eac56318fcea220ff9a6ebcd615d04 Mon Sep 17 00:00:00 2001 From: "Daniel (dB.) Doubrovkine" Date: Thu, 24 Oct 2024 16:53:53 -0400 Subject: [PATCH 3/3] Updated 2.18 to RC2. (#643) * Updated 2.18 to RC2. Signed-off-by: dblock --- .github/workflows/test-spec.yml | 2 +- CHANGELOG.md | 1 + spec/schemas/nodes._common.yaml | 8 ++++++++ 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test-spec.yml b/.github/workflows/test-spec.yml index 121632390..23f0fffc6 100644 --- a/.github/workflows/test-spec.yml +++ b/.github/workflows/test-spec.yml @@ -44,7 +44,7 @@ jobs: tests: plugins/query_insights - version: 2.18.0 hub: opensearchstaging - ref: '@sha256:4445e195c53992038891519dc3be0d273cdaad1b047943d68921168ed243e7e9' + ref: '@sha256:57a1cd1142d68c203e2e4aa0666d9a691e1e409a5d14aa4a8f5044036f05cf06' - version: 3.0.0 hub: opensearchstaging ref: '@sha256:727643acdfebed77bfdb26362dbcff536b7ea02a0cc4ae2da2521729171333de' diff --git a/CHANGELOG.md b/CHANGELOG.md index fcb22f02d..dbf448b1e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -105,6 +105,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) - Added a release workflow triggered on a tag ([#635](https://github.com/opensearch-project/opensearch-api-specification/pull/635)) - Added API spec for query insights plugin ([#625](https://github.com/opensearch-project/opensearch-api-specification/pull/625)) - Added API spec for `adjust_pure_negative` for bool queries ([#641](https://github.com/opensearch-project/opensearch-api-specification/pull/641)) +- Added `remote_store` to node `Stats` ([#643](https://github.com/opensearch-project/opensearch-api-specification/pull/643)) ### Changed diff --git a/spec/schemas/nodes._common.yaml b/spec/schemas/nodes._common.yaml index 9ba3c0621..4c071b743 100644 --- a/spec/schemas/nodes._common.yaml +++ b/spec/schemas/nodes._common.yaml @@ -276,12 +276,20 @@ components: $ref: '#/components/schemas/ShardSearchPipelineStats' segment_replication_backpressure: $ref: '#/components/schemas/ShardSegmentReplicationBackpressureStats' + remote_store: + $ref: '#/components/schemas/RemoteStoreStats' repositories: $ref: '#/components/schemas/ShardRepositoriesStats' admission_control: $ref: '#/components/schemas/ShardAdmissionControlStats' caches: $ref: '#/components/schemas/ShardCachesStats' + RemoteStoreStats: + type: object + properties: + last_successful_fetch_of_pinned_timestamps: + description: Timestamp for the last successful fetch of pinned timestamps. + $ref: '_common.yaml#/components/schemas/StringifiedEpochTimeUnitSeconds' ShardClusterManagerThrottlingStats: type: object properties: