Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/main' into security-tests
Browse files Browse the repository at this point in the history
  • Loading branch information
DarshitChanpura committed Jul 18, 2024
2 parents f5220c1 + e1cd369 commit cd4709b
Show file tree
Hide file tree
Showing 49 changed files with 1,459 additions and 135 deletions.
2 changes: 1 addition & 1 deletion .github/opensearch-cluster/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: '3'

services:
opensearch-cluster:
image: ${OPENSEARCH_DOCKER_HUB_PROJECT:-opensearchproject}/opensearch:${OPENSEARCH_VERSION:-latest}
image: ${OPENSEARCH_DOCKER_HUB_PROJECT:-opensearchproject}/opensearch:${OPENSEARCH_VERSION:-latest}${OPENSEARCH_DOCKER_REF}
ports:
- '9200:9200'
- '9600:9600'
Expand Down
6 changes: 5 additions & 1 deletion .github/workflows/test-spec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,15 @@ on:
- .github/workflows/test-spec.yml
- package*.json
- spec/**
- tests/**
- tools/src/tester/**
- tsconfig.json
pull_request:
paths:
- .github/workflows/test-spec.yml
- package*.json
- spec/**
- tests/**
- tools/src/tester/**
- tsconfig.json

Expand All @@ -21,15 +23,17 @@ jobs:
strategy:
matrix:
entry:
- {version: 1.3.17, admin_password: admin}
- {version: 2.0.0, admin_password: admin}
- {version: 2.15.0}
- {version: 2.16.0, hub: opensearchstaging}
- {version: 2.16.0, hub: opensearchstaging, ref: '@sha256:aa99ce3e2c9c94cd934cf98bab753a9920bfdbbb74bdc932d1b443d31c18d6b2'}

name: test-opensearch-spec (version=${{ matrix.entry.version }}, hub=${{ matrix.entry.hub || 'opensearchproject' }})
runs-on: ubuntu-latest

env:
OPENSEARCH_DOCKER_HUB_PROJECT: ${{ matrix.entry.hub || 'opensearchproject' }}
OPENSEARCH_DOCKER_REF: ${{ matrix.entry.ref }}
OPENSEARCH_VERSION: ${{ matrix.entry.version }}
OPENSEARCH_PASSWORD: ${{ matrix.entry.admin_password || 'myStrongPassword123!' }}

Expand Down
18 changes: 14 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Added tests for response payload ([#347](https://github.com/opensearch-project/opensearch-api-specification/pull/347))
- Added `cancel_after_time_interval` and `phase_took` in `_search` ([#353](https://github.com/opensearch-project/opensearch-api-specification/pull/353))
- Added support for testing `application/x-ndjson` payloads ([#355](https://github.com/opensearch-project/opensearch-api-specification/pull/355))
- Added SPECIFICATION_TESTING.md [#359](https://github.com/opensearch-project/opensearch-api-specification/pull/359)
- Added TESTING_GUIDE.md ([#359](https://github.com/opensearch-project/opensearch-api-specification/pull/359))
- Added StoryValidator to validate stories before running them ([#354](https://github.com/opensearch-project/opensearch-api-specification/issues/354))
- Added support for `text/plain` responses in `_cat` APIs ([#360](https://github.com/opensearch-project/opensearch-api-specification/pull/360))
- Added support for `application/yaml` responses ([#363](https://github.com/opensearch-project/opensearch-api-specification/pull/363))
Expand All @@ -42,16 +42,24 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Added `file` to `/_cache/clear` and `/{index}/_cache/clear` ([#396](https://github.com/opensearch-project/opensearch-api-specification/pull/396))
- Added a workflow to run tests against the next version of OpenSearch ([#409](https://github.com/opensearch-project/opensearch-api-specification/pull/409))
- Added support for skipping tests using semver range ([#410](https://github.com/opensearch-project/opensearch-api-specification/pull/410))
- Added `cluster_manager_timeout` to `HEAD /{index}` ([#421](https://github.com/opensearch-project/opensearch-api-specification/pull/421))
- Added missing fields to `/_nodes/stats` ([#415](https://github.com/opensearch-project/opensearch-api-specification/pull/415))
- Added missing metrics options to `/_nodes/stats` ([#422](https://github.com/opensearch-project/opensearch-api-specification/pull/422))
- Added tests against OpenSearch 1.3 ([#424](https://github.com/opensearch-project/opensearch-api-specification/pull/424))
- Added `is_hidden` to `/{index}/_alias/{name}` and `/{index}/_aliases/{name}` ([#429](https://github.com/opensearch-project/opensearch-api-specification/pull/429))
- Added `ignore_unmapped` to `GeoDistanceQuery` ([#427](https://github.com/opensearch-project/opensearch-api-specification/pull/427))
- Added missing variants of `indices.put_alias` ([#434](https://github.com/opensearch-project/opensearch-api-specification/pull/434))

### Changed

- Replaced Smithy with a native OpenAPI spec ([#189](https://github.com/opensearch-project/opensearch-api-specification/issues/189))
- Refactored spec tester internals to improve reusability ([#302](https://github.com/opensearch-project/opensearch-api-specification/pull/302))
- Renamed `main` release tag to `main-latest` ([#321](https://github.com/opensearch-project/opensearch-api-specification/pull/321))
- Replaced usages of `Opensearch` with `OpenSearch` ([#335](https://github.com/opensearch-project/opensearch-api-specification/pull/335))
- Prevented merger tool from printing warnings when used by tester tool [#359](https://github.com/opensearch-project/opensearch-api-specification/pull/359)
- Replaced the deprecated fs.rmdirSync with fs.rmSync [#359](https://github.com/opensearch-project/opensearch-api-specification/pull/359)
- Tester tool now provides better context for non-2XX responses when --verbose is used [#359](https://github.com/opensearch-project/opensearch-api-specification/pull/359)
- Prevented merger tool from printing warnings when used by tester tool ([#359](https://github.com/opensearch-project/opensearch-api-specification/pull/359))
- Replaced the deprecated fs.rmdirSync with fs.rmSync ([#359](https://github.com/opensearch-project/opensearch-api-specification/pull/359))
- Tester tool now provides better context for non-2XX responses when --verbose is used ([#359](https://github.com/opensearch-project/opensearch-api-specification/pull/359))
- Lock testing for next release of OpenSearch to a specific SHA ([#431](https://github.com/opensearch-project/opensearch-api-specification/pull/431))

### Deprecated

Expand All @@ -77,6 +85,8 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Fixed `/_data_stream` health status and required fields ([#401](https://github.com/opensearch-project/opensearch-api-specification/pull/401))
- Fixed query DSL `match` that supports a field name and value ([#405](https://github.com/opensearch-project/opensearch-api-specification/pull/405))
- Fixed `/_mapping` with `index` in query ([#385](https://github.com/opensearch-project/opensearch-api-specification/pull/385))
- Fixed duplicate `/_nodes/{node_id}` path ([#416](https://github.com/opensearch-project/opensearch-api-specification/pull/416))
- Fixed `_source` accepting an array of fields in `/_search` ([#430](https://github.com/opensearch-project/opensearch-api-specification/pull/430))

### Security

Expand Down
30 changes: 29 additions & 1 deletion CLIENT_GENERATOR_GUIDE.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,12 @@
- [Generate Clients for OpenSearch using OpenAPI Specification](#generate-clients-for-opensearch-using-openapi-specification)
- [The Grouping of API Operations](#the-grouping-of-api-operations)
- [Overloaded Name](#overloaded-name)
- [Handling Bulk Operations](#handling-bulk-operations)
- [Parameter Validation](#parameter-validation)
- [Global Parameters](#global-parameters)
- [Default Parameter Values](#default-parameter-values)
- [Duplicate Paths](#duplicate-paths)

# Generate Clients for OpenSearch using OpenAPI Specification

OpenSearch Clients are available in multiple programming languages. The biggest challenge with this is keeping the clients up to date with the latest changes in OpenSearch. To solve this problem, we're automating the process of generating clients for OpenSearch using the OpenAPI specification. While OpenAPI comes with many well established off-the-shelf client generators for most languages, the OpenSearch APIs come with a lot of quirkiness that makes it near impossible to use these off-the-shelf generators. For this reason, we've opted to write our own client generators that is specifically tailored to the OpenSearch APIs. This document will walk you through the process of generating clients from [the published OpenSearch OpenAPI spec](https://github.com/opensearch-project/opensearch-api-specification/releases), more specifically client API methods.
Expand Down Expand Up @@ -58,4 +67,23 @@ Some clients also check for the validity of query string parameter names to guar
All operations in the spec contain a set of parameters that are common across all operations. These parameters are denoted with `x-global: true` vendor extension. The generated clients should find a way to DRY these parameters in type definitions and method documentation.

## Default Parameter Values
Parameters can have default values either through schema or the `x-default` vendor extension. When both are present, `x-default` will take precedence.
Parameters can have default values either through schema or the `x-default` vendor extension. When both are present, `x-default` will take precedence.

## Duplicate Paths
OpenAPI does not allow duplicate paths, which makes it challenging to express certain OpenSearch APIs, such as `/_nodes/{node_id}` and `/_nodes/{metric}`. In this example the server expects either a `node_id` or a `metric`. The API specification handles this by defining a `/_nodes/{node_id_or_metric}` path, and decorating the `node_id_or_metric` type with a `title` field.

```yaml
name: node_id_or_metric
schema:
anyOf:
- title: node_id
$ref: '../schemas/_common.yaml#/components/schemas/NodeIds'
- title: metric
type: array
items:
$ref: '../schemas/nodes.info.yaml#/components/schemas/Metric'
```
See [#416](https://github.com/opensearch-project/opensearch-api-specification/pull/416) for a complete example.
Clients should generate multiple methods or a method that accepts multiple parameters in this case.
2 changes: 1 addition & 1 deletion DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ This repository includes several OpenAPI Specification Extensions to fill in any

## Writing Spec Tests

To assure the correctness of the spec, you must add tests for the spec, when making changes. Check [SPECIFICATION_TESTING.md](SPECIFICATION_TESTING.md) for more information.
To assure the correctness of the spec, you must add tests for the spec, when making changes. Check [TESTING_GUIDE.md](TESTING_GUIDE.md) for more information.

## Tools

Expand Down
File renamed without changes.
29 changes: 7 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,25 +12,25 @@
- [Welcome!](#welcome)
- [Project Resources](#project-resources)
- [Code of Conduct](#code-of-conduct)
- [Developer Guide](#developer-guide)
- [Client Generator Guide](#client-generator-guide)
- [Published Spec](#published-spec)
- [Security](#security)
- [License](#license)
- [Copyright](#copyright)

## Welcome!

`opensearch-api-specification` is an open source, community-driven collection of API model specifications for
[OpenSearch](https://github.com/opensearch-project/OpenSearch) APIs. The API models are written in OpenAPI format and are used to generate client libraries and documentation.
The `opensearch-api-specification` is an open source, community-driven collection of API model specifications for [OpenSearch](https://github.com/opensearch-project/OpenSearch) APIs. The API models are written in OpenAPI format and are used to generate client libraries and documentation. You can find the latest version of the API specification [here](https://github.com/opensearch-project/opensearch-api-specification/releases/download/main-latest/opensearch-openapi.yaml).

To contribute to this project or to track the developments head over to [Projects](https://github.com/opensearch-project/opensearch-api-specification/projects)
board. Follow the [Developer guide](DEVELOPER_GUIDE.md) and [Contributing guidelines](CONTRIBUTING.md) for instructions
To contribute to this project or to track the developments head over to [Projects](https://github.com/opensearch-project/opensearch-api-specification/projects) board. Follow the [Developer guide](DEVELOPER_GUIDE.md) and [Contributing guidelines](CONTRIBUTING.md) for instructions
on building and contributing to opensearch-api-specification.

## Project Resources

* [Current Release](https://github.com/opensearch-project/opensearch-api-specification/releases/download/main-latest/opensearch-openapi.yaml)
* [Developer Guide](DEVELOPER_GUIDE.md).
* [Client Generator Guide](CLIENT_GENERATOR_GUIDE.md).
* [Spec Publishing Guide](PUBLISHING_GUIDE.md).
* [Project Website](https://opensearch.org/)
* [API Playground](https://opensearch-project.github.io/opensearch-api-specification/)
* [Downloads](https://opensearch.org/downloads.html)
* [Documentation](https://opensearch.org/docs/)
* Need help? Try [Forums](https://forum.opensearch.org/)
Expand All @@ -41,21 +41,6 @@ on building and contributing to opensearch-api-specification.

This project has adopted the [Amazon Open Source Code of Conduct](CODE_OF_CONDUCT.md). For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq), or contact [[email protected]](mailto:[email protected]) with any additional questions or comments.

## Developer Guide

See [DEVELOPER_GUIDE](DEVELOPER_GUIDE.md).

## Client Generator Guide

See [CLIENT_GENERATOR_GUIDE](CLIENT_GENERATOR_GUIDE.md).

## Published Spec

OpenSearch API Specs are hosted at https://opensearch-project.github.io/opensearch-api-specification/. See [PUBLISHING_SPECS](PUBLISHING_SPECS.md) for more information.

Click [here](https://github.com/opensearch-project/opensearch-api-specification/releases/download/main-latest/opensearch-openapi.yaml) to download the latest OpenSearch OpenAPI yaml file.


## Security

If you discover a potential security issue in this project we ask that you notify OpenSearch Security directly via email to [email protected]. Please do **not** create a public GitHub issue.
Expand Down
4 changes: 2 additions & 2 deletions SPECIFICATION_TESTING.md → TESTING_GUIDE.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
<!-- TOC -->
- [Specification Testing](#specification-testing)
- [Spec Testing Guide](#spec-testing-guide)
- [Running Spec Tests Locally](#running-spec-tests-locally)
- [Writing Spec Tests](#writing-spec-tests)
- [Simple Test Story](#simple-test-story)
- [Using Output from Previous Chapters](#using-output-from-previous-chapters)
- [Managing Versions](#managing-versions)
<!-- TOC -->

# Specification Testing
# Spec Testing Guide

We have devised our own test framework to test the spec against an OpenSearch cluster. We're still adding more features to the framework as the needs arise, and this document will be updated accordingly. This test framework has also been integrated into the repo's CI/CD pipeline. Checkout the [test-spec](.github/workflows/test-spec.yml) workflow for more details.

Expand Down
Loading

0 comments on commit cd4709b

Please sign in to comment.