From 611aa7e7a04dee84644f5131fd92821044dd1b2c Mon Sep 17 00:00:00 2001 From: YANGDB Date: Wed, 11 Oct 2023 17:51:05 -0700 Subject: [PATCH 1/6] add 2.11 release notes Signed-off-by: YANGDB --- .../opensearch-sql.release-notes-2.11.0.0.md | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 release-notes/opensearch-sql.release-notes-2.11.0.0.md diff --git a/release-notes/opensearch-sql.release-notes-2.11.0.0.md b/release-notes/opensearch-sql.release-notes-2.11.0.0.md new file mode 100644 index 0000000000..f99a309297 --- /dev/null +++ b/release-notes/opensearch-sql.release-notes-2.11.0.0.md @@ -0,0 +1,55 @@ +Compatible with OpenSearch and OpenSearch Dashboards Version 2.10.0 + +### Features + +### Enhancements +* [Backport 2.x] Enable PPL lang and add datasource to async query API in https://github.com/opensearch-project/sql/pull/2195 +* [Backport 2.x] Refactor Flint Auth in https://github.com/opensearch-project/sql/pull/2201 +* [Backport 2.x] Add conf for spark structured streaming job in https://github.com/opensearch-project/sql/pull/2203 +* [Backport 2.x] Submit long running job only when auto_refresh = false in https://github.com/opensearch-project/sql/pull/2209 +* [Backport 2.x] Bug Fix, handle DESC TABLE response in https://github.com/opensearch-project/sql/pull/2213 +* [Backport 2.x] Drop Index Implementation in https://github.com/opensearch-project/sql/pull/2217 +* [Backport 2.x] Enable PPL Queries in https://github.com/opensearch-project/sql/pull/2223 +* [Backport 2.11] Read extra Spark submit parameters from cluster settings in https://github.com/opensearch-project/sql/pull/2236 +* [Backport 2.11] Spark Execution Engine Config Refactor in https://github.com/opensearch-project/sql/pull/2266 +* [Backport 2.11] Provide auth.type and auth.role_arn paramters in GET Datasource API response. in https://github.com/opensearch-project/sql/pull/2283 +* [Backport 2.x] Add support for `date_nanos` and tests. (#337) in https://github.com/opensearch-project/sql/pull/2020 +* [Backport 2.x] Applied formatting improvements to Antlr files based on spotless changes (#2017) by @MitchellGale in https://github.com/opensearch-project/sql/pull/2023 +* [Backport 2.x] Revert "Guarantee datasource read api is strong consistent read (#1815)" in https://github.com/opensearch-project/sql/pull/2031 +* [Backport 2.x] Add _primary preference only for segment replication enabled indices in https://github.com/opensearch-project/sql/pull/2045 +* [Backport 2.x] Changed allowlist config to denylist ip config for datasource uri hosts in https://github.com/opensearch-project/sql/pull/2058 + +### Bug Fixes +* [Backport 2.x] fix broken link for connectors doc in https://github.com/opensearch-project/sql/pull/2199 +* [Backport 2.x] Fix response codes returned by JSON formatting them in https://github.com/opensearch-project/sql/pull/2200 +* [Backport 2.x] Bug fix, datasource API should be case sensitive in https://github.com/opensearch-project/sql/pull/2202 +* [Backport 2.11] Minor fix in dropping covering index in https://github.com/opensearch-project/sql/pull/2240 +* [Backport 2.11] Fix Unit tests for FlintIndexReader in https://github.com/opensearch-project/sql/pull/2242 +* [Backport 2.11] Bug Fix , delete OpenSearch index when DROP INDEX in https://github.com/opensearch-project/sql/pull/2252 +* [Backport 2.11] Correctly Set query status in https://github.com/opensearch-project/sql/pull/2232 +* [Backport 2.x] Exclude generated files from spotless in https://github.com/opensearch-project/sql/pull/2024 +* [Backport 2.x] Fix mockito core conflict. in https://github.com/opensearch-project/sql/pull/2131 +* [Backport 2.x] Fix `ASCII` function and groom UT for text functions. (#301) in https://github.com/opensearch-project/sql/pull/2029 +* [Backport 2.x] Fixed response codes For Requests With security exception. in https://github.com/opensearch-project/sql/pull/2036 + +### Documentation +* [Backport 2.x] Datasource description in https://github.com/opensearch-project/sql/pull/2138 +* [Backport 2.11] Add documentation for S3GlueConnector. in https://github.com/opensearch-project/sql/pull/2234 + +### Infrastructure +* [Backport 2.x] bump aws-encryption-sdk-java to 1.71 in https://github.com/opensearch-project/sql/pull/2057 +* [Backport 2.x] Run IT tests with security plugin (#335) #1986 by @MitchellGale in https://github.com/opensearch-project/sql/pull/2022 + +### Refactoring +* [Backport 2.x] Merging Async Query APIs feature branch into main. in https://github.com/opensearch-project/sql/pull/2163 +* [Backport 2.x] Removed Domain Validation in https://github.com/opensearch-project/sql/pull/2136 +* [Backport 2.x] Check for existence of security plugin in https://github.com/opensearch-project/sql/pull/2069 +* [Backport 2.x] Always use snapshot version for security plugin download in https://github.com/opensearch-project/sql/pull/2061 +* [Backport 2.x] Add customized result index in data source etc in https://github.com/opensearch-project/sql/pull/2220 + +### Security +* [2.x] bump okhttp to 4.10.0 (#2043) by @joshuali925 in https://github.com/opensearch-project/sql/pull/2044 +* [2.x] bump okio to 3.4.0 by @joshuali925 in https://github.com/opensearch-project/sql/pull/2047 + +--- +**Full Changelog**: https://github.com/opensearch-project/sql/compare/2.3.0.0...v.2.11.0.0 \ No newline at end of file From 2b4ab28399572c7cedb088b6f4be4f6180294081 Mon Sep 17 00:00:00 2001 From: YANGDB Date: Wed, 11 Oct 2023 17:51:20 -0700 Subject: [PATCH 2/6] add 2.11 release notes Signed-off-by: YANGDB --- release-notes/opensearch-sql.release-notes-2.11.0.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-notes/opensearch-sql.release-notes-2.11.0.0.md b/release-notes/opensearch-sql.release-notes-2.11.0.0.md index f99a309297..7434f4e95a 100644 --- a/release-notes/opensearch-sql.release-notes-2.11.0.0.md +++ b/release-notes/opensearch-sql.release-notes-2.11.0.0.md @@ -1,4 +1,4 @@ -Compatible with OpenSearch and OpenSearch Dashboards Version 2.10.0 +Compatible with OpenSearch and OpenSearch Dashboards Version 2.11.0 ### Features From 61f96c8e60f5a7c299b04ccdc052954482d6cdcf Mon Sep 17 00:00:00 2001 From: YANGDB Date: Thu, 12 Oct 2023 09:02:50 -0700 Subject: [PATCH 3/6] update notes Signed-off-by: YANGDB --- .../opensearch-sql.release-notes-2.11.0.0.md | 74 +++++++++---------- 1 file changed, 37 insertions(+), 37 deletions(-) diff --git a/release-notes/opensearch-sql.release-notes-2.11.0.0.md b/release-notes/opensearch-sql.release-notes-2.11.0.0.md index 7434f4e95a..a560d5c8dd 100644 --- a/release-notes/opensearch-sql.release-notes-2.11.0.0.md +++ b/release-notes/opensearch-sql.release-notes-2.11.0.0.md @@ -3,53 +3,53 @@ Compatible with OpenSearch and OpenSearch Dashboards Version 2.11.0 ### Features ### Enhancements -* [Backport 2.x] Enable PPL lang and add datasource to async query API in https://github.com/opensearch-project/sql/pull/2195 -* [Backport 2.x] Refactor Flint Auth in https://github.com/opensearch-project/sql/pull/2201 -* [Backport 2.x] Add conf for spark structured streaming job in https://github.com/opensearch-project/sql/pull/2203 -* [Backport 2.x] Submit long running job only when auto_refresh = false in https://github.com/opensearch-project/sql/pull/2209 -* [Backport 2.x] Bug Fix, handle DESC TABLE response in https://github.com/opensearch-project/sql/pull/2213 -* [Backport 2.x] Drop Index Implementation in https://github.com/opensearch-project/sql/pull/2217 -* [Backport 2.x] Enable PPL Queries in https://github.com/opensearch-project/sql/pull/2223 -* [Backport 2.11] Read extra Spark submit parameters from cluster settings in https://github.com/opensearch-project/sql/pull/2236 -* [Backport 2.11] Spark Execution Engine Config Refactor in https://github.com/opensearch-project/sql/pull/2266 -* [Backport 2.11] Provide auth.type and auth.role_arn paramters in GET Datasource API response. in https://github.com/opensearch-project/sql/pull/2283 -* [Backport 2.x] Add support for `date_nanos` and tests. (#337) in https://github.com/opensearch-project/sql/pull/2020 -* [Backport 2.x] Applied formatting improvements to Antlr files based on spotless changes (#2017) by @MitchellGale in https://github.com/opensearch-project/sql/pull/2023 -* [Backport 2.x] Revert "Guarantee datasource read api is strong consistent read (#1815)" in https://github.com/opensearch-project/sql/pull/2031 -* [Backport 2.x] Add _primary preference only for segment replication enabled indices in https://github.com/opensearch-project/sql/pull/2045 -* [Backport 2.x] Changed allowlist config to denylist ip config for datasource uri hosts in https://github.com/opensearch-project/sql/pull/2058 +* Enable PPL lang and add datasource to async query API in https://github.com/opensearch-project/sql/pull/2195 +* Refactor Flint Auth in https://github.com/opensearch-project/sql/pull/2201 +* Add conf for spark structured streaming job in https://github.com/opensearch-project/sql/pull/2203 +* Submit long running job only when auto_refresh = false in https://github.com/opensearch-project/sql/pull/2209 +* Bug Fix, handle DESC TABLE response in https://github.com/opensearch-project/sql/pull/2213 +* Drop Index Implementation in https://github.com/opensearch-project/sql/pull/2217 +* Enable PPL Queries in https://github.com/opensearch-project/sql/pull/2223 +* Read extra Spark submit parameters from cluster settings in https://github.com/opensearch-project/sql/pull/2236 +* Spark Execution Engine Config Refactor in https://github.com/opensearch-project/sql/pull/2266 +* Provide auth.type and auth.role_arn paramters in GET Datasource API response. in https://github.com/opensearch-project/sql/pull/2283 +* Add support for `date_nanos` and tests. (#337) in https://github.com/opensearch-project/sql/pull/2020 +* Applied formatting improvements to Antlr files based on spotless changes (#2017) by @MitchellGale in https://github.com/opensearch-project/sql/pull/2023 +* Revert "Guarantee datasource read api is strong consistent read (#1815)" in https://github.com/opensearch-project/sql/pull/2031 +* Add _primary preference only for segment replication enabled indices in https://github.com/opensearch-project/sql/pull/2045 +* Changed allowlist config to denylist ip config for datasource uri hosts in https://github.com/opensearch-project/sql/pull/2058 ### Bug Fixes -* [Backport 2.x] fix broken link for connectors doc in https://github.com/opensearch-project/sql/pull/2199 -* [Backport 2.x] Fix response codes returned by JSON formatting them in https://github.com/opensearch-project/sql/pull/2200 -* [Backport 2.x] Bug fix, datasource API should be case sensitive in https://github.com/opensearch-project/sql/pull/2202 -* [Backport 2.11] Minor fix in dropping covering index in https://github.com/opensearch-project/sql/pull/2240 -* [Backport 2.11] Fix Unit tests for FlintIndexReader in https://github.com/opensearch-project/sql/pull/2242 -* [Backport 2.11] Bug Fix , delete OpenSearch index when DROP INDEX in https://github.com/opensearch-project/sql/pull/2252 -* [Backport 2.11] Correctly Set query status in https://github.com/opensearch-project/sql/pull/2232 -* [Backport 2.x] Exclude generated files from spotless in https://github.com/opensearch-project/sql/pull/2024 -* [Backport 2.x] Fix mockito core conflict. in https://github.com/opensearch-project/sql/pull/2131 -* [Backport 2.x] Fix `ASCII` function and groom UT for text functions. (#301) in https://github.com/opensearch-project/sql/pull/2029 -* [Backport 2.x] Fixed response codes For Requests With security exception. in https://github.com/opensearch-project/sql/pull/2036 +* fix broken link for connectors doc in https://github.com/opensearch-project/sql/pull/2199 +* Fix response codes returned by JSON formatting them in https://github.com/opensearch-project/sql/pull/2200 +* Bug fix, datasource API should be case sensitive in https://github.com/opensearch-project/sql/pull/2202 +* Minor fix in dropping covering index in https://github.com/opensearch-project/sql/pull/2240 +* Fix Unit tests for FlintIndexReader in https://github.com/opensearch-project/sql/pull/2242 +* Bug Fix , delete OpenSearch index when DROP INDEX in https://github.com/opensearch-project/sql/pull/2252 +* Correctly Set query status in https://github.com/opensearch-project/sql/pull/2232 +* Exclude generated files from spotless in https://github.com/opensearch-project/sql/pull/2024 +* Fix mockito core conflict. in https://github.com/opensearch-project/sql/pull/2131 +* Fix `ASCII` function and groom UT for text functions. (#301) in https://github.com/opensearch-project/sql/pull/2029 +* Fixed response codes For Requests With security exception. in https://github.com/opensearch-project/sql/pull/2036 ### Documentation -* [Backport 2.x] Datasource description in https://github.com/opensearch-project/sql/pull/2138 -* [Backport 2.11] Add documentation for S3GlueConnector. in https://github.com/opensearch-project/sql/pull/2234 +* Datasource description in https://github.com/opensearch-project/sql/pull/2138 +* Add documentation for S3GlueConnector. in https://github.com/opensearch-project/sql/pull/2234 ### Infrastructure -* [Backport 2.x] bump aws-encryption-sdk-java to 1.71 in https://github.com/opensearch-project/sql/pull/2057 -* [Backport 2.x] Run IT tests with security plugin (#335) #1986 by @MitchellGale in https://github.com/opensearch-project/sql/pull/2022 +* bump aws-encryption-sdk-java to 1.71 in https://github.com/opensearch-project/sql/pull/2057 +* Run IT tests with security plugin (#335) #1986 by @MitchellGale in https://github.com/opensearch-project/sql/pull/2022 ### Refactoring -* [Backport 2.x] Merging Async Query APIs feature branch into main. in https://github.com/opensearch-project/sql/pull/2163 -* [Backport 2.x] Removed Domain Validation in https://github.com/opensearch-project/sql/pull/2136 -* [Backport 2.x] Check for existence of security plugin in https://github.com/opensearch-project/sql/pull/2069 -* [Backport 2.x] Always use snapshot version for security plugin download in https://github.com/opensearch-project/sql/pull/2061 -* [Backport 2.x] Add customized result index in data source etc in https://github.com/opensearch-project/sql/pull/2220 +* Merging Async Query APIs feature branch into main. in https://github.com/opensearch-project/sql/pull/2163 +* Removed Domain Validation in https://github.com/opensearch-project/sql/pull/2136 +* Check for existence of security plugin in https://github.com/opensearch-project/sql/pull/2069 +* Always use snapshot version for security plugin download in https://github.com/opensearch-project/sql/pull/2061 +* Add customized result index in data source etc in https://github.com/opensearch-project/sql/pull/2220 ### Security -* [2.x] bump okhttp to 4.10.0 (#2043) by @joshuali925 in https://github.com/opensearch-project/sql/pull/2044 -* [2.x] bump okio to 3.4.0 by @joshuali925 in https://github.com/opensearch-project/sql/pull/2047 +* bump okhttp to 4.10.0 (#2043) by @joshuali925 in https://github.com/opensearch-project/sql/pull/2044 +* bump okio to 3.4.0 by @joshuali925 in https://github.com/opensearch-project/sql/pull/2047 --- **Full Changelog**: https://github.com/opensearch-project/sql/compare/2.3.0.0...v.2.11.0.0 \ No newline at end of file From d40304d6cb8cd1937aaa79c5d323ae54a39f25f9 Mon Sep 17 00:00:00 2001 From: YANGDB Date: Thu, 26 Oct 2023 20:57:54 -0700 Subject: [PATCH 4/6] add _async_query api doc and swagger Signed-off-by: YANGDB --- docs/dev/API/README.md | 91 ++++++++++++++++++++++++++++++++++++++ docs/dev/API/swagger.yaml | 92 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 183 insertions(+) create mode 100644 docs/dev/API/README.md create mode 100644 docs/dev/API/swagger.yaml diff --git a/docs/dev/API/README.md b/docs/dev/API/README.md new file mode 100644 index 0000000000..9187a84b75 --- /dev/null +++ b/docs/dev/API/README.md @@ -0,0 +1,91 @@ +# Async Query Interface API Documentation + +The Async Query Interface API enables the execution of queries on top of a Spark engine. This is specifically designed to work with the `S3Glue` data source connector and allows queries to be run asynchronously. + +The Async Query Interface is especially useful for running complex, time-consuming queries without blocking the main thread. The API provides endpoints for query creation, result retrieval, and query cancellation. + +--- + +## Table of Contents +1. [Configurations](#configurations) +2. [Async Query Creation](#async-query-creation) +3. [Query Result Retrieval](#query-result-retrieval) +4. [Query Cancellation](#query-cancellation) +5. [API Mock Tests](#api-mock-tests) + +--- + +## Configurations + +Before using the Async Query APIs, the Spark Execution Engine Config for Async Queries must be set under `plugins.query.executionengine.spark.config` in the cluster settings. For more details on this configuration, refer to the introductory section of this document. + +--- + +## Async Query Creation + +To create an async query, you can use the `POST /_plugins/_async_query` endpoint. + +**Examples** + +- [Query Creation Sample Request](../system/query_creation_request.json) +- [Query Creation Sample Response](../system/query_creation_response.json) + +--- + +## Query Result Retrieval + +To fetch the result of a previously created async query, use the `GET /_plugins/_async_query/{queryId}` endpoint. + +**Examples** + +- [Query Result Sample Request](../system/query_result_request.json) +- [Query Result Sample Response](../system/query_result_response.json) + +--- + +## Query Cancellation + +To cancel a running or queued query, use the `DELETE /_plugins/_async_query/{queryId}` endpoint. + +**Examples** + +- [Query Cancellation Sample Request](../system/query_cancellation_request.json) +- [Query Cancellation Sample Response](../system/query_cancellation_response.json) + +--- + +_This API is designed to work seamlessly with AWS EMRServerless as the Spark execution engine._ + +--- + +## API Mock Tests + +The [Swagger YAML file](swagger.yaml) describes the API. You can visualize the REST API using any of the numerous [online editors](https://editor.swagger.io/). + +### Setup Mock Server + +To test the API, you can use a Swagger-based mock library like Stoplight Prism. + +#### Running the Swagger Mock Server + +```bash +npm install -g @stoplight/prism-cli +``` + +Once installed, run the server using: + +```bash +prism mock swagger.yaml +``` + +#### Running the Swagger Mock Server as Docker + +Run the following one-liner Docker command: + +```bash +docker run -p 9200:4010 -v "$(pwd)/docs/API:/api" --name integ-prism -t stoplight/prism mock -h 0.0.0.0 /api/swagger.yaml +``` + +After the server has started, you can initiate a CURL request to test any of the API endpoints. + +![Swagger API Screenshot](../../img/swagger-api.png) diff --git a/docs/dev/API/swagger.yaml b/docs/dev/API/swagger.yaml new file mode 100644 index 0000000000..b07418a5d8 --- /dev/null +++ b/docs/dev/API/swagger.yaml @@ -0,0 +1,92 @@ +openapi: "3.0.0" +info: + title: "Async Query Interface Endpoints" + version: "1.0.0" + +paths: + /_plugins/_async_query: + post: + summary: "Create an async query" + description: "If the security plugin is enabled, this API can only be invoked by users with specific permissions." + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + datasource: + type: string + lang: + type: string + query: + type: string + sessionId: + type: string + responses: + "200": + description: "Query created successfully" + content: + application/json: + schema: + type: object + properties: + queryId: + type: string + sessionId: + type: string + + /_plugins/_async_query/{queryId}: + get: + summary: "Fetch async query result" + description: "If the security plugin is enabled, this API can only be invoked by users with specific permissions." + parameters: + - in: path + name: queryId + required: true + schema: + type: string + responses: + "200": + description: "Query result" + content: + application/json: + schema: + type: object + properties: + status: + type: string + schema: + type: array + items: + type: object + properties: + name: + type: string + type: + type: string + datarows: + type: array + items: + type: array + items: + type: string + total: + type: integer + size: + type: integer + + delete: + summary: "Cancel async query" + description: "If the security plugin is enabled, this API can only be invoked by users with specific permissions." + parameters: + - in: path + name: queryId + required: true + schema: + type: string + responses: + "200": + description: "Query cancelled successfully" +servers: + - url: "http://ec2-18-237-133-156.us-west-2.compute.amazonaws.com:9200" \ No newline at end of file From dc5f3b0e870b1c3f1deb936ba8fb90c700bf0084 Mon Sep 17 00:00:00 2001 From: YANGDB Date: Thu, 26 Oct 2023 21:25:37 -0700 Subject: [PATCH 5/6] add _async_query api doc and swagger Signed-off-by: YANGDB --- docs/dev/API/swagger.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dev/API/swagger.yaml b/docs/dev/API/swagger.yaml index b07418a5d8..6e3453a4f9 100644 --- a/docs/dev/API/swagger.yaml +++ b/docs/dev/API/swagger.yaml @@ -89,4 +89,4 @@ paths: "200": description: "Query cancelled successfully" servers: - - url: "http://ec2-18-237-133-156.us-west-2.compute.amazonaws.com:9200" \ No newline at end of file + - url: "http://my-backend-opensearch:9200" \ No newline at end of file From 9db4bd2bd64d0f60b915668500b118c683302ab0 Mon Sep 17 00:00:00 2001 From: YANGDB Date: Fri, 27 Oct 2023 11:00:29 -0700 Subject: [PATCH 6/6] make JsonVisitorImpl private Signed-off-by: YANGDB --- docs/dev/API/README.md | 32 +++++++++++++++++++++++++------- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/docs/dev/API/README.md b/docs/dev/API/README.md index 9187a84b75..a42be06165 100644 --- a/docs/dev/API/README.md +++ b/docs/dev/API/README.md @@ -27,10 +27,22 @@ To create an async query, you can use the `POST /_plugins/_async_query` endpoint **Examples** -- [Query Creation Sample Request](../system/query_creation_request.json) -- [Query Creation Sample Response](../system/query_creation_response.json) +- Query Creation Sample Request +``` + POST _plugins/_async_query + {"query":""CREATE EXTERNAL TABLE mys3.default.http_logs ( + `@timestamp` TIMESTAMP, + clientip STRING, + request STRING, + status INT, + size INT, + year INT, + month INT, + day INT) + USING json PARTITIONED BY(year, month, day) OPTIONS (path 's3://flint-data-dp-eu-west-1-beta/data/http_log/http_logs_partitioned_json_bz2/', compression 'bzip2')"" + ,"datasource":"mys3","lang":"sql"} +``` ---- ## Query Result Retrieval @@ -38,8 +50,11 @@ To fetch the result of a previously created async query, use the `GET /_plugins/ **Examples** -- [Query Result Sample Request](../system/query_result_request.json) -- [Query Result Sample Response](../system/query_result_response.json) +- Query Result Sample Request +``` +POST _plugins/_async_query + {"query":"SELECT * FROM mys3.default.aws_elb","datasource":"mys3","lang":"sql"} +``` --- @@ -49,8 +64,11 @@ To cancel a running or queued query, use the `DELETE /_plugins/_async_query/{que **Examples** -- [Query Cancellation Sample Request](../system/query_cancellation_request.json) -- [Query Cancellation Sample Response](../system/query_cancellation_response.json) +- Query Cancellation Sample Request + +``` +DELETE _plugins/_async_query/VjlvWTNBbE1Lem15czM= +``` ---