diff --git a/docs/dev/API/README.md b/docs/dev/API/README.md new file mode 100644 index 0000000000..a42be06165 --- /dev/null +++ b/docs/dev/API/README.md @@ -0,0 +1,109 @@ +# Async Query Interface API Documentation + +The Async Query Interface API enables the execution of queries on top of a Spark engine. This is specifically designed to work with the `S3Glue` data source connector and allows queries to be run asynchronously. + +The Async Query Interface is especially useful for running complex, time-consuming queries without blocking the main thread. The API provides endpoints for query creation, result retrieval, and query cancellation. + +--- + +## Table of Contents +1. [Configurations](#configurations) +2. [Async Query Creation](#async-query-creation) +3. [Query Result Retrieval](#query-result-retrieval) +4. [Query Cancellation](#query-cancellation) +5. [API Mock Tests](#api-mock-tests) + +--- + +## Configurations + +Before using the Async Query APIs, the Spark Execution Engine Config for Async Queries must be set under `plugins.query.executionengine.spark.config` in the cluster settings. For more details on this configuration, refer to the introductory section of this document. + +--- + +## Async Query Creation + +To create an async query, you can use the `POST /_plugins/_async_query` endpoint. + +**Examples** + +- Query Creation Sample Request +``` + POST _plugins/_async_query + {"query":""CREATE EXTERNAL TABLE mys3.default.http_logs ( + `@timestamp` TIMESTAMP, + clientip STRING, + request STRING, + status INT, + size INT, + year INT, + month INT, + day INT) + USING json PARTITIONED BY(year, month, day) OPTIONS (path 's3://flint-data-dp-eu-west-1-beta/data/http_log/http_logs_partitioned_json_bz2/', compression 'bzip2')"" + ,"datasource":"mys3","lang":"sql"} +``` + + +## Query Result Retrieval + +To fetch the result of a previously created async query, use the `GET /_plugins/_async_query/{queryId}` endpoint. + +**Examples** + +- Query Result Sample Request +``` +POST _plugins/_async_query + {"query":"SELECT * FROM mys3.default.aws_elb","datasource":"mys3","lang":"sql"} +``` + +--- + +## Query Cancellation + +To cancel a running or queued query, use the `DELETE /_plugins/_async_query/{queryId}` endpoint. + +**Examples** + +- Query Cancellation Sample Request + +``` +DELETE _plugins/_async_query/VjlvWTNBbE1Lem15czM= +``` + +--- + +_This API is designed to work seamlessly with AWS EMRServerless as the Spark execution engine._ + +--- + +## API Mock Tests + +The [Swagger YAML file](swagger.yaml) describes the API. You can visualize the REST API using any of the numerous [online editors](https://editor.swagger.io/). + +### Setup Mock Server + +To test the API, you can use a Swagger-based mock library like Stoplight Prism. + +#### Running the Swagger Mock Server + +```bash +npm install -g @stoplight/prism-cli +``` + +Once installed, run the server using: + +```bash +prism mock swagger.yaml +``` + +#### Running the Swagger Mock Server as Docker + +Run the following one-liner Docker command: + +```bash +docker run -p 9200:4010 -v "$(pwd)/docs/API:/api" --name integ-prism -t stoplight/prism mock -h 0.0.0.0 /api/swagger.yaml +``` + +After the server has started, you can initiate a CURL request to test any of the API endpoints. + +![Swagger API Screenshot](../../img/swagger-api.png) diff --git a/docs/dev/API/swagger.yaml b/docs/dev/API/swagger.yaml new file mode 100644 index 0000000000..6e3453a4f9 --- /dev/null +++ b/docs/dev/API/swagger.yaml @@ -0,0 +1,92 @@ +openapi: "3.0.0" +info: + title: "Async Query Interface Endpoints" + version: "1.0.0" + +paths: + /_plugins/_async_query: + post: + summary: "Create an async query" + description: "If the security plugin is enabled, this API can only be invoked by users with specific permissions." + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + datasource: + type: string + lang: + type: string + query: + type: string + sessionId: + type: string + responses: + "200": + description: "Query created successfully" + content: + application/json: + schema: + type: object + properties: + queryId: + type: string + sessionId: + type: string + + /_plugins/_async_query/{queryId}: + get: + summary: "Fetch async query result" + description: "If the security plugin is enabled, this API can only be invoked by users with specific permissions." + parameters: + - in: path + name: queryId + required: true + schema: + type: string + responses: + "200": + description: "Query result" + content: + application/json: + schema: + type: object + properties: + status: + type: string + schema: + type: array + items: + type: object + properties: + name: + type: string + type: + type: string + datarows: + type: array + items: + type: array + items: + type: string + total: + type: integer + size: + type: integer + + delete: + summary: "Cancel async query" + description: "If the security plugin is enabled, this API can only be invoked by users with specific permissions." + parameters: + - in: path + name: queryId + required: true + schema: + type: string + responses: + "200": + description: "Query cancelled successfully" +servers: + - url: "http://my-backend-opensearch:9200" \ No newline at end of file