forked from opensearch-project/opensearch-spark
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
update docker-compose.yml and different docs relating to te local spa…
…rk-flint tests Signed-off-by: YANGDB <[email protected]>
- Loading branch information
Showing
6 changed files
with
215 additions
and
56 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,3 @@ | ||
| # version for opensearch & opensearch-dashboards docker image | ||
| VERSION=2.9.0 | ||
| VERSION=2.10.0 | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one or more | ||
| # contributor license agreements. See the NOTICE file distributed with | ||
| # this work for additional information regarding copyright ownership. | ||
| # The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| # (the "License"); you may not use this file except in compliance with | ||
| # the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| # Set everything to be logged to the console | ||
| log4j.rootCategory=DEBUG, console | ||
| log4j.appender.console=org.apache.log4j.ConsoleAppender | ||
| log4j.appender.console.target=System.err | ||
| log4j.appender.console.layout=org.apache.log4j.PatternLayout | ||
| log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n | ||
|
|
||
| # Set the default spark-shell log level to WARN. When running the spark-shell, the | ||
| # log level for this class is used to overwrite the root logger's log level, so that | ||
| # the user can have different defaults for the shell and regular Spark apps. | ||
| log4j.logger.org.apache.spark.repl.Main=DEBUG | ||
|
|
||
| # Settings to quiet third party logs that are too verbose | ||
| log4j.logger.org.spark_project.jetty=WARN |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one or more | ||
| # contributor license agreements. See the NOTICE file distributed with | ||
| # this work for additional information regarding copyright ownership. | ||
| # The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| # (the "License"); you may not use this file except in compliance with | ||
| # the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| # Default system properties included when running spark-submit. | ||
| # This is useful for setting default environmental settings. | ||
|
|
||
| # Example: | ||
| # spark.master spark://master:7077 | ||
| # spark.eventLog.enabled true | ||
| # spark.eventLog.dir hdfs://namenode:8021/directory | ||
| # spark.serializer org.apache.spark.serializer.KryoSerializer | ||
| # spark.driver.memory 2g | ||
| # spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three" | ||
|
|
||
|
|
||
| #spark.jars.packages = org.postgresql:postgresql:9.4.1207.jar | ||
| #spark.driver.extraClassPath = /opt/bitnami/spark/jars/postgresql-9.4.1207.jar |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # Testing locally With Spark | ||
| This document is intended to review the local docker-compose based environment in-which the Flint/PPL - spark plugins can be testes and explored. | ||
|
|
||
| ## Overview | ||
| The following components are part of this testing environment | ||
|
|
||
| ### Livy | ||
| Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, all via a simple REST interface or an RPC client library. | ||
| Live provides a comprehensive [REST API](https://livy.apache.org/docs/latest/rest-api.html) to interact with spark cluster in a simplified way. | ||
|
|
||
| ## Test Tutorial | ||
| First we need to create a livy session | ||
| ``` | ||
| curl --location --request POST 'http://localhost:8998/sessions' \ | ||
| --header 'Content-Type: application/json' \ | ||
| --data-raw '{ | ||
| "kind": "sql", | ||
| "proxyUser": "a_user" | ||
| }' | ||
| ``` | ||
| This call will respond with a session Id in the following manner: | ||
| ```json5 | ||
| { | ||
| "id": 0, | ||
| "name": null, | ||
| "appId": null, | ||
| "owner": null, | ||
| "proxyUser": null, | ||
| "state": "starting", | ||
| "kind": "sql", | ||
| "appInfo": { | ||
| "driverLogUrl": null, | ||
| "sparkUiUrl": null | ||
| }, | ||
| "log": [ | ||
| "stdout: ", | ||
| "\nstderr: " | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| Once a session is created, we can submit a SQL query statement the following way: | ||
| ``` | ||
| curl --location --request POST 'http://localhost:8998/sessions/0/statements' \ | ||
| --header 'Content-Type: application/json' \ | ||
| --data-raw '{ | ||
| "code": "spark.sql(\"CREATE TABLE test_table (id INT, name STRING)\")" | ||
| }' | ||
| ``` | ||
|
|
||
| This call responds with the next ack | ||
| ```json5 | ||
| {"id":0,"code":"select 1","state":"waiting","output":null,"progress":0.0,"started":0,"completed":0} | ||
| ``` | ||
|
|
||
| Next we can Insert some data into that table: | ||
| ``` | ||
| curl --location --request POST 'http://localhost:8998/sessions/0/statements' \ | ||
| --header 'Content-Type: application/json' \ | ||
| --data-raw '{ | ||
| "code": "spark.sql(\"INSERT INTO test_table VALUES (1, 'John'), (2, 'Doe')\")" | ||
| }' | ||
| ``` | ||
|
|
||
| Now lets query the table using SQL: | ||
| ``` | ||
| curl --location --request POST 'http://localhost:8998/sessions/0/statements' \ | ||
| --header 'Content-Type: application/json' \ | ||
| --data-raw '{ | ||
| "code": "spark.sql(\"SELECT * FROM test_table\").show()" | ||
| }' | ||
| ``` | ||
|
|
||
| We can now see the Livy session created with the execution running: | ||
|
|
||
| ![Livy UI session Image]() | ||
|
|
||
| To get the response of this statement use the next API: | ||
| `curl --location --request GET http://localhost:8998/sessions/0/statements/0 | jq '.output.data.application/json.data'` | ||
|
|
||
| This would respond with the next results | ||
| ```text | ||
| % Total % Received % Xferd Average Speed Time Time Time Current | ||
| Dload Upload Total Spent Left Speed | ||
| 100 298 100 298 0 0 6610 0 --:--:-- --:--:-- --:--:-- 7641 | ||
| ``` |