Add some instructions to use Spark EMR docker image #965

normanj-bitquill · 2024-12-03T21:43:47Z

Description

Add some instruction on how to use the Spark EMR docker image to test the OpenSearch Spark PPL extension. Also included some sample files to help a user get started.

Related Issues

Check List

Updated documentation (docs/ppl-lang/README.md)
Implemented unit tests
Implemented tests for combination with other commands
New added source code should include a copyright header
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

* Also included some sample files Signed-off-by: Norman Jordan <[email protected]>

YANG-DB

@normanj-bitquill this is very nice

I'd also like to see this running using the spark docker image

YANG-DB · 2024-12-03T22:10:47Z

docs/spark-emr-docker.md

+Create a local build or copy of the OpenSearch Spark PPL extension. Make a note of the
+location of the Jar file as well as the name of the Jar file.
+
+## Run the Spark EMR Image


can you also please add a folder for emr based docker-compose ?

Added a docker-compose.yml for EMR.

YANG-DB · 2024-12-03T22:16:42Z

docker/spark-emr-sample/example-app/build.sbt

@@ -0,0 +1,8 @@
+name := "MyApp"


lets create a simple yet more complex use case application that will run multiple ppl queries and result with a report - similar to the next html report

This PR is intended to provide instructions so that a developer could run queries against docker images of Spark.

I think that your comment here is asking to generate an HTML report of the integration test results. See my comment below, I consider integration tests out of scope for this PR.

I can add more to the app if it will help a developer better understand how to run their queries. Perhaps loading a simple table.

YANG-DB · 2024-12-03T22:18:33Z

docker/spark-emr-sample/example-app/src/main/scala/MyApp.scala

+        println("Deploy Mode :" + spark.sparkContext.deployMode);
+        println("Master :" + spark.sparkContext.master);
+
+        spark.sql("CREATE table foo (id int, name varchar(100))").show()


we need a more generic mechanism of creating & loading table data and running a list of dedicated PPL queries based on the table's fields

we can use a simpler version of our IT tests for the first iteration

You are correct, but I consider that out of scope for this PR. This is work that is planned for follow up PRs. This PR is intended to provide instructions and minimal "code" for a developer to be able to run queries against docker images of Spark.

This PR is not intended to run the integration tests.

The follow up PRs would be:

Create docker files that are intended to be used by the integration tests. This would include containers for S3 and the dashboard server. Only focus on testing against Apache Spark release (not EMR).

Update integration tests to run against the docker containers. This includes loading data, running queries and verifying the output. Also includes running the integration tests from the SBT build.

Create docker files for running the integration tests against Spark EMR. This includes changes so that the user can choose whether integration tests run against Apache Spark or Spark EMR.

ok sound good !
thanks !

YANG-DB · 2024-12-05T18:36:17Z

docker/apache-spark-sample/docker-compose.yml

+        source: ./spark-defaults.conf
+        target: /opt/bitnami/spark/conf/spark-defaults.conf
+      - type: bind
+        source: ../../ppl-spark-integration/target/scala-2.12/ppl-spark-integration-assembly-0.7.0-SNAPSHOT.jar


@normanj-bitquill can u plz use docker .env file for parametrizing the ppl-spark-assembly-*.jar and maybe the PORTs ?

Added a sample .env file for the host ports and the PPL Jar location. * doesn't work in the PPL location in a .env file.

YANG-DB · 2024-12-05T18:40:36Z

docs/spark-docker.md

+The Bitnami Apache Spark image can be used to run a Spark cluster and also to run
+`spark-shell` for running queries.
+
+## Setup


plz add the docker .env configuration ppl-spark-assembly-*.jar step here

See the comment above. This was added.

YANG-DB · 2024-12-05T18:41:30Z

docs/spark-docker.md

+
+## Prepare OpenSearch Spark PPL Extension
+
+Create a local build or copy of the OpenSearch Spark PPL extension. Make a note of the


add sbt assembly command here

Added the sbt command.

YANG-DB · 2024-12-05T18:42:03Z

docs/spark-docker.md

+for them to use.
+
+```
+docker network create spark-network


why cant this be done in the docker-compose file ?

It can, and should be more clear in the documentation now. There are two sections, one for using docker compose and one for manual setup. docker network create ... is only needed for the manual setup.

YANG-DB · 2024-12-05T18:43:52Z

docs/spark-docker.md

+docker exec -it spark /opt/bitnami/spark/bin/spark-shell
+```
+
+Within the Spark Shell, you can submit queries, including PPL queries.


please give an example here and link to our example doc

Added an example and a link to the example doc.

YANG-DB · 2024-12-05T18:45:10Z

docs/spark-docker.md

+
+Within the Spark Shell, you can submit queries, including PPL queries.
+
+## Docker Compose Sample


IMO this should be the first step and the actual specific detailed instruction for running each spark element should be last

Moved up much higher in the file. There are two main sections, using docker compose and manual setup. Using docker compose comes first.

YANG-DB

@normanj-bitquill Mostly small comments
can u also add a docker-compose for emr ? if its too much effort lets add this in the next phase ...
thanks

YANG-DB · 2024-12-05T19:52:31Z

@normanj-bitquill
can u plz add sign-off for the next commit ?
8a3155bbe89a7775a2551fcaa1914a8e65eb0b8e The sign-off is missing.
thanks

Signed-off-by: Norman Jordan <[email protected]>

normanj-bitquill · 2024-12-05T22:02:00Z

@normanj-bitquill Mostly small comments can u also add a docker-compose for emr ? if its too much effort lets add this in the next phase ... thanks

This has been added. I feel it fits in this PR, since it allows developers to make use of what I have found so far.

normanj-bitquill · 2024-12-05T22:02:34Z

@normanj-bitquill can u plz add sign-off for the next commit ? 8a3155bbe89a7775a2551fcaa1914a8e65eb0b8e The sign-off is missing. thanks

Fixed the offending commit. All commits are not signed-off.

normanj-bitquill · 2024-12-05T22:03:08Z

@YANG-DB I have updated this. Take another look when you have time.

YANG-DB · 2024-12-06T00:55:04Z

@LantaoJin @penghuo @dai-chen
could u plz also review and give some feedback here ?
thanks

…ct#965) * Add some instructions to use Spark EMR docker image * Also included some sample files Signed-off-by: Norman Jordan <[email protected]> * Added instructions for using Bitnami Spark images Signed-off-by: Norman Jordan <[email protected]> * Added docker compose files for EMR Signed-off-by: Norman Jordan <[email protected]> --------- Signed-off-by: Norman Jordan <[email protected]>

Add some instructions to use Spark EMR docker image

c1edd9b

* Also included some sample files Signed-off-by: Norman Jordan <[email protected]>

normanj-bitquill requested review from dai-chen, mengweieric, vmmusings, penghuo, seankao-az, anirudha, kaituo, YANG-DB, noCharger, LantaoJin and ykmr1224 as code owners December 3, 2024 21:43

YANG-DB requested changes Dec 3, 2024

View reviewed changes

YANG-DB reviewed Dec 3, 2024

View reviewed changes

YANG-DB reviewed Dec 5, 2024

View reviewed changes

YANG-DB approved these changes Dec 5, 2024

View reviewed changes

normanj-bitquill mentioned this pull request Dec 5, 2024

[FEATURE]Add Docker-compose setting for IT testing #601

Closed

YANG-DB added documentation Improvements or additions to documentation Lang:PPL Pipe Processing Language support 0.7 labels Dec 5, 2024

normanj-bitquill added 2 commits December 5, 2024 13:55

Added instructions for using Bitnami Spark images

b4c0dfa

Signed-off-by: Norman Jordan <[email protected]>

Added docker compose files for EMR

9ac1925

Signed-off-by: Norman Jordan <[email protected]>

normanj-bitquill force-pushed the docker-instructions branch from 8a3155b to 9ac1925 Compare December 5, 2024 21:55

YANG-DB approved these changes Dec 6, 2024

View reviewed changes

YANG-DB mentioned this pull request Dec 6, 2024

Add Docker-compose setting for IT testing #606

Closed

YANG-DB merged commit 16fbfea into opensearch-project:main Dec 6, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add some instructions to use Spark EMR docker image #965

Add some instructions to use Spark EMR docker image #965

normanj-bitquill commented Dec 3, 2024

YANG-DB left a comment

YANG-DB Dec 3, 2024 •

edited

Loading

normanj-bitquill Dec 5, 2024

YANG-DB Dec 3, 2024

normanj-bitquill Dec 5, 2024

YANG-DB Dec 3, 2024

normanj-bitquill Dec 5, 2024

YANG-DB Dec 5, 2024

YANG-DB Dec 5, 2024

normanj-bitquill Dec 5, 2024

YANG-DB Dec 5, 2024

normanj-bitquill Dec 5, 2024

YANG-DB Dec 5, 2024

normanj-bitquill Dec 5, 2024

YANG-DB Dec 5, 2024

normanj-bitquill Dec 5, 2024

YANG-DB Dec 5, 2024

normanj-bitquill Dec 5, 2024

YANG-DB Dec 5, 2024

normanj-bitquill Dec 5, 2024

YANG-DB left a comment

YANG-DB commented Dec 5, 2024

normanj-bitquill commented Dec 5, 2024

normanj-bitquill commented Dec 5, 2024

normanj-bitquill commented Dec 5, 2024

YANG-DB commented Dec 6, 2024


		## Prepare OpenSearch Spark PPL Extension

		Create a local build or copy of the OpenSearch Spark PPL extension. Make a note of the


		Within the Spark Shell, you can submit queries, including PPL queries.

		## Docker Compose Sample

Add some instructions to use Spark EMR docker image #965

Add some instructions to use Spark EMR docker image #965

Conversation

normanj-bitquill commented Dec 3, 2024

Description

Related Issues

Check List

YANG-DB left a comment

Choose a reason for hiding this comment

YANG-DB Dec 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YANG-DB left a comment

Choose a reason for hiding this comment

YANG-DB commented Dec 5, 2024

normanj-bitquill commented Dec 5, 2024

normanj-bitquill commented Dec 5, 2024

normanj-bitquill commented Dec 5, 2024

YANG-DB commented Dec 6, 2024

YANG-DB Dec 3, 2024 •

edited

Loading