Skip to content

Commit

Permalink
update README.md with the additional spark extensions input argument
Browse files Browse the repository at this point in the history
Signed-off-by: YANGDB <[email protected]>
  • Loading branch information
YANG-DB committed Oct 4, 2023
1 parent a046696 commit f7394a8
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion spark-sql-application/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ FlintJob is designed for EMR Serverless Spark, executing SQL queries and storing
+ Spark 3.3.1
+ Scala 2.12.15
+ flint-spark-integration
+ ppl-spark-integration

## Usage

Expand All @@ -36,6 +37,7 @@ SQLJob
--class org.opensearch.sql.SQLJob \
--jars <flint-spark-integration-jar> \
sql-job.jar \
<spark-extensions-list> \
<spark-sql-query> \
<opensearch-index> \
<opensearch-host> \
Expand All @@ -52,7 +54,7 @@ aws emr-serverless start-job-run \
--application-id <application-id> \
--execution-role-arn <execution-role> \
--job-driver '{"sparkSubmit": {"entryPoint": "<flint-job-s3-path>", \
"entryPointArguments":["'<sql-query>'", "<result-index>", "<data-source-name>"], \
"entryPointArguments":["'<spark extensions>'","'<sql-query>'", "<result-index>", "<data-source-name>"], \
"sparkSubmitParameters":"--class org.opensearch.sql.FlintJob \
--conf spark.hadoop.fs.s3.customAWSCredentialsProvider=com.amazonaws.emr.AssumeRoleAWSCredentialsProvider \
--conf spark.emr-serverless.driverEnv.ASSUME_ROLE_CREDENTIALS_ROLE_ARN=<role-to-access-s3-and-opensearch> \
Expand Down

0 comments on commit f7394a8

Please sign in to comment.