Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Configure the Spark metrics properties while creating a s3 Glue Connector #2502

Closed
noCharger opened this issue Feb 5, 2024 · 0 comments
Labels
enhancement New feature or request untriaged

Comments

@noCharger
Copy link
Collaborator

Is your feature request related to a problem?

Currently, the user must configure the ExecutionEngine Config to use the s3 Glue Connector. Even with custom metrics support (opensearch-project/opensearch-spark#173 and opensearch-project/opensearch-spark#229), there is no direct way to configure Spark metrics (whether default or custom). The user must construct an image in EMR-S that includes an overriden metrics.properties file.

What solution would you like?

Allow users to configure Spark metrics properties in SQL plugin, for example.

plugins.query.executionengine.spark.config:
'{  "applicationId":"xxxxx",
    "executionRoleARN":"arn:aws:iam::***********:role/emr-job-execution-role",
    "region":"eu-west-1",
    "sparkSubmitParameter": "--conf spark.dynamicAllocation.enabled=false --conf spark.metrics.conf.*.sink.cloudwatch.class=org.apache.spark.metrics.sink.CloudWatchSink
spark.metrics.conf.*.sink.cloudwatch.namespace=OpenSearchSQLSpark"
}'
@noCharger noCharger added enhancement New feature or request untriaged labels Feb 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request untriaged
Projects
None yet
Development

No branches or pull requests

1 participant