OpenSearch domain as Catalog in Spark

[penghuo]

User interface

Catalog configuration

• user configure OpenSearch domain as a OpenSearchCatalog. then refer to this catalog by using name dev
• user configure OpenSearch domain access client.

spark.sql.catalog.dev=org.apache.opensearch.spark.OpenSearchCatalog
spark.sql.catalog.dev.opensearch.port: default is 9200.
spark.sql.catalog.dev.opensearch.scheme: default is http. valid values [http, https]
spark.sql.catalog.dev.opensearch.auth: default is noauth. valid values [noauth, sigv4, basic]
spark.sql.catalog.dev.opensearch.auth.username: basic auth username.
spark.sql.catalog.dev.opensearch.auth.password: basic auth password.
spark.sql.catalog.dev.opensearch.region: default is us-west-2. only been used when auth=sigv4
spark.sql.catalog.dev.opensearch.customAWSCredentialsProvider: default is empty.

Namespace

the only namespace supported is default

Supported catalog operation

read only catalog operations are supported.

• loadTable

Limitations

the following catalog operations are not supported.

• listTables
• createTable
• alterTable
• dropTable
• renameTable