Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE]Create MV index target using existing opensearch index #358

Closed
YANG-DB opened this issue May 29, 2024 · 3 comments
Closed

[FEATURE]Create MV index target using existing opensearch index #358

YANG-DB opened this issue May 29, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@YANG-DB
Copy link
Member

YANG-DB commented May 29, 2024

Is your feature request related to a problem?
There are some cases which existing indices or index templates are needed to be used as the ingestion target sync of the MV query.

For example:

  • existing index template contains specific data-types supported only in opensearch
  • existing index contains data that needs to be appended with the MV query results
  • existing index has former version of the MV and need to continue holding the data with the updated MV

What solution would you like?
The MV creation metadata specification would have the next fields: using_existing_index = opensearch_index_name

WITH (
  auto_refresh = true,
  refresh_interval = '15 Minute',
  checkpoint_location = '{s3_checkpoint_location}',
  watermark_delay = '1 Minute',
  using_existing_index = 'opensearch_index_name',
  extra_options = '{ "{table_name}": { "maxFilesPerTrigger": "10" }}'
)

For the index name to be compliant with Flint's naming convention - the MV name would be added to the index as an index alias

Do you have any additional context?


@YANG-DB YANG-DB added enhancement New feature or request untriaged labels May 29, 2024
@noCharger
Copy link
Collaborator

Although not explicitly stated in https://github.com/opensearch-project/opensearch-spark/blob/main/docs/index.md#flint-index-specification, the CREATE MATERIALIZED VIEW command does not indicate that any existing materialized view can be utilized. Alter the MV definition in with clause seems to be acheivable.

@penghuo
Copy link
Collaborator

penghuo commented May 29, 2024

Try to understand the use case of create MV with indexing index. and what is expected workflow?

existing index template contains specific data-types supported only in opensearch

We can extend Spark data type to support OpenSearch data type.

existing index contains data that needs to be appended with the MV query results

UNION can solve this problem.

@dai-chen dai-chen removed the untriaged label Jun 3, 2024
@dai-chen
Copy link
Collaborator

Closing as duplicate as #72.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants