Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add USING statement for covering / MV indices for using existing inde… #77

Conversation

YANG-DB
Copy link
Member

@YANG-DB YANG-DB commented Oct 18, 2023

Is your feature request related to a problem?
As a flint user, I'd like to use existing indices / index-templates for using as the index targets of the flint accelerated tables

What solution would you like?
Use existing index name to create the acceleration process - this will actually not create an index but use the given name as the target of the acceleration ETL store.

The next SQL syntax suggested:

CREATE (SKIPPING/COVERING/MV) INDEX
ON alb_logs USING ss4o_logs-elb-prod
WITH (
  auto_refresh = true,
  refresh_interval = '1 minute',
  checkpoint_location = 's3://test/'
)

Would initiate the acceleration ETL sync process without actually creating a new index in OpenSearch, it will use the ss4o_logs-elb-prod index (index template) as the data store for the acceleration content.

It may validate the following:

  • match source table (catalog) columns exist in the given index_mapping
  • match the index has an alias with the correct flint naming convention
  • match the index_template (mapping) has the correct covering - matadata fields exist (covering / MV columns)

Issues Resolved

#72

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…x / index template in opensearch

Signed-off-by: YANGDB <[email protected]>
add metadata creation parameters

Signed-off-by: YANGDB <[email protected]>
@dai-chen
Copy link
Collaborator

dai-chen commented Oct 18, 2023

Could you explain the semantics of this new syntax? So user already has index with Flint metadata, this actually does not CREATE but just alias their index name with expected Flint name. Why this has to be done in Flint extension?

@YANG-DB
Copy link
Member Author

YANG-DB commented Oct 19, 2023

this is to be done in flint since we do need to add the metadata section to an existing opensearch index
it will take an existing index and add the correct flint name (as an alias) according to the convention and the _meta columns that are accelerated

Using the same syntax as the existing SQL DDL statement (CREATE ... INDEX ON ... USING ...) will simplify the user process of enabling existing indices for supporting the MV / Covering accelerations.

In addition it will decouple the need for flint to be aware of the target index since it will use the alias name as the flint name convention thus making the flint actually agnostic whether the index was created directly or existed earlier

@dai-chen
Copy link
Collaborator

dai-chen commented Oct 19, 2023

I see. If so, could you do it as how we support index_settings option? I think you just need to add one more alias option in FlintSparkIndexOption and use it in FlintClient and FlintMetadata. And in existing client.createIndex()we just need to pass alias along with mapping and setting? Ref: https://opensearch.org/docs/latest/api-reference/index-apis/create-index/#request-body

PR for your reference: #44

@YANG-DB YANG-DB marked this pull request as ready for review October 19, 2023 17:22
@YANG-DB
Copy link
Member Author

YANG-DB commented Oct 20, 2023

remove this PR since the functionality will be moved to materialized view

@YANG-DB YANG-DB closed this Oct 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants