Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Support index mapping option in create index statement #772

Open
dai-chen opened this issue Oct 11, 2024 · 0 comments
Open

[FEATURE] Support index mapping option in create index statement #772

dai-chen opened this issue Oct 11, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@dai-chen
Copy link
Collaborator

Is your feature request related to a problem?

Currently, users cannot customize the index mapping for the OpenSearch index that stores the Flint index data created via Spark SQL.

What solution would you like?

Similarly as index_settings option introduced previously, a new option index_mappings can be added to the create statement in Spark SQL. This would allow users fine grain control over the OpenSearch index mappings.

For example:

CREATE MATERIALIZED VIEW mv_test
AS
SELECT ... 
WITH (
  index_mappings: '{ "_source": { "enabled": false } }'
)

Note that for the implementation, the index schema generated by the Flint index must be merged with the schema provided in the index_mappings option.

What alternatives have you considered?

Manually modifying the OpenSearch index after the materialized view is created. However, this approach won’t work for configurations that need to be set before indexing begins.

Do you have any additional context?

This feature would provide users the flexibility to optimize the OpenSearch index according to their specific needs. For example, it would allow them to:

  1. Disable the _source field to save space.
  2. Disable doc_values for specific fields to save space.
  3. Force certain field types (e.g., IP type) as a temporary workaround until SparkSQL supports them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant