Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EPIC] Zero-ETL - OpenSearch Table #185

Open
4 of 10 tasks
penghuo opened this issue Dec 8, 2023 · 1 comment
Open
4 of 10 tasks

[EPIC] Zero-ETL - OpenSearch Table #185

penghuo opened this issue Dec 8, 2023 · 1 comment
Labels

Comments

@penghuo
Copy link
Collaborator

penghuo commented Dec 8, 2023

This is meta issue to track the tasks of opensearch-project/OpenSearch#14524

Milestone I - Direct access OpenSearch index in Spark SQL

In milestone I, we want to achieve (1) Support direct query opensearch index use case. (2) Improve performance. #334. The task includes

@penghuo penghuo added enhancement New feature or request untriaged opensearch and removed untriaged labels Dec 8, 2023
@penghuo penghuo removed the untriaged label Dec 8, 2023
@penghuo penghuo changed the title [EPIC] Zero-ETL - Support OpenSearch Table in AWS Glue [EPIC] Zero-ETL - OpenSearch Table Jun 24, 2024
@dai-chen
Copy link
Collaborator

dai-chen commented Sep 9, 2024

Another requirement is to support writing to an OpenSearch table, allowing users to use the INSERT statement. This would enable users to control when and what data is loaded into the OpenSearch index.

For example:

# First batch
INSERT INTO os_table SELECT ... FROM logs WHERE id BETWEEN 1 AND 100;

# Second batch
INSERT INTO os_table SELECT ... FROM logs WHERE id BETWEEN 101 AND 200;

This is not a use case for Flint's materialized view feature. Currently there is no way to achieve in Flint Spark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants