Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Skipping Index does not apply to direct query #138

Closed
penghuo opened this issue Nov 7, 2023 · 0 comments
Closed

[BUG] Skipping Index does not apply to direct query #138

penghuo opened this issue Nov 7, 2023 · 0 comments
Assignees
@kaituo
Labels
0.1.1 bug Something isn't working

Comments

@penghuo
Copy link
Collaborator

penghuo commented Nov 7, 2023

What is the bug?
Skipping Index does not apply to direct query.

Log

23/11/07 06:57:35 INFO FlintSpark: Describing index name flint_spark_catalog_default_http_logs_150_skipping_index
23/11/07 06:57:35 INFO FlintOpenSearchClient: Checking if Flint index exists flint_spark_catalog_default_http_logs_150_skipping_index

Solution

  1. Short-term, add --conf spark.sql.defaultCatalog={datasource} in FlintREPL job.
val dataSource = conf.get("spark.flint.datasource.name", "unknown")
conf.set('spark.sql.defaultCatalog', dataSource)
  1. Long-term, plugin should add conf when submit job [BUG] Missing --conf spark.sql.defaultCatalog={datasource} when submit EMR-S job sql#2421.
@penghuo penghuo added bug Something isn't working untriaged 0.1.1 labels Nov 7, 2023
@kaituo kaituo removed the untriaged label Nov 7, 2023
@kaituo kaituo self-assigned this Nov 7, 2023
kaituo added a commit that referenced this issue Nov 8, 2023
@kaituo
Implement blue/green deployment support in REPL mode
2601cfb 
Features:
- Add mutual exclusivity in session-based emr-s jobs to ensure only one job runs at a time, enhancing system stability during blue/green deployments. This allows active job exclusion and seamless task pickup by new jobs. Details in #94.

Fixes:
- Resolve a bug where long-running queries failed to cancel post-index mapping verification, by introducing timely query cancellation checks within the REPL loop.
- Address issue #138 with the proposed short-term fix, improving reliability.

Tests:
- Conducted manual testing to validate blue/green deployment support and query cancellation.
- Extended unit tests to cover new features and bug fixes.

Signed-off-by: Kaituo Li <[email protected]>
@kaituo kaituo mentioned this issue Nov 8, 2023
Merged
kaituo added a commit that referenced this issue Nov 8, 2023
@kaituo
Implement blue/green deployment support in REPL mode
655075f 
Features:
- Add mutual exclusivity in session-based emr-s jobs to ensure only one job runs at a time, enhancing system stability during blue/green deployments. This allows active job exclusion and seamless task pickup by new jobs. Details in #94.

Fixes:
- Resolve a bug where long-running queries failed to cancel post-index mapping verification, by introducing timely query cancellation checks within the REPL loop.
- Address issue #138 with the proposed short-term fix, improving reliability.

Tests:
- Conducted manual testing to validate blue/green deployment support and query cancellation.
- Extended unit tests to cover new features and bug fixes.

Signed-off-by: Kaituo Li <[email protected]>
penghuo pushed a commit that referenced this issue Nov 11, 2023
@kaituo
Implement blue/green deployment support in REPL mode (#141)
1f31e56 
* Implement blue/green deployment support in REPL mode

Features:
- Add mutual exclusivity in session-based emr-s jobs to ensure only one job runs at a time, enhancing system stability during blue/green deployments. This allows active job exclusion and seamless task pickup by new jobs. Details in #94.

Fixes:
- Resolve a bug where long-running queries failed to cancel post-index mapping verification, by introducing timely query cancellation checks within the REPL loop.
- Address issue #138 with the proposed short-term fix, improving reliability.

Tests:
- Conducted manual testing to validate blue/green deployment support and query cancellation.
- Extended unit tests to cover new features and bug fixes.

Signed-off-by: Kaituo Li <[email protected]>

* add jobStartTime in session doc

Signed-off-by: Kaituo Li <[email protected]>

* read excluded jobs and customize timeout setting

Signed-off-by: Kaituo Li <[email protected]>

---------

Signed-off-by: Kaituo Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kaituo kaituo

Labels
0.1.1 bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@penghuo @kaituo @dai-chen