Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 0.1] Fix recover index bug when Flint data index is deleted accidentally #247

Merged
merged 2 commits into from
Feb 14, 2024

Conversation

dai-chen
Copy link
Collaborator

@dai-chen dai-chen commented Feb 7, 2024

Backport f4744ab from #241.

Note that the original PR depends on metadata log purge functionality introduced in #189. To avoid the possible impact and API changes, I only cherry-picked minimal code changes instead of the complete #189.

CREATE SKIPPING INDEX ON stream.lineitem_tiny
(l_shipdate VALUE_SET)
WITH ( auto_refresh = true );

# Delete Flint data index
DELETE flint_myglue_stream_lineitem_tiny_skipping_index

# Check index state metadata log
GET .query_execution_request_myglue/_search
      {
        "_index": ".query_execution_request_myglue",
        "_id": "ZmxpbnRfbXlnbHVlX3N0cmVhbV9saW5laXRlbV90aW55X3NraXBwaW5nX2luZGV4",
        "_score": 1,
        "_source": {
          "version": "1.0",
          "latestId": "ZmxpbnRfbXlnbHVlX3N0cmVhbV9saW5laXRlbV90aW55X3NraXBwaW5nX2luZGV4",
          "type": "flintindexstate",
          "state": "refreshing",
          "applicationId": "unknown",
          "jobId": "unknown",
          "dataSourceName": "myglue",
          "jobStartTime": 1706916312007,
          "lastUpdateTime": 1706916452474,
          "error": ""
        }
      }

spark-sql> RECOVER INDEX JOB flint_myglue_stream_lineitem_tiny_skipping_index;
24/02/02 23:48:31 WARN FlintSpark: Cleaning up metadata log as index data has been deleted

# The metadata log is gone
GET .query_execution_request_myglue/_search

…pensearch-project#241)

* Clean up metadata log in recover index API

Signed-off-by: Chen Dai <[email protected]>

* Await termination only if there is streaming job running

Signed-off-by: Chen Dai <[email protected]>

* Update user manual

Signed-off-by: Chen Dai <[email protected]>

---------

Signed-off-by: Chen Dai <[email protected]>
@dai-chen dai-chen added the 0.1.1 label Feb 7, 2024
@dai-chen dai-chen self-assigned this Feb 7, 2024
@dai-chen dai-chen marked this pull request as ready for review February 8, 2024 22:58
@penghuo penghuo merged commit 08bc3b2 into opensearch-project:0.1 Feb 14, 2024
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants