Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Flint index state management #95

Closed
Tracked by #3
penghuo opened this issue Oct 23, 2023 · 2 comments
Closed
Tracked by #3

[FEATURE] Flint index state management #95

penghuo opened this issue Oct 23, 2023 · 2 comments
Assignees
Labels
0.1.1 enhancement New feature or request

Comments

@penghuo
Copy link
Collaborator

penghuo commented Oct 23, 2023

  • CREATE INDEX, Spark streaming Job,
    1. CREATE NEW DOC in .query_execution_request_{mys3}, write docId into system.env.
    2. Flint-Extension, read system.env.

  type: streaming
    state = "running"
    jobId = “job1”
    appId = “app1”
    lastupdateTime = 0
  • RECOVER INDEX
    1. CREATE NEW DOC in .query_execution_request_{mys3}, write docId into system.env.
    2. Flint-Extension, read system.env.

    docID = FlintExt.state().docId()
    if (state == "running") {
      // occ
      updateState(indices, docId): {
        type: streaming
        state = "running"
        jobId = “job2”
        appId = “app2”
        lastupdateTime = 0
      }
    }
  • MANUAL REFRESH
    1. CREATE NEW DOC in .query_execution_request_{mys3}, write docId into system.env.
    2. Flint-Extension, read system.env.

    type: streaming
      state = "running"
      jobId = “job3”
      appId = “app3”
      lastupdateTime = 0
@penghuo penghuo added enhancement New feature or request untriaged 0.1.1 and removed untriaged labels Oct 23, 2023
@penghuo penghuo changed the title [FEATURE] Streaming Job support Blue/Green upgrade [FEATURE] Streaming Job support state maintain Oct 23, 2023
@penghuo
Copy link
Collaborator Author

penghuo commented Oct 24, 2023

  • index: read fome spark.flint.job.requestIndex
  • docId is stored in flint-index-metadata.
  • streaming doc
type: streaming
state = "running"
jobId = “jobId”
appId = “appId”
lastupdateTime = 0

@dai-chen dai-chen self-assigned this Oct 26, 2023
@dai-chen dai-chen changed the title [FEATURE] Streaming Job support state maintain [FEATURE] Flint index state management Oct 31, 2023
@dai-chen
Copy link
Collaborator

Closing. RECOVER command changes will be covered in #57.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.1.1 enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants