Terminate streaming job when index data is deleted #500
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dai-chen
force-pushed
the
terminate-streaming-job-if-data-deleted
branch
2 times, most recently
from
a5fc8cc to
30d0820
Compare
dai-chen
requested review from
rupal-bq,
vmmusings,
penghuo,
seankao-az,
anirudha,
kaituo and
YANG-DB
as code owners
noCharger
reviewed
Aug 5, 2024
integ-test/src/test/scala/org/opensearch/flint/spark/FlintSparkIndexMonitorITSuite.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Chen Dai <[email protected]>
Signed-off-by: Chen Dai <[email protected]>
penghuo
reviewed
Aug 6, 2024
flint-spark-integration/src/main/scala/org/opensearch/flint/spark/FlintSparkIndexMonitor.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Chen Dai <[email protected]>
dai-chen
force-pushed
the
terminate-streaming-job-if-data-deleted
branch
from
30d0820 to
68d4b72
Compare
noCharger
approved these changes
Aug 8, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds the capability to terminate the streaming job when the associated Flint data index no longer exists. The termination is managed by the index monitor rather than the Spark streaming listener, ensuring that the job can be terminated even if it remains idle without any new micro-batches being triggered.
Please note that this PR solely addresses the termination of the streaming job and the index monitor itself. The handling of dangling metadata log entries will be addressed in the upcoming PR for issue #356.
Issues Resolved
#244
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.