[Backport 2.x] [Writable Warm] Composite Directory implementation and integrating it with FileCache #14489
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… with FileCache (opensearch-project#12782) * Composite Directory POC Signed-off-by: Shreyansh Ray <[email protected]> * Refactor TransferManager interface to RemoteStoreFileTrackerAdapter Signed-off-by: Shreyansh Ray <[email protected]> * Implement block level fetch for Composite Directory Signed-off-by: Shreyansh Ray <[email protected]> * Removed CACHE state from FileTracker Signed-off-by: Shreyansh Ray <[email protected]> * Fixes after latest pull Signed-off-by: Shreyansh Ray <[email protected]> * Add new setting for warm, remove store type setting, FileTracker and RemoteStoreFileTrackerAdapter, CompositeDirectoryFactory and update Composite Directory implementation Signed-off-by: Shreyansh Ray <[email protected]> * Modify TransferManager - replace BlobContainer with Functional Interface to fetch an InputStream instead Signed-off-by: Shreyansh Ray <[email protected]> * Reuse OnDemandBlockSnapshotIndexInput instead of OnDemandBlockCompositeIndexInput Signed-off-by: Shreyansh Ray <[email protected]> * Modify constructors to avoid breaking public api contract and code review fixes Signed-off-by: Shreyansh Ray <[email protected]> * Add experimental annotations for newly created classes and review comment fixes Signed-off-by: Shreyansh Ray <[email protected]> * Use ref count as a temporary measure to prevent file from eviction until uploaded to Remote Signed-off-by: Shreyansh Ray <[email protected]> * Remove method level locks Signed-off-by: Shreyansh Ray <[email protected]> * Handle tmp file deletion Signed-off-by: Shreyansh Ray <[email protected]> * Nit fixes Signed-off-by: Shreyansh Ray <[email protected]> * Handle delete and close in Composite Directory, log current state of FileCache and correct it's clear method and modify unit and integration tests as per review comments Signed-off-by: Shreyansh Ray <[email protected]> * Refactor usages of WRITEABLE_REMOTE_INDEX_SETTING to TIERED_REMOTE_INDEX_SETTING Signed-off-by: Shreyansh Ray <[email protected]> * Add tests for FileCachedIndexInput and review comment fixes Signed-off-by: Shreyansh Ray <[email protected]> * Add additional IT for feature flag disabled Signed-off-by: Shreyansh Ray <[email protected]> * Move setting for Partial Locality type behind Feature Flag, fix bug for ref count via cloneMap in FullFileCachedIndexInput and other review fixes Signed-off-by: Shreyansh Ray <[email protected]> * Minor test and nit fixes Signed-off-by: Shreyansh Ray <[email protected]> * Add javadocs for FullFileCachedIndexInput Signed-off-by: Shreyansh Ray <[email protected]> * Minor precommit fixes Signed-off-by: Shreyansh Ray <[email protected]> --------- Signed-off-by: Shreyansh Ray <[email protected]>
rayshrey
requested review from
anasalkouz,
andrross,
Bukhtawar,
CEHENKLE,
dblock,
dbwiddis,
gbbafna,
kotwanikunal,
mch2,
msfroh,
nknize,
owaiskazi19,
reta,
Rishikesh1159,
sachinpkale,
saratvemulapalli,
shwetathareja,
sohami and
VachaShah
as code owners
github-actions
bot
added
enhancement
|
❌ Gradle check result for b42a50e: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
@rayshrey There are failed test which is being added by this PR. Can you please check on that. We will need to fix the test first before merging this PR.
|
|
❌ Gradle check result for 9e678a3: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
…g exception type caught Signed-off-by: Shreyansh Ray <[email protected]>
sohami
approved these changes
Jun 24, 2024
This was referenced Jun 25, 2024
kkewwei
pushed a commit
to kkewwei/OpenSearch
that referenced
this pull request
Jul 24, 2024
… integrating it with FileCache (opensearch-project#14489) * [Writable Warm] Composite Directory implementation and integrating it with FileCache (opensearch-project#12782) * Composite Directory POC Signed-off-by: Shreyansh Ray <[email protected]> * Refactor TransferManager interface to RemoteStoreFileTrackerAdapter Signed-off-by: Shreyansh Ray <[email protected]> * Implement block level fetch for Composite Directory Signed-off-by: Shreyansh Ray <[email protected]> * Removed CACHE state from FileTracker Signed-off-by: Shreyansh Ray <[email protected]> * Fixes after latest pull Signed-off-by: Shreyansh Ray <[email protected]> * Add new setting for warm, remove store type setting, FileTracker and RemoteStoreFileTrackerAdapter, CompositeDirectoryFactory and update Composite Directory implementation Signed-off-by: Shreyansh Ray <[email protected]> * Modify TransferManager - replace BlobContainer with Functional Interface to fetch an InputStream instead Signed-off-by: Shreyansh Ray <[email protected]> * Reuse OnDemandBlockSnapshotIndexInput instead of OnDemandBlockCompositeIndexInput Signed-off-by: Shreyansh Ray <[email protected]> * Modify constructors to avoid breaking public api contract and code review fixes Signed-off-by: Shreyansh Ray <[email protected]> * Add experimental annotations for newly created classes and review comment fixes Signed-off-by: Shreyansh Ray <[email protected]> * Use ref count as a temporary measure to prevent file from eviction until uploaded to Remote Signed-off-by: Shreyansh Ray <[email protected]> * Remove method level locks Signed-off-by: Shreyansh Ray <[email protected]> * Handle tmp file deletion Signed-off-by: Shreyansh Ray <[email protected]> * Nit fixes Signed-off-by: Shreyansh Ray <[email protected]> * Handle delete and close in Composite Directory, log current state of FileCache and correct it's clear method and modify unit and integration tests as per review comments Signed-off-by: Shreyansh Ray <[email protected]> * Refactor usages of WRITEABLE_REMOTE_INDEX_SETTING to TIERED_REMOTE_INDEX_SETTING Signed-off-by: Shreyansh Ray <[email protected]> * Add tests for FileCachedIndexInput and review comment fixes Signed-off-by: Shreyansh Ray <[email protected]> * Add additional IT for feature flag disabled Signed-off-by: Shreyansh Ray <[email protected]> * Move setting for Partial Locality type behind Feature Flag, fix bug for ref count via cloneMap in FullFileCachedIndexInput and other review fixes Signed-off-by: Shreyansh Ray <[email protected]> * Minor test and nit fixes Signed-off-by: Shreyansh Ray <[email protected]> * Add javadocs for FullFileCachedIndexInput Signed-off-by: Shreyansh Ray <[email protected]> * Minor precommit fixes Signed-off-by: Shreyansh Ray <[email protected]> --------- Signed-off-by: Shreyansh Ray <[email protected]> * Fix Writable Warm test for feature flag disabled condition by changing exception type caught Signed-off-by: Shreyansh Ray <[email protected]> --------- Signed-off-by: Shreyansh Ray <[email protected]> Signed-off-by: kkewwei <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Raising a manual back-port request for #12782
Related Issues
Resolves #12781
Check List
API changes companion pull request created, if applicable.Public documentation issue/PR created, if applicable.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.