[Writable Warm] Composite Directory implementation and integrating it with FileCache#12782

Merged

sohami merged 22 commits intoopensearch-project:mainfrom rayshrey:composite-directory-poc

Jun 20, 2024

+1,889-208

Commits on Jun 20, 2024

Composite Directory POC
rayshrey
committed
Refactor TransferManager interface to RemoteStoreFileTrackerAdapter
rayshrey
committed
Implement block level fetch for Composite Directory
rayshrey
committed
Removed CACHE state from FileTracker
rayshrey
committed
Fixes after latest pull
rayshrey
committed
Add new setting for warm, remove store type setting, FileTracker and RemoteStoreFileTrackerAdapter, CompositeDirectoryFactory and update Composite Directory implementation
rayshrey
committed
Modify TransferManager - replace BlobContainer with Functional Interface to fetch an InputStream instead
rayshrey
committed
Reuse OnDemandBlockSnapshotIndexInput instead of OnDemandBlockCompositeIndexInput
rayshrey
committed
Modify constructors to avoid breaking public api contract and code review fixes
rayshrey
committed
Add experimental annotations for newly created classes and review comment fixes
rayshrey
committed
Use ref count as a temporary measure to prevent file from eviction until uploaded to Remote
rayshrey
committed
Remove method level locks
rayshrey
committed
Handle tmp file deletion
rayshrey
committed
Nit fixes
rayshrey
committed
Handle delete and close in Composite Directory, log current state of FileCache and correct it's clear method and modify unit and integration tests as per review comments
rayshrey
committed
Refactor usages of WRITEABLE_REMOTE_INDEX_SETTING to TIERED_REMOTE_INDEX_SETTING
rayshrey
committed
Add tests for FileCachedIndexInput and review comment fixes
rayshrey
committed
Add additional IT for feature flag disabled
rayshrey
committed
Move setting for Partial Locality type behind Feature Flag, fix bug for ref count via cloneMap in FullFileCachedIndexInput and other review fixes
rayshrey
committed
Minor test and nit fixes
rayshrey
committed
Add javadocs for FullFileCachedIndexInput
rayshrey
committed
Minor precommit fixes
rayshrey
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Writable Warm] Composite Directory implementation and integrating it with FileCache#12782

Commits on Jun 20, 2024

Composite Directory POC

Refactor TransferManager interface to RemoteStoreFileTrackerAdapter

Implement block level fetch for Composite Directory

Removed CACHE state from FileTracker

Fixes after latest pull

Add new setting for warm, remove store type setting, FileTracker and RemoteStoreFileTrackerAdapter, CompositeDirectoryFactory and update Composite Directory implementation

Modify TransferManager - replace BlobContainer with Functional Interface to fetch an InputStream instead

Reuse OnDemandBlockSnapshotIndexInput instead of OnDemandBlockCompositeIndexInput

Modify constructors to avoid breaking public api contract and code review fixes

Add experimental annotations for newly created classes and review comment fixes

Use ref count as a temporary measure to prevent file from eviction until uploaded to Remote

Remove method level locks

Handle tmp file deletion

Nit fixes

Handle delete and close in Composite Directory, log current state of FileCache and correct it's clear method and modify unit and integration tests as per review comments

Refactor usages of WRITEABLE_REMOTE_INDEX_SETTING to TIERED_REMOTE_INDEX_SETTING

Add tests for FileCachedIndexInput and review comment fixes

Add additional IT for feature flag disabled

Move setting for Partial Locality type behind Feature Flag, fix bug for ref count via cloneMap in FullFileCachedIndexInput and other review fixes

Minor test and nit fixes

Add javadocs for FullFileCachedIndexInput

Minor precommit fixes