Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Remote Store] Support to emit multiple streams for a file content each responsible for processing a specific part of the file #6977

Conversation

raghuvanshraj
Copy link
Contributor

@raghuvanshraj raghuvanshraj commented Apr 4, 2023

Description

  • Offset based InputStream extensions to emit stream from a specific part of a file, which starts reading from a specific position and ensures that maximum length of content read doesn't exceed a specified limit. OffsetRangeFileInputStream achieves this for File objects, while OffsetRangeIndexInputStream achieves this for lucene's IndexInput construct
  • RemoteTransferContainer has utilities to open streams to specific parts of the file based on the type as mentioned in the previous point. It also manages post upload tasks by implementing an UploadFinalizer.
  • ResettableCheckedInputStream allows for individual parts to be reset through mark and reset in the event of upload failures

Issues Resolved

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 4, 2023

Gradle Check (Jenkins) Run Completed with:

@codecov-commenter
Copy link

Codecov Report

Merging #6977 (5b252f0) into main (1856090) will decrease coverage by 0.06%.
The diff coverage is 53.14%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6977      +/-   ##
============================================
- Coverage     70.71%   70.65%   -0.06%     
+ Complexity    59261    59249      -12     
============================================
  Files          4813     4822       +9     
  Lines        283767   283974     +207     
  Branches      40919    40935      +16     
============================================
- Hits         200656   200637      -19     
- Misses        66645    66882     +237     
+ Partials      16466    16455      -11     
Impacted Files Coverage Δ
...ch/common/blobstore/stream/write/WriteContext.java 0.00% <0.00%> (ø)
.../transfer/stream/ResettableCheckedInputStream.java 32.14% <32.14%> (ø)
...re/transfer/stream/OffsetRangeFileInputStream.java 34.88% <34.88%> (ø)
...e/transfer/stream/OffsetRangeIndexInputStream.java 53.33% <53.33%> (ø)
...on/blobstore/transfer/RemoteTransferContainer.java 65.43% <65.43%> (ø)
...er/src/main/java/org/opensearch/common/Stream.java 87.50% <87.50%> (ø)
...ain/java/org/opensearch/common/StreamProvider.java 100.00% <100.00%> (ø)
...nsearch/common/blobstore/stream/StreamContext.java 100.00% <100.00%> (ø)
...h/common/blobstore/stream/write/WritePriority.java 100.00% <100.00%> (ø)

... and 479 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 4, 2023

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-core-1 branch from f6dcb01 to 68fef72 Compare April 4, 2023 16:33
@github-actions
Copy link
Contributor

github-actions bot commented Apr 4, 2023

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-core-1 branch from 68fef72 to f1f03c7 Compare April 5, 2023 03:49
@raghuvanshraj raghuvanshraj changed the title Adding new abstractions in server module to support multipart upload [Remote Store] Support to emit multiple streams for a file content each responsible for processing a specific part of the file Apr 5, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

…for processing a specific part of the file

Signed-off-by: Raghuvansh Raj <[email protected]>
Signed-off-by: Raghuvansh Raj <[email protected]>
@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-core-1 branch from f1f03c7 to 9dbf037 Compare April 5, 2023 09:09
@github-actions
Copy link
Contributor

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants