Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add reader for S3 #26

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add reader for S3 #26

wants to merge 1 commit into from

Conversation

gauteh
Copy link
Owner

@gauteh gauteh commented Aug 17, 2023

The performance of this reader is probably going to depend on:

  • batching S3 requests
  • how the index is created (how does h5py do it?)
  • how the index is stored along with the data

Making the requests async, and decompression sync is probably the best strategy when that time comes.

Benchmark idea:

  • Set up Thredds and Minio in docker serving the file.

@gauteh gauteh changed the title set up s3 reader Add reader for S3 Aug 17, 2023
@codecov-commenter
Copy link

Codecov Report

Merging #26 (876cdd2) into main (0e63714) will decrease coverage by 0.27%.
The diff coverage is 0.00%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##             main      #26      +/-   ##
==========================================
- Coverage   57.00%   56.73%   -0.27%     
==========================================
  Files          21       22       +1     
  Lines        1056     1061       +5     
==========================================
  Hits          602      602              
- Misses        454      459       +5     
Files Changed Coverage Δ
src/reader/s3.rs 0.00% <0.00%> (ø)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants