Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CLI option to use the s3 file enlisting instead of the modified date to create daily/monthly files #70

Merged
merged 5 commits into from
Jan 22, 2024

Conversation

stijnvanhoey
Copy link
Collaborator

I could not fully test the concept due to access rights, but I could check the reading part of the public bucket. This PR adds an option path-s3-folder to recreate all daily/monthlmy files within a given S3 path, e.g. vph5_to_vpts --path-s3-folder uva/hdf5/nldhl/2019. It scans all folder recursively for h5 files and uses this as input to define the days to recreate vpts files.

@peterdesmet certainly open to a better name for the new CLI option :-)

@peterdesmet
Copy link
Member

@stijnvanhoey Picking this up again 😅

  1. I don't understand why the tests in this PR are failing for the last commit (which is a typo correction). Dependency changes?
  2. I tried locally, and the test are even failing on the main branch (7 failed, 84 passed)
  3. Once tests pass, how can I install and test the functionality of this PR?

@stijnvanhoey
Copy link
Collaborator Author

stijnvanhoey commented Dec 11, 2023

@stijnvanhoey Picking this up again 😅

1. I don't understand why the tests in this PR are failing for the last commit (which is a typo correction). Dependency changes?

2. I tried locally, and the test are even failing on the `main` branch (7 failed, 84 passed)

Is probably due to adjustment in one of the dependencies as it seems to fail on the pytest fixture I setup to mock AWS calls, see https://github.com/enram/vptstools/blob/main/tests/conftest.py#L71-L98 as all failing tests define

self = <[AttributeError("'MockHttpClientResponse' object has no attribute '_cache'") raised in repr()] MockHttpClientResponse object at 0x7fd75aaf0e50>

or

 E       AttributeError: 'MockHttpClientResponse' object has no attribute '_loop'

When testing with aiobotocore=2.5.0 (via s3fs 2023.5.0), these tests pass. I fixed the dependency to check in CI

@stijnvanhoey
Copy link
Collaborator Author

3. Once tests pass, how can I install and test the functionality of this PR?

afaik you can do custom runs of the vph5_to_vpts command in the uat/prd versions of the AWS deployment and pass the command to run. So instead of the default command that runs daily (vph5_to_vpts --modified-days-ago 3), you pass the command with the S3-path to run the conversion, e.g. vph5_to_vpts --path-s3-folder uva/hdf5/nldhl/2019 (see also vph5_to_vpts --help)

@peterdesmet peterdesmet merged commit 6173b18 into main Jan 22, 2024
4 checks passed
@peterdesmet peterdesmet deleted the SVH-key-based branch January 22, 2024 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants