-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add fetch
command to download expedition data based on space_time_region
from schedule.yaml
#83
Open
iuryt
wants to merge
70
commits into
OceanParcels:main
Choose a base branch
from
iuryt:fetch_its
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
70 commits
Select commit
Hold shift + click to select a range
7ac5d4d
add bbox and time range
iuryt 8c641cc
add AreaOfInterest class
iuryt 6b6b874
export AreaOfInterest
iuryt 6743f1f
add AreaOfInterest to Schedule class
iuryt 40d4d0d
fetch function for downloading data based on the area of interest
iuryt 3194056
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] aacbb38
import copernicusmarine
iuryt 5988c90
import datetime
iuryt fe77ea1
fix conflict
iuryt 3d7697e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] fafc74f
fix typo
iuryt ad97b93
fix conflict
iuryt 46af584
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] e092658
Update src/virtualship/cli/commands.py
iuryt bb776a6
Update src/virtualship/cli/commands.py
iuryt 870a2c2
patch `path` varname
VeckoTheGecko 8df718a
update _get_ship_config and _get_schedule
VeckoTheGecko c691a47
Add credential config
VeckoTheGecko 5fbf812
Add note to obtain credentials
VeckoTheGecko 391f5ef
Add copernicusmarine dep
VeckoTheGecko 17fef2b
Patch types
VeckoTheGecko 672fa6a
Update user messages in `virtualship fetch`
VeckoTheGecko 51451cf
Adding minimum and maximum depth to area_of_interest
erikvansebille bc88360
Setting None as default option for drifter endtime
erikvansebille f3a0dc9
Removing redundant import in text_drifter
erikvansebille 3b5edc6
Pin copernicusmarine < 2
VeckoTheGecko fb1effd
Validate domains for area_of_interest
VeckoTheGecko 97b4d8d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] bb8bacd
Make depth optional in AreaOfInterest
VeckoTheGecko 43f3329
Make AreaOfInterest optional in Schedule
VeckoTheGecko 368bd64
Rename AreaOfInterest to SpaceTimeRegion
VeckoTheGecko 123ad3d
Rename area_of_interest.py file
VeckoTheGecko 471d43a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] ef1e6bc
Add fetch utility functions and tests
VeckoTheGecko 3bb4e6a
Patch attr name
VeckoTheGecko 737e34f
pointing the right function to make it work
iuryt 6b2eb20
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 60952d8
remove temporary code
iuryt 1854258
fix typo
iuryt 77d40ab
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] ac61cf1
ask for username and password if not provided
iuryt 61ab177
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 34bea16
creates function that makes a sha256 from a dictionary
iuryt a125a11
create a data folder based on the hash from area_of_interest
iuryt 61a9c32
merge with pre-commit ¬¬
iuryt a2a61c8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 777f505
insert blank lines
iuryt f6908df
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 2a3c9ab
Update hash to use _fetch utils
VeckoTheGecko 66a1bb9
Avoid mixing matching creds
VeckoTheGecko 3012812
Download caching feature
VeckoTheGecko 4160437
Update input data loading with new data folder structure
VeckoTheGecko fd060d7
Handle download_cleanup on wrong credentials
VeckoTheGecko d2a7bc8
Copy schedule to download folde
VeckoTheGecko 18b86ec
Patch download command
VeckoTheGecko 524c650
Add `input_data` param for testing
VeckoTheGecko ad995c3
Add tests
VeckoTheGecko afe6112
Add fetch test
VeckoTheGecko 5669e41
Pin copernicusmarine >= 2
VeckoTheGecko bcfe01e
RTD explicit config key
VeckoTheGecko 0ef7cf0
Delete download_data script
VeckoTheGecko b830393
Update pyproject.toml
VeckoTheGecko 8d35d4b
Improve filename to hash conversion
VeckoTheGecko 9ffd8de
Rename to assert_complete_download
VeckoTheGecko cfc50cc
Error message when area of interest isn't defined
VeckoTheGecko 7132766
Add area of interest hash salting
VeckoTheGecko 8698d13
Update 'area of interest' to 'space-time region' throughout
VeckoTheGecko 04d7c13
Avoid circular import
VeckoTheGecko 93576c8
virtualship help documentation
VeckoTheGecko a2290b6
Update help messages
VeckoTheGecko File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -11,6 +11,7 @@ dependencies: | |
- pydantic >=2, <3 | ||
- pip | ||
- pyyaml | ||
- copernicusmarine >= 2 | ||
|
||
# linting | ||
- pre-commit | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,107 @@ | ||
from __future__ import annotations | ||
|
||
from pathlib import Path | ||
|
||
import click | ||
import pydantic | ||
import yaml | ||
|
||
CREDENTIALS_FILE = "credentials.yaml" | ||
|
||
|
||
class CredentialFileError(Exception): | ||
"""Exception raised for errors in the input file format.""" | ||
|
||
pass | ||
|
||
|
||
class Credentials(pydantic.BaseModel): | ||
"""Credentials to be used in `virtualship fetch` command.""" | ||
|
||
COPERNICUS_USERNAME: str | ||
COPERNICUS_PASSWORD: str | ||
|
||
@classmethod | ||
def from_yaml(cls, path: str | Path) -> Credentials: | ||
""" | ||
Load credentials from a yaml file. | ||
|
||
:param path: Path to the file to load from. | ||
:returns Credentials: The credentials. | ||
""" | ||
with open(path) as file: | ||
data = yaml.safe_load(file) | ||
|
||
if not isinstance(data, dict): | ||
raise CredentialFileError("Credential file is of an invalid format.") | ||
|
||
return cls(**data) | ||
|
||
def dump(self) -> str: | ||
""" | ||
Dump credentials to a yaml string. | ||
|
||
:param creds: The credentials to dump. | ||
:returns str: The yaml string. | ||
""" | ||
return yaml.safe_dump(self.model_dump()) | ||
|
||
def to_yaml(self, path: str | Path) -> None: | ||
""" | ||
Write credentials to a yaml file. | ||
|
||
:param path: Path to the file to write to. | ||
""" | ||
with open(path, "w") as file: | ||
file.write(self.dump()) | ||
|
||
|
||
def get_dummy_credentials_yaml() -> str: | ||
return ( | ||
Credentials( | ||
COPERNICUS_USERNAME="my_username", COPERNICUS_PASSWORD="my_password" | ||
) | ||
.dump() | ||
.strip() | ||
) | ||
|
||
|
||
def get_credentials_flow( | ||
username: str | None, password: str | None, creds_path: Path | ||
) -> tuple[str, str]: | ||
""" | ||
Execute flow of getting credentials for use in the `fetch` command. | ||
|
||
- If username and password are provided via CLI, use them (ignore the credentials file if exists). | ||
- If username and password are not provided, try to load them from the credentials file. | ||
- If no credentials are provided, print a message on how to make credentials file and prompt for credentials. | ||
|
||
:param username: The username provided via CLI. | ||
:param password: The password provided via CLI. | ||
:param creds_path: The path to the credentials file. | ||
""" | ||
if username and password: | ||
if creds_path.exists(): | ||
click.echo( | ||
f"Credentials file exists at '{creds_path}', but username and password are already provided. Ignoring credentials file." | ||
) | ||
return username, password | ||
|
||
try: | ||
creds = Credentials.from_yaml(creds_path) | ||
click.echo(f"Loaded credentials from '{creds_path}'.") | ||
return creds.COPERNICUS_USERNAME, creds.COPERNICUS_PASSWORD | ||
except FileNotFoundError: | ||
msg = f"""Credentials not provided. Credentials can be obtained from https://data.marine.copernicus.eu/register. Either pass in via `--username` and `--password` arguments, or via config file at '{creds_path}'. Config file should be YAML along following format: | ||
### {creds_path} | ||
|
||
{get_dummy_credentials_yaml().strip()} | ||
|
||
### | ||
VeckoTheGecko marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Prompting for credentials instead... | ||
""" | ||
click.echo(msg) | ||
username = click.prompt("username") | ||
password = click.prompt("password", hide_input=True) | ||
return username, password |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe explain what CLI is? Mot sure every user will know that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was intended more as developer documentation rather than user documentation. What do we think in terms of user documentation? Perhaps I can write a documentation page? (I imagine that we wouldn't want this information silo'd in a notebook, but rather link out to the doc page - I imagine there would be other course content in a notebook). Thoughts on how this fits with course content structure @ammedd?
Updating the docstring here wont appear to users unless its also displayed in some sort of user documentation.
Lets put that doc page in a different PR, this is already big enough and getting a bit much 😅