Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fetch command to download expedition data based on space_time_region from schedule.yaml #83

Open
wants to merge 70 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
7ac5d4d
add bbox and time range
iuryt Nov 13, 2024
8c641cc
add AreaOfInterest class
iuryt Nov 13, 2024
6b6b874
export AreaOfInterest
iuryt Nov 13, 2024
6743f1f
add AreaOfInterest to Schedule class
iuryt Nov 13, 2024
40d4d0d
fetch function for downloading data based on the area of interest
iuryt Nov 13, 2024
3194056
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 13, 2024
aacbb38
import copernicusmarine
iuryt Nov 13, 2024
5988c90
import datetime
iuryt Nov 13, 2024
fe77ea1
fix conflict
iuryt Nov 13, 2024
3d7697e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 13, 2024
fafc74f
fix typo
iuryt Nov 13, 2024
ad97b93
fix conflict
iuryt Nov 13, 2024
46af584
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 13, 2024
e092658
Update src/virtualship/cli/commands.py
iuryt Nov 15, 2024
bb776a6
Update src/virtualship/cli/commands.py
iuryt Nov 15, 2024
870a2c2
patch `path` varname
VeckoTheGecko Nov 21, 2024
8df718a
update _get_ship_config and _get_schedule
VeckoTheGecko Nov 21, 2024
c691a47
Add credential config
VeckoTheGecko Nov 21, 2024
5fbf812
Add note to obtain credentials
VeckoTheGecko Nov 21, 2024
391f5ef
Add copernicusmarine dep
VeckoTheGecko Nov 21, 2024
17fef2b
Patch types
VeckoTheGecko Nov 21, 2024
672fa6a
Update user messages in `virtualship fetch`
VeckoTheGecko Nov 21, 2024
51451cf
Adding minimum and maximum depth to area_of_interest
erikvansebille Nov 22, 2024
bc88360
Setting None as default option for drifter endtime
erikvansebille Nov 22, 2024
f3a0dc9
Removing redundant import in text_drifter
erikvansebille Nov 22, 2024
3b5edc6
Pin copernicusmarine < 2
VeckoTheGecko Nov 22, 2024
fb1effd
Validate domains for area_of_interest
VeckoTheGecko Dec 17, 2024
97b4d8d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 17, 2024
bb8bacd
Make depth optional in AreaOfInterest
VeckoTheGecko Dec 17, 2024
43f3329
Make AreaOfInterest optional in Schedule
VeckoTheGecko Dec 17, 2024
368bd64
Rename AreaOfInterest to SpaceTimeRegion
VeckoTheGecko Dec 17, 2024
123ad3d
Rename area_of_interest.py file
VeckoTheGecko Dec 17, 2024
471d43a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 17, 2024
ef1e6bc
Add fetch utility functions and tests
VeckoTheGecko Dec 17, 2024
3bb4e6a
Patch attr name
VeckoTheGecko Dec 17, 2024
737e34f
pointing the right function to make it work
iuryt Dec 21, 2024
6b2eb20
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 21, 2024
60952d8
remove temporary code
iuryt Dec 21, 2024
1854258
fix typo
iuryt Dec 21, 2024
77d40ab
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 21, 2024
ac61cf1
ask for username and password if not provided
iuryt Dec 21, 2024
61ab177
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 21, 2024
34bea16
creates function that makes a sha256 from a dictionary
iuryt Dec 21, 2024
a125a11
create a data folder based on the hash from area_of_interest
iuryt Dec 21, 2024
61a9c32
merge with pre-commit ¬¬
iuryt Dec 21, 2024
a2a61c8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 21, 2024
777f505
insert blank lines
iuryt Dec 21, 2024
f6908df
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 21, 2024
2a3c9ab
Update hash to use _fetch utils
VeckoTheGecko Jan 13, 2025
66a1bb9
Avoid mixing matching creds
VeckoTheGecko Jan 13, 2025
3012812
Download caching feature
VeckoTheGecko Jan 13, 2025
4160437
Update input data loading with new data folder structure
VeckoTheGecko Jan 13, 2025
fd060d7
Handle download_cleanup on wrong credentials
VeckoTheGecko Jan 13, 2025
d2a7bc8
Copy schedule to download folde
VeckoTheGecko Jan 13, 2025
18b86ec
Patch download command
VeckoTheGecko Jan 13, 2025
524c650
Add `input_data` param for testing
VeckoTheGecko Jan 13, 2025
ad995c3
Add tests
VeckoTheGecko Jan 13, 2025
afe6112
Add fetch test
VeckoTheGecko Jan 13, 2025
5669e41
Pin copernicusmarine >= 2
VeckoTheGecko Jan 13, 2025
bcfe01e
RTD explicit config key
VeckoTheGecko Jan 13, 2025
0ef7cf0
Delete download_data script
VeckoTheGecko Jan 13, 2025
b830393
Update pyproject.toml
VeckoTheGecko Jan 13, 2025
8d35d4b
Improve filename to hash conversion
VeckoTheGecko Jan 13, 2025
9ffd8de
Rename to assert_complete_download
VeckoTheGecko Jan 13, 2025
cfc50cc
Error message when area of interest isn't defined
VeckoTheGecko Jan 14, 2025
7132766
Add area of interest hash salting
VeckoTheGecko Jan 14, 2025
8698d13
Update 'area of interest' to 'space-time region' throughout
VeckoTheGecko Jan 14, 2025
04d7c13
Avoid circular import
VeckoTheGecko Jan 15, 2025
93576c8
virtualship help documentation
VeckoTheGecko Jan 15, 2025
a2290b6
Update help messages
VeckoTheGecko Jan 15, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .readthedocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

version: 2

sphinx:
configuration: docs/conf.py
build:
os: ubuntu-22.04
tools:
Expand Down
1 change: 1 addition & 0 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ dependencies:
- pydantic >=2, <3
- pip
- pyyaml
- copernicusmarine >= 2

# linting
- pre-commit
Expand Down
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ dependencies = [
"numpy >=1, < 2",
"pydantic >=2, <3",
"PyYAML",
"copernicusmarine >= 2",
]

[project.urls]
Expand Down Expand Up @@ -104,6 +105,8 @@ ignore = [
"D212",
# one-blank-line-before-class
"D203",
# First line of docstring should be in imperative mood
"D401",

# TODO: Remove later
"D100", "D103"
Expand Down
137 changes: 0 additions & 137 deletions scripts/download_data.py

This file was deleted.

107 changes: 107 additions & 0 deletions src/virtualship/cli/_creds.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
from __future__ import annotations

from pathlib import Path

import click
import pydantic
import yaml

CREDENTIALS_FILE = "credentials.yaml"


class CredentialFileError(Exception):
"""Exception raised for errors in the input file format."""

pass


class Credentials(pydantic.BaseModel):
"""Credentials to be used in `virtualship fetch` command."""

COPERNICUS_USERNAME: str
COPERNICUS_PASSWORD: str

@classmethod
def from_yaml(cls, path: str | Path) -> Credentials:
"""
Load credentials from a yaml file.

:param path: Path to the file to load from.
:returns Credentials: The credentials.
"""
with open(path) as file:
data = yaml.safe_load(file)

if not isinstance(data, dict):
raise CredentialFileError("Credential file is of an invalid format.")

return cls(**data)

def dump(self) -> str:
"""
Dump credentials to a yaml string.

:param creds: The credentials to dump.
:returns str: The yaml string.
"""
return yaml.safe_dump(self.model_dump())

def to_yaml(self, path: str | Path) -> None:
"""
Write credentials to a yaml file.

:param path: Path to the file to write to.
"""
with open(path, "w") as file:
file.write(self.dump())


def get_dummy_credentials_yaml() -> str:
return (
Credentials(
COPERNICUS_USERNAME="my_username", COPERNICUS_PASSWORD="my_password"
)
.dump()
.strip()
)


def get_credentials_flow(
username: str | None, password: str | None, creds_path: Path
) -> tuple[str, str]:
"""
Execute flow of getting credentials for use in the `fetch` command.

- If username and password are provided via CLI, use them (ignore the credentials file if exists).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe explain what CLI is? Mot sure every user will know that

Suggested change
- If username and password are provided via CLI, use them (ignore the credentials file if exists).
- If username and password are provided via Command Line Interface (CLI), use them (ignore the credentials file if exists).

Copy link
Collaborator

@VeckoTheGecko VeckoTheGecko Jan 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was intended more as developer documentation rather than user documentation. What do we think in terms of user documentation? Perhaps I can write a documentation page? (I imagine that we wouldn't want this information silo'd in a notebook, but rather link out to the doc page - I imagine there would be other course content in a notebook). Thoughts on how this fits with course content structure @ammedd?

Updating the docstring here wont appear to users unless its also displayed in some sort of user documentation.

Lets put that doc page in a different PR, this is already big enough and getting a bit much 😅

- If username and password are not provided, try to load them from the credentials file.
- If no credentials are provided, print a message on how to make credentials file and prompt for credentials.

:param username: The username provided via CLI.
:param password: The password provided via CLI.
:param creds_path: The path to the credentials file.
"""
if username and password:
if creds_path.exists():
click.echo(
f"Credentials file exists at '{creds_path}', but username and password are already provided. Ignoring credentials file."
)
return username, password

try:
creds = Credentials.from_yaml(creds_path)
click.echo(f"Loaded credentials from '{creds_path}'.")
return creds.COPERNICUS_USERNAME, creds.COPERNICUS_PASSWORD
except FileNotFoundError:
msg = f"""Credentials not provided. Credentials can be obtained from https://data.marine.copernicus.eu/register. Either pass in via `--username` and `--password` arguments, or via config file at '{creds_path}'. Config file should be YAML along following format:
### {creds_path}

{get_dummy_credentials_yaml().strip()}

###
VeckoTheGecko marked this conversation as resolved.
Show resolved Hide resolved

Prompting for credentials instead...
"""
click.echo(msg)
username = click.prompt("username")
password = click.prompt("password", hide_input=True)
return username, password
Loading
Loading