Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding a dataset that can only downloaded with credentials #45

Open
Datseris opened this issue Jun 9, 2024 · 0 comments
Open

Adding a dataset that can only downloaded with credentials #45

Datseris opened this issue Jun 9, 2024 · 0 comments
Labels
enhancement New feature or request question Further information is requested

Comments

@Datseris
Copy link

Datseris commented Jun 9, 2024

I typically work with data that one cannot download from a URL directly, and I cannot put them into a download URL as I don't have the rights to them. How do I add such a dataset to the DataToolkit.jl Data.toml file?

For example, I am downloading ERA5 data, which can be downloaded with a Julia script (via a PythonCall) like so:

import PythonCall
cdsapi = PythonCall.pyimport("cdsapi")

c = cdsapi.Client()
savepath = ...
config = Dict(
    "product_type" => producttype,
    "variable" => variables,
    "year" => year
...
)
c.retrieve(data, config, savepath) # does the download

this also requires me to have the file ~/.cdsapirc saved in my computer with contents:

url: https://cds.climate.copernicus.eu/api/v2
key: 64546:.... # my private key, accessed by going into my account online and copying it

Would such an approach be possible to make reproducible with DataToolkit.jl...? I doubt it, and not due to DataToolkit.jl's fault, but mainly due to the absolutely terrible-for-reproducibility system that these data have, and in fact even worse systems are prevalent in the whole of climate science :(

@tecosaur tecosaur added enhancement New feature or request question Further information is requested labels Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants