Dataverse data access module

[abearab]

Aim

Including a draft dataverse data collector as part of gget tool boxes.

e.g.

gget dataverse -j dataverse.json -o path-to-dir

Tasks

• make a python module and CLI for data collection from dataverse
• collect data for a list of datasets defined as a JSON
• build a JSON file for a list of datasets deposited in a given dataverse DOI at certain version
• collect data by providing a dataverse DOI and version

History

As part of GilbertLabUCSF/CanDI#34, related to #121, I draft a module to collect data from dataverse. Then I thought data collection from dataverse can be a more general functionality here as part of gget.

I mostly borrowed codes from gears/utils.py or TDC/load.py. However, there are other efforts such as pyDataverse and easyDataverse.

References

• https://guides.dataverse.org/en/latest/api/dataaccess.html
• https://github.com/mims-harvard/TDC/blob/main/tdc/utils/load.py

[abearab]

cc @amva13

[lauraluebbert]

Hi Abe, this is great work, thank you! I'm pretty busy right now but I'll try to start merging asap.

I just have two thoughts at the moment:

1. We'll need to add unittests for this module. Do you have any ideas on how they should look like?
2. Having a json file as input is a relatively complex input type. Is there perhaps a way around this to simplify the input to the module?

[abearab]

Hi Laura,

1. We'll need to add unittests for this module. Do you have any ideas on how they should look like?

I'll think about and add more commits asap.

2. Having a json file as input is a relatively complex input type. Is there perhaps a way around this to simplify the input to the module?

That's true. I think this can work with dataverse DOI and/or just fileId. However, I personally think a json file can be a good option to list all files related to a single project or topic. I'll try to add more commits in this regard.

I made this a draft PR while I'm adding more commits.

[lauraluebbert]

Oops sorry didn't mean to close this

[abearab]

Oops sorry didn't mean to close this

Oh no worries, I also didn't get chance to finish what we discussed before. I have the changes in a fork so I'll open another PR in the future. Thanks :)

[lauraluebbert]

No need, I'll reopen this one as soon as the unit tests are passing and I can make a new dev branch

[abearab]

mims-harvard/TDC#314 (comment)