Datasets and data sources that need API

[henrykironde]

• US census data #1539
• https://dev.socrata.com/foundry/opendata.utah.gov/35s3-nmpm
• https://walker-data.com/tidycensus/articles/spatial-data.html
• Add MODIS NDVI data Add MODIS NDVI data #545
• https://github.com/datasets Frictionless data's data package https://datahub.io/docs/getting-started/getting-data#python you can track the data added using Add datasets from data hub awesome collection #1404

[Aakash3101]

@henrykironde How should I proceed with this? In the issue mentioned, the dataset is uploaded as an R package on CRAN.

[henrykironde]

This is an issue related to the GSOC project https://github.com/weecology/retriever/wiki/GSoC-2021-Project-Ideas#data-retriever-support-for-loginapi.
In this case the data can be dowloaded using the R tool.

The first step would be getting to know how the R tool downloads the data.
In the documentation the demonstrate how you can access the date.
If you can run the tool by creating an R script that downloads the data. The next step would be to call retriever to install this data.
Since there are two languages, Retriever being Python, we can use rpy2 to call the R script that will basically download the data. the retriever will then take it from here.

There are many data sources that provide APIs or need login. Retriever can scale up to be able to do that. A good example is the Kaggle Api that we added.

Having said that, this issue would best be solved as part of the summer work. But feel free to check it out and let me know

[Aakash3101]

I would like to take the GSOC project Data Retriever: Support for Login/API. Can you please send me more information related to the API support in retriever. This would help me create a better proposal. Also, can you share an example of how the Kaggle API is called/used in retriever install? Should I communicate about my proposal to you privately on gitter?

[henrykironde]

Yes you can communicate privately on gitter.

How Kaggle api works https://technowhisp.com/kaggle-api-python-documentation/
Kaggle is a python package that enables users to download kaggle dataset using programing tools. Basically like a small Data Retriever. So can use the Data Retriever to call the package and download data. Then Retriever will clean it up and standardize it.

retriever/retriever/lib/engine.py

Line 512 in c4840b3

def download_from_kaggle(

[henrykironde]

CDC data uses API
https://data.cdc.gov/Disability-Health/DHDS-Prevalence-of-Disability-Status-and-Types-by-/qjg3-6acf

In the API they have away to list the datasets https://github.com/xmunoz/sodapy. That means we can add all their datasets with much ease.

[Aakash3101]

It uses the same API as opendata.utah.gov uses, which is the Socrata (Sodapy) API. So it would not add any new API, as I have already discussed the Socrata API for the project