R_SODA_API_for_VDH_data.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "R - SODA API for VDH data.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOAc+0/2NujfX/TOeIDObmt",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "ir",
      "display_name": "R"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/odu-cs625-datavis/public/blob/main/R_SODA_API_for_VDH_data.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vSwr6fnhvoG3"
      },
      "source": [
        "# Using the SODA API to access data from the Virginia Open Data Portal\n",
        "\n",
        "For more information on using the API, visit the dataset page you're interested in and click the API button. It should provide links to the API docs for that dataset and the Developer Portal.\n",
        "\n",
        "In this example, we'll access the VDH COVID-19 Public Use Dataset - Cases, available at https://data.virginia.gov/Government/VDH-COVID-19-PublicUseDataset-Cases/bre9-aqqr. The dataset name, which we'll need for the API, is the last part of the URI, `bre9-aqqr`.\n",
        "\n",
        "As of Sep 29, 2021, this dataset had 74.7k rows and 7 columns, with each row representing the overall count of COVID-19 cases, hospitalizations, deaths for each locality in Virginia by report date since reporting began for this dataset.\n",
        "\n",
        "The following example is based the API docs for this dataset, available at https://dev.socrata.com/foundry/data.virginia.gov/bre9-aqqr, and the RSocrata repo at https://github.com/Chicago/RSocrata"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xzuBXxT0MNBE"
      },
      "source": [
        "First, we have to install and load the `RSocrata` library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G3Z49BjoS5hp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ffb57962-b61d-4c3e-9554-c9bf64df51de"
      },
      "source": [
        "install.packages(\"RSocrata\")\n",
        "library(RSocrata)"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Installing package into ‘/usr/local/lib/R/site-library’\n",
            "(as ‘lib’ is unspecified)\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qkBrwtlxMKtv"
      },
      "source": [
        "Then, we specify the dataset we want to access. This corresponds to the set of characters at end of the dataset URI."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0_-k6ZdQxC4j"
      },
      "source": [
        "vdhDataset <- \"bre9-aqqr\""
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TMloCD-jMZM7"
      },
      "source": [
        "By default, the API only returns 1000 results. To access more, we have to set the $limit parameter."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FtvpGfHFMllt"
      },
      "source": [
        "fetchLimit <- 80000   # since there were 74.7k on Sep 30, request more than that"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i3IiCI55MfBW"
      },
      "source": [
        "Then, we construct the URI to request the data. There's nothing you need to change here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iQya2T0fMn5E"
      },
      "source": [
        "uri <- paste(\"https://data.virginia.gov/resource/\", vdhDataset, \".json?$limit=\", fetchLimit, sep=\"\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FZq4T5JGMrWv"
      },
      "source": [
        "Finally, we read in the data. Again, there's nothing for you to change."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zPR_pu86wCiF"
      },
      "source": [
        "df <- read.socrata(uri)"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nU7Oe_KIxyKt",
        "outputId": "f195e331-d993-4558-972f-b5b012e80229"
      },
      "source": [
        "str(df)"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "'data.frame':\t74879 obs. of  7 variables:\n",
            " $ report_date        : Date, format: \"2020-03-17\" \"2020-03-17\" ...\n",
            " $ fips               : chr  \"51001\" \"51003\" \"51005\" \"51007\" ...\n",
            " $ locality           : chr  \"Accomack\" \"Albemarle\" \"Alleghany\" \"Amelia\" ...\n",
            " $ vdh_health_district: chr  \"Eastern Shore\" \"Thomas Jefferson\" \"Alleghany\" \"Piedmont\" ...\n",
            " $ total_cases        : num  0 0 0 0 0 0 13 0 0 0 ...\n",
            " $ hospitalizations   : num  0 0 0 0 0 0 1 0 0 0 ...\n",
            " $ deaths             : num  0 0 0 0 0 0 0 0 0 0 ...\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5C9Fhkj0MxxY"
      },
      "source": [
        "To be able to use the data, we'll want to convert columns to the proper datatypes (datetime, int, etc.)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wk-HbST1K6UA"
      },
      "source": [
        "df$total_cases = as.numeric(as.character(df$total_cases))\n",
        "df$hospitalizations = as.numeric(as.character(df$hospitalizations))\n",
        "df$deaths = as.numeric(as.character(df$deaths))\n",
        "df$report_date = as.Date(df$report_date)"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v24Gtt2VM6XB",
        "outputId": "48488b2e-94a8-40bb-a1b5-4f53e8b8e9d6"
      },
      "source": [
        "str(df)"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "'data.frame':\t74879 obs. of  7 variables:\n",
            " $ report_date        : Date, format: \"2020-03-17\" \"2020-03-17\" ...\n",
            " $ fips               : chr  \"51001\" \"51003\" \"51005\" \"51007\" ...\n",
            " $ locality           : chr  \"Accomack\" \"Albemarle\" \"Alleghany\" \"Amelia\" ...\n",
            " $ vdh_health_district: chr  \"Eastern Shore\" \"Thomas Jefferson\" \"Alleghany\" \"Piedmont\" ...\n",
            " $ total_cases        : num  0 0 0 0 0 0 13 0 0 0 ...\n",
            " $ hospitalizations   : num  0 0 0 0 0 0 1 0 0 0 ...\n",
            " $ deaths             : num  0 0 0 0 0 0 0 0 0 0 ...\n"
          ]
        }
      ]
    }
  ]
}