sosulski_kristen_dwd_class16_NYCOpenData.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "toc": {
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": false,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": false
    },
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ajita-asthana/Node-tuts/blob/master/sosulski_kristen_dwd_class16_NYCOpenData.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Px_pvSRJ7vmc"
      },
      "source": [
        "## We will use the data set from NYC OpenData called \"New York City Leading Causes of Death\".\n",
        "\n",
        "\n",
        "The end point is: https://data.cityofnewyork.us/resource/jb7j-dtam.json\n",
        "\n",
        "The URL for the data source is at: https://data.cityofnewyork.us/Health/New-York-City-Leading-Causes-of-Death/jb7j-dtam"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WSJiqcwX7vmd"
      },
      "source": [
        "%matplotlib inline \n",
        "# It is a magic function that renders the figure in a notebook (instead of displaying the figure object reference).\n",
        "import requests\n",
        "import pandas as pd\n",
        "import numpy as np"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LJ-oOD1O7vmi"
      },
      "source": [
        "#### Fetching the data\n",
        "\n",
        "We fetch the data in JSON format using the NYC OpenData API:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MXAxkHbU7vmj"
      },
      "source": [
        "# Data set: New York City Leading Causes of Death\n",
        "url=  'https://data.cityofnewyork.us/resource/jb7j-dtam.json'\n",
        "results = requests.get(url).json() \n",
        "print(len(results)) \n",
        "#note limit is defaulting to 1000. Need to add parameter for 1272 to URL using ?$limit=1272\n",
        "#https://dev.socrata.com/foundry/data.cityofnewyork.us/jb7j-dtam"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u80n-Ak2oHsR"
      },
      "source": [
        "url=  'https://data.cityofnewyork.us/resource/jb7j-dtam.json?$limit=1272'\n",
        "results = requests.get(url).json()\n",
        "print(len(results))\n",
        "print(type(results))\n",
        "print(results[1271])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dk9GWU5A4FR0"
      },
      "source": [
        "Begin to look at the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-AX-P4If7vmm"
      },
      "source": [
        "print(results[1:3]) #exclusive when using : range\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IpF7DsVPquFz"
      },
      "source": [
        "print(results[0]['deaths'])\n",
        "print(results[0]['year'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "results[0]"
      ],
      "metadata": {
        "id": "fnIA2zrheBkS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-S06RRhu7vm7"
      },
      "source": [
        "### Creating a DataFrame from JSON data\n",
        "\n",
        "Let's create a pandas dataframe from the `results` "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X5XP_bJ17vm7"
      },
      "source": [
        "df = pd.DataFrame(results)\n",
        "df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UHsGMG125yz6"
      },
      "source": [
        "len(df.index)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EPYBrJ6_7vnX"
      },
      "source": [
        "### Renaming Columns\n",
        "\n",
        "We do not like some of these column names. Let's rename them.\n",
        "\n",
        "We will use a dictionary, for specifying the existing and the new names for the columns."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q7LAlIna7vnX"
      },
      "source": [
        "# This dictionary specifies as a key the existing name of the column, and as value the new name. Try this out. Change the key value pairs. \n",
        "renaming_dict = {\n",
        "    'leading_cause': 'cause', \n",
        "    'sex': 'gender'\n",
        "}\n",
        "\n",
        "df.rename(columns=renaming_dict, inplace=True)\n",
        "df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XLcOjQgQ7vna"
      },
      "source": [
        "### Converting Data Types"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RwADX3V77vna"
      },
      "source": [
        "df.dtypes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NbmyIBUg7vnd"
      },
      "source": [
        "# Let's convert year using astype('int')\n",
        "df['deaths'] = df['deaths'].astype('int')  \n",
        "df.dtypes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VMcbQFq67vng"
      },
      "source": [
        "##ALTERNATIVE TO ABOVE FOR THE YEAR COLUMN INSTEAD...\n",
        "\n",
        "df[\"year\"] = pd.to_numeric(df[\"year\"])\n",
        "df.dtypes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gUvCjCpV7vni"
      },
      "source": [
        "Sometimes, during the conversion of data, the cells contain values that cannot be properly converted. We can specify how we want pandas to handle such cases. By default, it will raise an exception, and will not allow us to convert the data to a new data type."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5dzLWpKJ7vnj"
      },
      "source": [
        "# This one will cause an error, as the \"deaths\" column contains non-numeric values.\n",
        "df[\"deaths\"] = pd.to_numeric(df[\"deaths\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df[\"deaths\"].iloc[188]"
      ],
      "metadata": {
        "id": "_7vfqcLsDuu5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "df.iloc[188:190]"
      ],
      "metadata": {
        "id": "C11BoMifDV2g"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DSVYEguY7vnm"
      },
      "source": [
        "We can pass the `errors` command to specify what should happen. From the [documentation of to_numeric](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_numeric.html), we get:\n",
        "* If ‘raise’, then invalid parsing will raise an exception\n",
        "* If ‘coerce’, then invalid parsing will be set as NaN\n",
        "* If ‘ignore’, then invalid parsing will return the input"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z_Eq0s757vnp"
      },
      "source": [
        "df[\"deaths\"] = pd.to_numeric(df[\"deaths\"], errors='coerce')\n",
        "df[\"death_rate\"] = pd.to_numeric(df[\"death_rate\"], errors='coerce')\n",
        "df[\"age_adjusted_death_rate\"] = pd.to_numeric(df[\"age_adjusted_death_rate\"], errors='coerce')\n",
        "df.dtypes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2dNX84SN7vns"
      },
      "source": [
        "df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hQ5OjMU57vnv"
      },
      "source": [
        "We will also mark the other values as Categorical."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zDE5OXXD7vnw"
      },
      "source": [
        "df[\"gender\"] = pd.Categorical(df[\"gender\"])\n",
        "df[\"race_ethnicity\"] = pd.Categorical(df[\"race_ethnicity\"])\n",
        "df[\"cause\"] = pd.Categorical(df[\"cause\"])\n",
        "df.dtypes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X1HvpRWD7vn8"
      },
      "source": [
        "### Exploratory Data Analysis"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6y1FK8ua7voD"
      },
      "source": [
        "df[\"gender\"].value_counts()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rYjejkgiaetA"
      },
      "source": [
        "# Let's be consistent with the gender nominal values\n",
        "df['gender'].replace({'Female': 'F', 'Male': 'M'}, inplace=True)\n",
        "df['gender'].value_counts()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J0By_IGz7voG"
      },
      "source": [
        "df['cause'].value_counts()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G7DdqFnf3cGm"
      },
      "source": [
        "df.groupby(by='gender').size()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kLZndizF48QE"
      },
      "source": [
        "df['gender'].nunique()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O2K3PAKt7voI"
      },
      "source": [
        "### Pivot Tables\n",
        "\n",
        "Let's create a pivot table now. We are going to put the \"leading cause\" as the row, with gender and race as columns. For the cell values we will use the number of deaths, and we are going to sum (`np.sum`) the values.\n",
        "\n",
        "_Note: You will also find the `pivot` and `crosstab` functions in Pandas. The `pivot_table` function is typically a more general version of both._"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Omr11BvU7voI"
      },
      "source": [
        "pivot = pd.pivot_table(df, \n",
        "                       values='deaths', \n",
        "                       index=['cause'], # rows\n",
        "                       columns=['gender', 'race_ethnicity'], # columns\n",
        "                       aggfunc=np.mean).copy() # aggregation function\n",
        "pivot"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SYaZqZaT7aXN"
      },
      "source": [
        "#Check the data type of pivot. \n",
        "\n",
        "type(pivot)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6EeGCkc57voL"
      },
      "source": [
        "And we can easily transpose the dataframe"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "r1vt4oEV7voL"
      },
      "source": [
        "pivot.transpose()\n",
        "# alternatively you could use pivot.T"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VH4OX2A1MNw7"
      },
      "source": [
        "pivot.transpose()['Influenza (Flu) and Pneumonia (J09-J18)'].sort_values().plot(kind='barh', title='Leading cause of death: Influenza and Pneumonia', color='#999999');"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S9_83nbt7voO"
      },
      "source": [
        "pivot.transpose()['Diseases of Heart (I00-I09, I11, I13, I20-I51)'].sort_values().plot(kind='barh', title='Leading cause of death: Diseases of the Heart', color='#932211');"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}