Lessons/Lesson15_Pandas-Reading.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ACIL5-c6rdot"
   },
   "source": [
    "# Reading Data with Pandas\n",
    "\n",
    "In the last lesson, we learned how `pandas` stores data as rows and columns in `DataFrames`. We previously used a small dataset that was hard-coded right in the notebook. But in the real world, we want to be able to use large datasets that can't be easily hard-coded or typed out by hand. One way that we can store large datasets as files is in the CSV format. This is a format which can be opened by many different programs like Excel, Google Sheets, or our Python programs, which allows us to share data easily. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZgNq_eJkt5SX"
   },
   "source": [
    "Let's start by importing pandas. We can use the `pd` nickname like before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZqBeAVEtuoHO"
   },
   "outputs": [],
   "source": [
    "# import the pandas package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OiDokkTnvq58"
   },
   "source": [
    "Now we're ready to read our dataset into Python with `pandas`! We'll use a function called `read_csv`.  Our dataset is in our GWC GitHub repository, and we need to tell `read_csv` exactly where to find it. `read_csv` will create a `DataFrame` for us. Let's call it `tips`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ydp4Ua9JwI2D"
   },
   "outputs": [],
   "source": [
    "# load the tips csv\n",
    "path = 'https://raw.githubusercontent.com/GWC-DCMB/ClubCurriculum/master/'\n",
    "tips = pd.read_csv(path + 'SampleData/tips.csv')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "G2tQtYbCw5Te"
   },
   "source": [
    "Since we saved the data to a variable, `pandas` didn't show us what it looks like. How would you view the beginning of the `tips` `DataFrame` without seeing every row? Try it below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 592,
     "status": "ok",
     "timestamp": 1563148643354,
     "user": {
      "displayName": "Kelly Sovacool",
      "photoUrl": "https://lh3.googleusercontent.com/-WQsOgn6cXFg/AAAAAAAAAAI/AAAAAAAAAVA/ol7X28qyOHc/s64/photo.jpg",
      "userId": "07259925140867310896"
     },
     "user_tz": 240
    },
    "id": "aO8F9zhDwUGo",
    "outputId": "1c2928de-9b8e-4b54-fcfe-dc4d0ddfc5e5"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>sex</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>16.99</td>\n",
       "      <td>1.01</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.34</td>\n",
       "      <td>1.66</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21.01</td>\n",
       "      <td>3.50</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23.68</td>\n",
       "      <td>3.31</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>24.59</td>\n",
       "      <td>3.61</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   total_bill   tip     sex smoker  day    time  size\n",
       "0       16.99  1.01  Female     No  Sun  Dinner     2\n",
       "1       10.34  1.66    Male     No  Sun  Dinner     3\n",
       "2       21.01  3.50    Male     No  Sun  Dinner     3\n",
       "3       23.68  3.31    Male     No  Sun  Dinner     2\n",
       "4       24.59  3.61  Female     No  Sun  Dinner     4"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View just the beginning of the tips DataFrame\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5gXl4EOlxUHo"
   },
   "source": [
    "Look at the column names of the `tips` `DataFrame`. We have `total_bill`, `tip`, `sex`, `smoker`, `day`, `time`, and `size`. Based on the column names and some of the values in the `DataFrame`, what do you think the rows each represent?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OcufortFyCdx"
   },
   "source": [
    "**The rows represent**: waiters and waitresses in a restaurant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XhE-xUlvzrFD"
   },
   "source": [
    "Now let's take a look at the end of the `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 938,
     "status": "ok",
     "timestamp": 1563148643714,
     "user": {
      "displayName": "Kelly Sovacool",
      "photoUrl": "https://lh3.googleusercontent.com/-WQsOgn6cXFg/AAAAAAAAAAI/AAAAAAAAAVA/ol7X28qyOHc/s64/photo.jpg",
      "userId": "07259925140867310896"
     },
     "user_tz": 240
    },
    "id": "8x_k5eVXxhyH",
    "outputId": "b0837489-81a7-4dc8-d852-7fd1803bccd5"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>sex</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>239</th>\n",
       "      <td>29.03</td>\n",
       "      <td>5.92</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240</th>\n",
       "      <td>27.18</td>\n",
       "      <td>2.00</td>\n",
       "      <td>Female</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>241</th>\n",
       "      <td>22.67</td>\n",
       "      <td>2.00</td>\n",
       "      <td>Male</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>242</th>\n",
       "      <td>17.82</td>\n",
       "      <td>1.75</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>243</th>\n",
       "      <td>18.78</td>\n",
       "      <td>3.00</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Thur</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     total_bill   tip     sex smoker   day    time  size\n",
       "239       29.03  5.92    Male     No   Sat  Dinner     3\n",
       "240       27.18  2.00  Female    Yes   Sat  Dinner     2\n",
       "241       22.67  2.00    Male    Yes   Sat  Dinner     2\n",
       "242       17.82  1.75    Male     No   Sat  Dinner     2\n",
       "243       18.78  3.00  Female     No  Thur  Dinner     2"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View the end of the tips DataFrame\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "naQNUtPqz9xK"
   },
   "source": [
    "Notice the numbers on the far left side of the `DataFrame`. `pandas` assigned a number to every row. What number did `pandas` assign to the very first row of the `DataFrame`? (Scroll up if you need to.) So how many rows do we have in this `DataFrame`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Vf8HXv5P0QAn"
   },
   "source": [
    "**Number of rows:** 244"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lwI6Tj780YQ_"
   },
   "source": [
    "The column of numbers that label the rows is called the `index` of the `DataFrame`. The `index` is an `attribute`, a special variable which belongs to variables of the `DataFrame` type. An example of an attribute would be if you had a variable `dog` with an attribute `dog.owner` to store the name of the person who owns the dog.\n",
    "\n",
    "We can view the `DataFrame`'s `index` like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 928,
     "status": "ok",
     "timestamp": 1563148643720,
     "user": {
      "displayName": "Kelly Sovacool",
      "photoUrl": "https://lh3.googleusercontent.com/-WQsOgn6cXFg/AAAAAAAAAAI/AAAAAAAAAVA/ol7X28qyOHc/s64/photo.jpg",
      "userId": "07259925140867310896"
     },
     "user_tz": 240
    },
    "id": "TEper3lH2bXx",
    "outputId": "04c2269a-11da-4879-cd48-b0dddcd51d0f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=244, step=1)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rE3A4CfS12ag"
   },
   "source": [
    "So our index starts at 0, ends at 244, and increases by 1 for each row.\n",
    "Another way to count the number of rows is to take the length of the index using the `len` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 915,
     "status": "ok",
     "timestamp": 1563148643722,
     "user": {
      "displayName": "Kelly Sovacool",
      "photoUrl": "https://lh3.googleusercontent.com/-WQsOgn6cXFg/AAAAAAAAAAI/AAAAAAAAAVA/ol7X28qyOHc/s64/photo.jpg",
      "userId": "07259925140867310896"
     },
     "user_tz": 240
    },
    "id": "_9syv7p2zyZJ",
    "outputId": "22150178-ad29-45fe-c592-c470e4c39228"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "244"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the length of the index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eVPrUNKh3oMH"
   },
   "source": [
    "Like the `index` labels the rows of the `DataFrame`, there is an `attribute` called `columns` that refers to the columns of the `DataFrame`. Let's take a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 901,
     "status": "ok",
     "timestamp": 1563148643723,
     "user": {
      "displayName": "Kelly Sovacool",
      "photoUrl": "https://lh3.googleusercontent.com/-WQsOgn6cXFg/AAAAAAAAAAI/AAAAAAAAAVA/ol7X28qyOHc/s64/photo.jpg",
      "userId": "07259925140867310896"
     },
     "user_tz": 240
    },
    "id": "OGoDPN5F21dq",
    "outputId": "3ad7f248-501c-404e-d306-f9287d4917f5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['total_bill', 'tip', 'sex', 'smoker', 'day', 'time', 'size'], dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0JJa0dOf329W"
   },
   "source": [
    "We could count the number of columns -- there aren't too many -- but what's the fun in that? Let's write a line of code to tell us the number of columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 887,
     "status": "ok",
     "timestamp": 1563148643725,
     "user": {
      "displayName": "Kelly Sovacool",
      "photoUrl": "https://lh3.googleusercontent.com/-WQsOgn6cXFg/AAAAAAAAAAI/AAAAAAAAAVA/ol7X28qyOHc/s64/photo.jpg",
      "userId": "07259925140867310896"
     },
     "user_tz": 240
    },
    "id": "zEN-Nmj93vti",
    "outputId": "bb6b95d3-3e4b-4c10-e5e3-2e72d2f9c991"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# length of the DataFrame's columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AU7lnXVPhR8n"
   },
   "source": [
    "Conveniently, we can also call `len` on the `DataFrame` itself. Try it here! Is the result equal to the number of rows or the number of columns?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 856,
     "status": "ok",
     "timestamp": 1563148643726,
     "user": {
      "displayName": "Kelly Sovacool",
      "photoUrl": "https://lh3.googleusercontent.com/-WQsOgn6cXFg/AAAAAAAAAAI/AAAAAAAAAVA/ol7X28qyOHc/s64/photo.jpg",
      "userId": "07259925140867310896"
     },
     "user_tz": 240
    },
    "id": "gUe_lc4qhbTC",
    "outputId": "3d3d9864-ad35-4b01-8758-f2b9c1471980"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "244"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use len on tips\n",
    "len(tips)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rYL5DRca4504"
   },
   "source": [
    "Based on the number of rows and columns, how many data points are in the `tips` `DataFrame`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 839,
     "status": "ok",
     "timestamp": 1563148643728,
     "user": {
      "displayName": "Kelly Sovacool",
      "photoUrl": "https://lh3.googleusercontent.com/-WQsOgn6cXFg/AAAAAAAAAAI/AAAAAAAAAVA/ol7X28qyOHc/s64/photo.jpg",
      "userId": "07259925140867310896"
     },
     "user_tz": 240
    },
    "id": "j9bCbWNJ4EYa",
    "outputId": "b063a0b8-b9cd-4e68-fdc5-2cf143847583"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1708"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate the number of data points in tips\n",
    "7 * 244"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yK106lOu5QAn"
   },
   "source": [
    "That's a lot more data than we've handled before. But that's nothing for `pandas` -- it can handle `DataFrames` with *millions* of rows! Data scientists use `pandas` to handle very large datasets from the real world.\n",
    "\n",
    "Instead of typing the number of rows and columns in the DataFrame, we could put both commands with `len` on the same line. Try it here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 824,
     "status": "ok",
     "timestamp": 1563148643736,
     "user": {
      "displayName": "Kelly Sovacool",
      "photoUrl": "https://lh3.googleusercontent.com/-WQsOgn6cXFg/AAAAAAAAAAI/AAAAAAAAAVA/ol7X28qyOHc/s64/photo.jpg",
      "userId": "07259925140867310896"
     },
     "user_tz": 240
    },
    "id": "DPj6aVwH5E6Q",
    "outputId": "53d89884-eaf6-4521-a230-750b0cd074b3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1708"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Multiply the length of rows & columns without typing numbers\n",
    "len(tips) * len(tips.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "znhZFN0a54aA"
   },
   "source": [
    "This way, if the `tips` data changes, we can quickly re-run the above cell to find the number of values in it, without having to manually type out the number of rows and columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5eGz8e7H5Xos"
   },
   "source": [
    "You just learned:\n",
    "\n",
    "- How to read datasets into `pandas` `DataFrames`.\n",
    "- The `index` and `columns` attributes of `DataFrames`.\n",
    "- How to find the number of rows, columns, and number of data points in a `DataFrame`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}