05022019_ABC_python_day2_session_4_wo_anwsers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## This is the etherpad connection ###\n",
    "\n",
    "https://public.etherpad-mozilla.org/p/ABC_PYTHON_2019"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## This material is adjusted from Software carpentry and can be found here.  \n",
    "http://swcarpentry.github.io/python-novice-gapminder/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running and Quitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions you should be able to answer after this session\n",
    "* How can I run a Python program and Jupyter Notebooks?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Objectives\n",
    "* Launch the Jupyter Notebook, create new notebooks, and exit the Notebook.\n",
    "* Create Markdown cells in a notebook.\n",
    "* Create and run Python cells in a notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Difference in file extensions\n",
    "* python programs are in plain text and by convention end in **.py**\n",
    "* in the following session and tomorrow afternoon we will use [jupyter notebook](https://jupyter.org/) as it nice to get to know python interactively and to mix code with comments + figures as shown here.\n",
    "* jupyter notebook comes within the [Anaconda](http://docs.continuum.io/anaconda/install/) package manager so you should be good to go. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the github for Benjamin's section.\n",
    "\n",
    "https://github.com/BenjaminSchwessinger/ABC_python\n",
    "\n",
    "For windows:  \n",
    "Go to the github page above. Click on download as zip. Move the zip to your desktop. Unzip on the Desktop. Start the Anaconda Navigator. Navigate to the Desktop and the ABC_python-master folder. Start the notebook 05022019_ABC_python_day2_session_4_wo_anwsers by double clicking on it.\n",
    "\n",
    "For Mac:  \n",
    "Application -> Terminal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the terminal do:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "git clone git@github.com:BenjaminSchwessinger/ABC_python.git\n",
    "\n",
    "cd ABC_python\n",
    "\n",
    "jupyter notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This has several advantages:\n",
    "* You can easily type, edit, and copy and paste blocks of code.\n",
    "* Tab complete allows you to easily access the names of things you are using and learn more about them.\n",
    "* **Shift+Tab** let's you access the docstrings of methods and functions.\n",
    "* It allows you to annotate your code with links, different sized text, bullets, etc. to make it more accessible to you and your collaborators.\n",
    "* It allows you to display figures next to the code that produces them to tell a complete story of the analysis.\n",
    "\n",
    "And some disadvantages:\n",
    "* Is not stored in plain text but JSON format.\n",
    "* Not the order of the cell but the order of execusion matters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='./fig/0_jupyter_notebook_example.jpg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Code vs. Text\n",
    "We often use the term “code” to mean “the source code of software written in a language such as Python”. A “code cell” in a Notebook is a cell that contains software; a “text cell” is one that contains ordinary prose written for human beings. \n",
    "  \n",
    "* If you press **\"Esc\"** and **\"Return\"** alternately, the outer border of your code cell will change from gray/blue to green. \n",
    "* These are the **Command** (blue) and **Edit** (green) modes of your notebook.\n",
    "* In Command mode, pressing the **\"H\"** key will provide a list of all the shortcut keys.\n",
    "* Command mode allows you to edit notebook-level features, and Edit mode changes the content of cells.\n",
    "* When in Command mode (esc/blue),\n",
    "\n",
    "    * The **\"B\"** key will make a new cell below the currently selected cell.\n",
    "    * The **\"A\"** key will make one above.\n",
    "    * The **\"X\"** key will delete the current cell.\n",
    "    * The **\"Z\"** key will undo your last cell deletion.\n",
    "* All actions can be done using the menus, but there are lots of keyboard shortcuts to speed things up.\n",
    "* If you remember the **\"Esc\"** and **\"H\"** shortcut, you will be able to find out all the rest.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Excerise I\n",
    "In the Jupyter notebook page are you currently in Command or Edit mode?\n",
    "Switch between the modes. Use the shortcuts to generate a new cell. Use the shortcuts to delete a cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the keyboard and mouse to select and edit cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Pressing the **\"Return\"** key turns the border green and engages Edit mode, which allows you to type within the cell.\n",
    "* Because we want to be able to write many lines of code in a single cell, pressing the **\"Return\"** key when in Edit mode (green) moves the cursor to the next line in the cell just like in a text editor.\n",
    "* We need some other way to tell the Notebook we want to run what’s in the cell.\n",
    "* Pressing **\"Shift+Return\"** together will execute the contents of the cell.\n",
    "* Notice that the **\"Return\"** and **\"Shift\"** keys on the right of the keyboard are right next to each other.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The Notebook will turn Markdown into pretty-printed documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notebooks can also render [Markdown](https://en.wikipedia.org/wiki/Markdown). Nice cheatsheet [here](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)\n",
    "    * A simple plain-text format for writing lists, links, and other things that might go into a web page.\n",
    "    * Equivalently, a subset of HTML that looks like what you’d send in an old-fashioned email.\n",
    "* Turn the current cell into a Markdown cell by entering the Command mode (**\"Esc\"**/blue) and press the **\"M\"** key.\n",
    "* In [ ]: will disappear to show it is no longer a code cell and you will be able to write in Markdown.\n",
    "* Turn the current cell into a Code cell by entering the Command mode (**\"Esc\"**/blue) and press the **\"Y\"** key.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Markdown does most of what HTML does."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "*   Use asterisks\n",
    "*   to create\n",
    "*   bullet lists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Use asterisks\n",
    "*   to create\n",
    "*   bullet lists."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "1.  Use numbers\n",
    "1.  to create\n",
    "1.  numbered lists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.  Use numbers\n",
    "1.  to create\n",
    "1.  numbered lists."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "*  You can use indents\n",
    "\t*  To create sublists \n",
    "\t*  of the same type\n",
    "*  Or sublists\n",
    "\t1. Of different\n",
    "\t1. types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*  You can use indents\n",
    "\t*  To create sublists \n",
    "\t*  of the same type\n",
    "*  Or sublists\n",
    "\t1. Of different\n",
    "\t1. types"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# A Level-1 Heading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Level-1 Heading"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "## A Level-2 Heading (etc.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Level-2 Heading (etc.)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Line breaks\n",
    "don't matter.\n",
    "\n",
    "But blank lines\n",
    "create new paragraphs.\n",
    "\n",
    "And two spaces with line break folds the end.  \n",
    "Hurray."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Line breaks\n",
    "don't matter.\n",
    "\n",
    "But blank lines\n",
    "create new paragraphs.  \n",
    "And two spaces with line break folds the end.  \n",
    "Hurray."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Excerise II"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "\n",
    "Create a nested list in a Markdown cell in a notebook that looks like this:\n",
    "\n",
    "    1. Get funding.\n",
    "    2. Do work.\n",
    "        o Design experiment.\n",
    "        o Collect data.\n",
    "        o Analyze.\n",
    "    3. Write up.\n",
    "    4. Publish.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Excerise III"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if you write some Python in a code cell and then you switch it to a Markdown cell? For example, put the following in a code cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 6*7 +12\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Key Points\n",
    "* Python programs are plain text files.\n",
    "* Use the Jupyter Notebook for editing and running Python.\n",
    "* The Notebook has Command and Edit modes.\n",
    "* Use the keyboard and mouse to select and edit cells.\n",
    "* The Notebook will turn Markdown into pretty-printed documentation.\n",
    "* Markdown does most of what HTML does."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions?!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Errors\n",
    "See http://swcarpentry.github.io/python-novice-inflammation/07-errors/index.html for initial lesson. And work from Juan Nunez-Iglesias https://github.com/aspp-apac/2019-debugging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "* How does Python report errors?\n",
    "* How can I handle errors in Python programs?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Objectives\n",
    "* To be able to read a traceback, and determine where the error took place and what type it is.\n",
    "* To be able to describe the types of situations in which syntax errors, indentation errors, name errors, index errors, and missing file errors occur.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Index Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [1, 2, 3]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Key Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {'a': 1, 'b': 2, 'c': 3}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Syntax Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def foo    \n",
    "    return x*x\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Identation Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def return_and_print(string):\n",
    "    for x in string:\n",
    "        print(x)\n",
    "     return(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TabError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def some_function():\n",
    "\tmsg = \"hello, world!\"\n",
    "\tprint(msg)\n",
    "        return msg\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Missing file error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('nonexistent-file.txt', mode='r') as fin:\n",
    "    for line in fin:\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise\n",
    "1. Read the code below, and (without running it) try to identify what the errors are.\n",
    "2. Run the code, and read the error message. Is it a SyntaxError or an IndentationError?\n",
    "3. Fix the error.\n",
    "4. Repeat steps 2 and 3, until you have fixed all the errors.\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "def another_function\n",
    "  print(\"Syntax errors are annoying.\")\n",
    "   print(\"But at least Python tells us about them!\")\n",
    "  print(\"So they are usually not too hard to fix.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "seasons = ['Spring', 'Summer', 'Fall', 'Winter']\n",
    "print('My favorite season is ', seasons[4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Key Points\n",
    "\n",
    "\n",
    "* Tracebacks can look intimidating, but they give us a lot of useful information about what went wrong in our program, including where the error occurred and what type of error it was.\n",
    "\n",
    "* An error having to do with the ‘grammar’ or syntax of the program is called a **SyntaxError**. If the issue has to do with how the code is indented, then it will be called an **IndentationError**.\n",
    "\n",
    "* A **NameError** will occur if you use a variable that has not been defined, either because you meant to use quotes around a string, you forgot to define the variable, or you just made a typo.\n",
    "\n",
    "* Containers like lists and strings will generate errors if you try to access items in them that do not exist. This type of error is called an **IndexError**.\n",
    "\n",
    "* Trying to read a file that does not exist will give you an **FileNotFoundError**. Trying to read a file that is open for writing, or writing to a file that is open for reading, will give you an **IOError**.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading Tabular Data into DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions you should be able to answer after this session\n",
    "* How can I read tabular data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Objectives\n",
    "* Import the Pandas library.\n",
    "* Use Pandas to load a simple CSV data set.\n",
    "* Get some basic information about a Pandas DataFrame.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the Pandas library to do statistics on tabular data.\n",
    "* Pandas is a widely-used Python library for statistics, particularly on tabular data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import the pandas library with an alias pd\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       country  gdpPercap_1952  gdpPercap_1957  gdpPercap_1962  \\\n",
      "0    Australia     10039.59564     10949.64959     12217.22686   \n",
      "1  New Zealand     10556.57566     12247.39532     13175.67800   \n",
      "\n",
      "   gdpPercap_1967  gdpPercap_1972  gdpPercap_1977  gdpPercap_1982  \\\n",
      "0     14526.12465     16788.62948     18334.19751     19477.00928   \n",
      "1     14463.91893     16046.03728     16233.71770     17632.41040   \n",
      "\n",
      "   gdpPercap_1987  gdpPercap_1992  gdpPercap_1997  gdpPercap_2002  \\\n",
      "0     21888.88903     23424.76683     26997.93657     30687.75473   \n",
      "1     19007.19129     18363.32494     21050.41377     23189.80135   \n",
      "\n",
      "   gdpPercap_2007  \n",
      "0     34435.36744  \n",
      "1     25185.00911  \n"
     ]
    }
   ],
   "source": [
    "#store one of the datasets in the variable data and print it e.g. oceania\n",
    "data = pd.read_csv('./data/gapminder_gdp_oceania.csv')\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>gdpPercap_1952</th>\n",
       "      <th>gdpPercap_1957</th>\n",
       "      <th>gdpPercap_1962</th>\n",
       "      <th>gdpPercap_1967</th>\n",
       "      <th>gdpPercap_1972</th>\n",
       "      <th>gdpPercap_1977</th>\n",
       "      <th>gdpPercap_1982</th>\n",
       "      <th>gdpPercap_1987</th>\n",
       "      <th>gdpPercap_1992</th>\n",
       "      <th>gdpPercap_1997</th>\n",
       "      <th>gdpPercap_2002</th>\n",
       "      <th>gdpPercap_2007</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Australia</td>\n",
       "      <td>10039.59564</td>\n",
       "      <td>10949.64959</td>\n",
       "      <td>12217.22686</td>\n",
       "      <td>14526.12465</td>\n",
       "      <td>16788.62948</td>\n",
       "      <td>18334.19751</td>\n",
       "      <td>19477.00928</td>\n",
       "      <td>21888.88903</td>\n",
       "      <td>23424.76683</td>\n",
       "      <td>26997.93657</td>\n",
       "      <td>30687.75473</td>\n",
       "      <td>34435.36744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>New Zealand</td>\n",
       "      <td>10556.57566</td>\n",
       "      <td>12247.39532</td>\n",
       "      <td>13175.67800</td>\n",
       "      <td>14463.91893</td>\n",
       "      <td>16046.03728</td>\n",
       "      <td>16233.71770</td>\n",
       "      <td>17632.41040</td>\n",
       "      <td>19007.19129</td>\n",
       "      <td>18363.32494</td>\n",
       "      <td>21050.41377</td>\n",
       "      <td>23189.80135</td>\n",
       "      <td>25185.00911</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  gdpPercap_1952  gdpPercap_1957  gdpPercap_1962  \\\n",
       "0    Australia     10039.59564     10949.64959     12217.22686   \n",
       "1  New Zealand     10556.57566     12247.39532     13175.67800   \n",
       "\n",
       "   gdpPercap_1967  gdpPercap_1972  gdpPercap_1977  gdpPercap_1982  \\\n",
       "0     14526.12465     16788.62948     18334.19751     19477.00928   \n",
       "1     14463.91893     16046.03728     16233.71770     17632.41040   \n",
       "\n",
       "   gdpPercap_1987  gdpPercap_1992  gdpPercap_1997  gdpPercap_2002  \\\n",
       "0     21888.88903     23424.76683     26997.93657     30687.75473   \n",
       "1     19007.19129     18363.32494     21050.41377     23189.80135   \n",
       "\n",
       "   gdpPercap_2007  \n",
       "0     34435.36744  \n",
       "1     25185.00911  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#nice visulization within notebooks\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Don't mind the errors."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Errors:\n",
    "FileNotFoundError: File b'gapminder_gdp_oceania.csv' does not exist\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gdpPercap_1952</th>\n",
       "      <th>gdpPercap_1957</th>\n",
       "      <th>gdpPercap_1962</th>\n",
       "      <th>gdpPercap_1967</th>\n",
       "      <th>gdpPercap_1972</th>\n",
       "      <th>gdpPercap_1977</th>\n",
       "      <th>gdpPercap_1982</th>\n",
       "      <th>gdpPercap_1987</th>\n",
       "      <th>gdpPercap_1992</th>\n",
       "      <th>gdpPercap_1997</th>\n",
       "      <th>gdpPercap_2002</th>\n",
       "      <th>gdpPercap_2007</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Australia</th>\n",
       "      <td>10039.59564</td>\n",
       "      <td>10949.64959</td>\n",
       "      <td>12217.22686</td>\n",
       "      <td>14526.12465</td>\n",
       "      <td>16788.62948</td>\n",
       "      <td>18334.19751</td>\n",
       "      <td>19477.00928</td>\n",
       "      <td>21888.88903</td>\n",
       "      <td>23424.76683</td>\n",
       "      <td>26997.93657</td>\n",
       "      <td>30687.75473</td>\n",
       "      <td>34435.36744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>New Zealand</th>\n",
       "      <td>10556.57566</td>\n",
       "      <td>12247.39532</td>\n",
       "      <td>13175.67800</td>\n",
       "      <td>14463.91893</td>\n",
       "      <td>16046.03728</td>\n",
       "      <td>16233.71770</td>\n",
       "      <td>17632.41040</td>\n",
       "      <td>19007.19129</td>\n",
       "      <td>18363.32494</td>\n",
       "      <td>21050.41377</td>\n",
       "      <td>23189.80135</td>\n",
       "      <td>25185.00911</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             gdpPercap_1952  gdpPercap_1957  gdpPercap_1962  gdpPercap_1967  \\\n",
       "country                                                                       \n",
       "Australia       10039.59564     10949.64959     12217.22686     14526.12465   \n",
       "New Zealand     10556.57566     12247.39532     13175.67800     14463.91893   \n",
       "\n",
       "             gdpPercap_1972  gdpPercap_1977  gdpPercap_1982  gdpPercap_1987  \\\n",
       "country                                                                       \n",
       "Australia       16788.62948     18334.19751     19477.00928     21888.88903   \n",
       "New Zealand     16046.03728     16233.71770     17632.41040     19007.19129   \n",
       "\n",
       "             gdpPercap_1992  gdpPercap_1997  gdpPercap_2002  gdpPercap_2007  \n",
       "country                                                                      \n",
       "Australia       23424.76683     26997.93657     30687.75473     34435.36744  \n",
       "New Zealand     18363.32494     21050.41377     23189.80135     25185.00911  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#now index it proper with using the index_col and check what else read_csv\n",
    "#can do (? and shift-tab) or help()\n",
    "data = pd.read_csv('./data/gapminder_gdp_oceania.csv', index_col='country')\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 2 entries, Australia to New Zealand\n",
      "Data columns (total 12 columns):\n",
      "gdpPercap_1952    2 non-null float64\n",
      "gdpPercap_1957    2 non-null float64\n",
      "gdpPercap_1962    2 non-null float64\n",
      "gdpPercap_1967    2 non-null float64\n",
      "gdpPercap_1972    2 non-null float64\n",
      "gdpPercap_1977    2 non-null float64\n",
      "gdpPercap_1982    2 non-null float64\n",
      "gdpPercap_1987    2 non-null float64\n",
      "gdpPercap_1992    2 non-null float64\n",
      "gdpPercap_1997    2 non-null float64\n",
      "gdpPercap_2002    2 non-null float64\n",
      "gdpPercap_2007    2 non-null float64\n",
      "dtypes: float64(12)\n",
      "memory usage: 208.0+ bytes\n"
     ]
    }
   ],
   "source": [
    "#look at the info()\n",
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['gdpPercap_1952', 'gdpPercap_1957', 'gdpPercap_1962', 'gdpPercap_1967',\n",
       "       'gdpPercap_1972', 'gdpPercap_1977', 'gdpPercap_1982', 'gdpPercap_1987',\n",
       "       'gdpPercap_1992', 'gdpPercap_1997', 'gdpPercap_2002', 'gdpPercap_2007'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#how to access the column names\n",
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>country</th>\n",
       "      <th>Australia</th>\n",
       "      <th>New Zealand</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>gdpPercap_1952</th>\n",
       "      <td>10039.59564</td>\n",
       "      <td>10556.57566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gdpPercap_1957</th>\n",
       "      <td>10949.64959</td>\n",
       "      <td>12247.39532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gdpPercap_1962</th>\n",
       "      <td>12217.22686</td>\n",
       "      <td>13175.67800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gdpPercap_1967</th>\n",
       "      <td>14526.12465</td>\n",
       "      <td>14463.91893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gdpPercap_1972</th>\n",
       "      <td>16788.62948</td>\n",
       "      <td>16046.03728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gdpPercap_1977</th>\n",
       "      <td>18334.19751</td>\n",
       "      <td>16233.71770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gdpPercap_1982</th>\n",
       "      <td>19477.00928</td>\n",
       "      <td>17632.41040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gdpPercap_1987</th>\n",
       "      <td>21888.88903</td>\n",
       "      <td>19007.19129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gdpPercap_1992</th>\n",
       "      <td>23424.76683</td>\n",
       "      <td>18363.32494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gdpPercap_1997</th>\n",
       "      <td>26997.93657</td>\n",
       "      <td>21050.41377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gdpPercap_2002</th>\n",
       "      <td>30687.75473</td>\n",
       "      <td>23189.80135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gdpPercap_2007</th>\n",
       "      <td>34435.36744</td>\n",
       "      <td>25185.00911</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "country           Australia  New Zealand\n",
       "gdpPercap_1952  10039.59564  10556.57566\n",
       "gdpPercap_1957  10949.64959  12247.39532\n",
       "gdpPercap_1962  12217.22686  13175.67800\n",
       "gdpPercap_1967  14526.12465  14463.91893\n",
       "gdpPercap_1972  16788.62948  16046.03728\n",
       "gdpPercap_1977  18334.19751  16233.71770\n",
       "gdpPercap_1982  19477.00928  17632.41040\n",
       "gdpPercap_1987  21888.88903  19007.19129\n",
       "gdpPercap_1992  23424.76683  18363.32494\n",
       "gdpPercap_1997  26997.93657  21050.41377\n",
       "gdpPercap_2002  30687.75473  23189.80135\n",
       "gdpPercap_2007  34435.36744  25185.00911"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#to transform data use .T\n",
    "data.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>country</th>\n",
       "      <th>Australia</th>\n",
       "      <th>New Zealand</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>12.000000</td>\n",
       "      <td>12.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>19980.595634</td>\n",
       "      <td>17262.622813</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>7815.405220</td>\n",
       "      <td>4409.009167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>10039.595640</td>\n",
       "      <td>10556.575660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>13948.900203</td>\n",
       "      <td>14141.858698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>18905.603395</td>\n",
       "      <td>16933.064050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>24318.059265</td>\n",
       "      <td>19517.996910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>34435.367440</td>\n",
       "      <td>25185.009110</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "country     Australia   New Zealand\n",
       "count       12.000000     12.000000\n",
       "mean     19980.595634  17262.622813\n",
       "std       7815.405220   4409.009167\n",
       "min      10039.595640  10556.575660\n",
       "25%      13948.900203  14141.858698\n",
       "50%      18905.603395  16933.064050\n",
       "75%      24318.059265  19517.996910\n",
       "max      34435.367440  25185.009110"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#use describe to get some summary stats about the data\n",
    "data.T.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Excerise IV\n",
    "Read the data in **gapminder_gdp_americas.csv** (which should be in the same directory as gapminder_gdp_oceania.csv, which is the data directory './data') into a variable called **americas** and display its summary statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "americas = pd.read_csv('./data/gapminder_gdp_americas.csv', index_col='country')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gdpPercap_1952</th>\n",
       "      <th>gdpPercap_1957</th>\n",
       "      <th>gdpPercap_1962</th>\n",
       "      <th>gdpPercap_1967</th>\n",
       "      <th>gdpPercap_1972</th>\n",
       "      <th>gdpPercap_1977</th>\n",
       "      <th>gdpPercap_1982</th>\n",
       "      <th>gdpPercap_1987</th>\n",
       "      <th>gdpPercap_1992</th>\n",
       "      <th>gdpPercap_1997</th>\n",
       "      <th>gdpPercap_2002</th>\n",
       "      <th>gdpPercap_2007</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>25.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>25.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>4079.062552</td>\n",
       "      <td>4616.043733</td>\n",
       "      <td>4901.541870</td>\n",
       "      <td>5668.253496</td>\n",
       "      <td>6491.334139</td>\n",
       "      <td>7352.007126</td>\n",
       "      <td>7506.737088</td>\n",
       "      <td>7793.400261</td>\n",
       "      <td>8044.934406</td>\n",
       "      <td>8889.300863</td>\n",
       "      <td>9287.677107</td>\n",
       "      <td>11003.031625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3001.727522</td>\n",
       "      <td>3312.381083</td>\n",
       "      <td>3421.740569</td>\n",
       "      <td>4160.885560</td>\n",
       "      <td>4754.404329</td>\n",
       "      <td>5355.602518</td>\n",
       "      <td>5530.490471</td>\n",
       "      <td>6665.039509</td>\n",
       "      <td>7047.089191</td>\n",
       "      <td>7874.225145</td>\n",
       "      <td>8895.817785</td>\n",
       "      <td>9713.209302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1397.717137</td>\n",
       "      <td>1544.402995</td>\n",
       "      <td>1662.137359</td>\n",
       "      <td>1452.057666</td>\n",
       "      <td>1654.456946</td>\n",
       "      <td>1874.298931</td>\n",
       "      <td>2011.159549</td>\n",
       "      <td>1823.015995</td>\n",
       "      <td>1456.309517</td>\n",
       "      <td>1341.726931</td>\n",
       "      <td>1270.364932</td>\n",
       "      <td>1201.637154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2428.237769</td>\n",
       "      <td>2487.365989</td>\n",
       "      <td>2750.364446</td>\n",
       "      <td>3242.531147</td>\n",
       "      <td>4031.408271</td>\n",
       "      <td>4756.763836</td>\n",
       "      <td>4258.503604</td>\n",
       "      <td>4140.442097</td>\n",
       "      <td>4439.450840</td>\n",
       "      <td>4684.313807</td>\n",
       "      <td>4858.347495</td>\n",
       "      <td>5728.353514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3048.302900</td>\n",
       "      <td>3780.546651</td>\n",
       "      <td>4086.114078</td>\n",
       "      <td>4643.393534</td>\n",
       "      <td>5305.445256</td>\n",
       "      <td>6281.290855</td>\n",
       "      <td>6434.501797</td>\n",
       "      <td>6360.943444</td>\n",
       "      <td>6618.743050</td>\n",
       "      <td>7113.692252</td>\n",
       "      <td>6994.774861</td>\n",
       "      <td>8948.102923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>3939.978789</td>\n",
       "      <td>4756.525781</td>\n",
       "      <td>5180.755910</td>\n",
       "      <td>5788.093330</td>\n",
       "      <td>6809.406690</td>\n",
       "      <td>7674.929108</td>\n",
       "      <td>8997.897412</td>\n",
       "      <td>7807.095818</td>\n",
       "      <td>8137.004775</td>\n",
       "      <td>9767.297530</td>\n",
       "      <td>8797.640716</td>\n",
       "      <td>11977.574960</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>13990.482080</td>\n",
       "      <td>14847.127120</td>\n",
       "      <td>16173.145860</td>\n",
       "      <td>19530.365570</td>\n",
       "      <td>21806.035940</td>\n",
       "      <td>24072.632130</td>\n",
       "      <td>25009.559140</td>\n",
       "      <td>29884.350410</td>\n",
       "      <td>32003.932240</td>\n",
       "      <td>35767.433030</td>\n",
       "      <td>39097.099550</td>\n",
       "      <td>42951.653090</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       gdpPercap_1952  gdpPercap_1957  gdpPercap_1962  gdpPercap_1967  \\\n",
       "count       25.000000       25.000000       25.000000       25.000000   \n",
       "mean      4079.062552     4616.043733     4901.541870     5668.253496   \n",
       "std       3001.727522     3312.381083     3421.740569     4160.885560   \n",
       "min       1397.717137     1544.402995     1662.137359     1452.057666   \n",
       "25%       2428.237769     2487.365989     2750.364446     3242.531147   \n",
       "50%       3048.302900     3780.546651     4086.114078     4643.393534   \n",
       "75%       3939.978789     4756.525781     5180.755910     5788.093330   \n",
       "max      13990.482080    14847.127120    16173.145860    19530.365570   \n",
       "\n",
       "       gdpPercap_1972  gdpPercap_1977  gdpPercap_1982  gdpPercap_1987  \\\n",
       "count       25.000000       25.000000       25.000000       25.000000   \n",
       "mean      6491.334139     7352.007126     7506.737088     7793.400261   \n",
       "std       4754.404329     5355.602518     5530.490471     6665.039509   \n",
       "min       1654.456946     1874.298931     2011.159549     1823.015995   \n",
       "25%       4031.408271     4756.763836     4258.503604     4140.442097   \n",
       "50%       5305.445256     6281.290855     6434.501797     6360.943444   \n",
       "75%       6809.406690     7674.929108     8997.897412     7807.095818   \n",
       "max      21806.035940    24072.632130    25009.559140    29884.350410   \n",
       "\n",
       "       gdpPercap_1992  gdpPercap_1997  gdpPercap_2002  gdpPercap_2007  \n",
       "count       25.000000       25.000000       25.000000       25.000000  \n",
       "mean      8044.934406     8889.300863     9287.677107    11003.031625  \n",
       "std       7047.089191     7874.225145     8895.817785     9713.209302  \n",
       "min       1456.309517     1341.726931     1270.364932     1201.637154  \n",
       "25%       4439.450840     4684.313807     4858.347495     5728.353514  \n",
       "50%       6618.743050     7113.692252     6994.774861     8948.102923  \n",
       "75%       8137.004775     9767.297530     8797.640716    11977.574960  \n",
       "max      32003.932240    35767.433030    39097.099550    42951.653090  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "americas.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Excerise V\n",
    "After reading the data for the Americas, use **help(americas.head)** and **help(americas.tail)** to find out what **DataFrame.head** and **DataFrame.tail** do.\n",
    " 1. What method call will display the first three rows of this data?\n",
    " 2. What method call will display the last three columns of this data? (Hint: you may need to change your view of the data.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>continent</th>\n",
       "      <th>gdpPercap_1952</th>\n",
       "      <th>gdpPercap_1957</th>\n",
       "      <th>gdpPercap_1962</th>\n",
       "      <th>gdpPercap_1967</th>\n",
       "      <th>gdpPercap_1972</th>\n",
       "      <th>gdpPercap_1977</th>\n",
       "      <th>gdpPercap_1982</th>\n",
       "      <th>gdpPercap_1987</th>\n",
       "      <th>gdpPercap_1992</th>\n",
       "      <th>gdpPercap_1997</th>\n",
       "      <th>gdpPercap_2002</th>\n",
       "      <th>gdpPercap_2007</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Argentina</th>\n",
       "      <td>Americas</td>\n",
       "      <td>5911.315053</td>\n",
       "      <td>6856.856212</td>\n",
       "      <td>7133.166023</td>\n",
       "      <td>8052.953021</td>\n",
       "      <td>9443.038526</td>\n",
       "      <td>10079.026740</td>\n",
       "      <td>8997.897412</td>\n",
       "      <td>9139.671389</td>\n",
       "      <td>9308.418710</td>\n",
       "      <td>10967.281950</td>\n",
       "      <td>8797.640716</td>\n",
       "      <td>12779.379640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bolivia</th>\n",
       "      <td>Americas</td>\n",
       "      <td>2677.326347</td>\n",
       "      <td>2127.686326</td>\n",
       "      <td>2180.972546</td>\n",
       "      <td>2586.886053</td>\n",
       "      <td>2980.331339</td>\n",
       "      <td>3548.097832</td>\n",
       "      <td>3156.510452</td>\n",
       "      <td>2753.691490</td>\n",
       "      <td>2961.699694</td>\n",
       "      <td>3326.143191</td>\n",
       "      <td>3413.262690</td>\n",
       "      <td>3822.137084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Brazil</th>\n",
       "      <td>Americas</td>\n",
       "      <td>2108.944355</td>\n",
       "      <td>2487.365989</td>\n",
       "      <td>3336.585802</td>\n",
       "      <td>3429.864357</td>\n",
       "      <td>4985.711467</td>\n",
       "      <td>6660.118654</td>\n",
       "      <td>7030.835878</td>\n",
       "      <td>7807.095818</td>\n",
       "      <td>6950.283021</td>\n",
       "      <td>7957.980824</td>\n",
       "      <td>8131.212843</td>\n",
       "      <td>9065.800825</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          continent  gdpPercap_1952  gdpPercap_1957  gdpPercap_1962  \\\n",
       "country                                                               \n",
       "Argentina  Americas     5911.315053     6856.856212     7133.166023   \n",
       "Bolivia    Americas     2677.326347     2127.686326     2180.972546   \n",
       "Brazil     Americas     2108.944355     2487.365989     3336.585802   \n",
       "\n",
       "           gdpPercap_1967  gdpPercap_1972  gdpPercap_1977  gdpPercap_1982  \\\n",
       "country                                                                     \n",
       "Argentina     8052.953021     9443.038526    10079.026740     8997.897412   \n",
       "Bolivia       2586.886053     2980.331339     3548.097832     3156.510452   \n",
       "Brazil        3429.864357     4985.711467     6660.118654     7030.835878   \n",
       "\n",
       "           gdpPercap_1987  gdpPercap_1992  gdpPercap_1997  gdpPercap_2002  \\\n",
       "country                                                                     \n",
       "Argentina     9139.671389     9308.418710    10967.281950     8797.640716   \n",
       "Bolivia       2753.691490     2961.699694     3326.143191     3413.262690   \n",
       "Brazil        7807.095818     6950.283021     7957.980824     8131.212843   \n",
       "\n",
       "           gdpPercap_2007  \n",
       "country                    \n",
       "Argentina    12779.379640  \n",
       "Bolivia       3822.137084  \n",
       "Brazil        9065.800825  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#head 3\n",
    "americas.head(n=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "country\n",
       "Argentina               5911.315053\n",
       "Bolivia                 2677.326347\n",
       "Brazil                  2108.944355\n",
       "Canada                 11367.161120\n",
       "Chile                   3939.978789\n",
       "Colombia                2144.115096\n",
       "Costa Rica              2627.009471\n",
       "Cuba                    5586.538780\n",
       "Dominican Republic      1397.717137\n",
       "Ecuador                 3522.110717\n",
       "El Salvador             3048.302900\n",
       "Guatemala               2428.237769\n",
       "Haiti                   1840.366939\n",
       "Honduras                2194.926204\n",
       "Jamaica                 2898.530881\n",
       "Mexico                  3478.125529\n",
       "Nicaragua               3112.363948\n",
       "Panama                  2480.380334\n",
       "Paraguay                1952.308701\n",
       "Peru                    3758.523437\n",
       "Puerto Rico             3081.959785\n",
       "Trinidad and Tobago     3023.271928\n",
       "United States          13990.482080\n",
       "Uruguay                 5716.766744\n",
       "Venezuela               7689.799761\n",
       "Name: gdpPercap_1952, dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#how to access individual columns e.g. americas.gdpPercap_1952\n",
    "americas['gdpPercap_1952']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Dominican Republic'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#how do access the index of a max/min column\n",
    "americas[americas['gdpPercap_1952'] == americas['gdpPercap_1952'].min()].index[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Excerise VI\n",
    "* What is the country with the lowest gdp in 2007?\n",
    "* What is the mean gdp in 2002?\n",
    "* What is the fold difference between the mean gdp in 2002 and the higest gdb in 2007?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Haiti'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "americas[americas['gdpPercap_2007'] == americas['gdpPercap_2007'].min()].index[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>continent</th>\n",
       "      <th>gdpPercap_1952</th>\n",
       "      <th>gdpPercap_1957</th>\n",
       "      <th>gdpPercap_1962</th>\n",
       "      <th>gdpPercap_1967</th>\n",
       "      <th>gdpPercap_1972</th>\n",
       "      <th>gdpPercap_1977</th>\n",
       "      <th>gdpPercap_1982</th>\n",
       "      <th>gdpPercap_1987</th>\n",
       "      <th>gdpPercap_1992</th>\n",
       "      <th>gdpPercap_1997</th>\n",
       "      <th>gdpPercap_2002</th>\n",
       "      <th>gdpPercap_2007</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [continent, gdpPercap_1952, gdpPercap_1957, gdpPercap_1962, gdpPercap_1967, gdpPercap_1972, gdpPercap_1977, gdpPercap_1982, gdpPercap_1987, gdpPercap_1992, gdpPercap_1997, gdpPercap_2002, gdpPercap_2007]\n",
       "Index: []"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "americas[americas['gdpPercap_2002'] == americas['gdpPercap_2002'].mean()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_2007 = americas['gdpPercap_2002'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_2007 = americas['gdpPercap_2002'].max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.209567053013284"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_2007 / mean_2007"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Excerise VII\n",
    "As well as the **read_csv** function for reading data from a file, Pandas provides a **to_csv** function to write dataframes to files. Applying what you’ve learned about reading from files, write one of your dataframes to a file called **processed.csv** in the **./data** folder. You can use **help** to get information on how to use **to_csv**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "americas.to_csv?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "americas.to_csv('./data/processed.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use help() function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Key points\n",
    "\n",
    "\n",
    "* Use the Pandas library to get basic statistics out of tabular data.\n",
    "* Use index_col to specify that a column’s values should be used as row headings.\n",
    "* Use DataFrame.info to find out more about a dataframe.\n",
    "* The DataFrame.columns variable stores information about the dataframe’s columns.\n",
    "* Use DataFrame.T to transpose a dataframe.\n",
    "* Use DataFrame.describe to get summary statistics about data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More on Pandas DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions you should be able to answer after this session\n",
    "* How can I slice, filter, and manipulate my data in a dataframe to do statistical analysis?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Objectives\n",
    "\n",
    "* Select individual values from a Pandas dataframe.\n",
    "* Select entire rows or entire columns from a dataframe.\n",
    "* Select a subset of both rows and columns from a dataframe in a single operation.\n",
    "* Select a subset of a dataframe by a single Boolean criterion.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Side note about Pandas DataFrames/Series\n",
    "A [DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) is a collection of [Series](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html); The DataFrame is the way Pandas represents a table, and Series is the data-structure Pandas use to represent a column.\n",
    "\n",
    "Pandas is built on top of the [Numpy](http://www.numpy.org/) library, which in practice means that most of the methods defined for Numpy Arrays apply to Pandas Series/DataFrames.\n",
    "\n",
    "What makes Pandas so attractive is the powerful interface to access individual records of the table, proper handling of missing values, and relational-databases operations between DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from IPython.display import Image\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selecting values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#now read in europe for this one\n",
    "data = pd.read_csv('data/gapminder_gdp_europe.csv', index_col='country')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#show head again\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use iloc by entry position (exlusive)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use loc by index and column name (inclusive)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#now print all columnns for Albania. Consider outcompletion\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get all values for a specific year\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#now use dot access of columns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#select data for certain years and countries 'Italy':'Poland', 'gdpPercap_1962':'gdpPercap_1972'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#select subsets and apply functions e.g. max()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Filter dataframes using Boolean masks\n",
    "* apply an comparison element per element\n",
    "* returns an identically shaped \"True/False\" dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use a subset of data to keep output readable.\n",
    "subset = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print subset dataframe\n",
    "print('Subset of data:\\n', subset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Which values were greater than 10000 e.g. subset > 10000 ?\n",
    "print('\\nWhere are values large?\\n', ___)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#make a mask to filter the subset dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use mask to filter subset e.g. subset[mask]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Get the value where the mask is true, and NaN (Not a Number) where it is false.\n",
    "* Useful because NaNs are ignored by operations like max, min, average, etc.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use describe to illustrate the NaN here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Select-Apply-Combine operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas vectorizing methods and grouping operations are features that provide users much flexibility to analyse their data.\n",
    "\n",
    "For instance, let’s say we want to have a clearer view on how the European countries split themselves according to their GDP.\n",
    "\n",
    "1. We may have a glance by splitting the countries in two groups during the years surveyed, those who presented a GDP higher than the European average and those with a lower GDP.\n",
    "1. We then estimate a wealthy score based on the historical (from 1962 to 2007) values, where we account how many times a country has participated in the groups of lower or higher GDP\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "#talk people through at the board.\n",
    "data.gdpPercap_1952.apply(lambda x: x> data.gdpPercap_1952.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create a higher mask\n",
    "mask_higher = \n",
    "mask_higher"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#aggregate the sum across the columns axis=1 e.g. mask_higher.aggregate('sum',axis=1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How does pandas handle axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='./fig/pandas_axis.png', height=800, width =800 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#now generate the wealth_score as % of times the country has been above the mean gdp of europe using e.g. /len(data.columns)*100, 0\n",
    "wealth_score = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#make a new column of the wealth_score\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#now groupby Wealth_score and count how many there are\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#now groupby wealth score and sum in each category\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise VIII\n",
    "Import the gapminder for Europe (again) into the variable **europe**. Write an expression that gives you the GDP for 2007 for the country Serbia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use read_csv and loc or iloc\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise IX\n",
    "1. Do the two statements below produce the same output?\n",
    "1. Based on this, what rule governs what is included (or not) in numerical slices and named slices in Pandas?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use this command below to check how the dataframe is setup\n",
    "europe.head()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "print(europe.iloc[0:2, 0:2])\n",
    "print(europe.loc['Albania':'Belgium', 'gdpPercap_1952':'gdpPercap_1962'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise X\n",
    "explain what each of the following does e.g. **first**, **second**, etc.?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "first = pd.read_csv('data/gapminder_all.csv', index_col='country')\n",
    "second = first[first['continent'] == 'Americas']\n",
    "third = second.drop('Puerto Rico')\n",
    "fourth = third.drop('continent', axis = 1)\n",
    "fourth.to_csv('result.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise X\n",
    "Explain in simple terms what **idxmin** and **idxmax** do. Use the **data** dataframe we generate here. When would you use these methods? When can these methods be an issue?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data.idxmin())\n",
    "print(data.idxmax())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise XI\n",
    "Use the europe dataframe you are generated before and try to answer the following question. Write an expression to select each of the following:\n",
    "\n",
    "1. GDP per capita for all countries in 1982.\n",
    "1. GDP per capita for Denmark for all years.\n",
    "1. GDP per capita for all countries for years after 1985.\n",
    "1. GDP per capita for each country in 2007 as a multiple of GDP per capita for that country in 1952.\n",
    "    * Make this a new column called \"Growth_1952-2007\".\n",
    "    * Which country had the biggest growth.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#check out the columns to see their names\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#or use the head method to see how the dataframe looks like\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#GDP per capita for all countries in 1982.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#GDP per capita for Denmark for all years.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#GDP per capita after 1985\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#GDP per capita for each country in 2007 as a multiple of GDP per capita for that country in 1952.\n",
    "#Make this a new column called \"Growth_1952-2007\".\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Which country had the biggest growth\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}