DS102-Statistical-Programming-in-R/DS102-L9-Data-Exploration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "50e6ddbb-52d3-4a02-9b69-100a4181e3b0",
   "metadata": {},
   "source": [
    "# DS102 Statistical Programming in R : Lesson Nine - Data Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1bd4bfc-9b15-4b80-87b4-b8b60eb56371",
   "metadata": {},
   "source": [
    "### Table of Contents <a class=\"anchor\" id=\"DS102L9_toc\"></a>\n",
    "\n",
    "* [Table of Contents](#DS102L9_toc)\n",
    "    * [Page 1 - Introduction](#DS102L9_page_1)\n",
    "    * [Page 2 - Data Exploration Part I](#DS102L9_page_2)\n",
    "    * [Page 3 - Data Exploration Part II](#DS102L9_page_3)\n",
    "    * [Page 4 - Data Exploration Part III](#DS102L9_page_4)\n",
    "    * [Page 5 - Key Terms](#DS102L9_page_5)\n",
    "    * [Page 6 - Hands On](#DS102L9_page_6)\n",
    "    * [Page 7 - The Terminal](#DS102L9_page_7)\n",
    "    * [Page 8 - Command Line Interface (CLI)](#DS102L9_page_8)\n",
    "    * [Page 9 - File Organization and Paths](#DS102L9_page_9)\n",
    "    * [Page 10 - Showing Files and Changing Directories](#DS102L9_page_10)\n",
    "    * [Page 11 - Learning the CLI](#DS102L9_page_11)\n",
    "    * [Page 12 - CLI Activity](#DS102L9_page_12)\n",
    "    * [Page 13 - Key Terms](#DS102L9_page_13)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef5a8c65-ed48-4354-a257-e01b0fd733d4",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 1 -  Introduction <a class=\"anchor\" id=\"DS102L9_page_1\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b0bb0cf7-e5c2-4931-a8fd-01fd52bff0b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/247057983\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7fb91029f940>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Exploration\n",
    "VimeoVideo('247057983', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b6ad526-95fa-43c4-bc43-1fe807708ed3",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "Now that you have all these newfound data skills in R, you will put them to use in a lesson on data exploration.  Often, data scientists are given specific questions that they need to answer, and that will guide their data analysis, but sometimes, you may be given a pile of data and are expected to make something out of it.  That's when data exploration comes in.  \n",
    "\n",
    "By the end of the first part of this lesson, you will be able to: \n",
    "\n",
    "* Employ ```ggplot``` to make effective and informative exploration graphs\n",
    "* Utilize data manipulation skills from ```dyplyr``` to wrangle and explore data\n",
    "* Compute summary statistics\n",
    "\n",
    "This lesson will also introduce you to some of the concepts you'll need in later modules about working with your computer through the command line, or terminal (depending on your operating system).  \n",
    "\n",
    "By the end of the second part of this lesson, you will understand: \n",
    "\n",
    "* How to use your terminal\n",
    "* File organization, directories, and paths\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>You may want to watch this <a href=\"https://vimeo.com/427596329\"> recorded live workshop </a> that goes over the material in this lesson regarding your terminal. </p>\n",
    "    </div>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8fd3c3d3-ccbd-42c8-a2d4-16ed34985b86",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/427596329\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7fb91029fb20>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Exploration\n",
    "VimeoVideo('427596329', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7990ac7a-5a2f-4744-af14-1b7e385ea484",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 2 - Data Exploration Part 1<a class=\"anchor\" id=\"DS102L9_page_2\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1f97ec3e-e5bc-4f02-adbb-a38a5963dfdd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/329835610\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7fb91029f310>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Exploration\n",
    "VimeoVideo('329835610', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612429df-e4ef-4d85-8460-5bc8914079c3",
   "metadata": {},
   "source": [
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO102-L09-pg2tutorial.zip)**.\n",
    "\n",
    "# Loading in and Examining the Data\n",
    "\n",
    "The **[GapMinder website](https://www.gapminder.org/)** has a large collection of statistics about health and economic development for countries around the world. Its founder, Hans Rosling, has developed compelling ways to present statistical data. An R data set is available called ```Gapminder```, and you will use what you have learned about data manipulation and graphing to explore that dataset.\n",
    "\n",
    "\n",
    "The first order of business is to install the ```gapminder``` package and make it available to R. You do this with the following commands:\n",
    "\n",
    "```{r}\n",
    "install.packages(\"gapminder\")\n",
    "library(gapminder)\n",
    "head(gapminder)\n",
    "```\n",
    "\n",
    "Here is the result: \n",
    "\n",
    "```text\n",
    "# A tibble: 6 x 6`\n",
    "\n",
    "country continent year lifeExp pop gdpPercap\n",
    "\n",
    "<fctr> <fctr> <int> <dbl> <int> <dbl>\n",
    "\n",
    "1 Afghanistan Asia 1952 28.801 8425333 779.4453\n",
    "\n",
    "2 Afghanistan Asia 1957 30.332 9240934 820.8530\n",
    "\n",
    "3 Afghanistan Asia 1962 31.997 10267083 853.1007\n",
    "\n",
    "4 Afghanistan Asia 1967 34.020 11537966 836.1971\n",
    "\n",
    "5 Afghanistan Asia 1972 36.088 13079460 739.9811\n",
    "\n",
    "6 Afghanistan Asia 1977 38.438 14880372 786.1134\n",
    "```\n",
    "\n",
    "From the package documentation, you will find that it has the following variables:\n",
    "\n",
    "* **```country```:** The country in question. This is a factor with 142 levels.\n",
    "* **```continent```:** The continent in which the country resides. This is a factor with 5 levels.\n",
    "* **```year```:** The year for the given data. The year values range from 1952 to 2007 in steps of 5 years.\n",
    "* **```lifeExy```:** The life expectancy at birth.\n",
    "* **```pop```:** The population of the country.\n",
    "* **```gdpPercap```:** The per capita gross domestic product of the country; this is a measure of individual productivity and wealth.\n",
    "\n",
    "You can see all of the countries in the data set if you just type in ```country```.  You see that it is a factor.  A factor has levels, which are the different values that the factor can take. The ```levels()``` function shows you the levels of the factor country:\n",
    "\n",
    "```{r}\n",
    "levels(gapminder$country)\n",
    "```\n",
    "\n",
    "And here is the output:\n",
    "\n",
    "```text\n",
    "[1] \"Afghanistan\" \"Albania\" \"Algeria\"\n",
    "\n",
    "[4] \"Angola\" \"Argentina\" \"Australia\"\n",
    "\n",
    "[7] \"Austria\" \"Bahrain\" \"Bangladesh\"\n",
    "\n",
    "[10] \"Belgium\" \"Benin\" \"Bolivia\"\n",
    "```\n",
    "\n",
    "You can see that there are 142 countries represented in the data set.\n",
    "\n",
    "```year``` is a numerical variable, so it does not have levels. You can see what values the year variable takes by using the ```unique()``` function. This function will return a vector of all the unique values in a vector:\n",
    "\n",
    "```{r}\n",
    "unique(gapminder$year)\n",
    "```\n",
    "\n",
    "[1] 1952 1957 1962 1967 1972 1977 1982 1987 1992 1997 2002 2007\n",
    "\n",
    "So you see that the data set starts in 1952 and includes data in five year increments until 2007.\n",
    "\n",
    "---\n",
    "\n",
    "# Graphing the Population\n",
    "\n",
    "You can get a feel for how the population of the countries has changed over the years by creating a box plot for each year value.\n",
    "\n",
    "```{r}\n",
    "ggplot(gapminder, aes(x = factor(year), y = pop)) + geom_boxplot()\n",
    "```\n",
    "\n",
    "This code yields the following graph:\n",
    "\n",
    "![A box plot showing how the population of countries has changed over the years. The x axis is labeled factor open parentheses year close parentheses and runs from nineteen fifty two to two thousand seven in increments of five years. The y axis is labeled pop, for population, and runs from zero e plus zero zero to more than 1 e plus zero nine. There is a separate box plot for each year indicated.](Media/exploration2.png)\n",
    "\n",
    "This shows that some years had a very high population, but it makes it quite difficult to see the population of the smaller countries. You can change the vertical scale to be logarithmic; this allows you to see the small values as well as the large ones:\n",
    "\n",
    "```{r}\n",
    "ggplot(gapminder, aes(x = factor(year), y = pop)) + geom_boxplot() +\n",
    "scale_y_log10()\n",
    "```\n",
    "\n",
    "See how things look much more spread out along the y axis? \n",
    "\n",
    "![A box plot showing how the population of countries has changed over the years. The x axis is labeled factor open parentheses year close parentheses and runs from nineteen fifty two to two thousand seven in increments of five years. The y axis is labeled pop, for population, and runs from less than one e plus zero five to one e plus zero nine. There is a separate box plot for each year indicated.](Media/exploration1.png)\n",
    "\n",
    "In the logarithmic scale, each grid line is ten times the one below it. 1e+06 is scientific notation for one million (1,000,000), so the grid line labeled 1e+06 represents 1 million people. The next grid line up, which is largely hidden by the boxes, is ten times one million, or 10 million (10,000,000). The next grid line up is ten times 10 million, or 100 million. And finally, the top line is ten times 100 million, which is 1 billion. You can see that around 1982 one country passed the 1 billion population mark, and around 1987 a second country passed this mark.\n",
    "\n",
    "---\n",
    "\n",
    "# Filtering and Arranging by Population\n",
    "\n",
    "You can find the largest countries in 2007 by first filtering the rows to get those for 2007 and then sorting these rows by the pop variable. This is done with the ```filter()``` and ```arrange()``` functions:\n",
    "\n",
    "```{r}\n",
    "gm.big <- gapminder %>%\n",
    "filter(year == 2007) %>%\n",
    "arrange(desc(pop))\n",
    "```\n",
    "\n",
    "These commands first use the ```gapminder``` data frame in the ```filter()``` function with a selection criteria of year == 2007; this finds all rows from 2007. These rows are then used in the ```arrange()``` function, which rearranges the order of the rows. The ```desc(pop)``` argument tells the ```arrange()``` function to order the rows in descending order of population; descending means that the largest number is first. You can look at the resulting tibble as follows; the argument ```n = 10 to head()``` indicates to print out the first 10 rows.\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>If you get an error when running this code, try loading the library dplyr! You'll need it to perform filtering and other operations.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "```{r}\n",
    "head(gm.big, n = 10)\n",
    "```\n",
    "\n",
    "Here is the result:\n",
    "\n",
    "```text\n",
    "# A tibble: 10 x 6\n",
    "\n",
    "country continent year lifeExp pop gdpPercap\n",
    "\n",
    "<fctr> <fctr> <int> <dbl> <int> <dbl>\n",
    "\n",
    "1 China Asia 2007 72.961 1318683096 4959.115\n",
    "\n",
    "2 India Asia 2007 64.698 1110396331 2452.210\n",
    "\n",
    "3 United States Americas 2007 78.242 301139947 42951.653\n",
    "\n",
    "4 Indonesia Asia 2007 70.650 223547000 3540.652\n",
    "\n",
    "5 Brazil Americas 2007 72.390 190010647 9065.801\n",
    "\n",
    "6 Pakistan Asia 2007 65.483 169270617 2605.948\n",
    "\n",
    "7 Bangladesh Asia 2007 64.062 150448339 1391.254\n",
    "\n",
    "8 Nigeria Africa 2007 46.859 135031164 2013.977\n",
    "\n",
    "9 Japan Asia 2007 82.603 127467972 31656.068\n",
    "\n",
    "10 Mexico Americas 2007 76.195 108700891 11977.575\n",
    "```\n",
    "\n",
    "The list of big countries is topped by China, then India, and then the United States. You can find the smallest countries by using the ```tail()``` function:\n",
    "\n",
    "```{r}\n",
    "tail(gm.big, n = 10)\n",
    "```\n",
    "\n",
    "Which provides the bottom 10 rows of data in the tibble:\n",
    "\n",
    "```text\n",
    "# A tibble: 10 x 6`\n",
    "\n",
    "country continent year lifeExp pop gdpPercap\n",
    "\n",
    "`<fctr> <fctr> <int> <dbl> <int> <dbl>`\n",
    "\n",
    "1 Swaziland Africa 2007 39.613 1133066 4513.4806\n",
    "\n",
    "2 Trinidad and Tobago Americas 2007 69.819 1056608 18008.5092\n",
    "\n",
    "3 Reunion Africa 2007 76.442 798094 7670.1226\n",
    "\n",
    "4 Comoros Africa 2007 65.152 710960 986.1479\n",
    "\n",
    "5 Bahrain Asia 2007 75.635 708573 29796.0483\n",
    "\n",
    "6 Montenegro Europe 2007 74.543 684736 9253.8961\n",
    "\n",
    "7 Equatorial Guinea Africa 2007 51.579 551201 12154.0897\n",
    "\n",
    "8 Djibouti Africa 2007 54.791 496374 2082.4816\n",
    "\n",
    "9 Iceland Europe 2007 81.757 301931 36180.7892\n",
    "\n",
    "10 Sao Tome and Principe Africa 2007 65.528 199579 1598.4351\n",
    "```\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Fun Fact!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Instead of printing the bottom of this dataset, you could also choose to arrange by population rather than population descending.  This would put the smallest countries first!</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Determining Outliers in Per Capita GDP\n",
    "\n",
    "You can plot the per capita GDP for each year:\n",
    "\n",
    "```{r}\n",
    "ggplot(gapminder, aes(x = factor(year), y = gdpPercap)) + geom_boxplot()\n",
    "```\n",
    "\n",
    "Providing this graph:\n",
    "\n",
    "![A box plot showing how the population of countries has changed over the years. The x axis is labeled factor open parentheses year close parentheses and runs from nineteen fifty two to two thousand seven in increments of five years. The y axis is labeled G D P per capita, and runs from zero to one hundred twenty thousand. There is a separate box plot for each year indicated.](Media/L09-GDPBoxplot.png)\n",
    "\n",
    "On this plot, you see that there are outliers with a high per capita GDP between 1952 and 1977. You can find out which country or countries these outliers represent by first calculating what values would be considered outliers. Thinking back a few lessons, you'll remember that you need to find the interquartile range, multiply it by 1.5, and then add it to the third quartile boundary: \n",
    "\n",
    "```text\n",
    "9325.5 - 1202.1 = 8123.4\n",
    "8123.4 x 1.5 = 12,185.1\n",
    "12,185.1 + 9325.5 = 13,988.25\n",
    "```\n",
    "\n",
    "with the ```filter()``` function by selecting all rows with a ```gdpPercap``` value greater than 13988:\n",
    "\n",
    "```{r}\n",
    "filter(gapminder, gdpPercap > 13988)\n",
    "```\n",
    "\n",
    "This yields the following results:\n",
    "\n",
    "```text\n",
    "# A tibble: 143 × 6\n",
    "\n",
    "  country  continent year  lifeExp    pop\t   gdpPercap\n",
    "\n",
    "1 Australia\tOceania\t1987\t76.320\t16257249\t21888.89\n",
    "2 Australia\tOceania\t1992\t77.560\t17481977\t23424.77\n",
    "3 Australia\tOceania\t1997\t78.830\t18565243\t26997.94\n",
    "4 Australia\tOceania\t2002\t80.370\t19546792\t30687.75\n",
    "5 Australia\tOceania\t2007\t81.235\t20434176\t34435.37\n",
    "6 Austria\tEurope\t1982\t73.180\t7574613\t    21597.08\n",
    "7 Austria\tEurope\t1987\t74.940\t7578903\t    23687.83\n",
    "8 Austria\tEurope\t1992\t76.040\t7914969\t    27042.02\n",
    "9 Austria\tEurope\t1997\t77.510\t8069876\t    29095.92\n",
    "10 Austria\tEurope\t2002\t78.980\t8148312\t    32417.61\n",
    "\n",
    "# ... with 133 more rows\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Graphing Life Expectancy\n",
    "\n",
    "Finally, you can plot the life expectancy values for each year:\n",
    "\n",
    "```{r}\n",
    "ggplot(gapminder, aes(x = factor(year), y = lifeExp)) + geom_boxplot()\n",
    "```\n",
    "\n",
    "![A box plot showing how the population of countries has changed over the years. The x axis is labeled factor open parentheses year close parentheses and runs from nineteen fifty two to two thousand seven in increments of five years. The y axis is labeled life expectancy and runs from twenty to eighty five. There is a separate box plot for each year indicated.](Media/L09-LifeBoxPLot.png)\n",
    "\n",
    "It appears that one country had a very low life expectancy in 1992; you can find this country as follows:\n",
    "\n",
    "```{r}\n",
    "filter(gapminder, lifeExp < 28)\n",
    "```\n",
    "\n",
    "Which yields a one-row tibble:\n",
    "\n",
    "```text\n",
    "# A tibble: 1 x 6\n",
    "\n",
    "country continent year lifeExp pop gdpPercap\n",
    "\n",
    "`<fctr> <fctr> <int> <dbl> <int> <dbl>`\n",
    "\n",
    "1 Rwanda Africa 1992 23.599 7290203 737.0686\n",
    "```\n",
    "---\n",
    "\n",
    "## Review\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fb8e5e22-74cf-416f-862e-37f38c208987",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from DS_Students import MultipleChoice\n",
    "    from ipynb.fs.full.DS102Questions import *\n",
    "except:\n",
    "    !pip install DS_Students\n",
    "    from DS_Students import MultipleChoice\n",
    "    from ipynb.fs.full.DS102Questions import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6bbb24aa-918f-4dda-b4db-c880a893d245",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2620fbbf81e04339b48086451ab2f5dd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '1. Question?\\n', 'output_type': 'stream'},)), Radio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L9P2Q1)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e2f1323-4b72-4038-8ef4-2480c533671c",
   "metadata": {},
   "source": [
    "<p style=\"text-align: center\">\n",
    "  <img src=\"Media/L09-LifeBoxPlot.png\" alt=\"Drawing\" style=\"width: 500px;\"/>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "22f40cf9-0823-4693-86a6-ebdf8c8f3f54",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d5b50d7d581047ef82103264dd587dd1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '2. Question?\\n', 'output_type': 'stream'},)), Radio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L9P2Q2)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1db90c90-1241-4ef4-bf86-8e754633124d",
   "metadata": {},
   "source": [
    "<p style=\"text-align: center\">\n",
    "  <img src=\"Media/L09-LifeBoxPlot.png\" alt=\"Drawing\" style=\"width: 500px;\"/>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "41800960-e6a4-494d-b392-13bca753a0c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5622b892ff044e25a074ff2b694e7e04",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '3. Question?\\n', 'output_type': 'stream'},)), Radio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5112ac6cd40f4d1c81736bb2ed1a9cfa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '4. Question?\\n', 'output_type': 'stream'},)), Radio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L9P2Q3, L9P2Q4)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832358fd-55fa-420a-8aa2-46fbf2097986",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 3 - Data Exploration Part II<a class=\"anchor\" id=\"DS102L9_page_3\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ff8ff4a0-e7a8-41f2-ab0c-5f4f88f49bef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/329835657\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7fb91029fa90>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Exploration\n",
    "VimeoVideo('329835657', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f564e3e7-3a7f-4936-b3a3-2e0a9ab78512",
   "metadata": {
    "tags": []
   },
   "source": [
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO102-L09-pg3tutorial.zip)**.\n",
    "\n",
    "# Comparing Countries\n",
    "\n",
    "---\n",
    "\n",
    "## Focusing on Angola\n",
    "\n",
    "You will continue your exploration of this data set by looking at the data for Angola. As you learned previously, you can get all rows in the ```gapminder``` data set that have data from Angola by using the condition ```country == \"Angola\"``` in the ```filter()``` function:\n",
    "\n",
    "```{r}\n",
    "gm_Angola <- filter(gapminder, country == \"Angola\")\n",
    "head(gm_Angola)\n",
    "```\n",
    "\n",
    "This provides you with the following data:\n",
    "\n",
    "```text\n",
    "# A tibble: 6 x 6\n",
    "\n",
    "country continent year lifeExp pop gdpPercap\n",
    "\n",
    "<fctr> <fctr> <int> <dbl> <int> <dbl>\n",
    "\n",
    "1 Angola Africa 1952 30.015 4232095 3520.610\n",
    "\n",
    "2 Angola Africa 1957 31.999 4561361 3827.940\n",
    "\n",
    "3 Angola Africa 1962 34.000 4826015 4269.277\n",
    "\n",
    "4 Angola Africa 1967 35.985 5247469 5522.776\n",
    "\n",
    "5 Angola Africa 1972 37.928 5894858 5473.288\n",
    "\n",
    "6 Angola Africa 1977 39.483 6162675 3008.647\n",
    "```\n",
    "\n",
    "As you can see, ```gm_Angola``` is now a tibble that has data only on Angola. You can use this tibble to plot life expectancy as a function of the year:\n",
    "\n",
    "```{r}\n",
    "ggplot(gm_Angola) + geom_line(aes(x = year, y = lifeExp)) +\n",
    "ylab(\"Life Expectancy\") + ggtitle(\"Life Expectancy in Angola\")\n",
    "```\n",
    "\n",
    "This creates the plot below:\n",
    "\n",
    "![A line graph titled life expectancy in Angola. The x axis is labeled year and runs from nineteen fifty to two thousand ten. The y axis is labeled life expectancy and runs from twenty eight to approximately forty three. The line rises at roughly forty five degrees from the bottom left, nineteen fifty, until approximately nineteen seventy seven, when it levels out until about nineteen eighty seven before slowly rising again.](Media/L09-AngolaLE.png)\n",
    "\n",
    "You can create a similar plot of per capita GDP for Angola:\n",
    "\n",
    "```{r}\n",
    "ggplot(gm_Angola) + geom_line(aes(x = year, y = gdpPercap)) +\n",
    "ylab(\"Per Capita GDP\") + ggtitle(\"GDP in Angola\")\n",
    "```\n",
    "\n",
    "![A line graph titled G D P in Angola. The x axis is labeled year and runs from nineteen fifty to two thousand ten. The y axis is labeled per capita G D P and runs from approximately two thousand to approximately five thousand seven hundred. The line starts just after nineteen fifty at three thousand five hundred, rises to approximately five thousand five hundred in approximately nineteen sixty seven, slightly decreases until approximately nineteen seventy two, sharply decreases until about nineteen eighty seven, then slightly increases and decreases to a low of less than two thousand five hundred until about nineteen ninety seven, and then rises up to approximately four thousand eight hundred by approximately two thousand seven.](Media/L09-AngolaGDP.png)\n",
    "\n",
    "It looks like between the years of 1977 and 2002, the GDP dropped significantly and the life expectancy did not improve. This coincides with a prolonged civil war in Angola.\n",
    "\n",
    "---\n",
    "\n",
    "## Comparing Four African Countries\n",
    "\n",
    "Next, you do a comparative investigation into four African countries: Angola, Ghana, Ethiopia, and South Africa. \n",
    "\n",
    "---\n",
    "\n",
    "### Filtering to African Countries\n",
    "\n",
    "The first thing to do is to create a data frame with the data from these four countries. You do this with the ```filter()``` function as follows:\n",
    "\n",
    "```{r}\n",
    "gm_Africa4 <- filter(gapminder,\n",
    "country %in% c(\"Angola\", \"Ghana\", \"Ethiopia\", \"South Africa\"))\n",
    "```\n",
    "\n",
    "In this case, the condition used for the ```filter()``` function is ```country %in% c(\"Angola\", \"Ghana\", \"Ethiopia\", \"South Africa\")```; this condition will be true if the value of country is in the vector you specified with the names \"Angola,\" \"Ghana,\" \"Ethiopia,\" and \"South Africa.\" This creates a tibble with only those four countries. \n",
    "\n",
    "---\n",
    "\n",
    "### Eliminating Extraneous Variables\n",
    "\n",
    "If you like, you can clean up the data frame ```gm_Africa4``` by eliminating the variables that you aren't using. This is not strictly necessary, but will may make it easier later on to work with this data.  If you decide you want to examine more data later, however, it will be a pain in the butt to go back. \n",
    "\n",
    "Nevertheless, here's how you'd get only the variables you're interested in at the moment: \n",
    "\n",
    "```{r}\n",
    "gm_AfricaClean <- select(gm_Africa4, country, year, lifeExp, gdpPercap)\n",
    "head(gm_AfricaClean)\n",
    "```\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Always give a new dataset a new name! This way it makes it easy to go backwards if you've made a mistake that's irreversible.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "You will not be using the population variable in your computations. Neither will you use the continent variable, since you now have data only from Africa. The ```select()``` function tells R what rows to keep. Here is the resulting tibble:\n",
    "\n",
    "```text\n",
    "# A tibble: 6 x 4\n",
    "\n",
    "country year lifeExp gdpPercap\n",
    "\n",
    "<fctr> <int> <dbl> <dbl>\n",
    "\n",
    "1 Angola 1952 30.015 3520.610\n",
    "\n",
    "2 Angola 1957 31.999 3827.940\n",
    "\n",
    "3 Angola 1962 34.000 4269.277\n",
    "\n",
    "4 Angola 1967 35.985 5522.776\n",
    "\n",
    "5 Angola 1972 37.928 5473.288\n",
    "\n",
    "6 Angola 1977 39.483 3008.647\n",
    "```\n",
    "\n",
    "--- \n",
    "\n",
    "### Combining Filtering and Selecting\n",
    "\n",
    "If you're feeling sassy, you can combine the filtering and selecting operations into one command using the ```%>%``` operator:\n",
    "\n",
    "```{r}\n",
    "gm_AfricaClean <- gapminder %>%\n",
    "filter(country %in% c(\"Angola\", \"Ghana\", \"Ethiopia\", \"South Africa\")) %>%\n",
    "select(country, year, lifeExp, gdpPercap)\n",
    "```\n",
    "\n",
    "In layman's terms, this code translates to: take the ```gapminder``` data frame and ```filter()``` it to get only the rows for ```Angola, Ghana, Ethiopia, and South Africa```; then, from the resulting data frame, select the ```county, year, lifeExp, and gdpPercap``` variables.\n",
    "\n",
    "---\n",
    "\n",
    "### Plotting Life Expectancy\n",
    "\n",
    "You can create a plot that shows the life expectancy in each of these four countries as a function of time using this command:\n",
    "\n",
    "```{r}\n",
    "ggplot(gm_AfricaClean) + geom_line(aes(x = year, y = lifeExp, color = country)) +\n",
    "ylab(\"Life Expectancy\") + ggtitle(\"Life Expectancy in Four Countries\")\n",
    "```\n",
    "\n",
    "![A line graph titled life expectancy in four countries. Four lines are plotted, one for each country, Angola, Ethiopia, Ghana, and South Africa. The x axis is labeled year and runs from nineteen fifty to two thousand ten. The y axis is labeled life expectancy and runs from thirty to approximately sixty three. The lines for Ghana, Angola, and Ethiopia rise steadily. The line for South Africa rises over time until about nineteen ninety two, when it drops.](Media/L09-FourLE.png)\n",
    "\n",
    "Note that the argument ```color = country``` in the ```aes()``` function assigns a different color to each country; that country's line is plotted in that color. You can create a plot of per capita GDP in a similar way:\n",
    "\n",
    "```{r}\n",
    "ggplot(gm_AfricaClean) + geom_line(aes(x = year, y = gdpPercap, color = country)) +\n",
    "ylab(\"Per Capita GDP\") + ggtitle(\"GDP in Four Countries\")\n",
    "```\n",
    "\n",
    "![A line graph titled G D P in Four Countries. Four lines are plotted, one for each country, Angola, Ethiopia, Ghana, and South Africa. The x axis is labeled year and runs from nineteen fifty to two thousand ten. The y axis is labeled per capita G D P and runs from zero to almost ten thousand. The lines for Ghana and Ethiopia are low, both below one thousand two hundred fifty G D P per capita, and remain relatively flat over time. The line for Angola begins at a higher G D P, at about three thousand five hundred fifty, and rises for about fifteen years before dropping until about nineteen ninety seven, at which point it rises sharply to nearly five thousand per capital G D P. The line for South Africa begins in nineteen fifty at about five thousand per capita G D P and rises to over eight thousand five hundred in about nineteen eighty two. It drops until about nineteen ninety two, then slightly rises until about two thousand and two, and then rises sharply](Media/L09-FourGDP.png)\n",
    "\n",
    "---\n",
    "\n",
    "### Using grid.arrange()\n",
    "\n",
    "When exploring the data, it may be helpful to have both plots together. You can do this as follows with the ```grid.arrange()``` function, which is part of the ```gridExtra``` package. To do this, first install and load ```gridExtra```:\n",
    "\n",
    "```{r}\n",
    "install.packages(\"gridExtra\")\n",
    "library(\"gridExtra\")\n",
    "```\n",
    "\n",
    "Then create the two plots and store them in the variables ```life_exp``` and ```GDP```, then arrange them as follows:\n",
    "\n",
    "```{r}\n",
    "life_exp <- ggplot(gm_AfricaClean) + geom_line(aes(x = year, y = lifeExp, color = country))\n",
    "GDP <- ggplot(gm_AfricaClean) + geom_line(aes(x = year, y = gdpPercap, color = country)) +\n",
    "ylab(\"Life Expectancy\")\n",
    "grid.arrange(life_exp, GDP, ncol = 1)\n",
    "```\n",
    "\n",
    "You call ```grid.arrange()``` with arguments ```life_exp``` and ```GDP```, which are the plots you want to put together. Then add on the argument ```ncol = 1``` to indicate that you want the plots to be in one column, so they are stacked vertically. Simply changing the order of the plots changes which one goes on top. You should get the following plot:\n",
    "\n",
    "![Two line graphs stacked. On the top, life expectancy over time in four countries, Angola, Ethiopia, Ghana, and South Africa. On the bottom, per capita G D P over time in the same four countries.](Media/L09-FourLEGDP.png)\n",
    "\n",
    "You can see from this plot that the life expectancy and the per capita GDP declined in South Africa beginning in the early 1990's. This is most likely a consequence of the devastating HIV/AIDS epidemic in sub-saharan Africa. You will also note that while per capita GDP in Ethiopia and Ghana did not increase dramatically over time, the life expectancy in these two countries did increase dramatically.\n",
    "\n",
    "---\n",
    "\n",
    "### Adding Additional Variables to the Plot\n",
    "\n",
    "You might try an alternate approach to representing both life expectancy and and per capita GDP on a single graph. You can plot the life expectancy both as lines and as points, and make the area of the points be proportional to the per capita GDP. You can do this as follows:\n",
    "\n",
    "```{r}\n",
    "ggplot(gm_AfricaClean, aes(x = year, y = lifeExp, color = country)) +\n",
    "geom_line() + geom_point(aes(size = gdpPercap)) +\n",
    "ylab(\"Life Expectancy\") + ggtitle(\"Life Expectancy and GDP in Four Countries\")\n",
    "```\n",
    "\n",
    "This yields the following graph:\n",
    "\n",
    "![A line graph that uses both lines and points to show the life expectancy and G D P per capita over time in four countries, Angola, Ethiopia, Ghana, and South Africa.](Media/L09-FourLEGDP2.png)\n",
    "\n",
    "You'll note that it is much harder with this plot to see the changes in GDP.\n",
    "\n",
    "---\n",
    "\n",
    "### Plotting GDP and Life Expectancy Against Each Other\n",
    "\n",
    "You might try another approach. You can start by plotting the per capita GDP on the horizontal axis and the life expectancy on the vertical axis as follows:\n",
    "\n",
    "```{r}\n",
    "ggplot(gm_AfricaClean, aes(x = gdpPercap, y = lifeExp, color = country)) +\n",
    "geom_point()\n",
    "```\n",
    "\n",
    "![A scatter plot that shows data for per capita G D P and life expectancy in four countries, Angola, Ethiopia, Ghana, and South Africa. Per capita G D P is plotted on the horizontal axis and life expectancy is plotted on the vertical axis. The dots are color coded to show which country they represent.](Media/L09-Rosling1.png)\n",
    "\n",
    "This creates an interesting plot that gives some indication of the changes in GDP and life expectancy over time. \n",
    "\n",
    "---\n",
    "\n",
    "### Connecting Points with geom_line() vs. geom_path()\n",
    "\n",
    "But it is difficult to see the evolution of these values over time. You might connect the series of points for each country with a line:\n",
    "\n",
    "\n",
    "```{r}\n",
    "ggplot(gm_AfricaClean, aes(x = gdpPercap, y = lifeExp, color = country)) +\n",
    "geom_point() + geom_line()\n",
    "```\n",
    "\n",
    "![A scatter plot that shows data for per capita G D P and life expectancy in four countries, Angola, Ethiopia, Ghana, and South Africa. Per capita G D P is plotted on the horizontal axis and life expectancy is plotted on the vertical axis. The dots are color coded to show which country they represent. The dots for each country are connected by a line that matches the color of the dots. Each line connects each point with the next point that has the closest horizontal distance, even though the points do not appear in the data frame in that order](Media/L09-Rosling2.png)\n",
    "\n",
    "This does not help you see the evolution of the values over time. Instead, it connects each point with the next point that has the closest horizontal distance, even though the points do not appear in the data frame in that order. You can fix this by using ```geom_path()``` instead of ```geom_line()```; ```geom_path()``` draws lines between adjacent values in the data frame. Making the change like this:\n",
    "\n",
    "```{r}\n",
    "ggplot(gm_AfricaClean, aes(x = gdpPercap, y = lifeExp, color = country)) +\n",
    "geom_point() + geom_path()\n",
    "```\n",
    "\n",
    "And this is the graphic you are provided with:\n",
    "\n",
    "![A scatter plot that shows data for per capita G D P and life expectancy in four countries, Angola, Ethiopia, Ghana, and South Africa. Per capita G D P is plotted on the horizontal axis and life expectancy is plotted on the vertical axis. The dots are color coded to show which country they represent. The dots for each country are connected by a line that matches the color of the dots. The lines are between adjacent values in the data frame.](Media/L09-Rosling3.png)\n",
    "\n",
    "This helps you see that life expectancy has generally increased (except in South Africa); but it is still hard to clearly see the progression of time. \n",
    "\n",
    "---\n",
    "\n",
    "### Making Points Transparent\n",
    "\n",
    "You can convey a sense of the progression of time by changing the transparency of each point, with earlier points being more transparent that later points. The transparency of a point is represented by the ```alpha=``` argument; you can set the ```alpha=``` argument for ```year``` as follows:\n",
    "\n",
    "```{r}\n",
    "ggplot(gm_AfricaClean, aes(x = gdpPercap, y = lifeExp, color = country)) +\n",
    "geom_point(aes(alpha = year)) + geom_path()\n",
    "```\n",
    "\n",
    "![A scatter plot that shows data for per capita G D P and life expectancy in four countries, Angola, Ethiopia, Ghana, and South Africa. Per capita G D P is plotted on the horizontal axis and life expectancy is plotted on the vertical axis. The dots are color coded to show which country they represent. The dots for each country are connected by a line that matches the color of the dots. The lines are between adjacent values in the data frame. For each line, earlier points are more transparent than later points.](Media/L09-Rosling4.png)\n",
    "\n",
    "This seems like you are making progress. It is now easier to see the progression of time, and the legend showing the year related to the transparency of the point provides a solid reference for time. However, it is too hard to see the early points because they are almost completely transparent; also, the points are small enough that it is hard to see the changes in transparency.\n",
    "\n",
    "You can change the range of transparency values used by introducing a ```scale_alpha()``` term in the plot; the arguments to this will be the lowest and highest transparency values to be used. You will use a vector of values from 0.3 to 1.0. You will also make the points larger with the ```size=``` argument:\n",
    "\n",
    "```{r}\n",
    "ggplot(gm_AfricaClean, aes(x = gdpPercap, y = lifeExp, color = country)) +\n",
    "geom_point(aes(alpha = year), size = 3) + geom_path() +\n",
    "scale_alpha(range = c(0.3, 1.0))\n",
    "```\n",
    "\n",
    "This gives the following:\n",
    "\n",
    "![A scatter plot that shows data for per capita G D P and life expectancy in four countries, Angola, Ethiopia, Ghana, and South Africa. Per capita G D P is plotted on the horizontal axis and life expectancy is plotted on the vertical axis. The dots are color coded to show which country they represent. The dots for each country are connected by a line that matches the color of the dots. The lines are between adjacent values in the data frame. For each line, earlier points are more transparent than later points. The size of the dots have been increased for added visibility.](Media/L09-Rosling5.png)\n",
    "\n",
    "---\n",
    "\n",
    "### Clearly Labeling the Graph\n",
    "\n",
    "Finally, make the labels on the horizontal and vertical axes meaningful in regular English: \n",
    "\n",
    "```{r}\n",
    "ggplot(gm_AfricaClean, aes(x = gdpPercap, y = lifeExp, color = country)) +\n",
    "geom_point(aes(alpha = year), size = 3) + geom_path() +\n",
    "scale_alpha(range = c(0.3, 1.0)) +\n",
    "xlab(\"Per capita GDP\") + ylab(\"Life Expectancy\")\n",
    "```\n",
    "\n",
    "You get the following:\n",
    "\n",
    "![A scatter plot that shows data for per capita G D P and life expectancy in four countries, Angola, Ethiopia, Ghana, and South Africa. Per capita G D P is plotted on the horizontal axis and life expectancy is plotted on the vertical axis. The dots are color coded to show which country they represent. The dots for each country are connected by a line that matches the color of the dots. The lines are between adjacent values in the data frame. For each line, earlier points are more transparent than later points. The size of the dots have been increased for added visibility. The labels on the axes have been changed for clarity, with the horizontal axis reading per capita G D P and the vertical axis reading life expectancy.](Media/L09-Rosling6.png)\n",
    "\n",
    "This is now quite an informative plot that shows the different paths that different African countries have taken. In both Ethiopia and Ghana, life expectancy has dramatically increased while GDP has moderately increased over time. In Angola, the increases in life expectancy came to an abrupt halt and GDP decreased dramatically, probably as a consequence of the civil war. In South Africa, the increases in life expectancy have later been reversed because of the HIV/AIDS crisis, and there have been economic impacts as well.\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Interpreting your plot like the above is one of the most important things you can do as a data scientist! Just producing a visual is not enough; you need to point what you want others to notice.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Review\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b1e9a15d-2c94-4572-bde5-3d39e5b519f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a3d8cf1d06345819fe64a715b00b149",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '1. Suppose you wish to compare the three most popul…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L9P3Q1)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "251cc4c7-1f77-487b-b59a-c5c786167566",
   "metadata": {},
   "source": [
    "<p style=\"text-align: center\">\n",
    "  <img src=\"Media/L09-ExerciseBig3.png\" alt=\"Drawing\" style=\"width: 500px;\"/>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b3d4880e-5b44-43b1-932a-60789ee8d148",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "25d7e9d2f10741fdaade1eb66a79076e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '2. Which of the following commands would produce th…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L9P3Q2)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aa07494-8270-45ff-840a-0f400f65507f",
   "metadata": {},
   "source": [
    "<p style=\"text-align: center\">\n",
    "  <img src=\"Media/L09-ExerciseBig3Points.png\" alt=\"Drawing\" style=\"width: 500px;\"/>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5f104f10-257f-40b4-bfa6-7a45ec4a2b08",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "37ac73fd3c4247b99b8c596eb1f68dc6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '3. Which of the following commands would produce th…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L9P3Q3)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1aade2-59af-4cbc-b5ed-da31e39f266d",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 4 - Data Exploration Part III<a class=\"anchor\" id=\"DS102L9_page_4\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c0bdbee2-1a02-484c-867d-cdd237bd4492",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/329835657\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7fb91029f370>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Exploration\n",
    "VimeoVideo('329835657', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eba2ed6e-98dd-43f7-8761-6ecc9a13af08",
   "metadata": {},
   "source": [
    "```c-lms\n",
    "topic: Statistical Summary\n",
    "video-id: Data Exploration Part III\n",
    "video-url-mp4: https://player.vimeo.com/external/329835741.hd.mp4?s=b15fcbe9c4e4cd4eb58a8603f71889e5c69a0b71&profile_id=175\n",
    "video-url-mp4-1080: https://player.vimeo.com/external/329835741.hd.mp4?s=b15fcbe9c4e4cd4eb58a8603f71889e5c69a0b71&profile_id=175\n",
    "video-url-mp4-720: https://player.vimeo.com/external/329835741.hd.mp4?s=b15fcbe9c4e4cd4eb58a8603f71889e5c69a0b71&profile_id=174\n",
    "video-url-mp4-540: https://player.vimeo.com/external/329835741.sd.mp4?s=05c08741b791c0a7fab8c653d9a7bce2118be465&profile_id=165\n",
    "video-url-mp4-360: https://player.vimeo.com/external/329835741.sd.mp4?s=05c08741b791c0a7fab8c653d9a7bce2118be465&profile_id=164\n",
    "```\n",
    "\n",
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO102-L09-pg4tutorial.zip)**.\n",
    "\n",
    "# A Statistical Summary\n",
    "\n",
    "The process of data exploration will often raise as many questions as it answers. In the case of your exploration of Angola, Ghana, Ethiopia, and South Africa, you may become curious how these countries compare to the rest of the countries in Africa. Are they typical? If not, how are they different?\n",
    "\n",
    "You might ask how the life expectancy and per capita GDP in these four countries compare to the median values for all African countries. You can extract and graph the median values.\n",
    "\n",
    "Use the following command to create a tibble that has the medians of life expectancy and per capita GDP for all of the countries in Africa:\n",
    "\n",
    "```{r}\n",
    "gm_medians <- gapminder %>%\n",
    "filter(continent == \"Africa\") %>%\n",
    "group_by(year) %>%\n",
    "summarise(life_med = median(lifeExp), gdp_med = median(gdpPercap))\n",
    "```\n",
    "\n",
    "This command does the following: it starts with the whole gapminder data frame and then uses ```filter()``` to get only the rows for which the continent variable is ```Africa```. It then groups this data by ```year```; the group for a given year is all of the rows that have that year in them. Finally, the ```summarise()``` function computes the median value of ```lifeExp``` for each group and stores this value in the ```life_med``` variable and computes the median value of ```gdpPercap``` for each group and stores this value in the ```gdp_med``` variable.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Note the non-conventional spelling of \"summarise.\" It is not a typo; the function was possibly created by someone in Europe or Australia.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n",
    "The result is the tibble below. Note that each row corresponds to one year and has the median values of life expectancy and GDP over all African countries for that year.\n",
    "\n",
    "```{r}\n",
    "gm_medians\n",
    "```\n",
    "\n",
    "```text\n",
    "# A tibble: 12 x 3\n",
    "year life.med gdp.med\n",
    "\n",
    "<int> <dbl> <dbl>\n",
    "\n",
    "1 1952 38.8330 987.0256\n",
    "\n",
    "2 1957 40.5925 1024.0230\n",
    "\n",
    "3 1962 42.6305 1133.7837\n",
    "\n",
    "4 1967 44.6985 1210.3764\n",
    "\n",
    "5 1972 47.0315 1443.3725\n",
    "\n",
    "6 1977 49.2725 1399.6388\n",
    "\n",
    "7 1982 50.7560 1323.7283\n",
    "\n",
    "8 1987 51.6395 1219.5856\n",
    "\n",
    "9 1992 52.4290 1161.6314\n",
    "\n",
    "10 1997 52.7590 1179.8831\n",
    "\n",
    "11 2002 51.2355 1215.6832\n",
    "\n",
    "12 2007 52.9265 1452.2671\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "Data exploration is one of those often-called-for but infrequently taught skills that you will use as a data scientist. Packages such as ```ggplot2``` and ```dplyr``` are of great assistance in the data exploration endeavor, and you'll be able to find anomalies in the data that you can then explore or explain further. \n",
    "\n",
    "---\n",
    "\n",
    "## Review\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "86f0fa6c-48bf-4a50-8a17-f1b429fd35f0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2474dcee664141998af2e97fcc985c62",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '1. Suppose you wish to create a data set of median …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b40aebc094824816a7e188107fe67510",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '2. True or False?\\n\\n   Functions end in (), argume…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L9P4Q1, L9P4Q2)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ead4ad3c-2fe6-457e-8924-1410e951bebd",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 5 - Key Terms<a class=\"anchor\" id=\"DS102L9_page_5\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9b8cbdc-2b64-4251-9619-950f1761d635",
   "metadata": {},
   "source": [
    "```c-lms\n",
    "topic: Key Terms\n",
    "```\n",
    "\n",
    "# Key R Code \n",
    "\n",
    "Below is a list and short description of the important R code learned in this lesson. Please read through and go back and review any concepts you do not fully understand. Great Work!\n",
    "\n",
    "---\n",
    "\n",
    "# Key R Code\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>color=</td>\n",
    "        <td>As an aesthetic argument in the ggplot() function geom_line(), you can add an additional variable that is plotted by color.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>grid.arrange()</td>\n",
    "        <td>Allows you to combine more than one ggplot() together. </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>ncol=</td>\n",
    "        <td>An argument for grid.arrange() that lets you specify how many columns you should make with your plots.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>size=</td>\n",
    "        <td>An argument to geom_point() that allows you to add an additional variable that is plotted by dot size.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>geom_path()</td>\n",
    "        <td>A function that draws a line between adjacent values in a data frame. </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>alpha=</td>\n",
    "        <td>An argument to the geom_point() aes() that allows you to set an additional variable on the graph by transparency level. </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>scale_alpha()</td>\n",
    "        <td>Allows you to set a range for transparency. </td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "---\n",
    "\n",
    "# Key R Packages\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>gridExtra</td>\n",
    "        <td>Allows you to put multiple graphs in one. </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd9e3b8a-9224-447f-8cf7-5035cae86421",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 6 - Hands On<a class=\"anchor\" id=\"DS102L9_page_6\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "41344b5f-e8cb-455f-93e9-c6e784df9058",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/420538888\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7fb91029ff40>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Exploration\n",
    "VimeoVideo('420538888', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a81e2098-28ef-4ff5-a155-afe824de4323",
   "metadata": {},
   "source": [
    "```c-lms\n",
    "activity-type: project\n",
    "activity-name: Lesson 9 Hands-On\n",
    "points: 12\n",
    "due-at: 81%\n",
    "close-at: end-of-module\n",
    "```\n",
    "\n",
    "For this Hands-On, you will compare five countries of your choice. You can see the countries in the ```gapminder``` data frame with the below command:\n",
    "\n",
    "`levels(gapminder$country)`\n",
    "\n",
    "This Hands-On **will** be graded, so be sure you complete all requirements. Please complete this Hands-On in a R script file, and submit it along with your presentation file below when completed.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Do not submit your project until you have completed all requirements, as you will not be able to resubmit.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "## Requirements\n",
    "\n",
    "Choose five countries from the resulting list after running the above command. Then use them to answer each of these questions:\n",
    "\n",
    "* Which country of the five you chose has the lowest per capita GDP in 1952? In 2007? \n",
    "* Which has the highest per capita GDP in 1952? In 2007? \n",
    "* Create a line plot with year on the horizontal axis and ```lifeExp``` on the vertical axis for the five countries; give each country a different color line. Describe the variations in life expectancy between the countries.\n",
    "* On the entire gapminder data frame, compute the median of lifeExp for each year. For what years is the life expectancy for your five countries above the median life expectancy for the entire gapminder data frame? \n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>You may want to watch <a href=\"https://vimeo.com/420538888\">this recorded live workshop </a> before beginning the hands-on,which goes over a similar example.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>You will need to load the library dplyr, ggplot, and gapminder to complete this hands on!</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n",
    "Create a presentation (MS Power Point or equivalent) to report out on your findings. Be sure to include any R code you used to get your findings and create your graphs. Add any additional insight you believe is warranted.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Be sure to zip and submit all your files when finished!</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>To zip your file on <b>Windows</b>, right click on the file and select \"Send to\", then select \"Compressed (zipped) folder\". For <b>Mac</b> users, right click on the file and select \"Compress\", then select your file from the options.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42cb7d8a-aced-42e5-ac24-32acf8c95473",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 7 - The Terminal<a class=\"anchor\" id=\"DS102L9_page_7\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "341f67bb-47bf-4412-b48c-889093d78167",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/322292263\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7fb920a105e0>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Exploration\n",
    "VimeoVideo('322292263', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf0debaf-5fae-4035-8e7d-ae62e62456cb",
   "metadata": {},
   "source": [
    "# The Terminal\n",
    "\n",
    "Most of the time people will interact with their computers using a graphical user interface (or GUI). This kind of interface is generally a very nice way to use a computer. Learning how to use a mouse or touchscreen is a reasonably low cost to pay to interact with a large number of different applications. It is not a surprise that GUIs are the dominant way people use computers today. However, as great as GUIs are for most use cases, there are several kinds of interactions where they feel somewhat clunky.\n",
    "\n",
    "Consider the following directory (also called a folder):\n",
    "\n",
    "![A computer window showing various files, a folder, the date each was created, and the size of each.](Media/directory1.png)\n",
    "\n",
    "Now imagine you wanted to delete specific files, say all the `.jpg` files. You would probably click on each ```.jpg``` file you see and press delete. Maybe you would sort by file type, and select all the ```.jpg``` files at once and then delete them. Either way, this is generally a more intensive operation than it would be using a `Command Line Interface`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7b3f800-0a14-4d13-8c32-390965c537ca",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 8 - Command Line Interface (CLI) <a class=\"anchor\" id=\"DS102L9_page_8\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "57728ec1-1107-4c80-a925-947a4857a75b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/293882367\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7fb9306e1190>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Command Line Interface\n",
    "VimeoVideo('293882367', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4597e4b4-ce22-448d-843e-eab5ba74a9c4",
   "metadata": {},
   "source": [
    "# Command Line Interface (CLI)\n",
    "\n",
    "Here's what the command line interface looks like:\n",
    "\n",
    "![A bash window on a mac that shows the command line interface. The first line shows the last login attend. The second line shows the user has entered C D forward slash. The third line shows the user has entered L S, which will list files and directories present in the directory in which the user currently is.](Media/cli.png)\n",
    "\n",
    "If a GUI makes it easy to navigate a large number of applications using a small set of inputs (moving the mouse, swiping, clicking, right-clicking, tapping, etc.), the CLI is the opposite. Every program has a different interface when using the CLI, but those interfaces are not limited to a small set of interactions; they can be anything typed on your keyboard.\n",
    "\n",
    "---\n",
    "\n",
    "## Deleting a Windows File\n",
    "\n",
    "With a CLI, **Windows** accomplishes this with the following command:\n",
    "\n",
    "```bash\n",
    "del *.jpg\n",
    "```\n",
    "\n",
    "`del` means to delete all files that match the pattern `*.jpg`.  So, anything that ends with `.jpg`, no matter what comes first, will be deleted.\n",
    "\n",
    "---\n",
    "\n",
    "## Deleting a Mac or Linux File\n",
    "\n",
    "On **MacOS** or **Linux** the command would be:\n",
    "\n",
    "```bash\n",
    "rm *.jpg\n",
    "```\n",
    "\n",
    "`rm` stands for remove.\n",
    "\n",
    "---\n",
    "\n",
    "## Commands\n",
    "\n",
    "Each command on a CLI often has more than one part. The required part is the name of the command to run. There are commands to do a vast number of tasks, from showing you the files in a directory to copying, renaming, moving, deleting, zipping, unzipping, and transferring them. Each of these commands is a tiny program that does a specialized task.\n",
    "\n",
    "![Example of a C L I command and parameter. The command is D E L. The parameter shown is asterisk dot J P G.](Media/cli-command1.png)\n",
    "\n",
    "The parameter(s) are arbitrary and determined by the command. Many commands support a pattern that matches one or more files in a directory. In this case, the parameter specifies a pattern that matches a file that starts with any name but ends in `.jpg`\n",
    "\n",
    "![Visualization of each part of a C L I command with parameters. D E L means delete all files that match the following pattern. Asterisk means starting with any name. Dot J P G means ending in dot J P G.](Media/cli-command2.png)\n",
    "\n",
    "In the example directory above, this would only match one file, `embarrassing_christmas_photo.jpg`. Another way to match this file would be to list the name explicitly.\n",
    "\n",
    "```bash\n",
    "del embarrassing_christmas_photo.jpg\n",
    "```\n",
    "\n",
    "Or, if you wanted to match only the `.jpg` files that are named a particular way you could use this:\n",
    "\n",
    "```bash\n",
    "del embarrassing*.jpg\n",
    "```\n",
    "\n",
    "to match any file that starts with `embarrassing` and ends with `.jpg`. You could even use:\n",
    "\n",
    "```bash\n",
    "del *christmas*.jpg\n",
    "```\n",
    "\n",
    "To delete any file that has `christmas` as part of the name and ends in `.jpg`:\n",
    "\n",
    "---\n",
    "\n",
    "## Review\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ae745d3c-d45c-4ba9-ac67-b7e8da02806e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "079c7df126a145798503b1cf1f2c9b7e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '1. True or False?\\n\\n   The * is a wild card that m…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L9P8Q1)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36943bcd-7e16-4af1-97d3-c09e81ae5e99",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 9 - File Organization and Paths<a class=\"anchor\" id=\"DS102L9_page_9\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8dcbd5f7-e1bb-41ee-8f72-66c4544efd37",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/294067838\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7fb9306e1220>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Command Line Interface\n",
    "VimeoVideo('294067838', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdf2cdde-9447-49b5-8936-17d80f46b45e",
   "metadata": {},
   "source": [
    "# Terminal Differences\n",
    "\n",
    "There are two major families of terminals, the MacOS/Linux versions and the Windows version. Both MacOS and Linux come from an operating system family known as UNIX, while Windows was developed independently. As a result, a large number of commands are the same between MacOS and Linux, while Windows uses different commands. The same structure of having a command followed by its parameters remains the same, but the particulars of various common commands are different.\n",
    "\n",
    "---\n",
    "\n",
    "## File Organization\n",
    "\n",
    "Before you can talk about using the terminal, it is important to understand how files are organized on a computer. You may have some familiarity with this structure if you've used the file browser application on your operating system to copy, move, rename, or delete files.\n",
    "\n",
    "![A finder window in a mac showing how folders and files are organized on a computer.](Media/file-browser.png)\n",
    "\n",
    "The organization of files and directories (aka folders) is a tree-like structure. There is a root starting point that holds some number of directories which, in turn, contain files and other directories.\n",
    "\n",
    "![A tree like graphic organizer with four levels. The top level is the root level. Below the root level is a level for directories. Below a directory are files and subdirectories. Below a subdirectory are files.](Media/directory-tree.png)\n",
    "\n",
    "On Windows, the root is a named \"disk;\" this is what the `C drive` is referencing. The ```C drive``` is the root of a tree of files and folders stored on your hard drive. On Windows, you can have multiple roots that are each identified by a different letter. On MacOS and Linux there is only ever one root which is `/` instead of a letter.\n",
    "\n",
    "---\n",
    "\n",
    "## Paths\n",
    "\n",
    "When describing the location of a file or directory, a *path* is used. This path is the unique set of directories that must be traversed to reach the file each separated by a symbol. So if you had a file on your desktop named `notable_quotations_of_13th_century_poets.txt` and your username is `prosescholar` the full path to the file would be:\n",
    "\n",
    "**Windows:**\n",
    "\n",
    "```bash\n",
    "C:\\Users\\prosescholar\\Desktop\\notable_quotations_of_13th_century_poets.txt\n",
    "```\n",
    "\n",
    "**MacOS / Linux:**\n",
    "\n",
    "```bash\n",
    "/Users/prosescholar/Desktop/notable_quotations_of_13th_century_poets.txt\n",
    "```\n",
    "\n",
    "You may have noticed that the path separator symbol is different between Windows (`\\`) and MacOS/Linux (`/`). This is a remnant of their different heritages and a constant source of confusion when switching between operating systems.\n",
    "\n",
    "Each section of the path describes a directory that is located inside the previous directory. The above example might be visualized as:\n",
    "\n",
    "![A visualization file path. Each section of the path describes a directory that is located inside the previous directory. Root directory to users directory to prose scholar directory to desktop directory to a file named notable quotations of thirteenth century poets dot T X T.](Media/path-to-file.png)\n",
    "\n",
    "The MacOS/Linux forward slash separator is used everywhere except Windows. If you look at a web page URL you may have seen the forward slash (`/`) in use there. Paths are very similar when used with websites so this concept is not just restricted to files on your hard drive.\n",
    "\n",
    "---\n",
    "\n",
    "## Review\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5d72e4fb-5bea-4e6e-80b6-4f168c434d50",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "73790bf51437425b93a3948641d2166a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '1. What is another word for a directory?\\n', 'outpu…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "090ced2134f04218aa2d442dc319481a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '2. True or False?\\n\\n   The path is a specific loca…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L9P9Q1, L9P9Q2)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a0d02d5-04fa-4220-9637-d02c3b32dce5",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 10 - Showing Files and Changing Directories<a class=\"anchor\" id=\"DS102L9_page_10\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ee718d17-4fbc-4765-8505-505571727306",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/294071245\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7fb91029f9d0>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Showing Files and Changing Directories\n",
    "VimeoVideo('294071245', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e7a0a16-0196-41f0-96d3-d9aac5650f67",
   "metadata": {},
   "source": [
    "# Showing Files and Changing Directories\n",
    "\n",
    "---\n",
    "\n",
    "## Opening the CLI\n",
    "\n",
    "Each operating system has a different way of opening its command line interface. On **Windows** this is called the `Command Prompt`, on **MacOS/Linux** it's called the `Terminal`. You will use the more general term `terminal` for the CLI on whatever operating system you are using.\n",
    "\n",
    "**Windows:**\n",
    "\n",
    "1. Click on the Windows Button\n",
    "2. In the search box, type `cmd`\n",
    "3. Press enter\n",
    "\n",
    "**MacOS:**\n",
    "\n",
    "1. Click on the magnifying glass in the upper right corner of the screen\n",
    "2. In the box type `terminal`\n",
    "3. Press enter\n",
    "\n",
    "**Ubuntu Linux:**\n",
    "\n",
    "1. Click on the Ubuntu button\n",
    "2. In the search box, type `terminal`\n",
    "3. Press enter\n",
    "\n",
    "---\n",
    "\n",
    "## Prompt\n",
    "\n",
    "When you open a `terminal` you will see a prompt which is where you can type commands.\n",
    "\n",
    "**Windows Prompt:**\n",
    "\n",
    "```bash\n",
    "C:\\Users\\dkoontz\\Desktop>\n",
    "```\n",
    "\n",
    "![A visualization of a terminal prompt on windows. The drive letter is C colon. The path to the current directory is users backslash D Koontz backslash desktop. The end of prompt is greater than symbol.](Media/windows-prompt.png)\n",
    "\n",
    "**Mac/Linux Prompt:**\n",
    "\n",
    "```bash\n",
    "Work-Laptop:Desktop dkoontz$\n",
    "```\n",
    "\n",
    "![A visualization of a terminal prompt on mac or linux. The name of the computer is work dash laptop colon. The current directory is desktop. The current user is D Koontz. The role is dollar sign.](Media/unix-prompt.png)\n",
    "\n",
    "As you can see, the prompts vary in the information they present. Windows came from a non-networked single-user world, where the MacOS/Linux terminal was borrowed from the world of UNIX which primarily existed as a shared system supporting multiple simultaneous users.  They do this across business or university networks, and with different permission levels between users.\n",
    "\n",
    "One of the essential features of the prompt is to tell you where on your hard drive, the terminal considers the `current directory`. When you use commands such as `del` or `rm` unless you specify a different location, the command will run in the current directory.\n",
    "\n",
    "---\n",
    "\n",
    "## Showing Files\n",
    "\n",
    "When you open your command prompt, it will default to your `home` directory. This is often the directory located at `C:\\Users\\<your user name>` on **Windows** or `/Users/<username>` on **MacOS/Linux**. On MacOS/Linux systems this directory has a special name `~`.\n",
    "\n",
    "To list the files in your home directory, you can use the following command:\n",
    "\n",
    "![The windows command to show files is D I R. The mac O S or linux command to show files is L S.](media/homeDirectory.png)\n",
    "\n",
    "This will show a listing of files, and depending on your operating system, it will display additional information such as size and last modified date. There are a lot of options for most commands such as `dir`/`ls` that will allow you to specify what kind of information you want. A common variant on **Windows** is `dir/w` which will show just the files names, and on **MacOS/Linux** `ls-al` which will show additional information beyond just file names.\n",
    "\n",
    "---\n",
    "\n",
    "## Changing Directories\n",
    "\n",
    "Generally, the files that are of interest are not in your home directory.  They tend to be in sub-directories such as Documents or Pictures. To use `dir`/`ls` in your Documents folder, the `terminal` must first change the current directory. This command is the same on both types of terminals, (`cd`).\n",
    "\n",
    "```bash\n",
    "cd Documents\n",
    "```\n",
    "\n",
    "The `cd` command instructs the terminal to switch the current directory to Documents. If you are in a different directory or do not have a ```Documents``` folder, you will see an error saying: `No such file or directory`.\n",
    "\n",
    "Once you have run the `cd Documents` command, you will see your prompt update to reflect that your current directory has changed to ```Documents```.\n",
    "\n",
    "In the same way that you can traverse into a sub-directory, you can also move up towards the root of the file system. Two special patterns that represent the current directory, and the parent directory.\n",
    "\n",
    "`.` is the current directory.\n",
    "\n",
    "`..` is the parent of the current directory.\n",
    "\n",
    "---\n",
    "\n",
    "## Relative versus Absolute Directories\n",
    "\n",
    "Remember, a command is running in the current directory if you do not specify somewhere else. As it turns out, your `terminal` is inserting a (`.\\`) or (`./`) at the beginning of your commands. This is why the current directory is used for commands such as `dir`/`ls`.\n",
    "\n",
    "`.\\dir` is the same as `dir` and `./ls` is the same as `ls`.\n",
    "\n",
    "So what about (`..`) then? You could use it as the location to change directories to using `cd ..`, which would move the current directory up back to its parent. So if the current directory was `/Users/prosescholar/Documents/awesome_poetry` and you used `cd ..` the resulting current directory would be `/Users/prosescholar/Documents`\n",
    "\n",
    "It is also possible to specify a location when using commands such as `dir`/`ls`. For example, if the current directory is `/Users/prosescholar/Documents` you could use `ls awesome_poetry` to list the files in `/Users/prosescholar/Documents/awesome_poetry`.\n",
    "\n",
    "Now examine how this works.\n",
    "\n",
    "First, the terminal will helpfully insert a `./` in front of the `awesome_poetry` parameter resulting in `ls ./awesome_poetry`. Next, the `./` will be expanded out into what it stands for, the current directory. This results in `ls /Users/prosescholar/Documents/awesome_poetry`\n",
    "\n",
    "A path that starts with either `/` on **MacOS/Linux** or a drive letter such as `C:\\` on **Windows** is known as an `absolute` path. You could run `ls /Users/prosescholar/Documents/awesome_poetry` from any directory and get the same result. The result of `ls awesome_poetry` would vary based on the current directory, in many directories it would fail due to there not being an `awesome_poetry` directory. Paths that are not absolute are called `relative`, as the results of using them depends on what the current directory is.\n",
    "\n",
    "---\n",
    "\n",
    "## Review\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "88f24425-1a71-4fb1-b36e-c4de55eafa90",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2a4db26a3c614feab5e60c499f956516",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '1. Which directory does \\x1b[31;1mdir test\\x1b[0m l…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ee29d92f95574d77b7ad3cc0f352d0b4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '2. A path starting with \\x1b[31;1mC:\\\\\\x1b[0m on Wi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7bb6dc9a19fa46a6acafc03df6b12276",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '3. Which command switches the current directory?\\n'…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L9P10Q1, L9P10Q2, L9P10Q3)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b31f697b-49a5-42d1-8c1e-a060a9c6dedd",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 11 - Learning the CLI<a class=\"anchor\" id=\"DS102L9_page_11\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97534345-1098-4918-a756-201adfbb8a61",
   "metadata": {},
   "source": [
    "# Learning the CLI\n",
    "\n",
    "The topic of using a terminal is broad and will be an ongoing process throughout your career. Most of the commands you will use on an everyday basis are a bit more involved than these few, but they are a great way to get a feel for how the terminal works. It is encouraged for you to experiment with the following commands to get a feel for them.\n",
    "\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Windows Command</th>\n",
    "        <th>MacOS/Linux Command</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>dir</td>\n",
    "        <td>ls</td>\n",
    "        <td>Lists the files in a current directory (by default the current directory)</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>cd</td>\n",
    "        <td>cd</td>\n",
    "        <td>Changes the directory where commands are run</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>move</td>\n",
    "        <td>mv</td>\n",
    "        <td>Moves a file to a new location (possibly with a new name)</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>copy</td>\n",
    "        <td>cp</td>\n",
    "        <td>Copies a file</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>del</td>\n",
    "        <td>rm</td>\n",
    "        <td>Deletes a file</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>..</td>\n",
    "        <td>..</td>\n",
    "        <td>Move current directory back to parent directory</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>md</td>\n",
    "        <td>mkdir</td>\n",
    "        <td>To create a directory</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "---\n",
    "\n",
    "## Review\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "39ad1faa-6ade-46b6-8f94-7e782aaa0d5d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "91424ef2d5b34bc2a08c5db1dda24fa6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(outputs=({'name': 'stdout', 'text': '1. Windows and MacOS have most commands in common\\n…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L9P11Q1)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cdffdcf-0f72-4901-bdb7-dedd4633faec",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 12 - CLI Activity <a class=\"anchor\" id=\"DS102L9_page_12\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31bc2460-99ea-4c1d-91c5-6e50a041fed8",
   "metadata": {},
   "source": [
    "# Activity\n",
    "\n",
    "Follow the instructions below to practice your newly learned skills!\n",
    "\n",
    "1. Create a new directory named `fast`.\n",
    "2. Switch the current directory to `fast`.\n",
    "3. Create a directory named `furious` within the `fast` directory.\n",
    "4. Switch the current directory to `furious`.\n",
    "5. Create a new empty file named `furiosa.txt`.\n",
    "   1. On **OSX/Linux**, you can use the `touch your_file.txt` command to create a new empty file.\n",
    "   2. On **Windows**, you can use the `type nul > your_file.txt` to create a new empty file.\n",
    "6. Switch back to the `fast` directory.\n",
    "\n",
    "When complete, your directory structure should look like:\n",
    "\n",
    "```text\n",
    "fast\n",
    "    furious\n",
    "            furiosa.txt\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a32ba04-e27f-4846-8e94-31d0ba21848f",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 13 - Key Terms <a class=\"anchor\" id=\"DS102L9_page_13\"></a>\n",
    "\n",
    "[Back to Top](#DS102L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ce3fdd9-9093-4f4f-b683-b54cc810eee7",
   "metadata": {},
   "source": [
    "```c-lms\n",
    "topic: Command Line Key Terms\n",
    "```\n",
    "\n",
    "# Key Terms\n",
    "\n",
    "Below, is a list and short description of the central keywords you have learned in this lesson. Please read through and go back and review any concepts you don't understand fully. Great Work!\n",
    "\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>CLI</td>\n",
    "        <td>Command Line Interface, a text based interface for giving instructions to an operating system.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Directory</td>\n",
    "        <td>A location on your hard drive that can contain files and other directories.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Folder</td>\n",
    "        <td>Another name for directory.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Command</td>\n",
    "        <td>An action for the CLI to carry out.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Parameter</td>\n",
    "        <td>Additional information to modify the behavior of a command.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Terminal</td>\n",
    "        <td>Another name for a CLI.</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31effb1a-277d-4690-bfba-bce22481fdb2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}