DS101-Basic-Statistics/DS101-L8-Other-Common-Distributions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "96a37145-48e7-431b-9cf2-ed568460a2b7",
   "metadata": {},
   "source": [
    "# Basic Statistics : Lesson Eight Other Common Distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "11ae5f66-9c86-4ed5-8110-3b68f01a2d71",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/231927586\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x224264e5670>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Science Software\n",
    "VimeoVideo('231927586', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3673fd5-0fef-4458-8d2d-7b0206debc4d",
   "metadata": {},
   "source": [
    "### Table of Contents <a class=\"anchor\" id=\"DS101L8_toc\"></a>\n",
    "* [Table of Contents](#DS101L8_toc)\n",
    "    * [Page 1 - Overview and Objectives](#DS101L8_page_1)\n",
    "    * [Page 2 - Poisson Distribution](#DS101L8_page_2)\n",
    "    * [Page 3 - Exponential Distribution](#DS101L8_page_3)\n",
    "    * [Page 4 - Bathtub Curve](#DS101L8_page_4)\n",
    "    * [Page 5 - Chi-Square Distribution](#DS101L8_page_5)\n",
    "    * [Page 6 - Conducting Chi-Square](#DS101L8_page_6)\n",
    "    * [Page 7 - Chi-Square Activity](#DS101L8_page_7)\n",
    "    * [Page 8 - Chi-Square Activity Solution](#DS101L8_page_8)\n",
    "    * [Page 9 - Key Terms](#DS101L8_page_9)\n",
    "    * [Page 10 - Lesson 8 Exam](#DS101L8_page_10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ed35d21-4dce-4e60-8a46-b0a1a1e5a422",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 1 - Overview and Objectives<a class=\"anchor\" id=\"DS101L8_page_1\"></a>\n",
    "\n",
    "[Back to Top](#DS101L8_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39a65390-0a8f-420d-ab4c-fb91537f6ad5",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "In this lesson, you will learn about additional distributions:\n",
    "\n",
    "* Poisson distribution\n",
    "* Bathtub curve\n",
    "* Exponential distribution\n",
    "* Chi-square distribution\n",
    "\n",
    "These distributions are not used as heavily as those from the last lesson, but still provide a lot of groundwork for many statistical techniques.\n",
    "\n",
    "Concept mastery at the end of this lesson will be assessed by an exam.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "048cf2a5-b4ea-431a-8af0-746d612dfdd7",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 2 - Poisson Distribution<a class=\"anchor\" id=\"DS101L8_page_2\"></a>\n",
    "\n",
    "[Back to Top](#DS101L8_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bc6b8295-6e25-424f-8eb5-b541b8ba4cb1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/318291070\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x224264fae50>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Science Software\n",
    "VimeoVideo('318291070', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b802a686-466b-4389-bff6-b970436c1c3d",
   "metadata": {},
   "source": [
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO101-L08-pg2tutorial.zip)**.\n",
    "\n",
    "# The Poisson Distribution\n",
    "\n",
    "The Poisson distribution is elegant and simple. It is used to model situations in which a given number of events would occur in a fixed interval of time or space. \n",
    "\n",
    "---\n",
    "\n",
    "## Examples Using the Poisson Distribution\n",
    "\n",
    "Examples are things like:\n",
    "\n",
    "* The number of bugs of a particular type in a square foot patch of lawn\n",
    "* The number of pieces of mail received at your home in a typical day\n",
    "* The number of customers arriving between 10:00am and 10:30am to a store\n",
    "* The number of defects found in 30 feet of pipe\n",
    "\n",
    "---\n",
    "\n",
    "## Assumptions for the Poisson Distribution\n",
    "\n",
    "There are some assumptions that must be met in order for the Poisson distribution to be a good model:\n",
    "\n",
    "* The occurrence of an event does not affect the probability of a following event taking place\n",
    "* The number of events can occur in a given interval is countable and a whole number\n",
    "* The long term rate at which events occur is constant\n",
    "* Two events cannot happen in exactly the same instant\n",
    "\n",
    "The Poisson distribution is simple, because there is only one parameter. The mean of the distribution is the same as the variance (which, if you recall, is just the standard deviation squared).\n",
    "\n",
    "The shape of the distribution is dependent on the value of the mean. For the Poisson distribution, the mean is represented by the Greek letter \"lambda,\" and it looks like this:\n",
    "\n",
    "![The Greek letter lambda, which is used to represent the mean for the Poisson distribution.](Media/L07-06.png)\n",
    "\n",
    "It is important to remember that lambda can take on any value, it does not need to be a whole number. However, the number of events in the time interval of interest **does** need to be a whole number (0, 1, 2...).\n",
    "\n",
    "For instance, you may own a fruit stand and you sell an average of 2.7 cantaloupes each hour, but the actual count is never 2.7. Sometimes you sell 4, sometimes 2, sometimes none at all.\n",
    "\n",
    "Take a look at the Poisson distribution for several different values of lambda:\n",
    "\n",
    "![The Poisson distribution for lambda values of one, two, three, four, five, six, seven, eight, and nine.](Media/L07-07.png)\n",
    "\n",
    "Focus on the Poisson distribution for lambda = 2, or the red distribution. Even though the distribution is drawn as a smooth curve, the only values it can actually take on are 0, 1, 2, 3, etc.\n",
    "\n",
    "In order to look at this graph and understand it, do the following: For a Poisson distribution with a lambda of 2, if you want to determine the probability of a 0, you would look vertically right above the 0 and see where it hits the curve.\n",
    "\n",
    "In this case, the curve is directly above the 0 at about 0.14, meaning that the probability of getting a zero when something is distributed as a Poisson with a lambda of 2 is equal to 0.14. How about a 1?\n",
    "\n",
    "By observing the graph directly vertically up from the 1 on the horizontal axis, and then extrapolating that point to the vertical axis, you can see that the probability of a 1 is about 0.26.\n",
    "\n",
    "If you find the probability using the eyeball method and the graph, you can construct the following probabilities:\n",
    "\n",
    "* 0 => p ~ 0.14\n",
    "* 1 => p ~ 0.26\n",
    "* 2 => p ~ 0.28\n",
    "* 3 => p ~ 0.16\n",
    "* 4 => p ~ 0.09\n",
    "* 5 => p ~ 0.04\n",
    "* 6 => p ~ 0.02\n",
    "* 7 or higher => p ~ 0.00\n",
    "\n",
    "If you add all these probabilities up, they should be pretty close to 1, because that forms a complete set of possible outcomes. Based on your eyeball analysis, the total of the probabilities is 0.99, which is not bad for a quick and dirty analysis.\n",
    "\n",
    "Eyeball analysis is fine for approximating, but there are two problems...it might not be as precise as you want it, and if your lambda is something like 2.7, then there is no graph to represent that.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "80c33ccb-f82c-4c42-98cc-e1c70a66ac6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from DS_Students import *\n",
    "    from ipynb.fs.full.DS101Questions import *\n",
    "except:\n",
    "    !pip install DS_Students\n",
    "    from DS_Students import *\n",
    "    from ipynb.fs.full.DS101Questions import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dc305305-4017-49b6-92d3-cde8f155fb25",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b06872eee0214b2188b0bee66510548b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Looking at two related me…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "62b2d26c27fb4b3fba98f0c5002e608d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Number of cattle on 30 ac…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L8P2Q1, L8P2Q2)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f535cf3-1895-42eb-961d-ac08975a1b09",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 3 - Exponential Distribution<a class=\"anchor\" id=\"DS101L8_page_3\"></a>\n",
    "\n",
    "[Back to Top](#DS101L8_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7ab833f7-fcd5-4879-ace2-064bf35064e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/318291050\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x224264e5e50>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Science Software\n",
    "VimeoVideo('318291050', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85667e5c-85c1-4097-9b90-0e3618512af1",
   "metadata": {},
   "source": [
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO101-L08-pg3tutorial.zip)**.\n",
    "\n",
    "# The Exponential Distribution\n",
    "\n",
    "The exponential distribution is a large family of distributions, but you will focus on only a single aspect, and it is somewhat related to the Poisson distribution discussed earlier in this lesson.\n",
    "\n",
    "The Poisson distribution described the number of events within a certain defined interval, whether time or distance. The responses were discrete, meaning that you could have 0 events, 1 event, 2 events, etc.\n",
    "\n",
    "The exponential distribution has a continuous response instead of discrete, because it is used to describe the amount of time between two sequential events. \n",
    "\n",
    "---\n",
    "\n",
    "## An Example\n",
    "\n",
    "A local fast-food chain has a drive thru window that is open 24 hours a day. During the hours of 10:30 AM and 11:00 PM, there is more likely than not a car sitting in the drive thru lane wanting some food. But in the dead of night, there usually aren't that many cars that come by.\n",
    "\n",
    "Suppose you are studying the hours from 2:00 AM to 4:00 AM. Historical data tells you that between those hours, you will typically have 17.8 customers in the drive thru window. With this information, you could create a Poisson distribution with mean (lambda) = 17.8.\n",
    "\n",
    "At this point, it is probably a good idea to keep an eye on units. The mean of 17.8 represents 17.8 cars per 2 hour period, or 17.8 cars per period.\n",
    "\n",
    "---\n",
    "\n",
    "## Shape of the Exponential Distribution\n",
    "\n",
    "The exponential distribution looks like this:\n",
    "\n",
    "---\n",
    "![The exponential distribution with three different colored curves representing three different lambda values, zero point five, one point zero, and one point five.](Media/L07-22.png)\n",
    "\n",
    "---\n",
    "Please note that the larger lambda is, the steeper the curve is on the left side. The three curves shown here are for lambda of 0.5, 1, and 1.5.\n",
    "\n",
    "Imagine what the curve would look like for lambda = 17.8, as in the above scenario. It would be so steep on the left side, that it would look almost vertical, and then approach the horizontal axis very quickly.\n",
    "\n",
    "What you did with the last calculation is similar to what you were doing earlier with the web applets for finding probabilities based on \"area under the curve\" and between various values of *z* or *t*.\n",
    "\n",
    "In this case, you use the above equation to calculate the area under the curve, between the values of *x* = 0 and whatever chosen value for *x* you have.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "85eeea64-3d24-4a18-827e-8be057fa2fde",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "168d451bf942423f932dc3202b728c30",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('The Poisson distribution …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L8P3Q1)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20f65092-e9b2-42c4-a4bf-6cc7ea8a1cbd",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 4 - Bathtub Curve<a class=\"anchor\" id=\"DS101L8_page_4\"></a>\n",
    "\n",
    "[Back to Top](#DS101L8_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fc02b9ff-f997-49dc-a561-15687f2f55c5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/318291016\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x224284d93d0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Science Software\n",
    "VimeoVideo('318291016', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66e37b58-1346-47ff-af1a-17b702d404b7",
   "metadata": {},
   "source": [
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO101-L08-pg4tutorial.zip)**.\n",
    "\n",
    "# The Bathtub Curve\n",
    "\n",
    "There is a probability density curve that is very popular in the electronics industry. It is called the Bathtub Curve, and it typically looks like this:\n",
    "\n",
    "![The bathtub curb placed on a chart. The x axis of the chart represents time. The y axis represents failure rate. The area of the chart has three vertical sections, decreasing failure rate on the left, constant failure rate in the middle, and increasing failure rate on the right. Four lines are plotted, including observed failure rate, the early quote infant mortality quote failure, constant open parentheses random close parentheses failures, and wear out failures.](Media/L07-18.png)\n",
    "\n",
    "The bathtub curve is the blue curve in the picture above. You can see two dashed lines - one in red and one in yellow - that added together make up the bathtub curve.\n",
    "\n",
    "---\n",
    "\n",
    "## \"Infant Mortality\" Failure Rate\n",
    "\n",
    "As noted in the diagram, the red dashed line represents the \"infant mortality\" failure rate of an electronic device. The name is kind of morbid, but the concept is probably intuitive to most readers. When anything is manufactured, there is a certain percentage that will result in a manufacturing fail.\n",
    "\n",
    "If your company manufactures rubber chickens, the fails are usually pretty easy to spot...maybe the eyes are printed in the middle of the back, or there are three legs. All you need is a couple of quality assurance (QA) inspectors to pull out the fails before the rubber chickens go wherever they go for retail.\n",
    "\n",
    "On the other hand, electronics are hard to evaluate by visual means. A manufacturing fail and a manufacturing success can look exactly the same.\n",
    "\n",
    "No manufacturer wants to send a bunch of manufacturing fails to a retailer for sale.  It is in everybody's best interest to separate out the manufacturing fails before sending inventory to the retailer.\n",
    "\n",
    "To prevent this as much as possible, the manufacturer will do what is called \"burn-in\". If you go to any semiconductor manufacturer in the world, it is a good bet they have a test floor as big as several football fields. On that test floor there will often be hundreds of testers, each holding as many as 250,000 die to be burned in.\n",
    "\n",
    "What they are looking for is to use the semiconductors enough in simulated workload, and cull out all of the infant fails. If you look on the red dashed curve, that would be the point at which the \"steepness\" of the failure rate flattens out.\n",
    "\n",
    "---\n",
    "\n",
    "## Wear Out Failures\n",
    "\n",
    "On the other side of the distribution, pretty much everything wears out eventually. The \"wear out failures\" as indicated by the dashed yellow line are relatively flat to begin with. Most stuff doesn't wear out immediately after manufacture.\n",
    "\n",
    "But as time goes on, the wear out failure rate will begin to increase, and the curve gets steeper. Even in industries where manufactured goods are not terribly complex, a good manufacturer wants a pretty good feel for when the wear out curve starts to steepen.\n",
    "\n",
    "They don't want their warranties or guarantees to extend into the steep part of the wear out curve; it costs them too much.\n",
    "\n",
    "As mentioned before, putting both of these curves together creates the bathtub curve. \n",
    "\n",
    "---\n",
    "\n",
    "## The Data Scientist's Role\n",
    "\n",
    "The job of the data scientist in this scenario could be complex modelling with great detail for a highly sophisticated piece of electronic equipment with lots of electronic parts, or it could be a pretty simple \"connect the dots\" type of curve if the complexity of the manufactured goods is low.\n",
    "\n",
    "There is also a concept called \"planned obsolescence.\" This means that a manufacturer doesn't want your product to last forever. They want it to wear out, or become outdated, so that you will have to replace it.\n",
    "\n",
    "Sometimes it is possible for a manufacturer to shorten the width of a bathtub curve by doing things like using plastic parts instead of metal parts at a critical point in a piece of equipment.\n",
    "\n",
    "In summary of the bathtub curve, companies that utilize a data scientist to help them not only define what their bathtub curve looks like, but use it to make good decisions about manufacturing, warranties, and extended warranties are ahead of their competitors.\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1ab37884-9acf-4129-aad8-85ace2a6fc9a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d75f747ba8674c799beecc6986b8f7a4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('It means only parents who…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0aa845ffd9c94ab6a3a4de1e740253eb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('As short as possible', 0)…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L8P4Q1, L8P4Q2)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f1ba1fb-b508-40d2-adae-da84198981c6",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 5 - Chi-Square Distribution<a class=\"anchor\" id=\"DS101L8_page_5\"></a>\n",
    "\n",
    "[Back to Top](#DS101L8_toc)\n",
    "    \n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7aef95b7-fa45-4035-ba5e-2cc044ebd959",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/318291034\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x224284d9160>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Science Software\n",
    "VimeoVideo('318291034', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c79a3b26-7939-45fe-a1f9-8cc63774a196",
   "metadata": {},
   "source": [
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO101-L08-pg5tutorial.zip)**.\n",
    "\n",
    "# Chi-Square Distribution\n",
    "\n",
    "The Chi-Square distribution is not really suitable for modeling. It is mostly used for hypothesis testing.\n",
    "\n",
    "In general terms, the chi-square distribution is a good way of measuring how an assumed distribution matches up with a sampled distribution when you have categorical, not continuous, data. Time to look at an example:\n",
    "\n",
    "> Suppose you are polling for a general election. You have three candidates: Abernathy, Boykins, and Calendario. As you are polling, you ask two questions.  First, who is your preferred candidate, and second, what age group are you in.\n",
    "\n",
    "After you have collected a bunch of data, you construct a table that looks like this:\n",
    "\n",
    "![A table showing the names of three political candidates, the number of people who vote for each and the age range of these votes, the totals by candidate, and the totals by age range.](Media/L07-24.png)\n",
    "\n",
    "If you scrutinize this table, you will realize that Abernathy is much more popular among the younger voters than Boykins, whereas Boykins appeals to the 30 - 49 year old group.\n",
    "\n",
    "However, if the differences were a bit more subtle, how would you decide if there was a candidate preference by voter age?\n",
    "\n",
    "In order to tackle a problem like this, you need to assume that there is **no difference** between age groups regarding candidate preference. With this assumption, you then calculate the \"expected\" counts in each cell.\n",
    "\n",
    "For example, 90 out of 272 voters overall like Abernathy, or about 33%. If each age group liked Abernathy at 33% also, then for the 18 - 29 year olds, you would expect there to be about 0.33 x 84 = 27.8 voters for Abernathy.\n",
    "\n",
    "Here is what the table of \"expected\" values would be:\n",
    "\n",
    "![A table showing the names of three political candidates, the number of people who vote for each and the age range of these votes, the totals by candidate, and the totals by age range. The table shows quote expected quote values.](Media/L07-25.png)\n",
    "\n",
    "If you notice, all of the column totals and row totals are still the same, it is just that the expected counts in each cell have been re-balanced.\n",
    "\n",
    "Without talking about a lot of theory, it turns out that if you do a bunch of math on the \"observed\" cell counts and \"expected\" cell counts in each cell, the total will be distributed as a Chi-Square variable. In fact, it is called the *Chi-Square statistic*.\n",
    "\n",
    "The formula for the calculation will not be presented here. There are tons of web applications that will calculate it for you, so there is no need to learn how to do it by hand.\n",
    "\n",
    "In this case, the Chi-Square statistic value is 44.4. Much like has been done with the *z*-score and the *t*-score, you can use a web based calculator to determine the probability that a Chi-Square statistic would be as large or larger than 44.4, and it turns out the probability is about 0.000001.\n",
    "\n",
    "In other words, it should essentially never happen. In terms of the voting preferences, if you assume there is no difference in preferences among age groups, and then get a probability this small, it forces you to go back and question the assumptions that were originally made. The data have led you to believe that there is in fact a difference in voting preferences among the different age groups.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d34419ae-103c-4d15-bc45-f700b9d4085f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "276e23d3c9ed498a88f3bde0ad860ffe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('When you are comparing tw…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L8P5Q1)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96e85e4c-fc00-47fd-bc9d-903f562e09f1",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 6 - Conducting Chi-Squares<a class=\"anchor\" id=\"DS101L8_page_6\"></a>\n",
    "\n",
    "[Back to Top](#DS101L8_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5b0d3415-c2cb-4b0a-8d35-3c2b7829f21d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/318290985\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x224284ea130>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Science Software\n",
    "VimeoVideo('318290985', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fab66b08-d74b-4a8e-9fbf-74b93f51b176",
   "metadata": {},
   "source": [
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO101-L08-pg6tutorial.zip)**.\n",
    "\n",
    "# The Test for Independence\n",
    "\n",
    "To conduct a test for independence of categorical variables, the `χ2` (Chi-Square) Test of Independence will be used. The fancy 'X' looking character (`χ`) is the Greek letter `Chi`. It is pronounced 'ki' (rhymes with 'tie', sounds like the first part of 'kite').\n",
    "\n",
    "---\n",
    "\n",
    "## What Does Independent Mean? \n",
    "\n",
    "The word \"independent\" means that the two variables are not associated with each other. In the case of a test of independence, it simply means that neither variable depends on the other.\n",
    "\n",
    "Testing for independence is a bit different than all of the other tests you have run so far. Up to this point, there has always been some sort of numerical comparison to be made, whether it is a mean value against a standard, or two means against each other. For a test of independence, this is a bit tougher than the other approaches. What happens here is independence is assumed, and then the number of expected counts in each cell of a table is calculated. Then, the expected counts are subtracted from the observed counts. Finally, those differences are added up, and they become the `χ2` test statistic. The test statistic is then converted into a *p* value.\n",
    "\n",
    "The calculation for the χ2 test statistic is too complex to do by hand, so you will be learning it in MS Excel now, and in other programming languages later.\n",
    "\n",
    "The larger the value of the χ2 test statistic, the smaller the *p* value will be. Once the χ2 test statistic gets large enough for the *p* value to be less than α, you will reject the null hypothesis, just like you did for *t*-tests.\n",
    "\n",
    "---\n",
    "\n",
    "## Contingency Tables and Degrees of Freedom\n",
    "\n",
    "The table below is called a *contingency table*. The contingency table may or may not include a row of totals at the bottom, and a column of totals on the right side.\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Fruit Condition</th>\n",
    "        <th>Bananas</th>\n",
    "        <th>Oranges</th>\n",
    "        <th>Grapes (Bunch)</th>\n",
    "        <th>Strawberries (Quart)</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Bruised</td>\n",
    "        <td>120</td>\n",
    "        <td>21</td>\n",
    "        <td>6</td>\n",
    "        <td>57</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Discolored</td>\n",
    "        <td>38</td>\n",
    "        <td>8</td>\n",
    "        <td>2</td>\n",
    "        <td>3</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Shriveled</td>\n",
    "        <td>0</td>\n",
    "        <td>0</td>\n",
    "        <td>12</td>\n",
    "        <td>10</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Over Ripe</td>\n",
    "        <td>6</td>\n",
    "        <td>15</td>\n",
    "        <td>7</td>\n",
    "        <td>23</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Pristine</td>\n",
    "        <td>354</td>\n",
    "        <td>644</td>\n",
    "        <td>150</td>\n",
    "        <td>228</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "This contingency table has no total row or column. There are 5 rows, and 4 columns. The number of rows and columns are integral to calculating the degrees of freedom, which as you will remember from doing *t*-tests, is important to calculating the statistical significance of the test.\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Even if the contingency table has a 'totals' row or a 'totals' column, those total categories don't get counted as rows or columns.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "To find the degrees of freedom, take the number of rows of data, and subtract 1. Then, take the number of columns of data, and subtract 1. Finally, multiply these two numbers together. For this example, the degrees of freedom are: \n",
    "\n",
    "```text\n",
    "(5-1) x (4-1) = 4 x 3 = 12\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Requirements for a χ2 Test of Independence\n",
    "\n",
    "To conduct a Chi-Square, your data must meet the following requirements: \n",
    "\n",
    "* Each **expected** cell count must be greater than or equal to 5. Please note that it is not dependent on the **actual** cell count.\n",
    "\n",
    "---\n",
    "\n",
    "## Using the Chi-Square Calculator in MS Excel\n",
    "\n",
    "The math for running the hypothesis test is not terribly difficult, but there are many moving parts. Rather than spending a ton of time walking you through creating a calculator, one has been prepared for you in MS Excel.\n",
    "\n",
    "**[MS Excel χ2 calculator](https://repo.exeterlms.com/documents/V2/DataScience/Basic-Statistics/ChiSquareTest.zip)**\n",
    "\n",
    "The χ2 calculator has four parts:\n",
    "\n",
    "* In the upper left, there is a place where the cell counts are entered, and these boxes are blue:\n",
    "\n",
    "    ![The upper left portion of the chi square calculator in Microsoft excel in which cell counts are entered. The portion has six columns and six rows as well as a column for total by row and a row for total by column.](Media/chisquare3.png)\n",
    "\n",
    "* The lower left portion takes the observed counts from each cell, and places them directly above the expected counts for each cell. The expected counts are automatically calculated each time a new cell in the blue area is filled in:\n",
    "\n",
    "    ![A portion of the chi square calculator in Microsoft excel. This portion takes the observed counts from each cell, and places them directly above the expected counts for each cell. The expected counts are automatically calculated each time a new cell in the upper left portion of the calculator is filled in](Media/chisquare2.png)\n",
    "\n",
    "* The lower right portion is where the χ2 statistic is calculated for each cell. The formula is <code>(observed - expected)<sup>2</sup>/expected</code>.\n",
    "\n",
    "    ![The lower right portion of the chi square calculator in Microsoft excel. The heading for this portion reads chi square sums. There are six columns and six rows.](Media/chisquare4.png)\n",
    "\n",
    "* The upper right portion is the results. The cells from the bottom right portion are added together, this is the value of the test statistic. The degrees of freedom are then calculated automatically, and finally the *p* value is calculated based on the test statistic and the degrees of freedom:\n",
    "\n",
    "    ![Chi square test for showing test statistic, degrees of freedom, and p value for Pearson chi square. Test statistic is four point eight seven three. Degree of freedom is one. P value is zero point zero two seven three.](Media/chisquare5.png)\n",
    "\n",
    "To run the test of independence, all that has to be done is the cell counts need to be entered into the blue region at the top left. The size of the table is limited to a 6 x 6 contingency table. This should be sufficient for just about any case, but if a need for a larger table arises, it can easily be expanded.\n",
    "\n",
    "---\n",
    "\n",
    "## Null and Alternative Hypotheses for Chi-Squares\n",
    "\n",
    "The null and alternative hypotheses for any Chi-Square Test of Independence, in general terms, are:\n",
    "\n",
    "* <b>H<sub>0</sub>:</b> The row variable and the column variable are independent. \n",
    "\n",
    "* <b>H<sub>a</sub>:</b> The row variable and the column variable are not independent.\n",
    "\n",
    "---\n",
    "\n",
    "## Case Study: Friday the 13th in Finland \n",
    "\n",
    "Superstitious, anyone? In this example, you will be determining whether there is any relationship between Friday the 13th, gender, and traffic accident deaths in Finland.  The contingency table data is located **[here](https://repo.exeterlms.com/documents/V2/DataScience/Basic-Statistics/friday13.zip)**.\n",
    "\n",
    "---\n",
    "\n",
    "### Case Study Question \n",
    "\n",
    "People often wonder whether two things influence each other. You may wonder whether the gender of the driver and whether the day was Friday the 13th influences the number of deaths related to traffic accidents.  This thought can be expressed as the question:\n",
    "\n",
    "> \"Do traffic accident deaths depend on the gender of the driver or whether the day is Friday the 13th?\"\n",
    "\n",
    "This question has only two possible answers: \n",
    "\n",
    "* **\"Yes\":** You can write the alternative hypothesis by expressing the answer \"yes\" as \"Traffic accident deaths, gender, and Friday the 13th are not independent.\"  This will be the alternative hypothesis.\n",
    "\n",
    "* **\"No\":** The answer \"no\" can be written as Traffic accident deaths, gender, and Friday the 13th are independent.\"  This will be the null hypothesis.\n",
    "\n",
    "---\n",
    "\n",
    "### Case Study Contingency Table\n",
    "\n",
    "The data from the study are summarized in the following contingency table:\n",
    "\n",
    "![A contingency table with data that explores the relationship between Friday the thirteenth, gender, and traffic accidents in Finland. The table shows how many males and how many females were involved in traffic accidents on Fridays that were the thirteenth day of the month and on Fridays that were not the thirteenth day of the month. Eighty two males were in accidents on Friday the thirteenth. Two thousand four hundred twenty three males were in accidents on non thirteenth Fridays. In total, two thousand five hundred and five males were in accidents on these days. Forty one females were in accidents on Friday the thirteenth. Seven hundred eighty nine females were in accidents on non thirteenth Fridays. In total, eight hundred thirty females were in accidents on these days. On Friday the thirteenth, one hundred and twenty three males and females combined were in accidents. On non thirteenth Fridays, three thousand two hundred and twelve  males and females combined were in accidents. The total number of accidents recorded for both males and females on both Friday the thirteenth and non thirteenth Fridays was three thousand three hundred thirty five.](Media/chi-square1.png)\n",
    "\n",
    "---\n",
    "\n",
    "### Case Study Calculations in MS Excel\n",
    "\n",
    "Go ahead and enter the data into the MS Excel spreadsheet results like so:\n",
    "\n",
    "![A portion of the chi square calculator in Microsoft excel in which data is entered. Some data is entered. Column one, row one, eighty two. Column one, row two, forty one. Column two, row one, two thousand for hundred twenty three. Column two, row two, seven hundred eighty nine. Column 1 total, one hundred twenty three. Column two total, three thousand two hundred twelve. Row one total, two thousand five hundred five. Row two total, eight hundred thirty. Total, three thousand three hundred thirty five.](Media/chi-square2.png)\n",
    "\n",
    "There are a couple of important points: \n",
    "\n",
    "* It doesn't really matter if the rows and columns are interchanged. For example, this table is a 2x2 table (2 rows and 2 columns), but if it had been entered with gender on top and Friday along the side, the results would be the same.\n",
    "\n",
    "* The totals at the bottom and on the right are automatically generated when the cells are filled in. Don't include the totals when filling in the blue portion of the table.\n",
    "\n",
    "Now, take a peek at the observed vs. expected counts in the lower left portion of the spreadsheet:\n",
    "\n",
    "![The lower portion of the chi square calculator in Microsoft excel showing observed versus expected counts. Column one, row one, observed eight two, expected ninety two point four. Column one, row two, observed forty one, expected thirty point six. Total column one, observed one hundred twenty three, expected one hundred twenty three point zero. Column two, row one, observed two thousand four hundred twenty three, expected two thousand four hundred twelve point six. Column two, row two, observed seven hundred eight nine, expected seven hundred ninety nine point four. Total column two, observed three thousand two hundred twelve, expected three thousand two hundred twelve point zero. Row one total, observed two thousand five hundred five, expected two thousand five hundred five point zero. Row two total, observed eight hundred thirty, expected eight hundred thirty point zero. Total, observed three thousand three hundred thirty five, expected three thousand three hundred thirty five point zero. ](Media/chi-square3.png)\n",
    "\n",
    "As you can see, each row now has two entries. The entry on top is the observed counts for that cell, and the entry on the bottom is the expected counts for that cell. For example, the entries on row 1, column 1 are 82 and 92.4. This entry corresponds to the counts in the cell where it was Friday the 13th and a male was driving. \n",
    "\n",
    "The expected counts are based on what would happen if there was no difference in gender.\n",
    "\n",
    "Since there are no labels for the rows or columns, referring to the original raw data is helpful to remember what is going on. Anytime the expected counts and observed counts differ greatly, there is reason to believe that the columns variable depends on the rows.\n",
    "\n",
    "Next, look at the portion of the spreadsheet where the `χ2` sums are calculated:\n",
    "\n",
    "![Portion of spreadsheet showing chi square sums. Column one, row one, one point one six eight. Column one, row two, three point five two five. Column two, row one, zero point zero four five. Column two, row two, zero point one three five.](Media/chi-square4.png)\n",
    "\n",
    "Each of these cells shows the contribution to the overall test statistic. The cells correspond to the original data, and give you an idea of where the biggest relative differences between the observed cell counts and the expected cell counts lie. Summing all of these values will give you a total value which is the χ2 test statistic, found in the upper right hand portion of the spreadsheet:\n",
    "\n",
    "![Chi square test for showing test statistic, degrees of freedom, and p value for Pearson chi square. Test statistic is four point eight seven three. Degree of freedom is one. P value is zero point zero two seven three.](Media/chi-square5.png)\n",
    "\n",
    "This table portion is the result of the analysis. The χ2 total is labeled as the \"Pearson Chi-Square\" value. The degrees of freedom are calculated, and the corresponding *p* value is given.\n",
    "\n",
    "Now you've completed your first Chi-Square analysis! \n",
    "\n",
    "---\n",
    "\n",
    "### Null and Alternative Hypotheses for the Case Study\n",
    "\n",
    "Once you apply the Chi-Square to a particular scenario, however, you can then get more specific in how you word the hypotheses:\n",
    "\n",
    "* <b>H<sub>0</sub>:</b> Gender and Friday the 13th are independent.\n",
    "\n",
    "* <b>H<sub>a</sub>:</b> Gender and Friday the 13th are not independent.\n",
    "\n",
    "If the row and column variable are independent, then no matter which row you consider, the proportion of observations in each column should be roughly the same. For example, if gender and Friday the 13th are independent, then the proportion of people who die in a traffic accident should be approximately the same in each row. That is, it will be approximately the same for Men and Women, and for Friday the 13th versus other days. \n",
    "\n",
    "For this analysis, let `α=0.05`.  The test statistic is: `χ2 = 4.87`, with <code><em>df</em> = 8</code>. And the *p* value is rounded to 0.03 in the output.\n",
    "\n",
    "Therefore, you should reject the null hypothesis, and conclude that there is sufficient evidence to suggest that gender is not independent of Friday the 13th.  This means that the number of traffic accident deaths in Finland is influenced by gender and/or Friday the 13th.  Taking a look at the difference between the actual cells and the expected cells, you can see that there are far more traffic accident deaths NOT on Friday the 13th then on Friday the 13th.  This should make sense to you - there are many more days that are Fridays that are NOT the 13th. This is a good indication that the data reporting is flawed; counts will be biased, because of the number of days.  A better way for someone to have reported this data was the percentage of total accidents per day that resulted in death.  \n",
    "\n",
    "You will also note that women seem to have fewer accidents than men.  This seems like a much stronger takeaway, so your main conclusion from this independent chi-square test should be:\n",
    "\n",
    ">Female drivers in Finland are significantly less likely to get into a fatal traffic accident, regardless of the day.\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Fun Fact!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>You won't calculate confidence intervals when working with contingency tables, because confidence intervals are centered around a mean.  Without a mean, it does not make sense to calculate a confidence interval.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "The distributions covered in this lesson are a teaser for what's to come - you will see them again accompanied with their test statistics in later modules! However, you were able to compute an independent chi-square, used for testing the independence of two categorical variables.\\\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Want a little more information on Chi-Squares in MS Excel? Then check out <a href=\"https://vimeo.com/405273502\">this recorded live workshop. </a></p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3502214a-5067-4df0-80a2-6f33191b3ca1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b669d3e0af2b49f2a2d8b3a961253ee7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('The means are different',…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c45790a1e17148cdae2c5ee61dcabdf5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('(number of rows - 1) * (n…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "519c84aa660a4413a71c21adef3d8c23",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Use of simple random samp…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff5d2e32a1804c55ac004e91d86b2df3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Rows and columns are depe…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L8P6Q1, L8P6Q2, L8P6Q3, L8P6Q4)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f26d624e-ea0e-49f1-918b-5b9c4e3389aa",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 7 - Chi-Square Activity<a class=\"anchor\" id=\"DS101L8_page_7\"></a>\n",
    "\n",
    "[Back to Top](#DS101L8_toc)\n",
    "    \n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "### Directions\n",
    "# Independent Chi-Square Activity\n",
    "Given the below contingency table on alien sightings, please use the independent chi-square to determine whether the number of extra-terrestrial contacts is independent of the continent.\n",
    "\n",
    "![.](Media/chisquare12.png)\n",
    "\n",
    "A contingency table with data on alien sightings in different continents.\n",
    "\n",
    "Please follow the steps of hypothesis testing below, using an α-level cut-off of .05 to indicate significance.\n",
    "\n",
    "Steps of Hypothesis Testing:\n",
    "\n",
    " * State the null and alternative hypothesis.\n",
    "\n",
    " * Check the requirements for the hypothesis test\n",
    "\n",
    " * Calculate the test statistic, degrees of freedom, and p-value of the hypothesis test.\n",
    "\n",
    " * Assess the statistical significance by comparing the p-value to the α-level.\n",
    "\n",
    " * Draw a correct conclusion for the hypothesis test.\n",
    "\n",
    "When the assignment is complete, please attach the spreadsheet with your answers and a document that discusses your findings.\n",
    "\n",
    "The example Chi-Square Test template provided **[here.](https://repo.exeterlms.com/documents/V2/DataScience/Basic-Statistics/ChiSquareTest.zip)**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b6974d7-b1ef-4d4b-bfac-47f019f82d06",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 8 - Chi-Square Activity Solution<a class=\"anchor\" id=\"DS101L8_page_8\"></a>\n",
    "\n",
    "[Back to Top](#DS101L8_toc)\n",
    "    \n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "    \n",
    "## Solution\n",
    "The MS Excel workbook to determine whether extra-terrestrial contacts is independent of the continent upon which they are seen is located **[here.](https://repo.exeterlms.com/documents/V2/DataScience/Basic-Statistics/AliensChi-squareSolution.zip)**\n",
    "\n",
    "## Findings\n",
    "Below you will find the answers to the Steps of Hypothesis Testing.\n",
    "\n",
    "### State the Hypotheses\n",
    "The null hypothesis is that extra-terrestrial contact is independent of continent.\n",
    "\n",
    "The alternative hypothesis is that extra-terrestrial contact is not independent of continent.\n",
    "\n",
    "## Calculate the Test Statistic, Degrees of Freedom, and p Value\n",
    "The Chi-Square value is 2.734.\n",
    "\n",
    "The degrees of freedom (df) is 5.\n",
    "\n",
    "The p value is .70.\n",
    "\n",
    "### Assess the Statistical Significance\n",
    "The alpha level is .05. Since the p value is > .05, this means that this test is not significant.\n",
    "\n",
    "### Draw Conclusions\n",
    "The null hypothesis should be accepted, and the alternative hypothesis rejected, because the p value was greater than .05. This means that there is no relationship between the number of extra-terrestrial contacts and the continent upon which those sightings took place."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bf9db9e-a1b5-43d0-876d-ca91cb5c2efc",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 9 - Key Terms<a class=\"anchor\" id=\"DS101L8_page_9\"></a>\n",
    "\n",
    "[Back to Top](#DS101L8_toc)\n",
    "    \n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "## Key Terms\n",
    "    \n",
    "Below is a list and short description of the important keywords learned in this lesson. Please read through and go back and review any concepts you do not fully understand. Great Work!\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Poisson Distribution</td>\n",
    "        <td>For situations where a limited number of events occur in a fixed interval of time or space (for predicting discrete outcomes).</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Lambda</td>\n",
    "        <td>The mean of the poisson distribution.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Bathtub Curve</td>\n",
    "        <td>A graph of failure rates over time for the manufacturing industry.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Infant Mortality Failure Rate</td>\n",
    "        <td>A manufacturing fail that happens early on.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Wear Out Failure Rate</td>\n",
    "      <td>A manufacturing fail that takes place over time as the product ages.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Exponential Distribution</td>\n",
    "      <td>Used to model the amount of time that takes place between events (for predicting continuous outcomes).</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Chi-Square Distribution</td>\n",
    "      <td>Used to test categorical outcomes, either against a population or against a second categorical variable.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Independent</td>\n",
    "      <td>Variables are not associated with each other.  One variable does not <em>depend</em> on the other in any way.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Contingency Table</td>\n",
    "      <td>A chart of frequencies based two different categorical variables.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Independent Chi-Square</td>\n",
    "      <td>Tests for the independence of two categorical variables.</td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "984fa016-5e95-4ec8-8159-0295516b95dc",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 10 - Lesson Eight Exam<a class=\"anchor\" id=\"DS101L8_page_10\"></a>\n",
    "\n",
    "[Back to Top](#DS101L8_toc)\n",
    "    \n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "    \n",
    "Lesson 8 Exam on the LMS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b7957fcc-171e-4e98-bfe8-ff9225e33fee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2096c07682cd4876b644d40bf2cbd5c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Poisson Distribution', 0)…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9f82314e2a7c48f89e9eebdbfd6b48cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Poisson Distribution', 0)…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "17fd591aeb844be7816d0fc8dba9831a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Poisson Distribution', 0)…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "284b4c58f07f4c54af45d0c491e1b9e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Poisson Distribution', 0)…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1dbe4867de8e47eabf3f7206e82ecbd5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Expected cell count less …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cc90188d95c843f0af5d14d4adc3b45f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('There is no difference be…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cb92e27ae6474b6bb751141a99b5e5b3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Accept the null hypothesi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a8498872cef94a038efb5654c596fa30",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Total sample size', 0), (…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L8P10Q1, L8P10Q2, L8P10Q3, L8P10Q4, L8P10Q5, L8P10Q6, L8P10Q7, L8P10Q8)\n",
    "except:\n",
    "    pass"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}