Analyze_ab_test_results_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze A/B Test Results\n",
    "\n",
    "You may either submit your notebook through the workspace here, or you may work from your local machine and submit through the next page.  Either way assure that your code passes the project [RUBRIC](https://review.udacity.com/#!/projects/37e27304-ad47-4eb0-a1ab-8c12f60e43d0/rubric).  **Please save regularly.**\n",
    "\n",
    "This project will assure you have mastered the subjects covered in the statistics lessons.  The hope is to have this project be as comprehensive of these topics as possible.  Good luck!\n",
    "\n",
    "## Table of Contents\n",
    "- [Introduction](#intro)\n",
    "- [Part I - Probability](#probability)\n",
    "- [Part II - A/B Test](#ab_test)\n",
    "- [Part III - Regression](#regression)\n",
    "\n",
    "\n",
    "<a id='intro'></a>\n",
    "### Introduction\n",
    "\n",
    "A/B tests are very commonly performed by data analysts and data scientists.  It is important that you get some practice working with the difficulties of these \n",
    "\n",
    "For this project, you will be working to understand the results of an A/B test run by an e-commerce website.  Your goal is to work through this notebook to help the company understand if they should implement the new page, keep the old page, or perhaps run the experiment longer to make their decision.\n",
    "\n",
    "**As you work through this notebook, follow along in the classroom and answer the corresponding quiz questions associated with each question.** The labels for each classroom concept are provided for each question.  This will assure you are on the right track as you work through the project, and you can feel more confident in your final submission meeting the criteria.  As a final check, assure you meet all the criteria on the [RUBRIC](https://review.udacity.com/#!/projects/37e27304-ad47-4eb0-a1ab-8c12f60e43d0/rubric).\n",
    "\n",
    "<a id='probability'></a>\n",
    "#### Part I - Probability\n",
    "\n",
    "To get started, let's import our libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "#We are setting the seed to assure you get the same answers on quizzes as we set up\n",
    "random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`1.` Now, read in the `ab_data.csv` data. Store it in `df`.  **Use your dataframe to answer the questions in Quiz 1 of the classroom.**\n",
    "\n",
    "a. Read in the dataset and take a look at the top few rows here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"ab_data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Use the cell below to find the number of rows in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(294478, 5)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. The number of unique users in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id         290584\n",
       "timestamp       294478\n",
       "group                2\n",
       "landing_page         2\n",
       "converted            2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "d. The proportion of users converted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11.966%\n"
     ]
    }
   ],
   "source": [
    "print(\"{0:.3%}\".format(df[\"converted\"].mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "e. The number of times the `new_page` and `treatment` don't match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['user_id', 'timestamp', 'group', 'landing_page', 'converted'], dtype='object')"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3893"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diff = df.query('(group == \"treatment\" and landing_page !=\"new_page\" ) or (group != \"treatment\" and landing_page ==\"new_page\")' )\n",
    "diff['user_id'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. Do any of the rows have missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().values.any() # There is no missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` For the rows where **treatment** does not match with **new_page** or **control** does not match with **old_page**, we cannot be sure if this row truly received the new or old page.  Use **Quiz 2** in the classroom to figure out how we should handle these rows.  \n",
    "\n",
    "a. Now use the answer to the quiz to create a new dataset that meets the specifications from the quiz.  Store your new dataframe in **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>936923</td>\n",
       "      <td>2017-01-10 15:20:49.083499</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>679687</td>\n",
       "      <td>2017-01-19 03:26:46.940749</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>719014</td>\n",
       "      <td>2017-01-17 01:48:29.539573</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>817355</td>\n",
       "      <td>2017-01-04 17:58:08.979471</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>839785</td>\n",
       "      <td>2017-01-15 18:11:06.610965</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>929503</td>\n",
       "      <td>2017-01-18 05:37:11.527370</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>834487</td>\n",
       "      <td>2017-01-21 22:37:47.774891</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>803683</td>\n",
       "      <td>2017-01-09 06:05:16.222706</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>944475</td>\n",
       "      <td>2017-01-22 01:31:09.573836</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>718956</td>\n",
       "      <td>2017-01-22 11:45:11.327945</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>644214</td>\n",
       "      <td>2017-01-22 02:05:21.719434</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>847721</td>\n",
       "      <td>2017-01-17 14:01:00.090575</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>888545</td>\n",
       "      <td>2017-01-08 06:37:26.332945</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>650559</td>\n",
       "      <td>2017-01-24 11:55:51.084801</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>935734</td>\n",
       "      <td>2017-01-17 20:33:37.428378</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>740805</td>\n",
       "      <td>2017-01-12 18:59:45.453277</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>759875</td>\n",
       "      <td>2017-01-09 16:11:58.806110</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>793849</td>\n",
       "      <td>2017-01-23 22:36:10.742811</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>905617</td>\n",
       "      <td>2017-01-20 14:12:19.345499</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>746742</td>\n",
       "      <td>2017-01-23 11:38:29.592148</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>892356</td>\n",
       "      <td>2017-01-05 09:35:14.904865</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>773302</td>\n",
       "      <td>2017-01-12 08:29:49.810594</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>913579</td>\n",
       "      <td>2017-01-24 09:11:39.164256</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>736159</td>\n",
       "      <td>2017-01-06 01:50:21.318242</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>690284</td>\n",
       "      <td>2017-01-13 17:22:57.182769</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294448</th>\n",
       "      <td>776137</td>\n",
       "      <td>2017-01-12 05:53:12.386730</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294449</th>\n",
       "      <td>883344</td>\n",
       "      <td>2017-01-22 23:15:58.645325</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294450</th>\n",
       "      <td>825594</td>\n",
       "      <td>2017-01-06 12:37:08.897784</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294451</th>\n",
       "      <td>875688</td>\n",
       "      <td>2017-01-14 07:19:49.042869</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294452</th>\n",
       "      <td>927527</td>\n",
       "      <td>2017-01-12 10:52:11.084740</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294453</th>\n",
       "      <td>789177</td>\n",
       "      <td>2017-01-17 18:17:56.215378</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294454</th>\n",
       "      <td>937338</td>\n",
       "      <td>2017-01-19 03:23:22.236666</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294455</th>\n",
       "      <td>733101</td>\n",
       "      <td>2017-01-23 12:52:58.711914</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294456</th>\n",
       "      <td>679096</td>\n",
       "      <td>2017-01-02 16:43:49.237940</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294457</th>\n",
       "      <td>691699</td>\n",
       "      <td>2017-01-09 23:42:35.963486</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294458</th>\n",
       "      <td>807595</td>\n",
       "      <td>2017-01-22 10:43:09.285426</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294459</th>\n",
       "      <td>924816</td>\n",
       "      <td>2017-01-20 10:59:03.481635</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294460</th>\n",
       "      <td>846225</td>\n",
       "      <td>2017-01-16 15:24:46.705903</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294461</th>\n",
       "      <td>740310</td>\n",
       "      <td>2017-01-10 17:22:19.762612</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294462</th>\n",
       "      <td>677163</td>\n",
       "      <td>2017-01-03 19:41:51.902148</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294463</th>\n",
       "      <td>832080</td>\n",
       "      <td>2017-01-19 13:18:27.352570</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294464</th>\n",
       "      <td>834362</td>\n",
       "      <td>2017-01-17 01:51:56.106436</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294465</th>\n",
       "      <td>925675</td>\n",
       "      <td>2017-01-07 20:38:26.346410</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294466</th>\n",
       "      <td>923948</td>\n",
       "      <td>2017-01-09 16:33:41.104573</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294467</th>\n",
       "      <td>857744</td>\n",
       "      <td>2017-01-05 08:00:56.024226</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294468</th>\n",
       "      <td>643562</td>\n",
       "      <td>2017-01-02 19:20:05.460595</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294469</th>\n",
       "      <td>755438</td>\n",
       "      <td>2017-01-18 17:35:06.149568</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294470</th>\n",
       "      <td>908354</td>\n",
       "      <td>2017-01-11 02:42:21.195145</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294471</th>\n",
       "      <td>718310</td>\n",
       "      <td>2017-01-21 22:44:20.378320</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294472</th>\n",
       "      <td>822004</td>\n",
       "      <td>2017-01-04 03:36:46.071379</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294473</th>\n",
       "      <td>751197</td>\n",
       "      <td>2017-01-03 22:28:38.630509</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294474</th>\n",
       "      <td>945152</td>\n",
       "      <td>2017-01-12 00:51:57.078372</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294475</th>\n",
       "      <td>734608</td>\n",
       "      <td>2017-01-22 11:45:03.439544</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294476</th>\n",
       "      <td>697314</td>\n",
       "      <td>2017-01-15 01:20:28.957438</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294477</th>\n",
       "      <td>715931</td>\n",
       "      <td>2017-01-16 12:40:24.467417</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>290585 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id                   timestamp      group landing_page  converted\n",
       "0        851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1        804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2        661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3        853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4        864975  2017-01-21 01:52:26.210827    control     old_page          1\n",
       "5        936923  2017-01-10 15:20:49.083499    control     old_page          0\n",
       "6        679687  2017-01-19 03:26:46.940749  treatment     new_page          1\n",
       "7        719014  2017-01-17 01:48:29.539573    control     old_page          0\n",
       "8        817355  2017-01-04 17:58:08.979471  treatment     new_page          1\n",
       "9        839785  2017-01-15 18:11:06.610965  treatment     new_page          1\n",
       "10       929503  2017-01-18 05:37:11.527370  treatment     new_page          0\n",
       "11       834487  2017-01-21 22:37:47.774891  treatment     new_page          0\n",
       "12       803683  2017-01-09 06:05:16.222706  treatment     new_page          0\n",
       "13       944475  2017-01-22 01:31:09.573836  treatment     new_page          0\n",
       "14       718956  2017-01-22 11:45:11.327945  treatment     new_page          0\n",
       "15       644214  2017-01-22 02:05:21.719434    control     old_page          1\n",
       "16       847721  2017-01-17 14:01:00.090575    control     old_page          0\n",
       "17       888545  2017-01-08 06:37:26.332945  treatment     new_page          1\n",
       "18       650559  2017-01-24 11:55:51.084801    control     old_page          0\n",
       "19       935734  2017-01-17 20:33:37.428378    control     old_page          0\n",
       "20       740805  2017-01-12 18:59:45.453277  treatment     new_page          0\n",
       "21       759875  2017-01-09 16:11:58.806110  treatment     new_page          0\n",
       "23       793849  2017-01-23 22:36:10.742811  treatment     new_page          0\n",
       "24       905617  2017-01-20 14:12:19.345499  treatment     new_page          0\n",
       "25       746742  2017-01-23 11:38:29.592148    control     old_page          0\n",
       "26       892356  2017-01-05 09:35:14.904865  treatment     new_page          1\n",
       "27       773302  2017-01-12 08:29:49.810594  treatment     new_page          0\n",
       "28       913579  2017-01-24 09:11:39.164256    control     old_page          1\n",
       "29       736159  2017-01-06 01:50:21.318242  treatment     new_page          0\n",
       "30       690284  2017-01-13 17:22:57.182769    control     old_page          0\n",
       "...         ...                         ...        ...          ...        ...\n",
       "294448   776137  2017-01-12 05:53:12.386730  treatment     new_page          0\n",
       "294449   883344  2017-01-22 23:15:58.645325  treatment     new_page          0\n",
       "294450   825594  2017-01-06 12:37:08.897784  treatment     new_page          0\n",
       "294451   875688  2017-01-14 07:19:49.042869    control     old_page          0\n",
       "294452   927527  2017-01-12 10:52:11.084740    control     old_page          0\n",
       "294453   789177  2017-01-17 18:17:56.215378    control     old_page          0\n",
       "294454   937338  2017-01-19 03:23:22.236666  treatment     new_page          0\n",
       "294455   733101  2017-01-23 12:52:58.711914  treatment     new_page          0\n",
       "294456   679096  2017-01-02 16:43:49.237940  treatment     new_page          0\n",
       "294457   691699  2017-01-09 23:42:35.963486  treatment     new_page          0\n",
       "294458   807595  2017-01-22 10:43:09.285426  treatment     new_page          0\n",
       "294459   924816  2017-01-20 10:59:03.481635    control     old_page          0\n",
       "294460   846225  2017-01-16 15:24:46.705903  treatment     new_page          0\n",
       "294461   740310  2017-01-10 17:22:19.762612    control     old_page          0\n",
       "294462   677163  2017-01-03 19:41:51.902148  treatment     new_page          0\n",
       "294463   832080  2017-01-19 13:18:27.352570    control     old_page          0\n",
       "294464   834362  2017-01-17 01:51:56.106436    control     old_page          0\n",
       "294465   925675  2017-01-07 20:38:26.346410  treatment     new_page          0\n",
       "294466   923948  2017-01-09 16:33:41.104573    control     old_page          0\n",
       "294467   857744  2017-01-05 08:00:56.024226    control     old_page          0\n",
       "294468   643562  2017-01-02 19:20:05.460595  treatment     new_page          0\n",
       "294469   755438  2017-01-18 17:35:06.149568    control     old_page          0\n",
       "294470   908354  2017-01-11 02:42:21.195145    control     old_page          0\n",
       "294471   718310  2017-01-21 22:44:20.378320    control     old_page          0\n",
       "294472   822004  2017-01-04 03:36:46.071379  treatment     new_page          0\n",
       "294473   751197  2017-01-03 22:28:38.630509    control     old_page          0\n",
       "294474   945152  2017-01-12 00:51:57.078372    control     old_page          0\n",
       "294475   734608  2017-01-22 11:45:03.439544    control     old_page          0\n",
       "294476   697314  2017-01-15 01:20:28.957438    control     old_page          0\n",
       "294477   715931  2017-01-16 12:40:24.467417  treatment     new_page          0\n",
       "\n",
       "[290585 rows x 5 columns]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = df.drop(df.query('(group == \"treatment\" and landing_page !=\"new_page\" ) or (group != \"treatment\" and landing_page ==\"new_page\")').index)\n",
    "df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Double Check all of the correct rows were removed - this should be 0\n",
    "df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`3.` Use **df2** and the cells below to answer questions for **Quiz3** in the classroom."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. How many unique **user_id**s are in **df2**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[\"user_id\"].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "b. There is one **user_id** repeated in **df2**.  What is it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(df2['user_id'].duplicated())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2.duplicated(\"user_id\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. What is the row information for the repeat **user_id**? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1899</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-09 05:37:58.781806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "1899   773192  2017-01-09 05:37:58.781806  treatment     new_page          0\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2.duplicated(['user_id'], keep=False)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Remove **one** of the rows with a duplicate **user_id**, but keep your dataframe as **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [user_id, timestamp, group, landing_page, converted]\n",
       "Index: []"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.drop(1899, inplace = True)\n",
    "df2[df2.duplicated(['user_id'], keep=False)]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [user_id, timestamp, group, landing_page, converted]\n",
       "Index: []"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2.duplicated(\"user_id\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(df2['user_id'].duplicated()) # there is no dublicated\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`4.` Use **df2** in the cells below to answer the quiz questions related to **Quiz 4** in the classroom.\n",
    "\n",
    "a. What is the probability of an individual converting regardless of the page they receive?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11.96%\n"
     ]
    }
   ],
   "source": [
    "converted_users = df2[\"converted\"].mean()\n",
    "print(\"{0:0.2%}\".format(converted_users))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Given that an individual was in the `control` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12.04%\n"
     ]
    }
   ],
   "source": [
    "individual_1 = df2.query('group == \"control\" and converted == 1')[\"user_id\"].nunique()\n",
    "\n",
    "# Get dataframe with all records from control group \n",
    "control_1 = df2.query('group == \"control\"')['user_id'].nunique()\n",
    "\n",
    "c1 = individual_1 / control_1\n",
    "print(\"{0:.2%}\".format(c1))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12.04%\n"
     ]
    }
   ],
   "source": [
    "# another check to ensure accuracy using mean\n",
    "print(\"{0:.2%}\".format(df2.query('group == \"control\"')[\"converted\"].mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Given that an individual was in the `treatment` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11.88%\n"
     ]
    }
   ],
   "source": [
    "individual_2 = df2.query('group == \"treatment\" and converted == 1')[\"user_id\"].nunique()\n",
    "\n",
    "# Get dataframe with all records from treatment group \n",
    "control_2 = df2.query('group == \"treatment\"')['user_id'].nunique()\n",
    "\n",
    "c2 = individual_2/control_2\n",
    "\n",
    "print(\"{0:.2%}\".format(c2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. What is the probability that an individual received the new page?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50.01%\n"
     ]
    }
   ],
   "source": [
    "individual_3 = df2.query('landing_page == \"new_page\"')[\"user_id\"].nunique()\n",
    "\n",
    "c3 = individual_3/df2.shape[0]\n",
    "print(\"{0:.2%}\".format(c3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. Consider your results from parts (a) through (d) above, and explain below whether you think there is sufficient evidence to conclude that the new treatment page leads to more conversions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. OverAll conversions: 11.96%\n",
    "2. Control Conversions: 12.04%\n",
    "3. Treatment Conversions: 11.88%\n",
    "The Treatment shows a decrease in conversions by 0.08%, while the control increases by 0.08%, However, this difference is so small, which cannot tell us which is the best, the old or the new page, may need more test to get more  evidence.\n",
    "Consider this test, old page is more effective than the new page, where both group recvieved the pages almost equally (50.01%).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ab_test'></a>\n",
    "### Part II - A/B Test\n",
    "\n",
    "Notice that because of the time stamp associated with each event, you could technically run a hypothesis test continuously as each observation was observed.  \n",
    "\n",
    "However, then the hard question is do you stop as soon as one page is considered significantly better than another or does it need to happen consistently for a certain amount of time?  How long do you run to render a decision that neither page is better than another?  \n",
    "\n",
    "These questions are the difficult parts associated with A/B tests in general.  \n",
    "\n",
    "\n",
    "`1.` For now, consider you need to make the decision just based on all the data provided.  If you want to assume that the old page is better unless the new page proves to be definitely better at a Type I error rate of 5%, what should your null and alternative hypotheses be?  You can state your hypothesis in terms of words or in terms of **$p_{old}$** and **$p_{new}$**, which are the converted rates for the old and new pages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**H_0**: The new page is worse, or only as good as the old page.\n",
    "$$H_0: P_{new} <= P_{old} $$\n",
    "\n",
    "**H_1**: The new page is better.\n",
    "$$H_1: P_{new} > P_{old} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` Assume under the null hypothesis, $p_{new}$ and $p_{old}$ both have \"true\" success rates equal to the **converted** success rate regardless of page - that is $p_{new}$ and $p_{old}$ are equal. Furthermore, assume they are equal to the **converted** rate in **ab_data.csv** regardless of the page. <br><br>\n",
    "\n",
    "Use a sample size for each page equal to the ones in **ab_data.csv**.  <br><br>\n",
    "\n",
    "Perform the sampling distribution for the difference in **converted** between the two pages over 10,000 iterations of calculating an estimate from the null.  <br><br>\n",
    "\n",
    "Use the cells below to provide the necessary parts of this simulation.  If this doesn't make complete sense right now, don't worry - you are going to work through the problems below to complete this problem.  You can use **Quiz 5** in the classroom to make sure you are on the right track.<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. What is the **conversion rate** for $p_{new}$ under the null? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_new = df2[\"converted\"].mean()\n",
    "p_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. What is the **conversion rate** for $p_{old}$ under the null? <br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_old = df2[\"converted\"].mean()\n",
    "p_old"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. What is $n_{new}$, the number of individuals in the treatment group?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id         145310\n",
       "timestamp       145310\n",
       "group           145310\n",
       "landing_page    145310\n",
       "converted       145310\n",
       "dtype: int64"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_new = df2.query('landing_page == \"new_page\" and group == \"treatment\"').count()\n",
    "n_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. What is $n_{old}$, the number of individuals in the control group?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id         145274\n",
       "timestamp       145274\n",
       "group           145274\n",
       "landing_page    145274\n",
       "converted       145274\n",
       "dtype: int64"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_old = df2.query('landing_page == \"old_page\" and group == \"control\"').count()\n",
    "n_old"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. Simulate $n_{new}$ transactions with a conversion rate of $p_{new}$ under the null.  Store these $n_{new}$ 1's and 0's in **new_page_converted**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11896634780813434"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_new_converted = np.random.binomial(1, p_new, n_new[0])\n",
    "n_new_converted.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. Simulate $n_{old}$ transactions with a conversion rate of $p_{old}$ under the null.  Store these $n_{old}$ 1's and 0's in **old_page_converted**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12112284372977959"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_old_converted = np.random.binomial(1, p_old, n_old[0])\n",
    "n_old_converted.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "g. Find $p_{new}$ - $p_{old}$ for your simulated values from part (e) and (f)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0021564959216452545"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_new_converted.mean() - n_old_converted.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "h. Create 10,000 $p_{new}$ - $p_{old}$ values using the same simulation process you used in parts (a) through (g) above. Store all 10,000 values in a NumPy array called **p_diffs**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_diffs = []\n",
    "for _ in range(10000):\n",
    "    n_new_converted = np.random.binomial(1, p_new, n_new[0])\n",
    "    n_old_converted = np.random.binomial(1, p_old, n_old[0])\n",
    "    p_diffs.append(n_new_converted.mean() - n_old_converted.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.00068612644627580321,\n",
       " -0.00090377785135109523,\n",
       " -0.000366905743480056,\n",
       " -0.0017089939230749279,\n",
       " 2.56575097609818e-05,\n",
       " -0.00071105396211956773,\n",
       " -0.0013027778413528951,\n",
       " 0.0021730282705838683,\n",
       " -0.00095200700041522812,\n",
       " 0.0012575441780357383,\n",
       " 0.00033495143219723633,\n",
       " 0.0014362127710172423,\n",
       " 0.0014981544365789551,\n",
       " 0.0010785106354670726,\n",
       " 0.00093401760067157724,\n",
       " -9.1659476337116463e-05,\n",
       " -0.0024663611436688965,\n",
       " -0.0010205269486056184,\n",
       " -0.00094501260704232815,\n",
       " -0.00025684236728408527,\n",
       " 0.00019739651350186849,\n",
       " -0.0015919532072250497,\n",
       " 0.001112952231669892,\n",
       " 0.0014638731453349846,\n",
       " -0.0004563798804668312,\n",
       " 0.0013609013687103655,\n",
       " 0.00034884130737269925,\n",
       " 0.0011885586633159362,\n",
       " -0.0013441848396062717,\n",
       " -0.00057315626367178407,\n",
       " 0.00047950244963741728,\n",
       " -0.00016732218525593312,\n",
       " -0.00057374461697819179,\n",
       " -0.00019475062899491913,\n",
       " -0.00070436653568914376,\n",
       " -0.0007936121528409612,\n",
       " -0.0008211565618693184,\n",
       " -0.00080049058092448999,\n",
       " 0.00056226699035902272,\n",
       " 0.00075490902173920571,\n",
       " -0.0026655354126363801,\n",
       " 0.00113350054195345,\n",
       " -0.00033251701380621057,\n",
       " -0.0010412526175670456,\n",
       " 0.00023220135391995023,\n",
       " -0.00058035359630109751,\n",
       " -0.00086245271094904929,\n",
       " 0.0005484640891505882,\n",
       " -0.0021772522873009237,\n",
       " -0.0011171796591307737,\n",
       " -0.0015577077287910329,\n",
       " 0.0024066748324484272,\n",
       " -0.00091078077158351922,\n",
       " -0.001502675188007091,\n",
       " -0.00077999684254181911,\n",
       " 0.0009889751050918133,\n",
       " -0.00013267253368098864,\n",
       " -0.002004892528187488,\n",
       " -0.0013925435969350114,\n",
       " -0.00072484322035278048,\n",
       " 0.00032806277188229815,\n",
       " 0.00052125052227133817,\n",
       " -0.0004013950900961516,\n",
       " -0.0002432236462411308,\n",
       " 0.0024963109797659366,\n",
       " 0.0007410992990431603,\n",
       " 0.00021132390685919822,\n",
       " -0.0014682608777547151,\n",
       " 5.9705165054282916e-05,\n",
       " 0.0022139560593325974,\n",
       " 0.00060351709439736279,\n",
       " 0.00068587234586232892,\n",
       " 0.00077549996632031437,\n",
       " -0.0015370690337966481,\n",
       " 0.0021658411701859503,\n",
       " -0.0019225014639125543,\n",
       " 0.0021594248978880348,\n",
       " -0.00063533497592796473,\n",
       " 0.00097537855388360462,\n",
       " 0.0028336438690823307,\n",
       " 0.00046579675462743442,\n",
       " 0.00078256427994123634,\n",
       " -0.00031853504854800763,\n",
       " 0.00039049558455198086,\n",
       " -0.00047008387010490094,\n",
       " 0.00042480245637448177,\n",
       " 0.00065184515503184659,\n",
       " 0.00067204727481916249,\n",
       " 0.00045892685340342965,\n",
       " 0.00022495797624925984,\n",
       " 0.00023871483241633129,\n",
       " -0.0010552499311723834,\n",
       " -7.7624644549925681e-05,\n",
       " -0.00069037263282761829,\n",
       " -0.00049083000352916073,\n",
       " -0.00088312551338148859,\n",
       " 0.0007893028675287489,\n",
       " 0.0011540676113279413,\n",
       " -0.00020168703972314739,\n",
       " 0.0010028291674573225,\n",
       " -0.0013922929072653217,\n",
       " 0.00021821938866174728,\n",
       " 0.001883615858017218,\n",
       " -0.0010480935274687214,\n",
       " 0.00011493041286322891,\n",
       " 0.00014934301774370551,\n",
       " 0.0012229797950559229,\n",
       " -0.0021633845819602066,\n",
       " 0.00095459831302129039,\n",
       " 0.00036956868170603951,\n",
       " 0.00025258083238514917,\n",
       " 0.00015625043714959097,\n",
       " 0.00052105610987442097,\n",
       " -0.00057348881119279049,\n",
       " -0.0016400800339750193,\n",
       " -0.00019484442444957228,\n",
       " -0.0020599301850871277,\n",
       " 0.00070666452432795179,\n",
       " -0.0005045356985391436,\n",
       " 0.00039717277875098156,\n",
       " 7.3634263783525777e-05,\n",
       " 0.0013813217761012159,\n",
       " -0.00064252207632588265,\n",
       " 0.00064476719843568897,\n",
       " 0.0010230636893107797,\n",
       " -0.00073854891536277723,\n",
       " -0.00071109830178905931,\n",
       " -0.00013990908986406814,\n",
       " 0.00030745306821024221,\n",
       " -0.0038905283044805006,\n",
       " 0.0022899137975905814,\n",
       " -0.00031181692542332773,\n",
       " -0.00018770507446494444,\n",
       " 0.00021138700561958157,\n",
       " 0.00017017271439119519,\n",
       " -0.0008623350402877511,\n",
       " 0.00099579213978685799,\n",
       " -0.00064905260854127012,\n",
       " 0.0010921447048571481,\n",
       " 0.0010303786926013914,\n",
       " 0.0020078112721988589,\n",
       " 0.0003830884911786292,\n",
       " -0.0023079680014665288,\n",
       " 0.0014707447519309025,\n",
       " 0.00024588828983899969,\n",
       " 0.0022142238027213212,\n",
       " -0.00040139679546806473,\n",
       " -0.0017296564932759578,\n",
       " -9.8645342850520179e-05,\n",
       " 0.00085131445333848366,\n",
       " 0.0010030440443170585,\n",
       " -0.0006421878234329631,\n",
       " 0.00099577508606782383,\n",
       " -0.00038066259964712745,\n",
       " 0.0017939421925178556,\n",
       " -0.0017090911292733657,\n",
       " 0.0011061351969748612,\n",
       " 0.00088585325573975471,\n",
       " -0.0006832929708595753,\n",
       " 0.0012987567638922254,\n",
       " -0.00232148440019532,\n",
       " -0.0016402625087686001,\n",
       " 0.0021937778147519266,\n",
       " -0.00086945904192525791,\n",
       " 0.00056921192794673336,\n",
       " 0.0013058091399098248,\n",
       " 0.00024576550306200373,\n",
       " -0.0010206872535644673,\n",
       " -0.00011228793910010582,\n",
       " -0.00064906454614457876,\n",
       " -0.0017986147220453164,\n",
       " 0.0023792992552384151,\n",
       " -0.00075922001242330339,\n",
       " -0.0014954676231463543,\n",
       " 0.0010715912784578924,\n",
       " 0.0015120784191924724,\n",
       " -0.0016540330079108795,\n",
       " -0.0017640622766688235,\n",
       " -0.0010827619381345854,\n",
       " -0.00040115974877358296,\n",
       " 0.00049340255704430347,\n",
       " -0.00072470167448487888,\n",
       " -0.00071808416830244914,\n",
       " 0.0015738529583077393,\n",
       " -0.00071783518400465873,\n",
       " -0.0017159081639684104,\n",
       " -0.0016539511500595627,\n",
       " 0.00064501447736158013,\n",
       " 0.00023913264853246685,\n",
       " -0.00095868589998615583,\n",
       " -0.00020874623722835772,\n",
       " 0.00015628113384383302,\n",
       " -0.00011910156305132424,\n",
       " -0.00069045790142274743,\n",
       " 0.0003349139140153834,\n",
       " 0.0011613962575937609,\n",
       " -0.00085549754112988763,\n",
       " 0.00047975996079471783,\n",
       " 0.0016152241437511761,\n",
       " 0.0017734228735566404,\n",
       " 9.4274664149809895e-05,\n",
       " -0.001991415353012449,\n",
       " -0.0018327493513056597,\n",
       " -0.00015377850057671227,\n",
       " -0.00033934428073266465,\n",
       " 0.00089274362142660602,\n",
       " -3.6427407040573456e-05,\n",
       " 4.6828565017642543e-06,\n",
       " -0.00028459142094076906,\n",
       " -0.0016400834447188317,\n",
       " -0.00049762998450517126,\n",
       " -0.00032544076258197996,\n",
       " 0.0002318176452418691,\n",
       " 0.0012711611937067796,\n",
       " 0.00021810512874427535,\n",
       " -0.00027067084907103633,\n",
       " -0.0006558696432363148,\n",
       " 0.00025233866957496964,\n",
       " -0.0014060582902918894,\n",
       " 0.00019047545112077513,\n",
       " -0.001764295912619479,\n",
       " 0.00064482006496469069,\n",
       " -0.00090396373688848841,\n",
       " -0.00039433930333475364,\n",
       " 0.0019320428302220666,\n",
       " 0.0010029263736557742,\n",
       " 0.00031463675786433387,\n",
       " 0.00096146650887340979,\n",
       " 0.0010855902974351156,\n",
       " -0.00017418185424854238,\n",
       " 0.0026616217736055992,\n",
       " -0.00018815529264723518,\n",
       " -0.0014130288084581583,\n",
       " 0.00021785443907458568,\n",
       " -0.0013374428412749606,\n",
       " 0.00082389624183092092,\n",
       " -0.00066273954446031957,\n",
       " 0.00023871312704441816,\n",
       " 0.00063101204764054453,\n",
       " -0.00055956141783547464,\n",
       " -0.00056646542649754772,\n",
       " -0.00029105714902384616,\n",
       " 0.0019869611110885227,\n",
       " -0.0012136686539532815,\n",
       " 0.00065898279964457518,\n",
       " 0.00081010016211009728,\n",
       " 0.0020351084023013249,\n",
       " 0.00017687889991255257,\n",
       " 0.0004864473872251418,\n",
       " 0.0017804292045328768,\n",
       " -0.0020188403860076226,\n",
       " -0.0013717906420231268,\n",
       " 5.2959755979159473e-05,\n",
       " 0.00032814974584931267,\n",
       " -0.0014750489211274032,\n",
       " 0.00041085630392623251,\n",
       " -4.3558230165691159e-05,\n",
       " 0.0021246285843294632,\n",
       " 0.00046611224842943444,\n",
       " 0.00063126444268213344,\n",
       " -0.00062860832594378857,\n",
       " 0.0012503997119353988,\n",
       " 0.0013536682232710845,\n",
       " -0.00088980952906811406,\n",
       " -0.00084865151511251402,\n",
       " -0.0018260465765281009,\n",
       " -0.0010757811877369072,\n",
       " 3.9046005597065392e-05,\n",
       " -0.00070423181130883916,\n",
       " 0.00087170757477890437,\n",
       " 0.00087884863013543146,\n",
       " 0.00094773523328488263,\n",
       " -0.0016816319888401376,\n",
       " -0.0014404947703789972,\n",
       " 0.0016774079721230545,\n",
       " 0.0024066321981508487,\n",
       " -0.00034612720798964103,\n",
       " -0.0011445654685721951,\n",
       " 0.0015944575458640836,\n",
       " -0.00011233057339765651,\n",
       " 0.00042439657786165486,\n",
       " -0.00016025957700692428,\n",
       " -0.0016952012540979167,\n",
       " -0.0013100212190235855,\n",
       " -2.2810391369518235e-05,\n",
       " -0.00061460930696655147,\n",
       " 0.00026638373359358369,\n",
       " 0.0010717703425076608,\n",
       " 0.00052094867144455992,\n",
       " -0.0011516178445897945,\n",
       " -0.0017915725782591263,\n",
       " 0.0018423879238136237,\n",
       " -0.0026035135945952498,\n",
       " 0.0002045171043955768,\n",
       " 0.00045209788110507632,\n",
       " -0.0031130749293885734,\n",
       " 0.00063115870962417164,\n",
       " 0.00086504913967080932,\n",
       " 0.0011956178608211326,\n",
       " 0.00087896630079671578,\n",
       " 0.0013056965853642521,\n",
       " 0.00053468847389258334,\n",
       " -0.0023493477137694757,\n",
       " 0.0021314626727435282,\n",
       " -0.0014681619661843642,\n",
       " -0.00042162620120582417,\n",
       " 0.00033532831938772045,\n",
       " 0.0012160024554020432,\n",
       " 0.0015601847814796233,\n",
       " -0.00042883035532274849,\n",
       " -0.0004630246725996906,\n",
       " 0.001140472765491618,\n",
       " -0.0010412355638480253,\n",
       " 0.0014294332545040783,\n",
       " 0.0022622994683142161,\n",
       " -0.00040143260827800453,\n",
       " 0.00060352221051306054,\n",
       " -0.00048386801222240206,\n",
       " -0.003030411005609232,\n",
       " -0.0029134879604206521,\n",
       " 0.0014296054970662359,\n",
       " 0.001188388126125664,\n",
       " -0.00175027472380751,\n",
       " 0.00082379391951674374,\n",
       " 0.00068608040123445402,\n",
       " -0.00096571951691282187,\n",
       " 0.00081707238564826534,\n",
       " -0.00031196529277886798,\n",
       " 0.00073455000773683943,\n",
       " 0.00052086851896514241,\n",
       " -0.00051824480429292497,\n",
       " 0.0020417924179879643,\n",
       " 0.00099587911375388638,\n",
       " 0.00046588202322256356,\n",
       " 0.0023719160370717085,\n",
       " -0.0018052970323600287,\n",
       " -0.00036029676415715028,\n",
       " -0.00040813367768366415,\n",
       " 0.0012644481866978113,\n",
       " -0.0003875785459124953,\n",
       " 0.0007410413163984747,\n",
       " -0.00093143140418124049,\n",
       " 0.00073453295401781915,\n",
       " -0.00087609701257052031,\n",
       " 0.0022345811113517744,\n",
       " -0.0024178147954308782,\n",
       " -0.00049061853741323713,\n",
       " 0.00061024886049726446,\n",
       " -0.0021354547588818412,\n",
       " -0.002500826615078347,\n",
       " -0.0016884336751880474,\n",
       " 0.00022523936261320543,\n",
       " 0.00032135317561711452,\n",
       " 0.0013470404848572454,\n",
       " 0.0018906255997372529,\n",
       " 3.2080603546494335e-05,\n",
       " 0.00031438436282275883,\n",
       " 0.001223019018609689,\n",
       " -0.0015025541066020082,\n",
       " 0.0010095046562844379,\n",
       " -0.0012339031758067526,\n",
       " 0.00054183635073673519,\n",
       " -0.00016042670345338406,\n",
       " 0.0010854572784267102,\n",
       " -0.00066976463452747548,\n",
       " -0.00098637356025432776,\n",
       " 0.0011199210444642616,\n",
       " 0.0021315496467105566,\n",
       " 0.0016013905458484856,\n",
       " 0.00066542976863673275,\n",
       " -0.00010509060647076462,\n",
       " 0.0013125409060097265,\n",
       " -0.00073156134347747426,\n",
       " 0.002097084175301106,\n",
       " 0.0011401180481358797,\n",
       " -0.00024992301027489117,\n",
       " 0.0013745098575218828,\n",
       " 0.00077553748450218118,\n",
       " -0.0002772781230220428,\n",
       " 0.0010510293251991126,\n",
       " 0.00080296763361306644,\n",
       " 0.0014292558958261953,\n",
       " -0.0014889729037409205,\n",
       " -0.0025763290190383287,\n",
       " -0.0017365963147479707,\n",
       " 0.0019385818892969642,\n",
       " 0.0032192434256446967,\n",
       " 0.0010647145557462628,\n",
       " 0.0011198477134724549,\n",
       " -0.00086942663985911661,\n",
       " -0.00021533816283224316,\n",
       " -7.1087290846927331e-05,\n",
       " 0.00010788997444895199,\n",
       " 0.0028608864272839096,\n",
       " 0.000445429213765558,\n",
       " -0.00029105032753623528,\n",
       " -0.0027893931631812752,\n",
       " -0.00080759411809916415,\n",
       " -0.0015715822556193748,\n",
       " -0.0012204311167474391,\n",
       " 0.00026612963318008165,\n",
       " 0.0018770529237356892,\n",
       " 0.002661512629803825,\n",
       " 0.00092003222466957579,\n",
       " 0.00024570922578923127,\n",
       " -0.0001193624849524233,\n",
       " 0.0010234184066665458,\n",
       " 0.00039714378742863876,\n",
       " -0.00070413289973848825,\n",
       " -0.00037397517321667573,\n",
       " -0.0011926513663964994,\n",
       " -1.603258022825349e-05,\n",
       " -0.00062862196891899647,\n",
       " -0.0013165022954537975,\n",
       " 0.00056926820521953359,\n",
       " 0.0013121469651002082,\n",
       " 0.0019664366760116098,\n",
       " 0.00059692005267776571,\n",
       " 0.0011337239456726961,\n",
       " -0.000469816126716191,\n",
       " 0.00045190858482387075,\n",
       " -0.00057310510251470936,\n",
       " -0.00011927892172919341,\n",
       " 0.0019936093139651945,\n",
       " -0.00058024956861504884,\n",
       " -0.0026107723206130612,\n",
       " -0.0010687850889920941,\n",
       " 0.0008169069645737187,\n",
       " 0.000651717252139139,\n",
       " 0.0016565850969631896,\n",
       " -0.0027209823587926729,\n",
       " -0.00036689380587673348,\n",
       " 0.00072043502347023114,\n",
       " -0.0011242371512640847,\n",
       " -0.00080728203504097651,\n",
       " -0.00041483304171742452,\n",
       " 0.00014924751691716698,\n",
       " -0.0023423925439503279,\n",
       " -0.003133972840912172,\n",
       " 0.00050706732312860714,\n",
       " 0.00075477941347459887,\n",
       " 0.00025965196749366815,\n",
       " -4.3524122727622849e-05,\n",
       " 0.00038336646679874853,\n",
       " 0.00075521257793786933,\n",
       " -0.0015850185018687346,\n",
       " -0.0016882563165101783,\n",
       " 0.00096151084854287361,\n",
       " 0.0026476312814878722,\n",
       " 0.0011747574674794148,\n",
       " 3.928134691963403e-05,\n",
       " 0.0036458679771542712,\n",
       " -0.00012627160973020801,\n",
       " 0.0012368577326280772,\n",
       " -0.0014060173613662241,\n",
       " 0.00169104204151313,\n",
       " 0.00095448916921951621,\n",
       " -0.00037381827900163922,\n",
       " -0.0027828012375774036,\n",
       " -0.0014818147946653454,\n",
       " -0.00022932183346234536,\n",
       " -0.00034616472617150784,\n",
       " 0.0017256746393690542,\n",
       " -0.0012270861411117079,\n",
       " 0.0004449141914509569,\n",
       " 0.00086491953140620248,\n",
       " 0.0028538596318448683,\n",
       " -0.00051852960140065518,\n",
       " -8.4692368914660032e-05,\n",
       " 0.00048655823639881524,\n",
       " 0.00029389232981198721,\n",
       " 0.00019763356019633638,\n",
       " 0.0013126483444395876,\n",
       " 0.0022624410141821316,\n",
       " -0.0010688191964301347,\n",
       " 0.00030758949796244606,\n",
       " -0.00073859154966032792,\n",
       " -0.0003391345199886403,\n",
       " 0.001656711294483984,\n",
       " -0.0015987821795234031,\n",
       " -0.0013099751739822085,\n",
       " 0.0019250791335433948,\n",
       " -0.00020856376243477692,\n",
       " -0.0012960597182282013,\n",
       " -0.00011910667916703588,\n",
       " -0.00054586936580071355,\n",
       " 0.00081008822450677476,\n",
       " -0.0035192603144163781,\n",
       " 0.0014157070950312628,\n",
       " -0.0014201613369551891,\n",
       " 0.0018012435528332316,\n",
       " 0.0012160143930053657,\n",
       " 0.0011748563790497657,\n",
       " 0.00030065138186233242,\n",
       " -0.00027042868626087069,\n",
       " 0.00087886909459826412,\n",
       " 0.00070667475655936118,\n",
       " -0.0010756601063318105,\n",
       " -0.00061472356688400953,\n",
       " 0.00093369699075389334,\n",
       " -0.0021839107224090187,\n",
       " -0.0010619970456194061,\n",
       " 0.00036251630568842619,\n",
       " -0.00010515711597497424,\n",
       " 0.00053474304579347043,\n",
       " -0.0019844192542676498,\n",
       " 0.0024620672067038191,\n",
       " -0.00027754245566696811,\n",
       " 0.0022210169622097209,\n",
       " 0.00010801958271355883,\n",
       " -0.00016733753360305414,\n",
       " 0.0017459125719663376,\n",
       " -2.9535335981809019e-05,\n",
       " -0.0001055169494464242,\n",
       " -0.00053203235715423847,\n",
       " -0.00066289302793155758,\n",
       " 0.00047955020005067961,\n",
       " -0.0012822551116478675,\n",
       " 0.0021796082585844173,\n",
       " 3.9103988241764864e-05,\n",
       " 0.00086512758677832757,\n",
       " 0.0020559636795273589,\n",
       " -0.0019361150688397971,\n",
       " 0.00049349294175513037,\n",
       " 0.00036933504575537013,\n",
       " -0.00010531230481811149,\n",
       " -0.00011894637420818699,\n",
       " 0.00049311605456466012,\n",
       " 0.00060400312538960721,\n",
       " 0.00072755902510773796,\n",
       " 0.00029409526906840067,\n",
       " 0.0011265163808119594,\n",
       " -0.001027732808094442,\n",
       " 0.002400268792381921,\n",
       " -0.00095871659668041176,\n",
       " 0.0005829432035352744,\n",
       " 0.00122334986075881,\n",
       " -0.00018119841745617427,\n",
       " -0.00027727641765015743,\n",
       " 0.0013814889025476756,\n",
       " -2.3507604249017877e-06,\n",
       " -0.00029120210563557403,\n",
       " -0.0002569344573668253,\n",
       " 0.00057567424528603972,\n",
       " 0.0010094125662016978,\n",
       " 0.00030077757938312688,\n",
       " -0.00056651999839843481,\n",
       " 0.0019798882706081045,\n",
       " 0.00019064939905483191,\n",
       " -0.00062845313710063744,\n",
       " 0.0017874031334429441,\n",
       " 0.0014778329407584695,\n",
       " 0.00072056804247863648,\n",
       " -0.0032920624269158621,\n",
       " -0.00013983575887226152,\n",
       " -0.0016538590599768088,\n",
       " -0.0016401670079420616,\n",
       " 0.0026199350943601762,\n",
       " 0.0010511384690008729,\n",
       " 0.0020833563104563912,\n",
       " -0.00034616813691530635,\n",
       " -0.00058717233636804145,\n",
       " 0.0013813200707293166,\n",
       " -0.00020161541410324002,\n",
       " 0.00098206768568594172,\n",
       " 0.0002042323072878327,\n",
       " 0.0002528298166829257,\n",
       " -0.00096565300740861226,\n",
       " 0.00041087506301716592,\n",
       " 0.00094060952627547656,\n",
       " 0.0016016190656834434,\n",
       " -0.0016747654983599314,\n",
       " 0.00018371469369851678,\n",
       " -0.00031189025641514823,\n",
       " 0.0015189414989288941,\n",
       " -0.0003390833588315656,\n",
       " 0.00052101688632066878,\n",
       " 0.00081014109103576259,\n",
       " -0.00053196243690623035,\n",
       " 0.0011748785488844976,\n",
       " 0.00045896607695716796,\n",
       " -0.00043597141067928946,\n",
       " 0.00047993049798497611,\n",
       " -0.0023009053932175061,\n",
       " 0.00023887513737518018,\n",
       " -0.00080029616852758667,\n",
       " -0.00018101423729070809,\n",
       " -0.00058018646985463773,\n",
       " 0.00021816993287657183,\n",
       " -0.0022321808003988169,\n",
       " -0.0010689760906451851,\n",
       " 0.00028721001949726099,\n",
       " 0.00059004332996613618,\n",
       " -0.000793641144163304,\n",
       " -0.0002224587537259376,\n",
       " -0.0001397914192027977,\n",
       " -1.5926847170291691e-05,\n",
       " -0.00069041015100947123,\n",
       " -0.0012135185812258559,\n",
       " 0.0015257721765991328,\n",
       " 0.00028694739222426269,\n",
       " -2.9502933915653839e-05,\n",
       " -0.0013924583283398684,\n",
       " -2.9663238874502729e-05,\n",
       " -0.00095862962271336949,\n",
       " -0.00080764186851244035,\n",
       " 0.00030051836285392708,\n",
       " -0.0014818915364009783,\n",
       " -0.0016883347636176965,\n",
       " -0.00019481713849912874,\n",
       " 0.00061049613942314174,\n",
       " -0.00057345470375474994,\n",
       " -0.00067649298988356477,\n",
       " 0.0023585343627232358,\n",
       " 0.00063111948607040558,\n",
       " 0.0019663599342759908,\n",
       " 0.00089264641522815436,\n",
       " 0.00015617710615777047,\n",
       " -0.00040843552851042852,\n",
       " 0.00013539004380784536,\n",
       " 0.0013538762786432096,\n",
       " 0.00066539566119867832,\n",
       " -0.002266489377593231,\n",
       " 0.0010645968850849785,\n",
       " -0.00091741533148498311,\n",
       " 0.00022481643038133048,\n",
       " 0.00041097738533131534,\n",
       " -0.00060085756691519165,\n",
       " 0.00029382752567969073,\n",
       " -0.00362244526252882,\n",
       " 0.0019043517592100684,\n",
       " 0.00025253478734377222,\n",
       " 0.00086525890041483366,\n",
       " -6.388143135808988e-05,\n",
       " 0.0023038002620222459,\n",
       " 0.0031981715661870413,\n",
       " 0.0011130784291907003,\n",
       " -0.0014886829905174648,\n",
       " 0.00044518193483968072,\n",
       " 0.0022140464440434382,\n",
       " -0.0012273231878061758,\n",
       " 6.6517083633629848e-05,\n",
       " 0.00016284065738157716,\n",
       " 0.0015669029046043031,\n",
       " -0.0019565934588753331,\n",
       " -0.00024270350781081806,\n",
       " -0.0022392280603007048,\n",
       " -0.0013371938569771702,\n",
       " 0.0010301160653283931,\n",
       " -0.00021530235002228948,\n",
       " 0.0019042886604496712,\n",
       " 0.00085133662317321557,\n",
       " 0.001732285324063873,\n",
       " -0.00044985958048285313,\n",
       " -5.0452006596327093e-05,\n",
       " -0.00050459368118382919,\n",
       " -0.00073848411123046687,\n",
       " -0.0021702340187213787,\n",
       " -0.001068945393950943,\n",
       " -0.0016538795244396554,\n",
       " -0.0010964778653759777,\n",
       " 0.0014157872475106803,\n",
       " 0.0021175727975680514,\n",
       " 0.0026201977216331745,\n",
       " -0.0023217214468897879,\n",
       " 0.00089261401316199918,\n",
       " -2.0864277800042297e-06,\n",
       " -0.0011930470126779169,\n",
       " 0.00039041202132875097,\n",
       " -0.0024109585371820813,\n",
       " -0.00028443282135381942,\n",
       " -0.0024521574800633467,\n",
       " 0.001904095953424681,\n",
       " 0.00014936007146273966,\n",
       " -0.0021563236790830831,\n",
       " -0.00024981045572931848,\n",
       " -0.0032575474997212361,\n",
       " -0.00077283361735054623,\n",
       " -0.0026587132618256376,\n",
       " -0.00040801600702239371,\n",
       " -0.00061445411812340034,\n",
       " -0.0011308546574465006,\n",
       " -0.00014662550761684878,\n",
       " 0.00019707590358417071,\n",
       " 0.00033512708550320625,\n",
       " 0.00057591299735239299,\n",
       " 0.00084430812236226116,\n",
       " -0.0013509967581855908,\n",
       " -0.0017915930427219728,\n",
       " -0.00067661407128866147,\n",
       " 0.0015189704902512369,\n",
       " -0.00086241007665147085,\n",
       " -0.0009378579087105654,\n",
       " -0.00059420595329472137,\n",
       " 0.001491274303123527,\n",
       " -0.001571329860577772,\n",
       " -0.00089678857409389301,\n",
       " -0.00027761919740258711,\n",
       " -0.00064912593953307673,\n",
       " 0.0011130733130749887,\n",
       " -0.00032534867249923993,\n",
       " 0.00013534058802266991,\n",
       " 0.0009751739092552919,\n",
       " 0.0017323194315019136,\n",
       " 0.0011403772646650795,\n",
       " -0.0010275366903256394,\n",
       " 0.0014707771539970577,\n",
       " -0.0010069627994635511,\n",
       " 0.00046614635586747499,\n",
       " -0.00092438243890745342,\n",
       " -0.00022918881445394002,\n",
       " -0.00058731217686405768,\n",
       " 0.00085135026614845122,\n",
       " -0.00031857427210177369,\n",
       " -0.0024250581731015686,\n",
       " 0.0018904533571750953,\n",
       " -2.2725122774389095e-05,\n",
       " 0.00091321860071835737,\n",
       " -0.0013994714808037156,\n",
       " -0.00071131317864878141,\n",
       " -0.0011584144148219927,\n",
       " -0.0010964608116569574,\n",
       " 0.00074804764737471108,\n",
       " -0.00087625390678557069,\n",
       " -3.6444460759579855e-05,\n",
       " 0.00095472792128589723,\n",
       " -0.00029832099115736921,\n",
       " -0.00078680876112113829,\n",
       " 0.0046021741253931686,\n",
       " -0.00040108300703796396,\n",
       " 0.00045207059515461889,\n",
       " -0.0028656288770593924,\n",
       " 0.001511960748531202,\n",
       " 0.0010578105470841898,\n",
       " 0.0018974136431099409,\n",
       " 0.00065180081536236889,\n",
       " 0.00050727878924454461,\n",
       " 0.00069311572353300543,\n",
       " -0.0027136673555020752,\n",
       " -0.00081441115279419496,\n",
       " 0.0013607871087928936,\n",
       " 0.00086501673760465414,\n",
       " -0.0010138719242413219,\n",
       " 0.00054846238377867507,\n",
       " 0.0013536767501305946,\n",
       " 0.0016632588804183918,\n",
       " -0.00021576280043598961,\n",
       " -0.00076619735207719697,\n",
       " 0.00033506910285850677,\n",
       " 0.00061731999560578343,\n",
       " 0.00035587492429936529,\n",
       " -0.00067676414401610097,\n",
       " 0.0012713641329631931,\n",
       " -0.0022251386566126269,\n",
       " 0.00088618409788887575,\n",
       " 0.00070693056234477636,\n",
       " -0.00039457805540112079,\n",
       " 0.0010787408606739296,\n",
       " 0.0010027745955564354,\n",
       " 0.0008168199906066903,\n",
       " 0.0020485804613606384,\n",
       " -0.00053209033979892406,\n",
       " 6.006499852574676e-05,\n",
       " 0.0015189500257884042,\n",
       " -0.0013717650614445825,\n",
       " 0.0017391449930564684,\n",
       " -0.0023558509600344335,\n",
       " -0.0015094717582392891,\n",
       " -0.00049772548533170979,\n",
       " 0.0029917829108711963,\n",
       " -0.001585052609306789,\n",
       " 0.00053468335777688558,\n",
       " 0.001787486696666174,\n",
       " -0.00027726789079064729,\n",
       " 0.00069325215328522316,\n",
       " 0.0006998116768229673,\n",
       " 0.00038349095894764373,\n",
       " -0.0027897700503717732,\n",
       " 0.0011334562022839861,\n",
       " 0.0021798077870970323,\n",
       " 0.00049324566282923921,\n",
       " 0.00054187045817478963,\n",
       " 3.2348346935218153e-05,\n",
       " 0.00087217825742402777,\n",
       " 0.00069295371320227117,\n",
       " -0.0016125407410623738,\n",
       " 0.0011608044935435546,\n",
       " -0.0001743029356536252,\n",
       " 0.00085122918474335452,\n",
       " 0.00042472400926696352,\n",
       " 0.0016153810379662126,\n",
       " 4.5885210126841991e-05,\n",
       " -0.0020738661053039675,\n",
       " -0.0013788498395283511,\n",
       " -0.0017499950428154776,\n",
       " -0.0014750489211274032,\n",
       " -0.00057348540044900587,\n",
       " -0.00089721491706955259,\n",
       " -0.0022115830343300835,\n",
       " 0.00050050268347516524,\n",
       " -0.0014130765588714345,\n",
       " 0.0013122782787367143,\n",
       " -0.00080745939371885955,\n",
       " 0.00076211147048424466,\n",
       " 0.0011951778748702513,\n",
       " 0.00032827082725440937,\n",
       " 0.0021453832446132332,\n",
       " 0.0013676501885292874,\n",
       " 0.0018839364679349296,\n",
       " 0.00056911642712019483,\n",
       " 1.1680660618476613e-05,\n",
       " -0.0024181763342742274,\n",
       " -0.0013373660995393416,\n",
       " -0.00031861861177123751,\n",
       " -0.00070410049767233307,\n",
       " -0.0011309313991821196,\n",
       " 0.0023103086244029153,\n",
       " 0.0015533813897598142,\n",
       " -0.00077295640412754218,\n",
       " -0.00074545804014054806,\n",
       " -0.0015231723371335604,\n",
       " 0.00087198043428332594,\n",
       " -0.0014267293873524295,\n",
       " 0.0017602015041670166,\n",
       " -0.001736492287061922,\n",
       " 0.00076895067501400738,\n",
       " -0.0009658081962517634,\n",
       " -0.0012618329988851179,\n",
       " -0.0011719444565260057,\n",
       " -0.00049070380600836627,\n",
       " 0.00045921335588305912,\n",
       " -0.0002360672425374688,\n",
       " -0.0009313512517018091,\n",
       " 0.00088593170284728684,\n",
       " 0.00043853714270680744,\n",
       " -0.00084881693618707454,\n",
       " -0.00067675902790037545,\n",
       " 0.0014775873672044776,\n",
       " 0.00011477351864819241,\n",
       " -0.0020531250879954194,\n",
       " -0.0020116976252791963,\n",
       " 0.0018493226291699388,\n",
       " 0.00060354267497590708,\n",
       " -0.00022920586817296029,\n",
       " -0.0036913744999758497,\n",
       " 0.0011818899959764317,\n",
       " 0.0013879461037712426,\n",
       " -0.00041533783180060235,\n",
       " -0.0015642297341469102,\n",
       " -0.00093130179591663365,\n",
       " -0.00099284610982502963,\n",
       " 0.0011404932299544646,\n",
       " -0.0017847725972831158,\n",
       " 0.0012989648192643505,\n",
       " -0.00018791142446515641,\n",
       " 0.00056917099902108192,\n",
       " 0.0007966025224722395,\n",
       " -0.0021494367241400442,\n",
       " -0.00062143998463679018,\n",
       " 0.00099575973772070281,\n",
       " 0.00097532909809842916,\n",
       " -0.0016127999575915736,\n",
       " -0.0009933679536272555,\n",
       " -0.0015780940287438427,\n",
       " -0.00099322640775932614,\n",
       " -0.00076577783058914828,\n",
       " -0.00062130867100029796,\n",
       " -0.001068884000562445,\n",
       " -0.00020850066367436582,\n",
       " 0.00039015109942763804,\n",
       " 0.00060342329894270963,\n",
       " -0.00054609618026375817,\n",
       " 0.00087185935287824312,\n",
       " 0.0003971932432138281,\n",
       " -0.00098654409744459992,\n",
       " 0.00080324731460509891,\n",
       " -0.00049087434319862455,\n",
       " -0.00055282453561987521,\n",
       " -0.00056646713186943309,\n",
       " 0.0016018168888241591,\n",
       " -0.00012592541923396594,\n",
       " 0.00016293274746430331,\n",
       " -0.00011906575024135668,\n",
       " 0.0017116858526232404,\n",
       " -0.0001260226254324176,\n",
       " 0.00050740669213723832,\n",
       " 0.00043133810470559475,\n",
       " 0.00035593802305976252,\n",
       " 0.00019758580978306017,\n",
       " 0.00010807244924253279,\n",
       " 0.00069295541857415655,\n",
       " -0.0004494349428790928,\n",
       " -0.0013163607495858681,\n",
       " -0.0014063328551682103,\n",
       " -0.00071060203856537774,\n",
       " 0.00010135262074593976,\n",
       " -0.00023600243840515844,\n",
       " -0.00021544048514639258,\n",
       " -0.00080060484084196193,\n",
       " 0.0013537432596347904,\n",
       " 6.6430109666587578e-05,\n",
       " -0.0001534323100804702,\n",
       " -0.00069039309729046483,\n",
       " 0.00078933185885109169,\n",
       " 0.0011884989752993513,\n",
       " -0.00019475403973873151,\n",
       " -0.00064202581310220108,\n",
       " -0.0012548573646031236,\n",
       " 0.00029402875956419106,\n",
       " -0.00095878481155650674,\n",
       " 0.0013470234311382112,\n",
       " -0.001509325096255662,\n",
       " -0.0013511894652105949,\n",
       " 0.0016014894574188504,\n",
       " 0.0010992277775689896,\n",
       " -0.0010207674060438848,\n",
       " -4.345079173581623e-05,\n",
       " 0.00017683626561500188,\n",
       " 0.0013880433099697082,\n",
       " -5.6972306580319043e-05,\n",
       " -0.001082697134002289,\n",
       " -0.00032543223572246982,\n",
       " -0.0014956449818242235,\n",
       " -0.0019365260634683357,\n",
       " -0.0024455689652032597,\n",
       " -0.0007179801406163866,\n",
       " 0.0039760853916926686,\n",
       " -0.0004009721578643044,\n",
       " 0.00042458416877094729,\n",
       " 0.00077565856590727789,\n",
       " 8.0567263767927777e-05,\n",
       " 0.00054147140114957359,\n",
       " -0.0010895329277882532,\n",
       " 0.00010806051163921027,\n",
       " 0.00025934158980739364,\n",
       " -0.00053211592037746835,\n",
       " 0.00027319224142910437,\n",
       " 0.00083072862487307275,\n",
       " -0.00015342037247714768,\n",
       " -0.00073160056703124032,\n",
       " 0.0013193903427709125,\n",
       " 0.00043145236462306669,\n",
       " -0.0013719168395439213,\n",
       " 0.00010117696743998372,\n",
       " 0.00024547900058237426,\n",
       " 0.0012090268211200628,\n",
       " -0.0008898658063409004,\n",
       " 0.0027511982329065238,\n",
       " -0.00069032658778624134,\n",
       " -0.00015354145388224438,\n",
       " -0.00097256554293019548,\n",
       " 0.0007688858708817109,\n",
       " 0.00052088898342796119,\n",
       " -0.0023696896740528078,\n",
       " 4.5123193115059745e-06,\n",
       " 0.0030195046785774365,\n",
       " -0.0013167683334706082,\n",
       " -0.00019519402568961286,\n",
       " 0.0011335568192262363,\n",
       " 0.00012872819795595181,\n",
       " 0.0022007312791991612,\n",
       " 0.00043835296254132738,\n",
       " 0.00019754829160120724,\n",
       " 0.00083082071495581278,\n",
       " 0.00081680123151574302,\n",
       " 0.0010507087152814149,\n",
       " -0.0014064829278956498,\n",
       " 0.00073449884657976472,\n",
       " 0.00091993672384302339,\n",
       " 0.00032116046859212433,\n",
       " 0.0002802360905871798,\n",
       " -0.00055967056163723494,\n",
       " 0.00010792919800271805,\n",
       " 0.00052791577886703023,\n",
       " -0.00069071711795196111,\n",
       " -0.00053906767945281764,\n",
       " 0.00094786654692138872,\n",
       " -0.00074565245253745138,\n",
       " -0.00086938741630536442,\n",
       " 6.705598115486211e-05,\n",
       " 0.00074798454861431385,\n",
       " -9.1336876818920443e-06,\n",
       " 0.00067913034753100399,\n",
       " -0.00064215712673872105,\n",
       " -0.0010826698480518454,\n",
       " -0.0030236587750464977,\n",
       " 0.0038521560155278523,\n",
       " 0.0013329510811691675,\n",
       " -0.0010826903125146781,\n",
       " -0.0012134077320521824,\n",
       " -0.0016333141604370632,\n",
       " 0.0014088644797576738,\n",
       " 0.00065144780337852981,\n",
       " 0.000631146772020863,\n",
       " 0.00058282723824588933,\n",
       " -0.0017087654032399702,\n",
       " 0.0024619580629020588,\n",
       " -0.0012068635568615732,\n",
       " 9.4310476959777456e-05,\n",
       " 0.0016014621714684069,\n",
       " ...]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_diffs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "i. Plot a histogram of the **p_diffs**.  Does this plot look like what you expected?  Use the matching problem in the classroom to assure you fully understand what was computed here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([    8.,    95.,   533.,  1597.,  2961.,  2779.,  1464.,   471.,\n",
       "           82.,    10.]),\n",
       " array([ -4.70288543e-03,  -3.75379936e-03,  -2.80471330e-03,\n",
       "         -1.85562723e-03,  -9.06541160e-04,   4.25449077e-05,\n",
       "          9.91630975e-04,   1.94071704e-03,   2.88980311e-03,\n",
       "          3.83888918e-03,   4.78797525e-03]),\n",
       " <a list of 10 Patch objects>)"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAEi5JREFUeJzt3X+MXedd5/H3p3Z/sBSIQybB2C4OXa+E8wdp10qz6v6RJWziJKgu0lZKVgtWiWQkEqnVsgKX/hG2bKQUlgZVW4ICsXB3W9xAW9VqvRtMthVCokmckqZ1TPA0Cc3UJjY4pEXVZuX0u3/cx5tr53rmznju3Eme90s6uud+z3POeR6PNJ855zn3OlWFJKk/r5t2ByRJ02EASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjq1dtodmM8ll1xSmzdvnnY3JOlV5dFHH/37qppZqN2qDoDNmzdz6NChaXdDkl5VkvztOO28BSRJnTIAJKlTCwZAkjcleTjJV5McTvKfW/3yJA8lOZrkU0ne0OpvbO9n2/bNQ8f6QKs/meT6SQ1KkrSwca4AXgR+qqp+ErgS2J7kauDDwN1VtQV4Hri1tb8VeL6q/jlwd2tHkq3AzcAVwHbgd5OsWc7BSJLGt2AA1MA/tbevb0sBPwX8SavvBd7d1ne097Tt1yZJq++rqher6mlgFrhqWUYhSVq0seYAkqxJ8hhwAjgIfAP4x6o63ZrMARva+gbgWYC2/QXgh4frI/aRJK2wsQKgql6qqiuBjQz+av+JUc3aa86z7Xz1syTZleRQkkMnT54cp3uSpCVY1FNAVfWPwJeAq4GLkpz5HMFG4FhbnwM2AbTtPwScGq6P2Gf4HPdW1baq2jYzs+DnGCRJSzTOU0AzSS5q698H/DRwBPgi8O9as53A59r6/vaetv1/1+A/Ht4P3NyeEroc2AI8vFwDkSQtzjifBF4P7G1P7LwOuL+qPp/kCWBfkv8C/BVwX2t/H/Dfk8wy+Mv/ZoCqOpzkfuAJ4DRwW1W9tLzDUW827/7C1M79zF03Te3c0nJYMACq6nHgbSPqTzHiKZ6q+j/Ae85zrDuBOxffTUnScvOTwJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6Nc53AUkaYVrfQ+R3EGm5eAUgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpxYMgCSbknwxyZEkh5O8r9V/Pcm3kjzWlhuH9vlAktkkTya5fqi+vdVmk+yezJAkSeMY5z+EOQ38clV9JckPAI8mOdi23V1V/3W4cZKtwM3AFcCPAn+W5F+0zR8D/i0wBzySZH9VPbEcA5EkLc6CAVBVx4Hjbf07SY4AG+bZZQewr6peBJ5OMgtc1bbNVtVTAEn2tbYGgCRNwaLmAJJsBt4GPNRKtyd5PMmeJOtabQPw7NBuc612vrokaQrGDoAkbwY+Dby/qr4N3AO8FbiSwRXCb59pOmL3mqd+7nl2JTmU5NDJkyfH7Z4kaZHGCoAkr2fwy/8TVfUZgKp6rqpeqqrvAb/Py7d55oBNQ7tvBI7NUz9LVd1bVduqatvMzMxixyNJGtM4TwEFuA84UlUfGaqvH2r2s8DX2/p+4OYkb0xyObAFeBh4BNiS5PIkb2AwUbx/eYYhSVqscZ4Ceifwc8DXkjzWar8G3JLkSga3cZ4BfhGgqg4nuZ/B5O5p4Laqegkgye3AA8AaYE9VHV7GsUiSFmGcp4D+gtH37w/Ms8+dwJ0j6gfm20+StHL8JLAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROLRgASTYl+WKSI0kOJ3lfq1+c5GCSo+11XasnyUeTzCZ5PMnbh461s7U/mmTn5IYlSVrIOFcAp4FfrqqfAK4GbkuyFdgNPFhVW4AH23uAG4AtbdkF3AODwADuAN4BXAXccSY0JEkrb8EAqKrjVfWVtv4d4AiwAdgB7G3N9gLvbus7gI/XwJeBi5KsB64HDlbVqap6HjgIbF/W0UiSxraoOYAkm4G3AQ8Bl1XVcRiEBHBpa7YBeHZot7lWO19dkjQFYwdAkjcDnwbeX1Xfnq/piFrNUz/3PLuSHEpy6OTJk+N2T5K0SGMFQJLXM/jl/4mq+kwrP9du7dBeT7T6HLBpaPeNwLF56mepqnuraltVbZuZmVnMWCRJizDOU0AB7gOOVNVHhjbtB848ybMT+NxQ/efb00BXAy+0W0QPANclWdcmf69rNUnSFKwdo807gZ8DvpbksVb7NeAu4P4ktwLfBN7Tth0AbgRmge8C7wWoqlNJfgN4pLX7UFWdWpZRSJIWbcEAqKq/YPT9e4BrR7Qv4LbzHGsPsGcxHZQkTYafBJakThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpU2un3QG9Nmze/YVpd0HSInkFIEmdMgAkqVMLBkCSPUlOJPn6UO3Xk3wryWNtuXFo2weSzCZ5Msn1Q/XtrTabZPfyD0WStBjjXAH8IbB9RP3uqrqyLQcAkmwFbgauaPv8bpI1SdYAHwNuALYCt7S2kqQpWXASuKr+PMnmMY+3A9hXVS8CTyeZBa5q22ar6imAJPta2ycW3WNJ0rK4kDmA25M83m4RrWu1DcCzQ23mWu18dUnSlCw1AO4B3gpcCRwHfrvVM6JtzVN/hSS7khxKcujkyZNL7J4kaSFLCoCqeq6qXqqq7wG/z8u3eeaATUNNNwLH5qmPOva9VbWtqrbNzMwspXuSpDEs6YNgSdZX1fH29meBM08I7Qc+meQjwI8CW4CHGVwBbElyOfAtBhPF//5COi71apofunvmrpumdm4tvwUDIMkfAdcAlySZA+4ArklyJYPbOM8AvwhQVYeT3M9gcvc0cFtVvdSOczvwALAG2FNVh5d9NJKksY3zFNAtI8r3zdP+TuDOEfUDwIFF9U6SNDF+EliSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnFgyAJHuSnEjy9aHaxUkOJjnaXte1epJ8NMlskseTvH1on52t/dEkOyczHEnSuMa5AvhDYPs5td3Ag1W1BXiwvQe4AdjSll3APTAIDOAO4B3AVcAdZ0JDkjQdCwZAVf05cOqc8g5gb1vfC7x7qP7xGvgycFGS9cD1wMGqOlVVzwMHeWWoSJJW0FLnAC6rquMA7fXSVt8APDvUbq7VzleXJE3Jck8CZ0St5qm/8gDJriSHkhw6efLksnZOkvSypQbAc+3WDu31RKvPAZuG2m0Ejs1Tf4WqureqtlXVtpmZmSV2T5K0kKUGwH7gzJM8O4HPDdV/vj0NdDXwQrtF9ABwXZJ1bfL3ulaTJE3J2oUaJPkj4BrgkiRzDJ7muQu4P8mtwDeB97TmB4AbgVngu8B7AarqVJLfAB5p7T5UVedOLEuSVtCCAVBVt5xn07Uj2hZw23mOswfYs6jeSZImxk8CS1KnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6tQFBUCSZ5J8LcljSQ612sVJDiY52l7XtXqSfDTJbJLHk7x9OQYgSVqa5bgC+DdVdWVVbWvvdwMPVtUW4MH2HuAGYEtbdgH3LMO5JUlLtHYCx9wBXNPW9wJfAn611T9eVQV8OclFSdZX1fEJ9KFbm3d/YdpdkPQqcaFXAAX8aZJHk+xqtcvO/FJvr5e2+gbg2aF951pNkjQFF3oF8M6qOpbkUuBgkr+ep21G1OoVjQZBsgvgLW95ywV2T5J0Phd0BVBVx9rrCeCzwFXAc0nWA7TXE635HLBpaPeNwLERx7y3qrZV1baZmZkL6Z4kaR5LvgJI8v3A66rqO239OuBDwH5gJ3BXe/1c22U/cHuSfcA7gBe8/y+9ukxrjumZu26aynlf6y7kFtBlwGeTnDnOJ6vqfyV5BLg/ya3AN4H3tPYHgBuBWeC7wHsv4NySpAu05ACoqqeAnxxR/wfg2hH1Am5b6vkkScvLTwJLUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1Km10+7Aa9Hm3V+YdhckaUEGgKRVb5p/VD1z101TO/ekeQtIkjq14gGQZHuSJ5PMJtm90ueXJA2saAAkWQN8DLgB2ArckmTrSvZBkjSw0lcAVwGzVfVUVf1fYB+wY4X7IEli5SeBNwDPDr2fA94xqZP5NI6kCzWt3yMrMfm80gGQEbU6q0GyC9jV3v5Tkicn3qvFuwT4+2l3Yoocf9/jB/8NJj7+fPiCdv+xcRqtdADMAZuG3m8Ejg03qKp7gXtXslOLleRQVW2bdj+mxfH3PX7w3+C1Mv6VngN4BNiS5PIkbwBuBvavcB8kSazwFUBVnU5yO/AAsAbYU1WHV7IPkqSBFf8kcFUdAA6s9HmX2aq+RbUCHL96/zd4TYw/VbVwK0nSa45fBSFJnTIAhiS5OMnBJEfb67rztNvZ2hxNsnOo/i+TfK19zcVHk+Sc/f5TkkpyyaTHshSTGn+S30ry10keT/LZJBet1JjGsdDXkyR5Y5JPte0PJdk8tO0Drf5kkuvHPeZqstzjT7IpyReTHElyOMn7Vm40izeJn3/btibJXyX5/ORHsURV5dIW4DeB3W19N/DhEW0uBp5qr+va+rq27WHgXzH4vMP/BG4Y2m8Tg8nvvwUumfZYV3L8wHXA2rb+4VHHneKY1wDfAH4ceAPwVWDrOW1+Cfi9tn4z8Km2vrW1fyNweTvOmnGOuVqWCY1/PfD21uYHgL/pafxD+/1H4JPA56c9zvMtXgGcbQewt63vBd49os31wMGqOlVVzwMHge1J1gM/WFV/WYOf/sfP2f9u4Fc454Nvq8xExl9Vf1pVp9v+X2bw+Y/VYpyvJxn+d/kT4Np2dbMD2FdVL1bV08BsO96r6StPln38VXW8qr4CUFXfAY4w+BaA1WgSP3+SbARuAv5gBcawZAbA2S6rquMA7fXSEW1GfZ3FhrbMjaiT5F3At6rqq5Po9DKayPjP8QsMrg5Wi/ONZ2SbFmQvAD88z77jHHO1mMT4/792u+RtwEPL2OflNKnx/w6DP/i+t/xdXj7d/YcwSf4M+JERmz447iFG1Op89ST/rB37ujGPP1ErPf5zzv1B4DTwiTHPtRIW7Pc8bc5XH/WH1Wq98pvE+Ac7JW8GPg28v6q+veQeTtayjz/JzwAnqurRJNdcYP8mqrsAqKqfPt+2JM8lWV9Vx9stjRMjms0B1wy93wh8qdU3nlM/BryVwf3Br7Y50Y3AV5JcVVV/dwFDWZIpjP/MsXcCPwNc224RrRYLfj3JUJu5JGuBHwJOLbDvQsdcLSYy/iSvZ/DL/xNV9ZnJdH1ZTGL87wLeleRG4E3ADyb5H1X1HyYzhAsw7UmI1bQAv8XZk6C/OaLNxcDTDCZA17X1i9u2R4CreXkS9MYR+z/D6p0Ensj4ge3AE8DMtMc4YjxrGUxkX87Lk4BXnNPmNs6eBLy/rV/B2ZOATzGYVFzwmKtlmdD4w2AO6HemPb5pjP+cfa9hFU8CT70Dq2lhcF/vQeBoez3zi20b8AdD7X6BwYTPLPDeofo24OsMngb4b7QP2p1zjtUcABMZf2v3LPBYW35v2mM9Z9w3MnhS5RvAB1vtQ8C72vqbgD9u43gY+PGhfT/Y9nuSs5/6esUxV+uy3OMH/jWDWySPD/3MX/HH0GpZJvHzH9q+qgPATwJLUqd8CkiSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUqf8HVbyfErK+njYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe8783dadd8>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(p_diffs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Does this plot look like what I expected?**\n",
    "\n",
    "Yes, according to **Central Limit Theorem**, given a sufficiently large sample size, the sampling distribution of the mean for a variable will approximate a normal distribution regardless of that variable's distribution in the population."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "j. What proportion of the **p_diffs** are greater than the actual difference observed in **ab_data.csv**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0015911138438704647"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute actual difference observed in original dataset ab_data.csv\n",
    "obs_diff = df2.query('group == \"treatment\"')[\"converted\"].mean() - df.query('group == \"control\"')[\"converted\"].mean()\n",
    "obs_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.00068612644627580321,\n",
       " -0.00090377785135109523,\n",
       " -0.000366905743480056,\n",
       " -0.0017089939230749279,\n",
       " 2.56575097609818e-05,\n",
       " -0.00071105396211956773,\n",
       " -0.0013027778413528951,\n",
       " 0.0021730282705838683,\n",
       " -0.00095200700041522812,\n",
       " 0.0012575441780357383,\n",
       " 0.00033495143219723633,\n",
       " 0.0014362127710172423,\n",
       " 0.0014981544365789551,\n",
       " 0.0010785106354670726,\n",
       " 0.00093401760067157724,\n",
       " -9.1659476337116463e-05,\n",
       " -0.0024663611436688965,\n",
       " -0.0010205269486056184,\n",
       " -0.00094501260704232815,\n",
       " -0.00025684236728408527,\n",
       " 0.00019739651350186849,\n",
       " -0.0015919532072250497,\n",
       " 0.001112952231669892,\n",
       " 0.0014638731453349846,\n",
       " -0.0004563798804668312,\n",
       " 0.0013609013687103655,\n",
       " 0.00034884130737269925,\n",
       " 0.0011885586633159362,\n",
       " -0.0013441848396062717,\n",
       " -0.00057315626367178407,\n",
       " 0.00047950244963741728,\n",
       " -0.00016732218525593312,\n",
       " -0.00057374461697819179,\n",
       " -0.00019475062899491913,\n",
       " -0.00070436653568914376,\n",
       " -0.0007936121528409612,\n",
       " -0.0008211565618693184,\n",
       " -0.00080049058092448999,\n",
       " 0.00056226699035902272,\n",
       " 0.00075490902173920571,\n",
       " -0.0026655354126363801,\n",
       " 0.00113350054195345,\n",
       " -0.00033251701380621057,\n",
       " -0.0010412526175670456,\n",
       " 0.00023220135391995023,\n",
       " -0.00058035359630109751,\n",
       " -0.00086245271094904929,\n",
       " 0.0005484640891505882,\n",
       " -0.0021772522873009237,\n",
       " -0.0011171796591307737,\n",
       " -0.0015577077287910329,\n",
       " 0.0024066748324484272,\n",
       " -0.00091078077158351922,\n",
       " -0.001502675188007091,\n",
       " -0.00077999684254181911,\n",
       " 0.0009889751050918133,\n",
       " -0.00013267253368098864,\n",
       " -0.002004892528187488,\n",
       " -0.0013925435969350114,\n",
       " -0.00072484322035278048,\n",
       " 0.00032806277188229815,\n",
       " 0.00052125052227133817,\n",
       " -0.0004013950900961516,\n",
       " -0.0002432236462411308,\n",
       " 0.0024963109797659366,\n",
       " 0.0007410992990431603,\n",
       " 0.00021132390685919822,\n",
       " -0.0014682608777547151,\n",
       " 5.9705165054282916e-05,\n",
       " 0.0022139560593325974,\n",
       " 0.00060351709439736279,\n",
       " 0.00068587234586232892,\n",
       " 0.00077549996632031437,\n",
       " -0.0015370690337966481,\n",
       " 0.0021658411701859503,\n",
       " -0.0019225014639125543,\n",
       " 0.0021594248978880348,\n",
       " -0.00063533497592796473,\n",
       " 0.00097537855388360462,\n",
       " 0.0028336438690823307,\n",
       " 0.00046579675462743442,\n",
       " 0.00078256427994123634,\n",
       " -0.00031853504854800763,\n",
       " 0.00039049558455198086,\n",
       " -0.00047008387010490094,\n",
       " 0.00042480245637448177,\n",
       " 0.00065184515503184659,\n",
       " 0.00067204727481916249,\n",
       " 0.00045892685340342965,\n",
       " 0.00022495797624925984,\n",
       " 0.00023871483241633129,\n",
       " -0.0010552499311723834,\n",
       " -7.7624644549925681e-05,\n",
       " -0.00069037263282761829,\n",
       " -0.00049083000352916073,\n",
       " -0.00088312551338148859,\n",
       " 0.0007893028675287489,\n",
       " 0.0011540676113279413,\n",
       " -0.00020168703972314739,\n",
       " 0.0010028291674573225,\n",
       " -0.0013922929072653217,\n",
       " 0.00021821938866174728,\n",
       " 0.001883615858017218,\n",
       " -0.0010480935274687214,\n",
       " 0.00011493041286322891,\n",
       " 0.00014934301774370551,\n",
       " 0.0012229797950559229,\n",
       " -0.0021633845819602066,\n",
       " 0.00095459831302129039,\n",
       " 0.00036956868170603951,\n",
       " 0.00025258083238514917,\n",
       " 0.00015625043714959097,\n",
       " 0.00052105610987442097,\n",
       " -0.00057348881119279049,\n",
       " -0.0016400800339750193,\n",
       " -0.00019484442444957228,\n",
       " -0.0020599301850871277,\n",
       " 0.00070666452432795179,\n",
       " -0.0005045356985391436,\n",
       " 0.00039717277875098156,\n",
       " 7.3634263783525777e-05,\n",
       " 0.0013813217761012159,\n",
       " -0.00064252207632588265,\n",
       " 0.00064476719843568897,\n",
       " 0.0010230636893107797,\n",
       " -0.00073854891536277723,\n",
       " -0.00071109830178905931,\n",
       " -0.00013990908986406814,\n",
       " 0.00030745306821024221,\n",
       " -0.0038905283044805006,\n",
       " 0.0022899137975905814,\n",
       " -0.00031181692542332773,\n",
       " -0.00018770507446494444,\n",
       " 0.00021138700561958157,\n",
       " 0.00017017271439119519,\n",
       " -0.0008623350402877511,\n",
       " 0.00099579213978685799,\n",
       " -0.00064905260854127012,\n",
       " 0.0010921447048571481,\n",
       " 0.0010303786926013914,\n",
       " 0.0020078112721988589,\n",
       " 0.0003830884911786292,\n",
       " -0.0023079680014665288,\n",
       " 0.0014707447519309025,\n",
       " 0.00024588828983899969,\n",
       " 0.0022142238027213212,\n",
       " -0.00040139679546806473,\n",
       " -0.0017296564932759578,\n",
       " -9.8645342850520179e-05,\n",
       " 0.00085131445333848366,\n",
       " 0.0010030440443170585,\n",
       " -0.0006421878234329631,\n",
       " 0.00099577508606782383,\n",
       " -0.00038066259964712745,\n",
       " 0.0017939421925178556,\n",
       " -0.0017090911292733657,\n",
       " 0.0011061351969748612,\n",
       " 0.00088585325573975471,\n",
       " -0.0006832929708595753,\n",
       " 0.0012987567638922254,\n",
       " -0.00232148440019532,\n",
       " -0.0016402625087686001,\n",
       " 0.0021937778147519266,\n",
       " -0.00086945904192525791,\n",
       " 0.00056921192794673336,\n",
       " 0.0013058091399098248,\n",
       " 0.00024576550306200373,\n",
       " -0.0010206872535644673,\n",
       " -0.00011228793910010582,\n",
       " -0.00064906454614457876,\n",
       " -0.0017986147220453164,\n",
       " 0.0023792992552384151,\n",
       " -0.00075922001242330339,\n",
       " -0.0014954676231463543,\n",
       " 0.0010715912784578924,\n",
       " 0.0015120784191924724,\n",
       " -0.0016540330079108795,\n",
       " -0.0017640622766688235,\n",
       " -0.0010827619381345854,\n",
       " -0.00040115974877358296,\n",
       " 0.00049340255704430347,\n",
       " -0.00072470167448487888,\n",
       " -0.00071808416830244914,\n",
       " 0.0015738529583077393,\n",
       " -0.00071783518400465873,\n",
       " -0.0017159081639684104,\n",
       " -0.0016539511500595627,\n",
       " 0.00064501447736158013,\n",
       " 0.00023913264853246685,\n",
       " -0.00095868589998615583,\n",
       " -0.00020874623722835772,\n",
       " 0.00015628113384383302,\n",
       " -0.00011910156305132424,\n",
       " -0.00069045790142274743,\n",
       " 0.0003349139140153834,\n",
       " 0.0011613962575937609,\n",
       " -0.00085549754112988763,\n",
       " 0.00047975996079471783,\n",
       " 0.0016152241437511761,\n",
       " 0.0017734228735566404,\n",
       " 9.4274664149809895e-05,\n",
       " -0.001991415353012449,\n",
       " -0.0018327493513056597,\n",
       " -0.00015377850057671227,\n",
       " -0.00033934428073266465,\n",
       " 0.00089274362142660602,\n",
       " -3.6427407040573456e-05,\n",
       " 4.6828565017642543e-06,\n",
       " -0.00028459142094076906,\n",
       " -0.0016400834447188317,\n",
       " -0.00049762998450517126,\n",
       " -0.00032544076258197996,\n",
       " 0.0002318176452418691,\n",
       " 0.0012711611937067796,\n",
       " 0.00021810512874427535,\n",
       " -0.00027067084907103633,\n",
       " -0.0006558696432363148,\n",
       " 0.00025233866957496964,\n",
       " -0.0014060582902918894,\n",
       " 0.00019047545112077513,\n",
       " -0.001764295912619479,\n",
       " 0.00064482006496469069,\n",
       " -0.00090396373688848841,\n",
       " -0.00039433930333475364,\n",
       " 0.0019320428302220666,\n",
       " 0.0010029263736557742,\n",
       " 0.00031463675786433387,\n",
       " 0.00096146650887340979,\n",
       " 0.0010855902974351156,\n",
       " -0.00017418185424854238,\n",
       " 0.0026616217736055992,\n",
       " -0.00018815529264723518,\n",
       " -0.0014130288084581583,\n",
       " 0.00021785443907458568,\n",
       " -0.0013374428412749606,\n",
       " 0.00082389624183092092,\n",
       " -0.00066273954446031957,\n",
       " 0.00023871312704441816,\n",
       " 0.00063101204764054453,\n",
       " -0.00055956141783547464,\n",
       " -0.00056646542649754772,\n",
       " -0.00029105714902384616,\n",
       " 0.0019869611110885227,\n",
       " -0.0012136686539532815,\n",
       " 0.00065898279964457518,\n",
       " 0.00081010016211009728,\n",
       " 0.0020351084023013249,\n",
       " 0.00017687889991255257,\n",
       " 0.0004864473872251418,\n",
       " 0.0017804292045328768,\n",
       " -0.0020188403860076226,\n",
       " -0.0013717906420231268,\n",
       " 5.2959755979159473e-05,\n",
       " 0.00032814974584931267,\n",
       " -0.0014750489211274032,\n",
       " 0.00041085630392623251,\n",
       " -4.3558230165691159e-05,\n",
       " 0.0021246285843294632,\n",
       " 0.00046611224842943444,\n",
       " 0.00063126444268213344,\n",
       " -0.00062860832594378857,\n",
       " 0.0012503997119353988,\n",
       " 0.0013536682232710845,\n",
       " -0.00088980952906811406,\n",
       " -0.00084865151511251402,\n",
       " -0.0018260465765281009,\n",
       " -0.0010757811877369072,\n",
       " 3.9046005597065392e-05,\n",
       " -0.00070423181130883916,\n",
       " 0.00087170757477890437,\n",
       " 0.00087884863013543146,\n",
       " 0.00094773523328488263,\n",
       " -0.0016816319888401376,\n",
       " -0.0014404947703789972,\n",
       " 0.0016774079721230545,\n",
       " 0.0024066321981508487,\n",
       " -0.00034612720798964103,\n",
       " -0.0011445654685721951,\n",
       " 0.0015944575458640836,\n",
       " -0.00011233057339765651,\n",
       " 0.00042439657786165486,\n",
       " -0.00016025957700692428,\n",
       " -0.0016952012540979167,\n",
       " -0.0013100212190235855,\n",
       " -2.2810391369518235e-05,\n",
       " -0.00061460930696655147,\n",
       " 0.00026638373359358369,\n",
       " 0.0010717703425076608,\n",
       " 0.00052094867144455992,\n",
       " -0.0011516178445897945,\n",
       " -0.0017915725782591263,\n",
       " 0.0018423879238136237,\n",
       " -0.0026035135945952498,\n",
       " 0.0002045171043955768,\n",
       " 0.00045209788110507632,\n",
       " -0.0031130749293885734,\n",
       " 0.00063115870962417164,\n",
       " 0.00086504913967080932,\n",
       " 0.0011956178608211326,\n",
       " 0.00087896630079671578,\n",
       " 0.0013056965853642521,\n",
       " 0.00053468847389258334,\n",
       " -0.0023493477137694757,\n",
       " 0.0021314626727435282,\n",
       " -0.0014681619661843642,\n",
       " -0.00042162620120582417,\n",
       " 0.00033532831938772045,\n",
       " 0.0012160024554020432,\n",
       " 0.0015601847814796233,\n",
       " -0.00042883035532274849,\n",
       " -0.0004630246725996906,\n",
       " 0.001140472765491618,\n",
       " -0.0010412355638480253,\n",
       " 0.0014294332545040783,\n",
       " 0.0022622994683142161,\n",
       " -0.00040143260827800453,\n",
       " 0.00060352221051306054,\n",
       " -0.00048386801222240206,\n",
       " -0.003030411005609232,\n",
       " -0.0029134879604206521,\n",
       " 0.0014296054970662359,\n",
       " 0.001188388126125664,\n",
       " -0.00175027472380751,\n",
       " 0.00082379391951674374,\n",
       " 0.00068608040123445402,\n",
       " -0.00096571951691282187,\n",
       " 0.00081707238564826534,\n",
       " -0.00031196529277886798,\n",
       " 0.00073455000773683943,\n",
       " 0.00052086851896514241,\n",
       " -0.00051824480429292497,\n",
       " 0.0020417924179879643,\n",
       " 0.00099587911375388638,\n",
       " 0.00046588202322256356,\n",
       " 0.0023719160370717085,\n",
       " -0.0018052970323600287,\n",
       " -0.00036029676415715028,\n",
       " -0.00040813367768366415,\n",
       " 0.0012644481866978113,\n",
       " -0.0003875785459124953,\n",
       " 0.0007410413163984747,\n",
       " -0.00093143140418124049,\n",
       " 0.00073453295401781915,\n",
       " -0.00087609701257052031,\n",
       " 0.0022345811113517744,\n",
       " -0.0024178147954308782,\n",
       " -0.00049061853741323713,\n",
       " 0.00061024886049726446,\n",
       " -0.0021354547588818412,\n",
       " -0.002500826615078347,\n",
       " -0.0016884336751880474,\n",
       " 0.00022523936261320543,\n",
       " 0.00032135317561711452,\n",
       " 0.0013470404848572454,\n",
       " 0.0018906255997372529,\n",
       " 3.2080603546494335e-05,\n",
       " 0.00031438436282275883,\n",
       " 0.001223019018609689,\n",
       " -0.0015025541066020082,\n",
       " 0.0010095046562844379,\n",
       " -0.0012339031758067526,\n",
       " 0.00054183635073673519,\n",
       " -0.00016042670345338406,\n",
       " 0.0010854572784267102,\n",
       " -0.00066976463452747548,\n",
       " -0.00098637356025432776,\n",
       " 0.0011199210444642616,\n",
       " 0.0021315496467105566,\n",
       " 0.0016013905458484856,\n",
       " 0.00066542976863673275,\n",
       " -0.00010509060647076462,\n",
       " 0.0013125409060097265,\n",
       " -0.00073156134347747426,\n",
       " 0.002097084175301106,\n",
       " 0.0011401180481358797,\n",
       " -0.00024992301027489117,\n",
       " 0.0013745098575218828,\n",
       " 0.00077553748450218118,\n",
       " -0.0002772781230220428,\n",
       " 0.0010510293251991126,\n",
       " 0.00080296763361306644,\n",
       " 0.0014292558958261953,\n",
       " -0.0014889729037409205,\n",
       " -0.0025763290190383287,\n",
       " -0.0017365963147479707,\n",
       " 0.0019385818892969642,\n",
       " 0.0032192434256446967,\n",
       " 0.0010647145557462628,\n",
       " 0.0011198477134724549,\n",
       " -0.00086942663985911661,\n",
       " -0.00021533816283224316,\n",
       " -7.1087290846927331e-05,\n",
       " 0.00010788997444895199,\n",
       " 0.0028608864272839096,\n",
       " 0.000445429213765558,\n",
       " -0.00029105032753623528,\n",
       " -0.0027893931631812752,\n",
       " -0.00080759411809916415,\n",
       " -0.0015715822556193748,\n",
       " -0.0012204311167474391,\n",
       " 0.00026612963318008165,\n",
       " 0.0018770529237356892,\n",
       " 0.002661512629803825,\n",
       " 0.00092003222466957579,\n",
       " 0.00024570922578923127,\n",
       " -0.0001193624849524233,\n",
       " 0.0010234184066665458,\n",
       " 0.00039714378742863876,\n",
       " -0.00070413289973848825,\n",
       " -0.00037397517321667573,\n",
       " -0.0011926513663964994,\n",
       " -1.603258022825349e-05,\n",
       " -0.00062862196891899647,\n",
       " -0.0013165022954537975,\n",
       " 0.00056926820521953359,\n",
       " 0.0013121469651002082,\n",
       " 0.0019664366760116098,\n",
       " 0.00059692005267776571,\n",
       " 0.0011337239456726961,\n",
       " -0.000469816126716191,\n",
       " 0.00045190858482387075,\n",
       " -0.00057310510251470936,\n",
       " -0.00011927892172919341,\n",
       " 0.0019936093139651945,\n",
       " -0.00058024956861504884,\n",
       " -0.0026107723206130612,\n",
       " -0.0010687850889920941,\n",
       " 0.0008169069645737187,\n",
       " 0.000651717252139139,\n",
       " 0.0016565850969631896,\n",
       " -0.0027209823587926729,\n",
       " -0.00036689380587673348,\n",
       " 0.00072043502347023114,\n",
       " -0.0011242371512640847,\n",
       " -0.00080728203504097651,\n",
       " -0.00041483304171742452,\n",
       " 0.00014924751691716698,\n",
       " -0.0023423925439503279,\n",
       " -0.003133972840912172,\n",
       " 0.00050706732312860714,\n",
       " 0.00075477941347459887,\n",
       " 0.00025965196749366815,\n",
       " -4.3524122727622849e-05,\n",
       " 0.00038336646679874853,\n",
       " 0.00075521257793786933,\n",
       " -0.0015850185018687346,\n",
       " -0.0016882563165101783,\n",
       " 0.00096151084854287361,\n",
       " 0.0026476312814878722,\n",
       " 0.0011747574674794148,\n",
       " 3.928134691963403e-05,\n",
       " 0.0036458679771542712,\n",
       " -0.00012627160973020801,\n",
       " 0.0012368577326280772,\n",
       " -0.0014060173613662241,\n",
       " 0.00169104204151313,\n",
       " 0.00095448916921951621,\n",
       " -0.00037381827900163922,\n",
       " -0.0027828012375774036,\n",
       " -0.0014818147946653454,\n",
       " -0.00022932183346234536,\n",
       " -0.00034616472617150784,\n",
       " 0.0017256746393690542,\n",
       " -0.0012270861411117079,\n",
       " 0.0004449141914509569,\n",
       " 0.00086491953140620248,\n",
       " 0.0028538596318448683,\n",
       " -0.00051852960140065518,\n",
       " -8.4692368914660032e-05,\n",
       " 0.00048655823639881524,\n",
       " 0.00029389232981198721,\n",
       " 0.00019763356019633638,\n",
       " 0.0013126483444395876,\n",
       " 0.0022624410141821316,\n",
       " -0.0010688191964301347,\n",
       " 0.00030758949796244606,\n",
       " -0.00073859154966032792,\n",
       " -0.0003391345199886403,\n",
       " 0.001656711294483984,\n",
       " -0.0015987821795234031,\n",
       " -0.0013099751739822085,\n",
       " 0.0019250791335433948,\n",
       " -0.00020856376243477692,\n",
       " -0.0012960597182282013,\n",
       " -0.00011910667916703588,\n",
       " -0.00054586936580071355,\n",
       " 0.00081008822450677476,\n",
       " -0.0035192603144163781,\n",
       " 0.0014157070950312628,\n",
       " -0.0014201613369551891,\n",
       " 0.0018012435528332316,\n",
       " 0.0012160143930053657,\n",
       " 0.0011748563790497657,\n",
       " 0.00030065138186233242,\n",
       " -0.00027042868626087069,\n",
       " 0.00087886909459826412,\n",
       " 0.00070667475655936118,\n",
       " -0.0010756601063318105,\n",
       " -0.00061472356688400953,\n",
       " 0.00093369699075389334,\n",
       " -0.0021839107224090187,\n",
       " -0.0010619970456194061,\n",
       " 0.00036251630568842619,\n",
       " -0.00010515711597497424,\n",
       " 0.00053474304579347043,\n",
       " -0.0019844192542676498,\n",
       " 0.0024620672067038191,\n",
       " -0.00027754245566696811,\n",
       " 0.0022210169622097209,\n",
       " 0.00010801958271355883,\n",
       " -0.00016733753360305414,\n",
       " 0.0017459125719663376,\n",
       " -2.9535335981809019e-05,\n",
       " -0.0001055169494464242,\n",
       " -0.00053203235715423847,\n",
       " -0.00066289302793155758,\n",
       " 0.00047955020005067961,\n",
       " -0.0012822551116478675,\n",
       " 0.0021796082585844173,\n",
       " 3.9103988241764864e-05,\n",
       " 0.00086512758677832757,\n",
       " 0.0020559636795273589,\n",
       " -0.0019361150688397971,\n",
       " 0.00049349294175513037,\n",
       " 0.00036933504575537013,\n",
       " -0.00010531230481811149,\n",
       " -0.00011894637420818699,\n",
       " 0.00049311605456466012,\n",
       " 0.00060400312538960721,\n",
       " 0.00072755902510773796,\n",
       " 0.00029409526906840067,\n",
       " 0.0011265163808119594,\n",
       " -0.001027732808094442,\n",
       " 0.002400268792381921,\n",
       " -0.00095871659668041176,\n",
       " 0.0005829432035352744,\n",
       " 0.00122334986075881,\n",
       " -0.00018119841745617427,\n",
       " -0.00027727641765015743,\n",
       " 0.0013814889025476756,\n",
       " -2.3507604249017877e-06,\n",
       " -0.00029120210563557403,\n",
       " -0.0002569344573668253,\n",
       " 0.00057567424528603972,\n",
       " 0.0010094125662016978,\n",
       " 0.00030077757938312688,\n",
       " -0.00056651999839843481,\n",
       " 0.0019798882706081045,\n",
       " 0.00019064939905483191,\n",
       " -0.00062845313710063744,\n",
       " 0.0017874031334429441,\n",
       " 0.0014778329407584695,\n",
       " 0.00072056804247863648,\n",
       " -0.0032920624269158621,\n",
       " -0.00013983575887226152,\n",
       " -0.0016538590599768088,\n",
       " -0.0016401670079420616,\n",
       " 0.0026199350943601762,\n",
       " 0.0010511384690008729,\n",
       " 0.0020833563104563912,\n",
       " -0.00034616813691530635,\n",
       " -0.00058717233636804145,\n",
       " 0.0013813200707293166,\n",
       " -0.00020161541410324002,\n",
       " 0.00098206768568594172,\n",
       " 0.0002042323072878327,\n",
       " 0.0002528298166829257,\n",
       " -0.00096565300740861226,\n",
       " 0.00041087506301716592,\n",
       " 0.00094060952627547656,\n",
       " 0.0016016190656834434,\n",
       " -0.0016747654983599314,\n",
       " 0.00018371469369851678,\n",
       " -0.00031189025641514823,\n",
       " 0.0015189414989288941,\n",
       " -0.0003390833588315656,\n",
       " 0.00052101688632066878,\n",
       " 0.00081014109103576259,\n",
       " -0.00053196243690623035,\n",
       " 0.0011748785488844976,\n",
       " 0.00045896607695716796,\n",
       " -0.00043597141067928946,\n",
       " 0.00047993049798497611,\n",
       " -0.0023009053932175061,\n",
       " 0.00023887513737518018,\n",
       " -0.00080029616852758667,\n",
       " -0.00018101423729070809,\n",
       " -0.00058018646985463773,\n",
       " 0.00021816993287657183,\n",
       " -0.0022321808003988169,\n",
       " -0.0010689760906451851,\n",
       " 0.00028721001949726099,\n",
       " 0.00059004332996613618,\n",
       " -0.000793641144163304,\n",
       " -0.0002224587537259376,\n",
       " -0.0001397914192027977,\n",
       " -1.5926847170291691e-05,\n",
       " -0.00069041015100947123,\n",
       " -0.0012135185812258559,\n",
       " 0.0015257721765991328,\n",
       " 0.00028694739222426269,\n",
       " -2.9502933915653839e-05,\n",
       " -0.0013924583283398684,\n",
       " -2.9663238874502729e-05,\n",
       " -0.00095862962271336949,\n",
       " -0.00080764186851244035,\n",
       " 0.00030051836285392708,\n",
       " -0.0014818915364009783,\n",
       " -0.0016883347636176965,\n",
       " -0.00019481713849912874,\n",
       " 0.00061049613942314174,\n",
       " -0.00057345470375474994,\n",
       " -0.00067649298988356477,\n",
       " 0.0023585343627232358,\n",
       " 0.00063111948607040558,\n",
       " 0.0019663599342759908,\n",
       " 0.00089264641522815436,\n",
       " 0.00015617710615777047,\n",
       " -0.00040843552851042852,\n",
       " 0.00013539004380784536,\n",
       " 0.0013538762786432096,\n",
       " 0.00066539566119867832,\n",
       " -0.002266489377593231,\n",
       " 0.0010645968850849785,\n",
       " -0.00091741533148498311,\n",
       " 0.00022481643038133048,\n",
       " 0.00041097738533131534,\n",
       " -0.00060085756691519165,\n",
       " 0.00029382752567969073,\n",
       " -0.00362244526252882,\n",
       " 0.0019043517592100684,\n",
       " 0.00025253478734377222,\n",
       " 0.00086525890041483366,\n",
       " -6.388143135808988e-05,\n",
       " 0.0023038002620222459,\n",
       " 0.0031981715661870413,\n",
       " 0.0011130784291907003,\n",
       " -0.0014886829905174648,\n",
       " 0.00044518193483968072,\n",
       " 0.0022140464440434382,\n",
       " -0.0012273231878061758,\n",
       " 6.6517083633629848e-05,\n",
       " 0.00016284065738157716,\n",
       " 0.0015669029046043031,\n",
       " -0.0019565934588753331,\n",
       " -0.00024270350781081806,\n",
       " -0.0022392280603007048,\n",
       " -0.0013371938569771702,\n",
       " 0.0010301160653283931,\n",
       " -0.00021530235002228948,\n",
       " 0.0019042886604496712,\n",
       " 0.00085133662317321557,\n",
       " 0.001732285324063873,\n",
       " -0.00044985958048285313,\n",
       " -5.0452006596327093e-05,\n",
       " -0.00050459368118382919,\n",
       " -0.00073848411123046687,\n",
       " -0.0021702340187213787,\n",
       " -0.001068945393950943,\n",
       " -0.0016538795244396554,\n",
       " -0.0010964778653759777,\n",
       " 0.0014157872475106803,\n",
       " 0.0021175727975680514,\n",
       " 0.0026201977216331745,\n",
       " -0.0023217214468897879,\n",
       " 0.00089261401316199918,\n",
       " -2.0864277800042297e-06,\n",
       " -0.0011930470126779169,\n",
       " 0.00039041202132875097,\n",
       " -0.0024109585371820813,\n",
       " -0.00028443282135381942,\n",
       " -0.0024521574800633467,\n",
       " 0.001904095953424681,\n",
       " 0.00014936007146273966,\n",
       " -0.0021563236790830831,\n",
       " -0.00024981045572931848,\n",
       " -0.0032575474997212361,\n",
       " -0.00077283361735054623,\n",
       " -0.0026587132618256376,\n",
       " -0.00040801600702239371,\n",
       " -0.00061445411812340034,\n",
       " -0.0011308546574465006,\n",
       " -0.00014662550761684878,\n",
       " 0.00019707590358417071,\n",
       " 0.00033512708550320625,\n",
       " 0.00057591299735239299,\n",
       " 0.00084430812236226116,\n",
       " -0.0013509967581855908,\n",
       " -0.0017915930427219728,\n",
       " -0.00067661407128866147,\n",
       " 0.0015189704902512369,\n",
       " -0.00086241007665147085,\n",
       " -0.0009378579087105654,\n",
       " -0.00059420595329472137,\n",
       " 0.001491274303123527,\n",
       " -0.001571329860577772,\n",
       " -0.00089678857409389301,\n",
       " -0.00027761919740258711,\n",
       " -0.00064912593953307673,\n",
       " 0.0011130733130749887,\n",
       " -0.00032534867249923993,\n",
       " 0.00013534058802266991,\n",
       " 0.0009751739092552919,\n",
       " 0.0017323194315019136,\n",
       " 0.0011403772646650795,\n",
       " -0.0010275366903256394,\n",
       " 0.0014707771539970577,\n",
       " -0.0010069627994635511,\n",
       " 0.00046614635586747499,\n",
       " -0.00092438243890745342,\n",
       " -0.00022918881445394002,\n",
       " -0.00058731217686405768,\n",
       " 0.00085135026614845122,\n",
       " -0.00031857427210177369,\n",
       " -0.0024250581731015686,\n",
       " 0.0018904533571750953,\n",
       " -2.2725122774389095e-05,\n",
       " 0.00091321860071835737,\n",
       " -0.0013994714808037156,\n",
       " -0.00071131317864878141,\n",
       " -0.0011584144148219927,\n",
       " -0.0010964608116569574,\n",
       " 0.00074804764737471108,\n",
       " -0.00087625390678557069,\n",
       " -3.6444460759579855e-05,\n",
       " 0.00095472792128589723,\n",
       " -0.00029832099115736921,\n",
       " -0.00078680876112113829,\n",
       " 0.0046021741253931686,\n",
       " -0.00040108300703796396,\n",
       " 0.00045207059515461889,\n",
       " -0.0028656288770593924,\n",
       " 0.001511960748531202,\n",
       " 0.0010578105470841898,\n",
       " 0.0018974136431099409,\n",
       " 0.00065180081536236889,\n",
       " 0.00050727878924454461,\n",
       " 0.00069311572353300543,\n",
       " -0.0027136673555020752,\n",
       " -0.00081441115279419496,\n",
       " 0.0013607871087928936,\n",
       " 0.00086501673760465414,\n",
       " -0.0010138719242413219,\n",
       " 0.00054846238377867507,\n",
       " 0.0013536767501305946,\n",
       " 0.0016632588804183918,\n",
       " -0.00021576280043598961,\n",
       " -0.00076619735207719697,\n",
       " 0.00033506910285850677,\n",
       " 0.00061731999560578343,\n",
       " 0.00035587492429936529,\n",
       " -0.00067676414401610097,\n",
       " 0.0012713641329631931,\n",
       " -0.0022251386566126269,\n",
       " 0.00088618409788887575,\n",
       " 0.00070693056234477636,\n",
       " -0.00039457805540112079,\n",
       " 0.0010787408606739296,\n",
       " 0.0010027745955564354,\n",
       " 0.0008168199906066903,\n",
       " 0.0020485804613606384,\n",
       " -0.00053209033979892406,\n",
       " 6.006499852574676e-05,\n",
       " 0.0015189500257884042,\n",
       " -0.0013717650614445825,\n",
       " 0.0017391449930564684,\n",
       " -0.0023558509600344335,\n",
       " -0.0015094717582392891,\n",
       " -0.00049772548533170979,\n",
       " 0.0029917829108711963,\n",
       " -0.001585052609306789,\n",
       " 0.00053468335777688558,\n",
       " 0.001787486696666174,\n",
       " -0.00027726789079064729,\n",
       " 0.00069325215328522316,\n",
       " 0.0006998116768229673,\n",
       " 0.00038349095894764373,\n",
       " -0.0027897700503717732,\n",
       " 0.0011334562022839861,\n",
       " 0.0021798077870970323,\n",
       " 0.00049324566282923921,\n",
       " 0.00054187045817478963,\n",
       " 3.2348346935218153e-05,\n",
       " 0.00087217825742402777,\n",
       " 0.00069295371320227117,\n",
       " -0.0016125407410623738,\n",
       " 0.0011608044935435546,\n",
       " -0.0001743029356536252,\n",
       " 0.00085122918474335452,\n",
       " 0.00042472400926696352,\n",
       " 0.0016153810379662126,\n",
       " 4.5885210126841991e-05,\n",
       " -0.0020738661053039675,\n",
       " -0.0013788498395283511,\n",
       " -0.0017499950428154776,\n",
       " -0.0014750489211274032,\n",
       " -0.00057348540044900587,\n",
       " -0.00089721491706955259,\n",
       " -0.0022115830343300835,\n",
       " 0.00050050268347516524,\n",
       " -0.0014130765588714345,\n",
       " 0.0013122782787367143,\n",
       " -0.00080745939371885955,\n",
       " 0.00076211147048424466,\n",
       " 0.0011951778748702513,\n",
       " 0.00032827082725440937,\n",
       " 0.0021453832446132332,\n",
       " 0.0013676501885292874,\n",
       " 0.0018839364679349296,\n",
       " 0.00056911642712019483,\n",
       " 1.1680660618476613e-05,\n",
       " -0.0024181763342742274,\n",
       " -0.0013373660995393416,\n",
       " -0.00031861861177123751,\n",
       " -0.00070410049767233307,\n",
       " -0.0011309313991821196,\n",
       " 0.0023103086244029153,\n",
       " 0.0015533813897598142,\n",
       " -0.00077295640412754218,\n",
       " -0.00074545804014054806,\n",
       " -0.0015231723371335604,\n",
       " 0.00087198043428332594,\n",
       " -0.0014267293873524295,\n",
       " 0.0017602015041670166,\n",
       " -0.001736492287061922,\n",
       " 0.00076895067501400738,\n",
       " -0.0009658081962517634,\n",
       " -0.0012618329988851179,\n",
       " -0.0011719444565260057,\n",
       " -0.00049070380600836627,\n",
       " 0.00045921335588305912,\n",
       " -0.0002360672425374688,\n",
       " -0.0009313512517018091,\n",
       " 0.00088593170284728684,\n",
       " 0.00043853714270680744,\n",
       " -0.00084881693618707454,\n",
       " -0.00067675902790037545,\n",
       " 0.0014775873672044776,\n",
       " 0.00011477351864819241,\n",
       " -0.0020531250879954194,\n",
       " -0.0020116976252791963,\n",
       " 0.0018493226291699388,\n",
       " 0.00060354267497590708,\n",
       " -0.00022920586817296029,\n",
       " -0.0036913744999758497,\n",
       " 0.0011818899959764317,\n",
       " 0.0013879461037712426,\n",
       " -0.00041533783180060235,\n",
       " -0.0015642297341469102,\n",
       " -0.00093130179591663365,\n",
       " -0.00099284610982502963,\n",
       " 0.0011404932299544646,\n",
       " -0.0017847725972831158,\n",
       " 0.0012989648192643505,\n",
       " -0.00018791142446515641,\n",
       " 0.00056917099902108192,\n",
       " 0.0007966025224722395,\n",
       " -0.0021494367241400442,\n",
       " -0.00062143998463679018,\n",
       " 0.00099575973772070281,\n",
       " 0.00097532909809842916,\n",
       " -0.0016127999575915736,\n",
       " -0.0009933679536272555,\n",
       " -0.0015780940287438427,\n",
       " -0.00099322640775932614,\n",
       " -0.00076577783058914828,\n",
       " -0.00062130867100029796,\n",
       " -0.001068884000562445,\n",
       " -0.00020850066367436582,\n",
       " 0.00039015109942763804,\n",
       " 0.00060342329894270963,\n",
       " -0.00054609618026375817,\n",
       " 0.00087185935287824312,\n",
       " 0.0003971932432138281,\n",
       " -0.00098654409744459992,\n",
       " 0.00080324731460509891,\n",
       " -0.00049087434319862455,\n",
       " -0.00055282453561987521,\n",
       " -0.00056646713186943309,\n",
       " 0.0016018168888241591,\n",
       " -0.00012592541923396594,\n",
       " 0.00016293274746430331,\n",
       " -0.00011906575024135668,\n",
       " 0.0017116858526232404,\n",
       " -0.0001260226254324176,\n",
       " 0.00050740669213723832,\n",
       " 0.00043133810470559475,\n",
       " 0.00035593802305976252,\n",
       " 0.00019758580978306017,\n",
       " 0.00010807244924253279,\n",
       " 0.00069295541857415655,\n",
       " -0.0004494349428790928,\n",
       " -0.0013163607495858681,\n",
       " -0.0014063328551682103,\n",
       " -0.00071060203856537774,\n",
       " 0.00010135262074593976,\n",
       " -0.00023600243840515844,\n",
       " -0.00021544048514639258,\n",
       " -0.00080060484084196193,\n",
       " 0.0013537432596347904,\n",
       " 6.6430109666587578e-05,\n",
       " -0.0001534323100804702,\n",
       " -0.00069039309729046483,\n",
       " 0.00078933185885109169,\n",
       " 0.0011884989752993513,\n",
       " -0.00019475403973873151,\n",
       " -0.00064202581310220108,\n",
       " -0.0012548573646031236,\n",
       " 0.00029402875956419106,\n",
       " -0.00095878481155650674,\n",
       " 0.0013470234311382112,\n",
       " -0.001509325096255662,\n",
       " -0.0013511894652105949,\n",
       " 0.0016014894574188504,\n",
       " 0.0010992277775689896,\n",
       " -0.0010207674060438848,\n",
       " -4.345079173581623e-05,\n",
       " 0.00017683626561500188,\n",
       " 0.0013880433099697082,\n",
       " -5.6972306580319043e-05,\n",
       " -0.001082697134002289,\n",
       " -0.00032543223572246982,\n",
       " -0.0014956449818242235,\n",
       " -0.0019365260634683357,\n",
       " -0.0024455689652032597,\n",
       " -0.0007179801406163866,\n",
       " 0.0039760853916926686,\n",
       " -0.0004009721578643044,\n",
       " 0.00042458416877094729,\n",
       " 0.00077565856590727789,\n",
       " 8.0567263767927777e-05,\n",
       " 0.00054147140114957359,\n",
       " -0.0010895329277882532,\n",
       " 0.00010806051163921027,\n",
       " 0.00025934158980739364,\n",
       " -0.00053211592037746835,\n",
       " 0.00027319224142910437,\n",
       " 0.00083072862487307275,\n",
       " -0.00015342037247714768,\n",
       " -0.00073160056703124032,\n",
       " 0.0013193903427709125,\n",
       " 0.00043145236462306669,\n",
       " -0.0013719168395439213,\n",
       " 0.00010117696743998372,\n",
       " 0.00024547900058237426,\n",
       " 0.0012090268211200628,\n",
       " -0.0008898658063409004,\n",
       " 0.0027511982329065238,\n",
       " -0.00069032658778624134,\n",
       " -0.00015354145388224438,\n",
       " -0.00097256554293019548,\n",
       " 0.0007688858708817109,\n",
       " 0.00052088898342796119,\n",
       " -0.0023696896740528078,\n",
       " 4.5123193115059745e-06,\n",
       " 0.0030195046785774365,\n",
       " -0.0013167683334706082,\n",
       " -0.00019519402568961286,\n",
       " 0.0011335568192262363,\n",
       " 0.00012872819795595181,\n",
       " 0.0022007312791991612,\n",
       " 0.00043835296254132738,\n",
       " 0.00019754829160120724,\n",
       " 0.00083082071495581278,\n",
       " 0.00081680123151574302,\n",
       " 0.0010507087152814149,\n",
       " -0.0014064829278956498,\n",
       " 0.00073449884657976472,\n",
       " 0.00091993672384302339,\n",
       " 0.00032116046859212433,\n",
       " 0.0002802360905871798,\n",
       " -0.00055967056163723494,\n",
       " 0.00010792919800271805,\n",
       " 0.00052791577886703023,\n",
       " -0.00069071711795196111,\n",
       " -0.00053906767945281764,\n",
       " 0.00094786654692138872,\n",
       " -0.00074565245253745138,\n",
       " -0.00086938741630536442,\n",
       " 6.705598115486211e-05,\n",
       " 0.00074798454861431385,\n",
       " -9.1336876818920443e-06,\n",
       " 0.00067913034753100399,\n",
       " -0.00064215712673872105,\n",
       " -0.0010826698480518454,\n",
       " -0.0030236587750464977,\n",
       " 0.0038521560155278523,\n",
       " 0.0013329510811691675,\n",
       " -0.0010826903125146781,\n",
       " -0.0012134077320521824,\n",
       " -0.0016333141604370632,\n",
       " 0.0014088644797576738,\n",
       " 0.00065144780337852981,\n",
       " 0.000631146772020863,\n",
       " 0.00058282723824588933,\n",
       " -0.0017087654032399702,\n",
       " 0.0024619580629020588,\n",
       " -0.0012068635568615732,\n",
       " 9.4310476959777456e-05,\n",
       " 0.0016014621714684069,\n",
       " ...]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_diffs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.90649999999999997"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Proportion of p_diffs greater than the actual difference\n",
    "# observedin original dataset\n",
    "(p_diffs > obs_diff).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "k. Please explain using the vocabulary you've learned in this course what you just computed in part **j.**  What is this value called in scientific studies?  What does this value mean in terms of whether or not there is a difference between the new and old pages?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<li>This is p-value, the probability of observing our statistics if the null hypothesis is true.</li>\n",
    "\n",
    "<li>A small p-value indicates strong evidence against the null hypothesis, and we reject the null hypothesis.</li>\n",
    "\n",
    "<li>A large p-value indicates weak evidence against the null hypothesus, so we fail to reject the null hypothesis.</li>\n",
    "\n",
    "<li> The p-value here is very big so we fail to reject the null hypothesis, which tell us that there is no conversion advantages with new pages.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "l. We could also use a built-in to achieve similar results.  Though using the built-in might be easier to code, the above portions are a walkthrough of the ideas that are critical to correctly thinking about statistical significance. Fill in the below to calculate the number of conversions for each page, as well as the number of individuals who received each page. Let `n_old` and `n_new` refer the the number of rows associated with the old page and new pages, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "convert_old = sum(df2.query(\"landing_page == 'old_page'\")[\"converted\"])\n",
    "convert_new = sum(df2.query(\"landing_page == 'new_page'\")[\"converted\"])\n",
    "n_old = df2.query(\"landing_page == 'old_page'\").shape[0]\n",
    "n_new = df2.query(\"landing_page == 'new_page'\").shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "m. Now use `stats.proportions_ztest` to compute your test statistic and p-value.  [Here](https://docs.w3cub.com/statsmodels/generated/statsmodels.stats.proportion.proportions_ztest/) is a helpful link on using the built in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.31092419842 0.189883374482\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "z_score, p_value = sm.stats.proportions_ztest([convert_old, convert_new], [n_old, n_new])\n",
    "print(z_score, p_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "n. What do the z-score and p-value you computed in the previous question mean for the conversion rates of the old and new pages?  Do they agree with the findings in parts **j.** and **k.**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **Z-value** is a test statistic for Z-tests that measures the difference between an observed statistic(the difference between conversion rates) and its hypothesized population parameter in units of the standard deviation,\n",
    "helps us decide whether or not to reject the null.\n",
    "which is 1.31 here, This is less than the critical 1.96, So we accept the null hypothesis.\n",
    "\n",
    "Also the large p-value tells us that there is no enough evidence to regect the null hypothesis.\n",
    "\n",
    "These values agree with the findings in parts j. and k."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='regression'></a>\n",
    "### Part III - A regression approach\n",
    "\n",
    "`1.` In this final part, you will see that the result you achieved in the A/B test in Part II above can also be achieved by performing regression.<br><br> \n",
    "\n",
    "a. Since each row is either a conversion or no conversion, what type of regression should you be performing in this case?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should use **Logistic Regression**, since it predicts a probability between categorical data\n",
    "\n",
    "And in this dataset we want to predict 1 of 2 possibles, whether a user will convert or not, depending on the page type, whether it is new or old page.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. The goal is to use **statsmodels** to fit the regression model you specified in part **a.** to see if there is a significant difference in conversion based on which page a customer receives. However, you first need to create in df2 a column for the intercept, and create a dummy variable column for which page each user received.  Add an **intercept** column, as well as an **ab_page** column, which is 1 when an individual receives the **treatment** and 0 if **control**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>Intercept</th>\n",
       "      <th>Control</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>old_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   Intercept  Control  ab_page  new_page  old_page  \n",
       "0          1        1        0         0         1  \n",
       "1          1        1        0         0         1  \n",
       "2          1        0        1         1         0  \n",
       "3          1        0        1         1         0  \n",
       "4          1        1        0         0         1  "
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[\"Intercept\"] = 1\n",
    "df2[['Control', 'ab_page']] = pd.get_dummies(df['group'])\n",
    "df2[['new_page', 'old_page']] = pd.get_dummies(df['landing_page'])\n",
    "df2.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>Intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>old_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>936923</td>\n",
       "      <td>2017-01-10 15:20:49.083499</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>679687</td>\n",
       "      <td>2017-01-19 03:26:46.940749</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>719014</td>\n",
       "      <td>2017-01-17 01:48:29.539573</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>817355</td>\n",
       "      <td>2017-01-04 17:58:08.979471</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>839785</td>\n",
       "      <td>2017-01-15 18:11:06.610965</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>929503</td>\n",
       "      <td>2017-01-18 05:37:11.527370</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>834487</td>\n",
       "      <td>2017-01-21 22:37:47.774891</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>803683</td>\n",
       "      <td>2017-01-09 06:05:16.222706</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>944475</td>\n",
       "      <td>2017-01-22 01:31:09.573836</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>718956</td>\n",
       "      <td>2017-01-22 11:45:11.327945</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>644214</td>\n",
       "      <td>2017-01-22 02:05:21.719434</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>847721</td>\n",
       "      <td>2017-01-17 14:01:00.090575</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>888545</td>\n",
       "      <td>2017-01-08 06:37:26.332945</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>650559</td>\n",
       "      <td>2017-01-24 11:55:51.084801</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>935734</td>\n",
       "      <td>2017-01-17 20:33:37.428378</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>740805</td>\n",
       "      <td>2017-01-12 18:59:45.453277</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>759875</td>\n",
       "      <td>2017-01-09 16:11:58.806110</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>793849</td>\n",
       "      <td>2017-01-23 22:36:10.742811</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>905617</td>\n",
       "      <td>2017-01-20 14:12:19.345499</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>746742</td>\n",
       "      <td>2017-01-23 11:38:29.592148</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>892356</td>\n",
       "      <td>2017-01-05 09:35:14.904865</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>773302</td>\n",
       "      <td>2017-01-12 08:29:49.810594</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>913579</td>\n",
       "      <td>2017-01-24 09:11:39.164256</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>736159</td>\n",
       "      <td>2017-01-06 01:50:21.318242</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>690284</td>\n",
       "      <td>2017-01-13 17:22:57.182769</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294448</th>\n",
       "      <td>776137</td>\n",
       "      <td>2017-01-12 05:53:12.386730</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294449</th>\n",
       "      <td>883344</td>\n",
       "      <td>2017-01-22 23:15:58.645325</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294450</th>\n",
       "      <td>825594</td>\n",
       "      <td>2017-01-06 12:37:08.897784</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294451</th>\n",
       "      <td>875688</td>\n",
       "      <td>2017-01-14 07:19:49.042869</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294452</th>\n",
       "      <td>927527</td>\n",
       "      <td>2017-01-12 10:52:11.084740</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294453</th>\n",
       "      <td>789177</td>\n",
       "      <td>2017-01-17 18:17:56.215378</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294454</th>\n",
       "      <td>937338</td>\n",
       "      <td>2017-01-19 03:23:22.236666</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294455</th>\n",
       "      <td>733101</td>\n",
       "      <td>2017-01-23 12:52:58.711914</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294456</th>\n",
       "      <td>679096</td>\n",
       "      <td>2017-01-02 16:43:49.237940</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294457</th>\n",
       "      <td>691699</td>\n",
       "      <td>2017-01-09 23:42:35.963486</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294458</th>\n",
       "      <td>807595</td>\n",
       "      <td>2017-01-22 10:43:09.285426</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294459</th>\n",
       "      <td>924816</td>\n",
       "      <td>2017-01-20 10:59:03.481635</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294460</th>\n",
       "      <td>846225</td>\n",
       "      <td>2017-01-16 15:24:46.705903</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294461</th>\n",
       "      <td>740310</td>\n",
       "      <td>2017-01-10 17:22:19.762612</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294462</th>\n",
       "      <td>677163</td>\n",
       "      <td>2017-01-03 19:41:51.902148</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294463</th>\n",
       "      <td>832080</td>\n",
       "      <td>2017-01-19 13:18:27.352570</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294464</th>\n",
       "      <td>834362</td>\n",
       "      <td>2017-01-17 01:51:56.106436</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294465</th>\n",
       "      <td>925675</td>\n",
       "      <td>2017-01-07 20:38:26.346410</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294466</th>\n",
       "      <td>923948</td>\n",
       "      <td>2017-01-09 16:33:41.104573</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294467</th>\n",
       "      <td>857744</td>\n",
       "      <td>2017-01-05 08:00:56.024226</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294468</th>\n",
       "      <td>643562</td>\n",
       "      <td>2017-01-02 19:20:05.460595</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294469</th>\n",
       "      <td>755438</td>\n",
       "      <td>2017-01-18 17:35:06.149568</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294470</th>\n",
       "      <td>908354</td>\n",
       "      <td>2017-01-11 02:42:21.195145</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294471</th>\n",
       "      <td>718310</td>\n",
       "      <td>2017-01-21 22:44:20.378320</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294472</th>\n",
       "      <td>822004</td>\n",
       "      <td>2017-01-04 03:36:46.071379</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294473</th>\n",
       "      <td>751197</td>\n",
       "      <td>2017-01-03 22:28:38.630509</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294474</th>\n",
       "      <td>945152</td>\n",
       "      <td>2017-01-12 00:51:57.078372</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294475</th>\n",
       "      <td>734608</td>\n",
       "      <td>2017-01-22 11:45:03.439544</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294476</th>\n",
       "      <td>697314</td>\n",
       "      <td>2017-01-15 01:20:28.957438</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294477</th>\n",
       "      <td>715931</td>\n",
       "      <td>2017-01-16 12:40:24.467417</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>290584 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id                   timestamp      group landing_page  \\\n",
       "0        851104  2017-01-21 22:11:48.556739    control     old_page   \n",
       "1        804228  2017-01-12 08:01:45.159739    control     old_page   \n",
       "2        661590  2017-01-11 16:55:06.154213  treatment     new_page   \n",
       "3        853541  2017-01-08 18:28:03.143765  treatment     new_page   \n",
       "4        864975  2017-01-21 01:52:26.210827    control     old_page   \n",
       "5        936923  2017-01-10 15:20:49.083499    control     old_page   \n",
       "6        679687  2017-01-19 03:26:46.940749  treatment     new_page   \n",
       "7        719014  2017-01-17 01:48:29.539573    control     old_page   \n",
       "8        817355  2017-01-04 17:58:08.979471  treatment     new_page   \n",
       "9        839785  2017-01-15 18:11:06.610965  treatment     new_page   \n",
       "10       929503  2017-01-18 05:37:11.527370  treatment     new_page   \n",
       "11       834487  2017-01-21 22:37:47.774891  treatment     new_page   \n",
       "12       803683  2017-01-09 06:05:16.222706  treatment     new_page   \n",
       "13       944475  2017-01-22 01:31:09.573836  treatment     new_page   \n",
       "14       718956  2017-01-22 11:45:11.327945  treatment     new_page   \n",
       "15       644214  2017-01-22 02:05:21.719434    control     old_page   \n",
       "16       847721  2017-01-17 14:01:00.090575    control     old_page   \n",
       "17       888545  2017-01-08 06:37:26.332945  treatment     new_page   \n",
       "18       650559  2017-01-24 11:55:51.084801    control     old_page   \n",
       "19       935734  2017-01-17 20:33:37.428378    control     old_page   \n",
       "20       740805  2017-01-12 18:59:45.453277  treatment     new_page   \n",
       "21       759875  2017-01-09 16:11:58.806110  treatment     new_page   \n",
       "23       793849  2017-01-23 22:36:10.742811  treatment     new_page   \n",
       "24       905617  2017-01-20 14:12:19.345499  treatment     new_page   \n",
       "25       746742  2017-01-23 11:38:29.592148    control     old_page   \n",
       "26       892356  2017-01-05 09:35:14.904865  treatment     new_page   \n",
       "27       773302  2017-01-12 08:29:49.810594  treatment     new_page   \n",
       "28       913579  2017-01-24 09:11:39.164256    control     old_page   \n",
       "29       736159  2017-01-06 01:50:21.318242  treatment     new_page   \n",
       "30       690284  2017-01-13 17:22:57.182769    control     old_page   \n",
       "...         ...                         ...        ...          ...   \n",
       "294448   776137  2017-01-12 05:53:12.386730  treatment     new_page   \n",
       "294449   883344  2017-01-22 23:15:58.645325  treatment     new_page   \n",
       "294450   825594  2017-01-06 12:37:08.897784  treatment     new_page   \n",
       "294451   875688  2017-01-14 07:19:49.042869    control     old_page   \n",
       "294452   927527  2017-01-12 10:52:11.084740    control     old_page   \n",
       "294453   789177  2017-01-17 18:17:56.215378    control     old_page   \n",
       "294454   937338  2017-01-19 03:23:22.236666  treatment     new_page   \n",
       "294455   733101  2017-01-23 12:52:58.711914  treatment     new_page   \n",
       "294456   679096  2017-01-02 16:43:49.237940  treatment     new_page   \n",
       "294457   691699  2017-01-09 23:42:35.963486  treatment     new_page   \n",
       "294458   807595  2017-01-22 10:43:09.285426  treatment     new_page   \n",
       "294459   924816  2017-01-20 10:59:03.481635    control     old_page   \n",
       "294460   846225  2017-01-16 15:24:46.705903  treatment     new_page   \n",
       "294461   740310  2017-01-10 17:22:19.762612    control     old_page   \n",
       "294462   677163  2017-01-03 19:41:51.902148  treatment     new_page   \n",
       "294463   832080  2017-01-19 13:18:27.352570    control     old_page   \n",
       "294464   834362  2017-01-17 01:51:56.106436    control     old_page   \n",
       "294465   925675  2017-01-07 20:38:26.346410  treatment     new_page   \n",
       "294466   923948  2017-01-09 16:33:41.104573    control     old_page   \n",
       "294467   857744  2017-01-05 08:00:56.024226    control     old_page   \n",
       "294468   643562  2017-01-02 19:20:05.460595  treatment     new_page   \n",
       "294469   755438  2017-01-18 17:35:06.149568    control     old_page   \n",
       "294470   908354  2017-01-11 02:42:21.195145    control     old_page   \n",
       "294471   718310  2017-01-21 22:44:20.378320    control     old_page   \n",
       "294472   822004  2017-01-04 03:36:46.071379  treatment     new_page   \n",
       "294473   751197  2017-01-03 22:28:38.630509    control     old_page   \n",
       "294474   945152  2017-01-12 00:51:57.078372    control     old_page   \n",
       "294475   734608  2017-01-22 11:45:03.439544    control     old_page   \n",
       "294476   697314  2017-01-15 01:20:28.957438    control     old_page   \n",
       "294477   715931  2017-01-16 12:40:24.467417  treatment     new_page   \n",
       "\n",
       "        converted  Intercept  ab_page  new_page  old_page  \n",
       "0               0          1        0         0         1  \n",
       "1               0          1        0         0         1  \n",
       "2               0          1        1         1         0  \n",
       "3               0          1        1         1         0  \n",
       "4               1          1        0         0         1  \n",
       "5               0          1        0         0         1  \n",
       "6               1          1        1         1         0  \n",
       "7               0          1        0         0         1  \n",
       "8               1          1        1         1         0  \n",
       "9               1          1        1         1         0  \n",
       "10              0          1        1         1         0  \n",
       "11              0          1        1         1         0  \n",
       "12              0          1        1         1         0  \n",
       "13              0          1        1         1         0  \n",
       "14              0          1        1         1         0  \n",
       "15              1          1        0         0         1  \n",
       "16              0          1        0         0         1  \n",
       "17              1          1        1         1         0  \n",
       "18              0          1        0         0         1  \n",
       "19              0          1        0         0         1  \n",
       "20              0          1        1         1         0  \n",
       "21              0          1        1         1         0  \n",
       "23              0          1        1         1         0  \n",
       "24              0          1        1         1         0  \n",
       "25              0          1        0         0         1  \n",
       "26              1          1        1         1         0  \n",
       "27              0          1        1         1         0  \n",
       "28              1          1        0         0         1  \n",
       "29              0          1        1         1         0  \n",
       "30              0          1        0         0         1  \n",
       "...           ...        ...      ...       ...       ...  \n",
       "294448          0          1        1         1         0  \n",
       "294449          0          1        1         1         0  \n",
       "294450          0          1        1         1         0  \n",
       "294451          0          1        0         0         1  \n",
       "294452          0          1        0         0         1  \n",
       "294453          0          1        0         0         1  \n",
       "294454          0          1        1         1         0  \n",
       "294455          0          1        1         1         0  \n",
       "294456          0          1        1         1         0  \n",
       "294457          0          1        1         1         0  \n",
       "294458          0          1        1         1         0  \n",
       "294459          0          1        0         0         1  \n",
       "294460          0          1        1         1         0  \n",
       "294461          0          1        0         0         1  \n",
       "294462          0          1        1         1         0  \n",
       "294463          0          1        0         0         1  \n",
       "294464          0          1        0         0         1  \n",
       "294465          0          1        1         1         0  \n",
       "294466          0          1        0         0         1  \n",
       "294467          0          1        0         0         1  \n",
       "294468          0          1        1         1         0  \n",
       "294469          0          1        0         0         1  \n",
       "294470          0          1        0         0         1  \n",
       "294471          0          1        0         0         1  \n",
       "294472          0          1        1         1         0  \n",
       "294473          0          1        0         0         1  \n",
       "294474          0          1        0         0         1  \n",
       "294475          0          1        0         0         1  \n",
       "294476          0          1        0         0         1  \n",
       "294477          0          1        1         1         0  \n",
       "\n",
       "[290584 rows x 9 columns]"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# After creating ab_page, we donot need the Control column\n",
    "df2.drop(\"Control\",axis = 1,inplace = True)\n",
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Use **statsmodels** to instantiate your regression model on the two columns you created in part b., then fit the model using the two columns you created in part **b.** to predict whether or not an individual converts. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "regression_model = sm.Logit(df2['converted'], df2[['Intercept', 'ab_page']])\n",
    "fitting = regression_model.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Provide the summary of your model below, and use it as necessary to answer the following questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290582</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     1</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Wed, 11 Nov 2020</td> <th>  Pseudo R-squ.:     </th>  <td>8.077e-06</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>16:46:16</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.1899</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th> <td>   -1.9888</td> <td>    0.008</td> <td> -246.669</td> <td> 0.000</td> <td>   -2.005</td> <td>   -1.973</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0150</td> <td>    0.011</td> <td>   -1.311</td> <td> 0.190</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290582\n",
       "Method:                           MLE   Df Model:                            1\n",
       "Date:                Wed, 11 Nov 2020   Pseudo R-squ.:               8.077e-06\n",
       "Time:                        16:46:16   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                    0.1899\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "Intercept     -1.9888      0.008   -246.669      0.000      -2.005      -1.973\n",
       "ab_page       -0.0150      0.011     -1.311      0.190      -0.037       0.007\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# I get errors without the code belos\n",
    "from scipy import stats\n",
    "stats.chisqprob = lambda chisq, df: stats.chi2.sf(chisq, df)\n",
    "\n",
    "fitting.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. What is the p-value associated with **ab_page**? Why does it differ from the value you found in **Part II**?<br><br>  **Hint**: What are the null and alternative hypotheses associated with your regression model, and how do they compare to the null and alternative hypotheses in **Part II**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our Hypothesis is :\n",
    "<li>$$H_0: P_{new} = P_{old} $$ </li>\n",
    "<li>$$H_1: P_{new} != P_{old} $$ </li>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The p-value associated with ab_page is 0.190, which is differ from partII, as A/B Test the null and alternative hypothesis are different from part 3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. Now, you are considering other things that might influence whether or not an individual converts.  Discuss why it is a good idea to consider other factors to add into your regression model.  Are there any disadvantages to adding additional terms into your regression model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** So far, It seems that the features we used to make our decision doesnot affect on user convertion.\n",
    "Therefore, it maybe a good idea to add more factors.\n",
    "But one of the most disavantages of adding additional term into regression model is collineary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "g. Now along with testing if the conversion rate changes for different pages, also add an effect based on which country a user lives in. You will need to read in the **countries.csv** dataset and merge together your datasets on the appropriate rows.  [Here](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.join.html) are the docs for joining tables. \n",
    "\n",
    "Does it appear that country had an impact on conversion?  Don't forget to create dummy variables for these country columns - **Hint: You will need two columns for the three dummy variables.** Provide the statistical output as well as a written response to answer this question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>834778</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>928468</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>822059</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>711597</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>710616</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id country\n",
       "0   834778      UK\n",
       "1   928468      US\n",
       "2   822059      UK\n",
       "3   711597      UK\n",
       "4   710616      UK"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_df = pd.read_csv(\"countries.csv\")\n",
    "country_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>Intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>old_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   Intercept  ab_page  new_page  old_page  \n",
       "0          1        0         0         1  \n",
       "1          1        0         0         1  \n",
       "2          1        1         1         0  \n",
       "3          1        1         1         0  \n",
       "4          1        0         0         1  "
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>Intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>old_page</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>834778</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-14 23:08:43.304998</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>928468</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-23 14:44:16.387854</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>822059</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 14:04:14.719771</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>711597</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-22 03:14:24.763511</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>710616</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 13:14:44.000513</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>909908</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-06 20:44:26.334764</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>811617</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-02 18:42:11.851370</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>938122</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-10 09:32:08.222716</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>887018</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-06 11:09:40.487196</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>820683</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-14 11:52:06.521342</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>697357</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-15 05:21:57.086762</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>748296</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-10 04:05:24.283883</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>666132</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-19 22:45:47.593706</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>668810</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-13 16:59:49.226184</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>940939</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-08 10:10:27.267660</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>646414</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-07 10:06:42.693231</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>907385</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-17 13:29:52.718288</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>698200</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-21 21:50:42.718525</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>738692</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-08 07:03:08.917332</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>724651</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-07 21:32:07.568614</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>662682</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-08 22:48:46.350610</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>639818</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-21 21:46:38.906148</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>920941</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-21 12:26:25.304141</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>804632</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-03 08:37:34.233194</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>684798</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-19 16:36:41.202599</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>766270</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-22 10:11:29.823099</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>857817</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-18 22:21:01.405272</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>750698</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-15 23:31:02.321891</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>721445</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-03 02:19:44.388206</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>744732</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-22 05:00:19.003615</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>834931</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-06 20:20:20.082513</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>667920</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-09 03:00:15.337219</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>869193</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-22 21:18:40.153019</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>737522</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-14 17:43:30.555545</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>937048</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 05:32:01.893266</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>689274</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-09 11:39:14.209283</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>916713</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-19 07:43:21.591415</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>894324</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-06 22:52:33.280021</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>816587</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-03 00:02:26.952613</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>905950</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-16 03:10:23.770071</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>865612</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-17 18:25:12.503775</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>766165</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-05 09:41:30.417265</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>664716</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-04 16:53:19.145462</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>893381</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-12 21:15:01.053176</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>746186</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-13 15:09:49.931508</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>815837</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-09 15:02:49.049787</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>646239</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-03 22:05:22.504709</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>703088</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-04 23:53:49.761544</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>758018</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-10 07:25:26.419988</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>663071</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-22 05:42:11.888107</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>635122</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-20 13:07:51.125996</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>757673</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-03 21:17:23.416244</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>870839</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-15 21:22:00.479523</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>659679</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-14 23:49:33.712246</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>674173</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-21 21:21:36.827588</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>653118</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-09 03:12:31.034796</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>878226</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-05 15:02:50.334962</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>799368</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-09 18:07:34.253935</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>655535</th>\n",
       "      <td>CA</td>\n",
       "      <td>2017-01-09 13:30:47.524512</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>934996</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-09 00:30:08.377677</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>290584 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        country                   timestamp      group landing_page  \\\n",
       "user_id                                                               \n",
       "834778       UK  2017-01-14 23:08:43.304998    control     old_page   \n",
       "928468       US  2017-01-23 14:44:16.387854  treatment     new_page   \n",
       "822059       UK  2017-01-16 14:04:14.719771  treatment     new_page   \n",
       "711597       UK  2017-01-22 03:14:24.763511    control     old_page   \n",
       "710616       UK  2017-01-16 13:14:44.000513  treatment     new_page   \n",
       "909908       UK  2017-01-06 20:44:26.334764  treatment     new_page   \n",
       "811617       US  2017-01-02 18:42:11.851370  treatment     new_page   \n",
       "938122       US  2017-01-10 09:32:08.222716  treatment     new_page   \n",
       "887018       US  2017-01-06 11:09:40.487196  treatment     new_page   \n",
       "820683       US  2017-01-14 11:52:06.521342  treatment     new_page   \n",
       "697357       US  2017-01-15 05:21:57.086762  treatment     new_page   \n",
       "748296       US  2017-01-10 04:05:24.283883  treatment     new_page   \n",
       "666132       UK  2017-01-19 22:45:47.593706  treatment     new_page   \n",
       "668810       UK  2017-01-13 16:59:49.226184  treatment     new_page   \n",
       "940939       US  2017-01-08 10:10:27.267660  treatment     new_page   \n",
       "646414       US  2017-01-07 10:06:42.693231    control     old_page   \n",
       "907385       US  2017-01-17 13:29:52.718288    control     old_page   \n",
       "698200       US  2017-01-21 21:50:42.718525  treatment     new_page   \n",
       "738692       US  2017-01-08 07:03:08.917332    control     old_page   \n",
       "724651       US  2017-01-07 21:32:07.568614    control     old_page   \n",
       "662682       US  2017-01-08 22:48:46.350610  treatment     new_page   \n",
       "639818       US  2017-01-21 21:46:38.906148  treatment     new_page   \n",
       "920941       US  2017-01-21 12:26:25.304141    control     old_page   \n",
       "804632       US  2017-01-03 08:37:34.233194    control     old_page   \n",
       "684798       UK  2017-01-19 16:36:41.202599    control     old_page   \n",
       "766270       UK  2017-01-22 10:11:29.823099  treatment     new_page   \n",
       "857817       UK  2017-01-18 22:21:01.405272  treatment     new_page   \n",
       "750698       UK  2017-01-15 23:31:02.321891  treatment     new_page   \n",
       "721445       US  2017-01-03 02:19:44.388206    control     old_page   \n",
       "744732       UK  2017-01-22 05:00:19.003615    control     old_page   \n",
       "...         ...                         ...        ...          ...   \n",
       "834931       UK  2017-01-06 20:20:20.082513  treatment     new_page   \n",
       "667920       US  2017-01-09 03:00:15.337219  treatment     new_page   \n",
       "869193       US  2017-01-22 21:18:40.153019    control     old_page   \n",
       "737522       US  2017-01-14 17:43:30.555545  treatment     new_page   \n",
       "937048       UK  2017-01-16 05:32:01.893266    control     old_page   \n",
       "689274       UK  2017-01-09 11:39:14.209283    control     old_page   \n",
       "916713       US  2017-01-19 07:43:21.591415  treatment     new_page   \n",
       "894324       US  2017-01-06 22:52:33.280021  treatment     new_page   \n",
       "816587       US  2017-01-03 00:02:26.952613  treatment     new_page   \n",
       "905950       US  2017-01-16 03:10:23.770071  treatment     new_page   \n",
       "865612       US  2017-01-17 18:25:12.503775    control     old_page   \n",
       "766165       US  2017-01-05 09:41:30.417265  treatment     new_page   \n",
       "664716       US  2017-01-04 16:53:19.145462  treatment     new_page   \n",
       "893381       UK  2017-01-12 21:15:01.053176    control     old_page   \n",
       "746186       UK  2017-01-13 15:09:49.931508  treatment     new_page   \n",
       "815837       US  2017-01-09 15:02:49.049787  treatment     new_page   \n",
       "646239       US  2017-01-03 22:05:22.504709    control     old_page   \n",
       "703088       US  2017-01-04 23:53:49.761544  treatment     new_page   \n",
       "758018       UK  2017-01-10 07:25:26.419988  treatment     new_page   \n",
       "663071       UK  2017-01-22 05:42:11.888107    control     old_page   \n",
       "635122       US  2017-01-20 13:07:51.125996    control     old_page   \n",
       "757673       UK  2017-01-03 21:17:23.416244  treatment     new_page   \n",
       "870839       US  2017-01-15 21:22:00.479523  treatment     new_page   \n",
       "659679       US  2017-01-14 23:49:33.712246  treatment     new_page   \n",
       "674173       US  2017-01-21 21:21:36.827588  treatment     new_page   \n",
       "653118       US  2017-01-09 03:12:31.034796    control     old_page   \n",
       "878226       UK  2017-01-05 15:02:50.334962    control     old_page   \n",
       "799368       UK  2017-01-09 18:07:34.253935    control     old_page   \n",
       "655535       CA  2017-01-09 13:30:47.524512  treatment     new_page   \n",
       "934996       UK  2017-01-09 00:30:08.377677    control     old_page   \n",
       "\n",
       "         converted  Intercept  ab_page  new_page  old_page  \n",
       "user_id                                                     \n",
       "834778           0          1        0         0         1  \n",
       "928468           0          1        1         1         0  \n",
       "822059           1          1        1         1         0  \n",
       "711597           0          1        0         0         1  \n",
       "710616           0          1        1         1         0  \n",
       "909908           0          1        1         1         0  \n",
       "811617           1          1        1         1         0  \n",
       "938122           1          1        1         1         0  \n",
       "887018           0          1        1         1         0  \n",
       "820683           0          1        1         1         0  \n",
       "697357           0          1        1         1         0  \n",
       "748296           0          1        1         1         0  \n",
       "666132           0          1        1         1         0  \n",
       "668810           0          1        1         1         0  \n",
       "940939           0          1        1         1         0  \n",
       "646414           0          1        0         0         1  \n",
       "907385           0          1        0         0         1  \n",
       "698200           0          1        1         1         0  \n",
       "738692           0          1        0         0         1  \n",
       "724651           0          1        0         0         1  \n",
       "662682           1          1        1         1         0  \n",
       "639818           0          1        1         1         0  \n",
       "920941           0          1        0         0         1  \n",
       "804632           0          1        0         0         1  \n",
       "684798           0          1        0         0         1  \n",
       "766270           0          1        1         1         0  \n",
       "857817           0          1        1         1         0  \n",
       "750698           0          1        1         1         0  \n",
       "721445           0          1        0         0         1  \n",
       "744732           0          1        0         0         1  \n",
       "...            ...        ...      ...       ...       ...  \n",
       "834931           0          1        1         1         0  \n",
       "667920           1          1        1         1         0  \n",
       "869193           0          1        0         0         1  \n",
       "737522           1          1        1         1         0  \n",
       "937048           0          1        0         0         1  \n",
       "689274           0          1        0         0         1  \n",
       "916713           0          1        1         1         0  \n",
       "894324           1          1        1         1         0  \n",
       "816587           0          1        1         1         0  \n",
       "905950           0          1        1         1         0  \n",
       "865612           0          1        0         0         1  \n",
       "766165           0          1        1         1         0  \n",
       "664716           0          1        1         1         0  \n",
       "893381           0          1        0         0         1  \n",
       "746186           0          1        1         1         0  \n",
       "815837           0          1        1         1         0  \n",
       "646239           0          1        0         0         1  \n",
       "703088           0          1        1         1         0  \n",
       "758018           1          1        1         1         0  \n",
       "663071           0          1        0         0         1  \n",
       "635122           0          1        0         0         1  \n",
       "757673           0          1        1         1         0  \n",
       "870839           0          1        1         1         0  \n",
       "659679           0          1        1         1         0  \n",
       "674173           0          1        1         1         0  \n",
       "653118           0          1        0         0         1  \n",
       "878226           0          1        0         0         1  \n",
       "799368           0          1        0         0         1  \n",
       "655535           0          1        1         1         0  \n",
       "934996           0          1        0         0         1  \n",
       "\n",
       "[290584 rows x 9 columns]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We will merge with user_id \n",
    "df_new = country_df.set_index('user_id').join(df2.set_index('user_id'), how = 'inner')\n",
    "df_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['UK', 'US', 'CA'], dtype=object)"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_new['country'].unique() # We have three countries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will create dummies values only for the unique three countries\n",
    "df_new[['CA', 'UK', 'US']] = pd.get_dummies(df_new['country'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366113\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "country_logit = sm.Logit(df_new['converted'], df_new[['Intercept', 'ab_page', 'CA', 'UK']])\n",
    "fitting_2 = country_logit.fit()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290580</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     3</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Wed, 11 Nov 2020</td> <th>  Pseudo R-squ.:     </th>  <td>2.323e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>17:30:13</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.1760</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th> <td>   -1.9893</td> <td>    0.009</td> <td> -223.763</td> <td> 0.000</td> <td>   -2.007</td> <td>   -1.972</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0149</td> <td>    0.011</td> <td>   -1.307</td> <td> 0.191</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA</th>        <td>   -0.0408</td> <td>    0.027</td> <td>   -1.516</td> <td> 0.130</td> <td>   -0.093</td> <td>    0.012</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>    0.0099</td> <td>    0.013</td> <td>    0.743</td> <td> 0.457</td> <td>   -0.016</td> <td>    0.036</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290580\n",
       "Method:                           MLE   Df Model:                            3\n",
       "Date:                Wed, 11 Nov 2020   Pseudo R-squ.:               2.323e-05\n",
       "Time:                        17:30:13   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                    0.1760\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "Intercept     -1.9893      0.009   -223.763      0.000      -2.007      -1.972\n",
       "ab_page       -0.0149      0.011     -1.307      0.191      -0.037       0.007\n",
       "CA            -0.0408      0.027     -1.516      0.130      -0.093       0.012\n",
       "UK             0.0099      0.013      0.743      0.457      -0.016       0.036\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitting_2.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The p-values are greater than 0.05, So we canot reject the null hypothesis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.0416437559600236, 1.0099491671175422)"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1/np.exp(-0.0408), np.exp(0.0099)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CA is 1.04 times LESS likely to convert\n",
    "\n",
    "UK is 1.01 times MORE likely to convert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "h. Though you have now looked at the individual factors of country and page on conversion, we would now like to look at an interaction between page and country to see if there significant effects on conversion.  Create the necessary additional columns, and fit the new model.  \n",
    "\n",
    "Provide the summary results, and your conclusions based on the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366113\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290581</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     2</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Wed, 11 Nov 2020</td> <th>  Pseudo R-squ.:     </th>  <td>2.364e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>17:38:59</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.08085</td>  \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th> <td>   -1.9963</td> <td>    0.006</td> <td> -322.049</td> <td> 0.000</td> <td>   -2.008</td> <td>   -1.984</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA_page</th>   <td>   -0.0752</td> <td>    0.038</td> <td>   -1.997</td> <td> 0.046</td> <td>   -0.149</td> <td>   -0.001</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK_page</th>   <td>    0.0149</td> <td>    0.017</td> <td>    0.862</td> <td> 0.389</td> <td>   -0.019</td> <td>    0.049</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290581\n",
       "Method:                           MLE   Df Model:                            2\n",
       "Date:                Wed, 11 Nov 2020   Pseudo R-squ.:               2.364e-05\n",
       "Time:                        17:38:59   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                   0.08085\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "Intercept     -1.9963      0.006   -322.049      0.000      -2.008      -1.984\n",
       "CA_page       -0.0752      0.038     -1.997      0.046      -0.149      -0.001\n",
       "UK_page        0.0149      0.017      0.862      0.389      -0.019       0.049\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_new['CA_page'] = df_new['CA']*df_new['ab_page']\n",
    "df_new['UK_page'] = df_new['UK']*df_new['ab_page']\n",
    "df_new['US_page'] = df_new['US']*df_new['ab_page']\n",
    "logit_mod = sm.Logit(df_new['converted'], df_new[['Intercept', 'CA_page', 'UK_page']])\n",
    "results = logit_mod.fit()\n",
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One p-value does present as statistically significant: the interaction of CA and ab_page (p = 0.046; p < 0.05)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.0780997492739288, 1.0150115583846535)"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1/np.exp(-0.0752), np.exp(0.0149)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CA is 1.078 times LESS likely to convert\n",
    "\n",
    "UK is 1.015 times MORE likely to convert\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='conclusions'></a>\n",
    "## Finishing Up\n",
    "\n",
    "> Congratulations!  You have reached the end of the A/B Test Results project!  You should be very proud of all you have accomplished!\n",
    "\n",
    "> **Tip**: Once you are satisfied with your work here, check over your report to make sure that it is satisfies all the areas of the rubric (found on the project submission page at the end of the lesson). You should also probably remove all of the \"Tips\" like this one so that the presentation is as polished as possible.\n",
    "\n",
    "\n",
    "## Directions to Submit\n",
    "\n",
    "> Before you submit your project, you need to create a .html or .pdf version of this notebook in the workspace here. To do that, run the code cell below. If it worked correctly, you should get a return code of 0, and you should see the generated .html file in the workspace directory (click on the orange Jupyter icon in the upper left).\n",
    "\n",
    "> Alternatively, you can download this report as .html via the **File** > **Download as** submenu, and then manually upload it into the workspace directory by clicking on the orange Jupyter icon in the upper left, then using the Upload button.\n",
    "\n",
    "> Once you've done this, you can submit your project by clicking on the \"Submit Project\" button in the lower right here. This will create and submit a zip file with this .ipynb doc and the .html or .pdf version you created. Congratulations!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from subprocess import call\n",
    "call(['python', '-m', 'nbconvert', 'Analyze_ab_test_results_notebook.ipynb'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}