solution_rossman_kaggle.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Kaggle Rossman Cometition\n",
    "\n",
    "This was a kaggle competition to forecast sales at a pharmacy chain/dept store in Europe. It was run back in 2015.\n",
    "\n",
    "The aim of this notebook is\n",
    "\n",
    "(a) to see how to use sklearn pipelines\n",
    "(b) to understand some aspects of feature engineering that come in with continuous and categorical variables\n",
    "(c) to capture results from validation\n",
    "(d) to investigate the use of categorical \"embeddings\" to improve performance of a multi-layer percepton (in another version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy.stats\n",
    "import scipy.special\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.mlab as mlab\n",
    "from matplotlib import cm\n",
    "import pandas as pd\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = Path('./data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We engage in some cleaning. A lot of cleaning of this dataset has already been done for us. Some features have been created. In particular we moved from dates to week-of-year, day-of week, etc. For example the 49th and 50th weeks of the year may have higher sales!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_csv(data/\"train_clean.csv.gz\", compression='gzip').drop(['index', 'PromoInterval'], axis=1)\n",
    "test_df = pd.read_csv(data/\"test_clean.csv.gz\", compression='gzip').drop(['index', 'PromoInterval'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Store</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>Promo</th>\n",
       "      <th>StateHoliday</th>\n",
       "      <th>SchoolHoliday</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>...</th>\n",
       "      <th>AfterStateHoliday</th>\n",
       "      <th>BeforeStateHoliday</th>\n",
       "      <th>AfterPromo</th>\n",
       "      <th>BeforePromo</th>\n",
       "      <th>SchoolHoliday_bw</th>\n",
       "      <th>StateHoliday_bw</th>\n",
       "      <th>Promo_bw</th>\n",
       "      <th>SchoolHoliday_fw</th>\n",
       "      <th>StateHoliday_fw</th>\n",
       "      <th>Promo_fw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>2015</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>105</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>2015</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>105</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>2015</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>115</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>2015</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>115</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>2015</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>105</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 90 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Store  DayOfWeek        Date  Open  Promo  StateHoliday  SchoolHoliday  \\\n",
       "0   1      1          4  2015-09-17   1.0      1         False              0   \n",
       "1   2      3          4  2015-09-17   1.0      1         False              0   \n",
       "2   3      7          4  2015-09-17   1.0      1         False              0   \n",
       "3   4      8          4  2015-09-17   1.0      1         False              0   \n",
       "4   5      9          4  2015-09-17   1.0      1         False              0   \n",
       "\n",
       "   Year  Month    ...     AfterStateHoliday  BeforeStateHoliday  AfterPromo  \\\n",
       "0  2015      9    ...                   105                   0           0   \n",
       "1  2015      9    ...                   105                   0           0   \n",
       "2  2015      9    ...                   115                   0           0   \n",
       "3  2015      9    ...                   115                   0           0   \n",
       "4  2015      9    ...                   105                   0           0   \n",
       "\n",
       "   BeforePromo  SchoolHoliday_bw  StateHoliday_bw  Promo_bw  SchoolHoliday_fw  \\\n",
       "0            0               0.0              0.0       4.0               0.0   \n",
       "1            0               0.0              0.0       4.0               0.0   \n",
       "2            0               0.0              0.0       4.0               0.0   \n",
       "3            0               0.0              0.0       4.0               0.0   \n",
       "4            0               0.0              0.0       4.0               0.0   \n",
       "\n",
       "   StateHoliday_fw  Promo_fw  \n",
       "0              0.0       1.0  \n",
       "1              0.0       1.0  \n",
       "2              0.0       1.0  \n",
       "3              0.0       1.0  \n",
       "4              0.0       1.0  \n",
       "\n",
       "[5 rows x 90 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Id', 'Store', 'DayOfWeek', 'Date', 'Open', 'Promo', 'StateHoliday',\n",
       "       'SchoolHoliday', 'Year', 'Month', 'Week', 'Day', 'Dayofweek',\n",
       "       'Dayofyear', 'Is_month_end', 'Is_month_start', 'Is_quarter_end',\n",
       "       'Is_quarter_start', 'Is_year_end', 'Is_year_start', 'Elapsed',\n",
       "       'StoreType', 'Assortment', 'CompetitionDistance',\n",
       "       'CompetitionOpenSinceMonth', 'CompetitionOpenSinceYear', 'Promo2',\n",
       "       'Promo2SinceWeek', 'Promo2SinceYear', 'State', 'file', 'week', 'trend',\n",
       "       'file_DE', 'week_DE', 'trend_DE', 'Date_DE', 'State_DE', 'Month_DE',\n",
       "       'Day_DE', 'Dayofweek_DE', 'Dayofyear_DE', 'Is_month_end_DE',\n",
       "       'Is_month_start_DE', 'Is_quarter_end_DE', 'Is_quarter_start_DE',\n",
       "       'Is_year_end_DE', 'Is_year_start_DE', 'Elapsed_DE', 'Max_TemperatureC',\n",
       "       'Mean_TemperatureC', 'Min_TemperatureC', 'Dew_PointC', 'MeanDew_PointC',\n",
       "       'Min_DewpointC', 'Max_Humidity', 'Mean_Humidity', 'Min_Humidity',\n",
       "       'Max_Sea_Level_PressurehPa', 'Mean_Sea_Level_PressurehPa',\n",
       "       'Min_Sea_Level_PressurehPa', 'Max_VisibilityKm', 'Mean_VisibilityKm',\n",
       "       'Min_VisibilitykM', 'Max_Wind_SpeedKm_h', 'Mean_Wind_SpeedKm_h',\n",
       "       'Max_Gust_SpeedKm_h', 'Precipitationmm', 'CloudCover', 'Events',\n",
       "       'WindDirDegrees', 'StateName', 'CompetitionOpenSince',\n",
       "       'CompetitionDaysOpen', 'CompetitionMonthsOpen', 'Promo2Since',\n",
       "       'Promo2Days', 'Promo2Weeks', 'AfterSchoolHoliday',\n",
       "       'BeforeSchoolHoliday', 'AfterStateHoliday', 'BeforeStateHoliday',\n",
       "       'AfterPromo', 'BeforePromo', 'SchoolHoliday_bw', 'StateHoliday_bw',\n",
       "       'Promo_bw', 'SchoolHoliday_fw', 'StateHoliday_fw', 'Promo_fw'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['Events'] = train_df['Events'].fillna('None')\n",
    "test_df['Events'] = test_df['Events'].fillna('None')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And in log-transforming the dependent variable because it is long-tailed. Histo it to see this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'Sales'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m//anaconda/envs/py3l/lib/python3.6/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_loc\u001b[0;34m(self, key, method, tolerance)\u001b[0m\n\u001b[1;32m   3077\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3078\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3079\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'Sales'",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-44-73ecb6141999>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# your code here\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_df\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Sales'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m;\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m//anaconda/envs/py3l/lib/python3.6/site-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m   2686\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_getitem_multilevel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2687\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2688\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_getitem_column\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2689\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2690\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_getitem_column\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m//anaconda/envs/py3l/lib/python3.6/site-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36m_getitem_column\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m   2693\u001b[0m         \u001b[0;31m# get column\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2694\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcolumns\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_unique\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2695\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_item_cache\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2696\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2697\u001b[0m         \u001b[0;31m# duplicate columns & possible reduce dimensionality\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m//anaconda/envs/py3l/lib/python3.6/site-packages/pandas/core/generic.py\u001b[0m in \u001b[0;36m_get_item_cache\u001b[0;34m(self, item)\u001b[0m\n\u001b[1;32m   2487\u001b[0m         \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcache\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2488\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mres\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2489\u001b[0;31m             \u001b[0mvalues\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2490\u001b[0m             \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_box_item_values\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalues\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2491\u001b[0m             \u001b[0mcache\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m//anaconda/envs/py3l/lib/python3.6/site-packages/pandas/core/internals.py\u001b[0m in \u001b[0;36mget\u001b[0;34m(self, item, fastpath)\u001b[0m\n\u001b[1;32m   4113\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4114\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misna\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 4115\u001b[0;31m                 \u001b[0mloc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   4116\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4117\u001b[0m                 \u001b[0mindexer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0misna\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m//anaconda/envs/py3l/lib/python3.6/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_loc\u001b[0;34m(self, key, method, tolerance)\u001b[0m\n\u001b[1;32m   3078\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3079\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3080\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maybe_cast_indexer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3081\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3082\u001b[0m         \u001b[0mindexer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_indexer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtolerance\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtolerance\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'Sales'"
     ]
    }
   ],
   "source": [
    "# your code here\n",
    "plt.hist(train_df['Sales']);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_resp = np.log(train_df['Sales'].copy())\n",
    "train_df = train_df.drop('Sales', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets get some idea about our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>Date</th>\n",
       "      <th>Customers</th>\n",
       "      <th>Open</th>\n",
       "      <th>Promo</th>\n",
       "      <th>StateHoliday</th>\n",
       "      <th>SchoolHoliday</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>...</th>\n",
       "      <th>AfterStateHoliday</th>\n",
       "      <th>BeforeStateHoliday</th>\n",
       "      <th>AfterPromo</th>\n",
       "      <th>BeforePromo</th>\n",
       "      <th>SchoolHoliday_bw</th>\n",
       "      <th>StateHoliday_bw</th>\n",
       "      <th>Promo_bw</th>\n",
       "      <th>SchoolHoliday_fw</th>\n",
       "      <th>StateHoliday_fw</th>\n",
       "      <th>Promo_fw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>555</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>2015</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>625</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>2015</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>67</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>821</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>2015</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>1498</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>2015</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>67</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>559</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>2015</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 90 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Store  DayOfWeek        Date  Customers  Open  Promo  StateHoliday  \\\n",
       "0      1          5  2015-07-31        555     1      1         False   \n",
       "1      2          5  2015-07-31        625     1      1         False   \n",
       "2      3          5  2015-07-31        821     1      1         False   \n",
       "3      4          5  2015-07-31       1498     1      1         False   \n",
       "4      5          5  2015-07-31        559     1      1         False   \n",
       "\n",
       "   SchoolHoliday  Year  Month    ...     AfterStateHoliday  \\\n",
       "0              1  2015      7    ...                    57   \n",
       "1              1  2015      7    ...                    67   \n",
       "2              1  2015      7    ...                    57   \n",
       "3              1  2015      7    ...                    67   \n",
       "4              1  2015      7    ...                    57   \n",
       "\n",
       "   BeforeStateHoliday  AfterPromo  BeforePromo  SchoolHoliday_bw  \\\n",
       "0                   0           0            0               5.0   \n",
       "1                   0           0            0               5.0   \n",
       "2                   0           0            0               5.0   \n",
       "3                   0           0            0               5.0   \n",
       "4                   0           0            0               5.0   \n",
       "\n",
       "   StateHoliday_bw  Promo_bw  SchoolHoliday_fw  StateHoliday_fw  Promo_fw  \n",
       "0              0.0       5.0               7.0              0.0       5.0  \n",
       "1              0.0       5.0               1.0              0.0       1.0  \n",
       "2              0.0       5.0               5.0              0.0       5.0  \n",
       "3              0.0       5.0               1.0              0.0       1.0  \n",
       "4              0.0       5.0               1.0              0.0       1.0  \n",
       "\n",
       "[5 rows x 90 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((844338, 90), (41088, 90))"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.shape, test_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         2015-07-31\n",
       "1         2015-07-31\n",
       "2         2015-07-31\n",
       "3         2015-07-31\n",
       "4         2015-07-31\n",
       "5         2015-07-31\n",
       "6         2015-07-31\n",
       "7         2015-07-31\n",
       "8         2015-07-31\n",
       "9         2015-07-31\n",
       "10        2015-07-31\n",
       "11        2015-07-31\n",
       "12        2015-07-31\n",
       "13        2015-07-31\n",
       "14        2015-07-31\n",
       "15        2015-07-31\n",
       "16        2015-07-31\n",
       "17        2015-07-31\n",
       "18        2015-07-31\n",
       "19        2015-07-31\n",
       "20        2015-07-31\n",
       "21        2015-07-31\n",
       "22        2015-07-31\n",
       "23        2015-07-31\n",
       "24        2015-07-31\n",
       "25        2015-07-31\n",
       "26        2015-07-31\n",
       "27        2015-07-31\n",
       "28        2015-07-31\n",
       "29        2015-07-31\n",
       "             ...    \n",
       "844308    2013-01-02\n",
       "844309    2013-01-02\n",
       "844310    2013-01-02\n",
       "844311    2013-01-02\n",
       "844312    2013-01-02\n",
       "844313    2013-01-02\n",
       "844314    2013-01-02\n",
       "844315    2013-01-02\n",
       "844316    2013-01-02\n",
       "844317    2013-01-02\n",
       "844318    2013-01-02\n",
       "844319    2013-01-02\n",
       "844320    2013-01-02\n",
       "844321    2013-01-01\n",
       "844322    2013-01-01\n",
       "844323    2013-01-01\n",
       "844324    2013-01-01\n",
       "844325    2013-01-01\n",
       "844326    2013-01-01\n",
       "844327    2013-01-01\n",
       "844328    2013-01-01\n",
       "844329    2013-01-01\n",
       "844330    2013-01-01\n",
       "844331    2013-01-01\n",
       "844332    2013-01-01\n",
       "844333    2013-01-01\n",
       "844334    2013-01-01\n",
       "844335    2013-01-01\n",
       "844336    2013-01-01\n",
       "844337    2013-01-01\n",
       "Name: Date, Length: 844338, dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.Date # latest date first"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Types of variables and cardinality\n",
    "\n",
    "We make a note of which variables are categorical and which are not. This is a choice. If cardinality is not too high, binning or categorizing can be beneficial. Often this will be true for integer valued variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_vars = ['Store', 'DayOfWeek', 'Year', 'Month', 'Day', 'StateHoliday', 'CompetitionMonthsOpen',\n",
    "    'Promo2Weeks', 'StoreType', 'Assortment', 'CompetitionOpenSinceYear', 'Promo2SinceYear',\n",
    "    'State', 'Week', 'Events', 'Promo_fw', 'Promo_bw', 'StateHoliday_fw', 'StateHoliday_bw',\n",
    "    'SchoolHoliday_fw', 'SchoolHoliday_bw', 'Promo', 'SchoolHoliday']\n",
    "\n",
    "cont_vars = ['CompetitionDistance', 'Max_TemperatureC', 'Mean_TemperatureC', 'Min_TemperatureC',\n",
    "   'Max_Humidity', 'Mean_Humidity', 'Min_Humidity', 'Max_Wind_SpeedKm_h', \n",
    "   'Mean_Wind_SpeedKm_h', 'CloudCover', 'trend', 'trend_DE',\n",
    "   'AfterStateHoliday', 'BeforeStateHoliday']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We look for missing data and store the column names where this happend in the continuous data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CompetitionDistance 2186\n",
      "CloudCover 68056\n"
     ]
    }
   ],
   "source": [
    "nacols=[]\n",
    "for v in cont_vars:\n",
    "    if np.sum(train_df[v].isnull()) > 0:\n",
    "        nacols.append(v)\n",
    "        print(v, np.sum(train_df[v].isnull()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And look at some cardinalities: since we have none below 10, we dont engage in binning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CompetitionDistance 655\n",
      "Max_TemperatureC 50\n",
      "Mean_TemperatureC 45\n",
      "Min_TemperatureC 40\n",
      "Max_Humidity 50\n",
      "Mean_Humidity 71\n",
      "Min_Humidity 93\n",
      "Max_Wind_SpeedKm_h 42\n",
      "Mean_Wind_SpeedKm_h 27\n",
      "CloudCover 10\n",
      "trend 67\n",
      "trend_DE 38\n",
      "AfterStateHoliday 136\n",
      "BeforeStateHoliday 147\n"
     ]
    }
   ],
   "source": [
    "# your code here\n",
    "for k in cont_vars:\n",
    "    print(k, train_df[k].unique().shape[0])\n",
    "    if train_df[k].unique().shape[0] < 10:\n",
    "        print(train_df[k].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do a similar looksie on the categorical variables. Some of these have many levels. Is there really that much information in 1115 store labels. Can we get some compression to increase our signal-to-noise?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Store 1115\n",
      "DayOfWeek 7\n",
      "[5 4 3 2 1 7 6]\n",
      "Year 3\n",
      "[2015 2014 2013]\n",
      "Month 12\n",
      "[ 7  6  5  4  3  2  1 12 11 10  9  8]\n",
      "Day 31\n",
      "[31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10  9  8\n",
      "  7  6  5  4  3  2  1]\n",
      "StateHoliday 2\n",
      "[False  True]\n",
      "CompetitionMonthsOpen 25\n",
      "[24  3 19  9  0 16 17  7 15 22 11 13  2 23 12  4 10  1 14 20  8 18  6 21\n",
      "  5]\n",
      "Promo2Weeks 26\n",
      "[ 0 25 17  8 13 24 16  7 12 23 15  6 11 22 14  5 10 21  4  9 20  3 19  2\n",
      " 18  1]\n",
      "StoreType 4\n",
      "['c' 'a' 'd' 'b']\n",
      "Assortment 3\n",
      "['a' 'c' 'b']\n",
      "CompetitionOpenSinceYear 23\n",
      "[2008 2007 2006 2009 2015 2013 2014 2000 2011 1900 2010 2005 1999 2003\n",
      " 2012 2004 2002 1961 1995 2001 1990 1994 1998]\n",
      "Promo2SinceYear 8\n",
      "[1900 2010 2011 2012 2009 2014 2015 2013]\n",
      "State 12\n",
      "['HE' 'TH' 'NW' 'BE' 'SN' 'SH' 'HB,NI' 'BY' 'BW' 'RP' 'ST' 'HH']\n",
      "Week 52\n",
      "Events 22\n",
      "['Fog' 'None' 'Rain' 'Rain-Thunderstorm' 'Fog-Rain'\n",
      " 'Rain-Hail-Thunderstorm' 'Fog-Rain-Thunderstorm' 'Thunderstorm'\n",
      " 'Rain-Hail' 'Fog-Thunderstorm' 'Rain-Snow' 'Fog-Rain-Hail-Thunderstorm'\n",
      " 'Snow' 'Rain-Snow-Hail' 'Rain-Snow-Hail-Thunderstorm'\n",
      " 'Rain-Snow-Thunderstorm' 'Fog-Rain-Snow' 'Fog-Snow' 'Snow-Hail'\n",
      " 'Fog-Rain-Snow-Hail' 'Fog-Rain-Hail' 'Fog-Snow-Hail']\n",
      "Promo_fw 6\n",
      "[5. 1. 2. 3. 4. 0.]\n",
      "Promo_bw 6\n",
      "[5. 4. 3. 2. 1. 0.]\n",
      "StateHoliday_fw 8\n",
      "[0. 1. 2. 3. 4. 5. 6. 7.]\n",
      "StateHoliday_bw 8\n",
      "[0. 1. 2. 3. 4. 5. 6. 7.]\n",
      "SchoolHoliday_fw 8\n",
      "[7. 1. 5. 4. 2. 3. 0. 6.]\n",
      "SchoolHoliday_bw 8\n",
      "[5. 7. 0. 2. 4. 1. 3. 6.]\n",
      "Promo 2\n",
      "[1 0]\n",
      "SchoolHoliday 2\n",
      "[1 0]\n"
     ]
    }
   ],
   "source": [
    "for k in cat_vars:\n",
    "    print(k, train_df[k].unique().shape[0])\n",
    "    if train_df[k].unique().shape[0] < 50:\n",
    "        print(train_df[k].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a validation set\n",
    "\n",
    "The construction of a validation or \"development\" set is not always a `test_train_split` deal. Here we create a validation set of \"latest\" data, cireesponding oin date and size to what we have in the test set. Hopefully this will make sure we have similar distributions of features and outcomes on both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "41395"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cut = train_df['Date'][(train_df['Date'] == train_df['Date'][len(test_df)])].index.max()\n",
    "cut"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_idx = range(cut)\n",
    "train_idx = list(np.setdiff1d(range(train_df.shape[0]), valid_idx))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "trdf = train_df.iloc[train_idx]\n",
    "vadf = train_df.iloc[valid_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((802943, 90), (41395, 90))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trdf.shape, vadf.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformation Pipelines\n",
    "\n",
    "Ok, now we'll use the new `ColumnTransformer`, with imputation, missing-data indicators, the new `OrdinalEncoder`, and the usual Standard Scaling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.impute import SimpleImputer,MissingIndicator\n",
    "from sklearn.pipeline import make_pipeline, make_union, Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "impu = SimpleImputer(strategy=\"median\") # create a median imputer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do the missing indicator separately as it creates a new column. It is possible to do this in the pipeline flow in `sklearn` using a union, but subsequent scaling wants to scale this indicator since the categorical list does not include the new columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "mi = MissingIndicator() # create, fit, and transform a missingness indicator\n",
    "mi.fit(trdf[nacols])\n",
    "Xtrmi = mi.transform(trdf[nacols])\n",
    "Xvami = mi.transform(vadf[nacols])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False,  True])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xtrmi[4460,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler, OrdinalEncoder\n",
    "ss = StandardScaler()\n",
    "oe = OrdinalEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "trdf_cat = trdf[cat_vars]\n",
    "trdf_cont = trdf[cont_vars]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We construct two pipelines, one for categoricals and one for continuous variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "cont_pipe = Pipeline([(\"imp\",impu), (\"scale\", ss)])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_pipe = Pipeline([(\"categorify\", oe)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And combine them here in a transformer list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformers = [('cat', cat_pipe, cat_vars),\n",
    "                    ('cont', cont_pipe, cont_vars)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we use a `ColumnTransformer` to combine these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "ct = ColumnTransformer(transformers=transformers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ColumnTransformer(n_jobs=None, remainder='drop', sparse_threshold=0.3,\n",
       "         transformer_weights=None,\n",
       "         transformers=[('cat', Pipeline(memory=None,\n",
       "     steps=[('categorify', OrdinalEncoder(categories='auto', dtype=<class 'numpy.float64'>))]), ['Store', 'DayOfWeek', 'Year', 'Month', 'Day', 'StateHoliday', 'CompetitionMonthsOpen', 'Promo2Weeks', 'StoreType', 'Assortment', 'CompetitionOpenSinceYear', 'P...ean_Wind_SpeedKm_h', 'CloudCover', 'trend', 'trend_DE', 'AfterStateHoliday', 'BeforeStateHoliday'])])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct.fit(trdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "Xtr = ct.transform(trdf)\n",
    "Xval = ct.transform(vadf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((802943, 37), (802943, 2))"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xtr.shape, Xtrmi.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We concatenate the old indicators back in. The transformer lists all the categoricals first, since thats the first item in `transformers`, so we pre-pend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(802943, 39)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xtrain = np.concatenate([Xtrmi, Xtr], axis=1)\n",
    "Xtrain.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(41395, 39)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xvalid = np.concatenate([Xvami, Xval], axis=1)\n",
    "Xvalid.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sklearn-pipelines lose our nice pandas names. so we bring them back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([(0, ('CompetitionDistance_missing', 'cont')),\n",
       "  (1, ('CloudCover_missing', 'cont')),\n",
       "  (2, ('Store', 'cat')),\n",
       "  (3, ('DayOfWeek', 'cat')),\n",
       "  (4, ('Year', 'cat')),\n",
       "  (5, ('Month', 'cat')),\n",
       "  (6, ('Day', 'cat')),\n",
       "  (7, ('StateHoliday', 'cat')),\n",
       "  (8, ('CompetitionMonthsOpen', 'cat')),\n",
       "  (9, ('Promo2Weeks', 'cat')),\n",
       "  (10, ('StoreType', 'cat')),\n",
       "  (11, ('Assortment', 'cat')),\n",
       "  (12, ('CompetitionOpenSinceYear', 'cat')),\n",
       "  (13, ('Promo2SinceYear', 'cat')),\n",
       "  (14, ('State', 'cat')),\n",
       "  (15, ('Week', 'cat')),\n",
       "  (16, ('Events', 'cat')),\n",
       "  (17, ('Promo_fw', 'cat')),\n",
       "  (18, ('Promo_bw', 'cat')),\n",
       "  (19, ('StateHoliday_fw', 'cat')),\n",
       "  (20, ('StateHoliday_bw', 'cat')),\n",
       "  (21, ('SchoolHoliday_fw', 'cat')),\n",
       "  (22, ('SchoolHoliday_bw', 'cat')),\n",
       "  (23, ('Promo', 'cat')),\n",
       "  (24, ('SchoolHoliday', 'cat')),\n",
       "  (25, ('CompetitionDistance', 'cont')),\n",
       "  (26, ('Max_TemperatureC', 'cont')),\n",
       "  (27, ('Mean_TemperatureC', 'cont')),\n",
       "  (28, ('Min_TemperatureC', 'cont')),\n",
       "  (29, ('Max_Humidity', 'cont')),\n",
       "  (30, ('Mean_Humidity', 'cont')),\n",
       "  (31, ('Min_Humidity', 'cont')),\n",
       "  (32, ('Max_Wind_SpeedKm_h', 'cont')),\n",
       "  (33, ('Mean_Wind_SpeedKm_h', 'cont')),\n",
       "  (34, ('CloudCover', 'cont')),\n",
       "  (35, ('trend', 'cont')),\n",
       "  (36, ('trend_DE', 'cont')),\n",
       "  (37, ('AfterStateHoliday', 'cont')),\n",
       "  (38, ('BeforeStateHoliday', 'cont'))],\n",
       " 39)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = trdf.columns\n",
    "actcols = []\n",
    "actcolcount = 0\n",
    "nacols_cat = []\n",
    "for k in nacols:\n",
    "    actcols.append((k+'_missing', 'cont'))\n",
    "    nacols_cat.append(k+'_missing')\n",
    "    actcolcount+=1\n",
    "for k in cat_vars+cont_vars:\n",
    "    if k in cat_vars:\n",
    "        actcols.append((k, \"cat\"))\n",
    "        actcolcount+=1\n",
    "    if k in cont_vars:\n",
    "        actcols.append((k, \"cont\"))\n",
    "        actcolcount+=1\n",
    "        \n",
    "list(enumerate(actcols)), actcolcount"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Time to learn\n",
    "\n",
    "We first split the y (the log of the y, really)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((802943,), (41395,))"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ytrain = train_resp[train_idx]\n",
    "yvalid = train_resp[list(valid_idx)]\n",
    "ytrain.shape, yvalid.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((802943, 39), (41395, 39))"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xtrain.shape, Xvalid.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import gzip\n",
    "to_save = dict(key=list(enumerate(actcols)), Xtrain=Xtrain, Xvalid=Xvalid, ytrain=ytrain, yvalid=yvalid)\n",
    "with gzip.open(\"data/processed.pkl.gz\", \"wb\") as fd:\n",
    "    pickle.dump(to_save, fd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "with gzip.open(\"data/processed.pkl.gz\", \"rb\") as fd:\n",
    "    reget = pickle.load(fd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['key', 'Xtrain', 'Xvalid', 'ytrain', 'yvalid'])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reget.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge\n",
    "from sklearn.model_selection import ParameterGrid\n",
    "param_grid = {'alpha': [1e-6, 1e-5, 1e-3, 0.001, 0.01, 0.1, 1, 10, 100]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtain the minimum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C {'alpha': 1e-06} MSE Valid 0.12686369496129776 Train 0.13746464595659655 VScore 0.260601355295936\n",
      "C {'alpha': 1e-05} MSE Valid 0.12686369496178454 Train 0.1374646459565966 VScore 0.2606013552930989\n",
      "C {'alpha': 0.001} MSE Valid 0.12686369501533346 Train 0.13746464595659655 VScore 0.26060135498100034\n",
      "C {'alpha': 0.001} MSE Valid 0.12686369501533346 Train 0.13746464595659655 VScore 0.26060135498100034\n",
      "C {'alpha': 0.01} MSE Valid 0.12686369550214394 Train 0.13746464595660104 VScore 0.2606013521437267\n",
      "C {'alpha': 0.1} MSE Valid 0.1268637003705061 Train 0.13746464595704072 VScore 0.26060132376949097\n",
      "C {'alpha': 1} MSE Valid 0.12686374907981293 Train 0.137464646000974 VScore 0.260601039877433\n",
      "C {'alpha': 10} MSE Valid 0.1268642387087756 Train 0.1374646503608408 VScore 0.26059818617691977\n",
      "C {'alpha': 100} MSE Valid 0.12686935825463155 Train 0.13746505506237522 VScore 0.26056834796931794\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "# your code here\n",
    "for p in ParameterGrid(param_grid):\n",
    "    est = Ridge()\n",
    "    est.set_params(**p)\n",
    "    est.fit(Xtrain, ytrain)\n",
    "    ypred = est.predict(Xvalid)\n",
    "    ypredtrain = est.predict(Xtrain)\n",
    "    mse = mean_squared_error(ypred, yvalid)\n",
    "    msetr = mean_squared_error(ypredtrain, ytrain)\n",
    "    print(\"C\", p, \"MSE Valid\", mse, \"Train\", msetr, 'VScore', est.score(Xvalid, yvalid))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}