Cross Validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "994d48a9",
   "metadata": {},
   "source": [
    "How to use cross-validation for better measures of model performance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4b05751",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e44aeb86",
   "metadata": {},
   "source": [
    "You will face choices about what predictive variables to use, what types of models to use, what arguments to supply to those models, etc. So far, you have made these choices in a data-driven way by measuring model quality with a validation (or holdout) set.\n",
    "\n",
    "But there are some drawbacks to this approach. To see this, imagine you have a dataset with 5000 rows. You will typically keep about 20% of the data as a validation dataset, or 1000 rows. But this leaves some random chance in determining model scores. That is, a model might do well on one set of 1000 rows, even if it would be inaccurate on a different 1000 rows."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43657431",
   "metadata": {},
   "source": [
    "## Gist"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3118d493",
   "metadata": {},
   "source": [
    "In general, larger validation set means less randomness or noise there will be in out measure of model quality, and more reliable it will be."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8b3984d",
   "metadata": {},
   "source": [
    "But we can only get larger validation set by removing rows from our training data, and smaller training data means worse models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb35b5ad",
   "metadata": {},
   "source": [
    "## What is cross-validation?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bf03d28",
   "metadata": {},
   "source": [
    "In cross-validation, we run our modeling process on different subsets of the data to get multiple measures of model quality.\n",
    "\n",
    "For example, we could begin by dividing the data into 5 pieces, each 20% of the full dataset. In this case, we say that we have broken the data into 5 \"folds\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b90cae5",
   "metadata": {},
   "source": [
    "Then, we run one experiment for each fold:\n",
    "\n",
    "1. In Experiment 1, we use the first fold as a validation (or holdout) set and everything else as training data. This gives us a measure of model quality based on a 20% holdout set.\n",
    "\n",
    "\n",
    "2. In Experiment 2, we hold out data from the second fold (and use everything except the second fold for training the model). The holdout set is then used to get a second estimate of model quality.\n",
    "\n",
    "\n",
    "3. We repeat this process, using every fold once as the holdout set. Putting this together, 100% of the data is used as holdout at some point, and we end up with a measure of model quality that is based on all of the rows in the dataset (even if we don't use all rows simultaneously)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28530988",
   "metadata": {},
   "source": [
    "## When to use cross-validation?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8462a8c8",
   "metadata": {},
   "source": [
    "Cross-validation gives a more accurate measure of model quality, which is especially important if you are making a lot of modeling decisions. However, it can take longer to run, because it estimates multiple models (one for each fold).\n",
    "\n",
    "So, given these tradeoffs, when should you use which approach?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fa27115",
   "metadata": {},
   "source": [
    "1. For small datasets, where extra computational burden isn't a big deal, you should run cross-validation.\n",
    "\n",
    "\n",
    "2. For larger datasets, a single validation set is sufficient. Your code will run faster, and you may have enough data that there's little need to re-use some of it for holdout."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e14b36b9",
   "metadata": {},
   "source": [
    "##### How much is larger dataset or How much is small dataset?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85cfb83f",
   "metadata": {},
   "source": [
    "There's no simple threshold for what constitutes a large vs. small dataset. But if your model takes a couple minutes or less to run, it's probably worth switching to cross-validation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a95bd4c",
   "metadata": {},
   "source": [
    "Alternatively, you can run cross-validation and see if the scores for each experiment seem close. If each experiment yields the same results, a single validation set is probably sufficient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "da3f2d4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Read the data\n",
    "data = pd.read_csv('datasets/melb_data.csv')\n",
    "\n",
    "# Select subset of predictors\n",
    "cols_to_use = ['Rooms', 'Distance', 'Landsize', 'BuildingArea', 'YearBuilt']\n",
    "X = data[cols_to_use]\n",
    "\n",
    "# Select target\n",
    "y = data.Price"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4628966f",
   "metadata": {},
   "source": [
    "Then, we define a pipeline that uses an imputer to fill in missing values and a random forest model to make predictions.\n",
    "\n",
    "While it's possible to do cross-validation without pipelines, it is quite difficult! Using a pipeline will make the code remarkably straightforward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "faf69438",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "my_pipeline = Pipeline(\n",
    "    steps=[\n",
    "        ('preprocessor', SimpleImputer()),\n",
    "        ('model', RandomForestRegressor(n_estimators=50,random_state=0))\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87fea746",
   "metadata": {},
   "source": [
    "We obtain the cross-validation scores with the cross_val_score() function from scikit-learn. We set the number of folds with the cv parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c21a5329",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE scores:\n",
      " [301628.7893587  303164.4782723  287298.331666   236061.84754543\n",
      " 260383.45111427]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# Multiply by -1 since sklearn calculates *negative* MAE\n",
    "scores = -1 * cross_val_score(my_pipeline, X, y,\n",
    "                              cv=5,\n",
    "                              scoring='neg_mean_absolute_error')\n",
    "\n",
    "print(\"MAE scores:\\n\", scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c753a1f",
   "metadata": {},
   "source": [
    "The scoring parameter chooses a measure of model quality to report: in this case, we chose negative mean absolute error (MAE). The docs for scikit-learn show a [list of options](https://scikit-learn.org/stable/modules/model_evaluation.html).\n",
    "\n",
    "It is a little surprising that we specify negative MAE. Scikit-learn has a convention where all metrics are defined so a high number is better. Using negatives here allows them to be consistent with that convention, though negative MAE is almost unheard of elsewhere."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db4c7003",
   "metadata": {},
   "source": [
    "We typically want a single measure of model quality to compare alternative models. So we take the average across experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "eb23bebf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average MAE score (across experiments):\n",
      "277707.3795913405\n"
     ]
    }
   ],
   "source": [
    "print(\"Average MAE score (across experiments):\")\n",
    "print(scores.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46a7240d",
   "metadata": {},
   "source": [
    "Using cross-validation yields a much better measure of model quality, with the added benefit of cleaning up our code: note that we no longer need to keep track of separate training and validation sets. So, especially for small datasets, it's a good improvement!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfbd954b",
   "metadata": {},
   "source": [
    "## Applying cross-validation and a method to choose best parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "b91c9988",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data\n",
    "train_data = pd.read_csv('datasets/melb_data.csv', index_col=False)\n",
    "test_data = train_data.iloc[10864:,:]\n",
    "train_data = train_data.iloc[:10864,:]\n",
    "\n",
    "# Remove rows with missing target, separate target from predictors\n",
    "train_data.dropna(axis=0, subset=['Price'], inplace=True)\n",
    "y = train_data.Price              \n",
    "train_data.drop(['Price'], axis=1, inplace=True)\n",
    "\n",
    "# Select numeric columns only\n",
    "numeric_cols = [cname for cname in train_data.columns if train_data[cname].dtype in ['int64', 'float64']]\n",
    "X = train_data[numeric_cols].copy()\n",
    "X_test = test_data[numeric_cols].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "5ebf0b11",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2716.0"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "13580*0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "d5a82a90",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10864"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "13580-2716"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "b62b48ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2716 entries, 10864 to 13579\n",
      "Data columns (total 21 columns):\n",
      " #   Column         Non-Null Count  Dtype  \n",
      "---  ------         --------------  -----  \n",
      " 0   Suburb         2716 non-null   object \n",
      " 1   Address        2716 non-null   object \n",
      " 2   Rooms          2716 non-null   int64  \n",
      " 3   Type           2716 non-null   object \n",
      " 4   Price          2716 non-null   float64\n",
      " 5   Method         2716 non-null   object \n",
      " 6   SellerG        2716 non-null   object \n",
      " 7   Date           2716 non-null   object \n",
      " 8   Distance       2716 non-null   float64\n",
      " 9   Postcode       2716 non-null   float64\n",
      " 10  Bedroom2       2716 non-null   float64\n",
      " 11  Bathroom       2716 non-null   float64\n",
      " 12  Car            2654 non-null   float64\n",
      " 13  Landsize       2716 non-null   float64\n",
      " 14  BuildingArea   1351 non-null   float64\n",
      " 15  YearBuilt      1540 non-null   float64\n",
      " 16  CouncilArea    1349 non-null   object \n",
      " 17  Lattitude      2716 non-null   float64\n",
      " 18  Longtitude     2716 non-null   float64\n",
      " 19  Regionname     2716 non-null   object \n",
      " 20  Propertycount  2716 non-null   float64\n",
      "dtypes: float64(12), int64(1), object(8)\n",
      "memory usage: 445.7+ KB\n"
     ]
    }
   ],
   "source": [
    "test_data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "deecc9a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 10864 entries, 0 to 10863\n",
      "Data columns (total 20 columns):\n",
      " #   Column         Non-Null Count  Dtype  \n",
      "---  ------         --------------  -----  \n",
      " 0   Suburb         10864 non-null  object \n",
      " 1   Address        10864 non-null  object \n",
      " 2   Rooms          10864 non-null  int64  \n",
      " 3   Type           10864 non-null  object \n",
      " 4   Method         10864 non-null  object \n",
      " 5   SellerG        10864 non-null  object \n",
      " 6   Date           10864 non-null  object \n",
      " 7   Distance       10864 non-null  float64\n",
      " 8   Postcode       10864 non-null  float64\n",
      " 9   Bedroom2       10864 non-null  float64\n",
      " 10  Bathroom       10864 non-null  float64\n",
      " 11  Car            10864 non-null  float64\n",
      " 12  Landsize       10864 non-null  float64\n",
      " 13  BuildingArea   5779 non-null   float64\n",
      " 14  YearBuilt      6665 non-null   float64\n",
      " 15  CouncilArea    10862 non-null  object \n",
      " 16  Lattitude      10864 non-null  float64\n",
      " 17  Longtitude     10864 non-null  float64\n",
      " 18  Regionname     10864 non-null  object \n",
      " 19  Propertycount  10864 non-null  float64\n",
      "dtypes: float64(11), int64(1), object(8)\n",
      "memory usage: 1.7+ MB\n"
     ]
    }
   ],
   "source": [
    "train_data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "a7eea2a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Rooms</th>\n",
       "      <th>Distance</th>\n",
       "      <th>Postcode</th>\n",
       "      <th>Bedroom2</th>\n",
       "      <th>Bathroom</th>\n",
       "      <th>Car</th>\n",
       "      <th>Landsize</th>\n",
       "      <th>BuildingArea</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>Lattitude</th>\n",
       "      <th>Longtitude</th>\n",
       "      <th>Propertycount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>2.5</td>\n",
       "      <td>3067.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>202.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-37.7996</td>\n",
       "      <td>144.9984</td>\n",
       "      <td>4019.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2.5</td>\n",
       "      <td>3067.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>156.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>1900.0</td>\n",
       "      <td>-37.8079</td>\n",
       "      <td>144.9934</td>\n",
       "      <td>4019.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>3067.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>1900.0</td>\n",
       "      <td>-37.8093</td>\n",
       "      <td>144.9944</td>\n",
       "      <td>4019.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>3067.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-37.7969</td>\n",
       "      <td>144.9969</td>\n",
       "      <td>4019.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>2.5</td>\n",
       "      <td>3067.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>142.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>-37.8072</td>\n",
       "      <td>144.9941</td>\n",
       "      <td>4019.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Rooms  Distance  Postcode  Bedroom2  Bathroom  Car  Landsize  BuildingArea  \\\n",
       "0      2       2.5    3067.0       2.0       1.0  1.0     202.0           NaN   \n",
       "1      2       2.5    3067.0       2.0       1.0  0.0     156.0          79.0   \n",
       "2      3       2.5    3067.0       3.0       2.0  0.0     134.0         150.0   \n",
       "3      3       2.5    3067.0       3.0       2.0  1.0      94.0           NaN   \n",
       "4      4       2.5    3067.0       3.0       1.0  2.0     120.0         142.0   \n",
       "\n",
       "   YearBuilt  Lattitude  Longtitude  Propertycount  \n",
       "0        NaN   -37.7996    144.9984         4019.0  \n",
       "1     1900.0   -37.8079    144.9934         4019.0  \n",
       "2     1900.0   -37.8093    144.9944         4019.0  \n",
       "3        NaN   -37.7969    144.9969         4019.0  \n",
       "4     2014.0   -37.8072    144.9941         4019.0  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cde419c9",
   "metadata": {},
   "source": [
    "For instance, the pipeline below will use SimpleImputer() to replace missing values in the data, before using RandomForestRegressor() to train a random forest model to make predictions. We set the number of trees in the random forest model with the n_estimators parameter, and setting random_state ensures reproducibility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "593b005a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "my_pipeline = Pipeline(steps=[\n",
    "    ('preprocessor', SimpleImputer()),\n",
    "    ('model', RandomForestRegressor(n_estimators=50, random_state=0))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abc49c60",
   "metadata": {},
   "source": [
    "You have also learned how to use pipelines in cross-validation. The code below uses the cross_val_score() function to obtain the mean absolute error (MAE), averaged across five different folds. Recall we set the number of folds with the cv parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7f1e79bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average MAE score: 209652.44360269542\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# Multiply by -1 since sklearn calculates *negative* MAE\n",
    "scores = -1 * cross_val_score(my_pipeline, X, y,\n",
    "                              cv=5,\n",
    "                              scoring='neg_mean_absolute_error')\n",
    "\n",
    "print(\"Average MAE score:\", scores.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e359543f",
   "metadata": {},
   "source": [
    "function get_score() that reports the average (over three cross-validation folds) MAE of a machine learning pipeline that uses:\n",
    "\n",
    "1. the data in X and y to create folds,\n",
    "2. SimpleImputer() (with all parameters left as default) to replace missing values, and\n",
    "3. RandomForestRegressor() (with random_state=0) to fit a random forest model.\n",
    "\n",
    "\n",
    "The n_estimators parameter supplied to get_score() is used when setting the number of trees in the random forest model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9cdd37d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_score(n_estimators):\n",
    "    my_pipeline = Pipeline(\n",
    "        steps=[\n",
    "            ('imputer', SimpleImputer()),\n",
    "            ('model', RandomForestRegressor(n_estimators=n_estimators, random_state=0))\n",
    "        ]\n",
    "    )\n",
    "    # Replace this body with your own code\n",
    "    score = -1*cross_val_score(my_pipeline ,X, y,\n",
    "                                cv = 3,\n",
    "                               scoring='neg_mean_absolute_error')\n",
    "    return score.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8013bba",
   "metadata": {},
   "source": [
    "Now, you will use the function that you defined in Step 1 to evaluate the model performance corresponding to eight different values for the number of trees in the random forest: 50, 100, 150, ..., 300, 350, 400.\n",
    "\n",
    "Store your results in a Python dictionary results, where results[i] is the average MAE returned by get_score(i)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "ba75f9b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{50: 212199.73510690444, 100: 210461.39694324185, 150: 209774.6374935497, 200: 209613.91470599835, 250: 209694.13550690585, 300: 209927.34020534402, 350: 209893.8286538119, 400: 209556.28714783164}\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "iterer = np.arange(50, 450, step=50)\n",
    "\n",
    "results = {x: get_score(x) for x in iterer}\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "178e0547",
   "metadata": {},
   "source": [
    "Visualize your results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "493afdf0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD4CAYAAADy46FuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmkUlEQVR4nO3de3xV9Znv8c+TCwGScMkFCNdwU27BW0Cs1rbaKoIdL6MW2yqd2tJp7bzqHDuttmc6tqed0545R9vOtFpbnGKrVetlasVqFamoo2BABAGByB25JOEWLgkkec4f+xfYprlsINm3fN+v137ttX7rt9Z+sl6BJ79n/fZa5u6IiIi0JSPRAYiISHJTohARkXYpUYiISLuUKEREpF1KFCIi0q6sRAfQ2YqKiry0tDTRYYiIpJSlS5dWu3txa9vSLlGUlpZSUVGR6DBERFKKmW1ua5tKTyIi0i4lChERaZcShYiItEuJQkRE2qVEISIi7VKiEBGRdilRiIhIu5QogvW7avneH1dT39CY6FBERJKKEkWwbe8RHnhtI69VVic6FBGRpKJEEVw4pog+PbN4ZsWORIciIpJUlCiCHlkZXDZxEC+s3qXyk4hIFCWKKDMnl1Bb18Cr61V+EhFppkQR5cLRkfLTfJWfRESOU6KI0iMrg8tVfhIR+QAlihZmTi6htr6BV9ap/CQiAkoUf+XCMUX07ZXNsytVfhIRASWKv5KdmcHlEwfywupd1B1T+UlERImiFTMnD46UnzT7SUREiaI1HxpdSL/eKj+JiIASRauyMzO4fMIglZ9ERIghUZjZMDNbaGarzWyVmX0ttF8f1pvMrDyq/yfMbKmZrQzvl0RtOy+0V5rZT83MQnuBmb1gZuvDe//QbqFfpZmtMLNzO/8UtG7G5BIO1jewaF1VvD5SRCQpxTKiaABud/cJwDTgVjObALwDXAssatG/Gviku5cBs4HfRG27F/giMDa8pof2O4AF7j4WWBDWAa6I6jsn7B8XKj+JiER0mCjcfYe7LwvLtcAaYIi7r3H3ta30f8vd3w+rq4BeZpZjZiVAH3d/w90deBC4OvS7CpgXlue1aH/QI94A+oXjdLnszAymTxzEi2t2q/wkIt3aSV2jMLNS4BxgcYy7/C2wzN3rgSHAtqht20IbwEB3b/7TfScwMCwPAba2sU90XHPMrMLMKqqqOq9UNKNM5ScRkZgThZnlAU8At7n7gRj6TwR+BHzpZAIKow0/yX3ud/dydy8vLi4+mV3bdcHoQvr3zma+yk8i0o3FlCjMLJtIknjI3Z+Mof9Q4CngZnd/LzRvB4ZGdRsa2gB2NZeUwvvuqH2GtbFPl8vOzGD6pEG8qNlPItKNxTLryYC5wBp3vzuG/v2A+cAd7v5ac3soLR0ws2nhmDcDfwibnyZy4ZvwHt1+c5j9NA3YH1WiiosZZSUcOtrIyyo/iUg3FcuI4kLgJuASM1seXjPM7Boz2wZcAMw3s+dD/68CY4DvRPUfELZ9BfgVUAm8B/wptP8Q+ISZrQc+HtYBngU2hP6/DPvH1QWjQvlJtx4XkW4qq6MO7v4qYG1sfqqV/t8Hvt/GsSqASa201wCXttLuwK0dxdiVskL56Q/L36fuWCM9szMTGY6ISNzpm9kxmFk2mMNHG/nLWpWfRKT7UaKIwbRRBRTk9tDsJxHplpQoYpCVGXny3YI1mv0kIt2PEkWMrpxcEspPuzvuLCKSRpQoYnT+yAIKc3vwjGY/iUg3o0QRo6zMDC6fNIiX3t3NkaMqP4lI96FEcRKuLFP5SUS6HyWKkzC1ufyk2U8i0o0oUZyE5i/fvbRG5ScR6T6UKE7SzMklHDnWyEKVn0Skm1CiOEnnjyykKE9fvhOR7kOJ4iRlZtjx8tPhow2JDkdEpMspUZyCmWWDI+Wnd3XvJxFJf0oUp2DqyAKK8nJ4VuUnEekGlChOQWaGccWkQSx4d5fKTyKS9pQoTtHMySXUHWvipXc1+0lE0psSxSmaUqryk4h0D0oUpygzw5hRFrn3k8pPIpLOlChOw4wylZ9EJP0pUZyGKaUFFOfnMF+3HheRNKZEcRoyM4wZ4dbjh+pVfhKR9KREcZpmlJVQ36Dyk4ikLyWK01ReWsAAlZ9EJI0pUZymyOynEhauVflJRNKTEkUnaC4/LVD5SUTSkBJFJygf0T+Un95PdCgiIp1OiaITZBwvP1VxUOUnEUkzShSdZObkEo42NLFgza5EhyIi0qk6TBRmNszMFprZajNbZWZfC+3Xh/UmMyuP6l8Y+h80s/9ocay/mNlaM1seXgNCe46ZPWpmlWa22MxKo/a5M7SvNbPLO+0n72TnDe/PwD6a/SQi6Scrhj4NwO3uvszM8oGlZvYC8A5wLfCLFv3rgH8GJoVXS59x94oWbbcAe919jJnNAn4EfMrMJgCzgInAYOBFMzvD3Rtj/PniJiPDuGJSCQ8v2cLB+gbycmI5tSIiya/DEYW773D3ZWG5FlgDDHH3Ne6+tpX+h9z9VSIJI1ZXAfPC8uPApWZmof0Rd693941AJTD1JI4bV1eq/CQiaeikrlGEktA5wOLT+Mz/DGWnfw7JAGAIsBXA3RuA/UBhdHuwLbQlpXOH92dQn548o/KTiKSRmBOFmeUBTwC3ufuBU/y8z7h7GfDh8LrpFI/TMrY5ZlZhZhVVVYl7jnVGhnFF2SBeXldFbd2xhMUhItKZYkoUZpZNJEk85O5PnuqHufv28F4LPMyJMtJ2YFj4rCygL1AT3R4MDW0tj3u/u5e7e3lxcfGphtcpTpSf9OU7EUkPscx6MmAusMbd7z7VDzKzLDMrCsvZwJVELogDPA3MDsvXAS+5u4f2WWFW1EhgLLDkVGOIh3OGqfwkIukllqk5FxIpEa00s+Wh7VtADvDvQDEw38yWu/vlAGa2CegD9DCzq4HLgM3A8yFJZAIvAr8Mx5sL/MbMKoE9RGY64e6rzOwxYDWR2Ve3JuOMp2jNX7777Rubqa07Rn7P7ESHJCJyWizyh3v6KC8v94qKlrNv42vp5r387b3/zT2fOotrzhma0FhERGJhZkvdvby1bfpmdhc4Z1g/Svr2ZP6KnYkORUTktClRdIHm8tOidVUc0OwnEUlxShRdZObkEo42NvHian35TkRSmxJFFzlnWD8G9+3Jsys1+0lEUpsSRRcxay4/VbP/iMpPIpK6lCi6kMpPIpIOlCi60NnD+jGkXy+Vn0QkpSlRdKFI+WkQi9ZXqfwkIilLiaKLzSgr4Vijq/wkIilLiaKLNZef5qv8JCIpSomii5kZMyeX8IrKTyKSopQo4qC5/PSCyk8ikoKUKOLgrKF9I+WnFe8nOhQRkZOmRBEHZsaVk0t4tbKa/YdVfhKR1KJEESfN5ac/r9YdZUUktShRxMnkoX0Z2l+zn0Qk9ShRxImZMbOshFfXq/wkIqlFiSKOZk4uoaHJeV7lJxFJIUoUcVQ2pC/DCnoxf4XKTyKSOpQo4qj51uOvVVaz7/DRRIcjIhITJYo4u7JsMA1Nzp9X6ct3IpIalCjibNKQPgwv6M0zmv0kIilCiSLOmstP/11Zzd5DKj+JSPJTokiAK8PsJ335TkRSgRJFAkwcHCk/zV+pRCEiyU+JIgGabz3+mspPIpIClCgSZGZZCY1NzvOrNKoQkeSmRJEgEwf3YURhb937SUSSnhJFgjTf++m/36thj8pPIpLEOkwUZjbMzBaa2WozW2VmXwvt14f1JjMrj+pfGPofNLP/aHGs88xspZlVmtlPzcxCe4GZvWBm68N7/9BuoV+lma0ws3M798dPrJmTVX4SkeQXy4iiAbjd3ScA04BbzWwC8A5wLbCoRf864J+Br7dyrHuBLwJjw2t6aL8DWODuY4EFYR3giqi+c8L+aWNCSR9KC3vzrMpPIpLEOkwU7r7D3ZeF5VpgDTDE3de4+9pW+h9y91eJJIzjzKwE6OPub7i7Aw8CV4fNVwHzwvK8Fu0PesQbQL9wnLTQPPtJ5ScRSWYndY3CzEqBc4DFp/BZQ4BtUevbQhvAQHdv/rN6JzAwap+tbewTHdccM6sws4qqqqpTCC1xZmj2k4gkuZgThZnlAU8At7n7ga4KKIw2/CT3ud/dy929vLi4uIsi6xoTSvowsihXtx4XkaQVU6Iws2wiSeIhd3/yFD9rOzA0an1oaAPY1VxSCu+7o/YZ1sY+aeHE7Kdqag7WJzocEZG/EsusJwPmAmvc/e5T/aBQWjpgZtPCMW8G/hA2Pw3MDsuzW7TfHGY/TQP2R5Wo0saMshKaHJ7XrcdFJAnFMqK4ELgJuMTMlofXDDO7xsy2ARcA883s+eYdzGwTcDfwOTPbFmZJAXwF+BVQCbwH/Cm0/xD4hJmtBz4e1gGeBTaE/r8M+6ed8SX5jCrKZf7K9xMdiojIX8nqqEOYwWRtbH6qjX1K22ivACa10l4DXNpKuwO3dhRjqmue/fSzhZVUH6ynKC8n0SGJiBynb2YniRPlJ81+EpHkokSRJMYNymdUsWY/iUjyUaJIEmbGlWUlvLGhhmrNfhKRJKJEkURmTI6Un557R+UnEUkeShRJ5MyB+YxW+UlEkowSRRJp/vLd4o01VNWq/CQiyUGJIsnMnDw4Un7S7CcRSRJKFEnmjIF5jBmQx/wV+vKdiCQHJYokY2bMKCthycY97K6t63gHEZEupkSRhK4Ms5+e1+wnEUkCShRJ6IyB+YwdkMd8PflORJKAEkWSmlFWwmKVn0QkCShRJKmZk0twfflORJKAEkWSOl5+0pfvRCTBlCiS2MzJJSzZtIfdB1R+EpHEUaJIYjPLIuWnP6n8JCIJpESRxMYOzOeMgZr9JCKJpUSR5GaWDebNTXvYpfKTiCSIEkWSmzl5UKT8pFGFiCSIEkWSGzMgnzMH5vPsSl2nEJHEUKJIATMnl/DmZpWfRCQxlChSwIzm2U8qP4lIAihRpIAxA/IYNyhfs59EJCGUKFLEzLIS3ty0l537VX4SkfhSokgRMyaXAPCndzSqEJH4UqJIEaOLQ/lJ934SkThTokghV04uoWLzXnbsP5LoUESkG1GiSCEzykL5Sd+pEJE46jBRmNkwM1toZqvNbJWZfS20Xx/Wm8ysvMU+d5pZpZmtNbPLo9o3mdlKM1tuZhVR7QVm9oKZrQ/v/UO7mdlPw7FWmNm5nfejp55RxXmML+mj2U8iElexjCgagNvdfQIwDbjVzCYA7wDXAouiO4dts4CJwHTg52aWGdXlY+5+trtHJ5c7gAXuPhZYENYBrgDGhtcc4N6T/PnSzpWTS1iq8pOIxFGHicLdd7j7srBcC6wBhrj7Gndf28ouVwGPuHu9u28EKoGpHXzMVcC8sDwPuDqq/UGPeAPoZ2YlHcWczprLT7qlh4jEy0ldozCzUuAcYHE73YYAW6PWt4U2AAf+bGZLzWxOVJ+B7t5cT9kJDIzhWNFxzTGzCjOrqKqqivXHSUkji3KZUNKH+SveT3QoItJNxJwozCwPeAK4zd0PnOLnXeTu5xIpKd1qZhe37ODuTiShxMzd73f3cncvLy4uPsXQUsfMySUs27KP9/ep/CQiXS+mRGFm2USSxEPu/mQH3bcDw6LWh4Y23L35fTfwFCdKUruaS0rhfXdHx+rOmstP+k6FiMRDLLOeDJgLrHH3u2M45tPALDPLMbORRC5ELzGzXDPLD8fMBS4jckG8eZ/ZYXk28Ieo9pvD7KdpwP6oElW3NbIolyml/fnpgvWs21Wb6HBEJM3FMqK4ELgJuCRMa11uZjPM7Boz2wZcAMw3s+cB3H0V8BiwGngOuNXdG4lcd3jVzN4GlgDz3f258Bk/BD5hZuuBj4d1gGeBDUQuiP8S+Mrp/8jp4SezzqFnj0w+/+s3qTlYn+hwRCSNWeSSQPooLy/3ioqKjjumgbe37uNT97/OxMF9eegL59MzO7PjnUREWmFmS1t8beE4fTM7hZ01rB9333A2Szfv5ZtPrCDdkr6IJAclihQ3o6yEf7r8TP6w/H1+uqAy0eGISBrKSnQAcvq+8tHRbKg6xD0vrmNkcS5/c9bgRIckImlEI4o0YGb867WTmFpawNd//zbLtuxNdEgikkaUKNJETlYm9910HiV9ezLnwQq27jmc6JBEJE0oUaSRgtwezJ09haMNTXxhXgW1dccSHZKIpAElijQzZkAe9372PN6rOshXH36LhsamRIckIilOiSINXTimiP919SReXlfF9+evSXQ4IpLiNOspTd04dTgbqg7yy1c2Mqo4l5svKE10SCKSopQo0tgdV4xnY/Vh7np6FcMLevPRMwckOiQRSUEqPaWxzAzjJ7POZtygPnz14bdYu1M3EBSRk6dEkeZyc7KY+7lyeocbCFbrBoIicpKUKLqBkr69mDt7CjWH6pnzYAV1xxoTHZKIpBAlim6ibGhffvyps1m2ZR/feFw3EBSR2ClRdCPTJ5Xwjeln8vTb7/PjF9cnOhwRSRGa9dTNfPkjo9lYdYifLFjPqOJcrjp7SKJDEpEkpxFFN2Nm/OCaMs4fWcA/Pb6CpZv3JDokEUlyShTdUI+sDO777HkM7tuTOQ8u1Q0ERaRdShTdVP/cHsz93BSONTbx+V+/yQHdQFBE2qBE0Y2NLs7jvpvOY2P1Id1AUETapETRzX1odBE/uGYSi9ZV8b1nVic6HBFJQpr1JHxqynA2VB3iF4s2MKool89dODLRIYlIElGiEAC+MX0cG6oP8b1nVjOiMJePjdMNBEUkQqUnAU7cQHB8SR/+4Xdv8e7OA4kOSUSShBKFHNe7RxZzZ08hNyeTW35dQVWtbiAoIkoU0sKgvj2ZO3sKew4d5Yu6gaCIoEQhrZg0pC/3fOps3t62j6///m2amnQDQZHuTIlCWjV90iC+OX0cz6zYwY8X6AaCIt2ZZj1Jm7508Sg2VB3kpwvWM6ool6vP0Q0ERbqjDkcUZjbMzBaa2WozW2VmXwvt14f1JjMrb7HPnWZWaWZrzezyqPbpoa3SzO6Iah9pZotD+6Nm1iO054T1yrC9tNN+cumQmfH9q8uYNqqAbzy+gopNuoGgSHcUS+mpAbjd3ScA04BbzWwC8A5wLbAounPYNguYCEwHfm5mmWaWCfwMuAKYANwY+gL8CLjH3ccAe4FbQvstwN7Qfk/oJ3HUfAPBIf17Mec3S9lSoxsIinQ3HSYKd9/h7svCci2wBhji7mvcfW0ru1wFPOLu9e6+EagEpoZXpbtvcPejwCPAVWZmwCXA42H/ecDVUceaF5YfBy4N/SWO+vXuwQOfm0Jjk/P5ebqBoEh3c1IXs0Pp5xxgcTvdhgBbo9a3hba22guBfe7e0KL9A8cK2/eH/i3jmmNmFWZWUVVVdTI/ksRoZFEu9332PDbXHOLWh5bpBoIi3UjMicLM8oAngNvcPam+tuvu97t7ubuXFxcXJzqctHXB6EJ+cE0Zr6yv5q4/rtJzt0W6iZhmPZlZNpEk8ZC7P9lB9+3AsKj1oaGNNtprgH5mlhVGDdH9m4+1zcyygL6hvyTIDeXDeK/qIL94eQOjivL4/EW6gaBIuotl1pMBc4E17n53DMd8GpgVZiyNBMYCS4A3gbFhhlMPIhe8n/bIn6ULgevC/rOBP0Qda3ZYvg54yfVnbMJ98/JxXD5xIN+fv5qX3t2V6HBEpItZR//vmtlFwCvASqC5MP0tIAf4d6AY2Acsd/fLwz7fBj5PZMbUbe7+p9A+A/gxkAk84O4/CO2jiFzcLgDeAj7r7vVm1hP4DZHrInuAWe6+ob14y8vLvaKiIvYzIKfk8NEGbvjF62ysOsTjX/4Q40v6JDokSXLb9h7mqWXbqa1vIC8ni/yeWeT3zCYvJ4s+YTm/ZxZ5PSPbcrIyEx1yt2JmS929vNVt6fYHuhJF/OzcX8dVP3uVTDP+66sXMiC/Z6JDkiTT0NjEgnd387slW3h5XWSiSU5WBnXHOp4M0SMrgz49s0JSyQ6JJYu8nMhyc3JpTizH+0T1790jE02UjI0ShXSZd7bv5/r7XueMQfk8OmcaPbP1V6DA9n1HeHTJFh6t2MquA/UM7JPDp8qHccOUYQzt35ujDU0crG/gYF0DB+qOUVvXQG3dMQ7WNxxfrq1roDZq/WDdB7cdPNpAR/99ZRgfSDR9PpBY/npEEz3Sye+ZRVFeDr16dI/faSUK6VJ/XrWTL/12KTMmlfDvN55DRob+guuOGhqbWLi2iocXb+YvYfTwkTOK+fTU4VwybgBZmZ17a7mmJufQ0ebk0cDB+mMcqGs7sUSSzrGo/pH1Y41t/x/YMzuDL108mr//yOi0TxjtJQrd60lO22UTB3HnFeP412ffZVRxLrdfdmaiQ5I4en/fER55cyuPvbmVnQfqGJCfw1c/NoYbyocxrKB3l31uRoaFv/yzT/kY7k59QxMHPpBYQkKpb+DltVX8ZMF6Hl+6jTtnjGNmWUm3LGVpRCGdwt2544mVPFqxlbtvOItrzx2a6JCkCzU2OQvDtYeFa3fjwMVji7lx6nAuHT+A7E4ePSTS4g013PXH1azZcYCpIwv4l09OYOLgvokOq9Op9CRxcbShidkPLGHp5r089MXzmVJakOiQpJPt2H+ER9/cyqNvbmXH/jqK83O4oXwos6YM79LRQ6I1NjmPvLmF//v8WvYfOcasqcO5/RNnUJiXk+jQOo0ShcTN/sPHuObnr7H38FH+69YLGVGYm+iQ5DQ1Njkvr9vNw4u38NK7u2ly+PDYIj5z/nAuHT8wrUYPHdl/+Bg/XrCOB1/fTG6PTG77+BncdMGItDgHShQSVxurD3HNz1+jMLcHT37lQvr2OvUasiTOzv11YfSwhff311GUd2L0MLwwfUcPsVi/q5bvPbOaV9ZXM2ZAHt+5cgIXn5Hatw9SopC4e2NDDTfNXcz5Iwv5z7+bkhZ/cXUHjU3OonVVPLR4Cy+9u+v46OHTU4fz8Qnda/TQEXfnxTW7+f781WyuOczHxw/kf84cT2lRao6ilSgkIX5fsZV/enwFnz5/OD+4elK3nC2SKnYdqOOxN7fyyJtb2b7vCEV5Pbi+fBizpgxT+bAD9Q2NPPDqJv7jpfUca3Q+f9FIvnrJGPJyUmtSqabHSkJcXz6MDdWHuPcv7zGyMJcvfHikkkUSaWxyXllfxcOLt7Dg3d00NjkXjSniWzPG84kJA+mRpdFDLHKyMvnyR0fzt+cO4UfPreW+l9/jiWXb+Ob0cVx7zpC0+F6RRhTSpZqanK88tIznVu1kcN+eXHxGMR8eW8xFY4ro21vXLhJh94E6HqvYyu+WREYPhbk9uK58KDdOGZ6yZZNk8taWvdz1x9W8vXUfZw3rx12fnMA5w/snOqwOqfQkCVV3rJEnl21n0boqXquspra+gQyDs4b14+KxxVx8RhFnDe3X6d/clROampxXKqt5ePFmXlwTGT18aHQhnz5/OJdNGKTRQydranKeems7P3zuXapq67n23CHcMX0cA/ok7/3QlCgkaTQ0NrF86z4Wrati0fpq3t62D3fo0zOLC8cUcfEZxVx8RjFD+vVKdKhpYXdtHb+v2Mbvlmxh294jFOT24PrzhjJr6nBGavTQ5Q7WN/CzhZXMfWUj2ZnGrZeM4ZaLRiblnXGVKCRp7Tt8lFcrqyOJY101Ow/UATCqOJeLxxbzkTOKOX9UAb176HJarJqanNfeq+bhxVt4YfUuGpqcC0aF0cPEgUn5n1S621R9iB88u4YXVu9iRGFvvh2uAyXTNTslCkkJ7k7l7oO8vK6KV9ZX88aGGuobmuiRmUF5af/IaGNsMeNL8pPqH1iyqKqt5/dLt/LIkq1s2XOY/r2zj89cGlWcl+jwBHhlfRXf/eNqKncf5MNji/jOlRMYOzA/0WEBShSSouqONfLmpj3HRxtrd9UCUJyfw4dDmeqisUUUpdFtFE7WkaONVGzew++WbOHPqyKjh2mjCrhx6nCmTxqk0UMSOtbYxG/f2Mw9L6zj0NFGbpo2gn/8+BkJn9yhRCFpYdeBuuPXNl5dX8Xew8cAmDSkDxePjcymOm9E/7S7MLvv8FE21xxm857DbK4+xOY9h9lSc5hNNYfYXVsPQL/e2Vx37lBuPH84ozV6SAk1B+u5+4V1/G7JFvr2yub2y87kxqnDyUzQdFolCkk7jU3Oqvf3Hx9tLNuyl4YmJ7dHJheMLjw+Dbe0sHfSl6mampzdtfVsrokkgc01h9hcc5gtew6zueYw+48c+0D/gX1yGFGQy4jC3owo7M2YAfl89MxiPTQqRa16fz/f/eNqlmzcw/iSPvzLJycwbVRh3ONQopC0V1t3jNffq2HR+kji2LLnMADDCnqFKbjFfGh04Wk9u+B0HGts4v19R9hUc5gtIRFsqjnMlj2H2LLn8AceDZqZYQzt34vhBZFEUFqYG5Yj7+n+AJ3uyN15duVO/vXZNWzfd4SZZSXcOWMcQ/vH755aShTS7WyqPsQr66t4eV01r79XzaGjjWRmGOcO73c8cUwa0rdTh/lHjjaGUcChUCoK7zWH2b7vCI1NJ/6t9czOOP6f/4iQEEYURkYJg/v10j2VuqkjRxu5f9EG7n25Enf40kdG8+U4PV1PiUK6taMNTSzbspdXwmhj5fb9APTvnX3iuxtjixnUt+MvQ8VyvaBZ317ZjCjszfCCMCoo7M2Igt6UFuUyID8n6Utikjjb9x3hfz+7hmdW7GBw357cOWM8V07u2qfrKVGIRKk5WB++u1HNovVVVIX/4M8cmM+Hx0YSR05WxgeSQEfXC4YX9qa0sDfDo0YI/Xr3SMSPJ2lkycY93PX0KlbvOMDU0gK+88kJTBrSNU/XU6IQaYO78+7O2uOjjSWb9nC04YPXC4b063X8wvGJi8i6XiDx0djkPFaxlX97fi17Dx9l1pThfP2yzn+6nhKFSIyOHI18d8OBUl0vkCSy/8gxfvLieh58fRO9wtP1bu7Ep+spUYiIpInK3bV875k1LFpXxejiXL7zyYl8pBOertdeotCfSiIiKWTMgHzm/d0U5s4up7HJmf3AEr4w7002Vh/qss9UohARSTFmxqXjB/L8P17MHVeM4/X3arjsnpf51SsbuuTzdEtOEZEUlZOVyd9/ZDTXnjuEf3tuLcMLuuYLeh2OKMxsmJktNLPVZrbKzL4W2gvM7AUzWx/e+4f2/mb2lJmtMLMlZjYp6libzGylmS03s4qo9raOZWb2UzOrDMc7t/NPgYhIahuQ35N/u/4sLps4qEuOH0vpqQG43d0nANOAW81sAnAHsMDdxwILwjrAt4Dl7j4ZuBn4SYvjfczdz25x0aStY10BjA2vOcC9J/sDiojI6ekwUbj7DndfFpZrgTXAEOAqYF7oNg+4OixPAF4K/d8FSs1sYAcf09axrgIe9Ig3gH5mVtLxjyUiIp3lpC5mm1kpcA6wGBjo7jvCpp1AczJ4G7g29J8KjACGhm0O/NnMlprZnKhDt3WsIcDWqH7bQlvLuOaYWYWZVVRVVZ3MjyQiIh2IOVGYWR7wBHCbux+I3uaRL2M0fyHjh0T+8l8O/APwFtAYtl3k7ucSKSndamYXt/ycFseKibvf7+7l7l5eXHz684lFROSEmGY9mVk2kSTxkLs/GZp3mVmJu+8I5aDdACGJ/F3Yz4CNwIawbXt4321mTwFTgUVtHQvYDgyLCmVoaBMRkTiJZdaTAXOBNe5+d9Smp4HZYXk28IfQv5+ZNd8N7QvAInc/YGa5ZpYf+uQClwHvtHes0H5zmP00DdgfVaISEZE4iGVEcSFwE7AylJMgMrPph8BjZnYLsBm4IWwbD8wzMwdWAbeE9oHAU+E2uVnAw+7+XNjW1rGeBWYAlcBhwkhFRETiR/d6EhGR7nVTQDOrIjIqORVFQHUnhtPVUineVIoVUiveVIoVUiveVIoVTi/eEe7e6mygtEsUp8PMKtrKqMkoleJNpVghteJNpVghteJNpVih6+LVTQFFRKRdShQiItIuJYoPuj/RAZykVIo3lWKF1Io3lWKF1Io3lWKFLopX1yhERKRdGlGIiEi7lChERKRd3TpRtPYgpbYeopSg+B4ws91m9k5UW1I+5KmNWO8ys+3h/C43sxlR2+4Msa41s8vjHOvJPowr0ee2rXiT7vyaWU+LPLDs7RDrd0P7SDNbHGJ6tPk2P2aWE9Yrw/bSeMXaQby/NrONUef27NCe0N+FEEOmmb1lZs+E9a4/t+7ebV/AJqCoRdv/Ae4Iy3cAP0pgfBcD5wLvdBQfkVud/AkwIg+YWpwEsd4FfL2VvhOI3I4+BxgJvAdkxjHWEuDcsJwPrAsxJeu5bSvepDu/4RzlheVsIo8kmAY8BswK7fcBXw7LXwHuC8uzgEfjfG7bivfXwHWt9E/o70KI4X8ADwPPhPUuP7fdekTRhrYeohR37r4I2NOiOSkf8tRGrG25CnjE3evdfSORe3lN7bLgWvCTfxhXos9tW/G2JWHnN5yjg2E1O7wcuAR4PLS3PLfN5/xx4FKzyA3h4qGdeNuS0N8FMxsKzAR+FdaNOJzb7p4oWnuQUlsPUUoWp/WQpwT4ahiiPxBVxkuaWC22h3Ela7yQhOc3lEaWE3lcwAtERjT73L2hlXiOxxq27wcK4xVra/G6e/O5/UE4t/eYWU7LeIN4/y78GPgG0BTWC4nDue3uiaLdByl5ZMyWtPOHkz0+Is84Hw2cDewA/l9Co2nBYn8YV1JoJd6kPL/u3ujuZxN5fsxUYFxiI2pfy3jNbBJwJ5G4pwAFwDcTF2GEmV0J7Hb3pfH+7G6dKDzqQUpA84OUdjUPJe2DD1FKFm3Fl3QPeXL3XeEfYRPwS06UPxIeq7XzMK6wPanObWvxJvP5DfHtAxYCFxAp0TQ/1iA6nuOxhu19gZr4RhoRFe/0UO5zd68H/pPkOLcXAn9jZpuAR4iUnH5CHM5tt00U1vaDlNp6iFKySJmHPLWo3V7DBx9UNSvMyhgJjAWWxDGuk3oYFwk+t23Fm4zn18yKzaxfWO4FfILINZWFwHWhW8tz23zOrwNeCqO5uGgj3nej/mAwIjX/6HObkN8Fd7/T3Ye6eymRi9MvuftniMe57ewr8qnyAkYRmRnyNpEHLH07tBcCC4D1wItAQQJj/B2RksIxIrXHW9qKj8gsjJ8RqQevBMqTINbfhFhWhF/akqj+3w6xrgWuiHOsFxEpK60AlofXjCQ+t23Fm3TnF5gMvBViegf4TmgfRSRZVQK/B3JCe8+wXhm2j4rzuW0r3pfCuX0H+C0nZkYl9HchKu6PcmLWU5efW93CQ0RE2tVtS08iIhIbJQoREWmXEoWIiLRLiUJERNqlRCEiIu1SohARkXYpUYiISLv+P+A6zML7o++hAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.plot(list(results.keys()), list(results.values()))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7498381e",
   "metadata": {},
   "source": [
    "which value for n_estimators seems best for the random forest model?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba8c2f0b",
   "metadata": {},
   "source": [
    "200"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e8e9ba6",
   "metadata": {},
   "source": [
    "We explored one method for choosing appropriate parameters in a machine learning model.\n",
    "\n",
    "If like to learn more about [hyperparameter optimization](https://en.wikipedia.org/wiki/Hyperparameter_optimization), you're encouraged to start with grid search, which is a straightforward method for determining the best combination of parameters for a machine learning model. Thankfully, scikit-learn also contains a built-in function GridSearchCV() that can make your grid search code very efficient!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23c65e0a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8d03254f",
   "metadata": {},
   "source": [
    "#### Author: Piyush Kumar\n",
    "[Github](https://github.com/styles3544)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "647a1a18",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}