homework_02-.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "b8da29999fa548f09465d0b7e18732e9",
     "grade": false,
     "grade_id": "cell-7186dfefece144c8",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from nose.tools import *\n",
    "\n",
    "np.random.seed(24680)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write your imports in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "a9ff523820808ea082177fae7c6bf6c4",
     "grade": false,
     "grade_id": "cell-98c4a3a245e45b8a",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.metrics import f1_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ensemble Models and Support Vector Machines Lab\n",
    "## Training and comparing different algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, we'll work with the bank dataset. This time, the data preprocessing steps have been done for you.\n",
    "\n",
    "The goal is to try and improve our predictions (if they can be improved at all) using different types of algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Read the data (1 point)\n",
    "This time you only need to read the data. The indicator variables have been separated out for you.\n",
    "\n",
    "Read the dataset and save it in the variable `bank_data`. The target column is `y`. Use the variables `bank_attributes` and `bank_labels` to save the attributes (explanatory variables, features, predictors), and labels (`y`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "5332a28d728ce835574c4a823df35a67",
     "grade": false,
     "grade_id": "cell-990a0dfca3a695e6",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "bank_data = pd.read_csv(\"data/bank.csv\")\n",
    "bank_attributes = bank_data.drop(\"y\", axis = 1)\n",
    "bank_labels = bank_data.y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "fbb74ffbafaa9dbd29f5ecf74c4ea33d",
     "grade": true,
     "grade_id": "cell-dccbbb44b14b188f",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(bank_data)\n",
    "assert_is_not_none(bank_attributes)\n",
    "assert_is_not_none(bank_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Normalize the data (1 point)\n",
    "Because both forests and SVMs are sensitive to non-scaled data, we need to normalize our dataset first.\n",
    "\n",
    "Rescale all columns in `bank_attributes` so they have mean = 0 and variance = 1. You can either look at the `sklearn` docs or do this yourself. When you're ready, overwrite the `bank_attributes` column. Make sure that you don't lose the column names in the process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "c353a4a1516901f4222f00fc29885cfb",
     "grade": false,
     "grade_id": "cell-7cc2e9c90f8705c5",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "scaler = MinMaxScaler(feature_range=(-1,1))\n",
    "bank_attributes = scaler.fit_transform(bank_attributes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "ee58ea1d1a912cdf0b8e594249301c55",
     "grade": true,
     "grade_id": "cell-918a6111ee668331",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(bank_attributes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Split the data (1 point)\n",
    "Use the standard 70% / 30% split. Since this is a classification problem, be sure to stratify the split according to the `bank_labels`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "c2a9c6e60a741863517ff47df38195db",
     "grade": false,
     "grade_id": "cell-68893dc2052dd87c",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "bank_attributes_train, bank_attributes_test, bank_labels_train, bank_labels_test = train_test_split(bank_attributes, bank_labels, train_size = 0.7, test_size = 0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "abc82412e81ffd8b54d9eba9a4f5c943",
     "grade": true,
     "grade_id": "cell-215de695ca1c45ff",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(bank_attributes_train)\n",
    "assert_is_not_none(bank_labels_train)\n",
    "\n",
    "assert_is_not_none(bank_attributes_test)\n",
    "assert_is_not_none(bank_labels_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Prepare the cross-validation folds (1 point)\n",
    "Use a stratified k-fold cross-validation split, with $k = 5$. Fit it to the train data. Save the trained cross-validator to the variable `k_fold`.\n",
    "\n",
    "The data should already be shuffled. There's no need to shuffle it again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "c9f2a480f45546e6abaf83a538698a7f",
     "grade": false,
     "grade_id": "cell-ef1a3912d43342f1",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "k_fold = StratifiedKFold(n_splits=5)\n",
    "k_fold = k_fold.get_n_splits(bank_attributes_train, bank_labels_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "51421f786ff3b725b11aacb56e7643c8",
     "grade": true,
     "grade_id": "cell-8c28c6c421c7705a",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(k_fold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Decision Tree (2 points)\n",
    "Use cross-validation to train and optimize the hyperparameters for a decision tree classifier.\n",
    "\n",
    "Use grid search with the following grid:\n",
    "* `max_depth`: 1, 5, 7, 15, 20\n",
    "* `min_samples_leaf`: 2, 5, 10, 12\n",
    "* `max_leaf_nodes`: 5, 10, 20\n",
    "\n",
    "Use the most appropriate scoring metric (remember that accuracy doesn't work in this case because the data is highly imbalanced; we need something which combines precision and recall). Use the cross-validation splits you just created.\n",
    "\n",
    "Save the grid results in `grid_search`. Save the best classifier in `tree_classifier`.\n",
    "\n",
    "Optionally, you can print and / or visualize the cross-validation results and the best chosen parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "937595c41be7032fb2dc438a72db8a70",
     "grade": false,
     "grade_id": "cell-3603341dc3a0e1b4",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision tree; best score: 0.42489819920550354\n"
     ]
    }
   ],
   "source": [
    "tuned_params = [\n",
    "    {\"max_depth\": [1,5,7,15,20]},\n",
    "    {\"min_samples_leaf\" : [2,5,10,12]}, \n",
    "    {\"max_leaf_nodes\": [5,10,20]}\n",
    "]\n",
    "\n",
    "grid_search = GridSearchCV(DecisionTreeClassifier(),\n",
    "                           tuned_params,\n",
    "                           scoring=\"f1\",\n",
    "                           cv=k_fold).fit(bank_attributes_train,bank_labels_train)\n",
    "\n",
    "tree_classifier = grid_search.best_estimator_\n",
    "\n",
    "print(\"Decision tree; best score:\", grid_search.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "f0a46b7e34d4f16e0c62f00ecf18393a",
     "grade": true,
     "grade_id": "cell-8a9d43f3f99d1c09",
     "locked": true,
     "points": 2,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(grid_search)\n",
    "assert_is_not_none(tree_classifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Random Forest (1 point)\n",
    "Use cross-validation to train and optimize the hyperparameters for a random forest classifier. Use the same technique as before.\n",
    "\n",
    "Use the following grid:\n",
    "* `n_estimators`: 100, 200, 300 \n",
    "* `max_depth`: 20, 50, 100\n",
    "\n",
    "Note that this grid is on the small side but this is mainly due to performance reasons. Also note that the training will take some time.\n",
    "\n",
    "Save the grid results in `grid_search`. Save the best classifier in `forest_classifier`.\n",
    "\n",
    "Optionally, you can print and / or visualize the cross-validation results and the best chosen parameters.\n",
    "\n",
    "Due to the relatively slow training, we've chosen low values for the parameters. The performance of the random forest will be worse than the decision tree. This is not necessarily the case in general, it's due to the parameters we've chosen to try."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "671769b328ea72590acfd5212698421c",
     "grade": false,
     "grade_id": "cell-965717809a95920b",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random forest; best score: 0.3290520952956107\n"
     ]
    }
   ],
   "source": [
    "tuned_params = [\n",
    "    {\"n_estimators\": [100,200,300]},\n",
    "    {\"max_depth\" : [20,50,100]}\n",
    "]\n",
    "\n",
    "grid_search = GridSearchCV(RandomForestClassifier(n_estimators=100),\n",
    "                           tuned_params,\n",
    "                           scoring=\"f1\",\n",
    "                           cv=k_fold).fit(bank_attributes_train,bank_labels_train)\n",
    "forest_classifier = grid_search.best_estimator_\n",
    "\n",
    "print(\"Random forest; best score:\", grid_search.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "14e77aede1b15c71b9e6802012516c1d",
     "grade": true,
     "grade_id": "cell-5cdc884e24e42fec",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(grid_search)\n",
    "assert_is_not_none(forest_classifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Linear SVM (1 point)\n",
    "Use cross-validation to train and optimize the hyperparameters for a linear support vector machine. Use the same technique as before.\n",
    "\n",
    "Use the following grid:\n",
    "* `C`: 0.1, 0.5, 0.8, 1, 1.5, 2, 6, 10, 15, 20\n",
    "\n",
    "Note that we're choosing relatively small values for `C`. This is allowed because our data is normalized.\n",
    "\n",
    "Save the grid results in `grid_search`. Save the best classifier in `linear_svm_classifier`. There are many ways to create a linear SVM classifier. Look up the `sklearn` docs to choose the fastest one (in terms of performance).\n",
    "\n",
    "Optionally, you can print and / or visualize the cross-validation results and the best chosen parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "e19571c5e83d0b7d29d953b9ace63b42",
     "grade": false,
     "grade_id": "cell-dbc74911ea58e898",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear SVM; best score: 0.3793616904074903\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n"
     ]
    }
   ],
   "source": [
    "tuned_params = [\n",
    "    {\"C\" : [0.1,0.5,0.8,1,1.5,2,6,10,15,20]}\n",
    "]\n",
    "grid_search = GridSearchCV(LinearSVC(),\n",
    "                           tuned_params, \n",
    "                           scoring=\"f1\",\n",
    "                           cv=k_fold).fit(bank_attributes_train,bank_labels_train)\n",
    "linear_svm_classifier = grid_search.best_estimator_\n",
    "\n",
    "print(\"Linear SVM; best score:\", grid_search.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "375ffa7de497664fe4089895b7dca13b",
     "grade": true,
     "grade_id": "cell-7e5b16fe3d5deb03",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(grid_search)\n",
    "assert_is_not_none(linear_svm_classifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Gaussian SVM (1 point)\n",
    "Use cross-validation to train and optimize the hyperparameters for an SVM with a Gaussian kernel. Use the same technique as before.\n",
    "\n",
    "Use the following grid:\n",
    "* `C`: 10, 15, 20, 50, 200\n",
    "* `gamma`: 0.001, 0.01, 0.1, 0.2\n",
    "\n",
    "Note that this time we give larger values of `C` because the governing parameter here is `gamma`.\n",
    "\n",
    "Save the grid results in `grid_search`. Save the best classifier in `gaussian_svm_classifier`.\n",
    "\n",
    "Optionally, you can print and / or visualize the cross-validation results and the best chosen parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "e33ae1cd0f54a05b532e9df39d850a1a",
     "grade": false,
     "grade_id": "cell-06ab026bf98b2e69",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "c:\\users\\ivan\\appdata\\local\\programs\\python\\python37-32\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gaussian SVM; best score: 0.43750547893420283\n"
     ]
    }
   ],
   "source": [
    "tuned_params = [\n",
    "    {\"C\" : [10,15,20,50,200]},\n",
    "    {\"gamma\": [0.001,0.01,0.1,0.2]}\n",
    "]\n",
    "grid_search = GridSearchCV(SVC(gamma='auto'),\n",
    "                           tuned_params,\n",
    "                           cv=k_fold,\n",
    "                           scoring=\"f1\").fit(bank_attributes_train,bank_labels_train)\n",
    "gaussian_svm_classifier = grid_search.best_estimator_\n",
    "\n",
    "print(\"Gaussian SVM; best score:\", grid_search.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "8c81b3a26de11990ff354e812c694a31",
     "grade": true,
     "grade_id": "cell-0ba2a8ae7d45a6ad",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(grid_search)\n",
    "assert_is_not_none(gaussian_svm_classifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Compare performance on the testing data (1 point)\n",
    "Now that you've trained all your models, you've got to select the best one. This should be done on the testing data.\n",
    "\n",
    "Use the appropriate scoring metric to get the testing scores for all your models. Don't forget to pass the **testing**, not the training data. Save all scores.\n",
    "\n",
    "Choose the best classifier, based on these scores (the one with the highest test score). Of course, this is not enough. We need to look at ROC curves, track performance through other measures, debug the sources of variance in testing results, try more hyperparameters, etc. However, this is enough for an introductory lab :).\n",
    "\n",
    "Optionally, you can think of combining them into a boosted model but this is out of the scope of this lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "e04bd8cb12cc90f38bff21bb552e4f3d",
     "grade": false,
     "grade_id": "cell-cf416134ae2b2a62",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing scores:\n",
      "Decision tree: 0.48936170212765956\n",
      "Random forest: 0.3113207547169811\n",
      "Linear SVM: 0.39111111111111113\n",
      "Gaussian SVM: 0.4131147540983607\n"
     ]
    }
   ],
   "source": [
    "tree_classifier_score, forest_classifier_score, linear_svm_classifier_score, gaussian_svm_classifier_score = None, None, None, None\n",
    "\n",
    "tree_classifier_score = f1_score(bank_labels_test,tree_classifier.predict(bank_attributes_test))\n",
    "forest_classifier_score = f1_score(bank_labels_test,forest_classifier.predict(bank_attributes_test))\n",
    "linear_svm_classifier_score = f1_score(bank_labels_test,linear_svm_classifier.predict(bank_attributes_test))\n",
    "gaussian_svm_classifier_score = f1_score(bank_labels_test,gaussian_svm_classifier.predict(bank_attributes_test))\n",
    "\n",
    "print(\"Testing scores:\")\n",
    "print(\"Decision tree:\", tree_classifier_score)\n",
    "print(\"Random forest:\", forest_classifier_score)\n",
    "print(\"Linear SVM:\", linear_svm_classifier_score)\n",
    "print(\"Gaussian SVM:\", gaussian_svm_classifier_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "4454d84b6f6e3b4ab3d4cadf37b2e1c0",
     "grade": false,
     "grade_id": "cell-bdeed4c0d761fe0e",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "best_classifier = \"tree\" # Replace empty string with \"tree\", \"forest\", \"linear SVM\" or \"gaussian SVM\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "7b497b75ac0a01e38900e2d05760f637",
     "grade": true,
     "grade_id": "cell-495fc5ef0619deb6",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_not_equal(best_classifier, \"\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}