Titanic (SVM).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Binary classification using Support Vector Machines\n",
    "\n",
    "One of the classic public datasets used to demonstrate binary classification is the [Titanic dataset](https://www.kaggle.com/c/titanic), which lists 891 of the the passengers aboard the RMS Titanic when it sank on April 15, 1912. The dataset includes the name of each passenger as well as other information such as the fare class, the fare price, the person's age and gender, and whether that person survived the sinking of the ship. In this example, we will build a binary-classification model that predicts whether a passenger will survive. We will use a Support Vector Machine (SVM) classifier as the basis for the model.\n",
    "\n",
    "![](Images/titanic.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and prepare the dataset\n",
    "\n",
    "The first step is to load the dataset and prepare it for training a machine-learning model. One of the reasons the Titanic dataset is popular is that it provides ample opportunity for data scientists to practice their data-cleaning skills."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('Data/titanic.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine whether any of the columns in the dataset are missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 891 entries, 0 to 890\n",
      "Data columns (total 12 columns):\n",
      " #   Column       Non-Null Count  Dtype  \n",
      "---  ------       --------------  -----  \n",
      " 0   PassengerId  891 non-null    int64  \n",
      " 1   Survived     891 non-null    int64  \n",
      " 2   Pclass       891 non-null    int64  \n",
      " 3   Name         891 non-null    object \n",
      " 4   Sex          891 non-null    object \n",
      " 5   Age          714 non-null    float64\n",
      " 6   SibSp        891 non-null    int64  \n",
      " 7   Parch        891 non-null    int64  \n",
      " 8   Ticket       891 non-null    object \n",
      " 9   Fare         891 non-null    float64\n",
      " 10  Cabin        204 non-null    object \n",
      " 11  Embarked     889 non-null    object \n",
      "dtypes: float64(2), int64(5), object(5)\n",
      "memory usage: 83.7+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filter the dataset to remove columns that won't be used, one-hot-encode the \"Sex\" and \"Pclass\" columns, and remove rows with missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Sex_male</th>\n",
       "      <th>Pclass_1</th>\n",
       "      <th>Pclass_2</th>\n",
       "      <th>Pclass_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>26.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Survived   Age  Sex_female  Sex_male  Pclass_1  Pclass_2  Pclass_3\n",
       "0         0  22.0           0         1         0         0         1\n",
       "1         1  38.0           1         0         1         0         0\n",
       "2         1  26.0           1         0         0         0         1\n",
       "3         1  35.0           1         0         1         0         0\n",
       "4         0  35.0           0         1         0         0         1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df[['Survived', 'Age', 'Sex', 'Pclass']]\n",
    "df = pd.get_dummies(df, columns=['Sex', 'Pclass'])\n",
    "df.dropna(inplace=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to split the data into two datasets: one for training and one for testing. We'll use a stratified split to create a balanced distribution of samples in the training dataset and the testing dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "x = df.drop('Survived', axis=1)\n",
    "y = df['Survived']\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, stratify=y, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build and train an SVM model\n",
    "\n",
    "Support-vector classifiers (classifiiers that use Support Vector Machines, or SVMs) frequently do better at fitting data than classifiers that rely on logistic regression. Let's try a support-vector classifier on the same dataset and see if it fares better. By default, scikit's `SVC` class does not provide probabilities, but we can ask it to do so by setting its `probability` parameter to `True`. Doing so increases training time, but that's not a big deal here since the dataset is relatively small."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(probability=True, random_state=0)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "model = SVC(probability=True, random_state=0)\n",
    "model.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get an accuracy score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5944055944055944"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try a linear kernel and see if the model fares better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8321678321678322"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = SVC(kernel='linear', probability=True, random_state=0)\n",
    "model.fit(x_train, y_train)\n",
    "model.score(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cross-validate the model using five folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7801339505564858\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_validate\n",
    "\n",
    "scores = cross_validate(model, x, y, cv=5)\n",
    "print(scores['test_score'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimize the model (tune hyperparameters)\n",
    "\n",
    "We achieved a cross-validated score of 78% using a linear kernel, but we might be able to do better by trying other kernels with a variety of `C` and `gamma` values. Let's use `GridSearchCV` to see if we can improve the model. We'll start with `C` and `gamma` values that range from 0.1 to 10 with three different kernel types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 75 candidates, totalling 375 fits\n",
      "[CV] END ....................C=0.1, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END .......................C=0.1, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.1, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.1, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.1, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.1, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...................C=0.1, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.1, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.1, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.1, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.1, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END .......................C=0.1, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.1, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.1, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.1, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.1, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...................C=0.1, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.1, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.1, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.1, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.1, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=0.1, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.1, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.1, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.1, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.1, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END .........................C=0.1, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.1, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.1, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.1, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.1, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=0.1, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.1, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.1, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.1, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.1, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END .........................C=0.1, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.1, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.1, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.1, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.1, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .....................C=0.1, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=0.1, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=0.1, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=0.1, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=0.1, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=0.1, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=0.1, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END .......................C=0.5, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.5, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.5, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.5, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.5, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...................C=0.5, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.5, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.5, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.5, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.5, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END .......................C=0.5, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.5, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.5, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.5, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=0.5, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...................C=0.5, gamma=0.5, kernel=sigmoid; total time=   0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] END ...................C=0.5, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.5, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.5, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ...................C=0.5, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=0.5, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.5, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.5, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.5, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.5, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END .........................C=0.5, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.5, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.5, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.5, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.5, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=0.5, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.5, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.5, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.5, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=0.5, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END .........................C=0.5, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.5, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.5, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.5, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=0.5, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .....................C=0.5, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=0.5, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=0.5, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=0.5, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=0.5, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=0.5, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=0.5, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END .........................C=1, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=1, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=1, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=1, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=1, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .....................C=1, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=1, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=1, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=1, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=1, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END .........................C=1, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=1, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=1, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=1, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=1, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .....................C=1, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=1, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=1, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=1, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=1, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ........................C=1, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=1, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=1, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=1, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=1, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ...........................C=1, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=1, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=1, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=1, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=1, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ........................C=1, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=1, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=1, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=1, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=1, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ...........................C=1, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=1, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=1, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=1, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=1, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=5, kernel=sigmoid; total time=   0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] END .......................C=1, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .......................C=1, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END ..........................C=1, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=1, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=1, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=1, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=1, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=1, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=5, gamma=0.1, kernel=linear; total time=   0.2s\n",
      "[CV] END ......................C=5, gamma=0.1, kernel=linear; total time=   0.1s\n",
      "[CV] END ......................C=5, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=5, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=5, gamma=0.1, kernel=linear; total time=   0.0s\n",
      "[CV] END .........................C=5, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=5, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=5, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=5, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=5, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .....................C=5, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=5, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=5, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=5, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=5, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=5, gamma=0.5, kernel=linear; total time=   0.2s\n",
      "[CV] END ......................C=5, gamma=0.5, kernel=linear; total time=   0.1s\n",
      "[CV] END ......................C=5, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=5, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END ......................C=5, gamma=0.5, kernel=linear; total time=   0.0s\n",
      "[CV] END .........................C=5, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=5, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=5, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=5, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=5, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .....................C=5, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=5, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=5, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=5, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=5, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ........................C=5, gamma=1, kernel=linear; total time=   0.2s\n",
      "[CV] END ........................C=5, gamma=1, kernel=linear; total time=   0.1s\n",
      "[CV] END ........................C=5, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=5, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=5, gamma=1, kernel=linear; total time=   0.0s\n",
      "[CV] END ...........................C=5, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=5, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=5, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=5, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=5, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ........................C=5, gamma=5, kernel=linear; total time=   0.2s\n",
      "[CV] END ........................C=5, gamma=5, kernel=linear; total time=   0.1s\n",
      "[CV] END ........................C=5, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=5, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ........................C=5, gamma=5, kernel=linear; total time=   0.0s\n",
      "[CV] END ...........................C=5, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=5, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=5, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=5, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ...........................C=5, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=10, kernel=linear; total time=   0.2s\n",
      "[CV] END .......................C=5, gamma=10, kernel=linear; total time=   0.1s\n",
      "[CV] END .......................C=5, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END .......................C=5, gamma=10, kernel=linear; total time=   0.0s\n",
      "[CV] END ..........................C=5, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=5, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=5, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=5, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=5, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END ......................C=5, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=5, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=5, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=5, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=5, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=10, gamma=0.1, kernel=linear; total time=   0.3s\n",
      "[CV] END .....................C=10, gamma=0.1, kernel=linear; total time=   0.2s\n",
      "[CV] END .....................C=10, gamma=0.1, kernel=linear; total time=   0.1s\n",
      "[CV] END .....................C=10, gamma=0.1, kernel=linear; total time=   0.1s\n",
      "[CV] END .....................C=10, gamma=0.1, kernel=linear; total time=   0.1s\n",
      "[CV] END ........................C=10, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=10, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=10, gamma=0.1, kernel=rbf; total time=   0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] END ........................C=10, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=10, gamma=0.1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ....................C=10, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=10, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=10, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=10, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=10, gamma=0.1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=10, gamma=0.5, kernel=linear; total time=   0.3s\n",
      "[CV] END .....................C=10, gamma=0.5, kernel=linear; total time=   0.2s\n",
      "[CV] END .....................C=10, gamma=0.5, kernel=linear; total time=   0.1s\n",
      "[CV] END .....................C=10, gamma=0.5, kernel=linear; total time=   0.1s\n",
      "[CV] END .....................C=10, gamma=0.5, kernel=linear; total time=   0.1s\n",
      "[CV] END ........................C=10, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=10, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=10, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=10, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ........................C=10, gamma=0.5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ....................C=10, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=10, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=10, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=10, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ....................C=10, gamma=0.5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=10, gamma=1, kernel=linear; total time=   0.3s\n",
      "[CV] END .......................C=10, gamma=1, kernel=linear; total time=   0.2s\n",
      "[CV] END .......................C=10, gamma=1, kernel=linear; total time=   0.1s\n",
      "[CV] END .......................C=10, gamma=1, kernel=linear; total time=   0.1s\n",
      "[CV] END .......................C=10, gamma=1, kernel=linear; total time=   0.1s\n",
      "[CV] END ..........................C=10, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=10, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=10, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=10, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=10, gamma=1, kernel=rbf; total time=   0.0s\n",
      "[CV] END ......................C=10, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=10, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=10, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=10, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=10, gamma=1, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .......................C=10, gamma=5, kernel=linear; total time=   0.3s\n",
      "[CV] END .......................C=10, gamma=5, kernel=linear; total time=   0.2s\n",
      "[CV] END .......................C=10, gamma=5, kernel=linear; total time=   0.1s\n",
      "[CV] END .......................C=10, gamma=5, kernel=linear; total time=   0.1s\n",
      "[CV] END .......................C=10, gamma=5, kernel=linear; total time=   0.1s\n",
      "[CV] END ..........................C=10, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=10, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=10, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=10, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ..........................C=10, gamma=5, kernel=rbf; total time=   0.0s\n",
      "[CV] END ......................C=10, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=10, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=10, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=10, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=10, gamma=5, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END ......................C=10, gamma=10, kernel=linear; total time=   0.3s\n",
      "[CV] END ......................C=10, gamma=10, kernel=linear; total time=   0.2s\n",
      "[CV] END ......................C=10, gamma=10, kernel=linear; total time=   0.1s\n",
      "[CV] END ......................C=10, gamma=10, kernel=linear; total time=   0.1s\n",
      "[CV] END ......................C=10, gamma=10, kernel=linear; total time=   0.1s\n",
      "[CV] END .........................C=10, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=10, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=10, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=10, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END .........................C=10, gamma=10, kernel=rbf; total time=   0.0s\n",
      "[CV] END .....................C=10, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=10, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=10, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=10, gamma=10, kernel=sigmoid; total time=   0.0s\n",
      "[CV] END .....................C=10, gamma=10, kernel=sigmoid; total time=   0.0s\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "model = SVC(probability=True, random_state=0)\n",
    "\n",
    "param_grid = {\n",
    "    'C': [0.1, 0.5, 1, 5, 10],\n",
    "    'gamma': [0.1, 0.5, 1, 5, 10],\n",
    "    'kernel': ['linear', 'rbf', 'sigmoid']\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(estimator=model, param_grid=param_grid, cv=5, verbose=2)\n",
    "grid_search.fit(x, y) # Train the model 75 times with 75 different parameter combinations\n",
    "\n",
    "best_model = grid_search.best_estimator_ # Get the model with the best performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the best parameter values from `GridSearchCV`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'C': 1, 'gamma': 0.5, 'kernel': 'rbf'}\n"
     ]
    }
   ],
   "source": [
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cross-validate the model again to see if the score improved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8179552841524671\n"
     ]
    }
   ],
   "source": [
    "scores = cross_validate(best_model, x, y, cv=5)\n",
    "print(scores['test_score'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're up from 78% to almost 82% with an RBF kernel. Let's try narrowing the range of `C` and `gamma` and see if we can improve the model further."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 9 candidates, totalling 45 fits\n",
      "[CV] END ....................................C=1, gamma=0.25; total time=   0.0s\n",
      "[CV] END ....................................C=1, gamma=0.25; total time=   0.0s\n",
      "[CV] END ....................................C=1, gamma=0.25; total time=   0.0s\n",
      "[CV] END ....................................C=1, gamma=0.25; total time=   0.0s\n",
      "[CV] END ....................................C=1, gamma=0.25; total time=   0.0s\n",
      "[CV] END .....................................C=1, gamma=0.5; total time=   0.0s\n",
      "[CV] END .....................................C=1, gamma=0.5; total time=   0.0s\n",
      "[CV] END .....................................C=1, gamma=0.5; total time=   0.0s\n",
      "[CV] END .....................................C=1, gamma=0.5; total time=   0.0s\n",
      "[CV] END .....................................C=1, gamma=0.5; total time=   0.0s\n",
      "[CV] END ....................................C=1, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=1, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=1, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=1, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=1, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=2, gamma=0.25; total time=   0.0s\n",
      "[CV] END ....................................C=2, gamma=0.25; total time=   0.0s\n",
      "[CV] END ....................................C=2, gamma=0.25; total time=   0.0s\n",
      "[CV] END ....................................C=2, gamma=0.25; total time=   0.0s\n",
      "[CV] END ....................................C=2, gamma=0.25; total time=   0.0s\n",
      "[CV] END .....................................C=2, gamma=0.5; total time=   0.0s\n",
      "[CV] END .....................................C=2, gamma=0.5; total time=   0.0s\n",
      "[CV] END .....................................C=2, gamma=0.5; total time=   0.0s\n",
      "[CV] END .....................................C=2, gamma=0.5; total time=   0.0s\n",
      "[CV] END .....................................C=2, gamma=0.5; total time=   0.0s\n",
      "[CV] END ....................................C=2, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=2, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=2, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=2, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=2, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=3, gamma=0.25; total time=   0.0s\n",
      "[CV] END ....................................C=3, gamma=0.25; total time=   0.0s\n",
      "[CV] END ....................................C=3, gamma=0.25; total time=   0.0s\n",
      "[CV] END ....................................C=3, gamma=0.25; total time=   0.0s\n",
      "[CV] END ....................................C=3, gamma=0.25; total time=   0.0s\n",
      "[CV] END .....................................C=3, gamma=0.5; total time=   0.0s\n",
      "[CV] END .....................................C=3, gamma=0.5; total time=   0.0s\n",
      "[CV] END .....................................C=3, gamma=0.5; total time=   0.0s\n",
      "[CV] END .....................................C=3, gamma=0.5; total time=   0.0s\n",
      "[CV] END .....................................C=3, gamma=0.5; total time=   0.0s\n",
      "[CV] END ....................................C=3, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=3, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=3, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=3, gamma=0.75; total time=   0.0s\n",
      "[CV] END ....................................C=3, gamma=0.75; total time=   0.0s\n"
     ]
    }
   ],
   "source": [
    "model = SVC(kernel='rbf', probability=True, random_state=0)\n",
    "\n",
    "param_grid = {\n",
    "    'C': [1, 2, 3],\n",
    "    'gamma': [0.25, 0.5, 0.75]\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(estimator=model, param_grid=param_grid, cv=5, verbose=2)\n",
    "grid_search.fit(x, y)\n",
    "\n",
    "best_model = grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the best parameter values this time?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'C': 3, 'gamma': 0.25}\n"
     ]
    }
   ],
   "source": [
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And what is the cross-validated score?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8221510883482714\n"
     ]
    }
   ],
   "source": [
    "scores = cross_validate(best_model, x, y, cv=5)\n",
    "print(scores['test_score'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're above 82% now. Make the best model that `GridSearchCV` found *the* model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = best_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use a confusion matrix to see how the model performs during testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x166d6dae1f0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVgAAAEoCAYAAAAdTumBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfwUlEQVR4nO3debwcVZ338c/3JkBCFkgIyQQQwhKDLCFAWAJOZJVFJYwQQKNPVBjExwH0kXHiuIyCAjrqgApKFCaRnQiYAApkApFNgSQk7EyUhEVCNrYQAiTh9/xRdaET7r1dfdN1u7r7++ZVr66qrjr169vw4/Spc04pIjAzs+prqXUAZmaNygnWzCwnTrBmZjlxgjUzy4kTrJlZTrrXOoCiU/eeoY371DoMq8CeH9q21iFYhebMmb0sIrbs7Pnd+m4XsWZVpmNj1dLbIuLIzl6rEk6wZWjjPmwy7IRah2EVuPf+X9Q6BKtQz430zIacH2tWZf7v9M25Fw3YkGtVwgnWzBqAQMVr8XSCNbP6J0CqdRTv4wRrZo2hpVutI3gfJ1gzawBuIjAzy4+bCMzMciBcgzUzy4dcgzUzy41vcpmZ5cE3uczM8uF+sGZmOXIN1swsD24iMDPLT4ubCMzMqk+4F4GZWT7cRGBmlh/3IjAzy4lrsGZmOZCHypqZ5cc1WDOzPMi9CMzMcuMmAjOzHBR0PtjiRWRmVrG0H2yWpVxJ0jBJc0uW1yR9RVJ/SdMlzU9f+5UrywnWzBpDa0+CcksZEfFURIyIiBHA3sAbwI3ABGBGRAwFZqTbHXKCNbPG0NIt21KZQ4G/RcQzwBhgcrp/MnBsuZPdBmtm9U8VDZUdIGlWyfbEiJjYzrEnAVen64MiYhFARCySNLDchZxgzawxZO9FsCwiRpYvThsDxwDf6GxIbiIws4YgKdNSgaOAORGxON1eLGlweq3BwJJyBTjBmlndS54YU/UE+yneax4AmAaMT9fHA1PLFeAEa2b1TxUsWYqTNgUOB24o2X0+cLik+el755crx22wZtYAREtL9eqLEfEGsMV6+5aT9CrIzAnWzBpChT//u4QTrJk1BCdYM7M8VNC+2pWcYM2s7omKewh0CSdYM2sITrBmZjmpZi+CanGCNbP65zZYM7P8uInAzCwHvsllZpYjJ1gzszwI1OIEa2aWC9dgzcxy4gRrZpYD3+QyM8tT8fKrE6yZNQC5icDMLDceKmtmlpfiVWCdYJvBTtsN5LJzv/Du9nZbbcF5E2/hnlnz+cmEk+i96SY8u2g5p357MitWvlnDSK09r654gzO+fxVP/G0REvz82+PYd/gOtQ6rUJquiUDSWuCR9DpPAOPTZ91kOXcr4GcRcXw77w8Bbo6I3TYgvtcjondnz68Xf31mCaPHJc9na2kRj//hB9xy5zwm/fAUvn3hjdw356+M+8T+nP7ZQzn3V7fUOFpry4Sf/I5DR+3C5B+ewtur17DqzbdrHVKhdOKJsV0i70aLVRExIk2CbwOnZTlJUveIeKG95Gqd95F9hrHw+aU89+LL7LTtQO6b81cAZj7wJJ84eERtg7M2vfb6Ku576G98dswoADbeqDub9dm0xlEVTw6P7d5gXdkqfDewk6Reki6T9KCkhySNAZD0OUlTJN0E3C5piKRH0/d2lfSApLmSHpY0NC2zm6RfS3pM0u2SeqbH7yjpVkmzJd0taed0//aS/pxe+5wu/OyF8cmP7s31t80G4MmnF3HU6N0BGHPoXmw9qF8tQ7N2PPP35QzYvDdf/t4VjB53Pmd8/0pWrnqr1mEVjlqUaclUlrS5pN9JelLSE5JGSeovabqk+elr2f9guiTBSuoOHEXSXPBN4I6I2Ac4GPhPSb3SQ0eRNCMcsl4RpwEXRsQIYCTwfLp/KHBRROwKvAIcl+6fCJweEXsDZwEXp/svBH6ZXvvFDuI9VdIsSbNizapOfuri2ah7N44avTu/n/EQAP9y9pWcMnY0d/726/TedBNWr15b4witLWvWrmXeU8/xheP/kbuunMCmPTbhgknTax1W4VS5BnshcGtE7AzsQdLEOQGYERFDgRnpdofyvsnVU9LcdP1u4FLgPuAYSWel+3sA26br0yPipTbK+TPwTUnbADdExPz0D7UgIlrLnw0MkdQbOACYUvLH3CR9PZD3kvDlwA/bCjoiJpIkaVo2HRiZP23BHXbALsx78jmWvrQCgPnPLOa40y8CYMdtB/LRD+9ay/CsHVsN7MdWAzdn5G5DADjm0BFcMNkJdh1V7AcrqS8wGvgcQES8Dbyd/to+KD1sMjAT+LeOyso7wa5Ka53vUvJXOC4inlpv/37AyrYKiYirJN0PfAy4TdIpwNNA6e+ktUBPklr5K+tft7S4TnyOhnD8ESO5/vbZ724P6NebZS+/jiTO+sIR/Pf199QwOmvPoAF92XpQP+YvXMzQIYO468GnGLb9P9Q6rEIRUEF+HSBpVsn2xLRS1WoHYCnw35L2IKm8nQkMiohFABGxSNLAcheqRTet24DTJZ0eESFpz4h4qKMTJO0APB0RP0vXh5Mk2PeJiNckLZA0NiKmpAl9eETMA+4FTgKuAMZV9VMVXM9NNuKgfXfmq+de/e6+444YySnHjwbg5plzufKmv9QqPCvjR2eN5dTvTOLt1WsZsvUALvrOZ2odUsFU9PN/WUSM7OD97sBeJM2M90u6kAzNAe0V1NXOAS4AHk6T30Lg42XOORH4jKTVJG2nZwN9Ozh+HPBLSd8CNgKuAeaR/F/oKklnAtdvwGeoO6veWs2Oh6/7a+aSa2ZyyTUzaxOQVWT3Ydtw5287/DXa9KrYQeB54PmIuD/d/h1Jgl0saXBaex0MLClXUK4Jtq0+phGxCvhiG/snAZNKthcCu6Xr5wHnrXfKS63vp8f8uGR9AXBkG9dYQHIjrdX5WT6HmRWckj7e1RARL0p6TtKwtCnzUODxdBlPkjfGA1PLleWRXGZW90T1EmzqdOBKSRuTNEd+nuT+znWSTgaeBcaWK8QJ1swaQjXHEKS9k9pqpz20knKcYM2sIRRxqKwTrJnVP1W3BlstTrBmVveSfrDFy7BOsGbWAFTtm1xV4QRrZg3BNVgzszy4DdbMLB9ugzUzy1EB86sTrJk1Bt/kMjPLQxXng60mJ1gzq3sVzgfbZZxgzawBFPOpsk6wZtYQCphfnWDNrDG4BmtmlgNVccLtanKCNbOG4BqsmVlOCphfnWDNrDG4BmtmlgdP9mJmlg+5H6yZWX66uReBmVk+qlmBlbQQWAGsBdZExEhJ/YFrgSHAQuCEiHi5o3JaqheSmVltKJ3sJctSgYMjYkREtD6+ewIwIyKGAjPS7Q45wZpZQ2hRtmUDjAEmp+uTgWPLndBuE4GknwPR3vsRcUaFwZmZ5aaC2ukASbNKtidGxMT1jgngdkkBXJK+PygiFgFExCJJA8tdqKM22FkdvGdmVhgCWrIn2GUlP/vbc2BEvJAm0emSnuxMXO0m2IiYXLotqVdErOzMRczM8lbNTgQR8UL6ukTSjcC+wGJJg9Pa62BgSdmYyh0gaZSkx4En0u09JF28YeGbmVVRxhtcWZoRJPWS1Kd1Hfgo8CgwDRifHjYemFqurCzdtC4AjkgLJyLmSRqd4Twzsy5TxW5ag4Ab02TcHbgqIm6V9CBwnaSTgWeBseUKytQPNiKeWy/zr604ZDOznFTYBtuhiHga2KON/cuBQyspK0uCfU7SAUBI2hg4g7S5wMysKAo4UjZTgj0NuBDYGvg7cBvw5TyDMjOrRN1OuB0Ry4BxXRCLmVmnVauJoJqy9CLYQdJNkpZKWiJpqqQduiI4M7OslHHpSlmGyl4FXAcMBrYCpgBX5xmUmVmlcpiLYINlSbCKiMsjYk26XEEHQ2jNzLpa0osg97kIKtbRXAT909U7JU0AriFJrCcCt3RBbGZm2dSgdppFRze5ZpMk1Naov1jyXgDn5BWUmVml6qoXQURs35WBmJl1VmsTQdFkGsklaTdgF6BH676I+G1eQZmZVaremggAkPQfwEEkCfYPwFHAPYATrJkVRvHSa7ZeBMeTjL99MSI+TzJGd5NcozIzq4CUDDTIsnSlLE0EqyLiHUlrJPUlmQPRAw3MrFDq6iZXiVmSNgd+TdKz4HXggTyDMjOrVAGbYDPNRfB/09VfSboV6BsRD+cblplZdqLrf/5n0dFAg706ei8i5uQTkplZhVR/NdifdPBeAIdUOZZC2n3YB/jDHR39KaxovjTFP7CaUV1104qIg7syEDOzDZGlS1RXyzTQwMysyAR0q9NeBGZmhVfA/FrIWrWZWUWk6s8HK6mbpIck3Zxu95c0XdL89LVfuTKyPNFAkj4j6Tvp9raS9s0cpZlZF8hhPtgzWfcBrxOAGRExFJiRbnccU4aLXAyMAj6Vbq8ALqooTDOznEnZlmxlaRvgY8BvSnaPASan65OBY8uVk6UNdr+I2EvSQwAR8XL6+G4zs0IQ0L263bQuAL4O9CnZNygiFgFExCJJA8sVkqUGu1pSN9LHxEjaEnin4nDNzHJUQQ12gKRZJcup65ajjwNLImL2hsaUpQb7M+BGYKCkH5DMrvWtDb2wmVm1qLKZspZFxMgO3j8QOEbS0SRzYPeVdAWwWNLgtPY6mGTiqw6VrcFGxJUkVeXzgEXAsRExJcunMDPrKtVqg42Ib0TENhExBDgJuCMiPgNMA8anh40HppYrK8uE29sCbwA3le6LiGfLh2pm1jW6oB/s+cB1kk4GngXGljshSxPBLbz38MMewPbAU8CunY/TzKx6kmdyVT/DRsRMYGa6vpzk4QOZZZmucPfS7XSWrS+2c7iZWdcTdCvgsKmKh8pGxBxJ++QRjJlZZ6mAT+XK0gb7/0o2W4C9gKW5RWRmVqF6fmx3aUfbNSRtstfnE46ZWefUXYJNBxj0joh/7aJ4zMw6pa4m3JbUPSLWdPToGDOzIqjHJoIHSNpb50qaBkwBVra+GRE35BybmVk2qt8Jt/sDy0mewdXaHzYAJ1gzK4R6rMEOTHsQPMp7ibVV5BqVmVmFCtgE22GC7Qb0hjY7lznBmlmBiJY66we7KCLO7rJIzMw6SdRfDbaA4ZqZtUHQvYCNsB0l2IomNTAzq5W6q8FGxEtdGYiZ2YbIYzatDVXxZC9mZkVUwPzqBGtm9U9ke8BgV3OCNbP6pzqbi8DMrF4I6OYEa2aWj+KlVydYM2sQBazAOsGaWSNQIdtgi3jjzcysIq29CLIsZcuSekh6QNI8SY9J+l66v7+k6ZLmp6/9ypXlBGtmDUFSpiWDt4BDImIPYARwpKT9gQnAjIgYCsxItzvkBGtm9U/JSK4sSzmReD3d3ChdAhgDTE73TwaOLVeWE6yZ1b0KmwgGSJpVspz6vvKkbpLmAkuA6RFxPzAoIhYBpK8Dy8Xlm1xm1hAquMm1LCJGdnRARKwFRkjaHLhR0m6dick1WDNrCMq4VCIiXgFmAkcCiyUNBkhfl5Q73wnWzBqClG0pX462TGuuSOoJHAY8CUwDxqeHjQemlivLTQRmVveqPFR2MDBZUjeSSuh1EXGzpD8D10k6GXgWGFuuICdYM2sAQlUaLBsRDwN7trF/ORU+iMAJ1swaQgEHcjnBmln9S7ppFS/DOsGaWf3LeAOrqznBmllDcII1M8uBJ9w2M8tRtXoRVJMTrJk1hAJWYJ1gm8GEH13DHX95gi02780fL/tXAP7rsj/yP/c9RotE/81786N/O4lBAzarcaRWSsA3DhvKK6tWc/G9C/nYLoP48A79WfHWGgCmPvIij724orZBFkgRa7C5DpWV9M10wtqHJc2VtF8VyjxGUtl5GDOW9Xr5o+rfJ4/Yh8vO/+d19p1y4sHc8puzuOnXX+OQUbvwi8un1yg6a88hQwfw4oo319k343+Xcu70+Zw7fb6TawkBLcq2dKXcEqykUcDHgb0iYjjJeN7nMp7bbs06IqZFxPnVibI57LvHjmzed9N19vXp1ePd9TfefLuA/+9vbpv33IjdBvfh3qdfqnUodUKZ/+lKeTYRDCaZFuwtgIhYBiBpITAyIpZJGgn8OCIOkvRdYCtgCLBM0o7AFyLisfS8mcDXgN2BkcA3gXnADhHxjqRNgaeAHYBtgYuALYE3gH+OiCclbQ9clX7uW3P87HXhJ5f+gRtvn0WfXj254qdfqnU4VmLsiMHc+PCLbLLRunWgg3YawH7b9ePZl1dx/bxFvLF6bY0iLJga1E6zyLOJ4HbgA5L+V9LFkj6S4Zy9gTER8WngGuAEeHdqsK0iYnbrgRHxKkmCbS33E8BtEbEamAicHhF7A2cBF6fHXAj8MiL2AV7c4E9Y57528tHcc+13OOawvbj89/fUOhxL7Ta4DyveXMOzr6xaZ/9df1vOt//wJOdOn8+rb67muD0G1yjC4kmaCKrzRINqyi3Bpo9c2Bs4FVgKXCvpc2VOmxYRrf9WXcd7s9WcAExp4/hrgRPT9ZPSa/QGDgCmpDOSX0JSmwY4ELg6Xb+8vSAkndo62/nyZUvLhFz/jjlkT26765Fah2GpHbfoxfCt+vL9o3fm5P23ZdjA3nxu3w+w4q01BMmzS+55+iWG9N+0XFFNJY/5YDdUrr0I0lnBZwIzJT1CMofiGt5L7D3WO2Vlybl/l7Rc0nCSJPrFNi4xDThPUn+SZH4H0At4JSJGtBdWhrgnktSC2WPPvcseX48WPr+UIdtsCcCM+x5jh23LPv3CusjUR19k6qPJD6yhW/bi8A9uyaQHnqNvj+689mbSg2DE1pvxwqtvdlRM8ylgE0FuCVbSMOCdiJif7hoBPAP0JEmGfwSOK1PMNcDXgc0i4n1VrIh4XdIDJD/9b04T+muSFkgaGxFTlDxHYnhEzAPuJanpXgGM2+APWSe+cs7l3D/vb7z86koOPOFszvzcEfzp/id4+rmltLSIrQb245yvHl/rMK2MTw4fzDab9yACXnpjNVfOfr7WIRVKEbtp5VmD7Q38PJ0ZfA3wV5Lmgg8Bl0r6d+D+MmX8jiR5ntPBMdeSNB8cVLJvHPBLSd8ieSLkNSTttWcCV0k6E7i+ws9Tty749mfft++Eoze4x5x1gflLVzJ/afLDbtIDmTrhNK0i3uTKLcGmN6QOaOOtu4EPtnH8d9vYt5j1YoyIScCkku3fsd6Pg4hYQPIMnfXLWwCMKtnl7l5mjaKZEqyZWVdJbmAVL8M6wZpZ/fN8sGZm+SlgfnWCNbMGUcAMm+tkL2ZmXSPbKK4sI7kkfUDSnZKeSCerOjPd31/SdEnz09d+5cpygjWzupd1FFfGSu4a4GsR8SFgf+DLknYBJgAzImIoMCPd7pATrJk1hipl2IhYFBFz0vUVwBPA1sAYYHJ62GTg2HJluQ3WzBpCBd20BkiaVbI9MR0e//4ypSHAniSDogZFxCJIkrCksuPLnWDNrCFU0E1rWUSMLF+eepOM+PxKRLymTvQDcxOBmTWEas6mJWkjkuR6ZUTckO5enE6d2jqF6pJy5TjBmln9E0jKtJQtKjnoUuCJiPhpyVvTSGYEJH2dWq4sNxGYWd0TVR3JdSDwWeCRdE5pgH8nmbvkOkknA8/y3nzV7XKCNbOGUK38GhH3dFDcoZWU5QRrZo2hgCO5nGDNrCF4Ni0zs5w01YTbZmZdygnWzKz6POG2mVlePOG2mVl+CphfnWDNrEEUMMM6wZpZA8g2mXZXc4I1s7pXyUQuXckJ1swaQwEzrBOsmTUEd9MyM8tJAZtgnWDNrAHIQ2XNzHJUvAzrBGtmda/KE25XjROsmTWEAuZXJ1gzawyuwZqZ5cTdtMzMcuIarJlZDlTQ6Qpbah2AmVk1KOM/ZcuRLpO0RNKjJfv6S5ouaX762i9LTE6wZtYYlHEpbxJw5Hr7JgAzImIoMCPdLssJ1swaQrXya0TcBby03u4xwOR0fTJwbJaY3AZrZg2hgjbYAZJmlWxPjIiJZc4ZFBGLACJikaSBWS7kBGtmdU+VTbi9LCJG5hlPKzcRmJmVt1jSYID0dUmWk5xgzawhtHbVKrd00jRgfLo+Hpia5SQnWDNrCFXspnU18GdgmKTnJZ0MnA8cLmk+cHi6XZbbYM2s/lVxoEFEfKqdtw6ttCwnWDOre56u0MwsR57sxcwsJ67BmpnlpID51QnWzBpEATOsE6yZNYQitsEqImodQ6FJWgo8U+s4cjAAWFbrIKwijfydbRcRW3b2ZEm3kvx9slgWEevPlpULJ9gmJWlWV43Hturwd1Z/PJLLzCwnTrBmZjlxgm1e5ea/tOLxd1Zn3AZrZpYT12DNzHLiBGtmlhMnWDOznDjBmpnlxAnWzCwnnougCUhaAbTbXSQi+nZhOJaBv7PG4ATbBCKiD4Cks4EXgctJ5h4aB/SpYWjWDn9njcH9YJuIpPsjYr9y+6w4/J3VN7fBNpe1ksZJ6iapRdI4YG2tg7IO+TurY06wzeXTwAnA4nQZm+6z4vJ3VsfcRGBmlhPXYJuIpA9KmiHp0XR7uKRv1Toua5+/s/rmBNtcfg18A1gNEBEPAyfVNCIrx99ZHXOCbS6bRsQD6+1bU5NILCt/Z3XMCba5LJO0I2kHdknHA4tqG5KV4e+sjvkmVxORtAPJpM0HAC8DC4DPRMTCWsZl7WvnOxsXEY34IM6G4wTbhCT1AloiYkWtY7GOSeoWEWv9ndUnD5VtIpI2AY4DhgDdpeQ58hFxdg3Dso4tSB9JfS1wR62Dscq4Dba5TAXGkNwkWVmyWHENA/4H+DJJsv2FpA/XOCbLyE0ETUTSoxGxW63jsM6R1A+4kKQNtlut47HyXINtLvdJ2r3WQVhlJH1E0sXAHKAHydBZqwOuwTYBSY+QdPPpDgwFngbeIpn+LiJieA3Dsw5IWgDMBa4DpkWEm3TqiBNsE5C0XUfvu8tPcUnqGxGv1ToO6xwn2CaSdlh/PiLeknQQMBz4bUS8Usu47P0kfT0ifiTp57TxZIOIOKMGYVmF3E2ruVwPjJS0E3ApMA24Cji6plFZW55IX2fVNArbIE6wzeWdiFgj6ZPABRHxc0kP1Tooe7+IuCldfTgi/B3VKfciaC6rJX0K+D/Azem+jWoYj5X3U0lPSjpH0q61DsYq4wTbXD4PjAJ+EBELJG0PXFHjmKwDEXEwcBCwFJgo6RHPB1s/fJPLrE6kfZi/DpwYERvXOh4rzwm2CUi6LiJOKOkPuw73gy0uSR8CTgSOB5YD1wDXR8SSmgZmmTjBNgFJgyNiUXv9Yd0Ptrgk/QW4GpgSES/UOh6rjBNsk5DUDbgtIg6rdSyWTfqd/TYixtU6Fusc3+RqEhGxFnhD0ma1jsWySb+zLSS5vbVOuR9sc3kTeETSdEqmKfSooEJ7BrhX0jTW/c5+WruQLCsn2OZyS7pY/XghXVqAPjWOxSrkNtgmI6knsG1EPFXrWMwanWuwTUTSJ4AfAxsD20saAZwdEcfUNDBrl6Q7abtr3SE1CMcq5ATbXL4L7AvMBIiIueloLiuus0rWe5A8U21NjWKxCjnBNpc1EfFq68MOU24jKrCImL3ernsl/akmwVjFnGCby6OSPg10kzQUOAO4r8YxWQck9S/ZbAFGAv9Qo3CsQr7J1UQkbQp8E/houus24PsR8WbtorKOpI+Maf2PdA2wkKTd/J6aBWWZuQbbBCT1AE4DdgIeAUZFhNvxCkzSPsBzEbF9uj2epP11IfB4DUOzCngkV3OYTPLT8hHgKJKeBFZslwBvA0gaDZxH8j2+CkysYVxWATcRNAFJj0TE7ul6d+CBiNirxmFZByTNi4g90vWLgKUR8d10e25EjKhheJaRa7DNYXXripsG6ka39H+GAIcCd5S856a9OuEvqjnsIan10c8CeqbbAiIi+tYuNGvH1cCfJC0DVgF3A6QPrHy1loFZdm4iMCsoSfsDg4HbI2Jluu+DQO+ImFPT4CwTJ1gzs5y4DdbMLCdOsGZmOXGCtQ0maa2kuZIelTQlHTHW2bImSTo+Xf+NpF06OPYgSQd04hoLJQ3Iun+9Y16v8FrflXRW+SOtETnBWjWsiogREbEbSef400rfTJ8tVbGIOCUiOhq1dBBQcYI16ypOsFZtdwM7pbXLOyVdRfKYmm6S/lPSg5IelvRFACV+IelxSbcAA1sLkjRT0sh0/UhJcyTNkzRD0hCSRP7VtPb8j5K2lHR9eo0HJR2YnruFpNslPSTpEpLuaR2S9HtJsyU9JunU9d77SRrLDElbpvt2lHRres7dknauyl/T6pr7wVrVpB3jjwJuTXftC+wWEQvSJPVqROwjaROSafduB/YEhgG7A4NIxtlftl65WwK/BkanZfWPiJck/Qp4PSJ+nB53FfBfEXGPpG1JJrP5EPAfwD0RcbakjwHrJMx2fCG9Rk/gQUnXR8RyoBcwJyK+Juk7adn/QjJ89bSImC9pP+BiwJNiNzknWKuGnpLmput3A5eS/HR/ICIWpPs/CgxvbV8FNgOGAqOBq9MnqL4gqXTEUqv9gbtay4qIl9qJ4zBgl5L5bvtK6pNe45PpubdIejnDZzpD0j+l6x9IY10OvANcm+6/ArhBUu/0804pufYmGa5hDc4J1qph1fpj49NEs7J0F3B6RNy23nFHU37Sb2U4BpImr1ERsaqNWDJ3+JZ0EEmyHhURb0iaSfI0gbZEet1XPD+Arc9tsNZVbgO+JGkjSEYkSeoF3AWclLbRDgYObuPcPwMfaX28Tckk1CtY90mrt5P8XCc9bkS6ehcwLt13FNCvTKybAS+nyXVnkhp0qxagtRb+aZKmh9eABZLGpteQpD3KXMOagBOsdZXfkLSvzpH0KMl0fN2BG4H5JFMp/hJ43+NQImIpSbvpDZLm8d5P9JuAf2q9yUXyhIaR6U20x3mvN8P3gNGS5pA0VTxbJtZbge6SHgbOAf5S8t5KYFdJs0naWM9O948DTk7jewwYk+FvYg3OQ2XNzHLiGqyZWU6cYM3McuIEa2aWEydYM7OcOMGameXECdbMLCdOsGZmOfn/mMu2abUFouIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from sklearn.metrics import plot_confusion_matrix\n",
    "\n",
    "plot_confusion_matrix(model, x_test, y_test, display_labels=['Perished', 'Survived'], cmap='Blues', xticks_rotation='vertical')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print a classification report to view precision, recall, and other metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.86      0.93      0.89        85\n",
      "           1       0.88      0.78      0.83        58\n",
      "\n",
      "    accuracy                           0.87       143\n",
      "   macro avg       0.87      0.85      0.86       143\n",
      "weighted avg       0.87      0.87      0.87       143\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print(classification_report(y_test, model.predict(x_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the model to make predictions\n",
    "\n",
    "Now use the model to predict whether a 30-year-old female traveling in first class will survive the voyage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "female = [[30, 1, 0, 1, 0, 0]]\n",
    "model.predict(female)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More to the point, what is the probability that a 30-year-old female traveling in first class will survive?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probability of survival: 84.1%\n"
     ]
    }
   ],
   "source": [
    "probability = model.predict_proba(female)[0][1]\n",
    "print(f'Probability of survival: {probability:.1%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How about a 60-year-old male traveling in third class?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probability of survival: 13.0%\n"
     ]
    }
   ],
   "source": [
    "male = [[60, 0, 1, 0, 0, 1]]\n",
    "probability = model.predict_proba(male)[0][1]\n",
    "print(f'Probability of survival: {probability:.1%}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}