SVM_Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<small><i>This notebook uses material from a tutotial given by [Jake Vanderplas](http://www.vanderplas.com) for PyCon 2014. Source and license info is on [GitHub](https://github.com/jakevdp/sklearn_pycon2014/).</i></small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Support Vector Machine tutorial\n",
    "===\n",
    "\n",
    "Support vector machines (or SVMs) are a popular supervised classification method. SVMs attempt to find a classification boundary that maximizes the separability of the classes. This means that it tries to maximize the distance between the boundary lines and the closest data points.\n",
    "\n",
    "Scikit-learn has a great [SVM tutorial](http://scikit-learn.org/stable/modules/svm.html) if you want more detailed information.\n",
    "\n",
    "Toy Dataset Illustration\n",
    "---\n",
    "\n",
    "Here, we will use a toy (or overly simple) dataset of two classes which can be perfectly separated with a single, straght line. \n",
    "\n",
    "<img src=\"Images/SVMBoundary.png\">\n",
    "\n",
    "The solid line is the decision boundary, dividing the red and blue classes. Notice that on either side of the boundary, there is a dotted line that passes through the closest datapoints. The distance between the solid boundary line and this dotted line is what an SVM tries to maximize. \n",
    "\n",
    "The points that touch the dotted lines are called \"support vectors\". These points are the only ones that matter when determining boundary locations. All other datapoints can be added, moved, or removed from the dataset without changing the classification boundary, as long as they do not cross that dotted line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setup\n",
    "===\n",
    "Tell matplotlib to print figures in the notebook. Then import numpy (for numerical data), pyplot (for plotting figures) ListedColormap (for plotting colors), neighbors (for the scikit-learn nearest-neighbor algorithm) and datasets (to download the iris dataset from scikit-learn).\n",
    "\n",
    "Also create the color maps to use to color the plotted data, and \"labelList\", which is a list of colored rectangles to use in plotted legends"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print figures in the notebook\n",
    "%matplotlib inline \n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.colors import ListedColormap\n",
    "from sklearn import datasets # Import the dataset from scikit-learn\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import train_test_split, KFold\n",
    "\n",
    "# Import patch for drawing rectangles in the legend\n",
    "from matplotlib.patches import Rectangle\n",
    "\n",
    "# Create color maps\n",
    "cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#0000FF'])\n",
    "cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#AAAAFF'])\n",
    "\n",
    "# Create a legend for the colors, using rectangles for the corresponding colormap colors\n",
    "labelList = []\n",
    "for color in cmap_bold.colors:\n",
    "    labelList.append(Rectangle((0, 0), 1, 1, fc=color))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the dataset\n",
    "===\n",
    "Import the dataset and store it to a variable called iris. Scikit-learn's explanation of the dataset is [here](http://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html). This dataset is similar to a python dictionary, with the keys: ['DESCR', 'target_names', 'target', 'data', 'feature_names']\n",
    "\n",
    "The data features are stored in iris.data, where each row is an example from a single flower, and each column is a single feature. The feature names are stored in iris.feature_names. Labels are stored as the numbers 0, 1, or 2 in iris.target, and the names of these labels are in iris.target_names.\n",
    "\n",
    "The dataset consists of measurements made on 50 examples from each of three different species of iris flowers (Setosa, Versicolour, and Virginica). Each example has four features (or measurements): [sepal](http://en.wikipedia.org/wiki/Sepal) length, sepal width, [petal](http://en.wikipedia.org/wiki/Petal) length, and petal width. All measurements are in cm.\n",
    "\n",
    "Below, we load the labels into y, the corresponding label names into labelNames, the data into X, and the names of the features into featureNames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import some data to play with\n",
    "iris = datasets.load_iris()\n",
    "\n",
    "# Store the labels (y), label names, features (X), and feature names\n",
    "y = iris.target       # Labels are stored in y as numbers\n",
    "labelNames = iris.target_names # Species names corresponding to labels 0, 1, and 2\n",
    "X = iris.data\n",
    "featureNames = iris.feature_names\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we plot the first two features from the dataset (sepal length and width). Normally we would try to use all useful features, but sticking with two allows us to visualize the data more easily.\n",
    "\n",
    "Then we plot the data to get a look at what we're dealing with. The colormap is used to determine what colors are used for each class when plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the data\n",
    "\n",
    "# Sepal length and width\n",
    "X_small = X[:,:2]\n",
    "# Get the minimum and maximum values with an additional 0.5 border\n",
    "x_min, x_max = X_small[:, 0].min() - .5, X_small[:, 0].max() + .5\n",
    "y_min, y_max = X_small[:, 1].min() - .5, X_small[:, 1].max() + .5\n",
    "\n",
    "plt.figure(figsize=(8, 6))\n",
    "\n",
    "# Plot the training points\n",
    "plt.scatter(X_small[:, 0], X_small[:, 1], c=y, cmap=cmap_bold)\n",
    "plt.xlabel('Sepal length (cm)')\n",
    "plt.ylabel('Sepal width (cm)')\n",
    "plt.title('Sepal width vs length')\n",
    "\n",
    "# Set the plot limits\n",
    "plt.xlim(x_min, x_max)\n",
    "plt.ylim(y_min, y_max)\n",
    "\n",
    "# Plot the legend\n",
    "plt.legend(labelList, labelNames)\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Support vector machines: training\n",
    "===\n",
    "Next, we train a SVM classifier on our data. \n",
    "\n",
    "The first line creates our classifier using the [SVC()](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) function. For now we can ignore the parameter kernel='linear', this just means the decision boundaries should be straight lines. The second line uses the fit() method to train the classifier on the features in X_small, using the labels in y.\n",
    "\n",
    "It is safe to ignore the parameter 'decision_function_shape'. This is not important for this tutorial, but its inclusion prevents warnings from Scikit-learn about future changes to the default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an instance of SVM and fit the data.\n",
    "clf = SVC(kernel='linear', decision_function_shape='ovo')\n",
    "clf.fit(X_small, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the classification boundaries\n",
    "===\n",
    "Now that we have our classifier, let's visualize what it's doing. \n",
    "\n",
    "First we plot the decision spaces, or the areas assigned to the different labels (species of iris). Then we plot our examples onto the space, showing where each point lies and the corresponding decision boundary.\n",
    "\n",
    "The colored background shows the areas that are considered to belong to a certain label. If we took sepal measurements from a new flower, we could plot it in this space and use the color to determine which type of iris our classifier believes it to be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h = .02  # step size in the mesh\n",
    "\n",
    "# Plot the decision boundary. For that, we will assign a color to each\n",
    "# point in the mesh [x_min, m_max]x[y_min, y_max].\n",
    "x_min, x_max = X_small[:, 0].min() - 1, X_small[:, 0].max() + 1\n",
    "y_min, y_max = X_small[:, 1].min() - 1, X_small[:, 1].max() + 1\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "                     np.arange(y_min, y_max, h))\n",
    "\n",
    "Z = clf.predict(np.c_[xx.ravel(), yy.ravel()]) # Make a prediction oat every point \n",
    "                                               # in the mesh in order to find the \n",
    "                                               # classification areas for each label\n",
    "\n",
    "# Put the result into a color plot\n",
    "Z = Z.reshape(xx.shape)\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.pcolormesh(xx, yy, Z, cmap=cmap_light)\n",
    "\n",
    "# Plot the training points\n",
    "plt.scatter(X_small[:, 0], X_small[:, 1], c=y, cmap=cmap_bold)\n",
    "plt.xlim(xx.min(), xx.max())\n",
    "plt.ylim(yy.min(), yy.max())\n",
    "plt.title(\"3-Class classification (SVM)\")\n",
    "plt.xlabel('Sepal length (cm)')\n",
    "plt.ylabel('Sepal width (cm)')\n",
    "\n",
    "# Plot the legend\n",
    "plt.legend(labelList, labelNames)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making predictions\n",
    "===\n",
    "\n",
    "Now, let's say we go out and measure the sepals of two new iris plants, and want to know what species they are. We're going to use our classifier to predict the flowers with the following measurements:\n",
    "\n",
    "Plant | Sepal length | Sepal width\n",
    "------|--------------|------------\n",
    "A     |4.3           |2.5\n",
    "B     |6.3           |2.1\n",
    "\n",
    "We can use our classifier's [predict()](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier.predict) function to predict the label for our input features. We pass in the variable examples to the predict() function, which is a list, and each element is another list containing the features (measurements) for a particular example. The output is a list of labels corresponding to the input examples.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add our new data examples\n",
    "examples = [[4.3, 2.5], # Plant A\n",
    "            [6.3, 2.1]] # Plant B\n",
    "\n",
    "\n",
    "# Create an instance of SVM and fit the data\n",
    "clf = SVC(kernel='linear', decision_function_shape='ovo')\n",
    "clf.fit(X_small, y)\n",
    "\n",
    "# Predict the labels for our new examples\n",
    "labels = clf.predict(examples)\n",
    "\n",
    "# Print the predicted species names\n",
    "print('A: ' + labelNames[labels[0]])\n",
    "print('B: ' + labelNames[labels[1]])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting our predictions\n",
    "---\n",
    "Now let's plot our predictions to see why they were classified that way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now plot the results\n",
    "h = .02  # step size in the mesh\n",
    "\n",
    "# Plot the decision boundary. For that, we will assign a color to each\n",
    "# point in the mesh [x_min, m_max]x[y_min, y_max].\n",
    "x_min, x_max = X_small[:, 0].min() - 1, X_small[:, 0].max() + 1\n",
    "y_min, y_max = X_small[:, 1].min() - 1, X_small[:, 1].max() + 1\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "                     np.arange(y_min, y_max, h))\n",
    "\n",
    "Z = clf.predict(np.c_[xx.ravel(), yy.ravel()]) # Make a prediction oat every point \n",
    "                                               # in the mesh in order to find the \n",
    "                                               # classification areas for each label\n",
    "\n",
    "# Put the result into a color plot\n",
    "Z = Z.reshape(xx.shape)\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.pcolormesh(xx, yy, Z, cmap=cmap_light)\n",
    "\n",
    "# Plot the training points\n",
    "plt.scatter(X_small[:, 0], X_small[:, 1], c=y, cmap=cmap_bold)\n",
    "plt.xlim(xx.min(), xx.max())\n",
    "plt.ylim(yy.min(), yy.max())\n",
    "plt.title(\"3-Class classification (SVM)\")\n",
    "plt.xlabel('Sepal length (cm)')\n",
    "plt.ylabel('Sepal width (cm)')\n",
    "\n",
    "# Display the new examples as labeled text on the graph\n",
    "plt.text(examples[0][0], examples[0][1],'A', fontsize=14)\n",
    "plt.text(examples[1][0], examples[1][1],'B', fontsize=14)\n",
    "\n",
    "# Plot the legend\n",
    "plt.legend(labelList, labelNames)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the support vectors in this example?\n",
    "===\n",
    "\n",
    "Below, we define a function to plot the solid decision boundary and corresponding dashed lines, as shown in the introductory picture. Because there are three classes to separate, there will now be three sets of lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_svc_decision_function(clf):\n",
    "    \"\"\"Plot the decision function for a 2D SVC\"\"\"\n",
    "    x = np.linspace(plt.xlim()[0], plt.xlim()[1], 30)\n",
    "    y = np.linspace(plt.ylim()[0], plt.ylim()[1], 30)\n",
    "    Y, X = np.meshgrid(y, x)\n",
    "    P = np.zeros((3,X.shape[0],X.shape[1]))\n",
    "    for i, xi in enumerate(x):\n",
    "        for j, yj in enumerate(y):\n",
    "            P[:, i,j] = clf.decision_function([[xi, yj]])[0]\n",
    "    for ind in range(3):\n",
    "        plt.contour(X, Y, P[ind,:,:], colors='k',\n",
    "                levels=[-1, 0, 1],\n",
    "                linestyles=['--', '-', '--'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we plot the lines on top of our previous plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Now plot the results\n",
    "h = .02  # step size in the mesh\n",
    "\n",
    "# Plot the decision boundary. For that, we will assign a color to each\n",
    "# point in the mesh [x_min, m_max]x[y_min, y_max].\n",
    "x_min, x_max = X_small[:, 0].min() - 1, X_small[:, 0].max() + 1\n",
    "y_min, y_max = X_small[:, 1].min() - 1, X_small[:, 1].max() + 1\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "                     np.arange(y_min, y_max, h))\n",
    "\n",
    "Z = clf.predict(np.c_[xx.ravel(), yy.ravel()]) # Make a prediction at every point \n",
    "                                               # in the mesh in order to find the \n",
    "                                               # classification areas for each label\n",
    "\n",
    "# Put the result into a color plot\n",
    "Z = Z.reshape(xx.shape)\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.pcolormesh(xx, yy, Z, cmap=cmap_light)\n",
    "\n",
    "# Plot the training points\n",
    "plt.scatter(X_small[:, 0], X_small[:, 1], c=y, cmap=cmap_bold)\n",
    "plt.xlim(xx.min(), xx.max())\n",
    "plt.ylim(yy.min(), yy.max())\n",
    "plt.title(\"3-Class classification (SVM)\")\n",
    "plt.xlabel('Sepal length (cm)')\n",
    "plt.ylabel('Sepal width (cm)')\n",
    "\n",
    "# Display the new examples as labeled text on the graph\n",
    "plt.text(examples[0][0], examples[0][1],'A', fontsize=14)\n",
    "plt.text(examples[1][0], examples[1][1],'B', fontsize=14)\n",
    "\n",
    "# Plot the legend\n",
    "plt.legend(labelList, labelNames)\n",
    "\n",
    "plot_svc_decision_function(clf) # Plot the decison function\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This plot is much more visually cluttered than our previous toy example. There are a few points worth noticing if you take a closer look. \n",
    "\n",
    "First, notice how the three solid lines run right along one of the decision boundaries. These are used to determine the boundaries between the classification areas (where the colors change). \n",
    "\n",
    "Additionally, while the parallel dotted lines still pas through one or more support vectors, there are now data points located between the decision boundary and the dotted line (and even on the wrong side of the decision boundary!). This happens when our data is not \"perfectly separable\". A perfectly separable dataset is one where the classes can be separated completely with a single, straight (or at least simple) line. While it makes for nice examples, real world machine learning uses are almost never perfectly separable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kernels: Changing The Decision Boundary Lines\n",
    "===\n",
    "\n",
    "In our previous example, all of the decision boundaries are straight lines. But what if our data is grouped into more circular clusters, maybe a curved line would separate the data better.\n",
    "\n",
    "SVMs use something called [kernels](http://scikit-learn.org/stable/modules/svm.html#kernel-functions) to determine the shape of the decision boundary. Remember that when we called the SVC() function we gave it a parameter kernel='linear', which made the boundaries straight. A different kernel, the radial basis function (RBF) groups data into circular clusters instead of dividing by straight lines.\n",
    "\n",
    "Below we show the same example as before, but with an RBF kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an instance of SVM and fit the data.\n",
    "clf = SVC(kernel='rbf', decision_function_shape='ovo') # Use the RBF kernel this time\n",
    "clf.fit(X_small, y)\n",
    "\n",
    "# Now plot the results\n",
    "h = .02  # step size in the mesh\n",
    "\n",
    "# Plot the decision boundary. For that, we will assign a color to each\n",
    "# point in the mesh [x_min, m_max]x[y_min, y_max].\n",
    "x_min, x_max = X_small[:, 0].min() - 1, X_small[:, 0].max() + 1\n",
    "y_min, y_max = X_small[:, 1].min() - 1, X_small[:, 1].max() + 1\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "                     np.arange(y_min, y_max, h))\n",
    "\n",
    "Z = clf.predict(np.c_[xx.ravel(), yy.ravel()]) # Make a prediction oat every point \n",
    "                                               # in the mesh in order to find the \n",
    "                                               # classification areas for each label\n",
    "\n",
    "# Put the result into a color plot\n",
    "Z = Z.reshape(xx.shape)\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.pcolormesh(xx, yy, Z, cmap=cmap_light)\n",
    "\n",
    "# Plot the training points\n",
    "plt.scatter(X_small[:, 0], X_small[:, 1], c=y, cmap=cmap_bold)\n",
    "plt.xlim(xx.min(), xx.max())\n",
    "plt.ylim(yy.min(), yy.max())\n",
    "plt.title(\"3-Class classification (SVM)\")\n",
    "plt.xlabel('Sepal length (cm)')\n",
    "plt.ylabel('Sepal width (cm)')\n",
    "\n",
    "# Display the new examples as labeled text on the graph\n",
    "plt.text(examples[0][0], examples[0][1],'A', fontsize=14)\n",
    "plt.text(examples[1][0], examples[1][1],'B', fontsize=14)\n",
    "\n",
    "# Plot the legend\n",
    "plt.legend(labelList, labelNames)\n",
    "\n",
    "plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The boundaries are very similar to before, but now they're curved instead of straight. Now let's add the decision boundaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create an instance of SVM and fit the data.\n",
    "clf = SVC(kernel='rbf', decision_function_shape='ovo') # Use the RBF kernel this time\n",
    "clf.fit(X_small, y)\n",
    "\n",
    "# Now plot the results\n",
    "h = .02  # step size in the mesh\n",
    "\n",
    "# Plot the decision boundary. For that, we will assign a color to each\n",
    "# point in the mesh [x_min, m_max]x[y_min, y_max].\n",
    "x_min, x_max = X_small[:, 0].min() - 1, X_small[:, 0].max() + 1\n",
    "y_min, y_max = X_small[:, 1].min() - 1, X_small[:, 1].max() + 1\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "                     np.arange(y_min, y_max, h))\n",
    "\n",
    "Z = clf.predict(np.c_[xx.ravel(), yy.ravel()]) # Make a prediction oat every point \n",
    "                                               # in the mesh in order to find the \n",
    "                                               # classification areas for each label\n",
    "\n",
    "# Put the result into a color plot\n",
    "Z = Z.reshape(xx.shape)\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.pcolormesh(xx, yy, Z, cmap=cmap_light)\n",
    "\n",
    "# Plot the training points\n",
    "plt.scatter(X_small[:, 0], X_small[:, 1], c=y, cmap=cmap_bold)\n",
    "plt.xlim(xx.min(), xx.max())\n",
    "plt.ylim(yy.min(), yy.max())\n",
    "plt.title(\"3-Class classification (SVM)\")\n",
    "plt.xlabel('Sepal length (cm)')\n",
    "plt.ylabel('Sepal width (cm)')\n",
    "\n",
    "# Display the new examples as labeled text on the graph\n",
    "plt.text(examples[0][0], examples[0][1],'A', fontsize=14)\n",
    "plt.text(examples[1][0], examples[1][1],'B', fontsize=14)\n",
    "\n",
    "# Plot the legend\n",
    "plt.legend(labelList, labelNames)\n",
    "\n",
    "plot_svc_decision_function(clf) # Plot the decison function\n",
    "\n",
    "plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the plot looks very different from before! The solid black lines are now all curves, but each decision boundary still falls along one part of those lines. And instead of having dotted lines parallel to the solid, there are smaller ellipsoids on either side of the solid line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What other kernels exist?\n",
    "---\n",
    "\n",
    "Scikit-learn comes with two other default kernels: polynomial and sigmoid. Advanced users can also creat their own kernels, but we will stick to the defaults for now.\n",
    "\n",
    "Below, modify our previous examples to try out the other kernels. How do they change the decision boundaries?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about my other features?\n",
    "===\n",
    "\n",
    "We've been looking at two features: the length and width of the plant's sepal. But what about the other two featurs, petal length and width? What does the graph look like when train on the petal length and width? How does it change when you change the SVM kernel?\n",
    "\n",
    "How would you plot our two new plants, A and B, on these new plots? Assume we have all four measurements for each plant, as shown below.\n",
    "\n",
    "Plant | Sepal length | Sepal width| Petal length | Petal width\n",
    "------|--------------|------------|--------------|------------\n",
    "A     |4.3           |2.5         | 1.5          | 0.5\n",
    "B     |6.3           |2.1         | 4.8          | 1.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using more than two features\n",
    "---\n",
    "\n",
    "Sticking to two features is great for visualization, but is less practical for solving real machine learning problems. If you have time, you can experiment with using more features to train your classifier. It gets much harder to visualize the results with 3 features, and nearly impossible with 4 or more. There are techniques that can help you visualize the data in some form, and there are also ways to reduce the number of features you use while still retaining (hopefully) the same amount of information. However, those techniques are beyond the scope of this class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluating The Classifier\n",
    "===\n",
    "\n",
    "In order to evaluate a classifier, we need to split our dataset into training data, which we'll show to the classifier so it can learn, and testing data, which we will hold back from training and use to test its predictions.\n",
    "\n",
    "Below, we create the training and testing datasets, using all four features. We then train our classifier on the training data, and get the predictions for the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)\n",
    "\n",
    "# Create an instance of SVM and fit the data\n",
    "clf = SVC(kernel='linear', decision_function_shape='ovo')\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Predict the labels of the test data\n",
    "predictions = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we evaluate how well the classifier did. The easiest way to do this is to get the average number of correct predictions, usually referred to as the accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "accuracy = np.mean(predictions == y_test )*100\n",
    "\n",
    "print('The accuracy is %.2f' % accuracy + '%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Comparing Models with Crossvalidation\n",
    "===\n",
    "\n",
    "To select the kernel to use in our model, we need to use crossvalidation. We can then get our final result using our test data.\n",
    "\n",
    "First we choose the kernels we want to investigate, then divide our training data into folds. We loop through the sets of training and validation folds. Each time, we train each model on the training data and evaluate on the validation data. We store the accuracy of each classifier on each fold so we can look at them later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose our kernels\n",
    "kernels = ['linear', 'rbf']\n",
    "\n",
    "# Create a dictionary of arrays to store accuracies\n",
    "accuracies = {}\n",
    "for kernel in kernels:\n",
    "    accuracies[kernel] = []\n",
    "\n",
    "# Loop through 5 folds\n",
    "kf = KFold(n_splits=5)\n",
    "for trainInd, valInd in kf.split(X_train):\n",
    "    X_tr = X_train[trainInd,:]\n",
    "    y_tr = y_train[trainInd]\n",
    "    X_val = X_train[valInd,:]\n",
    "    y_val = y_train[valInd]\n",
    "    \n",
    "    # Loop through each kernel\n",
    "    for kernel in kernels:\n",
    "        \n",
    "        # Create the classifier\n",
    "        clf = SVC(kernel=kernel, decision_function_shape='ovo')\n",
    "        \n",
    "        # Train the classifier\n",
    "        clf.fit(X_tr, y_tr) \n",
    "        \n",
    "        # Make our predictions\n",
    "        pred = clf.predict(X_val)\n",
    "    \n",
    "        # Calculate the accuracy\n",
    "        accuracies[kernel].append(np.mean(pred == y_val))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select a Model\n",
    "---\n",
    "\n",
    "To select a model, we look at the average accuracy across all folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for kernel in kernels:\n",
    "    print('%s: %.2f' % (kernel, np.mean(accuracies[kernel])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Final Evaluation\n",
    "---\n",
    "\n",
    "The linear kernel gives us the highest accuracy, so we select it as our best model. Now we can evaluate it on our training set and get our final accuracy rating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = SVC(kernel='linear', decision_function_shape='ovo')\n",
    "clf.fit(X_train, y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "\n",
    "accuracy = np.mean(predictions == y_test) * 100\n",
    "\n",
    "print('The final accuracy is %.2f' % accuracy + '%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sidenote: Randomness and Results\n",
    "===\n",
    "\n",
    "Every time you run this notebook, you will get slightly different results. Why? Because data is randomly divided among the training/testing/validation data sets. Running the code again will create a different division of the data, and will make the results slightly different. However, the overall outcome should remain consistent and have approximately the same values. If you have drastically different results when running an analysis multiple times, it suggests a problem with your model or that you need more data.\n",
    "\n",
    "If it's important that you get the exact same results every time you run the code, you can specify a random state in the `random_state` argument of `train_test_split()` and `KFold`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}