WK3_Machine_Learning_Task.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this programming task, we'll try our hand at supervised learning. We'll work with a simple dataset and we'll build, evaluate and use two different machine learning models.<br><br>\n",
    "The dataset that we're going to be using in this programming task is from the botany domain, so it has to do with plants. Note that in this week's Programming Assignment you'll get to work with medical data!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Importing _scikit-learn_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_scikit-learn_ is the most widely used Python library for machine learning. \n",
    "We first need to tell Python that we're going to be using scikit-learn, with the use of the import command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "## Part 2: Familiarising ourselves with the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "scikit-learn comes with a few small standard datasets that do not require to download and to read data from external websites. In this programming task, we are going to be using the **Iris Plants Dataset**. This dataset contains information about different iris flowers, i.e. sepal length, sepal width, petal length, petal width and species (with three possible values for species: setosa, versicolor and virginica). <br><br>\n",
    "The iris dataset is typically used for supervised learning tasks, and in particular for classification. The idea is that we have measurements (i.e. sepal length, sepal width, petal length and petal width) for which we know the correct species. So if we go out in nature and find some iris flowers and measure their sepal length, sepal width, petal length and petal width, then we can use the iris dataset to predict which species each flower belongs to. Nice, ha? And since there are three possible values for the iris species, it's a classification task.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the dataset with the use of the *load_iris* function. We'll call it *iris_dataset* (but you could call it anything you want)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "iris_dataset = load_iris()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting a sense of the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *iris_dataset* object that is returned by *load_iris* is a *Bunch* object, which contains some information about the dataset, as well as the actual data.\n",
    "Bunch objects are very similar to dictionaries (we were introduced to dictionaries in Week 1) and they contain keys and values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code below to print the keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"Keys of iris_dataset: \", iris_dataset.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are five types of information in the dataset:\n",
    "* DESCR\n",
    "* feature_names\n",
    "* target_names\n",
    "* data\n",
    "* target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a closer look at each one of them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_DESCR_ is a short description of the dataset. Run the code below to get an extract of the first 200 characters. If you want to get a bigger extract, all you need to do is change *200* to a larger number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(iris_dataset['DESCR'][:200] + \"\\n.......\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*feature_names* corresponds to the names of all the features in the dataset, in other words all the variables that we take into account when building our machine learning model.\n",
    "Run the code below to print the names of all features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"Feature names: \", iris_dataset['feature_names'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*target_names* corresponds to the class labels.\n",
    "By running the code below, we can see that there are three class labels: 'setosa', 'versicolor' and 'virginica'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"Target names: \", iris_dataset['target_names'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The actual data is contained in the *data* and *target* fields. \n",
    "*data* contains the values for the different features, e.g. sepal length."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code in the next cell to get the shape of *data*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(iris_dataset['data'].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that we have data for 150 iris flowers. For each flower case we have 4 features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code in the next cell to get the first three rows in *data*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"First three rows of data:\\n\", iris_dataset['data'][:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to this output, we get the following values for the first flower:\n",
    "* sepal length (cm): 5.1 \n",
    "* sepal width (cm): 3.5\n",
    "* petal length (cm): 1.4\n",
    "* petal width (cm): 0.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Small challenge**: What if you wanted to get the first 6 rows of data? Write your code below and run it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code in the next two cells to get the shape of *target* and the first two elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"Shape of target: \", iris_dataset['target'].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"First two elements in target: \", iris_dataset['target'][:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that *target* contains the species for each of the 150 iris flowers in the database. The species of the first two flowers is setosa, as 0 corresponds to setosa, 1 to versicolor and 2 to virginica. (How de we know this? It is a convention that elements in *target_names* appear in an increasing order, starting from 0.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "## Part 3: Splitting our dataset into training data and test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before using our model for previously unseen iris flowers, we need to know how well it performs. To do this, we split our labelled data in two parts: i) a training dataset that we use for building the model, and ii) a test dataset that we use for testing the accuracy of our model. We do this with the use of the *train_test_split* function, which shuffles the dataset randomly, and by default extracts 75% of the cases as training data and 25% of the cases as test data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code below to split the iris dataset into training and test data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    iris_dataset['data'], iris_dataset['target'], random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is standard nomenclature. *X* corresponds to data (as in *data* in *iris_dataset*) and *y* to labels (as in *target* in *iris_dataset*). So, for the training dataset we get *X_train* and *y_train*, while for the test dataset we get *X_test* and *y_test*.\n",
    "\n",
    "Note that if you wanted to split a different dataset called \"my_dataset\" into a training and a test dataset, then all you would need to do is substitute \"iris_dataset\" with \"my_dataset\" in the code above. \n",
    "\n",
    "By setting *random_state=0* we are making sure that, even though our dataset is randomly shuffled by the *train_test_split* function, we can reproduce our results by using the same fixed seed for the random number generator (in this case 0). So if in the future you want to reproduce the same training and test data, all you need to do is use *random_state=0*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code below to get the shape of *X_train* and *y_train*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"X_train shape: \", X_train.shape)\n",
    "print(\"y_train shape: \", y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"X_test shape: \", X_test.shape)\n",
    "print(\"y_test shape: \", y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion prompt**: What do the outputs from the two previous cells mean? Post your thoughts in the discussion forums!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that it is good practice to visualise our data to get a sense of how different features are related or to spot any abnormalities. We will not do this in this programming task so as to keep things simple and save time, but it is worth keeping in mind for the future."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> \n",
    "## Part 4: Creating our first model: K Nearest Neighbours"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now learn how to build a classification model for the iris dataset with the use of the k nearest neighbours algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To build a k nearest neighbours model, we will use the *KNeighborsClassifier* class from the *sklearn.neighbors* module.\n",
    "\n",
    "Run the code below to create a *KNeighborsClassifier* object called *knn* (but we could give it any name we want). Note that *n_neighbors=1* is setting the number of nearest neighbours to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "knn = KNeighborsClassifier(n_neighbors=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code below to build the model on the training set, i.e. *X_train* and *y_train*.\n",
    "\n",
    "You can ignore the output for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "knn.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now use the test dataset to evaluate the accuracy of our model. We can do this with the use of the *score* method, as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"Test set score: \", knn.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code in the next cell contains a variation of the previous code, in case you want to get the value of *knn.score(X_test, y_test)* rounded to three decimal places. If you wanted it rounded to two decimal places, then all you would need to do is change *{:.3f}* to *{:.2f}*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"Test set score rounded to three decimal places: {:.3f}\".format(knn.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*How is the accuracy of our model calculated?* Essentially, our model is used to make predictions for *X_test* and the values predicted are compared to the actual labels in *y_test*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the model to make predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now use our model to make a prediction about a previously unseen iris flower case. We will first import the numpy libary, then we will specify the previously unseen iris flower case (we'll call it *X_unseen*) and finally we will use the *predict* method on *X_unseen* to get the prediction (we'll call the result *prediction*, but we could use any name we want)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_unseen = np.array([[5.3, 2.7, 1, 0.3]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "prediction = knn.predict(X_unseen)\n",
    "\n",
    "print(\"Prediction label: \", prediction)\n",
    "print(\"Predicted target name: \", iris_dataset['target_names'][prediction])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to this output, the prediction for case *X_unseen* is setosa."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tweaking the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can play around with the model to try different numbers of k nearest neighbours, e.g. 3 or 4.\n",
    "\n",
    "**Challenge**: Provide some code below to build and evaluate such a model. All you need to do is reuse and modify parts of the code above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "## Part 5: Creating a different model: Decision Trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now learn how to build a classification model for the iris dataset with the use of the decision tree classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Important note*: We would normally use the same training and test data as before, so we would reuse *X_train*, *X_test*, *y_train* and *y_test* (so as to compare the results of the K Nearest Neighbours and Decision Tree models). However, in this programming task we will re-split the iris dataset into training and test data, in order to illustrate how we can get a new version by using a different fixed seed (in this case, 7). If you plan to use this notebook as a template for future machine learning projects, then you can delete the next cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    iris_dataset['data'], iris_dataset['target'], random_state=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To build a decision tree model, we will use the *DecisionTreeClassifier* class from the *sklearn.tree* module.\n",
    "\n",
    "Run the code below to create a *DecisionTreeClassifier* object called *tree* and to fit the model on the training set, (i.e. *X_train* and *y_train*). You can ignore the information outputed.\n",
    "\n",
    "Note that the decision tree classifier algorithm contains some randomness aspects (explaining these is beyond the scope of this course), so by setting *random_state=12* we can reproduce our results by using the same fixed seed for the random number generator (in this case 12)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "tree = DecisionTreeClassifier(random_state=12)\n",
    "tree.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code below to evaluate the accuracy of the decision tree model that we just built. We'll distinguish between accuracy on the training set and accuracy on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"Accuracy on training set: \", tree.score(X_train, y_train))\n",
    "print(\"Accuracy on test set: \", tree.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The decision tree built has accuracy 100% on the training dataset. This means that our decision tree is over-fitting the training data.\n",
    "\n",
    "In order to avoid overfitting (and hopefully improve the accuracy of the model on test data), we can stop before the entire tree is created. We can do this by setting the maximal depth of the tree.\n",
    "\n",
    "Run the code below to create a new version of the tree with maximal depth 3. Note that the only difference to the code in the previous cell is *max_depth=3*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "tree = DecisionTreeClassifier(max_depth=3, random_state=12)\n",
    "tree.fit(X_train, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: \", tree.score(X_train, y_train))\n",
    "print(\"Accuracy on test set: \", tree.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The new decision tree has lower accuracy on the training dataset, but higher accuracy on the test dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now use our decision tree to make a prediction for the previously unseen iris case *X_unseen*, which was defined earlier in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "prediction = tree.predict(X_unseen)\n",
    "\n",
    "print(\"Prediction label: \", prediction)\n",
    "print(\"Predicted target name: \", iris_dataset['target_names'][prediction])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to this output, the prediction for case *X_unseen* is setosa. This prediction is in line with the prediction that we got using the K Nearest Neighbours classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "## Part 6 (Optional): Practise further"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We highly recommend that you practise further with what you've learnt in this programming task. Here are some ideas to get you started:\n",
    "- Build a K Nearest Neighbours model for a different number of neighbours and evaluate it. \n",
    "- Build a Decision Tree model with a different maximal depth and evaluate it.\n",
    "- Build a Decision Tree model on the original training data (i.e. for the original split of data with random_state=0) and evaluate it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}