Anurag_Chatterjee_&_Aryan_Saraswat_Lab_3.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TXQs59G5QWLn"
      },
      "source": [
        "# CS3237 Lab 3 Statistical Methods\n",
        "\n",
        "\n",
        "|Student Nunmber|Name                  |\n",
        "|:--------------|:---------------------|\n",
        "|   A0200521E   |     Aryan Sarswat    |\n",
        "|   A0200533Y   | Anurag S. Chatterjee |\n",
        "\n",
        "In this lab you will also be do some experiments to familiarize yourself with the linear regression, Naive Bayes and Support Vector Machine library in SciKit Learn. \n",
        "\n",
        "Please work together as a team of 2 to complete this lab. You will need to submit ONE copy of this notebook per team, but please fill in the names of both team members above. This lab is worth 40 marks (but it is still 5% CA like your other labs):\n",
        "\n",
        "**DO NOT SUBMIT MORE THAN ONE COPY OF THIS LAB!**\n",
        "\n",
        "Let's now begin using statistical techniques in SciKit Learn. \n",
        "\n",
        "## 1. SciKit Learn Hands-on\n",
        "\n",
        "We will now run some experiments to familiarize you with the statistical learning tools in SciKit Learn.\n",
        "\n",
        "### 1.1 Linear Regression\n",
        "\n",
        "Let's begin by playing around with the linear regression we did for the Boston Housing Dataset during the lecture. \n",
        "\n",
        "#### 1.1.1 Finding Better Correlations\n",
        "\n",
        "_Question 0: (4 marks)_\n",
        "\n",
        "In the lecture we took a quick look at correlating housing prices and poverty levels.  Using the code cell below:\n",
        "\n",
        "    1. Recreate the regression example from the lecture Jupyter Notebook. Set random_state to 0 for train_test_split.\n",
        "    2. Add code to find the correlation between housing prices and the other independent variables in the dataset. Leave out CHAS (Charles River dummy variable) and MEDV.\n",
        "    3. As before save 33% of the data for testing and set random_state to 0.\n",
        "    4. Create a new simple (single independent variable) regression model with the independent variable with the absolute highest correlation. If poverty levels is the highest, then choose the next highest.\n",
        "    5. Compute and print the MSE for training data and testing data, and answer the questions after the code block.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:33:13.725774Z",
          "start_time": "2022-09-08T05:33:12.663505Z"
        },
        "id": "DOIv8eGWQWLr"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import scipy.stats as stats\n",
        "import sklearn"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2r6nO8cIQWLs"
      },
      "source": [
        "Load Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:33:13.817838Z",
          "start_time": "2022-09-08T05:33:13.725774Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nFPhca3cQWLs",
        "outputId": "8043d77e-beb3-440e-d628-b412b36e01b1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function load_boston is deprecated; `load_boston` is deprecated in 1.0 and will be removed in 1.2.\n",
            "\n",
            "    The Boston housing prices dataset has an ethical problem. You can refer to\n",
            "    the documentation of this function for further details.\n",
            "\n",
            "    The scikit-learn maintainers therefore strongly discourage the use of this\n",
            "    dataset unless the purpose of the code is to study and educate about\n",
            "    ethical issues in data science and machine learning.\n",
            "\n",
            "    In this special case, you can fetch the dataset from the original\n",
            "    source::\n",
            "\n",
            "        import pandas as pd\n",
            "        import numpy as np\n",
            "\n",
            "\n",
            "        data_url = \"http://lib.stat.cmu.edu/datasets/boston\"\n",
            "        raw_df = pd.read_csv(data_url, sep=\"\\s+\", skiprows=22, header=None)\n",
            "        data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])\n",
            "        target = raw_df.values[1::2, 2]\n",
            "\n",
            "    Alternative datasets include the California housing dataset (i.e.\n",
            "    :func:`~sklearn.datasets.fetch_california_housing`) and the Ames housing\n",
            "    dataset. You can load the datasets as follows::\n",
            "\n",
            "        from sklearn.datasets import fetch_california_housing\n",
            "        housing = fetch_california_housing()\n",
            "\n",
            "    for the California housing dataset and::\n",
            "\n",
            "        from sklearn.datasets import fetch_openml\n",
            "        housing = fetch_openml(name=\"house_prices\", as_frame=True)\n",
            "\n",
            "    for the Ames housing dataset.\n",
            "    \n",
            "  warnings.warn(msg, category=FutureWarning)\n"
          ]
        }
      ],
      "source": [
        "\"\"\"\n",
        "    Enter your code for part 1.1.1 here, and answer the questions\n",
        "    after this code cell.\n",
        "\"\"\"\n",
        "\n",
        "from sklearn.datasets import load_boston\n",
        "boston = load_boston()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create dataframe named bos\n",
        "bos = pd.DataFrame(boston.data)\n",
        "bos.columns = boston.feature_names\n",
        "bos.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "-VwnEd8MNiCg",
        "outputId": "a6b57a22-6e86-4dac-92ee-5c85db7b1ff5"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \\\n",
              "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   \n",
              "1  0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   \n",
              "2  0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   \n",
              "3  0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   \n",
              "4  0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   \n",
              "\n",
              "   PTRATIO       B  LSTAT  \n",
              "0     15.3  396.90   4.98  \n",
              "1     17.8  396.90   9.14  \n",
              "2     17.8  392.83   4.03  \n",
              "3     18.7  394.63   2.94  \n",
              "4     18.7  396.90   5.33  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5d163569-7864-41e2-85fc-cdbb64bbf5fa\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CRIM</th>\n",
              "      <th>ZN</th>\n",
              "      <th>INDUS</th>\n",
              "      <th>CHAS</th>\n",
              "      <th>NOX</th>\n",
              "      <th>RM</th>\n",
              "      <th>AGE</th>\n",
              "      <th>DIS</th>\n",
              "      <th>RAD</th>\n",
              "      <th>TAX</th>\n",
              "      <th>PTRATIO</th>\n",
              "      <th>B</th>\n",
              "      <th>LSTAT</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.00632</td>\n",
              "      <td>18.0</td>\n",
              "      <td>2.31</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.538</td>\n",
              "      <td>6.575</td>\n",
              "      <td>65.2</td>\n",
              "      <td>4.0900</td>\n",
              "      <td>1.0</td>\n",
              "      <td>296.0</td>\n",
              "      <td>15.3</td>\n",
              "      <td>396.90</td>\n",
              "      <td>4.98</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.02731</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>6.421</td>\n",
              "      <td>78.9</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>396.90</td>\n",
              "      <td>9.14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.02729</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>7.185</td>\n",
              "      <td>61.1</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>392.83</td>\n",
              "      <td>4.03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.03237</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>6.998</td>\n",
              "      <td>45.8</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>394.63</td>\n",
              "      <td>2.94</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.06905</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>7.147</td>\n",
              "      <td>54.2</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>396.90</td>\n",
              "      <td>5.33</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5d163569-7864-41e2-85fc-cdbb64bbf5fa')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-5d163569-7864-41e2-85fc-cdbb64bbf5fa button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-5d163569-7864-41e2-85fc-cdbb64bbf5fa');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Add labels\n",
        "bos[\"PRICE\"] = boston.target"
      ],
      "metadata": {
        "id": "IXbXoQvHN1Hr"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wzcLsXvBQWLt"
      },
      "source": [
        "Find correlations"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:42:18.734608Z",
          "start_time": "2022-09-08T05:42:18.723243Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IwQRKPvIQWLt",
        "outputId": "5fc9f02d-876f-450f-e9d8-c1a05322ef2e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CHAS       0.175260\n",
            "DIS        0.249929\n",
            "B          0.333461\n",
            "ZN         0.360445\n",
            "AGE        0.376955\n",
            "RAD        0.381626\n",
            "CRIM       0.388305\n",
            "NOX        0.427321\n",
            "TAX        0.468536\n",
            "INDUS      0.483725\n",
            "PTRATIO    0.507787\n",
            "RM         0.695360\n",
            "LSTAT      0.737663\n",
            "PRICE      1.000000\n",
            "Name: PRICE, dtype: float64\n"
          ]
        }
      ],
      "source": [
        "# Find the correlation of other variables with price\n",
        "correlation_matrix = bos.corr()\n",
        "\n",
        "# Find absolute values of correlations in ascending order\n",
        "print(abs(correlation_matrix['PRICE']).sort_values()) "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-zbtUhEoQWLu"
      },
      "source": [
        "Train test split on RM as it has the highest correlation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:41:29.114344Z",
          "start_time": "2022-09-08T05:41:29.101583Z"
        },
        "id": "C-zgk697QWLu"
      },
      "outputs": [],
      "source": [
        "# Linear Regression In 1 Independant Variable\n",
        "X = bos['RM'].values.reshape(-1, 1)\n",
        "Y = bos['PRICE'].values.reshape(-1, 1)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.33, random_state=0) #Save 33% of the data for testing and set random state to 0"
      ],
      "metadata": {
        "id": "FDMS2gMdOK8U"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aTd9A5kXQWLv"
      },
      "source": [
        "Train Linear Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:41:32.784648Z",
          "start_time": "2022-09-08T05:41:32.768411Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aS-lKBdMQWLv",
        "outputId": "b14da7ec-a7b7-4105-f1d1-c4e4f495f99a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LinearRegression()"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "from sklearn.linear_model import LinearRegression\n",
        "lm = LinearRegression()\n",
        "lm.fit(X_train, Y_train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SXxpFik-QWLw"
      },
      "source": [
        "Check metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:41:38.554553Z",
          "start_time": "2022-09-08T05:41:38.533815Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8cCL2yxiQWLw",
        "outputId": "6f0d2767-61b1-4267-98df-624fb5b6dd98"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MSE for training data: 43.1318. MSE for testing data: 44.6075.\n"
          ]
        }
      ],
      "source": [
        "from sklearn import metrics\n",
        "from sklearn.metrics import mean_squared_error\n",
        "Y_pred_train = lm.predict(X_train)\n",
        "Y_pred_test = lm.predict(X_test)\n",
        "train_mse = mean_squared_error(Y_train, Y_pred_train)\n",
        "test_mse = mean_squared_error(Y_test, Y_pred_test)\n",
        "print(\"MSE for training data: %3.4f. MSE for testing data: %3.4f.\" % (train_mse, test_mse))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EizTIBeQQWLw"
      },
      "source": [
        "Answer the following questions between the \\*\\* markdowns so that your answers appear in bold.\n",
        "\n",
        "***\n",
        "_Question 1: Which independent variable has the highest correlation? Did it have any effect on your training and test accuracy scores? Why or why not?_ (2 marks)\n",
        "\n",
        "***\n",
        "\n",
        "**Answer: RM has the second highest correlation value which is 0.695360. The higher the correlation coefficient the strong the linear relationship between the variable and the target. Thus choosing the highest correlation value leads to a higher training and test accuracy score overall. Correlation coefficient is proportional to the error, so the higher the correlation you take, means that error between datapoints is smaller, so your training and testing accuracy scores will be better.** \n",
        "\n",
        "#### 1.1.2 Creating Multivariate Linear Regressions ####\n",
        "\n",
        "SciKit learn can create linear regression models with multiple independent variables, and in this section we are going to explore how to do this, and whether or not it makes a difference in our Boston Dataset.\n",
        "\n",
        "One way to create a multivariate model is to:\n",
        "\n",
        "    1. Rank the independent variables by correlation, then create a linear model using the independent variable with the highest correlation. Measure the training and testing accuracy.\n",
        "    2. Add in the independent variable with the next highest correlation and create a new linear model.  Measure the training and testing accuracy.\n",
        "    3. Stop when either accuracy score levels off or goes down.\n",
        "\n",
        "Answer the following questions to help you along with creating your multivariate model:\n",
        "\n",
        "***\n",
        "\n",
        "_Question 2: Explain what the following code fragment does. You may refer to NumPy and SciKit Learn documentation_ (2 marks)\n",
        "\n",
        "```\n",
        "bos['PRICE'].values.reshape(-1, 1)\n",
        "```\n",
        "\n",
        "**Answer: The above code fragment helps to give a new shape to the boston housing prices array without changing its data type. So it takes the series named \"PRICE\" from the boston housing dataset and converts into a numpy array, from of dimensions (n,) to dimensions (n,1) so basically -1 is a placeholder which helps numpy to calculate the dimensions for us instead of us having to specify it.**\n",
        "\n",
        "_Question 3: Consult the NumPy documentation: What does the 'concatenate' function do? In particular what does 'axis=1' do?_ (2 marks)\n",
        "\n",
        "**Answer: The numpy.concatenate function, numpy.concatenate((a1, a2, ...), axis=0, out=None, dtype=None, casting=\"same_kind\") helps to join a sequence of numpy arrays of the same shape along a specified axis and merge it together into a single, compiled array. The a1 and a2 are arrays which you want to concatenate and must have the same shape, except in the dimension corresponding to axis (the first, by default). The out parameter, if provided, the destination to place the result. The shape must be correct, matching that of what concatenate would have returned if no out argument were specified. The axis parameter is the line along which the arrays will be joined/concatenated. If axis is None, arrays are flattened before use. Default is 0, and if the axis=1, then the numpy arrays a1 and a2, which are of the same shape, will be concatenated horizontally, along the column axis instead of the rows.**\n",
        "\n",
        "_Question 4: Given your answers to Questions 2 and 3, what does the following code do?_ (2 marks)\n",
        "\n",
        "```\n",
        "import numpy as np\n",
        "\n",
        "... Other code here ...\n",
        "\n",
        "X1 = bos['INDUS'].values.reshape(-1, 1)\n",
        "X2 = bos['CRIM'].values.reshape(-1, 1)\n",
        "X = np.concatenate((X1, X2), axis = 1)\n",
        "```\n",
        "\n",
        "**Answer: The above combines the X1 and X2 arrays into a single array such that the entries of X are (X1_i, X2_i) where i is the index of the respective X1 and X2 entries. The above code fragment helps to give a new shape to the boston housing prices values, with the series named \"INDUS\" (proportion of non-retail business acres per town) and the series named \"CRIM\" (per capita crime rate by town) array without changing its data type. So it will convert these 2 array dimensions, from (n,) to dimensions (n,1) so basically -1 is a placeholder which helps numpy to calculate the dimensions for us instead of us having to specify it. Once this is done, then, the last code fragment will help concatente both the numpy array values concatenated horizontally, along the column axis.**\n",
        "\n",
        "***\n",
        "\n",
        "Use the following code cell to follow the steps above to create models with one, two and three independent variables, printing the training and testing accuracy each time. Note that you have to run _train_test_split_ for each model. Set the _random_state_ parameter in _train_test_split_ to 0 each time. (4 marks)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:42:50.192121Z",
          "start_time": "2022-09-08T05:42:50.176117Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uS4TJQ5kQWLx",
        "outputId": "f212d74e-7a0b-4e50-dd84-3e11e5f273e7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INDUS      0.483725\n",
            "PTRATIO    0.507787\n",
            "RM         0.695360\n",
            "LSTAT      0.737663\n",
            "PRICE      1.000000\n",
            "Name: PRICE, dtype: float64\n"
          ]
        }
      ],
      "source": [
        "\"\"\"\n",
        "    Enter your code for part 1.1.2 here. \n",
        "\"\"\"\n",
        "\n",
        "#Get the top 3 independant variables\n",
        "correlation_matrix = bos.corr()\n",
        "print(abs(correlation_matrix['PRICE']).sort_values()[-5:])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:33:13.989804Z",
          "start_time": "2022-09-08T05:33:13.973805Z"
        },
        "id": "aD3rZqNQQWLx"
      },
      "outputs": [],
      "source": [
        "#For one independant varible refer to the first part of this question"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:43:07.966642Z",
          "start_time": "2022-09-08T05:43:07.946367Z"
        },
        "id": "E-gvWsBiQWLy"
      },
      "outputs": [],
      "source": [
        "# Linear Regression In 2 Independant Variables\n",
        "X1 = bos['PTRATIO'].values.reshape(-1, 1)\n",
        "X2 = bos['RM'].values.reshape(-1, 1)\n",
        "X = np.concatenate((X1, X2), axis = 1)\n",
        "Y = bos['PRICE'].values.reshape(-1, 1)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.33, random_state=0) #Save 33% of the data for testing and set random state to 0\n",
        "\n",
        "lm = LinearRegression()\n",
        "lm.fit(X_train,Y_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t7pqrmd5QgaR",
        "outputId": "e3d4acf6-4e1b-44f8-95a9-6db0ba31018b"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LinearRegression()"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn import metrics \n",
        "from sklearn.metrics import mean_squared_error\n",
        "Y_pred_train = lm.predict(X_train) \n",
        "Y_pred_test = lm.predict(X_test)   \n",
        "train_mse = mean_squared_error(Y_train, Y_pred_train)\n",
        "test_mse = mean_squared_error(Y_test, Y_pred_test)\n",
        "print(\"MSE for training data: %3.4f. MSE for testing data: %3.4f.\" % (train_mse, test_mse))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rfoeEU1yQtQi",
        "outputId": "ac70a6d9-cec3-4872-c78f-8a91714c9e5f"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MSE for training data: 34.5563. MSE for testing data: 42.3877.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Linear Regression In 3 Independant Variables\n",
        "X1 = bos['INDUS'].values.reshape(-1, 1)\n",
        "X2 = bos['PTRATIO'].values.reshape(-1, 1)\n",
        "X3 = bos['RM'].values.reshape(-1, 1)\n",
        "X = np.concatenate((X1, X2, X3), axis = 1)\n",
        "Y = bos['PRICE'].values.reshape(-1, 1)"
      ],
      "metadata": {
        "id": "_FipcjMGXh56"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.33, random_state=0) #Save 33% of the data for testing and set random state to 0\n",
        "\n",
        "lm = LinearRegression()\n",
        "lm.fit(X_train,Y_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QTLg-3b9X8ep",
        "outputId": "ebff37bc-918f-4be8-97d2-2e309847e52a"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LinearRegression()"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn import metrics \n",
        "from sklearn.metrics import mean_squared_error\n",
        "Y_pred_train = lm.predict(X_train) \n",
        "Y_pred_test = lm.predict(X_test)   \n",
        "train_mse = mean_squared_error(Y_train, Y_pred_train)\n",
        "test_mse = mean_squared_error(Y_test, Y_pred_test)\n",
        "print(\"MSE for training data: %3.4f. MSE for testing data: %3.4f.\" % (train_mse, test_mse))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uctk2mteX-xp",
        "outputId": "d8f02f56-25de-4e06-bc0c-473c2cd6e4bf"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MSE for training data: 32.1271. MSE for testing data: 41.0605.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QLyAdV1pQWLz"
      },
      "source": [
        "### 1.2 Creating a Naive Bayes Classifier ###\n",
        "\n",
        "We will now look at how to create a Naive Bayes Classifier, and later on a Support Vector Machine classifier. We will also explore the use of _GridSearchCV_ to optimize the choice of parameters for the SVC.\n",
        "\n",
        "#### 1.2.1 The Irises Dataset ###\n",
        "\n",
        "In this lab we will use the irises dataset to classify four categories of irises (a species of flowers). We will consider four factors:\n",
        "\n",
        "    1. Sepal length in cm\n",
        "    2. Sepal width in cm\n",
        "    3. Petal length in cm\n",
        "    4. Petal width in cm\n",
        "\n",
        "The image below shows what these mean:\n",
        "\n",
        "![iris.png](attachment:image.png)\n",
        "\n",
        "The code cell below loads up the Iris dataset, prints it out, then scales it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:44:14.967207Z",
          "start_time": "2022-09-08T05:44:14.942695Z"
        },
        "scrolled": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MLzTq7qSQWLz",
        "outputId": "0b3646eb-70cc-43fb-f49d-d2b8a0d40440"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Iris Data:\n",
            "[[5.1 3.5 1.4 0.2]\n",
            " [4.9 3.  1.4 0.2]\n",
            " [4.7 3.2 1.3 0.2]\n",
            " [4.6 3.1 1.5 0.2]\n",
            " [5.  3.6 1.4 0.2]\n",
            " [5.4 3.9 1.7 0.4]\n",
            " [4.6 3.4 1.4 0.3]\n",
            " [5.  3.4 1.5 0.2]\n",
            " [4.4 2.9 1.4 0.2]\n",
            " [4.9 3.1 1.5 0.1]\n",
            " [5.4 3.7 1.5 0.2]\n",
            " [4.8 3.4 1.6 0.2]\n",
            " [4.8 3.  1.4 0.1]\n",
            " [4.3 3.  1.1 0.1]\n",
            " [5.8 4.  1.2 0.2]\n",
            " [5.7 4.4 1.5 0.4]\n",
            " [5.4 3.9 1.3 0.4]\n",
            " [5.1 3.5 1.4 0.3]\n",
            " [5.7 3.8 1.7 0.3]\n",
            " [5.1 3.8 1.5 0.3]\n",
            " [5.4 3.4 1.7 0.2]\n",
            " [5.1 3.7 1.5 0.4]\n",
            " [4.6 3.6 1.  0.2]\n",
            " [5.1 3.3 1.7 0.5]\n",
            " [4.8 3.4 1.9 0.2]\n",
            " [5.  3.  1.6 0.2]\n",
            " [5.  3.4 1.6 0.4]\n",
            " [5.2 3.5 1.5 0.2]\n",
            " [5.2 3.4 1.4 0.2]\n",
            " [4.7 3.2 1.6 0.2]\n",
            " [4.8 3.1 1.6 0.2]\n",
            " [5.4 3.4 1.5 0.4]\n",
            " [5.2 4.1 1.5 0.1]\n",
            " [5.5 4.2 1.4 0.2]\n",
            " [4.9 3.1 1.5 0.2]\n",
            " [5.  3.2 1.2 0.2]\n",
            " [5.5 3.5 1.3 0.2]\n",
            " [4.9 3.6 1.4 0.1]\n",
            " [4.4 3.  1.3 0.2]\n",
            " [5.1 3.4 1.5 0.2]\n",
            " [5.  3.5 1.3 0.3]\n",
            " [4.5 2.3 1.3 0.3]\n",
            " [4.4 3.2 1.3 0.2]\n",
            " [5.  3.5 1.6 0.6]\n",
            " [5.1 3.8 1.9 0.4]\n",
            " [4.8 3.  1.4 0.3]\n",
            " [5.1 3.8 1.6 0.2]\n",
            " [4.6 3.2 1.4 0.2]\n",
            " [5.3 3.7 1.5 0.2]\n",
            " [5.  3.3 1.4 0.2]\n",
            " [7.  3.2 4.7 1.4]\n",
            " [6.4 3.2 4.5 1.5]\n",
            " [6.9 3.1 4.9 1.5]\n",
            " [5.5 2.3 4.  1.3]\n",
            " [6.5 2.8 4.6 1.5]\n",
            " [5.7 2.8 4.5 1.3]\n",
            " [6.3 3.3 4.7 1.6]\n",
            " [4.9 2.4 3.3 1. ]\n",
            " [6.6 2.9 4.6 1.3]\n",
            " [5.2 2.7 3.9 1.4]\n",
            " [5.  2.  3.5 1. ]\n",
            " [5.9 3.  4.2 1.5]\n",
            " [6.  2.2 4.  1. ]\n",
            " [6.1 2.9 4.7 1.4]\n",
            " [5.6 2.9 3.6 1.3]\n",
            " [6.7 3.1 4.4 1.4]\n",
            " [5.6 3.  4.5 1.5]\n",
            " [5.8 2.7 4.1 1. ]\n",
            " [6.2 2.2 4.5 1.5]\n",
            " [5.6 2.5 3.9 1.1]\n",
            " [5.9 3.2 4.8 1.8]\n",
            " [6.1 2.8 4.  1.3]\n",
            " [6.3 2.5 4.9 1.5]\n",
            " [6.1 2.8 4.7 1.2]\n",
            " [6.4 2.9 4.3 1.3]\n",
            " [6.6 3.  4.4 1.4]\n",
            " [6.8 2.8 4.8 1.4]\n",
            " [6.7 3.  5.  1.7]\n",
            " [6.  2.9 4.5 1.5]\n",
            " [5.7 2.6 3.5 1. ]\n",
            " [5.5 2.4 3.8 1.1]\n",
            " [5.5 2.4 3.7 1. ]\n",
            " [5.8 2.7 3.9 1.2]\n",
            " [6.  2.7 5.1 1.6]\n",
            " [5.4 3.  4.5 1.5]\n",
            " [6.  3.4 4.5 1.6]\n",
            " [6.7 3.1 4.7 1.5]\n",
            " [6.3 2.3 4.4 1.3]\n",
            " [5.6 3.  4.1 1.3]\n",
            " [5.5 2.5 4.  1.3]\n",
            " [5.5 2.6 4.4 1.2]\n",
            " [6.1 3.  4.6 1.4]\n",
            " [5.8 2.6 4.  1.2]\n",
            " [5.  2.3 3.3 1. ]\n",
            " [5.6 2.7 4.2 1.3]\n",
            " [5.7 3.  4.2 1.2]\n",
            " [5.7 2.9 4.2 1.3]\n",
            " [6.2 2.9 4.3 1.3]\n",
            " [5.1 2.5 3.  1.1]\n",
            " [5.7 2.8 4.1 1.3]\n",
            " [6.3 3.3 6.  2.5]\n",
            " [5.8 2.7 5.1 1.9]\n",
            " [7.1 3.  5.9 2.1]\n",
            " [6.3 2.9 5.6 1.8]\n",
            " [6.5 3.  5.8 2.2]\n",
            " [7.6 3.  6.6 2.1]\n",
            " [4.9 2.5 4.5 1.7]\n",
            " [7.3 2.9 6.3 1.8]\n",
            " [6.7 2.5 5.8 1.8]\n",
            " [7.2 3.6 6.1 2.5]\n",
            " [6.5 3.2 5.1 2. ]\n",
            " [6.4 2.7 5.3 1.9]\n",
            " [6.8 3.  5.5 2.1]\n",
            " [5.7 2.5 5.  2. ]\n",
            " [5.8 2.8 5.1 2.4]\n",
            " [6.4 3.2 5.3 2.3]\n",
            " [6.5 3.  5.5 1.8]\n",
            " [7.7 3.8 6.7 2.2]\n",
            " [7.7 2.6 6.9 2.3]\n",
            " [6.  2.2 5.  1.5]\n",
            " [6.9 3.2 5.7 2.3]\n",
            " [5.6 2.8 4.9 2. ]\n",
            " [7.7 2.8 6.7 2. ]\n",
            " [6.3 2.7 4.9 1.8]\n",
            " [6.7 3.3 5.7 2.1]\n",
            " [7.2 3.2 6.  1.8]\n",
            " [6.2 2.8 4.8 1.8]\n",
            " [6.1 3.  4.9 1.8]\n",
            " [6.4 2.8 5.6 2.1]\n",
            " [7.2 3.  5.8 1.6]\n",
            " [7.4 2.8 6.1 1.9]\n",
            " [7.9 3.8 6.4 2. ]\n",
            " [6.4 2.8 5.6 2.2]\n",
            " [6.3 2.8 5.1 1.5]\n",
            " [6.1 2.6 5.6 1.4]\n",
            " [7.7 3.  6.1 2.3]\n",
            " [6.3 3.4 5.6 2.4]\n",
            " [6.4 3.1 5.5 1.8]\n",
            " [6.  3.  4.8 1.8]\n",
            " [6.9 3.1 5.4 2.1]\n",
            " [6.7 3.1 5.6 2.4]\n",
            " [6.9 3.1 5.1 2.3]\n",
            " [5.8 2.7 5.1 1.9]\n",
            " [6.8 3.2 5.9 2.3]\n",
            " [6.7 3.3 5.7 2.5]\n",
            " [6.7 3.  5.2 2.3]\n",
            " [6.3 2.5 5.  1.9]\n",
            " [6.5 3.  5.2 2. ]\n",
            " [6.2 3.4 5.4 2.3]\n",
            " [5.9 3.  5.1 1.8]]\n"
          ]
        }
      ],
      "source": [
        "from sklearn.datasets import load_iris\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.naive_bayes import GaussianNB\n",
        "import numpy as np\n",
        "\n",
        "iris_data = load_iris()\n",
        "print(\"Iris Data:\")\n",
        "print(iris_data.data)\n",
        "scaler = StandardScaler()\n",
        "scaler.fit(iris_data.data)\n",
        "X = scaler.transform(iris_data.data)\n",
        "Y = iris_data.target\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DIUNVCfLQWLz"
      },
      "source": [
        "Answer the following questions:\n",
        "\n",
        "_Question 5: What does 'StandardScaler' do? What other types of scalers are available? What is the advantage of scaling your inputs?_ (2 marks)\n",
        "\n",
        "**Answer: StandardScalar calculates the mean and the standard deviation of the dataset and then scales all the samples in the dataset such that it is centered around the mean and scaled to the standard deviation i.e. unit variance. The advantage of scaling inputs lies in the fact that each feature will be given equal importance and the dataset will not give priority to one feature because it has larger values. Variables that are measured at different scales do not contribute equally to the model fitting & model learned function and might end up creating a bias. Thus, to deal with this potential problem feature-wise standardized (μ=0, σ=1) is usually used prior to model fitting, hence, StandardScalar() is used.**\n",
        "\n",
        "#### 1.2.2 Creating a Naive Bayes Classifier Model\n",
        "\n",
        "Recall that there are three major types of Naive Bayes classifiers:\n",
        "\n",
        "    1. Gaussian\n",
        "    2. Multinomial\n",
        "    3. Bernoulli\n",
        "    \n",
        "_Question 6: What type of model should we use here? Why?_ (2 marks)\n",
        "\n",
        "**Answer: We should use Gaussian Naive Bayes classifiers as the sepal and petal lengths and width are continuous in nature and the other classifiers (Multinomial and Bernoulli) do not support continuous values - with Multinomial supporting features which resembles a count and bernoulli for binary values.**\n",
        "\n",
        "\n",
        "Now complete the code in the code cell below, following these specifications: (4 marks)\n",
        "\n",
        "    1. Set aside 20% of the data for testing.\n",
        "    2. Use the appropriate type of Naive Bayes Classifier, adding in whatever import statements you require here.\n",
        "    3. Print out the training and testing accuracies.\n",
        "    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:33:14.085806Z",
          "start_time": "2022-09-08T05:33:14.069808Z"
        },
        "id": "zP4OQD6yQWL0"
      },
      "outputs": [],
      "source": [
        "\"\"\"\n",
        "    Enter your code for part 1.2.2 here. \n",
        "\"\"\"\n",
        "# Split the dataset\n",
        "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.20, random_state=0) #Sets aside 20% of data for testing with random state as 0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:33:14.097808Z",
          "start_time": "2022-09-08T05:33:14.085806Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kiOmFi5PQWL0",
        "outputId": "8f3a173d-8c0e-4929-e0e6-a3b12941a3d6"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GaussianNB()"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ],
      "source": [
        "from sklearn.naive_bayes import GaussianNB\n",
        "\n",
        "#Import and fit classifer\n",
        "\n",
        "clf = GaussianNB()\n",
        "\n",
        "clf.fit(X_train, Y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:33:14.113806Z",
          "start_time": "2022-09-08T05:33:14.097808Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Mt3ARKLXQWL0",
        "outputId": "27083240-0798-4749-bb44-f9160277c25f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy for training data: 0.9500. Accuracy for testing data: 0.9667.\n"
          ]
        }
      ],
      "source": [
        "# print out results\n",
        "from sklearn import metrics\n",
        "from sklearn.metrics import accuracy_score\n",
        "Y_pred_train = clf.predict(X_train)\n",
        "Y_pred_test = clf.predict(X_test)\n",
        "train_accuracy = accuracy_score(Y_train, Y_pred_train, normalize=True)\n",
        "test_accuracy = accuracy_score(Y_test, Y_pred_test, normalize=True)\n",
        "print(\"Accuracy for training data: %3.4f. Accuracy for testing data: %3.4f.\" % (train_accuracy, test_accuracy))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "98PghgHAQWL1"
      },
      "source": [
        "#### 1.2.3 Using Pipelines ####\n",
        "\n",
        "In the Naive Bayes Jupyter Notebook included with your Lecture 4 slides, we used a _Pipeline_ object to simplify our code. Using that example as a guide, rewrite your code above to use _Pipeline_. Some things to note:\n",
        "\n",
        "    1. The code will not be exactly the same (it will be much simpler). For example we are not using a CountVectorizer nor a TfidfTransformer. So just follow the principle. Remember to put your StandardScaler into the Pipeline.\n",
        "    2. When doing 'fit' on your model, you should input the _original_ data, not the scaled one, since we are incorporating the StandardScaler as part of our Pipeline.\n",
        "\n",
        "**Hint: Section 1.3.2 below shows you how to create a Pipeline for SVM**\n",
        "\n",
        "Use the code cell below to enter your new version using Pipelines. Remember to print out your training and testing accuracies. (4 marks)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:45:02.398827Z",
          "start_time": "2022-09-08T05:45:02.381843Z"
        },
        "id": "qPBvihYAQWL1"
      },
      "outputs": [],
      "source": [
        "\"\"\"\n",
        "    Enter your code for part 1.2.3 here. \n",
        "\"\"\"\n",
        "\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "iris_clf = Pipeline([('scaler', StandardScaler()),\n",
        "                    ('clf', GaussianNB()), ])"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "X_train, X_test, Y_train, Y_test = train_test_split(iris_data.data, Y, test_size = 0.20, random_state=0) #Split 20% of data into testing data with random state as 0\n",
        "\n",
        "iris_clf.fit(X_train,Y_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RKBMunSgUC0O",
        "outputId": "0990fa1b-084f-4159-8411-f87f799c4bd3"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(steps=[('scaler', StandardScaler()), ('clf', GaussianNB())])"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn import metrics\n",
        "from sklearn.metrics import accuracy_score\n",
        "iris_clf.fit(X_train, Y_train)\n",
        "Y_pred_train = iris_clf.predict(X_train)\n",
        "Y_pred_test = iris_clf.predict(X_test)\n",
        "train_accuracy = accuracy_score(Y_train, Y_pred_train, normalize=True)\n",
        "test_accuracy = accuracy_score(Y_test, Y_pred_test, normalize=True)\n",
        "print(\"Accuracy for training data: %3.4f. Accuracy for testing data: %3.4f.\" % (train_accuracy, test_accuracy))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Mw9d6eJDUKuD",
        "outputId": "9def51e2-5119-4aeb-e616-dfbc59a1c202"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy for training data: 0.9500. Accuracy for testing data: 0.9667.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xfwKUbxAQWL1"
      },
      "source": [
        "## 1.3 Creating a Support Vector Machine Classifier ###\n",
        "\n",
        "We will now create an SVM to perform our classification. There are two major SVM classifiers provided with SciKit Learn:\n",
        "\n",
        "    1. LinearSVC: An SVM that uses a linear decision boundary to classify.\n",
        "    2. SVC: An SVM that offers a wider variety of classification boundaries: Radial Basis Function (so-called 'kernel'), sigmoid, polynomials, and of course a linear boundary.\n",
        "    \n",
        "#### 1.3.1 Creating a Linear SVM ####\n",
        "\n",
        "Using your code from 1.2.3 as a guide, create a new Pipeline to train a LinearSVC with the following parameters:\n",
        "\n",
        "    - max_iter: 100000\n",
        "    - loss: hinge\n",
        "    - penalty: l2      (Note: This is 'el-two', and not 'twelve')\n",
        "    \n",
        "Use the code cell below to implement your SVM, printing out your training and testing accuraces. Please consult the SciKit Learn documentation on what these parameters mean. (4 marks)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:45:30.006781Z",
          "start_time": "2022-09-08T05:45:29.990812Z"
        },
        "id": "7yzn4xpMQWL1"
      },
      "outputs": [],
      "source": [
        "\"\"\"\n",
        "    Enter your code for part 1.3.1 here. \n",
        "\"\"\"\n",
        "from sklearn.svm import LinearSVC\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "svc_clf = Pipeline([('scaler', StandardScaler()),\n",
        "                   ('svm', LinearSVC(loss = 'hinge', penalty = 'l2',\n",
        "                    random_state = 0, max_iter = 100000))])"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "X_train, X_test, Y_train, Y_test = train_test_split(iris_data.data, Y, test_size = 0.20, random_state=0) #Split 20% of data into testing data with random state as 0\n",
        "\n",
        "svc_clf.fit(X_train,Y_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "26bmq3EtU3PL",
        "outputId": "9b8fd2e4-9389-40a6-8f80-1e263a380d5f"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(steps=[('scaler', StandardScaler()),\n",
              "                ('svm',\n",
              "                 LinearSVC(loss='hinge', max_iter=100000, random_state=0))])"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn import metrics\n",
        "from sklearn.metrics import accuracy_score\n",
        "Y_pred_train = svc_clf.predict(X_train)\n",
        "Y_pred_test = svc_clf.predict(X_test)\n",
        "train_accuracy = accuracy_score(Y_train, Y_pred_train, normalize=True)\n",
        "test_accuracy = accuracy_score(Y_test, Y_pred_test, normalize=True)\n",
        "print(\"Accuracy for training data: %3.4f. Accuracy for testing data: %3.4f.\" % (train_accuracy, test_accuracy))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2J-qayVnVD9_",
        "outputId": "04608588-e86f-42e2-b149-6735cf095fa5"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy for training data: 0.9083. Accuracy for testing data: 0.8667.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4hPF1UVyQWL1"
      },
      "source": [
        "***\n",
        "_Question 7: Play around with the loss and penalty parameters. E.g. try an 'l1' penalty with hinge loss, or 'l1' penalty with squared hinge loss. Does 'l2' work with the squared hinge loss function? Record your training and testing accuracies below_ (2 marks)\n",
        "\n",
        "**Answer: Training Accuracy:0.9083 Testing Accuracy: 0.8667.**\n",
        "\n",
        "***\n",
        "\n",
        "#### 1.3.2 Autotuning Hyperparameters ####\n",
        "\n",
        "In Question 7 you have played around with some of the hyperparameters for LinearSVC and may have found that it gives you different accuracy results. Selecting the right hyperparameters is always a challenge, but thankfully SciKit Learn gives us a very useful tool called \"GridSearchCV\". In the example below we see how to tweak the 'C' parameter, which controls penalties applied to the SVM parametrs, to various values of between 1 and 10. GridSearchCV will then select the C value that gives us the best possible training accuracy:\n",
        "\n",
        "```\n",
        "from sklearn import svm\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "\n",
        "params = {'C':[1,10]}\n",
        "\n",
        "svm_pipe_2 = Pipeline([('scaler', StandardScaler()), \n",
        "                    ('svm', GridSearchCV(svm.LinearSVC(max_iter = 100000), params)), ])\n",
        "svm_pipe_2.fit(X_train_1, Y_train_1)\n",
        "\n",
        "Y_train_pred_1 = svm_pipe_2.predict(X_train_1)\n",
        "Y_test_pred_1 = svm_pipe_2.predict(X_test_1)\n",
        "\n",
        "print(\"SVM Train Accuracy: %3.2f\" % np.mean(Y_train_pred_1 == Y_train_1))\n",
        "print(\"SVM Test Accuracy: %3.2f\" % np.mean(Y_test_pred_1 == Y_test_1))\n",
        "```\n",
        "\n",
        "Note that the code above will not run because it's missing several variables, including X_train_1, etc. Notice that GridSearchCV is created in the Pipeline and takes svm.LinearSVC as a parameter.\n",
        "\n",
        "The \"param\" variable is a dictionary that specifies which parameters to tune (in this case just simply 'C'), and what values to use (here \\[1, 10\\] means to use between 1 and 10). You can also specify labels instead of numeric values. E.g.:\n",
        "\n",
        "```\n",
        "params = {'kernel':('linear', 'poly')}\n",
        "```\n",
        "\n",
        "GridSearchCV will try 'linear' and 'poly', specified in the tuple after 'kernel', when tuning the SVM.\n",
        "\n",
        "Use the code cell below to create a Pipeline that uses SVC (instead of LinearSVC), and applies GridSearchCV to tune the following hyperparameters:\n",
        "\n",
        "    - C: From 1 to 10 as before\n",
        "    - kernel: 'linear', 'poly', 'rbf', 'sigmoid'\n",
        "    - decision_function_shape: 'ovr', 'ovo'\n",
        "    \n",
        "***\n",
        "_Question 8: Consult the SVC documentation and write down below what each hyperparameter means. Also what is a 'decision function shape', and what is the difference between 'ovr' and 'ovo' in our decision function shape?_ (2 marks)\n",
        "\n",
        "**Answer: C is the regularization parameter it is used to prevent overfitting by penalising very large weights. Kernel specifies the transformation which should be applied on the X train, this allows us to model some non-linearities. And lastly decision funtion shape decides how classication is performed. One vs one finds every permuation of classes and calculates the probability it is either of the two classes. One versus rest (ovr) calculates the probabilty of one class we being predicted against the rest, and this is repeated for all classes**\n",
        "\n",
        "***\n",
        "\n",
        "Remember to print out the training and testing accuracies. (4 marks)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-08T05:53:00.050469Z",
          "start_time": "2022-09-08T05:52:59.926436Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Hfi87Sw8QWL2",
        "outputId": "d1c798f5-e1ab-44d1-8bff-05f402b0ef52"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "SVM Train Accuracy: 0.96\n",
            "SVM Test Accuracy: 1.00\n"
          ]
        }
      ],
      "source": [
        "\"\"\"\n",
        "    Enter your code for part 1.3.2 here. \n",
        "\"\"\"\n",
        "\n",
        "from sklearn import svm\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "\n",
        "params = {'C':[1,10],\n",
        "         'kernel':('linear', 'poly', 'rbf', 'sigmoid'),\n",
        "          'decision_function_shape': ('ovr', 'ovo')\n",
        "         }\n",
        "\n",
        "svm_pipe_2 = Pipeline([('scaler', StandardScaler()), \n",
        "                    ('svm', GridSearchCV(svm.SVC(max_iter = 100000), params))])\n",
        "\n",
        "\n",
        "X_train_1, X_test_1, Y_train_1, Y_test_1 = train_test_split(iris_data.data, Y, test_size = 0.20, random_state=0)\n",
        "\n",
        "svm_pipe_2.fit(X_train_1, Y_train_1)\n",
        "\n",
        "\n",
        "\n",
        "Y_train_pred_1 = svm_pipe_2.predict(X_train_1)\n",
        "Y_test_pred_1 = svm_pipe_2.predict(X_test_1)\n",
        "\n",
        "print(\"SVM Train Accuracy: %3.2f\" % np.mean(Y_train_pred_1 == Y_train_1))\n",
        "print(\"SVM Test Accuracy: %3.2f\" % np.mean(Y_test_pred_1 == Y_test_1))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9o0oxYpIQWL2"
      },
      "source": [
        "### 1.4 Summary ###\n",
        "\n",
        "\n",
        "***\n",
        "_Question 9: Summarize in the table given below all the training and testing accuracies you've had in the previous section.  Give your thoughts on the performance of the various classifiers, and on using GridSearchCV to search for the right hyperparameters._\n",
        "\n",
        "| Method            | Training Accuracy | Testing Accuracy |\n",
        "|:-----------------:|:-----------------:|:----------------:|\n",
        "| Linear Regression |     43.1318       |      44.6075     |\n",
        "| LR (2 var)        |        34.5563    |   42.3877        |\n",
        "| LR (3 var)        |    32.1271               |    41.0605              |\n",
        "| Naive Bayes       |      0.9500             |     0.9667             |\n",
        "| LinearSVC         |        0.9083           |       0.8667           |\n",
        "| SVC (GridSearch)  |       0.96            |     1.00             |\n",
        "\n",
        "***\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XxUOjzEoQWL2"
      },
      "source": [
        "**FOR LAB TA ONLY**\n",
        "\n",
        "Q0:        ___/ 4\n",
        "\n",
        "Q1:        ___/ 2\n",
        "\n",
        "Q2:        ___/ 2\n",
        "\n",
        "Q3:        ___/ 2\n",
        "\n",
        "Q4:        ___/ 2\n",
        "\n",
        "Q4 Code:   ___/ 4 \n",
        "\n",
        "Q5:        ___/ 2\n",
        "\n",
        "Q6:        ___/ 2\n",
        "\n",
        "Q6 Code:   ___/ 4\n",
        "\n",
        "Q6 Code 2: ___/ 4 \n",
        "\n",
        "Q7:        ___/ 2\n",
        "\n",
        "Q8:        ___/ 2\n",
        "\n",
        "Q8 Code:   ___/ 4\n",
        "\n",
        "Q9:        ___/ 4\n",
        "\n",
        "Total:     ___/ 40"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.12"
    },
    "latex_envs": {
      "LaTeX_envs_menu_present": true,
      "autoclose": false,
      "autocomplete": true,
      "bibliofile": "biblio.bib",
      "cite_by": "apalike",
      "current_citInitial": 1,
      "eqLabelWithNumbers": true,
      "eqNumInitial": 1,
      "hotkeys": {
        "equation": "Ctrl-E",
        "itemize": "Ctrl-I"
      },
      "labels_anchors": false,
      "latex_user_defs": false,
      "report_style_numbering": false,
      "user_envs_cfg": false
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": false,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": false
    },
    "vscode": {
      "interpreter": {
        "hash": "055b62fcaec9821674a26809055da6bc29fe87d96b4c426e8bdfbe57b9f21334"
      }
    },
    "colab": {
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}