GridSearch_Exercise_430.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jhomolos/data_projects/blob/main/GridSearch_Exercise_430.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Non-linear Regression Methods**\n",
        "\n",
        "The new mission now is to predict house prices using a non-linear regression method. All the steps of training the models in Random Forest, AdaBoost and Gradient Boosting and extracting the feature importance for each algorithm will be carried out. Then we will implement the GridSearch method in the Gradient Boosting Regression model to get the best performance, so that R² wil increase and MAE will decrease.\n",
        "\n",
        "A description of the features follows:\n",
        "\n",
        "**ID:** unique key for each house\n",
        "\n",
        "**MSSubClass:** The building class\n",
        "\n",
        "**LotFrontage:** Linear feet of street connected to property\n",
        "\n",
        "**LotArea:** Lot size in square feet\n",
        "\n",
        "**OverallQual:** Overall material and finish quality\n",
        "\n",
        "**OverallCond:** Overall condition rating\n",
        "\n",
        "**YearBuilt:** Original construction date\n",
        "\n",
        "**YearRemodAdd:** Remodel date\n",
        "\n",
        "**MasVnrArea:** Masonry veneer area in square feet\n",
        "\n",
        "**ExterQual:** Exterior material quality\n",
        "\n",
        "**ExterCond:** Present condition of the material on the exterior\n",
        "\n",
        "**BsmtFinSF1:** Type 1 finished square feet\n",
        "\n",
        "**BsmtFinSF2:** Type 2 finished square feet\n",
        "\n",
        "**BsmtUnfSF:** Unfinished square feet of basement area\n",
        "\n",
        "**TotalBsmtSF:** Total square feet of basement area\n",
        "\n",
        "**1stFlrSF:** First Floor square feet\n",
        "\n",
        "**2ndFlrSF:** Second floor square feet\n",
        "\n",
        "**LowQualFinSF:** Low quality finished square feet (all floors)\n",
        "\n",
        "**GrLivArea:** Above grade (ground) living area square feet\n",
        "\n",
        "**BsmtFullBath:** Basement full bathrooms\n",
        "\n",
        "**BsmtHalfBath:** Basement half bathrooms\n",
        "\n",
        "**FullBath:** Full bathrooms above grade\n",
        "\n",
        "**HalfBath:** Half baths above grade\n",
        "\n",
        "**BedroomAbvGr:** Number of bedrooms above grade\n",
        "\n",
        "**KitchenAbvGr:** Number of kitches above grade\n",
        "\n",
        "**TotRmsAbvGrd:** Total rooms above grade (does not include bathrooms)\n",
        "\n",
        "**Fireplaces:** Number of fireplaces\n",
        "\n",
        "**GarageYrBlt:** Year garage was built\n",
        "\n",
        "**GarageCars:** Size of garage in car capacity\n",
        "\n",
        "**GarageArea:** Size of garage in square feet\n",
        "\n",
        "**WoodDeckSF:** Wood deck area in square feet\n",
        "\n",
        "**OpenPorchSF:** Open porch area in square feet\n",
        "\n",
        "**EnclosedPorch:** Enclosed porch area in square feet\n",
        "\n",
        "**3SsnPorch:** Three season porch area in square feet\n",
        "\n",
        "**ScreenPorch:** Screen porch area in square feet\n",
        "\n",
        "**PoolArea:** Pool area in square feet\n",
        "\n",
        "**MiscVal:** $Value of miscellaneous feature\n",
        "\n",
        "**MoSold:** Month Sold\n",
        "\n",
        "**YrSold:** Year Sold\n",
        "\n",
        "**SalePrice:** the property's sale price in dollars. This is the target variable\n",
        "that you're trying to predict.\n"
      ],
      "metadata": {
        "id": "njTJBSf4p1Cu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#importing libraries\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.ensemble import RandomForestRegressor, AdaBoostRegressor, GradientBoostingRegressor\n",
        "from sklearn import metrics"
      ],
      "metadata": {
        "id": "UiNEHjakqP7Q"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        },
        "id": "VrqFS6bQpoeo",
        "outputId": "bc0642a3-c03b-4817-ce79-1e9ba65e99c1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   Id  MSSubClass  LotFrontage  LotArea  OverallQual  OverallCond  YearBuilt  \\\n",
              "0   1          60         65.0     8450            7            5       2003   \n",
              "1   2          20         80.0     9600            6            8       1976   \n",
              "2   3          60         68.0    11250            7            5       2001   \n",
              "3   4          70         60.0     9550            7            5       1915   \n",
              "4   5          60         84.0    14260            8            5       2000   \n",
              "\n",
              "   YearRemodAdd  MasVnrArea ExterQual  ... WoodDeckSF  OpenPorchSF  \\\n",
              "0          2003       196.0        Gd  ...          0           61   \n",
              "1          1976         0.0        TA  ...        298            0   \n",
              "2          2002       162.0        Gd  ...          0           42   \n",
              "3          1970         0.0        TA  ...          0           35   \n",
              "4          2000       350.0        Gd  ...        192           84   \n",
              "\n",
              "   EnclosedPorch  3SsnPorch  ScreenPorch  PoolArea  MiscVal  MoSold  YrSold  \\\n",
              "0              0          0            0         0        0       2    2008   \n",
              "1              0          0            0         0        0       5    2007   \n",
              "2              0          0            0         0        0       9    2008   \n",
              "3            272          0            0         0        0       2    2006   \n",
              "4              0          0            0         0        0      12    2008   \n",
              "\n",
              "   SalePrice  \n",
              "0     208500  \n",
              "1     181500  \n",
              "2     223500  \n",
              "3     140000  \n",
              "4     250000  \n",
              "\n",
              "[5 rows x 40 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-23332e2d-874b-4966-a08a-8ad2bab36fe1\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>MSSubClass</th>\n",
              "      <th>LotFrontage</th>\n",
              "      <th>LotArea</th>\n",
              "      <th>OverallQual</th>\n",
              "      <th>OverallCond</th>\n",
              "      <th>YearBuilt</th>\n",
              "      <th>YearRemodAdd</th>\n",
              "      <th>MasVnrArea</th>\n",
              "      <th>ExterQual</th>\n",
              "      <th>...</th>\n",
              "      <th>WoodDeckSF</th>\n",
              "      <th>OpenPorchSF</th>\n",
              "      <th>EnclosedPorch</th>\n",
              "      <th>3SsnPorch</th>\n",
              "      <th>ScreenPorch</th>\n",
              "      <th>PoolArea</th>\n",
              "      <th>MiscVal</th>\n",
              "      <th>MoSold</th>\n",
              "      <th>YrSold</th>\n",
              "      <th>SalePrice</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>60</td>\n",
              "      <td>65.0</td>\n",
              "      <td>8450</td>\n",
              "      <td>7</td>\n",
              "      <td>5</td>\n",
              "      <td>2003</td>\n",
              "      <td>2003</td>\n",
              "      <td>196.0</td>\n",
              "      <td>Gd</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>61</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>2008</td>\n",
              "      <td>208500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>20</td>\n",
              "      <td>80.0</td>\n",
              "      <td>9600</td>\n",
              "      <td>6</td>\n",
              "      <td>8</td>\n",
              "      <td>1976</td>\n",
              "      <td>1976</td>\n",
              "      <td>0.0</td>\n",
              "      <td>TA</td>\n",
              "      <td>...</td>\n",
              "      <td>298</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>5</td>\n",
              "      <td>2007</td>\n",
              "      <td>181500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>60</td>\n",
              "      <td>68.0</td>\n",
              "      <td>11250</td>\n",
              "      <td>7</td>\n",
              "      <td>5</td>\n",
              "      <td>2001</td>\n",
              "      <td>2002</td>\n",
              "      <td>162.0</td>\n",
              "      <td>Gd</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>42</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>9</td>\n",
              "      <td>2008</td>\n",
              "      <td>223500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>70</td>\n",
              "      <td>60.0</td>\n",
              "      <td>9550</td>\n",
              "      <td>7</td>\n",
              "      <td>5</td>\n",
              "      <td>1915</td>\n",
              "      <td>1970</td>\n",
              "      <td>0.0</td>\n",
              "      <td>TA</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>35</td>\n",
              "      <td>272</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>2006</td>\n",
              "      <td>140000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>60</td>\n",
              "      <td>84.0</td>\n",
              "      <td>14260</td>\n",
              "      <td>8</td>\n",
              "      <td>5</td>\n",
              "      <td>2000</td>\n",
              "      <td>2000</td>\n",
              "      <td>350.0</td>\n",
              "      <td>Gd</td>\n",
              "      <td>...</td>\n",
              "      <td>192</td>\n",
              "      <td>84</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>12</td>\n",
              "      <td>2008</td>\n",
              "      <td>250000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 40 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-23332e2d-874b-4966-a08a-8ad2bab36fe1')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-23332e2d-874b-4966-a08a-8ad2bab36fe1 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-23332e2d-874b-4966-a08a-8ad2bab36fe1');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-f2fb65a8-09fc-4bbb-ac72-5b1113f44caf\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-f2fb65a8-09fc-4bbb-ac72-5b1113f44caf')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-f2fb65a8-09fc-4bbb-ac72-5b1113f44caf button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "house"
            }
          },
          "metadata": {},
          "execution_count": 20
        }
      ],
      "source": [
        "#reading excel file with dataset\n",
        "house = pd.read_excel(\"house.xlsx\")\n",
        "house.head()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# now we will verify the datatypes to see if some features need to be encoded from categorical to numerical\n",
        "house.dtypes"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-AAp9zqNru9u",
        "outputId": "35eea22e-e47c-4431-9a1b-adb3f4526c90"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Id                 int64\n",
              "MSSubClass         int64\n",
              "LotFrontage      float64\n",
              "LotArea            int64\n",
              "OverallQual        int64\n",
              "OverallCond        int64\n",
              "YearBuilt          int64\n",
              "YearRemodAdd       int64\n",
              "MasVnrArea       float64\n",
              "ExterQual         object\n",
              "ExterCond         object\n",
              "BsmtFinSF1         int64\n",
              "BsmtFinSF2         int64\n",
              "BsmtUnfSF          int64\n",
              "TotalBsmtSF        int64\n",
              "1stFlrSF           int64\n",
              "2ndFlrSF           int64\n",
              "LowQualFinSF       int64\n",
              "GrLivArea          int64\n",
              "BsmtFullBath       int64\n",
              "BsmtHalfBath       int64\n",
              "FullBath           int64\n",
              "HalfBath           int64\n",
              "BedroomAbvGr       int64\n",
              "KitchenAbvGr       int64\n",
              "TotRmsAbvGrd       int64\n",
              "Fireplaces         int64\n",
              "GarageYrBlt      float64\n",
              "GarageCars         int64\n",
              "GarageArea         int64\n",
              "WoodDeckSF         int64\n",
              "OpenPorchSF        int64\n",
              "EnclosedPorch      int64\n",
              "3SsnPorch          int64\n",
              "ScreenPorch        int64\n",
              "PoolArea           int64\n",
              "MiscVal            int64\n",
              "MoSold             int64\n",
              "YrSold             int64\n",
              "SalePrice          int64\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "the info method shows that there are two categorical columns: ExterQual and ExterCond, and both cannot be used without conversion."
      ],
      "metadata": {
        "id": "Ap6YFFdK1QS1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# using Label Encoder to transform categorical features to numerical.\n",
        "categoricalColumns = ['ExterQual', 'ExterCond']\n",
        "le = LabelEncoder()\n",
        "for col in categoricalColumns:\n",
        "    house[col] = le.fit_transform(house[col])\n",
        "house.head()\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        },
        "id": "fX6bHQrXzCUv",
        "outputId": "22834eb9-dafb-4d87-d9bc-3fad98721b6b"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   Id  MSSubClass  LotFrontage  LotArea  OverallQual  OverallCond  YearBuilt  \\\n",
              "0   1          60         65.0     8450            7            5       2003   \n",
              "1   2          20         80.0     9600            6            8       1976   \n",
              "2   3          60         68.0    11250            7            5       2001   \n",
              "3   4          70         60.0     9550            7            5       1915   \n",
              "4   5          60         84.0    14260            8            5       2000   \n",
              "\n",
              "   YearRemodAdd  MasVnrArea  ExterQual  ...  WoodDeckSF  OpenPorchSF  \\\n",
              "0          2003       196.0          2  ...           0           61   \n",
              "1          1976         0.0          3  ...         298            0   \n",
              "2          2002       162.0          2  ...           0           42   \n",
              "3          1970         0.0          3  ...           0           35   \n",
              "4          2000       350.0          2  ...         192           84   \n",
              "\n",
              "   EnclosedPorch  3SsnPorch  ScreenPorch  PoolArea  MiscVal  MoSold  YrSold  \\\n",
              "0              0          0            0         0        0       2    2008   \n",
              "1              0          0            0         0        0       5    2007   \n",
              "2              0          0            0         0        0       9    2008   \n",
              "3            272          0            0         0        0       2    2006   \n",
              "4              0          0            0         0        0      12    2008   \n",
              "\n",
              "   SalePrice  \n",
              "0     208500  \n",
              "1     181500  \n",
              "2     223500  \n",
              "3     140000  \n",
              "4     250000  \n",
              "\n",
              "[5 rows x 40 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-1d620681-2cd1-485b-9e5d-926ef52b9cb2\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>MSSubClass</th>\n",
              "      <th>LotFrontage</th>\n",
              "      <th>LotArea</th>\n",
              "      <th>OverallQual</th>\n",
              "      <th>OverallCond</th>\n",
              "      <th>YearBuilt</th>\n",
              "      <th>YearRemodAdd</th>\n",
              "      <th>MasVnrArea</th>\n",
              "      <th>ExterQual</th>\n",
              "      <th>...</th>\n",
              "      <th>WoodDeckSF</th>\n",
              "      <th>OpenPorchSF</th>\n",
              "      <th>EnclosedPorch</th>\n",
              "      <th>3SsnPorch</th>\n",
              "      <th>ScreenPorch</th>\n",
              "      <th>PoolArea</th>\n",
              "      <th>MiscVal</th>\n",
              "      <th>MoSold</th>\n",
              "      <th>YrSold</th>\n",
              "      <th>SalePrice</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>60</td>\n",
              "      <td>65.0</td>\n",
              "      <td>8450</td>\n",
              "      <td>7</td>\n",
              "      <td>5</td>\n",
              "      <td>2003</td>\n",
              "      <td>2003</td>\n",
              "      <td>196.0</td>\n",
              "      <td>2</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>61</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>2008</td>\n",
              "      <td>208500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>20</td>\n",
              "      <td>80.0</td>\n",
              "      <td>9600</td>\n",
              "      <td>6</td>\n",
              "      <td>8</td>\n",
              "      <td>1976</td>\n",
              "      <td>1976</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3</td>\n",
              "      <td>...</td>\n",
              "      <td>298</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>5</td>\n",
              "      <td>2007</td>\n",
              "      <td>181500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>60</td>\n",
              "      <td>68.0</td>\n",
              "      <td>11250</td>\n",
              "      <td>7</td>\n",
              "      <td>5</td>\n",
              "      <td>2001</td>\n",
              "      <td>2002</td>\n",
              "      <td>162.0</td>\n",
              "      <td>2</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>42</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>9</td>\n",
              "      <td>2008</td>\n",
              "      <td>223500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>70</td>\n",
              "      <td>60.0</td>\n",
              "      <td>9550</td>\n",
              "      <td>7</td>\n",
              "      <td>5</td>\n",
              "      <td>1915</td>\n",
              "      <td>1970</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>35</td>\n",
              "      <td>272</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>2006</td>\n",
              "      <td>140000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>60</td>\n",
              "      <td>84.0</td>\n",
              "      <td>14260</td>\n",
              "      <td>8</td>\n",
              "      <td>5</td>\n",
              "      <td>2000</td>\n",
              "      <td>2000</td>\n",
              "      <td>350.0</td>\n",
              "      <td>2</td>\n",
              "      <td>...</td>\n",
              "      <td>192</td>\n",
              "      <td>84</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>12</td>\n",
              "      <td>2008</td>\n",
              "      <td>250000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 40 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1d620681-2cd1-485b-9e5d-926ef52b9cb2')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-1d620681-2cd1-485b-9e5d-926ef52b9cb2 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-1d620681-2cd1-485b-9e5d-926ef52b9cb2');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-155e782e-edfe-4118-b6d3-418ee6ca027d\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-155e782e-edfe-4118-b6d3-418ee6ca027d')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-155e782e-edfe-4118-b6d3-418ee6ca027d button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "house"
            }
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Correcting null values\n",
        "mean_LotFrontage = house['LotFrontage'].mean()\n",
        "house['LotFrontage'].fillna(mean_LotFrontage, inplace=True)\n",
        "\n",
        "mean_MasVnrArea = house['MasVnrArea'].mean()\n",
        "house['MasVnrArea'].fillna(mean_MasVnrArea, inplace=True)\n",
        "\n",
        "mean_GarageYrBlt = house['GarageYrBlt'].mean()\n",
        "house['GarageYrBlt'].fillna(mean_GarageYrBlt, inplace=True)\n",
        "\n",
        "house.isna().sum()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WUSZlGKQ2dpc",
        "outputId": "85f552c9-6e62-4a23-f4a9-2df8f8f15146"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Id               0\n",
              "MSSubClass       0\n",
              "LotFrontage      0\n",
              "LotArea          0\n",
              "OverallQual      0\n",
              "OverallCond      0\n",
              "YearBuilt        0\n",
              "YearRemodAdd     0\n",
              "MasVnrArea       0\n",
              "ExterQual        0\n",
              "ExterCond        0\n",
              "BsmtFinSF1       0\n",
              "BsmtFinSF2       0\n",
              "BsmtUnfSF        0\n",
              "TotalBsmtSF      0\n",
              "1stFlrSF         0\n",
              "2ndFlrSF         0\n",
              "LowQualFinSF     0\n",
              "GrLivArea        0\n",
              "BsmtFullBath     0\n",
              "BsmtHalfBath     0\n",
              "FullBath         0\n",
              "HalfBath         0\n",
              "BedroomAbvGr     0\n",
              "KitchenAbvGr     0\n",
              "TotRmsAbvGrd     0\n",
              "Fireplaces       0\n",
              "GarageYrBlt      0\n",
              "GarageCars       0\n",
              "GarageArea       0\n",
              "WoodDeckSF       0\n",
              "OpenPorchSF      0\n",
              "EnclosedPorch    0\n",
              "3SsnPorch        0\n",
              "ScreenPorch      0\n",
              "PoolArea         0\n",
              "MiscVal          0\n",
              "MoSold           0\n",
              "YrSold           0\n",
              "SalePrice        0\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        " It shows also that the columns LotFrontage, MasVnrArea and GarageYrBlt contain null values."
      ],
      "metadata": {
        "id": "CjOS0OAh63FW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#verifying duplicates\n",
        "print(\"There are {} duplicates\".format(house.duplicated().sum()))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lc8LXWV94T6V",
        "outputId": "390a1335-fa84-4e4a-832c-7c4512e9ef91"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "There are 0 duplicates\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "For modeling purposes, we will drop the column 'Id', because it doesn't contain relevant information."
      ],
      "metadata": {
        "id": "mrI9wKAL4uBg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "house_id = house['Id']\n",
        "house.drop('Id', axis=1, inplace=True)"
      ],
      "metadata": {
        "id": "mrR1LsW844P9"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Now we will split our data into test and training\n",
        "x = house.iloc[:, :-1]\n",
        "y = house.iloc[:, -1]\n",
        "x_list = list(x.columns)\n",
        "x = np.array(x)\n",
        "y = np.array(y)\n",
        "\n",
        "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)"
      ],
      "metadata": {
        "id": "zE3N0PLE7fTD"
      },
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Random Forest Regression\n",
        "rfr = RandomForestRegressor(n_estimators=100, random_state=42)\n",
        "rfr.fit(x_train, y_train)\n",
        "y_pred_rfr = rfr.predict(x_test)\n",
        "r_sq_score = rfr.score(x_train, y_train)\n",
        "print(f\"R²: {r_sq_score}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nYCIbhbs8V94",
        "outputId": "b5c9418d-fb0f-4762-f604-05177dafd642"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "R²: 0.9785355978814705\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Metrics for the Random Forest Regression\n",
        "print('MAE: {}'.format(metrics.mean_absolute_error(y_test, y_pred_rfr)))\n",
        "print('MSE: {}'.format(metrics.mean_squared_error(y_test, y_pred_rfr)))\n",
        "print('RMSE: {}'.format(np.sqrt(metrics.mean_squared_error(y_test, y_pred_rfr))))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hT307XS781vk",
        "outputId": "69d8e4b3-4851-49ca-e654-6ad2df6bd7a1"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MAE: 17966.99546803653\n",
            "MSE: 839136437.8418005\n",
            "RMSE: 28967.85179887871\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# AdaBoost\n",
        "adb = AdaBoostRegressor(n_estimators=100, random_state=42)\n",
        "adb.fit(x_train, y_train)\n",
        "y_pred_adb = adb.predict(x_test)\n",
        "adb_sq_score = adb.score(x_train, y_train)\n",
        "print(f\"R²: {adb_sq_score}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ynJLuiW5BaU8",
        "outputId": "4398fd40-c2e1-4354-c4ed-51537dba302f"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "R²: 0.8719496242910916\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Metrics for AdaBoost Regression\n",
        "print(\"MAE: {}\".format(metrics.mean_absolute_error(y_test, y_pred_adb)))\n",
        "print(\"MSE: {}\".format(metrics.mean_squared_error(y_test, y_pred_adb)))\n",
        "print(\"RMSE: {}\".format(np.sqrt(metrics.mean_squared_error(y_test, y_pred_adb))))\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FMLsfkdfBv8d",
        "outputId": "78163a83-5d77-4647-a778-e5ec1facef50"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MAE: 24665.647787939375\n",
            "MSE: 1249478513.4101741\n",
            "RMSE: 35347.96335590177\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Gradient Boosting Regression\n",
        "gbr = GradientBoostingRegressor(n_estimators = 100, random_state = 42)\n",
        "gbr.fit(x_train, y_train)\n",
        "y_pred_gbr = gbr.predict(x_test)\n",
        "gbr_sq_score = gbr.score(x_train, y_train)\n",
        "print(f\"R²: {gbr_sq_score}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bMQ5YusaCYfl",
        "outputId": "c518abd5-081e-4e2b-9144-a4473ce16710"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "R²: 0.9641021607500599\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Metrics for Gradient Boosting Regression\n",
        "print(\"MAE: {}\".format(metrics.mean_absolute_error(y_test, y_pred_gbr)))\n",
        "print(\"MSE: {}\".format(metrics.mean_squared_error(y_test, y_pred_gbr)))\n",
        "print(\"RMSE: {}\".format(np.sqrt(metrics.mean_squared_error(y_test, y_pred_gbr))))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CYjbsUasCrwG",
        "outputId": "446f95ef-423a-40ec-e108-f4ba695d1a7c"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MAE: 17547.37690312577\n",
            "MSE: 822246541.5578891\n",
            "RMSE: 28674.8416134752\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Among all regressors, Gradient Boosting delivered the best results."
      ],
      "metadata": {
        "id": "KrFFiahnC5SK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Tree Visualization**"
      ],
      "metadata": {
        "id": "-X4cu-DKFbm_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Random Forest**"
      ],
      "metadata": {
        "id": "x8M3MSEIFijF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Import tools needed for visualization\n",
        "from sklearn.tree import export_graphviz\n",
        "import pydot\n",
        "# Pull out one tree from the forest\n",
        "tree = rfr.estimators_[5]\n",
        "# Export the image to a dot file\n",
        "export_graphviz(tree, out_file = 'tree_rfr.dot', feature_names = x_list, rounded = True, precision = 1)\n",
        "# Use dot file to create a graph\n",
        "(graph, ) = pydot.graph_from_dot_file('tree_rfr.dot')\n",
        "# Write graph to a png file\n",
        "graph.write_png('tree_rfr.png')"
      ],
      "metadata": {
        "id": "qsMIc7hIFhp-"
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **AdaBoost**"
      ],
      "metadata": {
        "id": "-LxtclUmFntU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Pull out one tree from AdaBoost\n",
        "tree = adb.estimators_[5]\n",
        "# Export the image to a dot file\n",
        "export_graphviz(tree, out_file = 'tree_adb.dot', feature_names = x_list, rounded = True, precision = 1)\n",
        "# Use dot file to create a graph\n",
        "(graph, ) = pydot.graph_from_dot_file('tree_adb.dot')\n",
        "# Write graph to a png file\n",
        "graph.write_png('tree_adb.png')"
      ],
      "metadata": {
        "id": "ab1PD4gsFqM8"
      },
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Feature Importance**"
      ],
      "metadata": {
        "id": "J0RMZz4sEDl0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Get numerical feature importances\n",
        "importances = list(rfr.feature_importances_)\n",
        "# List of tuples with variable and importance\n",
        "feature_importances = [(feature, round(importance, 2)) for feature, importance in zip(x_list, importances)]\n",
        "# Sort the feature importances by most important first\n",
        "feature_importances = sorted(feature_importances, key = lambda x: x[1], reverse = True)\n",
        "# Print out the feature and importances\n",
        "[print('Variable: {:20} Importance: {}'.format(*pair)) for pair in feature_importances];"
      ],
      "metadata": {
        "id": "_OGU7r2LEG8e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ff7b8da4-cdfd-4a71-db1d-6122fbdd3528"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Variable: OverallQual          Importance: 0.56\n",
            "Variable: GrLivArea            Importance: 0.12\n",
            "Variable: TotalBsmtSF          Importance: 0.04\n",
            "Variable: 2ndFlrSF             Importance: 0.04\n",
            "Variable: BsmtFinSF1           Importance: 0.03\n",
            "Variable: 1stFlrSF             Importance: 0.03\n",
            "Variable: LotArea              Importance: 0.02\n",
            "Variable: YearBuilt            Importance: 0.02\n",
            "Variable: GarageArea           Importance: 0.02\n",
            "Variable: LotFrontage          Importance: 0.01\n",
            "Variable: OverallCond          Importance: 0.01\n",
            "Variable: YearRemodAdd         Importance: 0.01\n",
            "Variable: BsmtUnfSF            Importance: 0.01\n",
            "Variable: FullBath             Importance: 0.01\n",
            "Variable: TotRmsAbvGrd         Importance: 0.01\n",
            "Variable: GarageYrBlt          Importance: 0.01\n",
            "Variable: GarageCars           Importance: 0.01\n",
            "Variable: WoodDeckSF           Importance: 0.01\n",
            "Variable: OpenPorchSF          Importance: 0.01\n",
            "Variable: MSSubClass           Importance: 0.0\n",
            "Variable: MasVnrArea           Importance: 0.0\n",
            "Variable: ExterQual            Importance: 0.0\n",
            "Variable: ExterCond            Importance: 0.0\n",
            "Variable: BsmtFinSF2           Importance: 0.0\n",
            "Variable: LowQualFinSF         Importance: 0.0\n",
            "Variable: BsmtFullBath         Importance: 0.0\n",
            "Variable: BsmtHalfBath         Importance: 0.0\n",
            "Variable: HalfBath             Importance: 0.0\n",
            "Variable: BedroomAbvGr         Importance: 0.0\n",
            "Variable: KitchenAbvGr         Importance: 0.0\n",
            "Variable: Fireplaces           Importance: 0.0\n",
            "Variable: EnclosedPorch        Importance: 0.0\n",
            "Variable: 3SsnPorch            Importance: 0.0\n",
            "Variable: ScreenPorch          Importance: 0.0\n",
            "Variable: PoolArea             Importance: 0.0\n",
            "Variable: MiscVal              Importance: 0.0\n",
            "Variable: MoSold               Importance: 0.0\n",
            "Variable: YrSold               Importance: 0.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Get numerical feature importances\n",
        "importances = list(adb.feature_importances_)\n",
        "# List of tuples with variable and importance\n",
        "feature_importances = [(feature, round(importance, 2)) for feature, importance in zip(x_list, importances)]\n",
        "# Sort the feature importances by most important first\n",
        "feature_importances = sorted(feature_importances, key = lambda x: x[1], reverse = True)\n",
        "# Print out the feature and importances\n",
        "[print('Variable: {:20} Importance: {}'.format(*pair)) for pair in feature_importances];"
      ],
      "metadata": {
        "id": "N2HBYySOFRBk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a346c3fa-7a4e-4b82-f64d-b2df7d1df3fe"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Variable: OverallQual          Importance: 0.22\n",
            "Variable: GrLivArea            Importance: 0.17\n",
            "Variable: 2ndFlrSF             Importance: 0.14\n",
            "Variable: GarageCars           Importance: 0.08\n",
            "Variable: OpenPorchSF          Importance: 0.04\n",
            "Variable: LotFrontage          Importance: 0.03\n",
            "Variable: LotArea              Importance: 0.03\n",
            "Variable: BsmtFinSF1           Importance: 0.03\n",
            "Variable: TotalBsmtSF          Importance: 0.03\n",
            "Variable: 1stFlrSF             Importance: 0.03\n",
            "Variable: GarageYrBlt          Importance: 0.03\n",
            "Variable: ScreenPorch          Importance: 0.03\n",
            "Variable: YearBuilt            Importance: 0.02\n",
            "Variable: MoSold               Importance: 0.02\n",
            "Variable: YearRemodAdd         Importance: 0.01\n",
            "Variable: BsmtFullBath         Importance: 0.01\n",
            "Variable: FullBath             Importance: 0.01\n",
            "Variable: BedroomAbvGr         Importance: 0.01\n",
            "Variable: Fireplaces           Importance: 0.01\n",
            "Variable: WoodDeckSF           Importance: 0.01\n",
            "Variable: PoolArea             Importance: 0.01\n",
            "Variable: MSSubClass           Importance: 0.0\n",
            "Variable: OverallCond          Importance: 0.0\n",
            "Variable: MasVnrArea           Importance: 0.0\n",
            "Variable: ExterQual            Importance: 0.0\n",
            "Variable: ExterCond            Importance: 0.0\n",
            "Variable: BsmtFinSF2           Importance: 0.0\n",
            "Variable: BsmtUnfSF            Importance: 0.0\n",
            "Variable: LowQualFinSF         Importance: 0.0\n",
            "Variable: BsmtHalfBath         Importance: 0.0\n",
            "Variable: HalfBath             Importance: 0.0\n",
            "Variable: KitchenAbvGr         Importance: 0.0\n",
            "Variable: TotRmsAbvGrd         Importance: 0.0\n",
            "Variable: GarageArea           Importance: 0.0\n",
            "Variable: EnclosedPorch        Importance: 0.0\n",
            "Variable: 3SsnPorch            Importance: 0.0\n",
            "Variable: MiscVal              Importance: 0.0\n",
            "Variable: YrSold               Importance: 0.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Get numerical feature importances\n",
        "importances = list(gbr.feature_importances_)\n",
        "# List of tuples with variable and importance\n",
        "feature_importances = [(feature, round(importance, 2)) for feature, importance in zip(x_list, importances)]\n",
        "# Sort the feature importances by most important first\n",
        "feature_importances = sorted(feature_importances, key = lambda x: x[1], reverse = True)\n",
        "# Print out the feature and importances\n",
        "[print('Variable: {:20} Importance: {}'.format(*pair)) for pair in feature_importances];"
      ],
      "metadata": {
        "id": "uLWAYxCPFW40",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "230ae15a-de4a-4516-cf15-08dbbb458515"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Variable: OverallQual          Importance: 0.52\n",
            "Variable: GrLivArea            Importance: 0.14\n",
            "Variable: GarageCars           Importance: 0.05\n",
            "Variable: BsmtFinSF1           Importance: 0.04\n",
            "Variable: TotalBsmtSF          Importance: 0.04\n",
            "Variable: 1stFlrSF             Importance: 0.03\n",
            "Variable: 2ndFlrSF             Importance: 0.03\n",
            "Variable: LotArea              Importance: 0.02\n",
            "Variable: YearBuilt            Importance: 0.02\n",
            "Variable: YearRemodAdd         Importance: 0.02\n",
            "Variable: OverallCond          Importance: 0.01\n",
            "Variable: ExterQual            Importance: 0.01\n",
            "Variable: FullBath             Importance: 0.01\n",
            "Variable: Fireplaces           Importance: 0.01\n",
            "Variable: GarageYrBlt          Importance: 0.01\n",
            "Variable: MSSubClass           Importance: 0.0\n",
            "Variable: LotFrontage          Importance: 0.0\n",
            "Variable: MasVnrArea           Importance: 0.0\n",
            "Variable: ExterCond            Importance: 0.0\n",
            "Variable: BsmtFinSF2           Importance: 0.0\n",
            "Variable: BsmtUnfSF            Importance: 0.0\n",
            "Variable: LowQualFinSF         Importance: 0.0\n",
            "Variable: BsmtFullBath         Importance: 0.0\n",
            "Variable: BsmtHalfBath         Importance: 0.0\n",
            "Variable: HalfBath             Importance: 0.0\n",
            "Variable: BedroomAbvGr         Importance: 0.0\n",
            "Variable: KitchenAbvGr         Importance: 0.0\n",
            "Variable: TotRmsAbvGrd         Importance: 0.0\n",
            "Variable: GarageArea           Importance: 0.0\n",
            "Variable: WoodDeckSF           Importance: 0.0\n",
            "Variable: OpenPorchSF          Importance: 0.0\n",
            "Variable: EnclosedPorch        Importance: 0.0\n",
            "Variable: 3SsnPorch            Importance: 0.0\n",
            "Variable: ScreenPorch          Importance: 0.0\n",
            "Variable: PoolArea             Importance: 0.0\n",
            "Variable: MiscVal              Importance: 0.0\n",
            "Variable: MoSold               Importance: 0.0\n",
            "Variable: YrSold               Importance: 0.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **GridSearch on Gradient Boosting Model**"
      ],
      "metadata": {
        "id": "yccyeqcKlMic"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# we import the GridSearchCV model\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "\n",
        "# we set some hyperparameters to be tested, and among them, with GridSearch, we will find the best ones\n",
        "parameters = {'learning_rate': [0.03],\n",
        "                  'subsample'    : [0.2],\n",
        "                  'n_estimators' : [100,500,1000, 1500],\n",
        "                  'max_depth'    : [8]\n",
        "                 }\n",
        "\n",
        "# we instantiate our GridSearch model\n",
        "grid_search = GridSearchCV(gbr, parameters, scoring='r2', cv=2, n_jobs=-1)"
      ],
      "metadata": {
        "id": "w9xPM7U-l72c"
      },
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# we initiate the search for the best hyparameters\n",
        "grid_search.fit(x_train, y_train)"
      ],
      "metadata": {
        "id": "IJcRa-yBmB2S",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 141
        },
        "outputId": "3a43d10b-8eff-419c-b327-b75af7a41e0f"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(cv=2, estimator=GradientBoostingRegressor(random_state=42),\n",
              "             n_jobs=-1,\n",
              "             param_grid={'learning_rate': [0.03], 'max_depth': [8],\n",
              "                         'n_estimators': [100, 500, 1000, 1500],\n",
              "                         'subsample': [0.2]},\n",
              "             scoring='r2')"
            ],
            "text/html": [
              "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=2, estimator=GradientBoostingRegressor(random_state=42),\n",
              "             n_jobs=-1,\n",
              "             param_grid={&#x27;learning_rate&#x27;: [0.03], &#x27;max_depth&#x27;: [8],\n",
              "                         &#x27;n_estimators&#x27;: [100, 500, 1000, 1500],\n",
              "                         &#x27;subsample&#x27;: [0.2]},\n",
              "             scoring=&#x27;r2&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=2, estimator=GradientBoostingRegressor(random_state=42),\n",
              "             n_jobs=-1,\n",
              "             param_grid={&#x27;learning_rate&#x27;: [0.03], &#x27;max_depth&#x27;: [8],\n",
              "                         &#x27;n_estimators&#x27;: [100, 500, 1000, 1500],\n",
              "                         &#x27;subsample&#x27;: [0.2]},\n",
              "             scoring=&#x27;r2&#x27;)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: GradientBoostingRegressor</label><div class=\"sk-toggleable__content\"><pre>GradientBoostingRegressor(random_state=42)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GradientBoostingRegressor</label><div class=\"sk-toggleable__content\"><pre>GradientBoostingRegressor(random_state=42)</pre></div></div></div></div></div></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(\" Results from Grid Search \" )\n",
        "print(\"\\n The best estimator across ALL searched params:\\n\",grid_search.best_estimator_)\n",
        "print(\"\\n The best score across ALL searched params:\\n\",grid_search.best_score_)\n",
        "print(\"\\n The best parameters across ALL searched params:\\n\",grid_search.best_params_)\n"
      ],
      "metadata": {
        "id": "mrGnu4qZmNoZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7e0d4d0c-82e6-4714-c72a-c3ece38e16ac"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " Results from Grid Search \n",
            "\n",
            " The best estimator across ALL searched params:\n",
            " GradientBoostingRegressor(learning_rate=0.03, max_depth=8, random_state=42,\n",
            "                          subsample=0.2)\n",
            "\n",
            " The best score across ALL searched params:\n",
            " 0.8446117603338402\n",
            "\n",
            " The best parameters across ALL searched params:\n",
            " {'learning_rate': 0.03, 'max_depth': 8, 'n_estimators': 100, 'subsample': 0.2}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "best_model = grid_search.best_estimator_"
      ],
      "metadata": {
        "id": "8QkvlAP3mRWR"
      },
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# we extract the best hyperparameters for our Gradient Boosting Regression model\n",
        "best_model.get_params()"
      ],
      "metadata": {
        "id": "E_1soLC0mTKC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1f4527a6-ad28-4ca5-e563-62baaf3e5ea2"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'alpha': 0.9,\n",
              " 'ccp_alpha': 0.0,\n",
              " 'criterion': 'friedman_mse',\n",
              " 'init': None,\n",
              " 'learning_rate': 0.03,\n",
              " 'loss': 'squared_error',\n",
              " 'max_depth': 8,\n",
              " 'max_features': None,\n",
              " 'max_leaf_nodes': None,\n",
              " 'min_impurity_decrease': 0.0,\n",
              " 'min_samples_leaf': 1,\n",
              " 'min_samples_split': 2,\n",
              " 'min_weight_fraction_leaf': 0.0,\n",
              " 'n_estimators': 100,\n",
              " 'n_iter_no_change': None,\n",
              " 'random_state': 42,\n",
              " 'subsample': 0.2,\n",
              " 'tol': 0.0001,\n",
              " 'validation_fraction': 0.1,\n",
              " 'verbose': 0,\n",
              " 'warm_start': False}"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# We instanciate our new Gradient Boosting Gregressor with the best hyperparameters\n",
        "gbr_tunned = GradientBoostingRegressor(alpha = 0.9,  ccp_alpha = 0.0,\n",
        " criterion = 'friedman_mse',\n",
        " init = None,\n",
        " learning_rate = 0.03,\n",
        " loss = 'squared_error',\n",
        " max_depth = 8,\n",
        " max_features = None,\n",
        " max_leaf_nodes = None,\n",
        " min_impurity_decrease = 0.0,\n",
        " min_samples_leaf = 1,\n",
        " min_samples_split = 2,\n",
        " min_weight_fraction_leaf = 0.0,\n",
        " n_estimators = 100,\n",
        " n_iter_no_change = None,\n",
        " random_state = None,\n",
        " subsample = 0.2,\n",
        " tol = 0.0001,\n",
        " validation_fraction = 0.1,\n",
        " verbose = 0,\n",
        " warm_start = False)\n",
        "gbr_tunned.fit(x_train, y_train)\n",
        "gbr_tunned_pred = gbr_tunned.predict(x_test)"
      ],
      "metadata": {
        "id": "B_L-quTbgEMr"
      },
      "execution_count": 43,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Calculate the absolute errors\n",
        "errors_gbr = abs(gbr_tunned_pred - y_test)\n",
        "r_sq = gbr_tunned.score(x_test, y_test)\n",
        "print('Coeficiente de Determinação (R²):', r_sq)\n",
        "\n",
        "# Print out the mean absolute error (MAE)\n",
        "print('MAE:', metrics.mean_absolute_error(y_test, gbr_tunned_pred))\n",
        "# Print out the mean squared error (MSE)\n",
        "print('MSE:', metrics.mean_squared_error(y_test, gbr_tunned_pred))\n",
        "# Print out the root mean squared error (RMSE)\n",
        "print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, gbr_tunned_pred)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ie-cR-uqgio1",
        "outputId": "be768bf1-0da7-4185-d636-35dc5053593d"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Coeficiente de Determinação (R²): 0.8977059781547373\n",
            "MAE: 16840.109038744242\n",
            "MSE: 784628516.598291\n",
            "RMSE: 28011.221262170828\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "With GridSearch we were able to find the best hyperparameters for the Gradient Boosting Regressor. Hence, we could minimize our error metrics."
      ],
      "metadata": {
        "id": "jWiDDV7_hZOh"
      }
    }
  ]
}