6_Feature_Encoding.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/glopez21/Deep-Learning-Intro/blob/main/6_Feature_Encoding.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N8MXZ33KEKHG"
      },
      "source": [
        "# Encoding a Feature Vector for Keras Deep Learning\n",
        "\n",
        "Neural networks can accept many types of data. We will begin with tabular data, where there are well-defined rows and columns. This data is what you would typically see in Microsoft Excel. Neural networks require numeric input. This numeric form is called a feature vector. Each input neurons receive one feature (or column) from this vector. Each row of training data typically becomes one vector. This section will see how to encode the following tabular data into a feature vector. You can see an example of tabular data below."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd"
      ],
      "metadata": {
        "id": "Cx07ygg0EqIv"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pd.set_option('display.max_columns', 7) \n",
        "pd.set_option('display.max_rows', 5)"
      ],
      "metadata": {
        "id": "-zzSKyrDEqFD"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "id": "kd1TJhoqEKHG",
        "outputId": "5e59a598-11ab-4604-a9d5-ee18603f7581"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "        id job area   income  ...  pop_dense  retail_dense     crime  product\n",
              "0        1  vv    c  50876.0  ...   0.885827      0.492126  0.071100        b\n",
              "1        2  kd    c  60369.0  ...   0.874016      0.342520  0.400809        c\n",
              "...    ...  ..  ...      ...  ...        ...           ...       ...      ...\n",
              "1998  1999  qp    c  67949.0  ...   0.909449      0.598425  0.117803        c\n",
              "1999  2000  pe    c  61467.0  ...   0.925197      0.539370  0.451973        c\n",
              "\n",
              "[2000 rows x 14 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-1e607764-c407-4180-8457-82739ab81d9f\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>job</th>\n",
              "      <th>area</th>\n",
              "      <th>income</th>\n",
              "      <th>...</th>\n",
              "      <th>pop_dense</th>\n",
              "      <th>retail_dense</th>\n",
              "      <th>crime</th>\n",
              "      <th>product</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>vv</td>\n",
              "      <td>c</td>\n",
              "      <td>50876.0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.885827</td>\n",
              "      <td>0.492126</td>\n",
              "      <td>0.071100</td>\n",
              "      <td>b</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>kd</td>\n",
              "      <td>c</td>\n",
              "      <td>60369.0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.874016</td>\n",
              "      <td>0.342520</td>\n",
              "      <td>0.400809</td>\n",
              "      <td>c</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1998</th>\n",
              "      <td>1999</td>\n",
              "      <td>qp</td>\n",
              "      <td>c</td>\n",
              "      <td>67949.0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.909449</td>\n",
              "      <td>0.598425</td>\n",
              "      <td>0.117803</td>\n",
              "      <td>c</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1999</th>\n",
              "      <td>2000</td>\n",
              "      <td>pe</td>\n",
              "      <td>c</td>\n",
              "      <td>61467.0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.925197</td>\n",
              "      <td>0.539370</td>\n",
              "      <td>0.451973</td>\n",
              "      <td>c</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2000 rows × 14 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1e607764-c407-4180-8457-82739ab81d9f')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-1e607764-c407-4180-8457-82739ab81d9f button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-1e607764-c407-4180-8457-82739ab81d9f');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "df = pd.read_csv('https://raw.githubusercontent.com/glopez21/Deep-Learning-Intro/main/data/job-simple-dataset.csv', na_values=['NA','?'])\n",
        "\n",
        "pd.set_option('display.max_columns', 9)\n",
        "pd.set_option('display.max_rows', 5)\n",
        "\n",
        "display(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5DQ3KRuPEKHH"
      },
      "source": [
        "You can make the following observations from the above data:\n",
        "* The target column is the column that you seek to predict.  There are several candidates here.  However, we will initially use the column \"product\".  This field specifies what product someone bought.\n",
        "* There is an ID column.  You should exclude his column because it contains no information useful for prediction.\n",
        "* Many of these fields are numeric and might not require further processing.\n",
        "* The income column does have some missing values.\n",
        "* There are categorical values: job, area, and product.\n",
        "\n",
        "To begin with, we will convert the job code into dummy variables."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 441
        },
        "id": "FiVWZtIyEKHH",
        "outputId": "8c94c653-7ffa-400b-b564-1ffcf23d3910"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(2000, 33)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "      job_11  job_al  job_am  job_ax  ...  job_rn  job_sa  job_vv  job_zz\n",
              "0          0       0       0       0  ...       0       0       1       0\n",
              "1          0       0       0       0  ...       0       0       0       0\n",
              "2          0       0       0       0  ...       0       0       0       0\n",
              "3          1       0       0       0  ...       0       0       0       0\n",
              "4          0       0       0       0  ...       0       0       0       0\n",
              "...      ...     ...     ...     ...  ...     ...     ...     ...     ...\n",
              "1995       0       0       0       0  ...       0       0       1       0\n",
              "1996       0       0       0       0  ...       0       0       0       0\n",
              "1997       0       0       0       0  ...       0       0       0       0\n",
              "1998       0       0       0       0  ...       0       0       0       0\n",
              "1999       0       0       0       0  ...       0       0       0       0\n",
              "\n",
              "[2000 rows x 33 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ef7e626e-acc4-4414-ace1-c653c622a9ae\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>job_11</th>\n",
              "      <th>job_al</th>\n",
              "      <th>job_am</th>\n",
              "      <th>job_ax</th>\n",
              "      <th>...</th>\n",
              "      <th>job_rn</th>\n",
              "      <th>job_sa</th>\n",
              "      <th>job_vv</th>\n",
              "      <th>job_zz</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1995</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1996</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1997</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1998</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1999</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2000 rows × 33 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ef7e626e-acc4-4414-ace1-c653c622a9ae')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ef7e626e-acc4-4414-ace1-c653c622a9ae button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ef7e626e-acc4-4414-ace1-c653c622a9ae');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "pd.set_option('display.max_columns', 7) \n",
        "pd.set_option('display.max_rows', 5)\n",
        "\n",
        "dummies = pd.get_dummies(df['job'],prefix=\"job\")\n",
        "print(dummies.shape)\n",
        "\n",
        "pd.set_option('display.max_columns', 9)\n",
        "pd.set_option('display.max_rows', 10)\n",
        "\n",
        "display(dummies)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dpEF0GKEEKHH"
      },
      "source": [
        "Because there are 33 different job codes, there are 33 dummy variables.  We also specified a prefix because the job codes (such as \"ax\") are not that meaningful by themselves.  Something such as \"job_ax\" also tells us the origin of this field.\n",
        "\n",
        "Next, we must merge these dummies back into the main data frame.  We also drop the original \"job\" field, as the dummies now represent it. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "niXxhSjiEKHH",
        "outputId": "ca492943-7c17-424c-dc97-96f6d22b8490"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "        id area   income     aspect  ...  job_rn  job_sa  job_vv  job_zz\n",
              "0        1    c  50876.0  13.100000  ...       0       0       1       0\n",
              "1        2    c  60369.0  18.625000  ...       0       0       0       0\n",
              "2        3    c  55126.0  34.766667  ...       0       0       0       0\n",
              "3        4    c  51690.0  15.808333  ...       0       0       0       0\n",
              "4        5    d  28347.0  40.941667  ...       0       0       0       0\n",
              "...    ...  ...      ...        ...  ...     ...     ...     ...     ...\n",
              "1995  1996    c  51017.0  38.233333  ...       0       0       1       0\n",
              "1996  1997    d  26576.0  33.358333  ...       0       0       0       0\n",
              "1997  1998    d  28595.0  39.425000  ...       0       0       0       0\n",
              "1998  1999    c  67949.0   5.733333  ...       0       0       0       0\n",
              "1999  2000    c  61467.0  16.891667  ...       0       0       0       0\n",
              "\n",
              "[2000 rows x 46 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-bacc85e0-d820-4282-ba06-535e8916699d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>area</th>\n",
              "      <th>income</th>\n",
              "      <th>aspect</th>\n",
              "      <th>...</th>\n",
              "      <th>job_rn</th>\n",
              "      <th>job_sa</th>\n",
              "      <th>job_vv</th>\n",
              "      <th>job_zz</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>c</td>\n",
              "      <td>50876.0</td>\n",
              "      <td>13.100000</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>c</td>\n",
              "      <td>60369.0</td>\n",
              "      <td>18.625000</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>c</td>\n",
              "      <td>55126.0</td>\n",
              "      <td>34.766667</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>c</td>\n",
              "      <td>51690.0</td>\n",
              "      <td>15.808333</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>d</td>\n",
              "      <td>28347.0</td>\n",
              "      <td>40.941667</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1995</th>\n",
              "      <td>1996</td>\n",
              "      <td>c</td>\n",
              "      <td>51017.0</td>\n",
              "      <td>38.233333</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1996</th>\n",
              "      <td>1997</td>\n",
              "      <td>d</td>\n",
              "      <td>26576.0</td>\n",
              "      <td>33.358333</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1997</th>\n",
              "      <td>1998</td>\n",
              "      <td>d</td>\n",
              "      <td>28595.0</td>\n",
              "      <td>39.425000</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1998</th>\n",
              "      <td>1999</td>\n",
              "      <td>c</td>\n",
              "      <td>67949.0</td>\n",
              "      <td>5.733333</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1999</th>\n",
              "      <td>2000</td>\n",
              "      <td>c</td>\n",
              "      <td>61467.0</td>\n",
              "      <td>16.891667</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2000 rows × 46 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bacc85e0-d820-4282-ba06-535e8916699d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-bacc85e0-d820-4282-ba06-535e8916699d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-bacc85e0-d820-4282-ba06-535e8916699d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "pd.set_option('display.max_columns', 7) \n",
        "pd.set_option('display.max_rows', 5)\n",
        "\n",
        "df = pd.concat([df,dummies],axis=1)\n",
        "df.drop('job', axis=1, inplace=True)\n",
        "\n",
        "pd.set_option('display.max_columns', 9)\n",
        "pd.set_option('display.max_rows', 10)\n",
        "\n",
        "display(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CavlfEJQEKHH"
      },
      "source": [
        "We also introduce dummy variables for the area column."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "e7ymVu1jEKHH",
        "outputId": "374fb1dd-3e96-4ded-90e3-a5854512e0a3"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "        id   income     aspect  subscriptions  ...  area_a  area_b  area_c  \\\n",
              "0        1  50876.0  13.100000              1  ...       0       0       1   \n",
              "1        2  60369.0  18.625000              2  ...       0       0       1   \n",
              "2        3  55126.0  34.766667              1  ...       0       0       1   \n",
              "3        4  51690.0  15.808333              1  ...       0       0       1   \n",
              "4        5  28347.0  40.941667              3  ...       0       0       0   \n",
              "...    ...      ...        ...            ...  ...     ...     ...     ...   \n",
              "1995  1996  51017.0  38.233333              1  ...       0       0       1   \n",
              "1996  1997  26576.0  33.358333              2  ...       0       0       0   \n",
              "1997  1998  28595.0  39.425000              3  ...       0       0       0   \n",
              "1998  1999  67949.0   5.733333              0  ...       0       0       1   \n",
              "1999  2000  61467.0  16.891667              0  ...       0       0       1   \n",
              "\n",
              "      area_d  \n",
              "0          0  \n",
              "1          0  \n",
              "2          0  \n",
              "3          0  \n",
              "4          1  \n",
              "...      ...  \n",
              "1995       0  \n",
              "1996       1  \n",
              "1997       1  \n",
              "1998       0  \n",
              "1999       0  \n",
              "\n",
              "[2000 rows x 49 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-95cca3ce-b391-4c7b-876d-ad99536f392d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>income</th>\n",
              "      <th>aspect</th>\n",
              "      <th>subscriptions</th>\n",
              "      <th>...</th>\n",
              "      <th>area_a</th>\n",
              "      <th>area_b</th>\n",
              "      <th>area_c</th>\n",
              "      <th>area_d</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>50876.0</td>\n",
              "      <td>13.100000</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>60369.0</td>\n",
              "      <td>18.625000</td>\n",
              "      <td>2</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>55126.0</td>\n",
              "      <td>34.766667</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>51690.0</td>\n",
              "      <td>15.808333</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>28347.0</td>\n",
              "      <td>40.941667</td>\n",
              "      <td>3</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1995</th>\n",
              "      <td>1996</td>\n",
              "      <td>51017.0</td>\n",
              "      <td>38.233333</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1996</th>\n",
              "      <td>1997</td>\n",
              "      <td>26576.0</td>\n",
              "      <td>33.358333</td>\n",
              "      <td>2</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1997</th>\n",
              "      <td>1998</td>\n",
              "      <td>28595.0</td>\n",
              "      <td>39.425000</td>\n",
              "      <td>3</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1998</th>\n",
              "      <td>1999</td>\n",
              "      <td>67949.0</td>\n",
              "      <td>5.733333</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1999</th>\n",
              "      <td>2000</td>\n",
              "      <td>61467.0</td>\n",
              "      <td>16.891667</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2000 rows × 49 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-95cca3ce-b391-4c7b-876d-ad99536f392d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-95cca3ce-b391-4c7b-876d-ad99536f392d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-95cca3ce-b391-4c7b-876d-ad99536f392d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "pd.set_option('display.max_columns', 7) \n",
        "pd.set_option('display.max_rows', 5)\n",
        "\n",
        "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
        "df.drop('area', axis=1, inplace=True)\n",
        "\n",
        "pd.set_option('display.max_columns', 9)\n",
        "pd.set_option('display.max_rows', 10)\n",
        "display(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rfdYgZPEEKHI"
      },
      "source": [
        "The last remaining transformation is to fill in missing income values. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "sctpj0CaEKHI"
      },
      "outputs": [],
      "source": [
        "med = df['income'].median()\n",
        "df['income'] = df['income'].fillna(med)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nrLOs3GwEKHI"
      },
      "source": [
        "There are more advanced ways of filling in missing values, but they require more analysis. The idea would be to see if another field might hint at what the income was. For example, it might be beneficial to calculate a median income for each area or job category. This technique is something to keep in mind for the class Kaggle competition.\n",
        "\n",
        "At this point, the Pandas dataframe is ready to be converted to Numpy for neural network training. We need to know a list of the columns that will make up *x* (the predictors or inputs) and *y* (the target). \n",
        "\n",
        "The complete list of columns is:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cEBPEjmwEKHI",
        "outputId": "d57af32b-6c56-44b3-c367-0bbb6d7dd60f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['id', 'income', 'aspect', 'subscriptions', 'dist_healthy', 'save_rate', 'dist_unhealthy', 'age', 'pop_dense', 'retail_dense', 'crime', 'product', 'job_11', 'job_al', 'job_am', 'job_ax', 'job_bf', 'job_by', 'job_cv', 'job_de', 'job_dz', 'job_e2', 'job_f8', 'job_gj', 'job_gv', 'job_kd', 'job_ke', 'job_kl', 'job_kp', 'job_ks', 'job_kw', 'job_mm', 'job_nb', 'job_nn', 'job_ob', 'job_pe', 'job_po', 'job_pq', 'job_pz', 'job_qp', 'job_qw', 'job_rn', 'job_sa', 'job_vv', 'job_zz', 'area_a', 'area_b', 'area_c', 'area_d']\n"
          ]
        }
      ],
      "source": [
        "print(list(df.columns))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4nXsWjjgEKHI"
      },
      "source": [
        "This data includes both the target and predictors.  We need a list with the target removed.  We also remove **id** because it is not useful for prediction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yZwAz4ZgEKHI",
        "outputId": "a1282b03-e00f-41b8-836c-7cac6a05cf49"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['income', 'aspect', 'subscriptions', 'dist_healthy', 'save_rate', 'dist_unhealthy', 'age', 'pop_dense', 'retail_dense', 'crime', 'job_11', 'job_al', 'job_am', 'job_ax', 'job_bf', 'job_by', 'job_cv', 'job_de', 'job_dz', 'job_e2', 'job_f8', 'job_gj', 'job_gv', 'job_kd', 'job_ke', 'job_kl', 'job_kp', 'job_ks', 'job_kw', 'job_mm', 'job_nb', 'job_nn', 'job_ob', 'job_pe', 'job_po', 'job_pq', 'job_pz', 'job_qp', 'job_qw', 'job_rn', 'job_sa', 'job_vv', 'job_zz', 'area_a', 'area_b', 'area_c', 'area_d']\n"
          ]
        }
      ],
      "source": [
        "x_columns = df.columns.drop('product').drop('id')\n",
        "print(list(x_columns))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SJ2E_gA2EKHJ"
      },
      "source": [
        "## Generate X and Y for a Classification Neural Network\n",
        "\n",
        "We can now generate *x* and *y*.  Note that this is how we generate y for a classification problem.  Regression would not use dummies and would encode the numeric value of the target."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "qMVUonKYEKHJ"
      },
      "outputs": [],
      "source": [
        "# Convert to numpy - Classification\n",
        "x_columns = df.columns.drop('product').drop('id')\n",
        "x = df[x_columns].values\n",
        "dummies = pd.get_dummies(df['product']) # Classification\n",
        "products = dummies.columns\n",
        "y = dummies.values"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yCRyAzBKEKHJ"
      },
      "source": [
        "We can display the *x* and *y* matrices."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CSd8zf6GEKHJ",
        "outputId": "7c61de87-b11c-4a36-a6a4-4aa159dc2896"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[5.08760000e+04 1.31000000e+01 1.00000000e+00 ... 0.00000000e+00\n",
            "  1.00000000e+00 0.00000000e+00]\n",
            " [6.03690000e+04 1.86250000e+01 2.00000000e+00 ... 0.00000000e+00\n",
            "  1.00000000e+00 0.00000000e+00]\n",
            " [5.51260000e+04 3.47666667e+01 1.00000000e+00 ... 0.00000000e+00\n",
            "  1.00000000e+00 0.00000000e+00]\n",
            " ...\n",
            " [2.85950000e+04 3.94250000e+01 3.00000000e+00 ... 0.00000000e+00\n",
            "  0.00000000e+00 1.00000000e+00]\n",
            " [6.79490000e+04 5.73333333e+00 0.00000000e+00 ... 0.00000000e+00\n",
            "  1.00000000e+00 0.00000000e+00]\n",
            " [6.14670000e+04 1.68916667e+01 0.00000000e+00 ... 0.00000000e+00\n",
            "  1.00000000e+00 0.00000000e+00]]\n",
            "[[0 1 0 ... 0 0 0]\n",
            " [0 0 1 ... 0 0 0]\n",
            " [0 1 0 ... 0 0 0]\n",
            " ...\n",
            " [0 0 0 ... 0 1 0]\n",
            " [0 0 1 ... 0 0 0]\n",
            " [0 0 1 ... 0 0 0]]\n"
          ]
        }
      ],
      "source": [
        "print(x)\n",
        "print(y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7QptcWpaEKHJ"
      },
      "source": [
        "The x and y values are now ready for a neural network.  Make sure that you construct the neural network for a classification problem.  Specifically,\n",
        "\n",
        "* Classification neural networks have an output neuron count equal to the number of classes.\n",
        "* Classification neural networks should use **categorical_crossentropy** and a **softmax** activation function on the output layer.\n",
        "\n",
        "## Generate X and Y for a Regression Neural Network\n",
        "\n",
        "The program generates the *x* values the say way for a regression neural network.  However, *y* does not use dummies.  Make sure to replace **income** with your actual target."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "3h2HPJtwEKHJ"
      },
      "outputs": [],
      "source": [
        "y = df['income'].values"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "IRdc_4nJFIib"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "display_name": "Python 3.9 (tensorflow)",
      "language": "python",
      "name": "tensorflow"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}