aula_7_feature_engineering.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Aula 7 - Feature Engineering.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MyaSGq65woLh",
        "colab_type": "text"
      },
      "source": [
        "![Codenation](https://forum.codenation.com.br/uploads/default/original/2X/2/2d2d2a9469f0171e7df2c4ee97f70c555e431e76.png)\n",
        "\n",
        "__Autor__: Kazuki Yokoyama (kazuki.yokoyama@ufrgs.br)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mi4xZxcfBA2U",
        "colab_type": "text"
      },
      "source": [
        "# _Feature engineering_\n",
        "\n",
        "![cover](https://venturebeat.com/wp-content/uploads/2018/07/feature_engineering.jpg?resize=680%2C198&strip=all)\n",
        "\n",
        "Neste módulo, trabalharemos a engenharia de _features_, que consiste em preparar os nossos dados para alimentar os algoritmos de ML adequadamente. Ao contrário do mundo dos tutoriais, na vida real os dados dificilmente estarão prontos para serem consumidos. Grande parte do tempo de um projeto de ML é gasto com a engenharia de _features_, e quanto melhor a qualidade desta etapa, maiores são as chances de melhores resultados nas etapas seguintes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cAxxSlo3QrZV",
        "colab_type": "text"
      },
      "source": [
        "## Importação das bibliotecas"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jMxYy1NkQwW6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import functools\n",
        "from math import sqrt\n",
        "\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import statsmodels.api as sm\n",
        "import scipy.stats as sct\n",
        "import seaborn as sns\n",
        "from sklearn.datasets import load_digits, fetch_20newsgroups\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.feature_extraction.text import (\n",
        "    CountVectorizer, TfidfTransformer, TfidfVectorizer\n",
        ")\n",
        "from sklearn.impute import SimpleImputer\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.preprocessing import (\n",
        "    OneHotEncoder, Binarizer, KBinsDiscretizer,\n",
        "    MinMaxScaler, StandardScaler, PolynomialFeatures\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xNbPRHkKQyv2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Algumas configurações para o matplotlib.\n",
        "%matplotlib inline\n",
        "\n",
        "from IPython.core.pylabtools import figsize\n",
        "\n",
        "\n",
        "figsize(12, 12)\n",
        "\n",
        "sns.set()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m8onCO86Q2Hm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "np.random.seed(1000)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EIEVdatWDh3Z",
        "colab_type": "text"
      },
      "source": [
        "## _One-hot encoding_\n",
        "\n",
        "Até aqui, nós praticamente ignoramos a existência de variáveis categóricas. Focamos nas variáveis numéricas porque elas são simples de lidar e bastante comuns. Ainda assim, variáveis categóricas são encontradas facilmente e precisamos de uma forma de trabalhar com elas.\n",
        "\n",
        "Uma das formas mais simples de representação de variáveis categóricas é através do método chamado _one-hot enconding_. Com ele, uma variável categórica com $h$ categorias é transformada em $h$ novas variáveis binárias (0 ou 1), onde a presença do 1 (_hot_) significa que aquela observação pertence àquela categoria, e 0 (_cold_) que não pertence. Veja um exemplo abaixo:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M1zv6xPDk4ym",
        "colab_type": "code",
        "outputId": "b9b41a48-556d-44e1-f142-708bae7a2d02",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "rows = 100\n",
        "\n",
        "height = np.random.normal(loc=1.70, scale=0.2, size=rows).round(3)\n",
        "score = np.random.normal(loc=7, scale=1, size=rows).round(2)\n",
        "courses = [\"Math\", \"Physics\", \"Biology\"]\n",
        "course = np.random.choice(courses, size=rows)\n",
        "\n",
        "data = pd.DataFrame({\"Height\": height, \"Score\": score, \"Course\": course})\n",
        "\n",
        "data.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Height</th>\n",
              "      <th>Score</th>\n",
              "      <th>Course</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.539</td>\n",
              "      <td>6.61</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.764</td>\n",
              "      <td>6.42</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.695</td>\n",
              "      <td>8.95</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.829</td>\n",
              "      <td>5.14</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.640</td>\n",
              "      <td>6.43</td>\n",
              "      <td>Physics</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.778</td>\n",
              "      <td>7.98</td>\n",
              "      <td>Physics</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1.679</td>\n",
              "      <td>7.90</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1.604</td>\n",
              "      <td>6.76</td>\n",
              "      <td>Physics</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1.819</td>\n",
              "      <td>7.44</td>\n",
              "      <td>Physics</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1.607</td>\n",
              "      <td>6.01</td>\n",
              "      <td>Physics</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Height  Score   Course\n",
              "0   1.539   6.61  Biology\n",
              "1   1.764   6.42  Biology\n",
              "2   1.695   8.95  Biology\n",
              "3   1.829   5.14  Biology\n",
              "4   1.640   6.43  Physics\n",
              "5   1.778   7.98  Physics\n",
              "6   1.679   7.90  Biology\n",
              "7   1.604   6.76  Physics\n",
              "8   1.819   7.44  Physics\n",
              "9   1.607   6.01  Physics"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nK_6LysZP6Lw",
        "colab_type": "text"
      },
      "source": [
        "Criamos um _data set_ que contém duas variáveis numéricas (`Height` e `Score`) e uma variável categórica (`Course`). Nosso objetivo com o _one-hot encoding_ é transformar a variável `Course` em uma sequência de variáveis numéricas binárias, cada uma descrevendo uma classe da variável. Neste caso, como temos três categorias para `Course` (Biology, Physics e Math), teremos três novas variáveis binárias.\n",
        "\n",
        "Vamos treinar esse _encoder_:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MDpY6XcNmYlw",
        "colab_type": "code",
        "outputId": "5fda81c9-000d-4557-cb3f-22d012b3e548",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        }
      },
      "source": [
        "one_hot_encoder = OneHotEncoder(sparse=False, dtype=np.int)\n",
        "\n",
        "#one_hot_encoder.fit(data[[\"Course\"]])\n",
        "\n",
        "#course_encoded = one_hot_encoder.transform(...)\n",
        "\n",
        "course_encoded = one_hot_encoder.fit_transform(data[[\"Course\"]])\n",
        "\n",
        "course_encoded[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1, 0, 0],\n",
              "       [1, 0, 0],\n",
              "       [1, 0, 0],\n",
              "       [1, 0, 0],\n",
              "       [0, 0, 1],\n",
              "       [0, 0, 1],\n",
              "       [1, 0, 0],\n",
              "       [0, 0, 1],\n",
              "       [0, 0, 1],\n",
              "       [0, 0, 1]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V-O0cMCyQqk4",
        "colab_type": "text"
      },
      "source": [
        "A saída é um `np.ndarray` com formato `(n, h)`, onde `n` é o número de observações no _data set_ e `h` é o número de categorias da variável codificada."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BP_QsDI6REl_",
        "colab_type": "code",
        "outputId": "10a0faf0-b05f-4ad8-f79d-7642d15862a7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "course_encoded.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(100, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eoRT2AR8RHNl",
        "colab_type": "text"
      },
      "source": [
        "No atributo `categories_` do _encoder_, temos as categorias da variável:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ziGE3VCinqM7",
        "colab_type": "code",
        "outputId": "2c77ac8b-ba1b-4479-97aa-b59cff8b78bf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "one_hot_encoder.categories_"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[array(['Biology', 'Math', 'Physics'], dtype=object)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y8V2WMjmRUkw",
        "colab_type": "text"
      },
      "source": [
        "Podemos criar as novas colunas que descrevem cada categoria. Repare que, para qualquer linha, apenas uma das colunas contém um 1, indicando a qual categoria aquela observação pertence. Isso acontece, obviamente, se as categorias forem mutuamente exclusivas (uma observação não pode pertencer a mais de uma categoria simultaneamente)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dGepWPRFoqc0",
        "colab_type": "code",
        "outputId": "dc6a6dff-007d-4f66-cbfb-2aad4c8a7448",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "columns_encoded = one_hot_encoder.categories_[0]\n",
        "\n",
        "data_encoded = pd.concat([data, pd.DataFrame(course_encoded, columns=columns_encoded)], axis=1)\n",
        "\n",
        "data_encoded.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Height</th>\n",
              "      <th>Score</th>\n",
              "      <th>Course</th>\n",
              "      <th>Biology</th>\n",
              "      <th>Math</th>\n",
              "      <th>Physics</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.539</td>\n",
              "      <td>6.61</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.764</td>\n",
              "      <td>6.42</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.695</td>\n",
              "      <td>8.95</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.829</td>\n",
              "      <td>5.14</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.640</td>\n",
              "      <td>6.43</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.778</td>\n",
              "      <td>7.98</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1.679</td>\n",
              "      <td>7.90</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1.604</td>\n",
              "      <td>6.76</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1.819</td>\n",
              "      <td>7.44</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1.607</td>\n",
              "      <td>6.01</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Height  Score   Course  Biology  Math  Physics\n",
              "0   1.539   6.61  Biology        1     0        0\n",
              "1   1.764   6.42  Biology        1     0        0\n",
              "2   1.695   8.95  Biology        1     0        0\n",
              "3   1.829   5.14  Biology        1     0        0\n",
              "4   1.640   6.43  Physics        0     0        1\n",
              "5   1.778   7.98  Physics        0     0        1\n",
              "6   1.679   7.90  Biology        1     0        0\n",
              "7   1.604   6.76  Physics        0     0        1\n",
              "8   1.819   7.44  Physics        0     0        1\n",
              "9   1.607   6.01  Physics        0     0        1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iIiVR7P4SHXz",
        "colab_type": "text"
      },
      "source": [
        "Como você deve imaginar, a maior parte da matriz retornada é composta por zeros, sendo apenas alguns elementos compostos de um. Dizemos que essa matriz é __esparsa__. É um grande desperdício de memória trabalhar diretamente como uma matriz esparsa assim. Por isso, o _default_ do `OneHotEncoder` é retornar uma `sparse matrix` do NumPy, economizando espaço em memória:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "muGSmJckraf3",
        "colab_type": "code",
        "outputId": "c8957d2b-68c4-4722-80ea-5e241c479a88",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "one_hot_encoder_sparse = OneHotEncoder(sparse=True) # sparse=True é o default.\n",
        "\n",
        "course_encoded_sparse = one_hot_encoder_sparse.fit_transform(data[[\"Course\"]])\n",
        "\n",
        "course_encoded_sparse"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<100x3 sparse matrix of type '<class 'numpy.float64'>'\n",
              "\twith 100 stored elements in Compressed Sparse Row format>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FOYl0Lx8TPJm",
        "colab_type": "text"
      },
      "source": [
        "Para acessar os dados dessa matriz, podemos convertê-la para um _array_ não esparso:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mtUziaQmrqTN",
        "colab_type": "code",
        "outputId": "bb7920ae-69a0-4543-97da-b1fc2746ddd0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        }
      },
      "source": [
        "course_encoded_sparse.toarray()[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1., 0., 0.],\n",
              "       [1., 0., 0.],\n",
              "       [1., 0., 0.],\n",
              "       [1., 0., 0.],\n",
              "       [0., 0., 1.],\n",
              "       [0., 0., 1.],\n",
              "       [1., 0., 0.],\n",
              "       [0., 0., 1.],\n",
              "       [0., 0., 1.],\n",
              "       [0., 0., 1.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zHGmVXu1uEvM",
        "colab_type": "text"
      },
      "source": [
        "## Binarização (_Binarization_)\n",
        "\n",
        "Binarização é o processo de discretizar uma variável numérica em dois níveis com base em um _threshold_. Isso pode ser útil, por exemplo, para tornar uma variável numérica contínua em uma variável binária alvo de duas classes (positiva ou negativa).\n",
        "\n",
        "No exemplo abaixo, vamos separar a variável `Height` em dois grupos, utilizando 1.80 m como _threshold_ de separação. Observações que possuam menos de 1.80 m terão valor 0, enquanto aquelas com mais de 1.80 m terão valor 1:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PeGrPpyWPcOw",
        "colab_type": "code",
        "outputId": "edb6b4c4-97e9-4914-f952-aa60c6dbbbc2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 208
        }
      },
      "source": [
        "tall = (data_encoded.Height > 1.80)\n",
        "\n",
        "tall[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    False\n",
              "1    False\n",
              "2    False\n",
              "3     True\n",
              "4    False\n",
              "5    False\n",
              "6    False\n",
              "7    False\n",
              "8     True\n",
              "9    False\n",
              "Name: Height, dtype: bool"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "94vcsMVguGvG",
        "colab_type": "code",
        "outputId": "b2b15447-7399-4309-b18a-3de5a183a41e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        }
      },
      "source": [
        "binarizer = Binarizer(threshold=1.80).fit(data_encoded[[\"Height\"]])\n",
        "\n",
        "height_binary = binarizer.transform(data_encoded[[\"Height\"]])\n",
        "\n",
        "height_binary[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.],\n",
              "       [0.],\n",
              "       [0.],\n",
              "       [1.],\n",
              "       [0.],\n",
              "       [0.],\n",
              "       [0.],\n",
              "       [0.],\n",
              "       [1.],\n",
              "       [0.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oND_xnxRV8wZ",
        "colab_type": "text"
      },
      "source": [
        "O `Binarizer` tem como saída uma matriz binária numérica. Podemos transformá-la em um vetor de _bool_:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iXbf50-4vdDR",
        "colab_type": "code",
        "outputId": "2f7dba40-f513-491a-e072-743ac0a8c88f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "height_bool = pd.DataFrame(height_binary.flatten().astype(bool), columns=[\"Tall\"])\n",
        "\n",
        "height_bool.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Tall</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    Tall\n",
              "0  False\n",
              "1  False\n",
              "2  False\n",
              "3   True\n",
              "4  False\n",
              "5  False\n",
              "6  False\n",
              "7  False\n",
              "8   True\n",
              "9  False"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nn9Gs9DhWNvi",
        "colab_type": "text"
      },
      "source": [
        "Vamos adicionar a nova variável `Tall`, que indica se a pessoa é alta (> 1.80 m), ao nosso _data set_:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xjOV0WlJy7DY",
        "colab_type": "code",
        "outputId": "af316c4b-4931-44cb-a4af-4fa51b3c93fc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "data_encoded = pd.concat([data_encoded, height_bool], axis=1)\n",
        "\n",
        "data_encoded.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Height</th>\n",
              "      <th>Score</th>\n",
              "      <th>Course</th>\n",
              "      <th>Biology</th>\n",
              "      <th>Math</th>\n",
              "      <th>Physics</th>\n",
              "      <th>Tall</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.539</td>\n",
              "      <td>6.61</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.764</td>\n",
              "      <td>6.42</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.695</td>\n",
              "      <td>8.95</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.829</td>\n",
              "      <td>5.14</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.640</td>\n",
              "      <td>6.43</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.778</td>\n",
              "      <td>7.98</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1.679</td>\n",
              "      <td>7.90</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1.604</td>\n",
              "      <td>6.76</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1.819</td>\n",
              "      <td>7.44</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1.607</td>\n",
              "      <td>6.01</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Height  Score   Course  Biology  Math  Physics   Tall\n",
              "0   1.539   6.61  Biology        1     0        0  False\n",
              "1   1.764   6.42  Biology        1     0        0  False\n",
              "2   1.695   8.95  Biology        1     0        0  False\n",
              "3   1.829   5.14  Biology        1     0        0   True\n",
              "4   1.640   6.43  Physics        0     0        1  False\n",
              "5   1.778   7.98  Physics        0     0        1  False\n",
              "6   1.679   7.90  Biology        1     0        0  False\n",
              "7   1.604   6.76  Physics        0     0        1  False\n",
              "8   1.819   7.44  Physics        0     0        1   True\n",
              "9   1.607   6.01  Physics        0     0        1  False"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2tOdmnNi23p4",
        "colab_type": "text"
      },
      "source": [
        "## Discretização (_Binning_)\n",
        "\n",
        "Discretização, como o nome diz, é o processo de discretizar ou separar em intervalos contínuos uma variável numérica. Isso pode ser útil para converter uma variável numérica em categórica, quando o valor exato numérico não for tão importante quanto o intervalo onde ele se encontra.\n",
        "\n",
        "Podemos criar _bins_ (_buckets_ ou intervalos) que contenham aproximadamente a mesma quantidade de observações, utilizando a estratégia `quantile` ou que sejam igualmente espaçados com a estratégia `uniform`.\n",
        "\n",
        "No exemplo a seguir, criamos quatro intervalos da variável `Score` com a estratégia `quantile`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xir4K6i522ZQ",
        "colab_type": "code",
        "outputId": "e902850a-d3dc-4d97-a80f-ad3dad1bb1a2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        }
      },
      "source": [
        "discretizer = KBinsDiscretizer(n_bins=4, encode=\"ordinal\", strategy=\"quantile\")\n",
        "\n",
        "discretizer.fit(data_encoded[[\"Score\"]])\n",
        "\n",
        "score_bins = discretizer.transform(data_encoded[[\"Score\"]])\n",
        "\n",
        "score_bins[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1.],\n",
              "       [1.],\n",
              "       [3.],\n",
              "       [0.],\n",
              "       [1.],\n",
              "       [3.],\n",
              "       [3.],\n",
              "       [2.],\n",
              "       [2.],\n",
              "       [0.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3hrP6E4xYXCs",
        "colab_type": "text"
      },
      "source": [
        "Os limites dos intervalos estão disponíveis no atributo `bin_edges_`. Isso pode ser útil para criarmos _labels_ para colunas do _data set_ por exemplo:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ScCmeNtn3-fF",
        "colab_type": "code",
        "outputId": "be1003a5-2d28-42d6-e76d-bc349e957e95",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "discretizer.bin_edges_"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([array([4.09  , 6.1975, 6.735 , 7.6   , 9.28  ])], dtype=object)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vGl5ONq2Yk7r",
        "colab_type": "text"
      },
      "source": [
        "A função `get_interval()` abaixo facilita a criação de _labels_ indicativas dos intervalos:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fvB70_vd4fSO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_interval(bin_idx, bin_edges):\n",
        "  return f\"{np.round(bin_edges[bin_idx], 2):.2f} ⊢ {np.round(bin_edges[bin_idx+1], 2):.2f}\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hn3eqHFbYtfm",
        "colab_type": "text"
      },
      "source": [
        "Cada um dos intervalos mostrados abaixo deve possuir aproximadamente a mesma quantidade de observações:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HX59pepN5ZQQ",
        "colab_type": "code",
        "outputId": "d5b3d4dc-c969-44cb-fa34-e31fad2dd818",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        }
      },
      "source": [
        "bin_edges_quantile = discretizer.bin_edges_[0]\n",
        "\n",
        "print(f\"Bins quantile\")\n",
        "print(f\"interval: #elements\\n\")\n",
        "for i in range(len(discretizer.bin_edges_[0])-1):\n",
        "    print(f\"{get_interval(i, bin_edges_quantile)}: {sum(score_bins[:, 0] == i)}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Bins quantile\n",
            "interval: #elements\n",
            "\n",
            "4.09 ⊢ 6.20: 25\n",
            "6.20 ⊢ 6.74: 25\n",
            "6.74 ⊢ 7.60: 25\n",
            "7.60 ⊢ 9.28: 25\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OQ0fli3IY2G6",
        "colab_type": "text"
      },
      "source": [
        "A _Series_ abaixo mostra alguns dos intervalos para os quais as observações foram encaixadas:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SZMBYjqR5-H6",
        "colab_type": "code",
        "outputId": "cba541dc-9f9e-48d8-eb87-fa54440ca353",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 208
        }
      },
      "source": [
        "score_intervals = pd.Series(score_bins.flatten().astype(np.int)).apply(get_interval, args=(bin_edges_quantile,))\n",
        "\n",
        "score_intervals.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    6.20 ⊢ 6.74\n",
              "1    6.20 ⊢ 6.74\n",
              "2    7.60 ⊢ 9.28\n",
              "3    4.09 ⊢ 6.20\n",
              "4    6.20 ⊢ 6.74\n",
              "5    7.60 ⊢ 9.28\n",
              "6    7.60 ⊢ 9.28\n",
              "7    6.74 ⊢ 7.60\n",
              "8    6.74 ⊢ 7.60\n",
              "9    4.09 ⊢ 6.20\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6gWE7IU6Y_9q",
        "colab_type": "text"
      },
      "source": [
        "Também podemos criar uma nova variável, `Score_interval`, no nosso _data set_ com os intervalos (que agora são categorias):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fomFOQbVA8eS",
        "colab_type": "code",
        "outputId": "1f065c4f-6da4-43ad-ebb7-b58706595871",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "data_encoded = pd.concat([data_encoded, pd.DataFrame(score_intervals, columns=[\"Score_interval\"])], axis=1)\n",
        "\n",
        "data_encoded.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Height</th>\n",
              "      <th>Score</th>\n",
              "      <th>Course</th>\n",
              "      <th>Biology</th>\n",
              "      <th>Math</th>\n",
              "      <th>Physics</th>\n",
              "      <th>Tall</th>\n",
              "      <th>Score_interval</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.539</td>\n",
              "      <td>6.61</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>6.20 ⊢ 6.74</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.764</td>\n",
              "      <td>6.42</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>6.20 ⊢ 6.74</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.695</td>\n",
              "      <td>8.95</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>7.60 ⊢ 9.28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.829</td>\n",
              "      <td>5.14</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>True</td>\n",
              "      <td>4.09 ⊢ 6.20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.640</td>\n",
              "      <td>6.43</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>6.20 ⊢ 6.74</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.778</td>\n",
              "      <td>7.98</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>7.60 ⊢ 9.28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1.679</td>\n",
              "      <td>7.90</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>7.60 ⊢ 9.28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1.604</td>\n",
              "      <td>6.76</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>6.74 ⊢ 7.60</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1.819</td>\n",
              "      <td>7.44</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "      <td>6.74 ⊢ 7.60</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1.607</td>\n",
              "      <td>6.01</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>4.09 ⊢ 6.20</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Height  Score   Course  Biology  Math  Physics   Tall Score_interval\n",
              "0   1.539   6.61  Biology        1     0        0  False    6.20 ⊢ 6.74\n",
              "1   1.764   6.42  Biology        1     0        0  False    6.20 ⊢ 6.74\n",
              "2   1.695   8.95  Biology        1     0        0  False    7.60 ⊢ 9.28\n",
              "3   1.829   5.14  Biology        1     0        0   True    4.09 ⊢ 6.20\n",
              "4   1.640   6.43  Physics        0     0        1  False    6.20 ⊢ 6.74\n",
              "5   1.778   7.98  Physics        0     0        1  False    7.60 ⊢ 9.28\n",
              "6   1.679   7.90  Biology        1     0        0  False    7.60 ⊢ 9.28\n",
              "7   1.604   6.76  Physics        0     0        1  False    6.74 ⊢ 7.60\n",
              "8   1.819   7.44  Physics        0     0        1   True    6.74 ⊢ 7.60\n",
              "9   1.607   6.01  Physics        0     0        1  False    4.09 ⊢ 6.20"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LldlZ92lZN1k",
        "colab_type": "text"
      },
      "source": [
        "Como dito, podemos utilizar a estratégia `uniform` para criar _bins_ igualmente espaçados, independente do número de observações que cada um possui. Também podemos especificar o tipo de codificação utilizada. No caso a seguir, utilizamos `encode=onehot-dense` para informar que queremos que a saída seja codificada como o _one-hot encode_ visto anteriormente:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E6L1qXuW-v-n",
        "colab_type": "code",
        "outputId": "956f9e9f-67ba-436f-f457-889ee2d1f3db",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        }
      },
      "source": [
        "discretizer_uniform = KBinsDiscretizer(n_bins=4, encode=\"onehot-dense\", strategy=\"uniform\")\n",
        "\n",
        "discretizer_uniform.fit(data_encoded[[\"Score\"]])\n",
        "\n",
        "score_bins_uniform = discretizer_uniform.transform(data_encoded[[\"Score\"]]).astype(np.int)\n",
        "\n",
        "score_bins_uniform[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 1, 0, 0],\n",
              "       [0, 1, 0, 0],\n",
              "       [0, 0, 0, 1],\n",
              "       [1, 0, 0, 0],\n",
              "       [0, 1, 0, 0],\n",
              "       [0, 0, 1, 0],\n",
              "       [0, 0, 1, 0],\n",
              "       [0, 0, 1, 0],\n",
              "       [0, 0, 1, 0],\n",
              "       [0, 1, 0, 0]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YapI8RuMZZfM",
        "colab_type": "text"
      },
      "source": [
        "Note como agora os intervalos são ligeiramente diferentes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P8gW9k-w-_CC",
        "colab_type": "code",
        "outputId": "731fca86-f052-4a93-e5bf-e13eec18ac8b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "bin_edges_uniform = discretizer_uniform.bin_edges_[0]\n",
        "\n",
        "bin_edges_uniform"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([4.09  , 5.3875, 6.685 , 7.9825, 9.28  ])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ieyy46EJAnb6",
        "colab_type": "code",
        "outputId": "99835fa9-8003-4060-afae-2c4de66685ff",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        }
      },
      "source": [
        "score_intervals_columns = [get_interval(i, bin_edges_uniform) for i in range(4)]\n",
        "\n",
        "print(f\"Bins uniform\")\n",
        "print(f\"interval: #elements\\n\")\n",
        "for i in range(len(discretizer_uniform.bin_edges_[0])-1):\n",
        "    print(f\"{get_interval(i, bin_edges_uniform)}: {sum(score_bins_uniform[:, i])}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Bins uniform\n",
            "interval: #elements\n",
            "\n",
            "4.09 ⊢ 5.39: 6\n",
            "5.39 ⊢ 6.68: 43\n",
            "6.68 ⊢ 7.98: 44\n",
            "7.98 ⊢ 9.28: 7\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WuWi-1U4Zzf_",
        "colab_type": "text"
      },
      "source": [
        "Podemos adicionar as novas variáveis binárias no _data set_:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P-v3UgiQB87S",
        "colab_type": "code",
        "outputId": "ad22d68f-c0e8-4a91-8838-842e7e2f5041",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "data_encoded = pd.concat([data_encoded, pd.DataFrame(score_bins_uniform, columns=score_intervals_columns)], axis=1)\n",
        "\n",
        "data_encoded.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Height</th>\n",
              "      <th>Score</th>\n",
              "      <th>Course</th>\n",
              "      <th>Biology</th>\n",
              "      <th>Math</th>\n",
              "      <th>Physics</th>\n",
              "      <th>Tall</th>\n",
              "      <th>Score_interval</th>\n",
              "      <th>4.09 ⊢ 5.39</th>\n",
              "      <th>5.39 ⊢ 6.68</th>\n",
              "      <th>6.68 ⊢ 7.98</th>\n",
              "      <th>7.98 ⊢ 9.28</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.539</td>\n",
              "      <td>6.61</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>6.20 ⊢ 6.74</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.764</td>\n",
              "      <td>6.42</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>6.20 ⊢ 6.74</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.695</td>\n",
              "      <td>8.95</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>7.60 ⊢ 9.28</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.829</td>\n",
              "      <td>5.14</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>True</td>\n",
              "      <td>4.09 ⊢ 6.20</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.640</td>\n",
              "      <td>6.43</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>6.20 ⊢ 6.74</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.778</td>\n",
              "      <td>7.98</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>7.60 ⊢ 9.28</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1.679</td>\n",
              "      <td>7.90</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>7.60 ⊢ 9.28</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1.604</td>\n",
              "      <td>6.76</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>6.74 ⊢ 7.60</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1.819</td>\n",
              "      <td>7.44</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "      <td>6.74 ⊢ 7.60</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1.607</td>\n",
              "      <td>6.01</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>4.09 ⊢ 6.20</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Height  Score   Course  ...  5.39 ⊢ 6.68  6.68 ⊢ 7.98  7.98 ⊢ 9.28\n",
              "0   1.539   6.61  Biology  ...            1            0            0\n",
              "1   1.764   6.42  Biology  ...            1            0            0\n",
              "2   1.695   8.95  Biology  ...            0            0            1\n",
              "3   1.829   5.14  Biology  ...            0            0            0\n",
              "4   1.640   6.43  Physics  ...            1            0            0\n",
              "5   1.778   7.98  Physics  ...            0            1            0\n",
              "6   1.679   7.90  Biology  ...            0            1            0\n",
              "7   1.604   6.76  Physics  ...            0            1            0\n",
              "8   1.819   7.44  Physics  ...            0            1            0\n",
              "9   1.607   6.01  Physics  ...            1            0            0\n",
              "\n",
              "[10 rows x 12 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jD8WM_-yzqSc",
        "colab_type": "text"
      },
      "source": [
        "## Normalização (_Scaling_)\n",
        "\n",
        "Normalização é o processo de colocar uma variável numérica em uma escala pré-determinada, geralmente $[0, 1]$, mas também é comum ser $[-1, 1]$.\n",
        "\n",
        "Para colocar no intervalo $[0, 1]$, basta subtrair cada valor da valor mínimo e dividir pela diferença do valor máximo e mínimo:\n",
        "\n",
        "$$x_{\\text{scaled}} = \\frac{x - x_{\\text{min}}}{x_{\\text{max}} - x_{\\text{min}}}$$\n",
        "\n",
        "Abaixo, escalamos a variável `Score` no intervalo $[0, 1]$:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nMM2mu-Qzwnv",
        "colab_type": "code",
        "outputId": "5c60c83b-13bf-431d-e77e-a2fb2e8af317",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        }
      },
      "source": [
        "minmax_scaler = MinMaxScaler(feature_range=(0, 1)) # Default feature_scale é (0, 1).\n",
        "\n",
        "minmax_scaler.fit(data_encoded[[\"Score\"]])\n",
        "\n",
        "score_normalized = minmax_scaler.transform(data_encoded[[\"Score\"]])\n",
        "\n",
        "score_normalized[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.48554913],\n",
              "       [0.44894027],\n",
              "       [0.93641618],\n",
              "       [0.20231214],\n",
              "       [0.45086705],\n",
              "       [0.7495183 ],\n",
              "       [0.73410405],\n",
              "       [0.51445087],\n",
              "       [0.64547206],\n",
              "       [0.3699422 ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FPr-37M2UBj4",
        "colab_type": "code",
        "outputId": "dc170301-56af-4cab-da7c-307c5cbb94a6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "score_normalized.min(), score_normalized.max()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.0, 0.9999999999999999)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Et6m_2Bbbq-n",
        "colab_type": "text"
      },
      "source": [
        "Adicionamos a variável `Score` normalizada ao nosso _data set_:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kaYvCQtK0fzi",
        "colab_type": "code",
        "outputId": "9f8ccb6c-d0b7-4445-96c9-490f284f2357",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "data_encoded = pd.concat([data_encoded, pd.DataFrame(score_normalized.flatten(), columns=[\"Score_normalized\"])], axis=1)\n",
        "\n",
        "data_encoded.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Height</th>\n",
              "      <th>Score</th>\n",
              "      <th>Course</th>\n",
              "      <th>Biology</th>\n",
              "      <th>Math</th>\n",
              "      <th>Physics</th>\n",
              "      <th>Tall</th>\n",
              "      <th>Score_interval</th>\n",
              "      <th>4.09 ⊢ 5.39</th>\n",
              "      <th>5.39 ⊢ 6.68</th>\n",
              "      <th>6.68 ⊢ 7.98</th>\n",
              "      <th>7.98 ⊢ 9.28</th>\n",
              "      <th>Score_normalized</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.539</td>\n",
              "      <td>6.61</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>6.20 ⊢ 6.74</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.485549</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.764</td>\n",
              "      <td>6.42</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>6.20 ⊢ 6.74</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.448940</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.695</td>\n",
              "      <td>8.95</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>7.60 ⊢ 9.28</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.936416</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.829</td>\n",
              "      <td>5.14</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>True</td>\n",
              "      <td>4.09 ⊢ 6.20</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.202312</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.640</td>\n",
              "      <td>6.43</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>6.20 ⊢ 6.74</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.450867</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.778</td>\n",
              "      <td>7.98</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>7.60 ⊢ 9.28</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0.749518</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1.679</td>\n",
              "      <td>7.90</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>7.60 ⊢ 9.28</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0.734104</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1.604</td>\n",
              "      <td>6.76</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>6.74 ⊢ 7.60</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0.514451</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1.819</td>\n",
              "      <td>7.44</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "      <td>6.74 ⊢ 7.60</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0.645472</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1.607</td>\n",
              "      <td>6.01</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>4.09 ⊢ 6.20</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.369942</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Height  Score   Course  ...  6.68 ⊢ 7.98  7.98 ⊢ 9.28  Score_normalized\n",
              "0   1.539   6.61  Biology  ...            0            0          0.485549\n",
              "1   1.764   6.42  Biology  ...            0            0          0.448940\n",
              "2   1.695   8.95  Biology  ...            0            1          0.936416\n",
              "3   1.829   5.14  Biology  ...            0            0          0.202312\n",
              "4   1.640   6.43  Physics  ...            0            0          0.450867\n",
              "5   1.778   7.98  Physics  ...            1            0          0.749518\n",
              "6   1.679   7.90  Biology  ...            1            0          0.734104\n",
              "7   1.604   6.76  Physics  ...            1            0          0.514451\n",
              "8   1.819   7.44  Physics  ...            1            0          0.645472\n",
              "9   1.607   6.01  Physics  ...            0            0          0.369942\n",
              "\n",
              "[10 rows x 13 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n7-msElsbveR",
        "colab_type": "text"
      },
      "source": [
        "Para avaliar se os valores encontrados conferem, podemos utilizar a função `normalize` abaixo:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EAfUGaFc061d",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def normalize(x, xmin, xmax):\n",
        "  return (x - xmin)/(xmax - xmin)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CXywxNX-b-0K",
        "colab_type": "text"
      },
      "source": [
        "A função `partial()` do módulo `functools` (_builtin_ do Python) permite \"congelar\" alguns parâmetros da função passaga como argumento, facilitando a invocação desta função quando tais parâmetros são constantes. No caso abaixo, \"congelamos\" os argumentos `xmin` e `xmax` da função `normalize()` com os valores mínimo e máximo da variável `Score`, respectivamente. Nas invocações subsequentes de `normalize` não precisaremos passar esses argumentos, somente o argumento \"não congelado\" `x`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UAlpigp21OVx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "normalize_score = functools.partial(normalize,\n",
        "                                    xmin=data_encoded.Score.min(),\n",
        "                                    xmax=data_encoded.Score.max())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nhR0rwUIctTa",
        "colab_type": "text"
      },
      "source": [
        "O valor abaixo realmente confere com aquele encontrado pelo `MinMaxScaler`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pMfk3jrU1mQV",
        "colab_type": "code",
        "outputId": "f9851c0d-9446-4f10-874e-cdba22b43722",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "normalize_score(data_encoded.Score[0]).round(6)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.485549"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HEcSQzWJ2Yum",
        "colab_type": "text"
      },
      "source": [
        "## Padronização (_Standardization_)\n",
        "\n",
        "Padronização é o processo de tornar a variável com média zero e variância um. Esse processo não deve ser confundido com a normalização descrita acima.\n",
        "\n",
        "O processo é simples, basta subtrair a média dos dados de cada observação e dividi-los pelo desvio-padrão:\n",
        "\n",
        "$$x_{\\text{standardized}} = \\frac{x - \\bar{x}}{s}$$\n",
        "\n",
        "onde $\\bar{x}$ indica a média amostral e $s$ o desvio-padrão amostral."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kXYXezCNdYue",
        "colab_type": "text"
      },
      "source": [
        "No exemplo abaixo, padronizamos a variável `Score`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qfhs3Eaq2dGV",
        "colab_type": "code",
        "outputId": "572aae65-5460-44d1-8134-dbc26f82e2d2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        }
      },
      "source": [
        "standard_scaler = StandardScaler()\n",
        "\n",
        "standard_scaler.fit(data_encoded[[\"Score\"]])\n",
        "\n",
        "score_standardized = standard_scaler.transform(data_encoded[[\"Score\"]])\n",
        "\n",
        "score_standardized[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[-0.20752554],\n",
              "       [-0.40839081],\n",
              "       [ 2.26628886],\n",
              "       [-1.76158843],\n",
              "       [-0.39781896],\n",
              "       [ 1.24081879],\n",
              "       [ 1.15624393],\n",
              "       [-0.0489477 ],\n",
              "       [ 0.66993854],\n",
              "       [-0.84183693]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SJJucIQddgME",
        "colab_type": "text"
      },
      "source": [
        "E adicionamos a variável padronizada ao nosso _data set_:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BAndWLe13RSr",
        "colab_type": "code",
        "outputId": "4a6231c1-f459-4307-ad14-24c4e46760cd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "data_encoded = pd.concat([data_encoded, pd.DataFrame(score_standardized.flatten(), columns=[\"Score_standardized\"])], axis=1)\n",
        "\n",
        "data_encoded.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Height</th>\n",
              "      <th>Score</th>\n",
              "      <th>Course</th>\n",
              "      <th>Biology</th>\n",
              "      <th>Math</th>\n",
              "      <th>Physics</th>\n",
              "      <th>Tall</th>\n",
              "      <th>Score_interval</th>\n",
              "      <th>4.09 ⊢ 5.39</th>\n",
              "      <th>5.39 ⊢ 6.68</th>\n",
              "      <th>6.68 ⊢ 7.98</th>\n",
              "      <th>7.98 ⊢ 9.28</th>\n",
              "      <th>Score_normalized</th>\n",
              "      <th>Score_standardized</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.539</td>\n",
              "      <td>6.61</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>6.20 ⊢ 6.74</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.485549</td>\n",
              "      <td>-0.207526</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.764</td>\n",
              "      <td>6.42</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>6.20 ⊢ 6.74</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.448940</td>\n",
              "      <td>-0.408391</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.695</td>\n",
              "      <td>8.95</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>7.60 ⊢ 9.28</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.936416</td>\n",
              "      <td>2.266289</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.829</td>\n",
              "      <td>5.14</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>True</td>\n",
              "      <td>4.09 ⊢ 6.20</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.202312</td>\n",
              "      <td>-1.761588</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.640</td>\n",
              "      <td>6.43</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>6.20 ⊢ 6.74</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.450867</td>\n",
              "      <td>-0.397819</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.778</td>\n",
              "      <td>7.98</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>7.60 ⊢ 9.28</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0.749518</td>\n",
              "      <td>1.240819</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1.679</td>\n",
              "      <td>7.90</td>\n",
              "      <td>Biology</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "      <td>7.60 ⊢ 9.28</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0.734104</td>\n",
              "      <td>1.156244</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1.604</td>\n",
              "      <td>6.76</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>6.74 ⊢ 7.60</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0.514451</td>\n",
              "      <td>-0.048948</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1.819</td>\n",
              "      <td>7.44</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "      <td>6.74 ⊢ 7.60</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0.645472</td>\n",
              "      <td>0.669939</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1.607</td>\n",
              "      <td>6.01</td>\n",
              "      <td>Physics</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>4.09 ⊢ 6.20</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.369942</td>\n",
              "      <td>-0.841837</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Height  Score   Course  ...  7.98 ⊢ 9.28  Score_normalized  Score_standardized\n",
              "0   1.539   6.61  Biology  ...            0          0.485549           -0.207526\n",
              "1   1.764   6.42  Biology  ...            0          0.448940           -0.408391\n",
              "2   1.695   8.95  Biology  ...            1          0.936416            2.266289\n",
              "3   1.829   5.14  Biology  ...            0          0.202312           -1.761588\n",
              "4   1.640   6.43  Physics  ...            0          0.450867           -0.397819\n",
              "5   1.778   7.98  Physics  ...            0          0.749518            1.240819\n",
              "6   1.679   7.90  Biology  ...            0          0.734104            1.156244\n",
              "7   1.604   6.76  Physics  ...            0          0.514451           -0.048948\n",
              "8   1.819   7.44  Physics  ...            0          0.645472            0.669939\n",
              "9   1.607   6.01  Physics  ...            0          0.369942           -0.841837\n",
              "\n",
              "[10 rows x 14 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_SgwGLgOdk5Q",
        "colab_type": "text"
      },
      "source": [
        "Note que, ao contrário da variável normalizada, é possível ter valores negativos e positivos, menores e maiores que um. Isso é bem óbvio, pois os dados agora têm média 0 e variância 1:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I0E9fwo93h9w",
        "colab_type": "code",
        "outputId": "2d9d5cdf-181b-4ca1-bea7-b382bf738ebd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "data_encoded.Score_standardized.mean(), data_encoded.Score_standardized.var()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(-1.2501111257279262e-15, 1.0101010101010102)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Av0cwG_Qd3Ow",
        "colab_type": "text"
      },
      "source": [
        "Novamente, para avaliar os resultados obtidos, podemos escrever nossa própria função de padronização:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "khwEkoks3-cS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def standardize(x, xmean, xstd):\n",
        "  return (x - xmean)/xstd"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "14w3018J4Gwy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "standardize_score = functools.partial(standardize,\n",
        "                                      xmean=data_encoded.Score.mean(),\n",
        "                                      xstd=data_encoded.Score.std())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UAGxoUK5d-22",
        "colab_type": "text"
      },
      "source": [
        "Como esperado, o valor confere com o encontrado:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dpaNVzOy4aCL",
        "colab_type": "code",
        "outputId": "fa0f42f0-32a5-48f4-f8d7-724350cdca86",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "standardize_score(data_encoded.Score[0])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "-0.20648530634442175"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2tO4OOJK7NY1",
        "colab_type": "text"
      },
      "source": [
        "## Criando um _Pipeline_\n",
        "\n",
        "Todo esse processo de transformar os dados pode ser bastante trabalhoso e entendiante. Para facilitar as coisas, o sklearn dispõe de um mecanismo de _pipeline_ que funciona como ao esteira de uma linha de montagem. Cada etapa desse _pipeline_ é uma transformação nos dados, de forma que, ao final do _pipeline_, temos os dados totalmente transformados. A vantagem é que agora especificamos todas as etapas, ou transformações, de uma só vez, e podemos reaproveitar esse _pipeline_ no futuro."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T1LyaI0-B2hV",
        "colab_type": "code",
        "outputId": "011176a0-ec92-4122-9fc4-3b3d0a3118c9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "data.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Height</th>\n",
              "      <th>Score</th>\n",
              "      <th>Course</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.539</td>\n",
              "      <td>6.61</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.764</td>\n",
              "      <td>6.42</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.695</td>\n",
              "      <td>8.95</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.829</td>\n",
              "      <td>5.14</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.640</td>\n",
              "      <td>6.43</td>\n",
              "      <td>Physics</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.778</td>\n",
              "      <td>7.98</td>\n",
              "      <td>Physics</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1.679</td>\n",
              "      <td>7.90</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1.604</td>\n",
              "      <td>6.76</td>\n",
              "      <td>Physics</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1.819</td>\n",
              "      <td>7.44</td>\n",
              "      <td>Physics</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1.607</td>\n",
              "      <td>6.01</td>\n",
              "      <td>Physics</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Height  Score   Course\n",
              "0   1.539   6.61  Biology\n",
              "1   1.764   6.42  Biology\n",
              "2   1.695   8.95  Biology\n",
              "3   1.829   5.14  Biology\n",
              "4   1.640   6.43  Physics\n",
              "5   1.778   7.98  Physics\n",
              "6   1.679   7.90  Biology\n",
              "7   1.604   6.76  Physics\n",
              "8   1.819   7.44  Physics\n",
              "9   1.607   6.01  Physics"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "86on9pLMeidf",
        "colab_type": "text"
      },
      "source": [
        "Para evitar bagunçar com nosso _data set_ original, criamos uma cópia (rasa) dele:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vdA8euCcZeq1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data_missing = data.copy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "snDUyWqEenh8",
        "colab_type": "text"
      },
      "source": [
        "E para tornar o exemplo mais interessante, adicionamos (ou removemos?) dados faltantes ao _data set_. Isso porque uma das transformações úteis que podemos aplicar no _pipeline_ é justamente a imputação de dados, ou seja, preencher dados faltantes.\n",
        "\n",
        "As variáveis numéricas faltantes são representadas por `np.nan`, enquanto a variável categórica é representada pela classe `Unknown`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LkVnbFAKS_fF",
        "colab_type": "code",
        "outputId": "6ba74eb6-0d60-419a-c39a-dd165cd49b60",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "unknown_height_idx = pd.Index(np.random.choice(data_missing.index, 10, replace=False))\n",
        "unknown_score_idx = pd.Index(np.random.choice(data_missing.index, 10, replace=False))\n",
        "unknown_course_idx = pd.Index(np.random.choice(data_missing.index, 10, replace=False))\n",
        "\n",
        "data_missing.loc[unknown_height_idx, \"Height\"] = np.nan\n",
        "data_missing.loc[unknown_score_idx, \"Score\"] = np.nan\n",
        "data_missing.loc[unknown_course_idx, \"Course\"] = \"Unknown\"\n",
        "\n",
        "data_missing_idx = unknown_height_idx | unknown_score_idx | unknown_course_idx\n",
        "\n",
        "data_missing.loc[data_missing_idx].head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Height</th>\n",
              "      <th>Score</th>\n",
              "      <th>Course</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.695</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Unknown</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.778</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Physics</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>NaN</td>\n",
              "      <td>7.44</td>\n",
              "      <td>Physics</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>1.539</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>NaN</td>\n",
              "      <td>5.44</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>NaN</td>\n",
              "      <td>8.08</td>\n",
              "      <td>Biology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>2.020</td>\n",
              "      <td>6.83</td>\n",
              "      <td>Unknown</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>1.691</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Math</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>2.085</td>\n",
              "      <td>6.96</td>\n",
              "      <td>Unknown</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>1.376</td>\n",
              "      <td>6.54</td>\n",
              "      <td>Unknown</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    Height  Score   Course\n",
              "2    1.695    NaN  Unknown\n",
              "5    1.778    NaN  Physics\n",
              "8      NaN   7.44  Physics\n",
              "11   1.539    NaN  Biology\n",
              "15     NaN   5.44  Biology\n",
              "24     NaN   8.08  Biology\n",
              "29   2.020   6.83  Unknown\n",
              "33   1.691    NaN     Math\n",
              "35   2.085   6.96  Unknown\n",
              "38   1.376   6.54  Unknown"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nmUJS9SzfC9Y",
        "colab_type": "text"
      },
      "source": [
        "Criamos o _pipeline_ com as seguintes etapas:\n",
        "\n",
        "1. Faça imputação dos dados, preenchendo os dados faltantes com a mediana dos dados presentes.\n",
        "2. Faça a normalização dos dados no intervalo _default_ $[0, 1]$.\n",
        "3. Crie novas variáveis através da expansão polinomial da variável original."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ypslSlEhGBr",
        "colab_type": "text"
      },
      "source": [
        "O `Pipeline` recebe uma lista de transformações representadas por tuplas de dois elementos. Cada tupla contém:\n",
        "\n",
        "* O nome para a etapa (ou transformação ou estimador). Isso vai ser útil para recuperar algumas informações do _pipeline_ mais a frente.\n",
        "* Um objeto da classe do transformador ou estimador, já com seus parâmetros configurados."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XqthBhA18ITd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "num_pipeline = Pipeline(steps=[\n",
        "    (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
        "    (\"minmax_scaler\", MinMaxScaler()),\n",
        "    (\"poly_features\", PolynomialFeatures(degree=2, include_bias=False))\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3UVr1XWCfZID",
        "colab_type": "text"
      },
      "source": [
        "Depois da especificação do nosso _pipeline_, podemos aplicá-lo simultaneamente a diversas variáveis (desde que as transformações especificadas façam sentido).\n",
        "\n",
        "No exemplo abaixo, aplicamos esse _pipeline_ às variáveis `Height` e `Score` ao mesmo tempo:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qh8kbymmDZqB",
        "colab_type": "code",
        "outputId": "0595019a-1288-4ea8-d18b-1d61dc44136b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        }
      },
      "source": [
        "pipeline_transformation = num_pipeline.fit_transform(data_missing[[\"Height\", \"Score\"]])\n",
        "\n",
        "pipeline_transformation[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.26553106, 0.48554913, 0.07050674, 0.12892838, 0.23575796],\n",
              "       [0.49098196, 0.44894027, 0.24106329, 0.22042158, 0.20154737],\n",
              "       [0.42184369, 0.48843931, 0.1779521 , 0.20604504, 0.23857296],\n",
              "       [0.55611222, 0.20231214, 0.30926081, 0.11250825, 0.0409302 ],\n",
              "       [0.36673347, 0.45086705, 0.13449344, 0.16534804, 0.2032811 ],\n",
              "       [0.50501002, 0.48843931, 0.25503512, 0.24666674, 0.23857296],\n",
              "       [0.40581162, 0.73410405, 0.16468307, 0.29790795, 0.53890875],\n",
              "       [0.33066132, 0.51445087, 0.10933691, 0.170109  , 0.26465969],\n",
              "       [0.41082164, 0.64547206, 0.16877442, 0.26517389, 0.41663418],\n",
              "       [0.33366733, 0.3699422 , 0.11133389, 0.12343763, 0.13685723]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HoNf9vDJfrW8",
        "colab_type": "text"
      },
      "source": [
        "Para ficar mais claro a saída do _pipeline_, podemos utilizar os nomes das _features_ geradas através do método `get_feature_names()`. Para tornar ainda mais claro, substituímos o que é chamado `x0` por `Height` e `x1` por `Score`, que é inferido pela ordem das variáveis no _pipeline_."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OJz5zvr2EeM3",
        "colab_type": "code",
        "outputId": "444fe35c-4e5e-4f9c-ef6a-152dd9bcd775",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "poly_features = num_pipeline.get_params()[\"poly_features\"].get_feature_names()\n",
        "  \n",
        "pipeline_columns = [old_name.replace(\"x0\", \"Height_n\").replace(\"x1\", \"Score_n\") for old_name in poly_features]\n",
        "\n",
        "pipeline_columns"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Height_n', 'Score_n', 'Height_n^2', 'Height_n Score_n', 'Score_n^2']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MBgEafF-gKA3",
        "colab_type": "text"
      },
      "source": [
        "Criamos um novo _data set_ com essas variáveis resultantes do _pipeline_:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q_xBepJGIAJm",
        "colab_type": "code",
        "outputId": "6126947b-ef3f-42db-84aa-4317ed5f79d3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 237
        }
      },
      "source": [
        "height_score_normalized_poly = pd.DataFrame(pipeline_transformation, columns=pipeline_columns)\n",
        "\n",
        "height_score_normalized_poly.head(6)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Height_n</th>\n",
              "      <th>Score_n</th>\n",
              "      <th>Height_n^2</th>\n",
              "      <th>Height_n Score_n</th>\n",
              "      <th>Score_n^2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.265531</td>\n",
              "      <td>0.485549</td>\n",
              "      <td>0.070507</td>\n",
              "      <td>0.128928</td>\n",
              "      <td>0.235758</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.490982</td>\n",
              "      <td>0.448940</td>\n",
              "      <td>0.241063</td>\n",
              "      <td>0.220422</td>\n",
              "      <td>0.201547</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.421844</td>\n",
              "      <td>0.488439</td>\n",
              "      <td>0.177952</td>\n",
              "      <td>0.206045</td>\n",
              "      <td>0.238573</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.556112</td>\n",
              "      <td>0.202312</td>\n",
              "      <td>0.309261</td>\n",
              "      <td>0.112508</td>\n",
              "      <td>0.040930</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.366733</td>\n",
              "      <td>0.450867</td>\n",
              "      <td>0.134493</td>\n",
              "      <td>0.165348</td>\n",
              "      <td>0.203281</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>0.505010</td>\n",
              "      <td>0.488439</td>\n",
              "      <td>0.255035</td>\n",
              "      <td>0.246667</td>\n",
              "      <td>0.238573</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Height_n   Score_n  Height_n^2  Height_n Score_n  Score_n^2\n",
              "0  0.265531  0.485549    0.070507          0.128928   0.235758\n",
              "1  0.490982  0.448940    0.241063          0.220422   0.201547\n",
              "2  0.421844  0.488439    0.177952          0.206045   0.238573\n",
              "3  0.556112  0.202312    0.309261          0.112508   0.040930\n",
              "4  0.366733  0.450867    0.134493          0.165348   0.203281\n",
              "5  0.505010  0.488439    0.255035          0.246667   0.238573"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9imGtnaygRiX",
        "colab_type": "text"
      },
      "source": [
        "Podemos também criar outro _pipeline_ para a variável categórica `Course`. Como se trata de uma variável de natureza completamente diferente, precisamos especificar um _pipeline_ diferente com as seguintes transformações:\n",
        "\n",
        "1. Preencha os dados faltantes (`None`) com a classe `Unknown`.\n",
        "2. Crie novas variáveis binárias com o `OneHotEncoder`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eZP_HTkchI5c",
        "colab_type": "text"
      },
      "source": [
        "Assim como no _pipeline_ anterior, especificamos cada etapa como uma tupla com um nome e um objeto de um transformador ou estimador:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NMv_2lV7KxTM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "cat_pipeline = Pipeline([\n",
        "    (\"imputer\", SimpleImputer(strategy=\"constant\", fill_value=\"Unknown\")),\n",
        "    (\"one_hot_encoder\", OneHotEncoder(sparse=False, dtype=np.int))\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wK66jYTShV52",
        "colab_type": "text"
      },
      "source": [
        "Após a especificação do _pipeline_, podemos aplicá-lo à nossa variável `Course`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KIFWvPS7LNUA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "course_pipeline_transformation = cat_pipeline.fit_transform(data_missing[[\"Course\"]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "quJ4ThBBhfBI",
        "colab_type": "text"
      },
      "source": [
        "Agora, utilizaremos o nome que demos à etapa do `OneHotEncoder` para recuperar esse transformador através do método `get_params()`. Depois de recuperado o `OneHotEncoder`, acessamos seu atributo `categories_` (primeiro índice `[0]`, pois poderíamos ter aplicado o _pipeline_ a mais de uma variável categórica):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zurb-NVWM4sX",
        "colab_type": "code",
        "outputId": "1e7c2960-6ffb-4285-bb2d-691157302850",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "course_columns = cat_pipeline.get_params()[\"one_hot_encoder\"].categories_[0]\n",
        "\n",
        "course_columns"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['Biology', 'Math', 'Physics', 'Unknown'], dtype=object)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ABQDGjU_iDGS",
        "colab_type": "text"
      },
      "source": [
        "Utilizamos a saída do _pipeline_ e os nomes das categorias recuperados do transformador para criar um novo `DataFrame`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3ec56uIcMvll",
        "colab_type": "code",
        "outputId": "5707acac-8d67-4d74-eb02-d73b98f6340a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "course_discretized = pd.DataFrame(course_pipeline_transformation, columns=course_columns)\n",
        "\n",
        "course_discretized.head(5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Biology</th>\n",
              "      <th>Math</th>\n",
              "      <th>Physics</th>\n",
              "      <th>Unknown</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Biology  Math  Physics  Unknown\n",
              "0        1     0        0        0\n",
              "1        1     0        0        0\n",
              "2        0     0        0        1\n",
              "3        1     0        0        0\n",
              "4        0     0        1        0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OeO6hmSEiL6N",
        "colab_type": "text"
      },
      "source": [
        "Por fim, combinamos as saídas dos dois _pipelines_ para criar um único `DataFrame`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d8tL_jS1NTf7",
        "colab_type": "code",
        "outputId": "8b39c1c3-e549-4cea-fade-7c8e90d290ba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "data_transformed = pd.concat([height_score_normalized_poly, course_discretized], axis=1)\n",
        "\n",
        "data_transformed.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Height_n</th>\n",
              "      <th>Score_n</th>\n",
              "      <th>Height_n^2</th>\n",
              "      <th>Height_n Score_n</th>\n",
              "      <th>Score_n^2</th>\n",
              "      <th>Biology</th>\n",
              "      <th>Math</th>\n",
              "      <th>Physics</th>\n",
              "      <th>Unknown</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.265531</td>\n",
              "      <td>0.485549</td>\n",
              "      <td>0.070507</td>\n",
              "      <td>0.128928</td>\n",
              "      <td>0.235758</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.490982</td>\n",
              "      <td>0.448940</td>\n",
              "      <td>0.241063</td>\n",
              "      <td>0.220422</td>\n",
              "      <td>0.201547</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.421844</td>\n",
              "      <td>0.488439</td>\n",
              "      <td>0.177952</td>\n",
              "      <td>0.206045</td>\n",
              "      <td>0.238573</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.556112</td>\n",
              "      <td>0.202312</td>\n",
              "      <td>0.309261</td>\n",
              "      <td>0.112508</td>\n",
              "      <td>0.040930</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.366733</td>\n",
              "      <td>0.450867</td>\n",
              "      <td>0.134493</td>\n",
              "      <td>0.165348</td>\n",
              "      <td>0.203281</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>0.505010</td>\n",
              "      <td>0.488439</td>\n",
              "      <td>0.255035</td>\n",
              "      <td>0.246667</td>\n",
              "      <td>0.238573</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>0.405812</td>\n",
              "      <td>0.734104</td>\n",
              "      <td>0.164683</td>\n",
              "      <td>0.297908</td>\n",
              "      <td>0.538909</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>0.330661</td>\n",
              "      <td>0.514451</td>\n",
              "      <td>0.109337</td>\n",
              "      <td>0.170109</td>\n",
              "      <td>0.264660</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>0.410822</td>\n",
              "      <td>0.645472</td>\n",
              "      <td>0.168774</td>\n",
              "      <td>0.265174</td>\n",
              "      <td>0.416634</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>0.333667</td>\n",
              "      <td>0.369942</td>\n",
              "      <td>0.111334</td>\n",
              "      <td>0.123438</td>\n",
              "      <td>0.136857</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Height_n   Score_n  Height_n^2  ...  Math  Physics  Unknown\n",
              "0  0.265531  0.485549    0.070507  ...     0        0        0\n",
              "1  0.490982  0.448940    0.241063  ...     0        0        0\n",
              "2  0.421844  0.488439    0.177952  ...     0        0        1\n",
              "3  0.556112  0.202312    0.309261  ...     0        0        0\n",
              "4  0.366733  0.450867    0.134493  ...     0        1        0\n",
              "5  0.505010  0.488439    0.255035  ...     0        1        0\n",
              "6  0.405812  0.734104    0.164683  ...     0        0        0\n",
              "7  0.330661  0.514451    0.109337  ...     0        1        0\n",
              "8  0.410822  0.645472    0.168774  ...     0        1        0\n",
              "9  0.333667  0.369942    0.111334  ...     0        1        0\n",
              "\n",
              "[10 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1NLD-pyliXWO",
        "colab_type": "text"
      },
      "source": [
        "Vale ressaltar que:\n",
        "\n",
        "* Poderíamos utilizar também o `ColumnTransformer` para compor (por isso, ele se encontra no módulo `sklearn.compose`) múltiplos `Pipeline` em diferentes variáveis.\n",
        "* Os `Pipeline` não servem apenas para a transformação dos dados de treinamento. Eles também podem (e devem) ser usados para submeter os dados de teste e até de produção aos mesmos procedimentos dos dados de treinamento."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbShR7kMZGwE",
        "colab_type": "text"
      },
      "source": [
        "## _Outliers_\n",
        "\n",
        "_Outliers_, os famosos \"pontos fora da curva\", são observações que não parecem seguir o mesmo padrão dos demais dados. Eles podem vir de distribuições diferentes, serem erros na coleta de dados, erros de medição etc.\n",
        "\n",
        "Eles influenciam nossas análises e os nossos algoritmos ao apresentar comportamento distoante do resto do _data set_, impactando na média, variância, funções de perda e custo etc. Se fizer sentido, eles devem ser removidos ou transformados antes de prosseguirmos com a análise.\n",
        "\n",
        "No entanto, devemos julgar com cautela sua remoção: __alguns _outliers_ são dados autênticos e devem ser estudados com atenção__. Por exemplo, a remoção de uma medição muito alta na temperatura de um reator seria um erro, pois essa medição pode estar nos indicando um potencial problema com o dispositivo.\n",
        "\n",
        "Abaixo estudamos algumas técnicas simples para encontrar _outliers_.\n",
        "\n",
        "![outlier](https://www.stats4stem.org/common/web/plugins/ckeditor/plugins/doksoft_uploader/userfiles/WithInfOutlier.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u3bsTDv0pAN4",
        "colab_type": "text"
      },
      "source": [
        "Começamos criando uma cópia da variável `Height` do nosso _data set_ para não impactar o original:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tQ7AQztcZkYx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "height_outlier = data.Height.copy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VQNHBAu4pHcp",
        "colab_type": "text"
      },
      "source": [
        "Adicionamos dez  _outliers_ que representam pessoas estranhamente baixas ou estranhamente altas para o padrão que estamos observando:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nX2R3V0HZI0w",
        "colab_type": "code",
        "outputId": "6acbd63c-820e-485a-cde4-72a69fefe13d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 208
        }
      },
      "source": [
        "height_outlier_idx = pd.Index(np.random.choice(height_outlier.index, 10, replace=False))\n",
        "\n",
        "too_short_idx = pd.Index(height_outlier_idx[:5])\n",
        "too_tall_idx = pd.Index(height_outlier_idx[5:])\n",
        "\n",
        "height_outlier[too_short_idx] = np.random.normal(loc=1.30, scale=0.5, size=5)\n",
        "height_outlier[too_tall_idx] = np.random.normal(loc=2.20, scale=0.5, size=5)\n",
        "\n",
        "outlier_idx = too_short_idx | too_tall_idx\n",
        "\n",
        "height_outlier[outlier_idx]\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "14    1.646795\n",
              "18    1.696510\n",
              "29    0.516665\n",
              "38    2.943781\n",
              "48    1.058498\n",
              "49    1.326605\n",
              "57    2.074231\n",
              "66    1.831315\n",
              "68    2.737088\n",
              "96    1.966029\n",
              "Name: Height, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mwNbTzDnpoDL",
        "colab_type": "text"
      },
      "source": [
        "Note que nem todos dados gerados se tornaram realmente _outliers_. Como geramos de uma distribuição aleatória, corremos esse risco.\n",
        "\n",
        "No entanto, temos alguns dados estranhos como 0.51 m e 2.73 m."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x5pwD_1EqRNZ",
        "colab_type": "text"
      },
      "source": [
        "No _boxplot_ padrão, os dados mais extremos são mostrados como pontos fora do alcance dos _whiskers_ (as barrinhas do _box plot_).\n",
        "\n",
        "No caso abaixo, notamos três pontos acima e três pontos abaixo do considerado \"dentro da faixa normal\"."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hRMVhYz3b2KH",
        "colab_type": "code",
        "outputId": "9e090cef-804c-4f17-958b-5e25154662db",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 695
        }
      },
      "source": [
        "sns.boxplot(height_outlier, orient=\"vertical\");"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAt0AAAKmCAYAAACR0iLwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3W+s3nV9//HXdZ22YbWupz1WOPyx\naLYQElgwcyEn7SAUsCSjnGLWQZg62YYy0G6LGe3449KCjpKFZGNk1c0QWZwhZFQ5woQBC3i64zLv\nLKWS2p9RKv0HHHrqiujsua7fDfXEyr9TPe/raq/zeNw61znf61yv3sGn33zOdTXa7XY7AABAmWa3\nBwAAQK8T3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0AwBA\nMdENAADF5nR7wEw7cODltFrtbs8AAKBHNZuNLFr0lqN6Ts9Fd6vVFt0AABxTHC8BAIBiohsAAIqJ\nbgAAKCa6AQCgmOgGAIBiohsAAIqJbgAAKCa6AQCgmOgGAIBiohsAAIp17GPgr7vuujz33HNpNpuZ\nP39+brnllpx55plHXDM5OZnbbrstX/3qV9NoNPLhD384a9as6dREAAAo0bHo3rRpU9761rcmSR57\n7LHceOON2bJlyxHXjIyMZNeuXXn00UczMTGR1atXZ2hoKKeeemqnZgIAwIzr2PGSnwZ3khw6dCiN\nRuNV1zz88MNZs2ZNms1mFi9enIsuuihf+cpXOjURAABKdOxOd5LcdNNN2bp1a9rtdv7pn/7pVT/f\nu3dvTj755KnHg4OD2bdv31G9xsDAgl96JwAAzKSORvcnP/nJJMkXv/jF3HHHHfnHf/zHGX+N8fFD\nabXaM/57AQAgSZrNxlHf6O3Ku5esXr06//Vf/5UDBw4c8f3BwcHs2bNn6vHevXtz0kkndXoeAADM\nqI5E98svv5y9e/dOPX7iiSeycOHC9Pf3H3HdJZdckvvvvz+tVisvvfRSHnvssaxcubITEwEAoExH\njpe88sor+dM//dO88soraTabWbhwYTZv3pxGo5Frrrkma9euzdlnn53h4eH8z//8T9773vcmSa6/\n/vqcdtppnZgIAABlGu12u6cOQDvTDQBApePmTDcAAMwmohsAAIqJbgAAKCa6AQCgmOgGmMUmJg7k\n9ts35uDBiW5PAehpohtgFhsZ2ZKdO3fkwQcf6PYUgJ4mugFmqYmJAxkdfTLtdjujo0+52w1QSHQD\nzFIjI1umPteg1Wq52w1QSHQDzFJjY1szOXk4STI5eThjY1u7vAigd4lugFlqaGhZ+vrmJEn6+uZk\naGhZlxcB9C7RDTBLrVp1eZrNRpKk2Wzmssve1+VFAL1LdAPMUv39i7J8+flpNBpZvvy8LFzY3+1J\nAD1rTrcHANA9q1Zdnt27n3OXG6BYo91ut7s9YiaNjx+a+mt8AACYac1mIwMDC47uOUVbAACAnxDd\nAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVE\nNwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx\n0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQ\nTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdALPYxMSB3H77xhw8ONHtKQA9\nTXQDzGIjI1uyc+eOPPjgA92eAtDTRDfALDUxcSCjo0+m3W5ndPQpd7sBColugFlqZGRLWq12kqTV\narnbDVBIdAPMUmNjWzM5eThJMjl5OGNjW7u8CKB3iW6AWWpoaFn6+uYkSfr65mRoaFmXFwH0LtEN\nMEutWnV5ms1GkqTZbOayy97X5UUAvUt0A8xS/f2Lsnz5+Wk0Glm+/LwsXNjf7UkAPWtOtwcA0D2r\nVl2e3bufc5cboFij3W63uz1iJo2PH5r6a3wAAJhpzWYjAwMLju45RVsAAICfEN0AAFBMdAMAQDHR\nDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBM\ndAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAU\nE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAA\nxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMA\nQDHRDQAAxeZ04kUOHDiQG264Ibt27cq8efOydOnSbNy4MYsXLz7iuvXr1+c///M/s2jRoiTJJZdc\nkj/5kz/pxEQAACjTaLfb7eoXmZiYyI4dO3LuuecmSTZt2pSDBw/mU5/61BHXrV+/PmeddVbe//73\n/8KvNT5+KK1W+T8JAIBZqtlsZGBgwdE9p2jLEfr7+6eCO0nOOeec7NmzpxMvDQAAXdfxM92tVitf\n+MIXsmLFitf8+T333JNVq1bluuuuy7e+9a0OrwMAgJnXkeMlP2vDhg3Zv39//v7v/z7N5pHNv3//\n/ixZsiTNZjNf/OIX87d/+7d57LHH0tfX18mJAAAwozoa3Zs2bcqOHTuyefPmzJs3702vP/fcc/PA\nAw/klFNOmfZrONMNAEClY/ZMd5Lceeedefrpp3P33Xe/bnDv379/6uuvfvWraTabOfHEEzs1EQAA\nSnTkTvfOnTtz6aWX5vTTT88JJ5yQJDn11FNz9913Z3h4OJ/5zGdy4okn5kMf+lDGx8fTaDSyYMGC\n3HDDDTnnnHOO6rXc6QYAoNIvcqe742e6q4luAAAqHdPHSwAAYLYS3QAAUEx0AwBAMdENAADFRDcA\nABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0AwBAMdEN\nAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0\nAwBAMdENAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT\n3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0A8xiExMHcvvt\nG3Pw4ES3pwD0NNENMIuNjGzJzp078uCDD3R7CkBPE90As9TExIGMjj6Zdrud0dGn3O0GKCS6AWap\nkZEtabXaSZJWq+VuN0Ah0Q0wS42Nbc3k5OEkyeTk4YyNbe3yIoDeJboBZqmhoWXp65uTJOnrm5Oh\noWVdXgTQu0Q3wCy1atXlaTYbSZJms5nLLntflxcB9C7RDTBL9fcvyvLl56fRaGT58vOycGF/tycB\n9Kw53R4AQPesWnV5du9+zl1ugGKNdrvd7vaImTQ+fmjqr/EBAGCmNZuNDAwsOLrnFG0BAAB+wvES\nmCW2bn0qo6NPdnsGx5iffiCO89z8vOXLz8+yZed1ewb0DHe6AWaxgwcP5uDBg92eAdDznOkGmMU2\nbbo1SbJu3S1dXgJw/HCmGwAAjkGiGwAAioluAAAoJroBAKCY6AYAgGKiGwAAioluAAAoJroBAKCY\n6AYAgGKiGwAAioluAAAoJroBAKCY6AYAgGKiGwAAioluAAAoJroBAKCY6AYAgGKiGwAAioluAAAo\nJroBAKCY6AYAgGKiGwAAioluAAAoJroBAKCY6AYAgGKiGwAAioluAAAoJroBAKCY6AYAgGKiGwAA\nioluAAAoJroBAKCY6AYAgGKiGwAAioluAAAoJroBAKCY6AYAgGKiGwAAioluAAAoJroBAKCY6AYA\ngGKiGwAAioluAAAoJroBAKCY6AYAgGKiGwAAioluAAAoJroBAKCY6AYAgGKiGwAAioluAAAoJroB\nAKCY6AYAgGJzOvEiBw4cyA033JBdu3Zl3rx5Wbp0aTZu3JjFixcfcd0rr7ySv/zLv8z27dvT19eX\ndevW5YILLujERAAAKNORO92NRiN//Md/nEceeSQjIyM57bTT8jd/8zevuu6zn/1sFixYkH//93/P\n5s2bc/PNN+fll1/uxEQAACjTkeju7+/PueeeO/X4nHPOyZ49e1513b/927/liiuuSJKcfvrpOeus\ns/LUU091YiIAAJTp+JnuVquVL3zhC1mxYsWrfrZnz56ccsopU48HBwezb9++Ts4DAIAZ15Ez3T/r\n1ltvzfz58/P+97+/5PcPDCwo+b0AvWju3L4kyZIlb+3yEoDe1tHo3rRpU5599tls3rw5zearb7Kf\nfPLJ2b1799QfWO7du/eIYynTMT5+KK1We0b2AvS6H/1oMknywgv/2+UlAMePZrNx1Dd6O3a85M47\n78zTTz+du+++O/PmzXvNay655JLcd999SZLvfOc72bZtW377t3+7UxMBAKBER6J7586d+fSnP53n\nn38+V155ZYaHh3P99dcnSYaHh7N///4kyR/90R/le9/7Xi6++OJ85CMfycaNG7NggeMiAAAc3zpy\nvOTXf/3Xs2PHjtf82Ze+9KWpr+fPn5+/+7u/68QkAADoGJ9ICQAAxUQ3AAAUE90AAFBMdAMAQDHR\nDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBM\ndAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAU\nE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFCs0W63290eMZPGxw+l1eqpf9JR+5d/uTff/e6z3Z4B\nHAd27frxfyve8Y6lXV4CHA9OO21prrrqg92e0XXNZiMDAwuO6jlzirbQRd/97rPZsfP/pe+E/m5P\nAY5xrcm+JMn/++6LXV4CHOsmfzDR7QnHNdHdo/pO6M/8pRd2ewYA0CO+/+zj3Z5wXHOmGwAAiolu\nAAAoJroBAKCY6AYAgGKiGwAAioluAAAoJroBAKCY6AYAgGKiGwAAioluAAAoNu3o/uxnP/ua37/n\nnntmbAwAAPSiaUf33Xff/Zrf/4d/+IcZGwMAAL1ozptdMDY2liRptVr52te+lna7PfWz5557Lm95\ny1vq1gEAQA940+i+6aabkiQ//OEPc+ONN059v9FoZMmSJbn55pvr1gEAQA940+h+4oknkiQ33HBD\n7rjjjvJBAADQa940un/qZ4O71Wod8bNm05ugAADA65l2dG/fvj0bN27Mjh078sMf/jBJ0m6302g0\n8swzz5QNBACA4920o3v9+vW54IIL8qlPfSonnHBC5SYAAOgp047u3bt358///M/TaDQq9wAAQM+Z\n9mHsiy++OKOjo5VbAACgJ73hne6/+Iu/mLqz/X//93/56Ec/mt/8zd/M2972tiOu864mAADw+t4w\nupcuXXrE41/7tV8rHQMAAL3oDaP7ox/9aKd2AABAz5r2H1L+9OPgf968efNy0kkn5ZRTTpmxUQAA\n0EumHd033XRTnn/++SRJf39/JiYmkiQDAwN58cUXc8YZZ+TOO+/M6aefXjIUAACOV9N+95Lf/d3f\nzQc+8IF8/etfz+joaL7+9a/nD/7gD3LllVfmv//7v3PWWWdlw4YNlVsBAOC4NO3ovvfee/Pxj398\n6oNxTjjhhPzZn/1ZPve5z2X+/PlZv359nn766bKhAABwvJp2dM+fPz/btm074nvbt2/Pr/zKr/z4\nFzWn/asAAGBWmfaZ7rVr1+YP//APs2LFigwODmbfvn35j//4j9xyyy1JfvyHlitXriwbCgAAx6tp\nR/fq1atz1lln5ZFHHsnzzz+f008/Pffdd9/Ue3dfcMEFueCCC8qGAgDA8Wra0Z38+MNxfEAOAAAc\nnTeM7ltuuSW33nprkiM/Ev7n+Rh4AAB4fW8Y3aeeeurU1z//kfAAAMD0vGF0f+QjH5n62kfCAwDA\nL+aoznRv3bo1Dz30UF566aVs3rw527Zty6FDhzI0NFS1DwAAjnvTju5//ud/zr333ps1a9bkkUce\nSfLjD8j55Cc/KbqPMQcPTmTyBxP5/rOPd3sKANAjJn8wkYMHj+p+LT9j2p9o87nPfS733HNPPvzh\nD099EM673vWufPvb3y4bBwAAvWDa/3fl5ZdfzuDgYJJMvYvJ4cOHM3fu3Jpl/MIWLuzPC987nPlL\nL+z2FACgR3z/2cezcGF/t2cct6Z9p/u3fuu38pnPfOaI7917770599xzZ3wUAAD0kmnf6b755ptz\n7bXX5v7778/LL7+clStX5i1veUs+/elPV+4DAIDj3rSj++1vf3v+9V//Ndu2bcvu3btz8skn5zd+\n4zde9wNzAACAH3vT6L7qqqveNKw///nPz9ggAADoNW8a3WvWrJn6ut1u59Zbb80nPvGJ0lEAANBL\n3jS6L7/88iMe//Vf//WrvgcAALy+ab97yU85ww0AAEfnqKMbAAA4Om96vGRsbOyIx4cPH87Xvva1\ntNvtqe/5GHgAAHh9bxrdN9100xGP+/v7c+ONN049bjQaefzxx2d+GQAA9Ig3je4nnniiEzsAAKBn\nOdMNAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAA\nUEx0AwBAMdENAADF5nR7ADUmfzCR7z/7eLdnAMe41uEfJEmac07o8hLgWDf5g4kkb+v2jOOW6O5B\np522tNsTgOPErl3PJknecZr/IQXezNs0xi+h0W63290eMZPGxw+l1eqpfxJAmU2bbk2SrFt3S5eX\nABw/ms1GBgYWHN1zirYAAAA/IboBAKCY6AYAgGKiGwAAioluAAAoJroBAKCY6AYAgGKiGwAAiolu\nAAAoJroBAKCY6AYAgGKiGwAAioluAAAoJroBAKCY6AYAgGIdi+5NmzZlxYoVOeOMM/LNb37zNa+5\n6667MjQ0lOHh4QwPD2fDhg2dmgcAAGXmdOqFLrzwwnzwgx/M7//+77/hdatXr866des6tAoAAOp1\nLLrf8573dOqlAADgmNKx6J6uhx56KKOjo1myZEk+9rGP5d3vfvdRPX9gYEHRMoDeM3duX5JkyZK3\ndnkJQG87pqL7yiuvzLXXXpu5c+dm69atue666/Lwww9n0aJF0/4d4+OH0mq1C1cC9I4f/WgySfLC\nC//b5SUAx49ms3HUN3qPqXcvWbJkSebOnZskWbZsWQYHB7Nz584urwIAgF/OMRXd+/fvn/r6mWee\nye7du/POd76zi4sAAOCX17HjJbfddlseffTRvPjii7n66qvT39+fhx56KNdcc03Wrl2bs88+O3fe\neWe2b9+eZrOZuXPn5o477siSJUs6NREAAEo02u12Tx2AdqYbYPo2bbo1SbJu3S1dXgJw/Djuz3QD\nAEAvEt0AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHR\nDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBM\ndAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAU\nE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAAxUQ3AAAUE90AAFBMdAMAQDHRDQAA\nxUQ3AAAUE90AAFBMdAMAQDHRDQAAxRrtdrvd7REzaXz8UFqtnvonwYzYuvWpjI4+2e0ZHGN27Xo2\nSfKOdyzt8hKONcuXn59ly87r9gw4JjWbjQwMLDiq58wp2gLAcWDhwoXdngAwK7jTDQAAR+EXudPt\nTDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0AwBA\nMdENAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QCz\n2MTEgdx++8YcPDjR7SkAPU10A8xiIyNbsnPnjjz44APdngLQ00Q3wCw1MXEgo6NPpt1uZ3T0KXe7\nAQqJboBZamRkS1qtdpKk1Wq52w1QSHQDzFJjY1szOXk4STI5eThjY1u7vAigd4lugFlqaGhZ+vrm\nJEn6+uZkaGhZlxcB9C7RDTBLrVp1eZrNRpKk2Wzmssve1+VFAL1LdAPMUv39i7J8+flpNBpZvvy8\nLFzY3+1JAD1rTrcHANA9q1Zdnt27n3OXG6BYo91ut7s9YiaNjx+a+mt8AACYac1mIwMDC47uOUVb\nAACAnxDdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx\n0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVEN8AsNjFxILffvjEHD050ewpATxPdALPYyMiW7Ny5\nIw8++EC3pwD0NNENMEtNTBzI6OiTabfbGR19yt1ugEKiG2CWGhnZklarnSRptVrudgMUEt0As9TY\n2NZMTh5OkkxOHs7Y2NYuLwLoXaIbYJYaGlqWvr45SZK+vjkZGlrW5UUAvUt0A8xSq1ZdnmazkSRp\nNpu57LL3dXkRQO8S3QCzVH//oixffn4ajUaWLz8vCxf2d3sSQM+a0+0BAHTPqlWXZ/fu59zlBijW\naLfb7W6PmEnj44em/hofAABmWrPZyMDAgqN7TtEWAADgJ0Q3AAAUE90AAFBMdAMAQDHRDQAAxToS\n3Zs2bcqKFStyxhln5Jvf/OZrXjM5OZkNGzbkoosuysUXX5z777+/E9MAAKBcR6L7wgsvzOc///mc\ncsopr3vNyMhIdu3alUcffTT33Xdf7rrrrjz33HOdmAcAAKU6Et3vec97Mjg4+IbXPPzww1mzZk2a\nzWYWL16ciy66KF/5ylc6MQ8AAEodM59IuXfv3px88slTjwcHB7Nv376j/j1H+0blAABQ7ZiJ7pni\nEykBAKh0XH8i5eDgYPbs2TP1eO/evTnppJO6uAgAAGbGMRPdl1xySe6///60Wq289NJLeeyxx7Jy\n5cpuzwIAgF9aR6L7tttuy3nnnZd9+/bl6quvzu/8zu8kSa655pps27YtSTI8PJxTTz01733ve/N7\nv/d7uf7663Paaad1Yh4AAJRqtNvtnjoA7Uw3AACVjusz3QAA0KtENwAAFBPdAABQTHQDAEAx0Q0A\nAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQD\nAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPd\nAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVE\nNwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx\n0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQ\nTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAA\nFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0A\nAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQD\nAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0AAMVENwAAFBPd\nAABQTHQDAEAx0Q0AAMVENwAAFBPdAABQTHQDAEAx0Q0wi01MHMjtt2/MwYMT3Z4C0NNEN8AsNjKy\nJTt37siDDz7Q7SkAPU10A8xSExMHMjr6ZNrtdkZHn3K3G6CQ6AaYpUZGtqTVaidJWq2Wu90AhUQ3\nwCw1NrY1k5OHkySTk4czNra1y4sAelfHovvb3/52rrjiiqxcuTJXXHFFvvOd77zqmrvuuitDQ0MZ\nHh7O8PBwNmzY0Kl5ALPO0NCy9PXNSZL09c3J0NCyLi8C6F1zOvVCf/VXf5Wrrroqw8PD+dKXvpRP\nfOITuffee1913erVq7Nu3bpOzQKYtVatujyjo09mcjJpNpu57LL3dXsSQM/qyJ3u8fHxfOMb38il\nl16aJLn00kvzjW98Iy+99FInXh6A19DfvyjLl5+fRqOR5cvPy8KF/d2eBNCzOnKne+/evTnxxBPT\n19eXJOnr68vb3/727N27N4sXLz7i2oceeiijo6NZsmRJPvaxj+Xd7373Ub3WwMCCGdsN0Os+9KEP\n5Pnn9+bqqz+YRYve2u05AD2rY8dLpuPKK6/Mtddem7lz52br1q257rrr8vDDD2fRokXT/h3j44em\n/hofgDczNx//+E05fDh54YX/7fYYgONCs9k46hu9HTleMjg4mP3792dycjJJMjk5meeffz6Dg4NH\nXLdkyZLMnTs3SbJs2bIMDg5m586dnZgIAABlOhLdAwMDOfPMM/PlL385SfLlL385Z5555quOluzf\nv3/q62eeeSa7d+/OO9/5zk5MBACAMo12u92Rsxjf+ta3sn79+nzve9/Lr/7qr2bTpk1517velWuu\nuSZr167N2WefnXXr1mX79u1pNpuZO3du1q5dm/PPP/+oXsfxEgAAKv0ix0s6Ft2dIroBAKh0zJ7p\nBgCA2Ux0AwBAMdENAADFRDcAABQT3QAAUEx0AwBAMdENAADFRDcAABQT3QAAUEx0AwBAMdENAADF\nRDcAABQT3QAAUEx0AwBAMdHAkSFcAAAAe0lEQVQNAADFRDcAABQT3QAAUEx0AwBAMdENAADF5nR7\nwExrNhvdngAAQA/7RXqz0W632wVbAACAn3C8BAAAioluAAAoJroBAKCY6AYAgGKiGwAAioluAAAo\nJroBAKCY6AYAgGKiGwAAioluAAAoJroBAKCY6AYAgGL/H0DD/OPJX0Z9AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 864x864 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MOKP49JMqTog",
        "colab_type": "text"
      },
      "source": [
        "Uma primeira abordagem bem simples é encontrar os pontos do _box plot_ acima.\n",
        "\n",
        "Tudo que estiver fora da faixa $[Q1 - 1.5 \\times \\text{IQR}, Q3 + 1.5 \\times \\text{IQR}]$ é considerado um ponto anômalo para aquele padrão:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z_h0zaVDce0N",
        "colab_type": "code",
        "outputId": "86b9e772-6438-4820-87ba-dab83a4b1dd8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "q1 = height_outlier.quantile(0.25)\n",
        "q3 = height_outlier.quantile(0.75)\n",
        "iqr = q3 - q1\n",
        "\n",
        "non_outlier_interval_iqr = [q1 - 1.5 * iqr, q3 + 1.5 * iqr]\n",
        "\n",
        "print(f\"Faixa considerada \\\"normal\\\": {non_outlier_interval_iqr}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Faixa considerada \"normal\": [1.18575, 2.24175]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wsuVvr8hq4Rc",
        "colab_type": "text"
      },
      "source": [
        "Agora podemos identificar quais pontos encontram-se fora desse intervalo, ou seja, podem ser considerados _outliers_:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hm78PWbhc9Dz",
        "colab_type": "code",
        "outputId": "ee3995ea-8a63-4c90-b3dd-57ba673887ee",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        }
      },
      "source": [
        "outliers_iqr = height_outlier[(height_outlier < non_outlier_interval_iqr[0]) | (height_outlier > non_outlier_interval_iqr[1])]\n",
        "\n",
        "outliers_iqr"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "29    0.516665\n",
              "38    2.943781\n",
              "48    1.058498\n",
              "68    2.737088\n",
              "91    2.272000\n",
              "92    1.164000\n",
              "Name: Height, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XcF70kmerGEq",
        "colab_type": "text"
      },
      "source": [
        "Se estivermos seguos de que esses pontos representam de fato _outliers_ e que sua remoção não traz prejuízo à nossa análise, então podemos removê-los:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BVRJS9DNeb9z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "height_no_outlier_iqr = height_outlier.drop(index=outliers_iqr.index)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "urvTyUfHrVrJ",
        "colab_type": "text"
      },
      "source": [
        "Uma segunda abordagem é observar as estatísticas descritivas dos dados.\n",
        "\n",
        "Repare no histograma abaixo:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bc_paOePfHJ5",
        "colab_type": "code",
        "outputId": "6840da1c-bae6-4465-8aa7-87f69928e182",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 726
        }
      },
      "source": [
        "sns.distplot(height_outlier);"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtMAAALFCAYAAAABe2+3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XmMpHd97/vPU/vaXb1U99RszNgH\nzATsw+U69uX4OueCcRjdeDSWUGJfjIxscGScyII/guePEcbEUjSjyFKwTEgskog/iJBBgLwQL/ci\nkTjnZEE+YLxjz3jGPb1WdXft+3P/6K72LN3T1dVV/Wzvl4TUU1Vd/bUfuvvj73yf788wTdMUAAAA\ngC3zWV0AAAAA4FSEaQAAAKBHhGkAAACgR4RpAAAAoEeEaQAAAKBHhGkAAACgR4RpAAAAoEeEaQAA\nAKBHhGkAAACgR4RpAAAAoEeEaQAAAKBHhGkAAACgR4RpAAAAoEcBqwvo1uJiSe22aXUZjjM2llA2\nW7S6DGyA62NvXB974/rYG9fH3rg+6/P5DI2MxLf0OY4J0+22SZjuEf/e7I3rY29cH3vj+tgb18fe\nuD79wZgHAAAA0CPCNAAAANAjwjQAAADQI8I0AAAA0CPCNAAAANAjwjQAAADQI8I0AAAA0CPCNAAA\nANAjwjQAAADQI8I0AAAA0CPCNAAAANAjwjQAAADQI8I0AAAA0CPCNAAAANAjwjQAAADQI8I0AAAA\n0CPCNAAAANAjwjQAAADQI8I0AAAA0CPCNAAAANAjwjQAAADQI8I0AAAA0KNNw/Ti4qLuuecefeYz\nn9GRI0f0p3/6p8rlcpe8rlKp6Ctf+YpuvvlmHT58WD//+c+7eg4AAABwqk3DtGEY+tKXvqRnn31W\nTz75pPbt26e//Mu/vOR13/3ud5VIJPT888/rO9/5jo4fP65SqbTpcwAAAIBTbRqmU6mUrr/++rU/\nf+xjH9O5c+cued3PfvYz3XbbbZKkAwcO6KMf/ah+8YtfbPocAAAA4FRbmplut9v6x3/8R33qU5+6\n5Llz585pz549a3/OZDKamZnZ9DkAAADAqQJbefGf//mfKxaL6fOf//yg6tnQ2Fhix7+mW6TTSatL\nwGVwfeyN62NvXB974/rYG9enP7oO0ydOnNC7776r73znO/L5Lm1o7969W1NTUxodHZUkTU9Pr42H\nXO65bmWzRbXb5pY+ByvfKPPzBavLwAa4PvbG9bE3ro+9cX3sjeuzPp/P2HIDt6sxj0ceeUS/+c1v\n9NhjjykUCq37msOHD+sHP/iBJOn06dN6+eWXdeONN276HAB4WbMtlWrNgfyv2bb6nw4A3G/TzvRb\nb72lv/mbv9GBAwd0++23S5L27t2rxx57TEePHtXf/u3fanJyUl/84hd17Ngx3XzzzfL5fPrmN7+p\nRGIl2V/uOQDwslqjqf94bXYg7/27hyYVCG9pmg8AsEWGaZqOmJ1gzKM3/DWOvXF97G0nrk+pNtgw\nHXdxmOb7x964PvbG9VnfwMY8AAAAAFyKMA0AAAD0iDANAAAA9IgwDQAAAPSIMA0AAAD0iDANAAAA\n9IgwDQAAAPSIMA0AAAD0iDANAAAA9IgwDQAAAPSIMA0AAAD0iDANAAAA9IgwDQAAAPSIMA0AAAD0\niDANAAAA9IgwDQAAAPSIMA0AAAD0iDANAAAA9IgwDQAAAPSIMA0AAAD0iDANAAAA9IgwDQAAAPSI\nMA0AAAD0iDANAAAA9IgwDQAAAPSIMA0AAAD0iDANAAAA9IgwDQAAAPSIMA0AAAD0iDANAAAA9Igw\nDQAAAPSIMA0AAAD0iDANAAAA9IgwDQAAAPSIMA0AAAD0iDANAAAA9IgwDQAAAPSIMA0AAAD0iDAN\nAAAA9IgwDQAAAPSIMA0AAAD0iDANAAAA9IgwDQAAAPSIMA0AAAD0iDANAAAA9IgwDQAAAPSIMA0A\nAAD0iDANAAAA9IgwDQAAAPSIMA0AAAD0iDANAAAA9IgwDQAAAPSIMA0ADtBottVum1aXAQC4SMDq\nAgAAl3duoaRf/OqcktGQPv27exUO+q0uCQCwis40ANiUaZp69XRO/+9/vqdw0K/FQk0v/MdZ1Rot\nq0sDAKwiTAOADTVbbb348oz+8/V57ZtM6Jb/dkD/18d3a7FQ1/P/cVa1OoEaAOyAMA0ANlOuNvTs\nv53VO+fy+tgHx/XfP7ZbwYBPe9MJffLju7VUqOv5/yRQA4AdEKYBwGb+xyuzWi7V9MmP79E1V47J\nMIy15/Z0AnVxNVAz8gEAliJMA4CNVGpNnVso6cP7R7RvIrHua/akE/rk/7ZHi4WaXn47u8MVAgDO\nR5gGABt5d6Yg05QO7h667Ov2pOPaP5nUb99bVrPV3qHqAAAXI0wDgI2cms4rlQhpJBne9LUf3p9S\nvdnWqen8DlQGAFgPYRoAbKJQrmt+qbppV7pjYiSqVCKkN84syTQ50AUArECYBgCbOD1dkCQdzHQX\npg3D0FX7U8rla1pYrg6yNADABgjTAGADpmnqnem8JkaiSkSDXX/eFbuHFfT79MaZpQFWBwDYCGEa\nAGxgsVDTcrGug5nklj4vGPDpij1DOj1dULXeHFB1AICNEKYBwAZOTRdkGNIHdm0tTEvSVftSapum\nfvve8gAqAwBcDmEaACxmmqZOTee1ezyuSCiw5c9PJcOaHI3qjTNLanMjIgDsKMI0AFhsbrGicrXZ\n9Y2H67lq/4hK1aam5kt9rAwAsBnCNABY7NR0XgG/seGJh93YP5FQNOznRkQA2GFd/X3iiRMn9Oyz\nz2pqakpPPvmkPvShD13ymq997Wt644031v78xhtv6LHHHtNNN92kRx99VN///vc1MTEhSfr4xz+u\nBx98sE//CADgXM1WW6dnCto3kVAw0Ht/w+cz9MG9Kf367awK5bqSsVAfqwQAbKSrMH3TTTfpzjvv\n1B133LHha06ePLn28euvv64vfOELuvHGG9ceu/XWW/XAAw9so1QAcJ/XTi+q3mhva8Sj48o9Q/r1\n21mdnS3qdw6O9qE6AMBmumqDXHvttcpkMl2/6Q9/+EMdOXJEoRCdEQC4nDfPLsnvM5QZj2/7vZKx\nkFKJkN5jbhoAdkzfZ6br9bqefPJJffazn73g8aefflpHjhzR3XffrZdeeqnfXxYAHOnMbEGjQ2H5\nfUZf3m9POqHZxbLqjVZf3g8AcHlb38G0iRdeeEG7d+/WoUOH1h67/fbbde+99yoYDOrFF1/Ufffd\np2eeeUYjIyNdv+/YWO835nhdOr31vbXYOVwfexvk9Wm12npvrqgPHxhVMhHpy3te9YFRvXIqp1yx\noVgsrPRorC/va1d8/9gb18feuD790fcw/aMf/eiSrnQ6nV77+IYbblAmk9Fbb72l6667ruv3zWaL\narfZn7pV6XRS8/MFq8vABrg+9jbo63NmtqB6s62hWFCFYrUv7xkL+xQK+vTbs4sql2uab7m3Q833\nj71xfeyN67M+n8/YcgO3r2MeMzMz+uUvf6kjR45c8Pjs7Ozax6+99pqmpqZ08ODBfn5pAHCc0zMr\nv8jGh/vTlZYkn2Fobzqh9+ZpQADATuiqM/3www/rueee08LCgu666y6lUik9/fTTuueee3T//ffr\n6quvliT9+Mc/1ic/+UkNDw9f8PmPPPKIXnnlFfl8PgWDQZ08efKCbjUAeNGp6byiYb+SsWBf33dP\nOq53zuV1ejqvq68Y6+t7AwAuZJimM86eZcyjN/w1jr1xfext0NfnG3//74qGA7r+dyb7+r71Rks/\n+P9+q09fu0//z00f7Ot72wnfP/bG9bE3rs/6LB/zAAB0p9FsaWq+pP2T/b8BKBT0a3Ikpt+8k+37\newMALkSYBgALnJktqtU29YFdg7mbfm86rulsWQtLlYG8PwBgBWEaACxwajovSQPpTEvS3omVv6b8\n1dt0pwFgkAjTAGCBU9N5DcdXTiwchKF4SOlUVL/67cJA3h8AsKLve6YBAJs7NV3QwcyQDKM/Jx+u\n56NXjuqf/9c55Qo1hUP+vr53OBhQgHYMABCmAWCnlatNzeTK+sRH+rvF42If3j+in/9ySk/966m+\nj5P87qFJBcL8CgEA+goAsMPenVmZlz6YGRro1zm4e0jBgE/vzZcG+nUAwMsI0wCww06tnnx4YMBh\n2u/3afd4XO/NFeWQIwUAwHEI0wCww05N55VORZSI9vfkw/XsGY+rWm9pqVgf+NcCAC8iTAPADjs9\nnR/4iEfHrtGYJGkmV96RrwcAXkOYBoAdtFyqK5uv6cCunQnTiVhQiWhQs4RpABgIwjQA7KDOYS1X\n7N6ZMC1Jk6NRzeTKzE0DwAAQpgFgB52ezsswpA8M6OTD9ewajaneaGupWNuxrwkAXkGYBoAddGq6\noN3j8b4fonI5k5256Wxlx74mAHgFYRoAdohpmjo1ndfBHZqX7khEV+amuQkRAPqPMA0AOyS7XFWx\n0tDBzM6NeHTsGo1pdpG5aQDoN8I0AOyQs/NFSer70d7d2DUWVb3RVq7A3DQA9BNhGgB2yEx2Zcwi\nMxbb8a/dmZtmRR4A9BdhGgB2yHSurKF4SLHI4E8+vFg8ElQyFlwL9ACA/iBMA8AOmcmV104ktMLK\n3HRFbeamAaBvCNMAsENmstaG6cnRmBrNthbzzE0DQL8QpgFgBxQrDRUrDUvmpTs6QZ4VeQDQP4Rp\nANgBnVllKzvTsUhAQzH2TQNAPxGmAWAHTOdKkqRdFnamO19/brGidpu5aQDoB8I0AOyAmWxZAb+h\n8eGIpXV05qZzhaqldQCAWxCmAWAHzOTKmhiJye+z9sfu+3PTFUvrAAC3IEwDwA6wei1eRzQc0HA8\npFn2TQNAXxCmAWDAmq225hYrlm7yON/kaFRzSxWZ7JsGgG0jTAPAgC0sV9Vqm7boTEtSOhVVo9nW\nUrFudSkA4HiEaQAYMDusxTvfxEhUkjS/xNw0AGwXYRoABswua/E6EtGgIiG/5hcJ0wCwXYRpABiw\nmWxZQ7Gg4pGg1aVIkgzDUDq1MjcNANgewjQADJhdNnmcLz0SVaHcUKXWtLoUAHA0wjQADNhMrmyb\nEY+OidTK4THMTQPA9hCmAWCAipWGCuWGdo3GrS7lAmNDEfkMaX6JkxABYDsI0wAwQDO51U0eNutM\n+/0+jQ5F6EwDwDYRpgFggDpr8TI2m5mWVlbkdXZgAwB6Q5gGgAGazpXk9xkaX51RtpN0Kqp221Qu\nz6gHAPSKMA0AAzSTLWtiJCq/z34/btMpDm8BgO2y3093AHARO67F64hFAkpEg5rj8BYA6BlhGgAG\npNVua26xosyYvTZ5nC+dWrkJ0TSZmwaAXhCmAWBAFpZWbu6za2daWjm8pVJrqVTh8BYA6AVhGgAG\nZNqma/HON7E6N83R4gDQG8I0AAxIZy2enTvTqWRYAb/BTYgA0CPCNAAMyEyupGQsqEQ0aHUpG/IZ\nhtKpKGEaAHpEmAaAAZnJ2neTx/nSqagW8zU1mm2rSwEAxyFMA8CA2Hkt3vnSqahMSQvLdKcBYKsI\n0wAwAOVqU/lywyFheuV0RvZNA8DWEaYBYAA6Xd7OKYN2Fgr6NZwIaWGZY8UBYKsI0wAwAPNLK8F0\nfLXra3fjQxFll6sc3gIAW0SYBoAB6HSmx4ft35mWpLHhiKr1lspVDm8BgK0gTAPAACwsVRUN+xWP\nBKwupSvjwysddEY9AGBrCNMAMADzyxWND0dlGIbVpXRlJBmWYUhZwjQAbAlhGgAGYGG5utbtdQK/\n36eRZFjZPGEaALaCMA0AfWaaphaWK47Y5HG+8WFuQgSArSJMA0Cf5csN1RttR3WmJWlsKKJ6s61C\nuWF1KQDgGIRpAOizhaXVTR4O60yPrYZ/5qYBoHuEaQDos/nOgS0O60ynEmH5fQYbPQBgCwjTANBn\nC50DWxyyY7rD5zM0OsRNiACwFYRpAOizheWKhmJBhUN+q0vZsrHhiHL5qtrchAgAXSFMA0CfzS9V\nHTcv3TE+HFGzZWq5WLe6FABwBMI0APTZwnLFcZs8OsaGuAkRALaCMA0AfdRum8rla47bMd0xFA8p\nGPBxEyIAdIkwDQB9lCtU1Wqbju1MG4ahsaEINyECQJcI0wDQR2ubPBzamZakseGwFvM1tdrchAgA\nmyFMA0AfOXXH9PnGhqNqm6YWCzWrSwEA2yNMA0AfLSxVZRjS6JBzw/Q4NyECQNcI0wDQRwvLFY0m\nwwr4nfvjNR4NKBz0E6YBoAvO/WkPADY0v1x13MmHFzMMQ2PDES2sjqwAADbWVZg+ceKEPvWpT+mq\nq67Sm2++ue5rHn30UX3iE5/Q0aNHdfToUT300ENrz1UqFX3lK1/RzTffrMOHD+vnP/95f6oHAJtZ\nWKpoPOXcEY+O8eGIlot1NVttq0sBAFsLdPOim266SXfeeafuuOOOy77u1ltv1QMPPHDJ49/97neV\nSCT0/PPP6/Tp07rjjjv03HPPKR6P91Y1ANhQo9nSUrGutMM709LKseKmpFy+qomRmNXlAIBtddWZ\nvvbaa5XJZHr+Ij/72c902223SZIOHDigj370o/rFL37R8/sBgB11DjpxQ2f6/ZMQ2egBAJfT15np\np59+WkeOHNHdd9+tl156ae3xc+fOac+ePWt/zmQympmZ6eeXBgDLrYVpF3Smo2G/IiE/6/EAYBNd\njXl04/bbb9e9996rYDCoF198Uffdd5+eeeYZjYyM9OX9x8YSfXkfL0qnk1aXgMvg+tjbVq5P9a0F\nSdKHrxzXWJeB2syVlUwMppMdDAa29d7jqaiWy/V13yMWCys9av34B98/9sb1sTeuT3/0LUyn0+m1\nj2+44QZlMhm99dZbuu6667R7925NTU1pdHRUkjQ9Pa3rr79+S++fzRbV5jSuLUunk5qfL1hdBjbA\n9bG3rV6fU+8tKeD3qVlraH6+2dXnlGtNFYqDWUHXaGzvvYdiQb1+pqTlfEU+n3HBc+VyTfOt1nZL\n3Ba+f+yN62NvXJ/1+XzGlhu4fRvzmJ2dXfv4tdde09TUlA4ePChJOnz4sH7wgx9Ikk6fPq2XX35Z\nN954Y7++NADYwsJSRWPDEfkMY/MXO8BIMqx221S+XLe6FACwra460w8//LCee+45LSws6K677lIq\nldLTTz+te+65R/fff7+uvvpqPfLII3rllVfk8/kUDAZ18uTJtW71F7/4RR07dkw333yzfD6fvvnN\nbyqRYGwDgLvML1UdfYz4xUaSYUnSYr6mVCJscTUAYE9dhenjx4/r+PHjlzz++OOPr3184sSJDT8/\nFovpW9/6Vg/lAYBzLCxXdHD3kNVl9M1wIiyfIeUKNR20uhgAsClOQASAPihXmypVm67qTPt9hoYT\nYTZ6AMBlEKYBoA86R2+Pp5y/Fu98I8mwFguDuUESANyAMA0AfTC/1Nkx7Z7OtLQSpiu1lqr17raT\nAIDXEKYBoA86nem0CzvTkhj1AIANEKYBoA8WlqqKhv2KR/q2vt8WRofe3+gBALgUYRoA+mB+uaLx\n4agMl+yY7oiEAoqG/crRmQaAdRGmAaAPsstV181Ld4wkI4x5AMAGCNMAsE2maWohX9XYkFvDdFjL\nxZpabdPqUgDAdgjTALBNlVpTtXpLY67tTIfVNqV8ie40AFyMMA0A25RdvTnPrZ3p0dWNHjluQgSA\nS7jrtnMAGIBmW6o1Nt6zPLVQkiRFowGValvbx+yEyYmheEg+n8HcNACsgzANAJuoNZr6j9dmN3z+\n9XcXJUlnZgqaX6xs6b3/64fS26ptJ/h8hlKJEGEaANbBmAcAbFOp2pTPMBQJ+a0uZWBWjhWvyTQd\n0EoHgB1EmAaAbSpVGopHA67bMX2+0WRE1XpLlVrL6lIAwFYI0wCwTaVqQ/FI0OoyBopjxQFgfYRp\nANimUrWpeNTdt6C8H6arFlcCAPZCmAaAbWi3TVWqTdd3psMhv2KRAJ1pALgIYRoAtqFcbcqUXN+Z\nlt6/CREA8D7CNABsQ6nakCTXd6allcNblkt1tVptq0sBANsgTAPANngpTI8kwzJNaalUt7oUALAN\nwjQAbEOpsnLioRfGPFKJlZsQl4uEaQDoIEwDwDaUqg2Fg34F/O7/cZqMh2QY0nKRuWkA6HD/T38A\nGKBipamEB7rSkuT3GUpGg1pmzAMA1hCmAWAbStWG4lH3z0t3DCfCjHkAwHkI0wDQI9M0V44S98DN\nhx3DiZDy5bqabPQAAEmEaQDoWb3ZVrNlKh7xxpiHJA3HQzJNaX6pYnUpAGALhGkA6FGpsroWz0Nj\nHp2NHrO5ssWVAIA9EKYBoEel6upaPA91pofiIUnSDGEaACQRpgGgZ17sTAcDPsUjAc1kCdMAIBGm\nAaBnpWpDPp+hSMhvdSk7ajgRZswDAFYRpgGgR6VKU/FIQIZhWF3KjhqOhzSbq6jdNq0uBQAsR5gG\ngB6Vqt5ai9cxnAip0WprIV+1uhQAsBxhGgB6VKo0FffI6YfnSyVWbkKcXihZXAkAWI8wDQA9aLdN\nlWtNb3am4yvr8aa5CREACNMA0ItyZy2eBzvT4ZBfyVhQ5+hMAwBhGgB6UayursXzYGdaknaNxjSd\nJUwDAGEaAHrQ2TGd8NCO6fNNjsZ0LluWabLRA4C3EaYBoAed0w9jHjr98Hy7xmKq1JpaKtatLgUA\nLEWYBoAelCoNRUJ+Bfze/DG6azQmSYx6APA8b/4WAIBtKlVXDmzxqvfDNBs9AHgbYRoAelCqNhT3\n6Ly0JA3FQ4qGA2z0AOB5hGkA2CLTNFWqePP0ww7DMLR7jI0eAECYBoAtqjfaarZMT495SFJmLK5z\njHkA8DjCNABsUamzY9rDYx6StHs8rnypruLqmkAA8CLCNABsUcnDpx+eLzPGRg8AIEwDwBZ1Dmzx\n8sy0JGXG45LY6AHA2wjTALBFpWpDPp+hSMhvdSmWGh+KKBTwsdEDgKcRpgFgi0qVlR3ThmFYXYql\nfD5Du0ZjOseYBwAPI0wDwBaVqt5ei3e+zHhc0wuMeQDwLsI0AGxRqdpUzONr8ToyYzFl81XVGi2r\nSwEASxCmAWAL2m1TFY8fJX6+yZGVjR7zixWLKwEAaxCmAWALKrWmTLHJo2NyNCpJml1k1AOANxGm\nAWALOjumYx7fMd0xkVrpTM/RmQbgUYRpANiCtdMP6UxLkmKRgJKxIJ1pAJ5FmAaALSh3Tj9kZnrN\n5EhMszk60wC8iTANAFtQqjYU9PsUDPDjs2NyJEpnGoBn8dsAALagVGkqFuXAlvNNjMa0VKyrVmc9\nHgDvIUwDwBaUqw1GPC4yObKy0WNuiVEPAN5DmAaALVg5sIWbD8/X2TU9m2PUA4D3EKYBoEutdlvV\neovO9EUmRtg1DcC7CNMA0KX3N3nQmT5fNBzQUDykWXZNA/AgwjQAdKlUWT2whc70JSZHoppjzAOA\nBxGmAaBLHNiyscmRmGa5ARGABxGmAaBLa2MeHCV+iYmRqJaLdVXrTatLAYAdRZgGgC6Vqg2Fg34F\n/PzovNjk6MpGjznmpgF4DL8RAKBLK2vx6EqvZ3JtowdhGoC3EKYBoEulCge2bKSzHm+O9XgAPIYw\nDQBdKlebike5+XA9kVBAw4mQZnN0pgF4C2EaALrQaLZVb7YZ87iMyVSUg1sAeE5XvxVOnDihZ599\nVlNTU3ryySf1oQ996JLXPPbYY3rmmWfk8/kUDAb11a9+VTfeeKMk6dixY/rXf/1XjYyMSJIOHz6s\nL3/5y338xwCAwWIt3uYmRmP69dtZq8sAgB3VVZi+6aabdOedd+qOO+7Y8DXXXHON7r77bkWjUb3+\n+uv6/Oc/r3/5l39RJBKRJP3xH/+xPv/5z/enagDYYe+ffkhneiOTI1HlS3VVak1Fw/x7AuANXY15\nXHvttcpkMpd9zY033qhodOUGlKuuukqmaWppaWn7FQKADXQ604x5bGxyhPV4ALxnIDPTP/nJT7R/\n/37t2rVr7bG///u/15EjR3Tffffp7bffHsSXBYCBef8occY8NtLZNc3cNAAv6XuL5d///d/1V3/1\nV/q7v/u7tce++tWvKp1Oy+fz6Sc/+Ym+9KUv6YUXXpDf7+/6fcfGEv0u1TPS6aTVJeAyuD72lk4n\nZebKarRMxSIBpYaifX3/YDCgZCLS1/fcifeOxcJKr4bnjuTqv5tivbVj/7/m+8feuD72xvXpj76G\n6Zdeekl/9md/pm9/+9u64oor1h6fnJxc+/jWW2/VX/zFX2hmZkZ79uzp+r2z2aLabbOf5XpCOp3U\n/HzB6jKwAa6PvXWuT7nW1FKhqlg4oEKx2tev0Wg0+/6eO/He5XJN863WJY+nEiGdOru0I/+/5vvH\n3rg+9sb1WZ/PZ2y5gdu3MY9f//rX+upXv6pvfetb+shHPnLBc7Ozs2sf//M//7N8Pt8FARsA7I7T\nD7szORLjFEQAntLVb4aHH35Yzz33nBYWFnTXXXcplUrp6aef1j333KP7779fV199tR566CFVq1V9\n/etfX/u8kydP6qqrrtIDDzygbDYrwzCUSCT013/91woE+KUEwBlM01S52tCe8bjVpdje5GhU/+ut\nBavLAIAd01WiPX78uI4fP37J448//vjaxz/60Y82/Px/+Id/2HplAGAT5VpTzZbJWrwuTI7ElC83\nWI8HwDM4AREANrFYqEmSYhwlvqmJETZ6APAWwjQAbGJpNUzTmd7c5MjKRo/ZHHPTALyBMA0Am1gk\nTHct3QnTdKYBeARhGgA2sVioyTCkCDPAmwoH/RpJhjkFEYBnEKYBYBNLhZpi4YB8hmF1KY4wORKl\nMw3AMwjTALCJxUJNcW4+7NrESFTzS4M5LAYA7IYwDQCbWCzUOLBlC8aHo8qX6qrVLz0hEQDchjAN\nAJfRNk0tFWuKR+hMdyudWrkJcWGZuWkA7keYBoDLKJTqarU5sGUrOmGaUQ8AXkCYBoDLyHUObCFM\ndy2dikiS5pfoTANwP8I0AFxGLr/SXWXMo3uJaFCRkJ8wDcATCNMAcBm5/OqBLVE6090yDEPjw1HC\nNABPIEwDwGXkClUF/T6Fg36rS3GUdCqihWVmpgG4H2EaAC4jl68plQzL4MCWLUmnVjrTpmlaXQoA\nDBRhGgAuI5evaiQZtroMx0nJt8vAAAAgAElEQVSnoqo328qX6laXAgADRZgGgMvIFVY609ga1uMB\n8ArCNABsoNVqa6lYozPdA9bjAfAKwjQAbCCbr8o0pZEEYXqrxocJ0wC8gTANABtYWA2CjHlsXTDg\n10gyrHmOFAfgcoRpANhAJ0wz5tGb9HCEmWkArkeYBoANEKa3p7MeDwDcjDANABuYX6ooEvIrGub0\nw16kU1EtFWpqNFtWlwIAA0OYBoANLCxVNDYUsboMxxpPRWRKnIQIwNUI0wCwgfmlikaGGPHoFbum\nAXgBYRoANrCwVNFoks50rzpheoGNHgBcjDANAOtoNFtaLtY1Sme6Z8PxkIIBHzchAnA1wjQArCNX\nqEkSneltMAxjdaMHYx4A3IswDQDryOVXwzSd6W0ZH47QmQbgaoRpAFhHLr/STR1lm8e2dHZNm6Zp\ndSkAMBCEaQBYR2fMgwNbtiediqpab6lYaVhdCgAMBGEaANaxmK8qGQspHPRbXYqjpVMrnX12TQNw\nK8I0AKwjV6gpPRK1ugzHe3/XNHPTANyJMA0A68jmq2tBEL0bH17pTBOmAbgVYRoA1pHL1zROmN62\nSCigoViQMA3AtQjTAHCRSq2pSq1JmO4Tdk0DcDPCNABcpLPJgzDdH531eADgRoRpALjI4uqOaWam\n+2M8FVUuX1Oz1ba6FADoO8I0AFyEznR/pVMRtU1z7d8rALgJYRoALpLLV2VIGhvm9MN+SA+zHg+A\nexGmAeAiuXxNQ4mQAn5+RPYDu6YBuBm/KQDgIrlCVWNDdKX7ZSQZlt9nEKYBuBJhGgAuks3XNJoM\nW12Ga/h8hsaGI1pgPR4AFyJMA8B5TNPUYr6qUTrTfZUejmhhmc40APchTAPAeUrVpurNNp3pPhsb\njmphmc40APchTAPAeXKrO6bpTPdXOhVRodxQtd60uhQA6KuA1QUAgJ3k8iu7kEeG6ExfjuEzVKp1\nH4yT8ZAk6ex8SbvH45d9bTgYUIBWDwCHIEwDwHlyhdXOdJLO9OXUGi396s35rl/f2eTxP1+Z0b6J\nxGVf+7uHJhUI8+sJgDPw3/4AcJ5cvia/z9DwaicV/ZGIBiVJxUrD4koAoL8I0wBwnlyhqlQiLJ/P\nsLoUV4mE/Ar4DRXLhGkA7kKYBoDz5PI1jTEv3XeGYSgeDdKZBuA6hGkAOE+OHdMDkyBMA3AhwjQA\nrGqbphYLNTZ5DAhhGoAbEaYBYFW+VFerbbLJY0AS0aAazbZqjZbVpQBA3xCmAWBVZ8f0KJ3pgVjb\n6MFNiABchDANAKs6px+OMTM9EIkY6/EAuA9hGgBWcZT4YLFrGoAbEaYBYFU2X1M46Fc8wul7gxAO\n+hUM+AjTAFyFMA0Aq1bW4oVlGBzYMihs9ADgNoRpAFiVzVeZlx4wwjQAtyFMA8AqDmwZvEQ0qFKl\nIdM0rS4FAPqCMA0AkuqNlvLlBkeJD1giFlSzZapaZ9c0AHcgTAOApMVCZ8c0nelBSrLRA4DLEKYB\nQCvz0hJhetDihGkALkOYBgC9H6YZ8xgsTkEE4DaEaQDQ+0eJjyTpTA9SMOBTJOSnMw3ANQjTAKCV\nTR7D8ZCCAX4sDhrr8QC4Cb81AECsxdtJccI0ABchTAOAVo4SZ156Z3R2TbfZNQ3ABQjTADzPNE06\n0zsoGQ2qbUqVatPqUgBg2zYN0ydOnNCnPvUpXXXVVXrzzTfXfU2r1dJDDz2kT3/607r55pv1xBNP\ndPUcANhBsdJQvdnmKPEdkoixHg+AewQ2e8FNN92kO++8U3fccceGr3nyySd15swZPffcc1paWtKt\nt96qT3ziE9q7d+9lnwMAO+hs8qAzvTMS5+2anrS4FgDYrk0709dee60ymcxlX/PMM8/oD//wD+Xz\n+TQ6OqpPf/rT+qd/+qdNnwMAO1jbMT3MzPROiEdX+jh0pgG4QV9mpqenp7V79+61P2cyGc3MzGz6\nHADYAacf7iy/z6dYOMDBLQBcYdMxD7sYG0tYXYJjpdNJq0vAZXB9rFdptBUK+HTF/lEZhnHBc+l0\nUmaurGRiMEE7GAx48r2HE2FV6q113yMWCys9Guvqffj+sTeuj71xffqjL2E6k8no3LlzuuaaayRd\n2I2+3HNbkc0W1W6zRmmr0umk5ucLVpeBDXB97OG92YJGkmEtLBQveLxzfcq1pgrF6kC+dqPhzfeO\nhv2azZXXfY9yuab5VmvT9+D7x964PvbG9Vmfz2dsuYHblzGPw4cP64knnlC73VYul9MLL7ygz3zm\nM5s+BwB2wFq8nZeIBlWuNmmSAHC8TcP0ww8/rN/7vd/TzMyM7rrrLv3BH/yBJOmee+7Ryy+/LEk6\nevSo9u7dq9///d/XH/3RH+lP/uRPtG/fvk2fAwA7yOarrMXbYYloUKakUpW5aQDOtumYx/Hjx3X8\n+PFLHn/88cfXPvb7/XrooYfW/fzLPQcAVmu22soX6xrl9MMddf56vGQsZHE1ANA7TkAE4GmLhZpM\nic70DlsL02z0AOBwhGkAnpbrrMUbJkzvpFgkIMNg1zQA5yNMA/C0tQNb6EzvKJ/PUDwSJEwDcDzC\nNABPy3aOEk8yM73TElHCNADnI0wD8LRcvqpkLKhQ0G91KZ5DmAbgBoRpAJ6WZce0ZRKxoCq1lpqt\nttWlAEDPCNMAPC2XrzEvbZHORo8S3WkADkaYBuBZpmmudKaZl7ZEIrpy1AGjHgCcjDANwLPKtaZq\n9RZjHhZJRFcOaykQpgE4GGEagGdll1fX4rFj2hLRsF8+n8GYBwBHI0wD8KxcYXUtHkeJW8IwjJWN\nHpyCCMDBCNMAPCvHgS2WS0QDzEwDcDTCNADPyuar8vsMDcVDVpfiWYlokJlpAI5GmAbgWbl8TaND\nYfkMw+pSPCsRDareaKvebFldCgD0hDANwLOy+SojHhZLxFb+VoCbEAE4FWEagGflOP3Qcp1d0wVu\nQgTgUIRpAJ7Uare1WKixycNi75+C2LS4EgDoDWEagCct5msyTTZ5WC0c9CvgN9joAcCxCNMAPCm7\nuhZvfDhqcSXe1tk1zUYPAE5FmAbgSQucfmgbKwe31K0uAwB6QpgG4ElrR4kzM225RCyoUqUp0zSt\nLgUAtowwDcCTFvJVDcdDCgb8VpfieYloUI1WW7VG2+pSAGDLCNMAPCm7XGXEwyY6Gz24CRGAExGm\nAXhSNl/VOGHaFgjTAJwsYHUBANAPzbZUa3S3q7htmsrlq7rmyjGVaht/jpkrq1xrqs0o70AlYoRp\nAM5FmAbgCrVGU//x2mxXry1Xm2q2TOXL9ct+TjIRUaFY1X/9ULpfZWIdoYBfoaBPRU5BBOBAjHkA\n8JzSage0M14A6yWiQTrTAByJMA3AczqhLREhTNtFIhpc+48cAHASwjQAzylWV0JbnM60bXQ60+ya\nBuA0hGkAnlOqNBQO+hUM8CPQLhKxoFptU5Vay+pSAGBL+E0CwHOKlabiUe6/thPW4wFwKsI0AM8p\nVRrcfGgzhGkATkWYBuAppmmqVG0ozs2HtkKYBuBUhGkAnlJrtNRsmXSmbSbg9yka9hOmATgOYRqA\npxQrKyceMjNtP/FIkINbADgOYRqAp3Bgi30lYhzcAsB5CNMAPKUTptkxbT/JaFClakPtNrumATgH\nYRqApxQrDQX9PoXYMW07iWhQpiktFWtWlwIAXeO3CQBPKVZXdkwbhmF1KbhIIrbytwULy1WLKwGA\n7hGmAXgKO6btKxkNSZIWlioWVwIA3SNMA/CUYqXBvLRNxaIB+Qw60wCchTANwDPqjZYazTadaZvy\nGYbi0SCdaQCOQpgG4BmlKps87C4ZC9GZBuAohGkAntE5sCUR4cAWu0rGglpYrsg0WY8HwBkI0wA8\no8iOadtLxoKq1FoqVZtWlwIAXSFMA/CMUqUhv89QJOS3uhRsIBlb2egxt8jcNABnIEwD8IzOWjx2\nTNtXcvVvDeaWyhZXAgDdIUwD8IxiZeXAFthX5+CWeTrTAByCMA3AM0rVhuIR5qXtLOD3aTgeYswD\ngGMQpgF4QrPVVrXeYse0A4ynIppj1zQAhyBMA/AENnk4x/hwlDANwDEI0wA8odTZMc3MtO2lU1Et\nF+uqNVpWlwIAmyJMA/CE0mpnmjEP+xtPRSRJ83SnATgAYRqAJxQrDfkMKRqmM21348NRSWz0AOAM\nhGkAnlCsNhSLsGPaCTqdaeamATgBYRqAJ3QObIH9xSNBxcIBwjQARyBMA/CEYqWxdiAI7C89EmXM\nA4AjEKYBuF6z1Valxo5pJ5lIsR4PgDMQpgG4XmfHdJIw7RgTI1Fll6tqtdtWlwIAl0WYBuB6xfLq\nWjzGPBwjnYqq1TaVy9esLgUALoswDcD1iuyYdpyJ1Mp6PEY9ANgdYRqA6xXKDQX8hiIhv9WloEsT\nI+yaBuAMhGkArldcXYvHjmnnSCXDCvh9dKYB2B5hGoDrFdkx7Tg+w1A6FaEzDcD2CNMAXM00TRXL\n7Jh2ojTr8QA4AGEagKvVGi01Wm0loyGrS8EWdXZNm6ZpdSkAsCHCNABXW9vkQWfacdIjUdXqLRVW\nVxsCgB0RpgG4WieIMTPtPKzHA+AEhGkArsaOaediPR4AJwh086JTp07p2LFjWlpaUiqV0okTJ3Tg\nwIELXvO1r31Nb7zxxtqf33jjDT322GO66aab9Oijj+r73/++JiYmJEkf//jH9eCDD/bvnwIANlAs\nNxQJ+RUM0DtwmvHhqAzRmQZgb12F6QcffFCf+9zndPToUf30pz/V17/+dX3ve9+74DUnT55c+/j1\n11/XF77wBd14441rj91666164IEH+lQ2AHSHtXjOFQz4NDIU1hydaQA2tmmrJpvN6tVXX9Utt9wi\nSbrlllv06quvKpfLbfg5P/zhD3XkyBGFQtw9D8BaBdbiOdpEKqp5OtMAbGzTMD09Pa3JyUn5/SvH\n8Pr9fk1MTGh6enrd19frdT355JP67Gc/e8HjTz/9tI4cOaK7775bL730Uh9KB4DLa7dNlaoNJelM\nOxa7pgHYXVdjHlvxwgsvaPfu3Tp06NDaY7fffrvuvfdeBYNBvfjii7rvvvv0zDPPaGRkpOv3HRtL\n9LtUz0ink1aXgMvg+vSHmSsrmYhc8Fi+VJdpSuMjsUue61YyEVEwGOj58zfDe18qFgsrPRqTJF2x\nb0T//OtpxZMRxSKX/kcR3z/2xvWxN65Pf2wapjOZjGZnZ9VqteT3+9VqtTQ3N6dMJrPu63/0ox9d\n0pVOp9NrH99www3KZDJ66623dN1113VdaDZbVLvN4v6tSqeTmp8vWF0GNsD16Z9yralCsXrBY7PZ\nsiQp4NMlz3UjmYioUKyq0bj0vfuF975UuVzTfKslSUqGV/5W9DdvzulgZuiC1/H9Y29cH3vj+qzP\n5zO23MDddMxjbGxMhw4d0lNPPSVJeuqpp3To0CGNjo5e8tqZmRn98pe/1JEjRy54fHZ2du3j1157\nTVNTUzp48OCWCgWArSpU6pLE6YcOtmu1Qz2z+h9GAGA3XY15fOMb39CxY8f07W9/W0NDQzpx4oQk\n6Z577tH999+vq6++WpL04x//WJ/85Cc1PDx8wec/8sgjeuWVV+Tz+RQMBnXy5MkLutUAMAjFckOG\nIcUifZ9oww6ZGInKMKTpHGEagD119Rvmyiuv1BNPPHHJ448//vgFf/7yl7+87ud3wjcA7KRCpaF4\nJCifz7C6FPQo4PcpPRzVDGEagE1xigEA1yqyFs8Vdo3FGPMAYFuEaQCuVaywFs8Ndo3GNLdYVtvk\nJnQA9kOYBuBKjWZb1XqL0w9dYNdoTPVmW7n8YLaHAMB2EKYBuFKx0pAkxjxcYG2jB3PTAGyIMA3A\nlTphmjEP59s1xno8APZFmAbgSsUynWm3GI6HFAn56UwDsCXCNABXKlTqCvgNhYN+q0vBNhmGoV2j\nMc0SpgHYEGEagCsVyw0lYyEZBjum3WDXWIzONABbIkwDcKVipcEmDxfZNRpTNl9TrdGyuhQAuABh\nGoDrmKZJmHaZzkYPRj0A2A1hGoDrVOstNVsmNx+6COvxANgVYRqA67AWz30mCdMAbIowDcB1CqzF\nc51w0K+xoTBhGoDtEKYBuM7a6Yd0pl1l12iMg1sA2A5hGoDrFMp1RcN+Bfz8iHOTXaNxzeTKMk3T\n6lIAYA2/aQC4TmF1xzTcZddYTNV6S8ulutWlAMAawjQA1ymU6xoiTLvO2kYPRj0A2AhhGoCrNJpt\nVWotJePMS7sN6/EA2BFhGoCr5MsrIwB0pt1nZCisUMBHmAZgK4RpAK7SWYuXZC2e6/gMQ5OjMcI0\nAFshTANwlcLqzWncgOhOk6zHA2AzhGkArpJfXYsXDPDjzY12jcY0v1xRo9m2uhQAkESYBuAyhXKD\neWkXy4zGZJrS3FLF6lIAQBJhGoDL5Et1JeOEabfaNcZ6PAD2QpgG4Br1ZkvVeoubD13s/fV4JYsr\nAYAVhGkArlEorWzyYMzDvaLhgIbjITZ6ALANwjQA1yh0dkxzYIurZcZimmbMA4BNEKYBuEZ+dcd0\nIkpn2s32jCc0tVBSu21aXQoAEKYBuEehVFc0HGAtnsvtmYirVm9pbpHuNADr8RsHgGvky3UNcfOh\n6+1NJyRJ707nLa4EAAjTAFykUG6wFs8D9ozHJUmnZwjTAKxHmAbgCpVaU9V6i860B0TDAY0PR/Tu\ndMHqUgCAMA3AHeZXT8RLshbPE/amEzrNmAcAGyBMA3CFTpgeYszDE/ak45qaL6rRbFtdCgCPI0wD\ncIX3O9OMeXjB3nRC7bap6SwnIQKwFmEagCvML1YUCwcU8PNjzQv2plduQpyaJ0wDsBa/dQC4wvxS\nVUlOPvSMydGYAn5D780XrS4FgMcRpgG4wvxSRUPcfOgZAb9PeyeSeo/ONACLEaYBOF652lSx0mBe\n2mMOZIboTAOwHGEagOPNrh4rzSYPb/lAZkiLhZpK1YbVpQDwMMI0AMfrhGl2THvLgcyQJG5CBGAt\nwjQAx5tbZC2eF31g10qYZtQDgJUI0wAcbzZXUSoRYi2ex4ynIoqGA9yECMBS/OYB4Hhzi2WlR6JW\nl4EdZhiG9qTjdKYBWIowDcDxZhcrmkgRpr1obzqhqfmSTNO0uhQAHkWYBuBopWpDxUpD44RpT9qb\njqtSayqXr1ldCgCPIkwDcLTOzYdpwrQn7U0nJHETIgDrEKYBONpsbmUtHmHam/ak45II0wCsQ5gG\n4GjT2bIMgzDtVfFIUCPJMLumAViGMA3A0c5lS5pIRRUM8OPMq/amE3SmAViG3z4AHG06W1ZmLG51\nGbDQ3nRc09mymq221aUA8CDCNADHarbams2VlRmPWV0KLLQ3nVCrbWpmdX4eAHYSYRqAY80vVdRq\nm9pNZ9rTuAkRgJUI0wAc69zCSidy9zhh2ssyY3H5fYbem+MmRAA7jzANwLGmsyvhadcoYx5eFgz4\ntHs8rndn8laXAsCDCNMAHOtctqTRobCi4YDVpcBiBzNJnZ4pcKw4gB1HmAbgWNMLbPLAigOZIZWq\nTc0tVawuBYDHEKYBOFLbNDWdK3HzISRJB3cNSZJOTTPqAWBnEaYBOFJuuap6o81aPEha2egRDPh0\nerpgdSkAPIYwDcCRzmVXN3nQmYakgN+n/ZMJOtMAdhxhGoAjdTZ5sBYPHQd3Dend2YJabU5CBLBz\nCNMAHOncQknJWFCJaNDqUmATBzNDqjfaml7gJEQAO4cwDcCRprNs8sCFDmSSkrgJEcDOIkwDcBzT\nNDWdLTHigQtMjsYUDft1aoabEAHsHMI0AMfJl+oqVZvKjLHJA+/zGYYO7BqiMw1gRxGmATjO2iYP\nOtO4yIFMUu/NFdVochMigJ1BmAbgOGubPJiZxkUO7hpSq23q7FzR6lIAeARhGoDjnFsoKRLyK5UI\nWV0KbOZghpMQAewswjQAx5nOlrV7PC7DMKwuBTYzOhTWUCyo04RpADukqzB96tQp3XbbbfrMZz6j\n2267TadPn77kNY8++qg+8YlP6OjRozp69KgeeuihtecqlYq+8pWv6Oabb9bhw4f185//vG//AAC8\n51y2xM2HWJdhGDqQGWKjB4AdE+jmRQ8++KA+97nP6ejRo/rpT3+qr3/96/re9753yetuvfVWPfDA\nA5c8/t3vfleJRELPP/+8Tp8+rTvuuEPPPfec4nHmHQFsTbna0HKxzrw0NnQwM6SX386qUmsqGu7q\n1xwA9GzTznQ2m9Wrr76qW265RZJ0yy236NVXX1Uul+v6i/zsZz/TbbfdJkk6cOCAPvrRj+oXv/hF\njyUD8LLOJo8MmzywgYOZpExJZ2bpTgMYvE3D9PT0tCYnJ+X3+yVJfr9fExMTmp6evuS1Tz/9tI4c\nOaK7775bL7300trj586d0549e9b+nMlkNDMz04/6AXjM9EJnkwdjHljfgbWbEAnTAAavb3//dfvt\nt+vee+9VMBjUiy++qPvuu0/PPPOMRkZG+vL+Y2OJvryPF6XTSatLwGVwfbZmsdxQMODTh//LhPy+\n929ANHNlJRORvn+9ZCKiYDAwkPeWxHuvIxYLKz3a3X8srff9k5Y0MRLV9GKF7y+L8e/f3rg+/bFp\nmM5kMpqdnVWr1ZLf71er1dLc3JwymcwFr0un02sf33DDDcpkMnrrrbd03XXXaffu3ZqamtLo6Kik\nlW739ddfv6VCs9mi2m1zS5+DlW+U+Xm6M3bF9dm6d95b0q7RmHLZC/cIl2tNFYrVvn6tZCKiQrGq\nRqP/793Be1+qXK5pvtXa9HWX+/7ZP5HQ66ezfH9ZiJ9v9sb1WZ/PZ2y5gbvpmMfY2JgOHTqkp556\nSpL01FNP6dChQ2vBuGN2dnbt49dee01TU1M6ePCgJOnw4cP6wQ9+IEk6ffq0Xn75Zd14441bKhQA\npJUd02zywGYOZoY0v1RVsdKwuhQALtfVmMc3vvENHTt2TN/+9rc1NDSkEydOSJLuuece3X///br6\n6qv1yCOP6JVXXpHP51MwGNTJkyfXutVf/OIXdezYMd18883y+Xz65je/qUSCsQ0AW1NrtJRdrur/\nvCaz+YvhaQfOO7zl6ivGLK4GgJt1FaavvPJKPfHEE5c8/vjjj6993AnY64nFYvrWt77VQ3kA8L7p\nbEmmOEYcmzuYScpnGHrz7BJhGsBAcQIiAMc4O7syJ71vkr/ZwuVFQgEdyCT15tklq0sB4HKEaQCO\ncWauqHDIr3QqanUpcICr9qV0ajqvemPzmxkBoFeEaQCOcXa2oH3phHyGsfmL4Xkf2pdSs2Xq7XN5\nq0sB4GKEaQCOYJqmzs4XGfFA1z64NyXDkN44s2h1KQBcjDANwBEWlquq1FraP0GYRndikYD2TzA3\nDWCwCNMAHOHM6s2H+yc5sQvdu2p/Sm+fy6vRbFtdCgCXIkwDcISzcwUZhrRnnLV46N5V+1JqNNs6\nNc3cNIDBIEwDcIQzs0XtGo0pFPRbXQoc5IP7UjLE3DSAwSFMA3CEs3MFRjywZYloUHvSCb3B3DSA\nAenqBEQAsFKp2lA2X9OnuPnQEwyfoVKtuenrzFxZ5S5ed+WeIf2P38woX64rFgkpQBsJQB8RpgHY\nHicfekut0dKv3pzf9HXJRESFYrWr96w323r2387o//5vBxQI86sPQP/w3+cAbO/M3GqYnmDMA1s3\nObpyYubMYtniSgC4EWEagO2dnS1oOB7ScDxkdSlwoEgooOF4SLO5itWlAHAhwjQA2zszx8mH2J7J\n0ajmFytqtU2rSwHgMoRpALbWbLV1bqGk/Yx4YBsmR2NqtNqaWh0ZAoB+IUwDsLVzCyW12qb205nG\nNkyOxCRJb00tW1wJALchTAOwtbNrNx8SptG7WCSgZCyo377HvmkA/UWYBmBrZ2aLCgV9a51FoFeT\nozG9PZVXm7lpAH1EmAZga2fnCtqbTsjnM6wuBQ63azSmSq2pd2cLVpcCwEUI0wBsyzRNnZktaj8j\nHuiD3eMxGZJefjtrdSkAXIQwDcC2svmqyrWm9k2yyQPbFwkF9IFdSf36HcI0gP4hTAOwrc7Nh3Sm\n0S8fOTiqU+fyypfrVpcCwCUI0wBs6+xsUYakvWnCNPrjdw6OypT0G7rTAPqEMA3Ats7MFTUxGlM4\n5Le6FLjE3omEhuIh/Zq5aQB9QpgGYFtnZguMeKCvfIaha64Y02/eyanVbltdDgAXIEwDsKVCua6F\n5ao+sIubD9Ff11w5pnKtqben8laXAsAFCNMAbOmdcytB58rdQxZXArf5nQOj8vsMvczcNIA+IEwD\nsKW3zy3LZxg6sIswjf6KRQL64N5h/eq3hGkA20eYBmBLb0/ltW8iwc2HGIirrxzTe/NF5fJVq0sB\n4HCEaQC2026bemc6ryv30JXGYFxzxZgkMeoBYNsI0wBsZ2qhpFq9pSt3D1tdClxq93hcY0MRVuQB\n2DbCNADbefvcsiTRmcbAGIaha64c06unF9VosiIPQO8I0wBs5+2pZSWiQaVTUatLgYtdfeWYao2W\n3nxvyepSADgYYRqA7bxzLq//smdYhmFYXQpc7NAHRhTw+/RrtnoA2AbCNABbKVYams6WdQX7pTFg\n4aBfH/5ASr96e0GmaVpdDgCHIkwDsJW1w1r2cPMhBu/jH0xrbrGis3NFq0sB4FCEaQC28s65ZRmG\ndDDDMeIYvP/9qrR8hqF/f23O6lIAOFTA6gIA4HxvTy1rbzqhSIgfT+g/w2eoVGuu/dnn9+mq/Sn9\n26uzOvx/7N/WnH44GFCAFhXgOfy2AmAbbXPlsJbrf2eX1aXApWqNln715vwFj40kw3rt3UX90/98\nV+Pb2CDzu4cmFQjzaxXwGv4bGoBtTC+UVKm1/v/27jy8rfLOF/j3HC2WZcm2bGvzviR2nHjJSlYT\nkgJJh9CkTANtgU6fXsIDDM2dtIWklxamQO9t2nk6bZkULrRl67SluYUUkkBCCBAnIZB98ZbE+yLL\ni+RV1n7uHw4G1wlxHIUHxxMAACAASURBVNtHsr6ff2xZR6+/1vHR+enV+74HOZx8SJMo3ayDKAio\na+2VOwoRhSEW00QUMqo5+ZBkoFYpkGyMQZ2tl6t6ENE1YzFNRCGjurkbMRolzAZerIUmV6ZFD5fH\njzbngNxRiCjMsJgmopBR3dKDHF6shWSQZtJBIXKoBxFdOxbTRBQSXG4fWjr6OV6aZKFSikg16VDf\n2otgkEM9iGj0WEwTUUiosQ2Ol87meGmSSaZFD7c3ALvTJXcUIgojLKaJKCTUNPdAAJBtZc80ySPF\nGAOlQkCtjUM9iGj0WEwTUUi42NyNZGMMorlOL8lEqRCRZtKhwd6LAId6ENEosZgmItn5A0Gcb+pC\nXlq83FEowmVZY+H1BWHr7Jc7ChGFCRbTRCS76uZueH1BzMxMkDsKRThrUgzUShG1l9Y8JyK6GhbT\nRCS78jonBAGYkc6eaZKXQhSQaY1Fg70PHm9A7jhEFAZYTBOR7MrrHciyxkKrUckdhQi5aXEIBCVU\nt3TLHYWIwgCLaSKSlcvtR21LL2ZmGuSOQgQASIjVwBivwfnGbl5enIiuisU0EcmqqtGJoCRhZgbH\nS1PoyE2LR0+/F3YHLy9ORF+MxTQRyaqizgm1UkQOL9ZCISTDoodaJaKqsUvuKEQU4lhME5Gsyuud\nyE2Lh0rJlyMKHUqFiJzkODTYezHg8csdh4hCGM9eRCQbZ68HLR39XBKPQlJuWjwkCbjYxImIRHRl\nLKaJSDYV9Q4A4ORDCklxOjUsCVpcaOpGkBMRiegKWEwTkWzK65zQRauQatLJHYXosnLT4tA34IOt\ng1dEJKLLYzFNRLKQJAnldQ7MzDRAFAS54xBdVppZD41agapGDvUgostjMU1EsrB1utDV5+V4aQpp\nClHAtNQ4NLf1oX/AJ3ccIgpBLKaJSBbldZfGS2dwvDSFtumpcZAAVDVwmTwiGonFNBHJorzOCVN8\nNJLio+WOQvSF9Fo1Mix6VDV0weMNyB2HiEIMi2kimnSBYBCVDU6u4kFhoygnEb5AEBX1TrmjEFGI\nYTFNRJOu1tYLtzfA8dIUNgz6KKSbdaiod8LrY+80EX2GxTQRTbqyWgcEADM4XprCSFFOInz+ICrZ\nO01En8Nimogm3cnz7chJiYMuWiV3FKJRS4jVINWkQ3m9E14/e6eJaJBS7gBEFDn8QaC5vRcNbX34\n6o3Z6Pf4x63tIC9QR5OgKCcRuz/qQ1V9FwpzEofdJ4jCuP5Pf16USgklu7+IQhKLaSKaNB6fH28d\nrgMACACOVtjHre3iXOO4tUV0JUlxGqQYY1Be58SMDANUn6twPb4ATp9vn5DfuyDfDGUUT9lEoYjv\nc4loUjW09iIxNgo6LYd4UHgqykmExxdAVSPXnSaiUfZM19bWYsuWLejq6kJ8fDy2bt2KzMzMYdts\n27YNu3fvhiiKUKlU2LRpE0pKSgAAW7ZsweHDh2EwDE42Wr16NR588MHx/UuIKOQ5ez3o6HZjzvQk\nuaMQjZkxPhrWRC3Kax2YkR4PpYL9UkSRbFTF9BNPPIFvfvObWLt2Lf7+97/j8ccfxyuvvDJsm6Ki\nInznO99BdHQ0Kisrcc899+DgwYPQaDQAgPvvvx/33HPP+P8FRBQ2Tl/sAABkWPQyJyG6PsXTEvHO\nx404V+PAbL45JIpoV3073dnZifLycqxZswYAsGbNGpSXl8PhcAzbrqSkBNHRg1cyy8vLgyRJ6Ori\nR2BE9JnTFzoQr1MjNkYtdxSi62IyaJFp0eNcrQO9Lq/ccYhIRlftmbbZbDCbzVAoFAAAhUIBk8kE\nm82GhITLX3Bhx44dSE9Ph8ViGfrZiy++iNdeew1paWn4/ve/j5ycnGsKmpiou6bt6TNGI3sBQ1mk\n7B9nrxvVzd2Yn2+GXqcZ9/ZVKuWEtKvXaSasbWDickdK29eaYTxz3zQvDf+9pxLHz3dgzdKsCX1O\ntNooGBO0E9L2RIqU17dwxf0zPsZ9avAnn3yCX//61/jDH/4w9LNNmzbBaDRCFEXs2LED9913H/bt\n2zdUoI9GZ2cfglz76poZjXq0t/fKHYOuIJL2zwcnmyEBsCREo7fPPe7t+3z+cW9Xr9Ogt889IW1/\nim2Pve1P989EtD1axdMScayyHeU1HchJiZ2w58Tl8qA9EF5rW0fS61s44v65PFEUrrkD96rDPKxW\nK+x2OwKXDuJAIIC2tjZYrdYR2548eRKPPPIItm3bhuzs7KGfm81miOLgr1q3bh1cLhdaW1uvKSgR\nhbfjVW0wxmsQr+MQD5o6ZqQbEK9T45OKNnh4mXGiiHTVYjoxMRH5+fnYuXMnAGDnzp3Iz88fMcTj\nzJkz2LRpE37zm99g1qxZw+6z2z9bS7a0tBSiKMJsNo9HfiIKA30DPlQ2dKF4mhGCIMgdh2jciKKA\nhbPMcLn92PdJo9xxiEgGoxrm8e///u/YsmULfvvb3yI2NhZbt24FAGzYsAEbN25EYWEhfvKTn8Dt\nduPxxx8fetzPf/5z5OXlYfPmzejs7IQgCNDpdHj22WehVHLxeaJIcepCBwJBCbOnJ6HN6ZI7DtG4\nMhu0yEmJxQcnm7FmSQbidVFyRyKiSTSqijYnJwfbt28f8fMXXnhh6Pu//e1vV3z8Sy+9dO3JiGjK\nOF7VhsTYKKSbdSymaUqam2tES0c/Pilvwy0LUvkJDFEE4UrzRDShBjx+lNU5MDfXxAKDpqzoKCVu\nW5KJVocLlQ1cFpYokrCYJqIJdepCB/wBCfPyjHJHIZpQiwosSDXG4HhlOzp7JmZVDyIKPSymiWhC\nfXiqGSZDNKalxskdhWhCCYKAJYUWRKkVKD3VAp8/KHckIpoELKaJaMI0d/TjfFM3lhcnQ+QQD4oA\nGrUSJUVW9Lh8+KTCfvUHEFHYYzFNRBPmw1PNUIgClhaOXJeeaKqyJGpRlJOI6uYe1LT0yB2HiCYY\ni2kimhBeXwAfnWvFvDwjYmN4oRaKLEU5iTAZonGkrBU9/V654xDRBGIxTUQT4lhVG/rdfiyfnSJ3\nFKJJJ4oCSoqsEEUBB063wB/g+GmiqYrFNBFNiA9OtcBsiMaM9Hi5oxDJIiZahaWFVjh6PDhSZock\nSXJHIqIJwGKaiMZdc3sfLjZ1Y/nsFK4tTREtzaRD8bRE1LT0cP1poimKxTQRjbsPT7VAqRCwtNAi\ndxQi2RXlJCLVpMOxyja0OngFUKKphsU0EY0rry+Aw+daMS/PBL2WEw+JBEHAsiIL9Fo1DpxqQd+A\nT+5IRDSOWEwT0bg6WtkGl8ePm2Ynyx2FKGSolQqsmJOMQEDChyebOSGRaAphMU1E4+qDU82wJGiR\nm8aJh0SfF6eLwrJiKzp7PPiYExKJpgwW00Q0bmptPahu7sHy2cmceEh0GWkm3eAFXTghkWjKYDFN\nRONmR2ktYjRK3FjMIR5EV1I8LRGpxhhOSCSaIlhME9G4uNjcjbM1nfjyogxERynljkMUsgYnJFqH\nJiT2c0IiUVhjMU1E4+KNAzXQa1X40txUuaMQhTy16rMJiR+c5BUSicIZi2kium5VDU5U1DvxT4sy\nEKVWyB2HKCzE6aKwtMiCzh43JyQShTEW00R0XSRJwhultYjTqbFiTorccYjCSrpZPzQh8Xxjt9xx\niGgMWEwT0XUpr3fifGMX1izOhFrFXmmia1U8LRHJSVocq2xDV59H7jhEdI1YTBPRmEmShB2lNTDo\no3BjsVXuOERhSRAELCmwQqkQUXrahkCQ46eJwgmLaSIas7M1DlQ39+D2JZlQKdkrTTRWWo0SSwot\ncPZ6cPJ8h9xxiOgasJgmojEJBiW8caAGSXEaLCtirzTR9Uoz6ZCbFofyOidaOvrljkNEo8RimojG\nZM/RBtTbe3HH8mwoFXwpIRoP82eYEBujxqGzrXB7A3LHIaJR4BmQiK5Zc0c/3jhQi7m5RizMN8sd\nh2jKUCpElBRZ4fH6caSslcvlEYUBFtNEdE0CwSD+sKscGrUC967KgyAIckcimlIS4zSYnWtEg70P\ntbYeueMQ0VWwmCaia/L2kQbU2npx76o8xMWo5Y5DNCXNzDQgKU6DoxXtcHv9cschoi/AYpqIRq2p\nrQ9/P1iLBTNMWDDDJHccoilLFAQsLrDA5w/gWGW73HGI6AuwmCaiUfEHgvjdrnLEaJS459ZcueMQ\nTXkGfRRmZSeipqUHFfUOueMQ0RWwmCaiUXnzUC0a7H24d9UM6LUc3kE0GYqyExAbo8Zf9l2Ah6t7\nEIUkFtNEdFUfnmrGzsP1WFpowbw8o9xxiCKGQiFi8SwzHD0e7DhYI3ccIroMFtNE9IWOVbbhlT1V\nKMxOxL+sniF3HKKIY07QYmmhBXuPNqKulat7EIUaFtNEdEVldQ48/1YZcpLj8NBXC3hxFiKZfGVZ\nNmJj1Hjp7UoEgkG54xDR5/DMSESXVdPSg//621lYErT4n+uLEKVSyB2JKGJpNUrcfXMuGux92H+8\nWe44RPQ5LKaJaISmtj78avtp6LUqfO+u2YjRqOSORBTx5uUZUZidiDdKa+Ds9cgdh4guYTFNRMMc\nKWvF068eg0Ih4Adfn414XZTckYgIgCAIuPvWXASCEl7bf0HuOER0CYtpIgIA+PxBvLqnCs+/VY5M\nsx6P/8sCmAxauWMR0eeY4qNx2+IMfFLRhrJarj1NFApYTBMROroG8H/+eBzvn2zG6oXpeOSbc2DQ\ns0eaKBR9eWEGzIZovLq3Cj4/154mkhuLaaIIFggG8cGpZvzkpaOwOwfw8B2FuHPFNChEvjQQhSqV\nUsQ9t+ahzTmAtz9ukDsOUcRTyh2AiCafJEk4U92Jv75/EbZOF6anxuF/3JbPYR1EYWJWVgJuyDdh\n5+F6LJpp5rFLJCMW00QRpr61F399/yIq6p0wG6Lx8B2FmDM9CYIgyB2NiK7BXSun40x1J/747nls\nWl/MY5hIJiymiSJAUJJwrqYT7x5tRFmdE7poFe6+JRfLZyfzQixEYcqgj8JXS7Lx5/cu4HhVO+bP\nMMkdiSgisZgmmsI83gAOn7Ph3WNNaHW4EK9T444bs7Fybiq0Gh7+ROFu5bwUHDprw5/fu4BZWQmI\njuJxTTTZeNQRTUGOHjfeO96ED0+1wOXxI8uqx/1fmYn5eSb2RBNNIQpRxL2r8vDTV4/jzUO1uGvl\ndLkjEUUcFtMRxB8EPD7/hLQdpVJCyRptmNE835LDBZfn2veJSqmEzz/ycbW2HnxwohmnLrRDAjB7\nWhJumpuCLGssBEGAxx+Exx8cU9vjIShNSLNEES0nJQ43Fifj3aNNWFpgRapJJ3eksDeR58uJfI3l\nuVgeLKYjiMfnx9EK+4S0vSDfDCU/XhxmNM+3XqdBb5/7mtsuzjXi9Pl2AIMrc7R0uHCmuhPtXQNQ\nKUXMyDBgRoYBumgVOrvd6Owe/e/4fNvjrTjXOCHtEkW6r92UgxPn2/HK3ipsuXsuRE5GvC4Teb6c\nyNdYnovlwWecKExJkoSm9n6cqe5EZ7cbWo0SC/JNmJYSBxW7Jogiii5ahfUrcvDi7kocOmtDSVGy\n3JGIIgaLaaIwVFHnwM7D9XD2eqCLVmHxLDOyU+KgENkbRRSplhZaUXrGhu3vV2POdCN00Sq5IxFF\nBHZfEYWRnn4v3jvehN+9WQ5/IIilhRasK8nC9LR4FtJEEU4UBHzr1jy43H78df9FueMQRQz2TBOF\nAa8/gDMXO1FZ74RCFHH7skzEatUsoIlomFSTDl9elI5dH9VjzvQkzOE8BaIJx55pohBX39qLHQdq\nUV7nRHZyHNbdmIWb5qaykCaiy1q7LAvpZh1eeqcSPf1eueMQTXkspolClNvrx4FTLfjwVAtiNEr8\n0+IMLCm08KIMRPSFlAoRG9bMxIAngJferoQkcU1KoonEYpooBDXYe/HmwTo02HsxZ3oSvrwoA0lx\nGrljEVGYSDHq8LXl2Th1sQOlZ2xyxyGa0tjFRRRCvL4APi63o9bWi4TYKNyyIA0GfZTcsYgoDN28\nIA2nLnbgz+9dwIwMA0zx0XJHIpqS2DNNFCLanC68dagOda29KJ6WiH9alMFCmojGTBQE/I/bZkIU\ngN/vLEeQlyAlmhAspolkFgxKOH2xA3s+boQgCPjywnQUT0uCyAmGRHSdEuM0uPuWXFxo6saOg7Vy\nxwk7kiTBHwhiwONHT78X3X1eeHwBWcehB4JBeLwBDHj86Bvwoaffi55+L/yBoGyZIh2HeUxhLrcf\nrZ39aGrtxYDHD0efBw32XsTFREGvVbFYCwG9Li/2Hm1Em3MA2cmxuGGmCWqlQu5YRDSFLJ5lQWVD\nF3YeroM1QYvFBRa5I4Ukjy+AmuZuXGjqRkWDE3W2Xnj9AVyubhYFAZooBaLVCsREqxCrVSM2Ro04\n3eDXKNX1vY77A0H0ugYL5V6Xd/D7S19dbv8VH/f2kQZYErQwJ2iRkhSD2dOSkMj5NhOOxfQU4vMH\ncL6pG+dqOnGuxoHmjv4rbiuKAuJi1DDoo2CMj0amRY8oNYu4yVTf2osjZXYEgxKWFVmQnRwndyQi\nmoIEQcC3VuWho2sAL75dgcQ4DXLT4uWOFRIGPH4cKbfj8Dkb6my9CAQlCACSjTFIN+sQpVZApRSh\nVopQXerocHv9GPAE4Pb64fYE0N3nRVNbHz4/ikatEqGLViFGoxr8Gq2EUiFCIQoIAmiy9wIYLODd\nngDc3sH2XG7/YMHsGV4wa9QK6LUqWBK00GtVUCsVEEUBoihAIQKSBPQP+KBSKtDZ48bRCjs+cPvx\n3++eR5ZVj/l5JszLM8Jk0E7SMxtZWEyHuaAkoazWgfdPNKO83gGvLwilQkBuWjwWzTIj1RKHgM8P\nbZQSEAWcq+lAd58XXX0edPV60epwoaalB0cr7Eg16ZCdHIsUo45rGE8gnz+Io5VtuNjUDZNBi6WF\nZui1arljEdEUplSIeOirhfjpq8fxX6+fxY++NS9iCytJklDX2osPTzXj4/I2eHwBpBp1WL0wHdNT\n4zEtJRaSIOBohX3UbQaDEvoGfOi+NOSib8A3OATD5YWtsx/+wGeV9qGzrSMer1QI0KiViI5SwJqo\nhT5GDb12sMdbr1VBPcqe7gX5ZsRcWj7V7nDhWFUbjle1Y/sH1dj+QTWyrLFYvTAd83gxn3HFYjpM\neXwBHD7Xin3HGmHrdCFOp0ZJYTIKshMwI90w1MtsNOrR3j74DrjfMzjsIylu+IxuR48bNS09qGnp\nQYO9D2qViOmp8ZiZaeCaxuPM0eNG6Wkbuvu9KMhOwLLZqXC5PHLHIqIIoItW4d++VoSnXzmGX20/\ng8e+NQ8xGpXcsSaNJEk4Xd2JNw/Woq61F2qViIX5ZiyfnYIsqx6C8FknUr/nykMpLkcUBcTGDA7x\nuNzv9fqC8AeDCAYlTEszoLy2E5CAKLUCGrUCSsX4T2EzJ2hx2+JM3LY4Ex1dAzhW1Y4PTzXj2R3n\nYIqPxj9/aTqKMw2jLtTpylgphZm+AR/2Hm3A+yea0e/2I8Oix4bbZ2LBDNOYD8aEWA0SYjWYm2uE\nrdOFi83dKK91oLLeielpcSjISoA2gl5wJ0JQklBR58TJ8x2IUitwy4JUWBNj+AkAEU0qc4IWD99R\niP/4yyn89o1z+Lf1xVApp/ZaBJIkobzOiTdKa1DT0gNjvAb33JqLRTMt0GomvgwSBAFRagWiMFi0\nWhK1sHdeeRjmREiKj8bqhem4dUEaTl5ox+4jDXj2b2eg16pw64I0fGleKjRqloRjxWcuTLjcPuz5\npBHvHmuExxvA3FwjblmQhumpccPeTV8PURSQYoxBijEGPf1enK3pRFVDF843dGNaaiwKshOhi2ZR\nfa26+jw4fLYVHd1upJl0WFxg5osWEckmL92Ab395Bn6/qwL/+ddTePiOwinbYVLV4MQbpbU439iF\nhNgo/MvqPCwttE5IT3A4EEUB8/JMmJtrRFuvF396pxJ/+7AGe4824rZFGVgxN2VobDiNHs/oIW7A\n48e7xxqx55NGDHj8mJ9nxNplWUgx6ib098bGqLG00IqinESU1TpwsakbF5t6kJsWh8KcRA7/GIVA\nIIiz1Z04fbETKqWIkiIrMv/ho0QiIjksLbRCFAX8YVcF/vcfT2DT+uIptepDdUs3dhyoQVmdE3Ex\natx9Sy5uLE6e8r3woyUIAgpykrDpzmJUN3fjjdIa/GX/Rew52og1SzJRUhS5bzjGghVRiHJ7/Xjv\neBPe+bgB/W4/5kxPwtplWUg36yc1h16rxqJZFhRkJ+JMdSeqGrtwsbkbeekGFGQlcAWQK6hp6cFL\n71Siqa0PGRY9bsg38Q0IEYWUxbMsiNdF4b9eP4unXz2Gf/taMTIsk3uOGW/1rb3YUVqD09Wd0EWr\ncOeKaVgxN+W6l6qbynJS4vCDr89BZb0Tr5fW4NU9VXj7SD2+sjQLiwvMUIgsqq+GZ/cQ4/EFsP9E\nE94+0oC+AR+KchKxriQLmZZYWXPpolVYUmBBQVYCTl/sQFmtA+cbu5CXHo/8DIOs2UJJW9cAXv+w\nGp9UtEGvVWH57OSwPzkR0dSVn2HA/7pnLv5z+2n87E8n8ODaAhTlJMod65rVtPRg10d1OHmhAzEa\nJf55eTbHAV+jGRkG/DB9Ls7VOvD6gRr8YXcFdh2px7plWViQb4LIT1WviP9lIaJvwIcPTzXj3WNN\n6On3oiArAWtLspATYmsPx8aoUVKcjIJsD85UD65nXVHnRJtzALcvyURC7NT5mPBa9A348NahOuw/\n0QSFKOD2JZm4cU4yzlZ3yh2NiOgLpRh1eOze+fj19tP49fbTKCm24o7lOYgN8SU7JUlCZUMXdn1U\nh/I6J2I0SqxdloVb5qdNysTCqUgQBBRmJ6IgKwEnL3TgjdIa/N83y7DzozqsviEdN+SbOVTmMvjf\nJrOm9j7sO9aIj8rs8PmDmJlpwFfWFYT8gvoGfRSWz05Gd58X52o7UXrGhoNnbFg0c3CZoZyU2IgY\nG9zWNYAPTjTjw9MtcHv9WFZoxbqSbBj0Ude8tBIRkVwM+ihsuWcu3jxYh3ePNeJ4VTu+emM2bpqd\nEnJXy/X6Ajh+vh37jzehuqUHcTFq3LliGpbPTuZwunEiCALm5hoxe3oSjla04a3Ddfj9rgpsf/8i\nbpqTghVzUhCni5I7Zsjgf50M+t0+nDzfgY/KWlFR74RKKWLxLAtunp+K1AmeWDje4nSDExW/tXoG\nDpxswcGzNhw61wprohYlRclYUmC57Lqb4SwoSSivdeC94004U9156UUnCV9ZlhV2+4+I6FMatRJ3\nrpyGpUVW/PfeKvxx73kcON2CO1dMw4wMg6wf80uShHp7L0pP23Ck3I4Bjx/GeA3uvTUXy4qsXIFi\ngoiCgIUzzbgh34TyOifePdaINw/VYddH9ViQb8LCfDNmZSVE/GRFFtOT5NMC+lhVG8pqHQgEJSTG\navDPy7OxfHZK2C85lxCrwd235uKO5dk4WtmG0jMt+Ov7F/G3D6sxKyth8GOj7ASYw/SKW/5AEBca\nu3C6uhMnzrejo9uNWK0Ka5Zk4qY5KTDo+Q6diKaGlKQYPPKNOTha2YbX9l/Ef/zlFJLiNFhSYMGS\nAsukXTnRHwiiurkb52odOH2xA03t/VApRczPM6KkKBm56fEcxztJBEHArKwEzMpKgN3hwr5jTfio\nrBVHyuzQRikxZ3oS5s8wRWxhPapiura2Flu2bEFXVxfi4+OxdetWZGZmDtsmEAjg6aefRmlpKQRB\nwP3334/169df9b6pqrPbjQvNXZeWlOtGY3sfJAlIjNXglvlpmD/DNOKKS1NBdJQSNxYn48biZDR3\n9OPgmRacvNCBM5fGDpsM0SjMSsS01Dikm3UwG7Qh9xEiMPgxYnNHP+rtvSivc6KsthMDngCUCgEz\n0g346o3ZmJ9n4tgxIpqSBEHADflmFE9Lwomqdhw6Z8Nbh+rw5qE6TE+NQ/G0JGRa9Mi06Mdtjeoe\nlxfNbX1obO9HZb0TFQ1OeLwBKEQBOcmxuHdVHhbmm6bsmtjhwpygxd235uKuL01DeZ0DRyvacOJC\nBw6da4VaKSLLGotpqXGYnhqHnJS4iLjK5qiK6SeeeALf/OY3sXbtWvz973/H448/jldeeWXYNm+9\n9RYaGhqwd+9edHV1Yd26dVi8eDFSU1O/8L5wJEkS3N4Ael1e9A744OzxwO50we4YuPTVhR6XDwAQ\npVIgOzkWty/JRFFO0pQsoK8kJSkGd62cjrtWTofd6cK5GgfO1nSi9EwL3jvRBABQq0SkGnVIM+mQ\nFDd4JcYEfRQMsRoYdFETWqx6fAE4ez1w9Ljh6PHA0euG3eFCg70Ptk4XgpIEYHAoy4IZJhTnJCE/\n08DZ4UQUMaJUCiwusGBxgQWOHjc+KmvFR2V2/L8Pqoe2MRuikWHRIzFWA71WjdgYFfRaNVIH/Ojq\ncgEYHB4HCfD5g+gd8KGn34uefi96XV60dQ2gqb0fPf3eoTaT4jRYPGtwBan8DAPHQocgpUJEUU4S\ninKS8C1/EOV1DpTVDV6X4u0jDdh16RyaFKeB2RANU4IWZoMWZkM04nVR0GtV0EWrpsTlzK/639nZ\n2Yny8nK8+OKLAIA1a9bgqaeegsPhQEJCwtB2u3fvxvr16yGKIhISEnDzzTfjnXfewX333feF943W\nZPdeur0BvFFag54+D3wBCT5/EL5AEF5fAP1uHwIBacRj9Fo1kuI0mJ4Wj5SkGGRaYmFNjIYo8xqN\nnz53SoU4Ye/olQrxC/eRNTEG1sQY3LIgDYFgEHbHAGyd/WjucMHW0Yf61l5U1DtHPE6lFKFRKxEd\npYBGrUSUSoRCIUIpilAoBCgVwmXfnASDEvwBCYFgEIGgBL8/CLc3ALfXD7cvCLfXD78/OOJxcboo\n5KTGoWR2MlKTYpCcFAODPmpMb4BG83xHRykR8F/7Ppnofcm2B326f8Itd6S0PZbjJxRyj7VtuT/F\nS4qPxu1Ls3D7LRZkvwAACjZJREFU0iy43H40tfehub0PTW39aOroQ2N732XPjVciANBGqxCvi8Ky\nIgssCTGwJmhhSdRCF62SteMpUv5Pxut/KkqtwJxcI+bkGgEAXl8QDW29qG/thd05gI6uAVQ396Cs\n1jHisWqVAjFRSqhUIpQKESqlAiqFgHSLHqtvSB+XfNdiLM/JVYtpm80Gs9kMhWLwnYNCoYDJZILN\nZhtWTNtsNiQnJw/dtlqtaG1tvep9o2UwxFzT9uPh4TvnTPrvnAiJiZ9Niku1hsZSeyZjLArlDjEJ\nJvL5zk6duPW92TbbZtuh13YoSQSQlhLaq05dr3B9/b4Wn68PxpvVEouFRRPWfEjhgE8iIiIiojG6\najFttVpht9sRCAQADE4mbGtrg9VqHbFdS0vL0G2bzQaLxXLV+4iIiIiIwtVVi+nExETk5+dj586d\nAICdO3ciPz9/2BAPAFi9ejW2b9+OYDAIh8OBffv2YdWqVVe9j4iIiIgoXAmSJF11tkB1dTW2bNmC\nnp4exMbGYuvWrcjOzsaGDRuwceNGFBYWIhAI4Mknn8ShQ4cAABs2bMBdd90FAF94HxERERFRuBpV\nMU1ERERERCNxAiIRERER0RixmCYiIiIiGiMW00REREREY8RimoiIiIhojHix+ymgtrYWW7ZsQVdX\nF+Lj47F161ZkZmYO2+aZZ57Bn/70J5hMJgDA3Llz8cQTT8iQNrJs3boVe/bsQXNzM9566y3k5uaO\n2CYQCODpp59GaWkpBEHA/fffj/Xr18uQNvKMZv/w2JGP0+nEo48+ioaGBqjVamRkZODJJ58csTTr\nwMAAfvjDH6KsrAwKhQKbN2/GihUrZEodOUa7f7Zs2YLDhw/DYBi86t/q1avx4IMPyhE5Ij300ENo\namqCKIrQarX48Y9/jPz8/GHb8Dx0nSQKe/fee6+0Y8cOSZIkaceOHdK99947Ypvf/OY30s9+9rPJ\njhbxjh49KrW0tEgrVqyQqqqqLrvNG2+8IX3nO9+RAoGA1NnZKZWUlEiNjY2TnDQyjWb/8NiRj9Pp\nlI4cOTJ0+2c/+5n0wx/+cMR2zzzzjPTYY49JkiRJtbW10pIlS6S+vr5JyxmpRrt/Nm/eLL366quT\nGY0+p6enZ+j7d999V1q3bt2IbXgeuj4c5hHmOjs7UV5ejjVr1gAA1qxZg/LycjgcDpmTEQDMnz9/\nxNVC/9Hu3buxfv16iKKIhIQE3HzzzXjnnXcmKWFkG83+IfnEx8dj4cKFQ7dnz5497Gq6n3r77beH\nrl2QmZmJgoICHDhwYNJyRqrR7h+Sl16vH/q+r68PgiCM2IbnoevDYR5hzmazwWw2Q6FQAAAUCgVM\nJhNsNtuIj9p27dqFgwcPwmg04rvf/S7mzJkjR2T6BzabDcnJyUO3rVYrWltbZUxE/4jHjvyCwSD+\n/Oc/Y+XKlSPua2lpQUpKytBtHkOT74v2DwC8+OKLeO2115CWlobvf//7yMnJmeSEke2xxx7DoUOH\nIEkSfve73424n+eh68NiOkJ8/etfxwMPPACVSoVDhw7hoYcewu7du4fGsBHR5fHYCQ1PPfUUtFot\n7rnnHrmj0GV80f7ZtGkTjEYjRFHEjh07cN9992Hfvn1DnUA08X76058CAHbs2IGf//zneOGFF2RO\nNLVwmEeYs1qtsNvtCAQCAAYnEbS1tY346NpoNEKlUgEAli5dCqvVigsXLkx6XhrJarUO+2jUZrPB\nYrHImIg+j8eO/LZu3Yr6+nr86le/giiOPG0lJyejubl56DaPocl1tf1jNpuHfr5u3Tq4XC72espk\n3bp1+Pjjj+F0Oof9nOeh68NiOswlJiYiPz8fO3fuBADs3LkT+fn5I4Z42O32oe8rKirQ3NyMrKys\nSc1Kl7d69Wps374dwWAQDocD+/btw6pVq+SORZfw2JHXL3/5S5w7dw7btm2DWq2+7DarV6/Ga6+9\nBgCoq6vD2bNnUVJSMpkxI9Zo9s/nj6HS0lKIogiz2TxZESNaf38/bDbb0O39+/cjLi4O8fHxw7bj\neej6CJIkSXKHoOtTXV2NLVu2oKenB7Gxsdi6dSuys7OxYcMGbNy4EYWFhdi8eTPKysogiiJUKhU2\nbtyI5cuXyx19ynv66aexd+9edHR0wGAwID4+Hrt27Rq2bwKBAJ588kkcOnQIALBhw4ahyVQ0sUaz\nf3jsyOfChQtYs2YNMjMzodFoAACpqanYtm0b1q5di+effx5msxkulwtbtmxBRUUFRFHEI488gptv\nvlnm9FPfaPfPt7/9bXR2dkIQBOh0Ojz66KOYPXu2zOkjQ0dHBx566CEMDAxAFEXExcVh8+bNmDVr\nFs9D44jFNBERERHRGHGYBxERERHRGLGYJiIiIiIaIxbTRERERERjxGKaiIiIiGiMWEwTEREREY0R\ni2kioiniueeew2OPPTaqbZ955hn84Ac/mOBERERTH4tpIqIQsnLlShw+fHjYz15//XV84xvfuOpj\nH3jggaHLBk9EDiIiGonFNBERERHRGLGYJiIKI3a7Hd/97nexaNEirFy5Eq+88srQff84dGPHjh1Y\nsWIFFi5ciG3bto3obfb5fHj00UcxZ84c3HbbbTh79iwA4JFHHkFLSwseeOABzJkzBy+88MLk/YFE\nRGGGxTQRUZgIBoN48MEHkZeXhwMHDuDll1/Gyy+/jNLS0hHbXrx4ET/5yU/wi1/8AqWlpejr64Pd\nbh+2zf79+3Hbbbfh2LFjWLlyJZ566ikAwC9+8QskJyfjueeew8mTJ7Fhw4ZJ+fuIiMKRUu4AREQ0\n3L/+679CoVAM3fb5fJg5cybOnj0Lh8OBhx9+GACQlpaGO++8E7t370ZJScmwNt555x2sWLEC8+fP\nBwBs3LgRr7766rBt5s2bh+XLlwMA1q5di5dffnki/ywioimJxTQRUYjZtm0blixZMnT79ddfx/bt\n29Hc3Iy2trahAhkAAoHAsNufamtrg8ViGbodHR2N+Pj4YdskJSUNfa/RaODxeOD3+6FU8tRARDRa\nfMUkIgoTVqsVqamp2Lt371W3NZlMqK2tHbrtdrvR1dU1kfGIiCISx0wTEYWJoqIixMTE4Pnnn4fb\n7UYgEMD58+dx5syZEduuWrUK+/fvx4kTJ+D1evHMM89AkqRR/66kpCQ0NjaOZ3wioimJxTQRUZhQ\nKBR47rnnUFlZiS996UtYtGgRfvSjH6Gvr2/EttOnT8ePf/xjfO9730NJSQm0Wi0SEhKgVqtH9bvu\nv/9+PPvss5g/fz5+//vfj/efQkQ0ZQjStXRVEBFRWOrv78eCBQuwZ88epKWlyR2HiGjKYM80EdEU\ntX//fgwMDMDlcmHr1q3Izc1Famqq3LGIiKYUFtNERFPUe++9h5KSEpSUlKC+vh6//OUvIQiC3LGI\niKYUDvMgIiIiIhoj9kwTEREREY0Ri2kiIiIiojFiMU1ERERENEYspomIiIiIxojFNBERERHRGLGY\nJiIiIiIao/8Phj/Mf0+w3mIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 864x864 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jI9ToieVrisQ",
        "colab_type": "text"
      },
      "source": [
        "Dá para perceber que a maior parte dos dados concentra-se em torno da média (~ 1.7 m) e que apenas algumas observações encontram-se bastante distantes dela."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q49-oFz4gBHs",
        "colab_type": "code",
        "outputId": "f968b883-a1e3-4ead-963a-19d9f25e9d9e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "height_outlier_mean = height_outlier.mean()\n",
        "height_outlier_std = height_outlier.std()\n",
        "\n",
        "height_outlier_mean, height_outlier_std"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1.7181251474953014, 0.2948590174540895)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dTtLF6P2rvIh",
        "colab_type": "text"
      },
      "source": [
        "Um jeito de procurar por _outliers_ é ver quem se encontra fora do intervalo $[\\bar{x} - k * \\sigma, \\bar{x} + k * \\sigma]$, onde $k$ geralmente é 1.5, 2.0, 2.5 ou até 3.0.\n",
        "\n",
        "Abaixo utilizamos o $k = 2$, pois esse valor faz sentido (alturas menores que 1.12 m ou maiores que 2.30 m fogem do nosso padrão):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cI8gL-QrgK1s",
        "colab_type": "code",
        "outputId": "6c472ac1-ea23-4dd3-b833-91969a62f92d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "non_outlier_interval_dist = [height_outlier_mean - 2 * height_outlier_std, height_outlier_mean + 2 * height_outlier_std]\n",
        "\n",
        "non_outlier_interval_dist"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[1.1284071125871225, 2.3078431824034804]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b5A37brPsVPw",
        "colab_type": "text"
      },
      "source": [
        "Novamente, conhecendo o intervalo, podemos identificar as observações que caem foram dele e removê-las:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W6jVe5TMglf5",
        "colab_type": "code",
        "outputId": "c270dcb7-d46a-4dd8-94b3-c3d610269282",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "source": [
        "outliers_dist = height_outlier[(height_outlier < non_outlier_interval_dist[0]) | (height_outlier > non_outlier_interval_dist[1])]\n",
        "\n",
        "outliers_dist"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "29    0.516665\n",
              "38    2.943781\n",
              "48    1.058498\n",
              "68    2.737088\n",
              "Name: Height, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jqYD2d3chJTK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "height_no_outlier_dist = height_outlier.drop(index=outliers_dist.index)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8IL5fWP1sePM",
        "colab_type": "text"
      },
      "source": [
        "Até agora, nossas métodos de identificação de _outlier_ foram baseadas em estatísticas descritivas do nosso _data set_ (quantis, média e variância). Porém, alguns testes de hipóteses também existem.\n",
        "\n",
        "Um deles é o teste de Grubb. Esse é um teste bastante simples, cuja estatística de teste $G$ depende dos valores extremos do conjunto e da média amostral:\n",
        "\n",
        "$$G = \\frac{\\vert x_{\\text{\\{min ou max\\}}} - \\bar{x}\\vert}{s}$$\n",
        "\n",
        "onde $\\bar{x}$ é a média amostral e $s$ é o desvio-padrão da amostra.\n",
        "\n",
        "A hipótese nula, $H_{0}$, é de que não existem _outliers_ no _data set_. O teste de Grubb assume que os dados originam-se de uma distribuição normal, então pode ser válido testar essa hipótese antes.\n",
        "\n",
        "Rejeitamos a hipótese nula se o valor de $G$ encontrado for superior ao valor crítico do teste, que é dado por\n",
        "\n",
        "$$G_{\\text{crítico}} = \\frac{n - 1}{\\sqrt{n}} \\sqrt{\\frac{t_{\\alpha',n-2}^{2}}{n - 2 + t_{\\alpha',n-2}^{2}}}$$\n",
        "\n",
        "onde $n$ é o tamanho da amostra, $t$ é um valor com distribuição t-Student e $\\alpha'$ é $\\alpha/2n$ se o teste for bilateral (procuramos _outliers_ muito acima ou muito abaixo) ou $\\alpha/n$ se o teste for unilateral (acreditamos que o _outlier_, se houver, está em somente uma das extremidades da distribuição)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RNveH7ftxMOV",
        "colab_type": "text"
      },
      "source": [
        "Abaixo criamos algumas funções que nos auxiliam nos cálculos e na exibição dos resultados:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ir61-q0ckV6K",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def grubb_test(g, n, alpha=0.05, tailed='two-tailed'):\n",
        "  if tailed == 'two-tailed':\n",
        "    critical = ((n - 1)/sqrt(n)) * sqrt(sct.t.isf(alpha/(2*n), n-2)**2/(n - 2 + sct.t.isf(alpha/(2*n), n-2)**2))\n",
        "    \n",
        "    return (g, critical, g > critical)\n",
        "  elif tailed == 'one-tailed':\n",
        "    critical = ((n - 1)/sqrt(n)) * sqrt(sct.t.isf(alpha/(n), n-2)**2/(n - 2 + sct.t.isf(alpha/(n), n-2)**2))\n",
        "    \n",
        "    return (g, critical, g > critical)\n",
        "  else:\n",
        "    raise ValueError(f\"Invalid tailed argument\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c--VvSPuuHaM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def grubb_summary(result, decimals=10):\n",
        "  return (\n",
        "    f\"Null hypothesis: there is no outliers in the data set\\n\"\n",
        "    f\"Test statistic: {np.round(result[0], decimals)}, \"\n",
        "    f\"Grubb's critical value: {np.round(result[1], decimals)}, \"\n",
        "    f\"Reject: {result[2]}\"\n",
        "  )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d8nFGEVuqgdC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def next_outlier_candidate(data):\n",
        "  sample_distances = (data - data.mean()).abs()\n",
        "  candidate_idx = sample_distances.idxmax()\n",
        "  candidate_value = data[candidate_idx]\n",
        "  candidate_statistic = sample_distances.max()/data.std()\n",
        "  \n",
        "  return (candidate_idx, candidate_value, candidate_statistic, len(data))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MRZwuyOOxU7U",
        "colab_type": "text"
      },
      "source": [
        "Ao executarmos o teste de Grubb no nosso conjunto de alturas, encontramos alguns valores onde a hipótese nula é rejeitada, ou seja, há evidência de que o valor extremo é um _outlier_."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rz-yVWFlt-M6",
        "colab_type": "code",
        "outputId": "cb11e99b-2195-45d7-9089-fdf292a65e1c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 434
        }
      },
      "source": [
        "height_outlier_grubb = height_outlier.copy()\n",
        "outliers_grubb = pd.Series()\n",
        "has_outlier = True\n",
        "\n",
        "while has_outlier:\n",
        "  outlier_candidate = next_outlier_candidate(height_outlier_grubb)\n",
        "\n",
        "  print(f\"Index: {outlier_candidate[0]}, \"\n",
        "        f\"Value: {np.round(outlier_candidate[1], 3)}, \"\n",
        "        f\"Test statistic: {np.round(outlier_candidate[2], 3)}, \"\n",
        "        f\"Sample size: {outlier_candidate[3]}\\n\")\n",
        "\n",
        "  result = grubb_test(outlier_candidate[2], outlier_candidate[3])\n",
        "\n",
        "  print(grubb_summary(result, 3))\n",
        "\n",
        "  has_outlier = result[2]\n",
        "\n",
        "  if has_outlier:\n",
        "    height_outlier_grubb = height_outlier_grubb.drop(index=outlier_candidate[0])\n",
        "    outliers_grubb.at[outlier_candidate[0]] = outlier_candidate[1]\n",
        "    \n",
        "  print(f\"\\n\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Index: 38, Value: 2.944, Test statistic: 4.157, Sample size: 100\n",
            "\n",
            "Null hypothesis: there is no outliers in the data set\n",
            "Test statistic: 4.157, Grubb's critical value: 3.384, Reject: True\n",
            "\n",
            "\n",
            "Index: 29, Value: 0.517, Test statistic: 4.421, Sample size: 99\n",
            "\n",
            "Null hypothesis: there is no outliers in the data set\n",
            "Test statistic: 4.421, Grubb's critical value: 3.381, Reject: True\n",
            "\n",
            "\n",
            "Index: 68, Value: 2.737, Test statistic: 4.219, Sample size: 98\n",
            "\n",
            "Null hypothesis: there is no outliers in the data set\n",
            "Test statistic: 4.219, Grubb's critical value: 3.377, Reject: True\n",
            "\n",
            "\n",
            "Index: 48, Value: 1.058, Test statistic: 2.96, Sample size: 97\n",
            "\n",
            "Null hypothesis: there is no outliers in the data set\n",
            "Test statistic: 2.96, Grubb's critical value: 3.374, Reject: False\n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "49MMneSg-DCj",
        "colab_type": "code",
        "outputId": "a98df152-223e-43e1-ced9-d113a40b879f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        }
      },
      "source": [
        "outliers_grubb"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "38    2.943781\n",
              "29    0.516665\n",
              "68    2.737088\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_hajYam661Zd",
        "colab_type": "text"
      },
      "source": [
        "Abaixo comparamos os _outliers_ encontrados por cada um dos três métodos:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l3P2Bavg-zMK",
        "colab_type": "code",
        "outputId": "25065774-49a4-4509-fe92-70a4d32c8cd2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        }
      },
      "source": [
        "outliers = pd.Series({\"IQR\": outliers_iqr.index.values,\n",
        "                      \"Normal\": outliers_dist.index.values,\n",
        "                      \"Grubb\": outliers_grubb.index.values})\n",
        "\n",
        "outliers.apply(np.sort)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "IQR       [29, 38, 48, 68, 91, 92]\n",
              "Normal            [29, 38, 48, 68]\n",
              "Grubb                 [29, 38, 68]\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1oMEwGs_DHJW",
        "colab_type": "text"
      },
      "source": [
        "## _Features_ de texto\n",
        "\n",
        "Dados textuais são muito ricos e muito fáceis de serem encontrados. Diversos _data sets_ são compostos por documentos textuais e ainda um simples _scrapper_ pode coletar dezenas de milhares de documentos da Internet. Coleções de documentos são frequentemente chamadas de _corpus_ (plural, _corpora_).\n",
        "\n",
        "Nosso objetivo aqui é somente mostrar como preprocessar de forma simples _features_ textuais. Para isso, utilizaremos o _data set_ 20 newsgroups, que contém milhares de documentos categorizados em 20 grupos (desde astronomia até carros)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XItMVwyq8Dp9",
        "colab_type": "text"
      },
      "source": [
        "Abaixo escolhemos somente três grupos para restringir nosso escopo:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "usWrDfLvMNxw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "categories = [\"sci.crypt\", \"sci.med\", \"sci.space\"]\n",
        "\n",
        "newsgroups = fetch_20newsgroups(subset=\"train\", categories=categories, shuffle=True, random_state=42)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4uNwK5uREAn7",
        "colab_type": "text"
      },
      "source": [
        "Temos agora um _corpus_ com 1782 documentos:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_lUWgt06EtnR",
        "colab_type": "code",
        "outputId": "f82dd8b7-5f76-477c-9173-ee35d0c7e0aa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "len(newsgroups.data)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1782"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xh326fr28Jyc",
        "colab_type": "text"
      },
      "source": [
        "Um exemplo de documento desse _corpus_ é mostrado abaixo:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vsfaD72_M52H",
        "colab_type": "code",
        "outputId": "fb895197-8753-49e6-a631-e7716ad8c8ee",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        }
      },
      "source": [
        "document_idx = 4\n",
        "documents_total = len(newsgroups.data)\n",
        "\n",
        "print(f\"> Document {document_idx} of {documents_total}:\\n\\n{newsgroups.data[document_idx]}\")\n",
        "print(f\"> Category: {newsgroups.target_names[newsgroups.target[document_idx]]}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "> Document 4 of 1782:\n",
            "\n",
            "From: billc@col.hp.com (Bill Claussen)\n",
            "Subject: Re: Should I be angry at this doctor?\n",
            "Organization: HP Colorado Springs Division\n",
            "Lines: 5\n",
            "Distribution: na\n",
            "NNTP-Posting-Host: hpcspe17.col.hp.com\n",
            "\n",
            "\n",
            "Report them to your local BBB (Better Business Bureau).\n",
            "\n",
            "Bill Claussen\n",
            "\n",
            "\n",
            "> Category: sci.med\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6liTZFzv8Nas",
        "colab_type": "text"
      },
      "source": [
        "Quando trabalhando com dados textuais, uma representação simples é ter:\n",
        "\n",
        "* Cada documento em uma linha.\n",
        "* Cada palavra (ou termo) em uma coluna.\n",
        "\n",
        "Por exemplo, se nosso vocábulário (conjunto de todas palavras ou termos do _corpus_) tiver tamanho 10000 e tivermos 100 documentos, então nosso _data set_ será composto de 100 linhas e 10000 colunas."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qLBi7mFU8mLI",
        "colab_type": "text"
      },
      "source": [
        "O valor de cada célula, $x_{i, j}$, (interseção da linha $i$ com a coluna $j$) do _data set_ depende da tranformação que aplicarmos.\n",
        "\n",
        "A transformação mais simples é a contagem de palavras no documento, ou seja, $x_{i, j}$ indica o número de ocorrências da palavra $j$ no documento $i$.\n",
        "\n",
        "Isso pode ser obtido no sklearn pelo `CountVectorizer`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4E6FmUUhNs8b",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "count_vectorizer = CountVectorizer()\n",
        "newsgroups_counts = count_vectorizer.fit_transform(newsgroups.data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TSylOCPKjLmh",
        "colab_type": "code",
        "outputId": "d7b6e6b8-f227-4ec5-a34a-2cf93fc8ebb5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "type(newsgroups_counts)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "scipy.sparse.csr.csr_matrix"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M4rtFrsF9CgR",
        "colab_type": "text"
      },
      "source": [
        "Abaixo escolhemos dez palavras contidas no _corpus_ para exemplificar:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kmxzJhkSUpIZ",
        "colab_type": "code",
        "outputId": "613a8241-c25e-4d5d-9830-1cee04671fc4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "words_idx = sorted([count_vectorizer.vocabulary_.get(f\"{word.lower()}\") for word in\n",
        "                    [u\"clipper\", u\"Kapor\",\n",
        "                     u\"monitor\", u\"gibberish\",\n",
        "                     u\"Banks\", u\"private\",\n",
        "                     u\"study\", u\"group\",\n",
        "                     u\"Colorado\", u\"Business\"]])\n",
        "\n",
        "pd.DataFrame(newsgroups_counts[:5, words_idx].toarray(), columns=np.array(count_vectorizer.get_feature_names())[words_idx])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>banks</th>\n",
              "      <th>business</th>\n",
              "      <th>clipper</th>\n",
              "      <th>colorado</th>\n",
              "      <th>gibberish</th>\n",
              "      <th>group</th>\n",
              "      <th>kapor</th>\n",
              "      <th>monitor</th>\n",
              "      <th>private</th>\n",
              "      <th>study</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>4</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   banks  business  clipper  colorado  ...  kapor  monitor  private  study\n",
              "0      0         0        2         0  ...      1        0        0      0\n",
              "1      0         0        0         0  ...      0        2        0      0\n",
              "2      3         0        0         0  ...      0        0        1      0\n",
              "3      0         0        0         0  ...      0        0        0      2\n",
              "4      0         1        0         1  ...      0        0        0      0\n",
              "\n",
              "[5 rows x 10 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C7WuoRgP9WE9",
        "colab_type": "text"
      },
      "source": [
        "Por exemplo, o valor 2 na interseção do documento 0 com a coluna `clipper` indica que a palavra _clipper_ aparece duas vezes no documento 0. Obviamente é possível que uma mesma palavra apareça em múltiplos documentos e mais óbvio ainda que um documento contenha múltiplas palavras."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UQzj-_QT9p7e",
        "colab_type": "text"
      },
      "source": [
        "O problema com essa abordagem é que não temos como medir relevância dos termos. E se o  termo é super comum e aparece em quase todos documentos? E se o termo aparece muitas vezes no mesmo documento, mas poucas vezes nos outros?\n",
        "\n",
        "Essas perguntas não podem ser respondidas simplesmente com a contagem de termos acima. Para isso, precisamos do tf-idf."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXBnOFk___QK",
        "colab_type": "text"
      },
      "source": [
        "O tf-idf é uma estatística baseada no _corpus_ composta de outras duas estatísticas:\n",
        "\n",
        "* $\\text{tf}(t, d)$, ou _term frequency_, é uma medida de quantas vezes o termo $t$ aparece no documento $d$. Algumas opções estão disponíveis, mas a mais simples é a contagem do número de ocorrências do termo no documento, $f_{t, d}$, exatamente o que computamos acima. Essa é a forma como sklearn define $tf$:\n",
        "\n",
        "$$\\text{tf}(t, d) = f_{t, d}$$\n",
        "\n",
        "* $\\text{idf}(t)$, ou _inverse document frequency_, é uma medida de relevância do termo em todos documentos do _corpus_. O sklearn a computa, seguindo valores _default_, da seguinte forma:\n",
        "\n",
        "$$\\text{idf}(t) = \\log{\\frac{1+n}{1 + d_{t}}} + 1$$\n",
        "\n",
        "onde $n$ é o número de documentos no _corpus_ e $d_{t}$ é o número de documentos no _corpus_ que contêm o termo $t$ ($0 < d_{t} \\leq n$).\n",
        "\n",
        "O tf-idf é calculado multiplicando esses dois valores:\n",
        "\n",
        "$$\\text{tf-idf}(t, d) = \\text{tf}(t, d) \\times \\text{idf}(t) = f_{t, d} \\times \\log{\\frac{1+n}{1 + d_{t}}} + 1$$\n",
        "\n",
        "O sklearn também normaliza todos documentos resultantes, ou seja todas linhas da matriz, para terem norma unitária. Em outras palavras, os elementos do vetor de tf-idf do documento $i$ são dados por:\n",
        "\n",
        "$$\\text{tf-idf}(i, j)_{\\text{normalizado}} = \\frac{\\text{tf-idf}(i, j)}{\\sqrt{\\text{tf-idf}(i, 1)^{2} + \\text{tf-idf}(i, 2)^{2} + \\cdots + \\text{tf-idf}(i, T)^{2}}}$$\n",
        "\n",
        "onde $T$ é o número de termos do _corpus_, ou seja, o tamanho do vocabulário."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bWpYWUMjCH8l",
        "colab_type": "text"
      },
      "source": [
        "O tf-idf é sempre um valor não negativo e quanto mais alto, maior a relevância do termo.\n",
        "\n",
        "Note como o tf aumenta de acordo com o número de ocorrências do termo no documento: quanto mais frequente o termo, mas relevante ele parece ser.\n",
        "\n",
        "O idf é uma medida de \"raridade\" do termo através de todo _corpus_: quanto mais alto, menos o termo aparece no _corpus_ e consequentemente mais informação ele traz.\n",
        "\n",
        "Multiplicando os dois, temos uma medida do quão relevante aquele termo é para aquele documento no _corpus_."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b_N2VQnwDaey",
        "colab_type": "text"
      },
      "source": [
        "O sklearn provê um transformador, `TfidfTransformer`, que transforma de uma matriz de frequências, como a retornada pelo `CountVectorizer`, e retorna uma matriz de tf-idf:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fyxgx0YhVwtF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tfidf_transformer = TfidfTransformer()\n",
        "\n",
        "tfidf_transformer.fit(newsgroups_counts)\n",
        "\n",
        "newsgroups_tfidf = tfidf_transformer.transform(newsgroups_counts)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "evk8smtLWNtO",
        "colab_type": "code",
        "outputId": "bf99b51a-e276-480c-dee9-13713e85a00b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "pd.DataFrame(newsgroups_tfidf[:5, words_idx].toarray(), columns=np.array(count_vectorizer.get_feature_names())[words_idx])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>banks</th>\n",
              "      <th>business</th>\n",
              "      <th>clipper</th>\n",
              "      <th>colorado</th>\n",
              "      <th>gibberish</th>\n",
              "      <th>group</th>\n",
              "      <th>kapor</th>\n",
              "      <th>monitor</th>\n",
              "      <th>private</th>\n",
              "      <th>study</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.081293</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.096368</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.109894</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.179352</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.148152</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.048551</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.145223</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.083477</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.117248</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.131568</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      banks  business   clipper  ...   monitor   private     study\n",
              "0  0.000000  0.000000  0.081293  ...  0.000000  0.000000  0.000000\n",
              "1  0.000000  0.000000  0.000000  ...  0.179352  0.000000  0.000000\n",
              "2  0.148152  0.000000  0.000000  ...  0.000000  0.048551  0.000000\n",
              "3  0.000000  0.000000  0.000000  ...  0.000000  0.000000  0.083477\n",
              "4  0.000000  0.117248  0.000000  ...  0.000000  0.000000  0.000000\n",
              "\n",
              "[5 rows x 10 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h9hI18kYDsuA",
        "colab_type": "text"
      },
      "source": [
        "Também podemos obter a matriz de tf-idf diretamente do _corpus_ sem ter que passar pela matriz de frequência com o transformador `TfidfVectorizer`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wPV4xrxzWlA-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tfidf_vectorizer = TfidfVectorizer()\n",
        "\n",
        "tfidf_vectorizer.fit(newsgroups.data)\n",
        "\n",
        "newsgroups_tfidf_vectorized = tfidf_vectorizer.transform(newsgroups.data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WAQ20ew-Wx5V",
        "colab_type": "code",
        "outputId": "fd781f7a-198a-444f-bfb8-baee26469ef0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "pd.DataFrame(newsgroups_tfidf_vectorized[:5, words_idx].toarray(), columns=np.array(count_vectorizer.get_feature_names())[words_idx])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>banks</th>\n",
              "      <th>business</th>\n",
              "      <th>clipper</th>\n",
              "      <th>colorado</th>\n",
              "      <th>gibberish</th>\n",
              "      <th>group</th>\n",
              "      <th>kapor</th>\n",
              "      <th>monitor</th>\n",
              "      <th>private</th>\n",
              "      <th>study</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.081293</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.096368</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.109894</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.179352</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.148152</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.048551</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.145223</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.083477</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.117248</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.131568</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      banks  business   clipper  ...   monitor   private     study\n",
              "0  0.000000  0.000000  0.081293  ...  0.000000  0.000000  0.000000\n",
              "1  0.000000  0.000000  0.000000  ...  0.179352  0.000000  0.000000\n",
              "2  0.148152  0.000000  0.000000  ...  0.000000  0.048551  0.000000\n",
              "3  0.000000  0.000000  0.000000  ...  0.000000  0.000000  0.083477\n",
              "4  0.000000  0.117248  0.000000  ...  0.000000  0.000000  0.000000\n",
              "\n",
              "[5 rows x 10 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RLFGR7A_D0px",
        "colab_type": "text"
      },
      "source": [
        "Note como a matriz acima é exatamente igual a retornada pelo `TfidfTransformer`.\n",
        "\n",
        "O resultado (igual da matriz de frequência) é um _data set_ com 1782 documentos e 33796 termos:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8I_w7yLeYnRe",
        "colab_type": "code",
        "outputId": "e1162574-03a2-4368-c3b6-517759bb973f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "newsgroups_tfidf_vectorized.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1782, 33796)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NjPMTtkUwrS1",
        "colab_type": "text"
      },
      "source": [
        "## Referências\n",
        "\n",
        "* [Feature engineering](https://jakevdp.github.io/PythonDataScienceHandbook/05.04-feature-engineering.html)\n",
        "\n",
        "* [Feature Scaling with scikit-learn](http://benalexkeen.com/feature-scaling-with-scikit-learn/)\n",
        "\n",
        "* [Anthony Goldbloom gives you the secret to winning Kaggle competitions](https://www.import.io/post/how-to-win-a-kaggle-competition/)\n",
        "\n",
        "* [What are some best practices in Feature Engineering?](https://www.quora.com/What-are-some-best-practices-in-Feature-Engineering)\n",
        "\n",
        "* [Discover Feature Engineering, How to Engineer Features and How to Get Good at It](https://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/)\n",
        "\n",
        "* [Fundamental Techniques of Feature Engineering for Machine Learning](https://towardsdatascience.com/feature-engineering-for-machine-learning-3a5e293a5114)\n",
        "\n",
        "* [Feature Engineering Cookbook for Machine Learning](https://medium.com/@michaelabehsera/feature-engineering-cookbook-for-machine-learning-7bf21f0bcbae)\n",
        "\n",
        "* [A Simple Guide to Scikit-learn Pipelines](https://medium.com/vickdata/a-simple-guide-to-scikit-learn-pipelines-4ac0d974bdcf)\n",
        "\n",
        "* [Outlier detection with Scikit Learn](https://www.mikulskibartosz.name/outlier-detection-with-scikit-learn/)\n",
        "\n",
        "* [Working With Text Data](https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html)\n",
        "\n",
        "* [WTF is TF-IDF?](https://www.kdnuggets.com/2018/08/wtf-tf-idf.html)\n"
      ]
    }
  ]
}