examples/protodash/Protodash-CDC.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Protodash: NHANES (CDC) data example\n",
    "- This notebook shows an example of how to use the ProtodashExplainer defined in [AIX360](https://github.com/IBM/AIX360/) to generate prototypes from (training/test) data. The notebook uses one of the [NHANES CDC questionnaire dataset](https://wwwn.cdc.gov/nchs/nhanes/search/datapage.aspx?Component=Questionnaire&CycleBeginYear=2013) related to incomes of individuals.\n",
    "- ProtodashExplainer is an implementation of the [Protodash algorithm](https://arxiv.org/abs/1707.01212)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Protodash Explainer examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import statements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "from aix360.algorithms.protodash import ProtodashExplainer, get_Gaussian_Data\n",
    "from aix360.datasets import CDCDataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load NHANES dataset from CDC "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "nhanes = CDCDataset()\n",
    "nhanes_files = nhanes.get_csv_file_names()\n",
    "(nhanesinfo, _, _) = nhanes._cdc_files_info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"explore\"></a>\n",
    "#### Explore NHANES Income questionnaire dataset\n",
    "\n",
    "Now let us explore the income questionnaire dataset and find out the types of responses received in the survey. Each column in this dataset corresponds to a question and each row denotes the answers given by a respondent to those questions. Both column names and answers by respondents are encoded. For example, 'SEQN' denotes the sequence number assigned to a respondent and 'IND235' corresponds to a question about monthly family income. As seen below, in most cases a value of 1 implies \"Yes\" to the question, while a value of 2 implies \"No\". More details about the income questionaire and how questions and answers are encoded can be seen [here](https://wwwn.cdc.gov/Nchs/Nhanes/2013-2014/INQ_H.htm)\n",
    "\n",
    "|Column  |Description                    | Values and Meaning|\n",
    "|-------|----------------------------|---------|\n",
    "|SEQN   | Respondent sequence number |\n",
    "|INQ020 | Income from wages/salaries |1->Yes, 2->No, 7->Refused, 9->Don't know|\n",
    "|INQ012 | Income from self employment|1->Yes, 2->No, 7->Refused, 9->Don't know|\n",
    "|INQ030 | Income from Social Security or RR |1->Yes, 2->No, 7->Refused, 9->Don't know|\n",
    "|INQ060 | Income from other disability pension |1->Yes, 2->No, 7->Refused, 9->Don't know|\n",
    "|INQ080 | Income from retirement/survivor pension |1->Yes, 2->No, 7->Refused, 9->Don't know|\n",
    "|INQ090 | Income from Supplemental Security Income |1->Yes, 2->No, 7->Refused, 9->Don't know|\n",
    "|INQ132 | Income from state/county cash assistance |1->Yes, 2->No, 7->Refused, 9->Don't know|\n",
    "|INQ140 | Income from interest/dividends or rental |1->Yes, 2->No, 7->Refused, 9->Don't know|\n",
    "|INQ150 | Income from other sources |1->Yes, 2->No, 7->Refused, 9->Don't know|\n",
    "|IND235 | Monthly family income |1-12->Increasing income brackets, 77->Refused, 99->Don't know|\n",
    "|INDFMMPI | Family monthly poverty level index |0-5->Higher value more affluent|\n",
    "|INDFMMPC | Family monthly poverty level category |1-3->Increasing INDFMMPI brackets, 7->Refused, 9->Don't know|\n",
    "|INQ244 | Family has savings more than $5000 |1->Yes, 2->No, 7->Refused, 9->Don't know|\n",
    "|IND247 | Total savings/cash assets for the family |1-6->Increasing savings brackets, 77->Refused, 99->Don't know|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answers given by some respondents to the income questionnaire:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Respondent sequence number</th>\n",
       "      <th>Income from wages/salaries</th>\n",
       "      <th>Income from self employment</th>\n",
       "      <th>Income from Social Security or RR</th>\n",
       "      <th>Income from other disability pension</th>\n",
       "      <th>Income from retirement/survivor pension</th>\n",
       "      <th>Income from Supplemental Security Income</th>\n",
       "      <th>Income from state/county cash assistance</th>\n",
       "      <th>Income from interest/dividends or rental</th>\n",
       "      <th>Income from other sources</th>\n",
       "      <th>Monthly family income</th>\n",
       "      <th>Family monthly poverty level index</th>\n",
       "      <th>Family monthly poverty level category</th>\n",
       "      <th>Family has savings more than $5000</th>\n",
       "      <th>Total savings/cash assets for the family</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>73557.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.86</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>73558.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.92</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>73559.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>4.37</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>73560.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2.52</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>73561.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>5.00</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Respondent sequence number  Income from wages/salaries  \\\n",
       "0                     73557.0                         2.0   \n",
       "1                     73558.0                         1.0   \n",
       "2                     73559.0                         2.0   \n",
       "3                     73560.0                         1.0   \n",
       "4                     73561.0                         2.0   \n",
       "\n",
       "   Income from self employment  Income from Social Security or RR  \\\n",
       "0                          2.0                                1.0   \n",
       "1                          1.0                                1.0   \n",
       "2                          2.0                                1.0   \n",
       "3                          2.0                                2.0   \n",
       "4                          2.0                                1.0   \n",
       "\n",
       "   Income from other disability pension  \\\n",
       "0                                   2.0   \n",
       "1                                   2.0   \n",
       "2                                   2.0   \n",
       "3                                   2.0   \n",
       "4                                   2.0   \n",
       "\n",
       "   Income from retirement/survivor pension  \\\n",
       "0                                      2.0   \n",
       "1                                      2.0   \n",
       "2                                      1.0   \n",
       "3                                      2.0   \n",
       "4                                      2.0   \n",
       "\n",
       "   Income from Supplemental Security Income  \\\n",
       "0                                       2.0   \n",
       "1                                       2.0   \n",
       "2                                       2.0   \n",
       "3                                       2.0   \n",
       "4                                       2.0   \n",
       "\n",
       "   Income from state/county cash assistance  \\\n",
       "0                                       2.0   \n",
       "1                                       2.0   \n",
       "2                                       2.0   \n",
       "3                                       2.0   \n",
       "4                                       2.0   \n",
       "\n",
       "   Income from interest/dividends or rental  Income from other sources  \\\n",
       "0                                       2.0                        2.0   \n",
       "1                                       1.0                        2.0   \n",
       "2                                       1.0                        2.0   \n",
       "3                                       2.0                        1.0   \n",
       "4                                       2.0                        2.0   \n",
       "\n",
       "   Monthly family income  Family monthly poverty level index  \\\n",
       "0                    4.0                                0.86   \n",
       "1                    5.0                                0.92   \n",
       "2                   10.0                                4.37   \n",
       "3                    9.0                                2.52   \n",
       "4                   11.0                                5.00   \n",
       "\n",
       "   Family monthly poverty level category  Family has savings more than $5000  \\\n",
       "0                                    1.0                                 9.0   \n",
       "1                                    1.0                                 1.0   \n",
       "2                                    3.0                                 NaN   \n",
       "3                                    3.0                                 NaN   \n",
       "4                                    3.0                                 NaN   \n",
       "\n",
       "   Total savings/cash assets for the family  \n",
       "0                                       NaN  \n",
       "1                                       NaN  \n",
       "2                                       NaN  \n",
       "3                                       NaN  \n",
       "4                                       NaN  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# replace encoded column names by the associated question text. \n",
    "df_inc = nhanes.get_csv_file('INQ_H.csv')\n",
    "df_inc.columns[0]\n",
    "dict_inc = {\n",
    "'SEQN': 'Respondent sequence number', \n",
    "'INQ020': 'Income from wages/salaries',\n",
    "'INQ012': 'Income from self employment',\n",
    "'INQ030':'Income from Social Security or RR',\n",
    "'INQ060':  'Income from other disability pension', \n",
    "'INQ080':  'Income from retirement/survivor pension',\n",
    "'INQ090':  'Income from Supplemental Security Income',\n",
    "'INQ132':  'Income from state/county cash assistance', \n",
    "'INQ140':  'Income from interest/dividends or rental', \n",
    "'INQ150':  'Income from other sources',\n",
    "'IND235':  'Monthly family income',\n",
    "'INDFMMPI':  'Family monthly poverty level index', \n",
    "'INDFMMPC':  'Family monthly poverty level category',\n",
    "'INQ244':  'Family has savings more than $5000',\n",
    "'IND247':  'Total savings/cash assets for the family'\n",
    "}\n",
    "qlist = []\n",
    "for i in range(len(df_inc.columns)):\n",
    "    qlist.append(dict_inc[df_inc.columns[i]])\n",
    "df_inc.columns = qlist\n",
    "print(\"Answers given by some respondents to the income questionnaire:\")\n",
    "df_inc.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of respondents to Income questionnaire: 10175\n",
      "Distribution of answers to 'monthly family income' and 'Family savings' questions:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlwAAAE8CAYAAAAVAG93AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhAElEQVR4nO3df7RdZX3n8feXRFC0kiBphiahYWqsxc6AmIF0aKeMWAg/lqEdi2BHItJJZxVbO7aV0LqGLpVZsdMllWmlzZho6ChIUYdUsJAi1OW0IAlQfgUkxWCS8iMawLa02uB3/thP4Hi9N/fcu89z7jkn79daZ929n72fvZ99z93nfO7ez947MhNJkiTVc9BMN0CSJGnUGbgkSZIqM3BJkiRVZuCSJEmqzMAlSZJUmYFLkiSpMgOXJFUSEdsj4r6IuCciNpeywyNiU0Q8Un7OLeUREVdExLaIuDciju9Yzsoy/yMRsXKmtkfS9MUg34friCOOyMWLF890M6QD0pYtW76RmfNmuh3DLCK2A0sz8xsdZb8L7MnMNRGxGpibmRdHxBnArwBnACcCH8nMEyPicGAzsBRIYAvwhsx8eqL1+tkpzYz9fW7O7ndjpmLx4sVs3rx5ppshHZAi4rGZbsOIWgGcXIY3ALcBF5fyq7L5L/j2iJgTEUeWeTdl5h6AiNgELAeunmgFfnZKM2N/n5ueUpSkehK4OSK2RMSqUjY/Mx8vw08A88vwAmBHR92dpWyi8u8REasiYnNEbN69e3cvt0FSDwz0ES5JGnI/mZm7IuIHgU0R8VDnxMzMiOhJv47MXAusBVi6dOng9hWRDlAe4ZKkSjJzV/n5FPA54ATgyXKqkPLzqTL7LmBRR/WFpWyicklDxMAlSRVExMsj4gf2DQOnAvcDG4F9VxquBK4vwxuB88vVisuAZ8upx5uAUyNibrmi8dRSJmmIeEpRkuqYD3wuIqD5rP1UZv55RNwJXBsRFwKPAeeU+W+kuUJxG/AccAFAZu6JiA8Ad5b53r+vA72k4WHgkqQKMvNR4Nhxyr8JnDJOeQIXTbCs9cD6XrdRUv94SlGSJKkyA5ckSVJlBi5JkqTKDFySJEmVGbgkSZIqG6qrFBevvmG/07evObNPLZGk0TPZZ2yv+ZmtA8mkR7giYn1EPBUR93eU/c+IeCgi7o2Iz0XEnI5pl0TEtoh4OCJO6yhfXsq2RcTqnm+JJEnSgOrmlOInaJ5M32kT8OOZ+W+BrwKXAETEMcC5wOtKnY9GxKyImAX8IXA6cAxwXplXkiRp5E0auDLzS8CeMWU3Z+beMno7zbO9AFYA12TmtzPzazR3TD6hvLZl5qOZ+R3gmjKvJEnSyOtFp/l3Al8owwuAHR3TdpayicolSZJGXqvAFRG/DewFPtmb5kBErIqIzRGxeffu3b1arCRJ0oyZduCKiHcAZwG/UJ4BBrALWNQx28JSNlH598nMtZm5NDOXzps3b7rNkyRJGhjTClwRsRx4L/DmzHyuY9JG4NyIOCQijgaWAF+hecr9kog4OiIOpulYv7Fd0yVJkobDpPfhioirgZOBIyJiJ3ApzVWJhwCbIgLg9sz8r5n5QERcCzxIc6rxosx8viznXcBNwCxgfWY+UGF7JEmSBs6kgSszzxuneN1+5r8MuGyc8huBG6fUOkmSpBHgo30kSZIqM3BJkiRVZuCSJEmqzMAlSZJUmYFLkiSpMgOXJElSZQYuSZKkygxckiRJlRm4JEmSKjNwSZIkVWbgkiRJqszAJUmSVJmBS5IkqTIDlyRJUmUGLkmSpMoMXJIkSZUZuCRJkiozcEmSJFVm4JIkSarMwCVJklSZgUuSJKkyA5ckSVJlBi5JkqTKDFySJEmVGbgkSZIqM3BJkiRVZuCSJEmqzMAlSZJUmYFLkiSpMgOXJElSZQYuSaooImZFxN0R8fkyfnRE3BER2yLi0xFxcCk/pIxvK9MXdyzjklL+cEScNkObIqkFA5ck1fVuYGvH+IeAyzPz1cDTwIWl/ELg6VJ+eZmPiDgGOBd4HbAc+GhEzOpT2yX1iIFLkiqJiIXAmcDHyngAbwSuK7NsAM4uwyvKOGX6KWX+FcA1mfntzPwasA04oS8bIKlnDFySVM/vA+8FvlvGXwU8k5l7y/hOYEEZXgDsACjTny3zv1A+Tp0XRMSqiNgcEZt3797d482Q1NbsyWaIiPXAWcBTmfnjpexw4NPAYmA7cE5mPl3+G/sIcAbwHPCOzLyr1FkJvK8s9oOZuYE+W7z6hv1O377mzD61RNKoi4h9n5tbIuLk2uvLzLXAWoClS5dm7fVJmppJAxfwCeAPgKs6ylYDt2TmmohYXcYvBk4HlpTXicCVwIkloF0KLAUS2BIRGzPz6V5tSD8Y2CRNwUnAmyPiDOClwCtp/iGdExGzy1GshcCuMv8uYBGwMyJmA4cB3+wo36ezjqQhMekpxcz8ErBnTHFnX4OxfRCuysbtNB8sRwKnAZsyc08JWZtoOn9K0kjKzEsyc2FmLqbp9P7FzPwF4FbgLWW2lcD1ZXhjGadM/2JmZik/t1zFeDTNP7Rf6dNmSOqRbo5wjWd+Zj5ehp8A5pfhifoadNUHAZp+CMAqgKOOOmqazZOkgXUxcE1EfBC4G1hXytcBfxIR22j+yT0XIDMfiIhrgQeBvcBFmfl8/5stqY3pBq4XZGZGRM/6C9gPQdKoyczbgNvK8KOMc5VhZv4z8PMT1L8MuKxeCyXVNt2rFJ8spwopP58q5RP1NbAPgiRJOmBNN3B19jUY2wfh/GgsA54tpx5vAk6NiLkRMRc4tZRJkiSNvG5uC3E1cDJwRETspLnacA1wbURcCDwGnFNmv5HmlhDbaG4LcQFAZu6JiA8Ad5b53p+ZYzvijzyvcpQk6cA0aeDKzPMmmHTKOPMmcNEEy1kPrJ9S6yRJkkaAd5qXJEmqzMAlSZJUmYFLkiSpMgOXJElSZQYuSZKkygxckiRJlRm4JEmSKjNwSZIkVWbgkiRJqszAJUmSVJmBS5IkqTIDlyRJUmUGLkmSpMoMXJIkSZUZuCRJkiozcEmSJFVm4JIkSarMwCVJklSZgUuSJKkyA5ckSVJlBi5JkqTKDFySJEmVGbgkSZIqmz3TDVD3Fq++Yb/Tt685s08tkSRJU+ERLkmSpMoMXJIkSZUZuCRJkiozcEmSJFVm4JIkSarMwCVJklSZgUuSJKkyA5ckSVJlBi5JkqTKWgWuiPhvEfFARNwfEVdHxEsj4uiIuCMitkXEpyPi4DLvIWV8W5m+uCdbIEmSNOCmHbgiYgHwq8DSzPxxYBZwLvAh4PLMfDXwNHBhqXIh8HQpv7zMJ0mSNPLanlKcDbwsImYDhwKPA28ErivTNwBnl+EVZZwy/ZSIiJbrlyRJGnjTDlyZuQv4PeDrNEHrWWAL8Exm7i2z7QQWlOEFwI5Sd2+Z/1VjlxsRqyJic0Rs3r1793SbJ0mSNDDanFKcS3PU6mjgh4CXA8vbNigz12bm0sxcOm/evLaLkyRJmnFtTim+CfhaZu7OzH8BPgucBMwppxgBFgK7yvAuYBFAmX4Y8M0W65ckSRoKsyefZUJfB5ZFxKHAPwGnAJuBW4G3ANcAK4Hry/wby/hfl+lfzMxssX5N0eLVN+x3+vY1Z/apJZIkHVja9OG6g6bz+13AfWVZa4GLgfdExDaaPlrrSpV1wKtK+XuA1S3aLUmSNDTaHOEiMy8FLh1T/Chwwjjz/jPw823WJ0nDIiJeCnwJOITms/a6zLw0Io6mOQPwKpoLjd6emd+JiEOAq4A30HS3eGtmbi/LuoTm1jrPA7+amTf1e3skteOd5iWpjm8Db8zMY4HjgOURsYwp3qswIo6hucfh62guTPpoRMzq54ZIaq/VES4dWOwDJnWv9FH9hzL6kvJKmnsVvq2UbwB+B7iS5qrv3ynl1wF/UO5VuAK4JjO/DXytdMs4gaY/rKQh4REuSaokImZFxD3AU8Am4G+Z+r0KXygfp07nuryHoTTADFySVElmPp+Zx9HcIucE4LUV1+U9DKUBZuCSpMoy8xmaW+b8BFO/V+EL5ePUkTQkDFySVEFEzIuIOWX4ZcDPAFt58V6FMP69CuF771W4ETg3Ig4pVzguAb7Sl42Q1DN2mpekOo4ENpQrCg8Crs3Mz0fEg8A1EfFB4G6+916Ff1I6xe+huTKRzHwgIq4FHgT2Ahdl5vN93hZJLRm4JKmCzLwXeP045VO+V2FmXgZc1us2SuofTylKkiRVZuCSJEmqzMAlSZJUmYFLkiSpMjvNq298NJAk6UDlES5JkqTKDFySJEmVGbgkSZIqM3BJkiRVZuCSJEmqzMAlSZJUmYFLkiSpMgOXJElSZQYuSZKkygxckiRJlRm4JEmSKjNwSZIkVWbgkiRJqszAJUmSVJmBS5IkqTIDlyRJUmUGLkmSpMoMXJIkSZUZuCRJkiozcEmSJFXWKnBFxJyIuC4iHoqIrRHxExFxeERsiohHys+5Zd6IiCsiYltE3BsRx/dmEyRJkgZb2yNcHwH+PDNfCxwLbAVWA7dk5hLgljIOcDqwpLxWAVe2XLckSdJQmHbgiojDgP8ArAPIzO9k5jPACmBDmW0DcHYZXgFclY3bgTkRceR01y9JkjQs2hzhOhrYDXw8Iu6OiI9FxMuB+Zn5eJnnCWB+GV4A7Oiov7OUSZIkjbQ2gWs2cDxwZWa+HvhHXjx9CEBmJpBTWWhErIqIzRGxeffu3S2aJ0mSNBjaBK6dwM7MvKOMX0cTwJ7cd6qw/HyqTN8FLOqov7CUfY/MXJuZSzNz6bx581o0T5IkaTBMO3Bl5hPAjoj40VJ0CvAgsBFYWcpWAteX4Y3A+eVqxWXAsx2nHiVJkkbW7Jb1fwX4ZEQcDDwKXEAT4q6NiAuBx4Bzyrw3AmcA24DnyrySJEkjr1Xgysx7gKXjTDplnHkTuKjN+iRJkoaRd5qXJEmqzMAlSZJUmYFLkiSpMgOXJElSZQYuSZKkygxckiRJlRm4JEmSKmt741OpbxavvmG/07evObNPLZEkaWo8wiVJklSZgUuSJKkyA5ckSVJlBi5JqiAiFkXErRHxYEQ8EBHvLuWHR8SmiHik/JxbyiMiroiIbRFxb0Qc37GslWX+RyJi5Uxtk6TpM3BJUh17gV/PzGOAZcBFEXEMsBq4JTOXALeUcYDTgSXltQq4EpqABlwKnAicAFy6L6RJGh4GLkmqIDMfz8y7yvDfA1uBBcAKYEOZbQNwdhleAVyVjduBORFxJHAasCkz92Tm08AmYHn/tkRSLxi4JKmyiFgMvB64A5ifmY+XSU8A88vwAmBHR7WdpWyi8rHrWBURmyNi8+7du3u7AZJaM3BJUkUR8QrgM8CvZea3OqdlZgLZi/Vk5trMXJqZS+fNm9eLRUrqIQOXJFUSES+hCVufzMzPluIny6lCys+nSvkuYFFH9YWlbKJySUPEwCVJFUREAOuArZn54Y5JG4F9VxquBK7vKD+/XK24DHi2nHq8CTg1IuaWzvKnljJJQ8RH++iA4aOB1GcnAW8H7ouIe0rZbwFrgGsj4kLgMeCcMu1G4AxgG/AccAFAZu6JiA8Ad5b53p+Ze/qyBZJ6xsAlSRVk5peBmGDyKePMn8BFEyxrPbC+d62T1G+eUpQkSarMwCVJklSZgUuSJKkyA5ckSVJldpqXuuRVjpKk6fIIlyRJUmUGLkmSpMoMXJIkSZUZuCRJkiozcEmSJFVm4JIkSarMwCVJklSZgUuSJKkyA5ckSVJlrQNXRMyKiLsj4vNl/OiIuCMitkXEpyPi4FJ+SBnfVqYvbrtuSZKkYdCLI1zvBrZ2jH8IuDwzXw08DVxYyi8Eni7ll5f5JEmSRl6rwBURC4EzgY+V8QDeCFxXZtkAnF2GV5RxyvRTyvySJEkjre3Dq38feC/wA2X8VcAzmbm3jO8EFpThBcAOgMzcGxHPlvm/0bnAiFgFrAI46qijWjZPGhw+/FqSDlzTPsIVEWcBT2Xmlh62h8xcm5lLM3PpvHnzerloSZKkGdHmCNdJwJsj4gzgpcArgY8AcyJidjnKtRDYVebfBSwCdkbEbOAw4Jst1i9JkjQUpn2EKzMvycyFmbkYOBf4Ymb+AnAr8JYy20rg+jK8sYxTpn8xM3O665ckSRoWNe7DdTHwnojYRtNHa10pXwe8qpS/B1hdYd2SJEkDp22neQAy8zbgtjL8KHDCOPP8M/DzvVifJEnSMPFO85IkSZUZuCRJkiozcEmSJFVm4JIkSarMwCVJklSZgUuSJKkyA5ckSVJlBi5JkqTKDFySJEmVGbgkSZIqM3BJkiRV1pNnKUqqb/HqG/Y7ffuaM/vUEknSVHmES5IkqTIDlyRJUmUGLkmSpMoMXJIkSZUZuCRJkiozcEmSJFVm4JIkSarMwCVJklSZgUuSJKkyA5ckSVJlBi5JqiAi1kfEUxFxf0fZ4RGxKSIeKT/nlvKIiCsiYltE3BsRx3fUWVnmfyQiVs7Etkhqz8AlSXV8Alg+pmw1cEtmLgFuKeMApwNLymsVcCU0AQ24FDgROAG4dF9IkzRcDFySVEFmfgnYM6Z4BbChDG8Azu4ovyobtwNzIuJI4DRgU2buycyngU18f4iTNAQMXJLUP/Mz8/Ey/AQwvwwvAHZ0zLezlE1U/n0iYlVEbI6Izbt37+5tqyW1ZuCSpBmQmQlkD5e3NjOXZubSefPm9WqxknrEwCVJ/fNkOVVI+flUKd8FLOqYb2Epm6hc0pAxcElS/2wE9l1puBK4vqP8/HK14jLg2XLq8Sbg1IiYWzrLn1rKJA2Z2TPdAEn9sXj1Dfudvn3NmX1qyYEhIq4GTgaOiIidNFcbrgGujYgLgceAc8rsNwJnANuA54ALADJzT0R8ALizzPf+zBzbEV/SEDBwSVIFmXneBJNOGWfeBC6aYDnrgfU9bJqkGeApRUmSpMoMXJIkSZVNO3BFxKKIuDUiHoyIByLi3aV8yo+ukCRJGmVtjnDtBX49M48BlgEXRcQxTPHRFZIkSaNu2oErMx/PzLvK8N8DW2nugDzVR1dIkiSNtJ704YqIxcDrgTuY+qMrxi7Lx1NIkqSR0jpwRcQrgM8Av5aZ3+qcNp1HV/h4CkmSNGpaBa6IeAlN2PpkZn62FE/10RWSJEkjrc1VigGsA7Zm5oc7Jk310RWSJEkjrc2d5k8C3g7cFxH3lLLfYoqPrpAkSRp10w5cmfllICaYPKVHV0iSJI0y7zQvSZJUmYFLkiSpMgOXJElSZQYuSZKkygxckiRJlRm4JEmSKjNwSZIkVWbgkiRJqszAJUmSVJmBS5IkqbI2z1KUJGloLF59Q1/Xt33NmX1dnwabR7gkSZIqM3BJkiRVZuCSJEmqzMAlSZJUmYFLkiSpMgOXJElSZQYuSZKkygxckiRJlRm4JEmSKjNwSZIkVWbgkiRJqszAJUmSVJmBS5IkqTIDlyRJUmUGLkmSpMoMXJIkSZUZuCRJkiozcEmSJFU2e6YbIEmS2lu8+oa+rm/7mjP7ur5h5xEuSZKkygxckiRJlXlKUZIkDbx+njKtcbq070e4ImJ5RDwcEdsiYnW/1y9Jw8jPTmm49TVwRcQs4A+B04FjgPMi4ph+tkGSho2fndLw6/cRrhOAbZn5aGZ+B7gGWNHnNkjSsPGzUxpy/Q5cC4AdHeM7S5kkaWJ+dkpDLjKzfyuLeAuwPDN/sYy/HTgxM9/VMc8qYFUZ/VHg4f0s8gjgGy2aZH3rW39iP5yZ81osXz1S4bOz19r+LQ46t2+49XP7Jvzc7PdViruARR3jC0vZCzJzLbC2m4VFxObMXDrdxljf+taffn31VU8/O3tt1P+W3L7hNijb1+9TincCSyLi6Ig4GDgX2NjnNkjSsPGzUxpyfT3ClZl7I+JdwE3ALGB9Zj7QzzZI0rDxs1Mafn2/8Wlm3gjc2KPFtT18bn3rW19Docefnb026n9Lbt9wG4jt62uneUmSpAORz1KUJEmqzMAlSZJUmYFLkiSpsr53mh8EEXE4QGbumUbd+bx4h+ddmfnkMNUvy5j29veifhuD8PuT5L407Eb5/RvUbTtgOs1HxFHA7wKnAM8AAbwS+CKwOjO3T1L/OOCPgMN48YaDC8uyfjkz7xrw+m23v1X9tgbg9/dammfXvbATAxszc+sUtuEwYPmYZdyUmc9MYRnT+iDpRfslaL8vDYtB/dJua5Tfv4HftswcqhfwWuBi4Iryuhj4sS7q/TXwVmBWR9ksmhsI3t5F/XtoHqUxtnwZ8DdDUL/t9req3+a9m+nfX2nnPcBq4D+X1+p9ZV22/3zgb4ErgfeV1x+VsvO7qH8ccDuwFfiL8nqolB1fu/2+fO17td0XB/3VZl8bhtcov3+Dvm1DdYQrIi4GzgOuoXl4KzTp9Vzgmsxcs5+6j2TmkqlO67L+tsx89RDXb7v93dSf9nvXxfqr/v4i4qvA6zLzX8aUHww8MNm2l3kfpvkgeGZM+Vzgjsx8zST17wF+KTPvGFO+DPjjzDy2Zvulfdrui4Ouzb42DEb5/Rv0bRu2PlwXMv4Xx4eBB4D9fWlviYiPAhuAHaVsEbASuLuLdX8hIm4ArhpT/3zgz4egftvtb1u/zXsHM/v7+y7wQ8BjY8qPLNO6EcB4/918t0ybzMvHfgEAZObtEfHySer2ov3SPm33xUHXZl8bBqP8/g30tg3bEa6HgNMy87Ex5T8M3JyZP7qfugfTfOl39mPZCfwZsC4zv93F+k9n/H4wXd39eSbrt93+HtSf9nvXMe+M/P4iYjnwB8AjvLgTHwW8GnhXZk66I0fESuC/AzePWcbPAB/IzE9MUv8K4EcY/4Pka5n5rprtlzq13RcHWZt9bViM+Ps3sNs2bIHLL44hNezvXUQcBJzA9+7Ed2bm81NYxlzgNL6/0/zTXdZvE7hbt186UAzyl7aG11AFLqjzxRERZ2Xm51vUX5WZ035W0wDUb7v9XdWv9aU/078/SQ33peE2yu/fIGzb0N34NDO/m5m3Z+Znyuv2HvyX/u9a1u+mD84g12+7/V3Vr/TewQz+/iJi2kG1YxmtPgQiYlWLuq3bL3Vouy8OtDb72pAY5fdvxrdt6I5wTSQiPp+ZZ00yzwlAZuadEXEMzT2RHurylMyvAp/LzB2TzTtB/ROBrZn5rYh4Gc1l+ccDDwL/IzOfneLyfpLmaNH9mXnzNNt0VWae3+W8B9NcUfh3mfkXEfE24N/TXDq9dmxn+Cm2Y9L3rsz3r4Gfo+lP8TzwVeBTmfmtLtfzWpqja3dk5j90lC+f7inNiDgyMx+fTt2OZbwhM7e0qP9LmfnH06zbuv068NTYl4ZBm31tULT9Lht0bb8nahqlwLXfL46IuBQ4nebKzE3AicCtNJ2Wb8rMyyZZ/rPAP9LcN+lq4E8zc/cU2vcAcGxm7i1HNJ4DrqO5keixmflzk9T/SmaeUIb/C3AR8DngVODPuritwsaxRcB/pLlxKZn55knqf5Lmd3cozU3kXgF8trSfzHzH/upPsuxJv/TLh8RZwJeAM2iujHwG+FmaG9rd1kX9i2gC4nHAuzPz+jLtrsw8frrtn2kRcUFmfnym26EDwyjvS5MZhX2t7XfZIGv7PVFdDsDNyvrxAu6juVHnocC3gFeW8pcB93ZR/26aU7CnAuuA3TSXma4EfqCL+ls7hu8aM+2ebtbfMXwnMK8Mvxy4r4v6dwH/BzgZ+Ony8/Ey/NNd1L+3/JwNPEm5ASpNcJv099er968MHwrcVoaP6vzdTFL/FWV4MbCZ5ouCyerThMv309y+4tny3t8OvGMK7T+M5tYXDwF7gG/SfGGtAea0/N18vYv3/n3Aj9R+n3yN/qvNvjTsr8n2tWF4tf0uG+RX2++J2q+hug9XRLwSuITmhplfyMxPdUz7aGb+8n6q782mv9BzEfG3WQ4vZuY/RUQ39yLKzPwuzWX9N0fES2iOmJ0H/B4wb5L693f8d/Q3EbE0MzdHxGuAbk7HHVSucjuI5sjk7tKof4yIvV3UXwq8G/ht4Dcz856I+KfM/Msu6u5b/8E0Ae9QmgCxBzgEeEmXyxhXRHwhM0/vYtbZNIeID6EJQWTm18t7MZmDspz6yMztEXEycF25LcVk5/Y/SXM08TTgHJrfwTXA+yLiNZn5W12s/1qao4knZ+YTABHxr2g+5K6l+fCbUETcO9EkYP4k654LzAFujYgnaP6r/XRm/l0X7ZbGarMvDbyW+9owaPtdNujafE9UNVSBC/g4zW0FPgO8MyL+E/C2bO4BtWySut+JiEMz8zngDfsKo3m+XTeB63s+SLLps7QR2BgRh3ZR/xeBj0TE+4BvAH8dETtobpHwi13UPwzYUtqR+07DRcQrxrZtPGUHuzwi/rT8fJKpvf/raI7OzKIJbX8aEY/S/N6vmaxyREx0miFoTktM5mPAnRFxB/BTwIfKcufRBL/JPBkRx2XmPQCZ+Q8RcRawHvg3k9RdnC/eJ+vDEXFnZn4gIi6g6YPXTeBanJkf6iwowetDEfHOLurPpwl8Y28hEcBfTVL36cz8DeA3IuKnaD5Y74qIrcDVOaJXJamaNvvSMGizrw2Dtt9lg6zt90RVQ9WHKyLuyczjOsZ/m+Y87ZuBTbmfvgMRcUiOc3POiDgCODIz75tk3a/JzK9Ou/EvLueVwNE0YWdntnwgatlB5mfm16ZY70zgpC6Pzuyr80MAmfl3ETEHeBPNIfavdFH3eeAvGT8cLsvMl3WxjNcBP0ZzocBD3ba71F1Ic5TziXGmnZSZ/28/df8KeG9mfjkiVtD0BTitTHs4u7tp6800z2TbsO89j+bhuO8AfiYz3zRJ/XXAxzPzy+NM+1Rmvm0/de/OzNePKZtF03/xrZl5wWTtl/Zpsy8Ngzb72jDo1XfZoGrzPVHbsAWurTSPh/luR9k7gN+k6VPwwzPVNu1fRNwP/GxmPjLOtB2ZuWgGmtWViDgW+N/AEpp+XO/MzK+W/5rOy8wruljGXJorU1cAP1iKn6T5z3JNdnnz0+mIiGsy89xay5ckTW7YAtfv0jwG5i/GlC8H/lf6EN6BFRFvoenc//A4087OzP/b/1Z1LyJ+hBcvNd5LDy81rn3l06hfBi5Jw2CoAtf+jMLlugeqQX/val9qHBFfz8yjWjZzf8sf2cvAJWlYjFLgqvqlpXoG/b2LiPuA4zLz+dJn7sbMPDkijgKuH9s/aoJl7O/Kp9dk5iE9bPLYdd9Nc6HIm4C30vR53EITvj6bmX9fa92SpMZQXaV4AFyuO7JG4L1re6nxTF75NOqXgUvSwBuqwMXoX647yob5vevFpcafp7mw456xEyLitt40c0KjfBm4JA2FoTqlOOqX646yYX/vBvlS48mM+mXgkjQMhipwSZIkDaODZroBkiRJo87AJUmSVJmBS5IkqTIDlyRJUmX/HxVHwhJkOxj/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Number of respondents to Income questionnaire:\", df_inc.shape[0])\n",
    "print(\"Distribution of answers to \\'monthly family income\\' and \\'Family savings\\' questions:\")\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10,5))\n",
    "fig.subplots_adjust(wspace=0.5)\n",
    "hist1 = df_inc['Monthly family income'].value_counts().plot(kind='bar', ax=axes[0])\n",
    "hist2 = df_inc['Family has savings more than $5000'].value_counts().plot(kind='bar', ax=axes[1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"study1\"></a>\n",
    "#### Summarize NHANES Income Questionnaire dataset using Prototypes\n",
    "\n",
    "Consider a social scientist who would like to quickly obtain a summary report of this dataset in terms of types of people that span this dataset. Is it possible to summarize this dataset by looking at answers given by a few representative/prototypical respondents? \n",
    "\n",
    "We now show how the ProtodashExplainer can be used to obtain a few prototypical respondents (about 10 in this example) that span the diverse set of individuals answering the income questionnaire making it easy for the social scientist to summarize the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert pandas dataframe to numpy\n",
    "data = df_inc.to_numpy()\n",
    "\n",
    "#sort the rows by sequence numbers in 1st column \n",
    "idx = np.argsort(data[:, 0])  \n",
    "data = data[idx, :]\n",
    "\n",
    "# replace nan's (missing values) with 0's\n",
    "original = data\n",
    "original[np.isnan(original)] = 0\n",
    "\n",
    "# delete 1st column (sequence numbers)\n",
    "original = original[:, 1:]\n",
    "\n",
    "# one hot encode all features as they are categorical\n",
    "onehot_encoder = OneHotEncoder(sparse=False)\n",
    "onehot_encoded = onehot_encoder.fit_transform(original)\n",
    "\n",
    "explainer = ProtodashExplainer()\n",
    "\n",
    "# call protodash explainer\n",
    "# S contains indices of the selected prototypes\n",
    "# W contains importance weights associated with the selected prototypes \n",
    "(W, S, _) = explainer.explain(onehot_encoded, onehot_encoded, m=10) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Respondent sequence number</th>\n",
       "      <th>Income from wages/salaries</th>\n",
       "      <th>Income from self employment</th>\n",
       "      <th>Income from Social Security or RR</th>\n",
       "      <th>Income from other disability pension</th>\n",
       "      <th>Income from retirement/survivor pension</th>\n",
       "      <th>Income from Supplemental Security Income</th>\n",
       "      <th>Income from state/county cash assistance</th>\n",
       "      <th>Income from interest/dividends or rental</th>\n",
       "      <th>Income from other sources</th>\n",
       "      <th>Monthly family income</th>\n",
       "      <th>Family monthly poverty level index</th>\n",
       "      <th>Family monthly poverty level category</th>\n",
       "      <th>Family has savings more than $5000</th>\n",
       "      <th>Total savings/cash assets for the family</th>\n",
       "      <th>Weights of Prototypes</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>73565.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>5.00</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7475</th>\n",
       "      <td>81032.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.08</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2449</th>\n",
       "      <td>76006.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2912</th>\n",
       "      <td>76469.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.32</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6895</th>\n",
       "      <td>80452.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.86</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3899</th>\n",
       "      <td>77456.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2.71</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1475</th>\n",
       "      <td>75032.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.65</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>690</th>\n",
       "      <td>74247.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>3.05</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5077</th>\n",
       "      <td>78634.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.44</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>73689.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>4.30</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Respondent sequence number  Income from wages/salaries  \\\n",
       "8                        73565.0                         1.0   \n",
       "7475                     81032.0                         2.0   \n",
       "2449                     76006.0                         1.0   \n",
       "2912                     76469.0                         1.0   \n",
       "6895                     80452.0                         1.0   \n",
       "3899                     77456.0                         1.0   \n",
       "1475                     75032.0                         1.0   \n",
       "690                      74247.0                         2.0   \n",
       "5077                     78634.0                         1.0   \n",
       "132                      73689.0                         1.0   \n",
       "\n",
       "      Income from self employment  Income from Social Security or RR  \\\n",
       "8                             2.0                                2.0   \n",
       "7475                          2.0                                1.0   \n",
       "2449                          1.0                                2.0   \n",
       "2912                          2.0                                2.0   \n",
       "6895                          2.0                                2.0   \n",
       "3899                          2.0                                1.0   \n",
       "1475                          2.0                                2.0   \n",
       "690                           2.0                                2.0   \n",
       "5077                          2.0                                2.0   \n",
       "132                           2.0                                2.0   \n",
       "\n",
       "      Income from other disability pension  \\\n",
       "8                                      2.0   \n",
       "7475                                   2.0   \n",
       "2449                                   2.0   \n",
       "2912                                   1.0   \n",
       "6895                                   2.0   \n",
       "3899                                   2.0   \n",
       "1475                                   2.0   \n",
       "690                                    2.0   \n",
       "5077                                   2.0   \n",
       "132                                    2.0   \n",
       "\n",
       "      Income from retirement/survivor pension  \\\n",
       "8                                         2.0   \n",
       "7475                                      2.0   \n",
       "2449                                      2.0   \n",
       "2912                                      1.0   \n",
       "6895                                      2.0   \n",
       "3899                                      2.0   \n",
       "1475                                      2.0   \n",
       "690                                       2.0   \n",
       "5077                                      1.0   \n",
       "132                                       2.0   \n",
       "\n",
       "      Income from Supplemental Security Income  \\\n",
       "8                                          2.0   \n",
       "7475                                       2.0   \n",
       "2449                                       2.0   \n",
       "2912                                       2.0   \n",
       "6895                                       1.0   \n",
       "3899                                       2.0   \n",
       "1475                                       2.0   \n",
       "690                                        2.0   \n",
       "5077                                       2.0   \n",
       "132                                        2.0   \n",
       "\n",
       "      Income from state/county cash assistance  \\\n",
       "8                                          2.0   \n",
       "7475                                       2.0   \n",
       "2449                                       2.0   \n",
       "2912                                       2.0   \n",
       "6895                                       1.0   \n",
       "3899                                       2.0   \n",
       "1475                                       2.0   \n",
       "690                                        2.0   \n",
       "5077                                       2.0   \n",
       "132                                        2.0   \n",
       "\n",
       "      Income from interest/dividends or rental  Income from other sources  \\\n",
       "8                                          2.0                        2.0   \n",
       "7475                                       1.0                        2.0   \n",
       "2449                                       2.0                        1.0   \n",
       "2912                                       2.0                        2.0   \n",
       "6895                                       2.0                        2.0   \n",
       "3899                                       2.0                        2.0   \n",
       "1475                                       1.0                        2.0   \n",
       "690                                        2.0                        2.0   \n",
       "5077                                       2.0                        2.0   \n",
       "132                                        1.0                        2.0   \n",
       "\n",
       "      Monthly family income  Family monthly poverty level index  \\\n",
       "8                      12.0                                5.00   \n",
       "7475                    3.0                                1.08   \n",
       "2449                    1.0                                0.00   \n",
       "2912                    6.0                                1.32   \n",
       "6895                    5.0                                0.86   \n",
       "3899                    7.0                                2.71   \n",
       "1475                    4.0                                1.65   \n",
       "690                     8.0                                3.05   \n",
       "5077                    2.0                                0.44   \n",
       "132                    11.0                                4.30   \n",
       "\n",
       "      Family monthly poverty level category  \\\n",
       "8                                       3.0   \n",
       "7475                                    1.0   \n",
       "2449                                    1.0   \n",
       "2912                                    2.0   \n",
       "6895                                    1.0   \n",
       "3899                                    3.0   \n",
       "1475                                    2.0   \n",
       "690                                     3.0   \n",
       "5077                                    1.0   \n",
       "132                                     3.0   \n",
       "\n",
       "      Family has savings more than $5000  \\\n",
       "8                                    NaN   \n",
       "7475                                 2.0   \n",
       "2449                                 2.0   \n",
       "2912                                 1.0   \n",
       "6895                                 2.0   \n",
       "3899                                 NaN   \n",
       "1475                                 2.0   \n",
       "690                                  NaN   \n",
       "5077                                 2.0   \n",
       "132                                  NaN   \n",
       "\n",
       "      Total savings/cash assets for the family  Weights of Prototypes  \n",
       "8                                          NaN                   0.18  \n",
       "7475                                       1.0                   0.10  \n",
       "2449                                       1.0                   0.15  \n",
       "2912                                       NaN                   0.07  \n",
       "6895                                       1.0                   0.09  \n",
       "3899                                       NaN                   0.12  \n",
       "1475                                       3.0                   0.06  \n",
       "690                                        NaN                   0.09  \n",
       "5077                                       1.0                   0.07  \n",
       "132                                        NaN                   0.07  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display the prototypes along with their computed weights\n",
    "inc_prototypes = df_inc.iloc[S, :].copy()\n",
    "# Compute normalized importance weights for prototypes\n",
    "inc_prototypes[\"Weights of Prototypes\"] = np.around(W/np.sum(W), 2) \n",
    "inc_prototypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation:\n",
    "The 10 people shown above (i.e. 5 prototypes) are representative of the income questionnaire according to Protodash. Firstly, in the distribution plot for family finance related questions we saw that there roughly were 5 times as many people not having savings in excess of $5000 compared with others. Our prototypes also have a similar spread which is reassuring. Also for monthly family income we get a more even spread over the more commonly occuring categories. This is kind of a spot check to see if our prototypes actually match the distribution of values in the dataset.\n",
    "\n",
    "Looking at the other questions in the questionnaire and the corresponding answers given by the prototypical people above the social scientist realizes that most people are employeed (3rd question) and work for an organization earning through salary/wages (1st two questions). Most of them are also young (5th question) and fit to work (4th question). However, they don't seem to have much savings (last question). These insights that the social scientist has acquired from studying the prototypes could be conveyed also to the appropriate government authorities that affect future public policy decisions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Summarize Gaussian (simulated) data using prototypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(300, 100) (4000, 100)\n"
     ]
    }
   ],
   "source": [
    "# generate normalized gaussian data X, Y with 100 features and 300 & 4000 observations respectively\n",
    "(X, Y) = get_Gaussian_Data(100, 300, 4000)\n",
    "print(X.shape, Y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "(W, S, setValues) = explainer.explain(X, Y, m=5, kernelType='Gaussian', sigma=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3940 2539 2168 2189 1170] [0.20611992 0.24524021 0.19131892 0.17175183 0.16123833]\n"
     ]
    }
   ],
   "source": [
    "print(S, W)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Print prototypes\n",
    "#print(Y[S, :]) "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}