13_ensemble/13_02_ensemble.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification of Newsgroups by Text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Newsgroups Data\n",
    "\n",
    "We will take a look at some of the twenty newsgroups dataset, another common dataset for classification.\n",
    "\n",
    "More specifically, it is a list of emails within 20 different newsgroups. We will just focus on a few categories in particular. \n",
    "\n",
    "There will most likely be some difficulties or misclassification between emails in `atheism` and `religion` as well as between `comp.graphics` and `space`. Let's see..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "# We will use four of the twenty newsgroups\n",
    "categories = ['alt.atheism',\n",
    "              'talk.religion.misc',\n",
    "              'comp.graphics',\n",
    "              'sci.space']\n",
    "\n",
    "twenty_train_subset = fetch_20newsgroups(subset='train', categories=categories)\n",
    "twenty_test_subset = fetch_20newsgroups(subset='test', categories=categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have lists of messages (as strings) in the `.data` members."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u\"From: rych@festival.ed.ac.uk (R Hawkes)\\nSubject: 3DS: Where did all the texture rules go?\\nLines: 21\\n\\nHi,\\n\\nI've noticed that if you only save a model (with all your mapping planes\\npositioned carefully) to a .3DS file that when you reload it after restarting\\n3DS, they are given a default position and orientation.  But if you save\\nto a .PRJ file their positions/orientation are preserved.  Does anyone\\nknow why this information is not stored in the .3DS file?  Nothing is\\nexplicitly said in the manual about saving texture rules in the .PRJ file. \\nI'd like to be able to read the texture rule information, does anyone have \\nthe format for the .PRJ file?\\n\\nIs the .CEL file format available from somewhere?\\n\\nRych\\n\\n======================================================================\\nRycharde Hawkes\\t\\t\\t\\temail: rych@festival.ed.ac.uk\\nVirtual Environment Laboratory\\nDept. of Psychology\\t\\t\\tTel  : +44 31 650 3426\\nUniv. of Edinburgh\\t\\t\\tFax  : +44 31 667 0150\\n======================================================================\\n\",\n",
       " u\"Subject: Re: Biblical Backing of Koresh's 3-02 Tape (Cites enclosed)\\nFrom: kmcvay@oneb.almanac.bc.ca (Ken Mcvay)\\nOrganization: The Old Frog's Almanac\\nLines: 20\\n\\nIn article <20APR199301460499@utarlg.uta.edu> b645zaw@utarlg.uta.edu (stephen) writes:\\n\\n>Seems to me Koresh is yet another messenger that got killed\\n>for the message he carried. (Which says nothing about the \\n\\nSeems to be, barring evidence to the contrary, that Koresh was simply\\nanother deranged fanatic who thought it neccessary to take a whole bunch of\\nfolks with him, children and all, to satisfy his delusional mania. Jim\\nJones, circa 1993.\\n\\n>In the mean time, we sure learned a lot about evil and corruption.\\n>Are you surprised things have gotten that rotten?\\n\\nNope - fruitcakes like Koresh have been demonstrating such evil corruption\\nfor centuries.\\n-- \\nThe Old Frog's Almanac - A Salute to That Old Frog Hisse'f, Ryugen Fisher \\n     (604) 245-3205 (v32) (604) 245-4366 (2400x4) SCO XENIX 2.3.2 GT \\n  Ladysmith, British Columbia, CANADA. Serving Central Vancouver Island  \\nwith public access UseNet and Internet Mail - home to the Holocaust Almanac\\n\"]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twenty_train_subset.data[0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 3]\n",
      "['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc']\n"
     ]
    }
   ],
   "source": [
    "print twenty_train_subset.target[0:2]\n",
    "print twenty_train_subset.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Count Vectorization\n",
    "\n",
    "Recall how we can generate features from text using the CountVectorizer. We can \"fit\" a body of text using a count vectorizer, which will give the CountVectorizer a total vocabulary. When we call transform again using the fitted vectorizer, it will return a matrix (sparse matrix) of size `document x vocabulary`, where each element is the frequency of that word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Turn the text documents into vectors of word frequencies\n",
    "vectorizer = CountVectorizer()\n",
    "vectorizer.fit(twenty_train_subset.data) \n",
    "X_train = vectorizer.transform(twenty_train_subset.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>00</th>\n",
       "      <th>000</th>\n",
       "      <th>0000</th>\n",
       "      <th>00000</th>\n",
       "      <th>000000</th>\n",
       "      <th>000005102000</th>\n",
       "      <th>000021</th>\n",
       "      <th>000062david42</th>\n",
       "      <th>0000vec</th>\n",
       "      <th>0001</th>\n",
       "      <th>...</th>\n",
       "      <th>zwakke</th>\n",
       "      <th>zware</th>\n",
       "      <th>zwarte</th>\n",
       "      <th>zxmkr08</th>\n",
       "      <th>zyeh</th>\n",
       "      <th>zyklon</th>\n",
       "      <th>zyxel</th>\n",
       "      <th>ªl</th>\n",
       "      <th>º_________________________________________________º_____________________º</th>\n",
       "      <th>ºnd</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2005</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2011</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2025</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2026</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2027</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2028</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2029</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2030</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2031</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2032</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2033</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2034 rows × 34118 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      00  000  0000  00000  000000  000005102000  000021  000062david42  \\\n",
       "0      0    0     0      0       0             0       0              0   \n",
       "1      0    0     0      0       0             0       0              0   \n",
       "2      0    0     0      0       0             0       0              0   \n",
       "3      0    0     0      0       0             0       0              0   \n",
       "4      0    0     0      0       0             0       0              0   \n",
       "5      0    0     0      0       0             0       0              0   \n",
       "6      0    0     0      0       0             0       0              0   \n",
       "7      0    0     0      0       0             0       0              0   \n",
       "8      0    0     0      0       0             0       0              0   \n",
       "9      0    0     0      0       0             0       0              0   \n",
       "10     0    0     0      0       0             0       0              0   \n",
       "11     0    0     0      0       0             0       0              0   \n",
       "12     0    0     0      0       0             0       0              0   \n",
       "13     0    0     0      0       0             0       0              0   \n",
       "14     0    0     0      0       0             0       0              0   \n",
       "15     0    0     0      0       0             0       0              0   \n",
       "16     0    0     0      0       0             0       0              0   \n",
       "17     0    0     0      0       0             0       0              0   \n",
       "18     0    0     0      0       0             0       0              0   \n",
       "19     0    0     0      0       0             0       0              0   \n",
       "20     0    0     0      0       0             0       0              0   \n",
       "21     0    0     0      0       0             0       0              0   \n",
       "22     0    0     0      0       0             0       0              0   \n",
       "23     0    4     0      0       0             0       0              0   \n",
       "24     0    0     0      0       0             0       0              0   \n",
       "25     0    0     0      0       0             0       0              0   \n",
       "26     0    0     0      0       0             0       0              0   \n",
       "27     0    0     0      0       0             0       0              0   \n",
       "28     0    0     0      0       0             0       0              0   \n",
       "29     0    0     0      0       0             0       0              0   \n",
       "...   ..  ...   ...    ...     ...           ...     ...            ...   \n",
       "2004   0    0     0      0       0             0       0              0   \n",
       "2005   0    0     0      0       0             0       0              0   \n",
       "2006   0    0     0      0       0             0       0              0   \n",
       "2007   0    0     0      0       0             0       0              0   \n",
       "2008   0    0     0      0       0             0       0              0   \n",
       "2009   0    0     0      0       0             0       0              0   \n",
       "2010   0    0     0      0       0             0       0              0   \n",
       "2011   0    2     0      0       0             0       0              0   \n",
       "2012   0    0     0      0       0             0       0              0   \n",
       "2013   0    0     0      0       0             0       0              0   \n",
       "2014   0    0     0      0       0             0       0              0   \n",
       "2015   0    0     0      0       0             0       0              0   \n",
       "2016   0    0     0      0       0             0       0              0   \n",
       "2017   0    0     0      0       0             0       0              0   \n",
       "2018   0    0     0      0       0             0       0              0   \n",
       "2019   0    0     0      0       0             0       0              0   \n",
       "2020   1    0     0      0       0             0       0              0   \n",
       "2021   0    0     0      0       0             0       0              0   \n",
       "2022   0    0     0      0       0             0       0              0   \n",
       "2023   0    0     0      0       0             0       0              0   \n",
       "2024   0    0     0      0       0             0       0              0   \n",
       "2025   0    0     0      0       0             0       0              0   \n",
       "2026   0    0     0      0       0             0       0              0   \n",
       "2027   0    0     0      0       0             0       0              0   \n",
       "2028   0    0     0      0       0             0       0              0   \n",
       "2029   0    0     0      0       0             0       0              0   \n",
       "2030   0    0     0      0       0             0       0              0   \n",
       "2031   0    0     0      0       0             0       0              0   \n",
       "2032   0    0     0      0       0             0       0              0   \n",
       "2033   0    0     0      0       0             0       0              0   \n",
       "\n",
       "      0000vec  0001 ...   zwakke  zware  zwarte  zxmkr08  zyeh  zyklon  zyxel  \\\n",
       "0           0     0 ...        0      0       0        0     0       0      0   \n",
       "1           0     0 ...        0      0       0        0     0       0      0   \n",
       "2           0     0 ...        0      0       0        0     0       0      0   \n",
       "3           0     0 ...        0      0       0        0     0       0      0   \n",
       "4           0     0 ...        0      0       0        0     0       0      0   \n",
       "5           0     0 ...        0      0       0        0     0       0      0   \n",
       "6           0     0 ...        0      0       0        0     0       0      0   \n",
       "7           0     0 ...        0      0       0        0     0       0      0   \n",
       "8           0     0 ...        0      0       0        0     0       0      0   \n",
       "9           0     0 ...        0      0       0        0     0       0      0   \n",
       "10          0     0 ...        0      0       0        0     0       0      0   \n",
       "11          0     0 ...        0      0       0        0     0       0      0   \n",
       "12          0     0 ...        0      0       0        0     0       0      0   \n",
       "13          0     0 ...        0      0       0        0     0       0      0   \n",
       "14          0     0 ...        0      0       0        0     0       0      0   \n",
       "15          0     0 ...        0      0       0        0     0       0      0   \n",
       "16          0     0 ...        0      0       0        0     0       0      0   \n",
       "17          0     0 ...        0      0       0        0     0       0      0   \n",
       "18          0     0 ...        0      0       0        0     0       0      0   \n",
       "19          0     0 ...        0      0       0        0     0       0      0   \n",
       "20          0     0 ...        0      0       0        0     0       0      0   \n",
       "21          0     0 ...        0      0       0        0     0       0      0   \n",
       "22          0     0 ...        0      0       0        0     0       0      0   \n",
       "23          0     0 ...        0      0       0        0     0       0      0   \n",
       "24          0     0 ...        0      0       0        0     0       0      0   \n",
       "25          0     0 ...        0      0       0        0     0       0      0   \n",
       "26          0     0 ...        0      0       0        0     0       0      0   \n",
       "27          0     0 ...        0      0       0        0     0       0      0   \n",
       "28          0     0 ...        0      0       0        0     0       0      0   \n",
       "29          0     0 ...        0      0       0        0     0       0      0   \n",
       "...       ...   ... ...      ...    ...     ...      ...   ...     ...    ...   \n",
       "2004        0     0 ...        0      0       0        0     0       0      0   \n",
       "2005        0     0 ...        0      0       0        0     0       0      0   \n",
       "2006        0     0 ...        0      0       0        0     0       0      0   \n",
       "2007        0     0 ...        0      0       0        0     0       0      0   \n",
       "2008        0     0 ...        0      0       0        0     0       0      0   \n",
       "2009        0     0 ...        0      0       0        0     0       0      0   \n",
       "2010        0     0 ...        0      0       0        0     0       0      0   \n",
       "2011        0     0 ...        0      0       0        0     0       0      0   \n",
       "2012        0     0 ...        0      0       0        0     0       0      0   \n",
       "2013        0     0 ...        0      0       0        0     0       0      0   \n",
       "2014        0     0 ...        0      0       0        0     0       0      0   \n",
       "2015        0     0 ...        0      0       0        0     0       0      0   \n",
       "2016        0     0 ...        0      0       0        0     0       0      0   \n",
       "2017        0     0 ...        0      0       0        0     0       0      0   \n",
       "2018        0     0 ...        0      0       0        0     0       0      0   \n",
       "2019        0     0 ...        0      0       0        0     0       0      0   \n",
       "2020        0     0 ...        0      0       0        0     0       0      0   \n",
       "2021        0     0 ...        0      0       0        0     0       0      0   \n",
       "2022        0     0 ...        0      0       0        0     0       0      0   \n",
       "2023        0     0 ...        0      0       0        0     0       0      0   \n",
       "2024        0     0 ...        0      0       0        0     0       0      0   \n",
       "2025        0     0 ...        0      0       0        0     0       0      0   \n",
       "2026        0     0 ...        0      0       0        0     0       0      0   \n",
       "2027        0     0 ...        0      0       0        0     0       0      0   \n",
       "2028        0     0 ...        0      0       0        0     0       0      0   \n",
       "2029        0     0 ...        0      0       0        0     0       0      0   \n",
       "2030        0     0 ...        0      0       0        0     0       0      0   \n",
       "2031        0     0 ...        0      0       0        0     0       0      0   \n",
       "2032        0     0 ...        0      0       0        0     0       0      0   \n",
       "2033        0     0 ...        0      0       0        0     0       0      0   \n",
       "\n",
       "      ªl  \\\n",
       "0      0   \n",
       "1      0   \n",
       "2      0   \n",
       "3      0   \n",
       "4      0   \n",
       "5      0   \n",
       "6      0   \n",
       "7      0   \n",
       "8      0   \n",
       "9      0   \n",
       "10     0   \n",
       "11     0   \n",
       "12     0   \n",
       "13     0   \n",
       "14     0   \n",
       "15     0   \n",
       "16     0   \n",
       "17     0   \n",
       "18     0   \n",
       "19     0   \n",
       "20     0   \n",
       "21     0   \n",
       "22     0   \n",
       "23     0   \n",
       "24     0   \n",
       "25     0   \n",
       "26     0   \n",
       "27     0   \n",
       "28     0   \n",
       "29     0   \n",
       "...   ..   \n",
       "2004   0   \n",
       "2005   0   \n",
       "2006   0   \n",
       "2007   0   \n",
       "2008   0   \n",
       "2009   0   \n",
       "2010   0   \n",
       "2011   0   \n",
       "2012   0   \n",
       "2013   0   \n",
       "2014   0   \n",
       "2015   0   \n",
       "2016   0   \n",
       "2017   0   \n",
       "2018   0   \n",
       "2019   0   \n",
       "2020   0   \n",
       "2021   0   \n",
       "2022   0   \n",
       "2023   0   \n",
       "2024   0   \n",
       "2025   0   \n",
       "2026   0   \n",
       "2027   0   \n",
       "2028   0   \n",
       "2029   0   \n",
       "2030   0   \n",
       "2031   0   \n",
       "2032   0   \n",
       "2033   0   \n",
       "\n",
       "      º_________________________________________________º_____________________º  \\\n",
       "0                                                     0                           \n",
       "1                                                     0                           \n",
       "2                                                     0                           \n",
       "3                                                     0                           \n",
       "4                                                     0                           \n",
       "5                                                     0                           \n",
       "6                                                     0                           \n",
       "7                                                     0                           \n",
       "8                                                     0                           \n",
       "9                                                     0                           \n",
       "10                                                    0                           \n",
       "11                                                    0                           \n",
       "12                                                    0                           \n",
       "13                                                    0                           \n",
       "14                                                    0                           \n",
       "15                                                    0                           \n",
       "16                                                    0                           \n",
       "17                                                    0                           \n",
       "18                                                    0                           \n",
       "19                                                    0                           \n",
       "20                                                    0                           \n",
       "21                                                    0                           \n",
       "22                                                    0                           \n",
       "23                                                    0                           \n",
       "24                                                    0                           \n",
       "25                                                    0                           \n",
       "26                                                    0                           \n",
       "27                                                    0                           \n",
       "28                                                    0                           \n",
       "29                                                    0                           \n",
       "...                                                 ...                           \n",
       "2004                                                  0                           \n",
       "2005                                                  0                           \n",
       "2006                                                  0                           \n",
       "2007                                                  0                           \n",
       "2008                                                  0                           \n",
       "2009                                                  0                           \n",
       "2010                                                  0                           \n",
       "2011                                                  0                           \n",
       "2012                                                  0                           \n",
       "2013                                                  0                           \n",
       "2014                                                  0                           \n",
       "2015                                                  0                           \n",
       "2016                                                  0                           \n",
       "2017                                                  0                           \n",
       "2018                                                  0                           \n",
       "2019                                                  0                           \n",
       "2020                                                  0                           \n",
       "2021                                                  0                           \n",
       "2022                                                  0                           \n",
       "2023                                                  0                           \n",
       "2024                                                  0                           \n",
       "2025                                                  0                           \n",
       "2026                                                  0                           \n",
       "2027                                                  0                           \n",
       "2028                                                  0                           \n",
       "2029                                                  0                           \n",
       "2030                                                  0                           \n",
       "2031                                                  0                           \n",
       "2032                                                  0                           \n",
       "2033                                                  0                           \n",
       "\n",
       "      ºnd  \n",
       "0       0  \n",
       "1       0  \n",
       "2       0  \n",
       "3       0  \n",
       "4       0  \n",
       "5       0  \n",
       "6       0  \n",
       "7       0  \n",
       "8       0  \n",
       "9       0  \n",
       "10      0  \n",
       "11      0  \n",
       "12      0  \n",
       "13      0  \n",
       "14      0  \n",
       "15      0  \n",
       "16      0  \n",
       "17      0  \n",
       "18      0  \n",
       "19      0  \n",
       "20      0  \n",
       "21      0  \n",
       "22      0  \n",
       "23      0  \n",
       "24      0  \n",
       "25      0  \n",
       "26      0  \n",
       "27      0  \n",
       "28      0  \n",
       "29      0  \n",
       "...   ...  \n",
       "2004    0  \n",
       "2005    0  \n",
       "2006    0  \n",
       "2007    0  \n",
       "2008    0  \n",
       "2009    0  \n",
       "2010    0  \n",
       "2011    0  \n",
       "2012    0  \n",
       "2013    0  \n",
       "2014    0  \n",
       "2015    0  \n",
       "2016    0  \n",
       "2017    0  \n",
       "2018    0  \n",
       "2019    0  \n",
       "2020    0  \n",
       "2021    0  \n",
       "2022    0  \n",
       "2023    0  \n",
       "2024    0  \n",
       "2025    0  \n",
       "2026    0  \n",
       "2027    0  \n",
       "2028    0  \n",
       "2029    0  \n",
       "2030    0  \n",
       "2031    0  \n",
       "2032    0  \n",
       "2033    0  \n",
       "\n",
       "[2034 rows x 34118 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.DataFrame( X_train.toarray(), columns=vectorizer.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, this makes a matrix of word counts, where each row is a document and each column is the word, the cell matrix[document, word] contains the count of word in document.\n",
    "\n",
    "We can expand on this by setting the ngram_range. This parameter allows us to set each column not only as one word, but possibly sequences of words.\n",
    "\n",
    "We can also provide an argument for `stopwords` that will strip out any english style stop words such as \"and\", \"or\", \"other\", etc for words that don't provide significance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Include every 1-gram, 2-gram, and 3-gram\n",
    "vectorizer = CountVectorizer(ngram_range=(1,3),stop_words='english')\n",
    "vectorizer.fit(twenty_train_subset.data)\n",
    "X_train = vectorizer.transform(twenty_train_subset.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, we could use a tf-idf representation, which stands for Term Frequency - Inverse Document Frequency.\n",
    "\n",
    "This value is the product of two intermediate values, the Term Frequency and the Inverse Document Frequency.\n",
    "\n",
    "The Term Frequency is equivalent to the CountVectorizer features, the number of times or count that a word appear in the document. This is our most basic representation of text.\n",
    "\n",
    "To establish Inverse Document Frequency, first let's define Document Frequency. This is the percentage of documents that a particular word appears in. For example, the word `the` might appear in 100% of documents, while words like `Syria` would likely have low document frequency. Inverse Document Frequency is simply 1 / Document Frequency (although often the log is also taken).\n",
    "\n",
    "So tf-idf is Term Frequency * Inverse Document Frequency, or similar to Term Frequency / Document Frequency. The intuition is that words that have high weight are those that appear a lot in this document and/or appear in very few other documents (somehow unique to this document)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "\n",
    "vectorizer = TfidfVectorizer()\n",
    "\n",
    "# This is short hand for calling fit, and then transform\n",
    "X_train = vectorizer.fit_transform(twenty_train_subset.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>00</th>\n",
       "      <th>000</th>\n",
       "      <th>0000</th>\n",
       "      <th>00000</th>\n",
       "      <th>000000</th>\n",
       "      <th>000005102000</th>\n",
       "      <th>000021</th>\n",
       "      <th>000062david42</th>\n",
       "      <th>0000vec</th>\n",
       "      <th>0001</th>\n",
       "      <th>...</th>\n",
       "      <th>zwakke</th>\n",
       "      <th>zware</th>\n",
       "      <th>zwarte</th>\n",
       "      <th>zxmkr08</th>\n",
       "      <th>zyeh</th>\n",
       "      <th>zyklon</th>\n",
       "      <th>zyxel</th>\n",
       "      <th>ªl</th>\n",
       "      <th>º_________________________________________________º_____________________º</th>\n",
       "      <th>ºnd</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.082467</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2005</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2011</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.029123</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020</th>\n",
       "      <td>0.069354</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2025</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2026</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2027</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2028</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2029</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2030</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2031</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2032</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2033</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2034 rows × 34118 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            00       000  0000  00000  000000  000005102000  000021  \\\n",
       "0     0.000000  0.000000     0      0       0             0       0   \n",
       "1     0.000000  0.000000     0      0       0             0       0   \n",
       "2     0.000000  0.000000     0      0       0             0       0   \n",
       "3     0.000000  0.000000     0      0       0             0       0   \n",
       "4     0.000000  0.000000     0      0       0             0       0   \n",
       "5     0.000000  0.000000     0      0       0             0       0   \n",
       "6     0.000000  0.000000     0      0       0             0       0   \n",
       "7     0.000000  0.000000     0      0       0             0       0   \n",
       "8     0.000000  0.000000     0      0       0             0       0   \n",
       "9     0.000000  0.000000     0      0       0             0       0   \n",
       "10    0.000000  0.000000     0      0       0             0       0   \n",
       "11    0.000000  0.000000     0      0       0             0       0   \n",
       "12    0.000000  0.000000     0      0       0             0       0   \n",
       "13    0.000000  0.000000     0      0       0             0       0   \n",
       "14    0.000000  0.000000     0      0       0             0       0   \n",
       "15    0.000000  0.000000     0      0       0             0       0   \n",
       "16    0.000000  0.000000     0      0       0             0       0   \n",
       "17    0.000000  0.000000     0      0       0             0       0   \n",
       "18    0.000000  0.000000     0      0       0             0       0   \n",
       "19    0.000000  0.000000     0      0       0             0       0   \n",
       "20    0.000000  0.000000     0      0       0             0       0   \n",
       "21    0.000000  0.000000     0      0       0             0       0   \n",
       "22    0.000000  0.000000     0      0       0             0       0   \n",
       "23    0.000000  0.082467     0      0       0             0       0   \n",
       "24    0.000000  0.000000     0      0       0             0       0   \n",
       "25    0.000000  0.000000     0      0       0             0       0   \n",
       "26    0.000000  0.000000     0      0       0             0       0   \n",
       "27    0.000000  0.000000     0      0       0             0       0   \n",
       "28    0.000000  0.000000     0      0       0             0       0   \n",
       "29    0.000000  0.000000     0      0       0             0       0   \n",
       "...        ...       ...   ...    ...     ...           ...     ...   \n",
       "2004  0.000000  0.000000     0      0       0             0       0   \n",
       "2005  0.000000  0.000000     0      0       0             0       0   \n",
       "2006  0.000000  0.000000     0      0       0             0       0   \n",
       "2007  0.000000  0.000000     0      0       0             0       0   \n",
       "2008  0.000000  0.000000     0      0       0             0       0   \n",
       "2009  0.000000  0.000000     0      0       0             0       0   \n",
       "2010  0.000000  0.000000     0      0       0             0       0   \n",
       "2011  0.000000  0.029123     0      0       0             0       0   \n",
       "2012  0.000000  0.000000     0      0       0             0       0   \n",
       "2013  0.000000  0.000000     0      0       0             0       0   \n",
       "2014  0.000000  0.000000     0      0       0             0       0   \n",
       "2015  0.000000  0.000000     0      0       0             0       0   \n",
       "2016  0.000000  0.000000     0      0       0             0       0   \n",
       "2017  0.000000  0.000000     0      0       0             0       0   \n",
       "2018  0.000000  0.000000     0      0       0             0       0   \n",
       "2019  0.000000  0.000000     0      0       0             0       0   \n",
       "2020  0.069354  0.000000     0      0       0             0       0   \n",
       "2021  0.000000  0.000000     0      0       0             0       0   \n",
       "2022  0.000000  0.000000     0      0       0             0       0   \n",
       "2023  0.000000  0.000000     0      0       0             0       0   \n",
       "2024  0.000000  0.000000     0      0       0             0       0   \n",
       "2025  0.000000  0.000000     0      0       0             0       0   \n",
       "2026  0.000000  0.000000     0      0       0             0       0   \n",
       "2027  0.000000  0.000000     0      0       0             0       0   \n",
       "2028  0.000000  0.000000     0      0       0             0       0   \n",
       "2029  0.000000  0.000000     0      0       0             0       0   \n",
       "2030  0.000000  0.000000     0      0       0             0       0   \n",
       "2031  0.000000  0.000000     0      0       0             0       0   \n",
       "2032  0.000000  0.000000     0      0       0             0       0   \n",
       "2033  0.000000  0.000000     0      0       0             0       0   \n",
       "\n",
       "      000062david42  0000vec  0001 ...   zwakke  zware  zwarte  zxmkr08  zyeh  \\\n",
       "0                 0        0     0 ...        0      0       0        0     0   \n",
       "1                 0        0     0 ...        0      0       0        0     0   \n",
       "2                 0        0     0 ...        0      0       0        0     0   \n",
       "3                 0        0     0 ...        0      0       0        0     0   \n",
       "4                 0        0     0 ...        0      0       0        0     0   \n",
       "5                 0        0     0 ...        0      0       0        0     0   \n",
       "6                 0        0     0 ...        0      0       0        0     0   \n",
       "7                 0        0     0 ...        0      0       0        0     0   \n",
       "8                 0        0     0 ...        0      0       0        0     0   \n",
       "9                 0        0     0 ...        0      0       0        0     0   \n",
       "10                0        0     0 ...        0      0       0        0     0   \n",
       "11                0        0     0 ...        0      0       0        0     0   \n",
       "12                0        0     0 ...        0      0       0        0     0   \n",
       "13                0        0     0 ...        0      0       0        0     0   \n",
       "14                0        0     0 ...        0      0       0        0     0   \n",
       "15                0        0     0 ...        0      0       0        0     0   \n",
       "16                0        0     0 ...        0      0       0        0     0   \n",
       "17                0        0     0 ...        0      0       0        0     0   \n",
       "18                0        0     0 ...        0      0       0        0     0   \n",
       "19                0        0     0 ...        0      0       0        0     0   \n",
       "20                0        0     0 ...        0      0       0        0     0   \n",
       "21                0        0     0 ...        0      0       0        0     0   \n",
       "22                0        0     0 ...        0      0       0        0     0   \n",
       "23                0        0     0 ...        0      0       0        0     0   \n",
       "24                0        0     0 ...        0      0       0        0     0   \n",
       "25                0        0     0 ...        0      0       0        0     0   \n",
       "26                0        0     0 ...        0      0       0        0     0   \n",
       "27                0        0     0 ...        0      0       0        0     0   \n",
       "28                0        0     0 ...        0      0       0        0     0   \n",
       "29                0        0     0 ...        0      0       0        0     0   \n",
       "...             ...      ...   ... ...      ...    ...     ...      ...   ...   \n",
       "2004              0        0     0 ...        0      0       0        0     0   \n",
       "2005              0        0     0 ...        0      0       0        0     0   \n",
       "2006              0        0     0 ...        0      0       0        0     0   \n",
       "2007              0        0     0 ...        0      0       0        0     0   \n",
       "2008              0        0     0 ...        0      0       0        0     0   \n",
       "2009              0        0     0 ...        0      0       0        0     0   \n",
       "2010              0        0     0 ...        0      0       0        0     0   \n",
       "2011              0        0     0 ...        0      0       0        0     0   \n",
       "2012              0        0     0 ...        0      0       0        0     0   \n",
       "2013              0        0     0 ...        0      0       0        0     0   \n",
       "2014              0        0     0 ...        0      0       0        0     0   \n",
       "2015              0        0     0 ...        0      0       0        0     0   \n",
       "2016              0        0     0 ...        0      0       0        0     0   \n",
       "2017              0        0     0 ...        0      0       0        0     0   \n",
       "2018              0        0     0 ...        0      0       0        0     0   \n",
       "2019              0        0     0 ...        0      0       0        0     0   \n",
       "2020              0        0     0 ...        0      0       0        0     0   \n",
       "2021              0        0     0 ...        0      0       0        0     0   \n",
       "2022              0        0     0 ...        0      0       0        0     0   \n",
       "2023              0        0     0 ...        0      0       0        0     0   \n",
       "2024              0        0     0 ...        0      0       0        0     0   \n",
       "2025              0        0     0 ...        0      0       0        0     0   \n",
       "2026              0        0     0 ...        0      0       0        0     0   \n",
       "2027              0        0     0 ...        0      0       0        0     0   \n",
       "2028              0        0     0 ...        0      0       0        0     0   \n",
       "2029              0        0     0 ...        0      0       0        0     0   \n",
       "2030              0        0     0 ...        0      0       0        0     0   \n",
       "2031              0        0     0 ...        0      0       0        0     0   \n",
       "2032              0        0     0 ...        0      0       0        0     0   \n",
       "2033              0        0     0 ...        0      0       0        0     0   \n",
       "\n",
       "      zyklon  zyxel  ªl  \\\n",
       "0          0      0   0   \n",
       "1          0      0   0   \n",
       "2          0      0   0   \n",
       "3          0      0   0   \n",
       "4          0      0   0   \n",
       "5          0      0   0   \n",
       "6          0      0   0   \n",
       "7          0      0   0   \n",
       "8          0      0   0   \n",
       "9          0      0   0   \n",
       "10         0      0   0   \n",
       "11         0      0   0   \n",
       "12         0      0   0   \n",
       "13         0      0   0   \n",
       "14         0      0   0   \n",
       "15         0      0   0   \n",
       "16         0      0   0   \n",
       "17         0      0   0   \n",
       "18         0      0   0   \n",
       "19         0      0   0   \n",
       "20         0      0   0   \n",
       "21         0      0   0   \n",
       "22         0      0   0   \n",
       "23         0      0   0   \n",
       "24         0      0   0   \n",
       "25         0      0   0   \n",
       "26         0      0   0   \n",
       "27         0      0   0   \n",
       "28         0      0   0   \n",
       "29         0      0   0   \n",
       "...      ...    ...  ..   \n",
       "2004       0      0   0   \n",
       "2005       0      0   0   \n",
       "2006       0      0   0   \n",
       "2007       0      0   0   \n",
       "2008       0      0   0   \n",
       "2009       0      0   0   \n",
       "2010       0      0   0   \n",
       "2011       0      0   0   \n",
       "2012       0      0   0   \n",
       "2013       0      0   0   \n",
       "2014       0      0   0   \n",
       "2015       0      0   0   \n",
       "2016       0      0   0   \n",
       "2017       0      0   0   \n",
       "2018       0      0   0   \n",
       "2019       0      0   0   \n",
       "2020       0      0   0   \n",
       "2021       0      0   0   \n",
       "2022       0      0   0   \n",
       "2023       0      0   0   \n",
       "2024       0      0   0   \n",
       "2025       0      0   0   \n",
       "2026       0      0   0   \n",
       "2027       0      0   0   \n",
       "2028       0      0   0   \n",
       "2029       0      0   0   \n",
       "2030       0      0   0   \n",
       "2031       0      0   0   \n",
       "2032       0      0   0   \n",
       "2033       0      0   0   \n",
       "\n",
       "      º_________________________________________________º_____________________º  \\\n",
       "0                                                     0                           \n",
       "1                                                     0                           \n",
       "2                                                     0                           \n",
       "3                                                     0                           \n",
       "4                                                     0                           \n",
       "5                                                     0                           \n",
       "6                                                     0                           \n",
       "7                                                     0                           \n",
       "8                                                     0                           \n",
       "9                                                     0                           \n",
       "10                                                    0                           \n",
       "11                                                    0                           \n",
       "12                                                    0                           \n",
       "13                                                    0                           \n",
       "14                                                    0                           \n",
       "15                                                    0                           \n",
       "16                                                    0                           \n",
       "17                                                    0                           \n",
       "18                                                    0                           \n",
       "19                                                    0                           \n",
       "20                                                    0                           \n",
       "21                                                    0                           \n",
       "22                                                    0                           \n",
       "23                                                    0                           \n",
       "24                                                    0                           \n",
       "25                                                    0                           \n",
       "26                                                    0                           \n",
       "27                                                    0                           \n",
       "28                                                    0                           \n",
       "29                                                    0                           \n",
       "...                                                 ...                           \n",
       "2004                                                  0                           \n",
       "2005                                                  0                           \n",
       "2006                                                  0                           \n",
       "2007                                                  0                           \n",
       "2008                                                  0                           \n",
       "2009                                                  0                           \n",
       "2010                                                  0                           \n",
       "2011                                                  0                           \n",
       "2012                                                  0                           \n",
       "2013                                                  0                           \n",
       "2014                                                  0                           \n",
       "2015                                                  0                           \n",
       "2016                                                  0                           \n",
       "2017                                                  0                           \n",
       "2018                                                  0                           \n",
       "2019                                                  0                           \n",
       "2020                                                  0                           \n",
       "2021                                                  0                           \n",
       "2022                                                  0                           \n",
       "2023                                                  0                           \n",
       "2024                                                  0                           \n",
       "2025                                                  0                           \n",
       "2026                                                  0                           \n",
       "2027                                                  0                           \n",
       "2028                                                  0                           \n",
       "2029                                                  0                           \n",
       "2030                                                  0                           \n",
       "2031                                                  0                           \n",
       "2032                                                  0                           \n",
       "2033                                                  0                           \n",
       "\n",
       "      ºnd  \n",
       "0       0  \n",
       "1       0  \n",
       "2       0  \n",
       "3       0  \n",
       "4       0  \n",
       "5       0  \n",
       "6       0  \n",
       "7       0  \n",
       "8       0  \n",
       "9       0  \n",
       "10      0  \n",
       "11      0  \n",
       "12      0  \n",
       "13      0  \n",
       "14      0  \n",
       "15      0  \n",
       "16      0  \n",
       "17      0  \n",
       "18      0  \n",
       "19      0  \n",
       "20      0  \n",
       "21      0  \n",
       "22      0  \n",
       "23      0  \n",
       "24      0  \n",
       "25      0  \n",
       "26      0  \n",
       "27      0  \n",
       "28      0  \n",
       "29      0  \n",
       "...   ...  \n",
       "2004    0  \n",
       "2005    0  \n",
       "2006    0  \n",
       "2007    0  \n",
       "2008    0  \n",
       "2009    0  \n",
       "2010    0  \n",
       "2011    0  \n",
       "2012    0  \n",
       "2013    0  \n",
       "2014    0  \n",
       "2015    0  \n",
       "2016    0  \n",
       "2017    0  \n",
       "2018    0  \n",
       "2019    0  \n",
       "2020    0  \n",
       "2021    0  \n",
       "2022    0  \n",
       "2023    0  \n",
       "2024    0  \n",
       "2025    0  \n",
       "2026    0  \n",
       "2027    0  \n",
       "2028    0  \n",
       "2029    0  \n",
       "2030    0  \n",
       "2031    0  \n",
       "2032    0  \n",
       "2033    0  \n",
       "\n",
       "[2034 rows x 34118 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame( X_train.toarray(), columns= vectorizer.get_feature_names() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can put this together with our other tricks as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vectorizer = TfidfVectorizer(stop_words='english', ngram_range=(1,3))\n",
    "X_train = vectorizer.fit_transform(twenty_train_subset.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For computational ease, let's just look at single word patterns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vectorizer = TfidfVectorizer(stop_words='english') \n",
    "X_train = vectorizer.fit_transform(twenty_train_subset.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise - Naive Bayes vs. Decision Trees\n",
    "\n",
    "We originally reviewed the CountVectorizer concept during our exploration of Naive Bayes. This time, we'll use a tf-idf vectorizer to create our X_train matrix, and compare the performance of Naive Bayes and Decision Trees.\n",
    "\n",
    "\n",
    "1. Using Naive Bayes, what is your cross validation score on the training data (5 fold using StratifiedShuffleSplit) with a tf-idf matrix?\n",
    "2. What is your score on the test set when you fit on the entire training set using Naive Bayes?\n",
    "3. With a similar setup, how well do decision trees perform on training and test? Try comparing the performance of the trees with various max_depth.\n",
    "\n",
    "\n",
    "**Notes:**\n",
    "* Using MultinomialNB may provide the best results when creating a Naive Bayes model.\n",
    "* Note, you will have to define y_train, X_test, and y_test variables.\n",
    "* You will also have to `transform` the data into a tf-idf matrix for the test set (**Don't call fit again! Remember that `fit` is run on the training set to get a total vocabulary. Once we have this vocabulary, we run transform on new data to transform the known words**)\n",
    "* Remember not to contaminate your training and test sets. Ideally, the test set should be looked at once at the end. Otherwise, you could end up fitting to the test set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "from sklearn.cross_validation import StratifiedShuffleSplit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Set up data\n",
    "vectorizer = TfidfVectorizer(stop_words='english') \n",
    "\n",
    "vectorizer.fit(twenty_train_subset.data)\n",
    "X_train = vectorizer.transform(twenty_train_subset.data)\n",
    "y_train = twenty_train_subset.target\n",
    "\n",
    "\n",
    "X_test  = vectorizer.transform( twenty_test_subset.data )\n",
    "y_test  = twenty_test_subset.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TrainScore: 0.920588235294\n",
      "TestScore: 0.875092387288\n"
     ]
    }
   ],
   "source": [
    "# Multinomial Version\n",
    "model_nb = MultinomialNB()\n",
    "model_nb.fit(X_train,y_train)\n",
    "print \"TrainScore:\",np.mean(cross_val_score(model_nb,X_train,y_train,\n",
    "                                            cv=StratifiedShuffleSplit(y_train,5)))\n",
    "print \"TestScore:\", model_nb.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TrainScore: 0.81862745098\n",
      "TestScore: 0.70214338507\n"
     ]
    }
   ],
   "source": [
    "# DTree Version\n",
    "model_dt = DecisionTreeClassifier()\n",
    "model_dt.fit(X_train,y_train)\n",
    "print \"TrainScore:\",np.mean(cross_val_score(model_dt,X_train,y_train,\n",
    "                                            cv=StratifiedShuffleSplit(y_train,5)))\n",
    "print \"TestScore:\", model_dt.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.tree._tree.Tree at 0x115176e00>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bagging (Bootstrap Aggregation)\n",
    "\n",
    "Let's explore how bootstrap aggregation can give us stronger results.\n",
    "\n",
    "Bagging tends to work well when the base classifiers (weak learners) have a low bias but high variance. By averaging the predictions over these base classifiers, we can reduce the overall variance of the final predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import BaggingClassifier\n",
    "\n",
    "# Let's view the arguments in the model\n",
    "# By default, DecisionTreeClassifier used, \n",
    "#  but can also use any base classifier\n",
    "\n",
    "\n",
    "model_bag = BaggingClassifier(   )   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BaggingClassifier(base_estimator=None, bootstrap=True,\n",
       "         bootstrap_features=False, max_features=1.0, max_samples=1.0,\n",
       "         n_estimators=10, n_jobs=1, oob_score=False, random_state=None,\n",
       "         verbose=0)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_bag.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.74722838137472281"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_bag.score(X_train,y_train)\n",
    "model_bag.score(X_test,y_test) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise - Bagging\n",
    "\n",
    "* How does bagging compare to a base decision tree classifier with 1, 5, 10, 20, and 30 estimators on the training set? \n",
    "* How about on the testing set?\n",
    "* Plot the improvements on both training and testing sets\n",
    "\n",
    "If you finish early, feel free to build a bagging ensemble using a different base classifier, such as NaiveBayes or SVM. You may see that SVM does poorly with a default rbf kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "5\n",
      "10\n",
      "20\n",
      "30\n",
      "50\n"
     ]
    }
   ],
   "source": [
    "n_est = [1,5,10,20,30,50]\n",
    "\n",
    "train_res, test_res = [], []\n",
    "\n",
    "for i in n_est:\n",
    "    print i\n",
    "    \n",
    "    model_bag = BaggingClassifier(base_estimator=svm.SVC() \n",
    "    model_bag.fit(X_train,y_train)\n",
    "    train_res.append(model_bag.score(X_train,y_train))\n",
    "    test_res.append(model_bag.score(X_test,y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x15a0eacd0>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEZCAYAAACJjGL9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcFNW5//HPlxl2AQUTVBCJibu5rgG3GydKDJqoMXH3\nZ6LXEK+Jmt2IN+rEmOuSxMS4JCZBMW64R0iuEUVHicEFRUARV1ABxRVZBJxhnt8fVQPN0FN0D9PT\ns3zfr1e9purUqaqnz8z00+fU0ooIzMzMmtKl3AGYmVnb5kRhZmaZnCjMzCyTE4WZmWVyojAzs0xO\nFGZmlsmJwlqEpGclfb7ccVj5+W+h43Gi6EAkzZX0kaQlkt6X9HdJg1vj2BGxc0Q8Uqr9S6qSVC/p\nrFIdo6ORdIikm9L5v0o6NKNutaTa9G+nYXq/gGOMlfSL3LJS/S2kfwNvtPR+bf2cKDqWAL4SEX2A\nzYGFwBXlDanFfBN4H/hGax5UqdY8ZgvaA3gynd8deCqjbgC3RESfnKl/ySNsRZIqyx1De+VE0UFF\nxErgTmDHhjJJX5Y0TdKHkl6XdH7uNpK+Iek1Se9K+lnaQzkwXddT0vVpT2WWpLNyP92ldQ9I56sl\n3ZbWX5wOReyRU3f3NI7Fab1bG38qbRRXb+DrwHeAbXL3la4flca0WNJzknZLy7eUdJekt9PXdEVO\nfDfkbD807a10SZdrJF0o6VFgGbC1pJNzjvGKpG83iuFwSc+kbfuypC9JOkrS1Eb1fijpb3le4zGS\nnmxU9gNJ96Tzh6SvbbGkeZJ+1FR75dgTeDptv/4RsSCjrtIp/0rpt5IWpq9vhqSd0jY4Hjgr7YE0\nxNr4b+F2STeksc+QtI2k0en+Xpf0xZzj5G3n9DXcC2yRHmuxpM0kdZf0O0nz0+m3krql21SlbXWW\npDeBMQW0meUTEZ46yATMAQ5M53sB1wNjc9bvD+yUzn8WeAs4PF3eEVgC7AN0BX4FfAwckK6/GHgI\n6AcMAmYArzc6dkPdamA5MJLkzed/gSnpum7Aa8AZQAVwBLASuCDjdZ0IzCf5YDMe+H3OuqOAecAe\n6fKngSHpvqcDvwF6At2BfdI65wM35OxjKFAPdEmXa4C5wA7pMSuBQ4BPpes/T5JAdkuXhwGLctp+\nC2C79LW+B2yfc6xpwBF5XmNPYDHwmZyyJ4Gj0/k3gX3T+X4Nx26ivWYDHwB1aVxL0t/lB8Afmtim\nOrdNGq37EjAV6Jsubwdsls5f1/h318TfwhfT38n1wKvA6HT5W8CrOdtmtfP+wBuNjnUB8G9g03R6\ntCEeoAqoBS4i+ZvuUe7/0fY6lT0ATy34y0ze3Jakbwgfp2+gO2fU/x1wWTp/HnBTzrqeJG/gDf/w\nrwBfzFl/Su4/bZ43h4k563YEPkrnPw/MaxTH5MZvNo3WP5AT57HA20BFunwfcEaebfZO63XJs26t\nN0XWTRQPAdXraeu7gTPT+WuA3zRR72rgwnR+J5Lhs65N1L0BODed34YkcfRIl18Dvk36Zl3A38II\n4K6c+I5cT/3q9Pf9Qc40KV13APACMLxxe5Ikil80Kmv8t3BfzrpD079Rpct90rbP+7oatXMV6yaK\nl4GROcsHAXNy6q8EurXW/2BHnTz01LEESQ9hE5JP0GcAD0saCCBpuKSH0qGYRcCpwIB02y1IEkuy\no4jlJJ+GyVmfeyJxHtkW5sx/BPRIh3a2IOkd5HqDJoY9JG1J8g9/c1o0HugBfDldHkySxBrbEngt\nIurXE2dT1jppKulgSY9Jek/SBySffBvarqkYIPkEfXw6fyJwa0TUNlH3ZuC4dP544O6IWJEufz09\n5tx0aGyvfDuQdGka39+Bg9L5U4A/S8oaeiKNbZOc6UCAiHgQuBK4Clgo6RpJfdazr1xv58wvB96N\n9J08XQbYKI0/q53z2YIkiTZ4PS1r8E5EfFxErJaHE0UHFYm7gVXAvmnxzcDfgMERsTHwR9a8QS8g\necMDknMSrP0P+ibJm2+D3PlivEkydJVrCEmSy+dEkr/TCek48yskieKkdP0bwGfybPcGMERSRZ51\nS0mG5hpslqfO6ngkdSc533Mp8Mk0Ef8fa9quqRiIiMeBj5VcLnocSa+hKQ8An5C0C0nPqSE5EhFT\nI+KrwCdIfoe3NXG8s9L45pAMw+1PMuy3SURskW+bnNfb5DmKiLgiIvYk6R1uC/wkZ7sWUUA75zvW\nApIeYYMhadnq0Fsqvs7MiaLjEay+WudwYBPg+XTdRsAHEfGxpGGs+aQLyT/ooZL2Tk8GVrP2G8dt\nwGhJG0saBJxO8/4JpwCrJJ0uqTKN8XMZ9b+ZxrJLzvR14BBJ/YG/AD9WcoJckj4jaQjwOElSulhS\nL0k9JO2T7vMZ4PNKTnb3Ixkvbyz3tXdLp3eBekkHkwxxNBgDnCzpAEldJA2StF3O+r+SfCL/OCL+\n3dQLTXsatwO/Jvm93Q8gqaukEyT1i4hVJEM3q5raT/ppv09ELCS58mlqU3WbeL2N97dn2hvtStI7\nXJFz/IXA1gXsvxDra+eFwABJfXPKbgF+JmlTSZuSDKFmJWNrBieKjmeCpCXAh8AvgG9EREOi+A5w\ngaTFwLnArQ0bRcRzJENV40g+kS0hGTJYmVa5gGS4aQ4wkeQNrakufbBuEon0OB8DXyMZDvkAOIFk\nmGSdfaXDK1sCV0XE2znTBJKx6WMj4g7glySfvhcDdwGbpENOh5J80n+d5FP/0WkMD6SvfQbJCeMJ\nTcWb1l8CnEmSLN8n6Rnck7P+SeBk4LckJ48fIvlk2+AGkvMTNzbRXrluBg4Ebm80bPb/gDmSPiQ5\nV3FCxj52Izlp3jBfSKII4BitfR/F4vTNty/wJ5LXPpfkjfxX6XZjgB0lfSDprib222Tb5i4X0M6z\nSRLDq0quvtsMuDB9fTPSaWpa1tSxrBkaTiiVZufStSRjyW9HxGebqPN74GCSTyonRcS0tHwkycnW\nCuAvEXFJyQK1dUjaiOSN/DMR8Vqe9aeRXJHzhRY41uPA1RFx/Ybuqy1Kh/EWkly909S5DLM2q9Q9\niutILpHMS9IhJG9E25B8SvpDWl5B0lUfSTImepykHUoca6cn6dB0mKY3yfDHjIYkkV6zvm86tLId\n8EOSK1Kac5zPp/urlPRNYGfgny31Otqg04AnnCSsvSrpnYoRMVnS0Iwqh5FcFUJEPJ6Of28GfAp4\nOSLmAkgaBxzOmrF2K43DSMbTRTIkc2zOum4kJ78/RTK8cgvJpZ/NsR3J8EJvkpPTR6bj6R2OpLkk\nwx9fLXMoZs1W7lvaB7HuJZeDyH8p5vBWjKtTiohRwKgm1r1OcpNeSxznz8CfW2JfbV1EDC13DGYb\nqi2czG6vz9ExM+sUyt2jmM/a1+MPJuk9dGXda/bXucFLkq9oMDNrhogo+EN6uRPFeJLr8cell0Iu\nioiFkt4jefjbUJJLNY9hzR2raynlVVvtSXV1NdXV1eUOI1ME1NXBypVrphUr1l5uibJZs6oZMqSa\njz+moKm2Frp2hW7d8k9Z61p7qqyEYp5l2x7+LlqL22INFflA5JImCkm3kNwZuqmSJ42eT9JbICKu\niYj/U/JUzJdJHv51crquTtLpJM/xqQDG5NwLYEWor2/5N+INKZOgRw/o3n3NtL7lfGW9e8OAAfnr\n3XYbnHJK4W++XbsW9+Zr1tmU+qqnvL2ARnVOb6L8XpLHClseH34ITz4Jjz+eTFOmwF//uu6bc23t\num+mhb4ZNy7baKPmb9swVbZCH3baNDjggNIfx6yzKPfQkxWgrg5mzlyTFB5/HF5/HXbbDYYPhxNP\nhCOOqOI//3PdN+du3Trfp+Wqqqpyh9BmuC3WcFs0X0nvzC41SdGe488nAubNWzspPP00DBmSJIWG\naeedkyETM7NiSSrqZLYTRZktXQpTp65JCo89lvQgcpPC5z4HG29c7kjN2q5iT852JvneI50o2rBV\nq2DWrLV7C6+8ArvssnZiGDq08w0XmW2I9I2v3GG0OU21ixNFG/Lmm2snhalTYbPN1k4Ku+ySnEcw\ns+ZzosjPiYK2lSg++ig5l9AwfPT447BsGQwbliSEvfZK5vv3L3ekZh2PE0V+ThS0jUTxy1/CnXfC\nCy/ATjut3Vv4zGc8hGTWGpwo8mupROHLYzfAPffA9dcn0267JZemmpm1pNNOO41Bgwbxs5/9rGwx\nuEfRTIsXJz2IG2+E/fcvSwhmlmrLPYqhQ4dy7bXXckAZ7gJtqR5FW3h6bLs0ejSMHOkkYWbZspJY\nXV1dK0fTPE4UzfDoo3D33XDppeWOxMzashNPPJHXX3+dQw89lD59+vCrX/2KLl26cO2117LVVlsx\nYsQIAI466ig233xzNt54Y/bff39mzZq1eh8nnXQS5557LgA1NTUMHjyYyy67jIEDB7LFFlswduzY\nkr8OJ4oirVwJo0bB5ZfDJpuUOxoza8tuuOEGhgwZwt///neWLFnC0UcfDcAjjzzC7Nmzue+++wD4\n8pe/zMsvv8w777zD7rvvzgknnLB6H5LWuqFw4cKFLF68mAULFjBmzBi++93v8uGHH5b0dThRFOni\ni2GbbeDII8sdiZkVSmqZaUM1DEFVV1fTs2dPunfvDiS9ht69e9O1a1fOP/98pk+fzpIlS9bZDqBr\n166cd955VFRUcPDBB7PRRhvxwgsvbHhwGXzVUxGefx6uvDJ5OqkvezVrP9raee4tt1zzvWz19fWc\nc8453HHHHbzzzjt06ZJ8fn/33Xfp06fPOtsOGDBgdR2AXr16sXTp0pLG6x5FgerrkyGn6moYPLjc\n0ZhZe5HvOVS5ZTfddBPjx49n0qRJfPjhh8yZMwdYuxdR7mdZOVEU6JprkmRx2mnljsTM2pOBAwfy\nyiuvNLl+6dKldO/enf79+7Ns2TLOOeectdZHRNkv/XWiKMD8+XDeefDnP0MXt5iZFWH06NFceOGF\n9O/fnzvvvHOd3sE3vvENttpqKwYNGsTOO+/M3nvvvVadxiezy9G78A13BTjiCPiP/4Cf/7zkhzKz\nZmjLN9yVkx/h0Uruugtmz4Zx48odiZlZeThRZFi0CM48M0kS6VVsZmadjoeeMpx6anIZ7B//WLJD\nmFkL8NBTfh56KrFHHoF//AOee67ckZiZlZev4cljxQr49rfhiiugX79yR2NmVl4lTRSSRkqaLekl\nST/Ns34TSXdLmi7pcUk75aybK2mGpGmSnihlnI397//CjjsmVzuZmXV2JTtHIakCeAEYAcwHngSO\ni4jnc+r8ClgcEb+QtB1wVUSMSNfNAfaIiPczjtHi5yiefRa+8AWYPh222KJFd21mJeJzFPm1h++j\nGAa8HBFzI6IWGAcc3qjODsBDABHxAjBU0idy1rfqnSWrViWP6fjFL5wkzMwalDJRDALeyFmel5bl\nmg58DUDSMGAroOFJSgE8IGmqpFEljHO1P/wBKiuT8xNmZpYo5VVPhfQDLwYulzQNmAlMA1al6/aL\niAVpD+N+SbMjYnLjHVRXV6+er6qqoqqqqlnBvvFGcuf15Ml+TIeZtZyW+CrUsWPHMmbMGCZPXuct\nsCA1NTXU1NQ0+/ilTBTzgS1zlrck6VWsFhFLgP9qWE7PS7yarluQ/nxH0t0kQ1mZiaK5IuC734Uz\nzoDtt9/g3ZmZrdYWzp80/hD98yKfR1TKz85TgW0kDZXUDTgGGJ9bQVK/dB3p8NLDEbFUUi9JfdLy\n3sBBJD2OkrjjDnjlFTj77FIdwcw6o8ZfhfrrX/+axx57jH322YdNNtmEXXfdlYcffnh1/bFjx/Lp\nT3+avn37svXWW3PzzTcze/Zs/vu//5spU6bQp08f+vfv3/ovpOERtqWYgINJrnx6GRidlp0KnJrO\n752unw3cAfRLyz8FPJNOzzZsm2f/saHefz9i880jHn10g3dlZmXSEu8FpTJ06NCYNGlSRETMmzcv\nBgwYEPfee29ERNx///0xYMCAePfdd2Pp0qXRt2/fePHFFyMi4q233ornnnsuIiLGjh0b++23X9HH\nbqpd0vKC38tLemd2RNwL3Nuo7Jqc+SnAdnm2mwPsWsrYGvzkJ/C1r8E++7TG0cysHPTzlrmAMs7f\nsCGkG2+8kUMOOYSRI0cCMGLECPbcc0/+8Y9/cOSRR9KlSxdmzpzJ4MGDGThwIAMHDkyOW+ahq079\nCI+aGpg4Mbl3wsw6rg19g28pr732GrfffjsTJkxYXVZXV8cBBxxAr169uPXWW/n1r3/NKaecwr77\n7stvfvMbtttunc/Sra7TXt+zfHlyGeyVV0LfvuWOxsw6qtwvGhoyZAgnnngiH3zwweppyZIlnHXW\nWQAcdNBBTJw4kbfeeovtt9+eUaNGrbOPcui0ieLCC2GXXeCww8odiZl1ZLlfhXrCCScwYcIEJk6c\nyKpVq1ixYgU1NTXMnz+ft99+m3vuuYdly5bRtWtXevfuTUVFxep9zJs3j9ra2rK8hk75mPEZM2DE\niOQxHZtvXoLAzKxVtYVLUJsyfvx4zjjjDBYvXsy5557Lfvvtx1lnncXMmTOpqKhg+PDh/OEPf6Cy\nspJjjz2WZ555BknstttuXH311Wy//fbU1tZyxBFHMGXKFCoqKnj77bcLOnZLPcKj0yWKVatg772T\nYadvfatEgZlZq2rLiaKc2sOzntqkK6+EXr3glFPKHYmZWfvQqXoUr70Ge+wB//43bLttCQMzs1bl\nHkV+7lEUKQK+8x34wQ+cJMzMitFpEsWttyYP/vvJT8odiZlZ+9Iphp7eew923hn+9jcYPrwVAjOz\nVuWhp/x81ROFJ4qTT05uqrv88lYIysxanRNFfi2VKDr8IzwmTYIHH/RjOsw6unLfvdyRdehEsXw5\nnHoqXH019OlT7mjMrFTcmyitDj30dPbZySWxt9zSikGZmbVxHnpKTZsG112XPK7DzMyar0NeHltX\nB6NGwcUXQ/o4dzMza6YOmSh+/3vo1w9OOqnckZiZtX8dcuhpzz2Tx4f7Iggzsw3XoU9mm5nZuvys\nJzMza1FOFGZmlsmJwszMMjlRmJlZppImCkkjJc2W9JKkn+ZZv4mkuyVNl/S4pJ0K3dbMzFpHyRKF\npArgSmAksCNwnKQdGlU7B3g6InYBvgFcXsS2ZmbWCkrZoxgGvBwRcyOiFhgHHN6ozg7AQwAR8QIw\nVNInC9zWzMxaQSkTxSDgjZzleWlZrunA1wAkDQO2AgYXuK2ZmbWCUt6ZXcidcBcDl0uaBswEpgGr\nCtwWgOrq6tXzVVVVVFVVFRWkmVlHV1NTQ01NTbO3L9md2ZL2AqojYmS6PBqoj4hLMraZA3wW2LmQ\nbX1ntplZ8drSndlTgW0kDZXUDTgGGJ9bQVK/dB2SRgEPR8TSQrY1M7PWUbKhp4iok3Q6cB9QAYyJ\niOclnZquv4bkiqaxkgJ4Fjgla9tSxWpmZk3zQwHNzDqZtjT0ZGZmHYAThZmZZXKiMDOzTE4UZmaW\nyYnCzMwyOVGYmVkmJwozM8vkRGFmZpmcKMzMLJMThZmZZXKiMDOzTE4UZmaWyYnCzMwyOVGYmVkm\nJwozM8vkRGFmZpmcKMzMLJMThZmZZXKiMDOzTE4UZmaWyYnCzMwyOVGYmVkmJwozM8tU0kQhaaSk\n2ZJekvTTPOv7SZog6RlJz0o6KWfdXEkzJE2T9EQp4zQzs6YpIkqzY6kCeAEYAcwHngSOi4jnc+qc\nA/SJiNGSNk3rD4yIOklzgD0i4v2MY0Sp4jcz66gkEREqtH4pexTDgJcjYm5E1ALjgMMb1akH+qbz\nfYH3IqIuZ33BL8TMzEqjlIliEPBGzvK8tCzXlcCOkhYA04Hv5awL4AFJUyWNKmGcZmaWobKE+y5k\nTGgk8HREfEHSp4H7Je0SEUuAfSPiTUmfSMtnR8Tkxjuorq5ePV9VVUVVVVXLRG9m1kHU1NRQU1PT\n7O1LeY5iL6A6Ikamy6OB+oi4JKfO34GLIuLRdHkS8NOImNpoX+cDSyPiN43KfY7CzKxIbekcxVRg\nG0lDJXUDjgHGN6rzOsnJbiQNBLYDXpXUS1KftLw3cBAws4SxmplZE0o29JReuXQ6cB9QAYyJiOcl\nnZquvwb4BTBW0gySE9dnRcT7krYG7pLUEONNETGxVLGamVnTChp6ktQL2DIiXih9SIXz0JOZWfFa\nfOhJ0mHANJKeAZJ2k9R4CMnMzDqoQs5RVAPDgQ8AImIasHUJYzIzszakkERRGxGLGpXVlyIYMzNr\newo5mf2cpBOASknbAGcC/y5tWGZm1lYU0qM4HdgJWAncAiwGvl/KoMzMrO3IvOpJUiVwf0R8ofVC\nKpyvejIzK16LXvWUPqCvXtLGGxyZmZm1S4Wco1gGzJR0fzoPEBFxZunCMjOztqKQRHFXOjWM8YjC\nHvhnZmYdQKF3ZncHtk0XZ6ffL1F2PkdhZla8Ys9RrLdHIakKuB54LS0aIumbEfFw80I0M7P2ZL09\nCklPk3yF6Qvp8rbAuIjYvRXiy+QehZlZ8UrxmPHK3IcBRsSLlPYLj8zMrA0p5A3/KUl/AW4kOZF9\nAsl3TZiZWSdQyNBTD+C7wL5p0WTg6ohYWeLY1stDT2ZmxSt26KmQRNEbWBERq9LlCqB7RHy0QZG2\nACcKM7PileIcxYNAz5zlXsADxQZmZmbtUyGJontELG1YiIglJMnCzMw6gUISxTJJezQsSNoTWF66\nkMzMrC0p5Kqn7wO3SXozXd4MOLZ0IZmZWVvSZI9C0jBJm0fEk8AOwDjgY5Lvzn61leIzM7Myyxp6\nuobky4oA9gL+B7iK5Luz/1TiuMzMrI3IGnrqEhHvp/PHANdExJ3AnZKmlz40MzNrC7J6FBWSuqbz\nI4CHctYV9AgPSSMlzZb0kqSf5lnfT9IESc9IelbSSYVua2ZmrSPrDf8W4GFJ7wIfkdyRjaRtgEXr\n23F6Y96VJElmPvCkpPER8XxOte8Cz0bEoZI2BV6QdCPJ912sb1szM2sFTSaKiPilpAdJrnKaGBH1\n6SoBZxSw72HAyxExF0DSOOBwIPfNvh7om873Bd6LiDpJexewrZmZtYLMIaSImJKn7MUC9z0IeCNn\neR4wvFGdK4EJkhYAfYCji9jWzMxaQSkfF17IQ5hGAk9HxBckfRq4X9IuxRykurp69XxVVRVVVVXF\nbG5m1uHV1NRQU1PT7O0L+irUZu1Y2guojoiR6fJooD4iLsmp83fgooh4NF2eBPyUJIFlbpuW+6GA\nZmZFKsVDAZtrKrCNpKGSupFcYju+UZ3XSU5YI2kgsB3JzXyFbGtmZq2gZENP6Unp00nu5K4AxkTE\n85JOTddfA/wCGCtpBslJ8rMa7t3It22pYjUzs6aVbOipNXjoycyseG1p6MnMzDoAJwozM8vkRGFm\nZpmcKMzMLJMThZmZZXKiMDOzTE4UZmaWyYnCzMwyOVGYmVkmJwozM8vkRGFmZpmcKMzMLJMThZmZ\nZXKiMDOzTE4UZmaWyYnCzMwyOVGYmVkmJwozM8vkRGFmZpmcKMzMLJMThZmZZXKiMDOzTJXlDsA2\n3LzF85j82mR6VPagZ9ee9Kzs2eTPHpU9kFTukM2sHSlpopA0EvgdUAH8JSIuabT+x8AJObHsAGwa\nEYskzQUWA6uA2ogYVspY26P3l7/PRZMv4tpnrqVqaBWr6lexvG45y2uXN/nz41Uf072ye2YyWf2z\nkDoF/Kzs4s8jZu2ZIqI0O5YqgBeAEcB84EnguIh4von6XwG+HxEj0uU5wB4R8X7GMaJU8bdlH9V+\nxOWPXc5lj13G13f4Ouftfx5b9NmioG3ro54VdSsyk0lBP4uo20VdiksuG5iYuld0d6/JLIMkIqLg\nf5JSftQbBrwcEXMBJI0DDgfyJgrgeOCWRmX+b89Ru6qWa6ddywWPXMC+W+7Lo//1KNsO2LaofXRR\nF3p17UWvrr1KFOXaIoLa+tpmJ6RFKxYVnJg+qv2I5bXLqa2vpYu6UNmlkgpVUNmlMpnvkjOflhda\nllu+TpmasU0RZRsSb/eK7vTs2pMu8ulIa75SJopBwBs5y/OA4fkqSuoFfAn4Tk5xAA9IWgVcExF/\nLlWgbV1EcMesO/ifB/+HIf2GcM+x97DnFnuWO6yCSKJbRTe6VXSjH/1a5ZgRwapYxar6VdTV11FX\nX8eqSOazyhqWS1G2sn5lYdsWEGcxZStXrWR57XJ6VPagd7fe9O7am97detOra6/V86vLKnuttdy7\na1pvPWXuwXV8pUwUxYwJHQr8KyIW5ZTtGxFvSvoEcL+k2RExufGG1dXVq+erqqqoqqpqZrht06RX\nJ3H2pLOpj3quOuQqvvjpL5Y7pDZPEpVKPlV3p3u5wym7iGB53XKWfbyMZbXLVv/8qPajJsveWvrW\nmnXrqVtbX5s/8TQuy0pSGWVdK7qWuwnbvZqaGmpqapq9fSnPUewFVEfEyHR5NFDf+IR2uu5u4NaI\nGNfEvs4HlkbEbxqVd9hzFE8teIrRk0YzZ9EcLvzChRy101EePrA2qa6+bq1EkpVUcsuWfbyMj+rW\nX7eLuqy/h1NA4sm3fa+uvTrl/1Wx5yhKmSgqSU5mHwgsAJ4gz8lsSf2AV4HBEbE8LesFVETEEkm9\ngYnAzyNiYqNtO1yieOm9l/jZQz9j8muTOffz5/Kt3b/lT1TWaUUEH6/6uPgEVPvR6t5QVt3ltcvp\nXtm9oN5QIcNwjcva6uXobeZkdkTUSToduI/k8tgxEfG8pFPT9dekVb8K3NeQJFIDgbvTBq4Ebmqc\nJDqaN5e8yQUPX8Dts27nh3v/kGsPu5be3XqXOyyzspJE98rudK/sTv+e/Vt8/w1XARaTgBYuW8iy\nD7KH5RrqNgzLZfZwihiGa1zWraJbi7dJPiXrUbSGjtCj+HDFh1z66KX88ak/cvKuJzN6v9EM6DWg\n3GGZWQsoZFhurR5QvoSVUVdSYT2cRknme3t9r230KCzbiroVXPXEVVzy6CV8ZduvMO3UaQzpN6Tc\nYZlZC6rsUknf7n3p271vi+87d1iuqWTTuGzRikXMXzK/6GO5R9HKVtWv4q/T/8r5Neez++a788sD\nfslOn9whWmGVAAAMNElEQVSp3GGZWSfSZs5R2NoigvEvjOecB89hQM8BjDtyHPtsuU+5wzIzWy8n\nilYw+bXJnD3pbJasXMKlIy7lkG0OaZNXQpiZ5eNEUUIzF85k9KTRPPfOc1xQdQHHf/Z4KrpUlDss\nM7OiOFGUwNxFcznvofOY+MpERu83mjuPvpPulb5D2MzaJyeKFvTOsnf45eRfcsOMGzhj2Bm8dMZL\n9Onep9xhmZltECeKFrBk5RIum3IZVzxxBcd/9nhmfWcWAzcaWO6wzMxahBPFBogI/vTUn6h+uJoD\nP3UgT4x6gq032brcYZmZtSgnimZatGIRJ99zMvMXz+feE+5l1812LXdIZmYl0fkem9gCpi6Yyu7X\n7M6QvkP413/9y0nCzDo09yiKEBFc/eTV/Pzhn3P1l6/myB2PLHdIZmYl50RRoMUrFzNqwihefO9F\n/n3Kv/lM/8+UOyQzs1bhoacCTH9rOnv+aU826bEJU06Z4iRhZp2KexQZIoIx08YwetJofvel33HC\nf5xQ7pDMzFqdE0UTln28jNP+cRpPv/k0k0+ezPabbl/ukMzMysJDT3nMemcWw/4yjIouFTz+rced\nJMysU3OiaOSG6Tew/9j9+fHeP+a6w6/z15GaWafnoafU8trlnHnvmUx+fTIPfuNBPjvws+UOycys\nTXCPAnjxvRfZa8xeLK1dypOjnnSSMDPL0ekTxW3P3cZ+1+7HaXuexs1fu9lPezUza6TTDj2trFvJ\njyb+iH++/E/++f/+ye6b717ukMzM2qROmSjmfDCHo24/iq023oqnvv0U/Xr0K3dIZmZtVkmHniSN\nlDRb0kuSfppn/Y8lTUunmZLqJG1cyLbN9bfZf2P4X4Zz4n+cyB1H3eEkYWa2HoqI0uxYqgBeAEYA\n84EngeMi4vkm6n8F+H5EjCh0W0lRaPy1q2o5+4GzufP5O7n1yFsZPnh4c1+amVm7JomIUKH1Szn0\nNAx4OSLmAkgaBxwO5E0UwPHALc3cNtPrH77OMXccw6a9NuXpU5+mf8/+zdmNmVmnVMqhp0HAGznL\n89KydUjqBXwJuLPYbdfn/176P4b9eRhHbH8E9xx7j5OEmVmRStmjKGZM61DgXxGxqNhtq6urV89X\nVVVRVVUFQF19Hec+eC43zryRO46+g/2G7FdEOGZmHUdNTQ01NTXN3r6U5yj2AqojYmS6PBqoj4hL\n8tS9G7g1IsYVs21T5ygWLFnAcXceR4/KHtxwxA18svcnW/rlmZm1W8WeoyhloqgkOSF9ILAAeIL8\nJ6T7Aa8CgyNieZHb5k0UZz9wNr279uac/zyHii4VLf7azMzaszaTKNJgDgZ+B1QAYyLiIkmnAkTE\nNWmdbwJfiojj17dtnv3nTRQRgVRwG5iZdSptKlGUWjGXx5qZWaLYRNHpn/VkZmbZnCjMzCyTE4WZ\nmWVyojAzs0xOFGZmlsmJwszMMjlRmJlZJicKMzPL5ERhZmaZnCjMzCyTE4WZmWVyojAzs0xOFGZm\nlsmJwszMMjlRmJlZJicKMzPL5ERhZmaZnCjMzCyTE4WZmWVyojAzs0xOFGZmlsmJwszMMpU0UUga\nKWm2pJck/bSJOlWSpkl6VlJNTvlcSTPSdU+UMk4zM2tayRKFpArgSmAksCNwnKQdGtXZGLgKODQi\ndgaOzFkdQFVE7BYRw0oVZ0dRU1NT7hDaDLfFGm6LNdwWzVfKHsUw4OWImBsRtcA44PBGdY4H7oyI\neQAR8W6j9SphfB2K/wnWcFus4bZYw23RfKVMFIOAN3KW56VlubYB+kt6SNJUSSfmrAvggbR8VAnj\nNDOzDJUl3HcUUKcrsDtwINALmCLpsYh4CdgvIhZI+gRwv6TZETG5hPGamVkeiijk/bwZO5b2Aqoj\nYmS6PBqoj4hLcur8FOgZEdXp8l+Af0bEHY32dT6wNCJ+06i8NMGbmXVwEVHw0H4pexRTgW0kDQUW\nAMcAxzWqcw9wZXriuzswHLhMUi+gIiKWSOoNHAT8vPEBinmhZmbWPCVLFBFRJ+l04D6gAhgTEc9L\nOjVdf01EzJb0T2AGUA/8OSJmSdoauEtSQ4w3RcTEUsVqZmZNK9nQk5mZdQzt9s7sQm7m66gkXStp\noaSZOWX9Jd0v6UVJE9N7VDo0SVumV8w9l96weWZa3hnbooekxyU9k7ZFdVre6dqigaSK9IbdCely\np2yLfDcvF9sW7TJRFHIzXwd3Hclrz3U2cH9EbAtMSpc7ulrgBxGxE7AX8N3076DTtUVErAC+EBG7\nArsCIyUNpxO2RY7vAbNYcwVmZ22LfDcvF9UW7TJRUNjNfB1WepnwB42KDwOuT+evB77aqkGVQUS8\nFRHPpPNLgedJ7tXpdG0BEBEfpbPdSC49DzppW0gaDBwC/IU1N+52yrZINb7wp6i2aK+JopCb+Tqb\ngRGxMJ1fCAwsZzCtLb26bjfgcTppW0jqIukZktc8MSKeoJO2BfBb4CckF8k06Kxtke/m5aLaopSX\nx5aSz8BniIjoTPeYSNoIuBP4XnpJ9ep1naktIqIe2FVSP+BuSTs3Wt8p2kLSV4C3I2KapKp8dTpL\nW6T2jYg3c29ezl1ZSFu01x7FfGDLnOUtSXoVndlCSZsBSNoceLvM8bQKSV1JksQNEfG3tLhTtkWD\niPgQeAj4Ep2zLfYBDpM0B7gFOEDSDXTOtiAi3kx/vgPcTTJ0X1RbtNdEsfpmPkndSG7mG1/mmMpt\nPPDNdP6bwN8y6nYISroOY4BZEfG7nFWdsS02bbhyRVJP4Isk52w6XVtExDkRsWVEfAo4FngwIk6k\nE7aFpF6S+qTzDTcvz6TItmi391FIOhj4HWtu5ruozCG1Gkm3APsDm5KML55Hcpf7bcAQYC5wdEQs\nKleMrUHSfsAjJDdsNvwhjwaeoPO1xWdJTkpWkHwAvDUiLpTUn07WFrkk7Q/8KCIO64xtIelTJL0I\nWHPz8kXFtkW7TRRmZtY62uvQk5mZtRInCjMzy+REYWZmmZwozMwskxOFmZllcqIwM7NMThTWYUi6\nSFKVpK9KKurJoJI+kT6m+ylJ+zZaV5M+0n5aOt2WsZ9d0nt8GpYPbanH4Ev6fnoznVmrcqKwjmQY\n8BjJzYiPFLntgcCMiNgjIh5ttC6A49PHNO8WEUdn7Gc3kqeWJhtGTMj9nvgN9D2gVzEbSPL/uG0w\n/xFZuyfpUknTgc8BU4BTgD9I+lmeukMlPShpuqQH0i8/2hW4BDg87TH0yHeYPPs6StLM9MuCatLn\nTl0AHJPu52hJJ0m6Iq0/VtLVkqZIekXS/kq+hGqWpOty9nu1pCe19hcQnQlsATwkaVJadlz6hTQz\nJV2cs/1SSb9OnyS7t6SLlXy503RJv2pmM1tnFhGePLX7CdgTuJzkMQX/yqg3ATgxnT8ZuDud/ybw\n+ya2qQFmA9PS6ZK0fAaweTrfN99+0uUr0vmxwM3p/GHAh8BOJEloKrBLum6T9GcFycP9dk6X5wD9\n0/ktgNeAAWm9ScDh6bp64Mh0fgAwOyeevuX+XXlqf5N7FNZR7EHyxr0DycPwmrIXcHM6fyOwXzov\n8vQaUo2HnhrOOTwKXC/pW6x5ZP/69jMhnX8WWBgRz0VEAM8BQ9N1x0h6CniaJJHsmGdfnwMeioj3\nImIVcBPw+XTdKpIn6kKSjFZIGiPpCGB5E7GZNam9fh+FGZCcPCb5pD4YeJdkDF+Sngb2ieQrQtfZ\nrCWOHRGnSRoGfBl4StIeBWz2cfqzHliZU14PVKQPcfsRsGdEfJgOSeUbCgvWfh1izYMRV6TJh4io\nS2M8EDgSOD2dNyuYexTWrkXE9IjYDXgxInYAHgQOiojdm0gS/yZ59DTACRR+0jvfOYpPR8QTEXE+\n8A5JsloM9Mnabj3H6AMsAxZLGggcnLN+CdA3nX8S2F/SACXfIX8s8HCeGHsDG0fEvcAPgV2KiMcM\ncI/COoD0m7veTxe3j4jZGdXPAK6T9BOSL2s5OS0Psr858SZJDcM270TEQcClkrYheYN/ICJmSHoD\nOFvSNOCiPPttah6SLxubkW47m+Trfv+Vs/5PwD8lzY+IA9NLgB9Kj//3iGgY1srdbx/gnvQEvYAf\nZLxGs7z8mHEzM8vkoSczM8vkRGFmZpmcKMzMLJMThZmZZXKiMDOzTE4UZmaWyYnCzMwyOVGYmVmm\n/w+Sf+4FiBigYQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10c62f510>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "plt.plot(n_est,train_res,label=\"train\")\n",
    "plt.plot(n_est,test_res, label=\"test\" ) \n",
    "plt.xlabel(\"# of Estimators\"), plt.ylabel(\"Score\"), plt.title(\"Bagging Accuracy vs # Estimator\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bagging works very well when the base classifier has a low bias but a high variance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------\n",
    "# Random Forests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random Forests are very popular ensemble classifiers. They are relatively simple to use (very few parameters to set and easy to avoid overfitting). \n",
    "\n",
    "The only parameter we are really worried about is the number of trees we want to create - n_estimators in sklearn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Default # of vars is sqrt(features)\n",
    "model_rf = RandomForestClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use predict using our 20-newsgroup dataset above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "            oob_score=False, random_state=None, verbose=0,\n",
       "            warm_start=False)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_rf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.99508357915437562"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_rf.score(X_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the results and parameters of each individual estimator in the random forest through `estimators_`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "             max_features='auto', max_leaf_nodes=None, min_samples_leaf=1,\n",
       "             min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "             random_state=591477841, splitter='best'),\n",
       " DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "             max_features='auto', max_leaf_nodes=None, min_samples_leaf=1,\n",
       "             min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "             random_state=1521646388, splitter='best')]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_rf.estimators_[0:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random Forests can quantify the importance of features. Unlike with logistic regression, we don't have coeffcients that tell us relative impact. But we can keep track of what features give us the best splits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0.014357010133987014, u'space'), (0.012158020805175331, u'nasa'), (0.011401028761047831, u'christian'), (0.0098207943619235977, u'writes'), (0.0089695826037428823, u'moon'), (0.0084204948430040959, u'file'), (0.0079924041564997782, u'keith'), (0.0079806783417498313, u'launch'), (0.0078274256240183863, u'looking'), (0.0076880337595739657, u'access'), (0.007435626854264593, u'political'), (0.0068661468518484686, u'say'), (0.0067547888254878997, u'god'), (0.0060847947468401584, u'orbit'), (0.0060524954430212722, u'image'), (0.0059407654127502322, u'atheists'), (0.0054410337013916701, u'digex'), (0.0053384400199401431, u'toronto'), (0.0051268680343381692, u'atheism'), (0.0051179307653781477, u'color')]\n"
     ]
    }
   ],
   "source": [
    "# This prints the top 10 most important features\n",
    "print sorted(zip(model_rf.feature_importances_, \n",
    "                 vectorizer.get_feature_names()), \n",
    "             reverse=True)[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise - Random Forest Estimator Size\n",
    "\n",
    "* Compare training vs testing performance as you increase the number of estimators (i.e. from 1 - 100 for instance). Plotting will help visualize this.\n",
    "* In addition, anecdotally compare the performance to that of bagged trees.\n",
    "\n",
    "Extra: If you finish early, look at the top feature importances for a random forest. Is this similar to the feature importances coming out of a bagged model? Since the BaggingClassifier is generic and does not have a feature\\_importances\\_ attribute, you may need to roll your own baggingclassifier (i.e. generate your own function to bootstrap samples as well as create decision trees through a for loop, while accessing the feature\\_importances\\_ of each individual tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<2034x33815 sparse matrix of type '<type 'numpy.float64'>'\n",
       "\twith 229714 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[np.random.choice(range(X_train.shape[0]),X_train.shape[0]),:].copy()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For ensemble methods, we would like our base classifiers to be *better than random*, as well as *uncorrelated*.\n",
    "\n",
    "Bagging reduces variance by averaging over many trees. Random forests purposefully add \"randomness\" through the random features and decreases correlations of these trees. Think of random forest as an improved bagging method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "# Boosting\n",
    "\n",
    "Adaboost is a boosting implementation by Freund and Schapire that fit a sequence of weak learners. Each step improves upon the last one by overweighting misclassified samples and underweighting correct samples. The weak learners are also given weights based on their error rates.\n",
    "\n",
    "There are many components in AdaBoost--Let's see how they function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Default decision tree classifier\n",
    "#  Notice the learning rate\n",
    "model_boost = AdaBoostClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SAMME works off of discrete misclassification errors, while SAMME.R uses the predicted probabilities to reweight the samples and base classifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None,\n",
       "          learning_rate=1.0, n_estimators=50, random_state=None)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_boost.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7464892830746489"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_boost.score(X_train,y_train)\n",
    "model_boost.score(X_test,y_test) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boosting tends to perform better when the base classifier has a high bias but low variance.\n",
    "\n",
    "I.e. Random forest works well with **OVERFIT** models, while Boosting works well with **UNDERFIT** models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise - Stumps vs. Dense Trees\n",
    "\n",
    "Remember that in order for ensemble learners to work well, they require that the base classifiers are better than random guessing, and tend to get 'different questions wrong'. \n",
    "\n",
    "Boosting often does surprisingly well when using decision tree stumps (i.e. Decision Tree Classifiers with depth=1).\n",
    "\n",
    "Create a plot showing in sample and out of sample performance for an adaboost ensemble classifier that uses base_estimator as Decision Tree Classifiers with max depth = 1 and 10, as you increase the number of estimators.\n",
    "\n",
    "The x-axis should be n_estimators while y-axis is the score (or error = 1 - score). There should be two lines, one for stump and 1 for 10 node trees.\n",
    "\n",
    "Warning: Make sure your code works correctly--this can take time! I encourage you to try printing out diagnostics too to help you see that it is working."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, just using stumps, observe the performance in and out of sample as you increase the n_estimators from 1 to 100. You may want to test various ranges, rather than every number between 1 and 100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you finish early, feel free to play around with the GBDT (Gradient Boosted Decision Tree), aka GBT (Gradient Boosted Tree), aka GBM (Gradient Boosted Model) within sklearn. [GBDT](http://scikit-learn.org/stable/modules/ensemble.html#gradient-tree-boosting)\n",
    "\n",
    "Gradient boosting can be thought of as a generic form of boosting that can take in any differentiable loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}