ml_bayes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "This guide was written in Python 3.6.\n",
    "\n",
    "### Python and Pip\n",
    "\n",
    "If you haven't already, please download [Python](https://www.python.org/downloads/) and [Pip](https://pip.pypa.io/en/stable/installing/).\n",
    "\n",
    "Let's install the modules we'll need for this tutorial. Open up your terminal and enter the following commands to install the needed python modules: \n",
    "\n",
    "```\n",
    "pip3 install scikit-learn==0.18.1\n",
    "pip3 install nltk==3.2.4\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this tutorial set, we'll review the Naive Bayes Algorithm used in the field of machine learning. Naive Bayes works on Bayes Theorem of probability to predict the class of a given data point, and is extremely fast compared to other classification algorithms. \n",
    "\n",
    "Because it works with an assumption of independence among predictors, the Naive Bayes model is easy to build and particularly useful for large datasets. Along with its simplicity, Naive Bayes is known to outperform even some of the most sophisticated classification methods.\n",
    "\n",
    "This tutorial assumes you have prior programming experience in Python and probablility. While I will overview some of the priciples in probability, this tutorial is **not** intended to teach you these fundamental concepts. If you need some background on this material, please see my tutorial [here](https://github.com/lesley2958/intro-stats).\n",
    "\n",
    "\n",
    "### Bayes Theorem\n",
    "\n",
    "Recall Bayes Theorem, which provides a way of calculating the *posterior probability*: \n",
    "\n",
    "$$ P(A \\mid B) = \\frac{P(B \\mid A) \\, P(A)}{P(B)} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we go into more specifics of the Naive Bayes Algorithm, we'll go through an example of classification to determine whether a sports team will play or not based on the weather. Specifically, we'll classify whether or not a team will play if it is sunny.\n",
    "\n",
    "To start, we'll load in the data, which you can find [here](https://github.com/lesley2958/ml-bayes/blob/master/data/weather.csv)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "f1 = pd.read_csv(\"./data/weather.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we go any further, let's take a look at the dataset we're working with. It consists of 2 columns (excluding the indices), *weather* and *play*. The *weather* column consists of one of three possible weather categories: `sunny`, `overcast`, and `rainy`. The *play* column is a binary value of `yes` or `no`, and indicates whether or not the sports team played that day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Weather</th>\n",
       "      <th>Play</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Sunny</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Overcast</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Rainy</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Weather Play\n",
       "0     Sunny   No\n",
       "1  Overcast  Yes\n",
       "2     Rainy  Yes"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Frequency Table\n",
    "\n",
    "If you recall from probability theory, frequencies are an important part of eventually calculating the probability of a given class. In this section of the tutorial, we'll first convert the dataset into different frequency tables, using the `groupby()` function. First, we retrieve the frequences of each combination of weather and play columns: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weather   Play\n",
      "Overcast  Yes     4\n",
      "Rainy     No      3\n",
      "          Yes     2\n",
      "Sunny     No      2\n",
      "          Yes     3\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "df = f1.groupby(['Weather','Play']).size()\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It will also come in handy to split the frequencies by weather and yes/no. Let's start with the three weather frequencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Play\n",
      "Weather       \n",
      "Overcast     4\n",
      "Rainy        5\n",
      "Sunny        5\n"
     ]
    }
   ],
   "source": [
    "df2 = f1.groupby('Weather').count()\n",
    "print(df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now for the frequencies of yes and no:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      Weather\n",
      "Play         \n",
      "No          5\n",
      "Yes         9\n"
     ]
    }
   ],
   "source": [
    "df1 = f1.groupby('Play').count()\n",
    "print(df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Likelihood Table\n",
    "\n",
    "The frequencies of each class are important in calculating the likelihood, or the probably that a certain class will occur. Using the frequency tables we just created, we'll find the likelihoods of each weather condition and yes/no. We'll accomplish this by adding a new column that takes the frequency column and divides it by the total data occurances:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      Weather  Likelihood\n",
      "Play                     \n",
      "No          5    0.357143\n",
      "Yes         9    0.642857\n",
      "          Play  Likelihood\n",
      "Weather                   \n",
      "Overcast     4    0.285714\n",
      "Rainy        5    0.357143\n",
      "Sunny        5    0.357143\n"
     ]
    }
   ],
   "source": [
    "df1['Likelihood'] = df1['Weather']/len(f1)\n",
    "df2['Likelihood'] = df2['Play']/len(f1)\n",
    "print(df1)\n",
    "print(df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we're able to use the Naive Bayesian equation to calculate the posterior probability for each class. The highest posterior probability is the outcome of prediction.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculation\n",
    "\n",
    "Now, let's get back to our question: *Will the team play if the weather is sunny?*\n",
    "\n",
    "From this question, we can construct Bayes Theorem. Because the *know* factor is that it is sunny, the $P(A \\mid B)$ becomes $P(Yes \\mid Sunny)$. From there, it's just a matter of plugging in probabilities. \n",
    "\n",
    "$$ P(Yes \\mid Sunny) = \\frac{P(Sunny \\mid Yes) \\, P(Yes)}{P(Sunny)} $$\n",
    "\n",
    "Since we already created some likelihood tables, we can just index `P(Sunny)` and `P(Yes)` off the tables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ps = df2['Likelihood']['Sunny']\n",
    "py = df1['Likelihood']['Yes']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That leaves us with $P(Sunny \\mid Yes)$. This is the probability that the weather is sunny given that the players played that day. In `df`, we see that the total number of `yes` days under `sunny` is 3. We take this number and divide it by the total number of `yes` days, which we can get from `df`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "psy = df['Sunny']['Yes']/df1['Weather']['Yes']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, we can just plug these variables into bayes theorem: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6\n"
     ]
    }
   ],
   "source": [
    "p = (psy*py)/ps\n",
    "print(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tells us that there's a 60% likelihood of the team playing if it's sunny. Because this is a binary classification of yes or no, a value greater than 50% indicates a team *will* play. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naive Bayes Evaluation\n",
    "\n",
    "Every classifier has pros and cons, whether that be in terms of computational power, accuracy, etc. In this section, we'll review the pros and cons of Naive Bayes.\n",
    "\n",
    "#### Pros\n",
    "\n",
    "Naive Bayes is incredibly easy and fast in predicting the class of test data. It also performs well in multi-class prediction.\n",
    "\n",
    "When the assumption of independence is true, the Naive Bayes classifier performs better thanother models like logistic regression. It does this, and with less need of a lot of data.\n",
    "\n",
    "Naive Bayes also performs well with categorical input variables compared to numerical variable(s), which is why we're able to use it for text classification. For numerical variables, normal distribution must be assumed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cons\n",
    "\n",
    "If a categorical variable has a category not observed in the training data set, then model will assign a 0 probability and will be unable to make a prediction. This is referred to as â€œZero Frequencyâ€. To solve this, we can use the smoothing technique, such as Laplace estimation.\n",
    "\n",
    "Another limitation of Naive Bayes is the assumption of independent predictors. In real life, it is almost impossible that we get a set of predictors which are completely independent.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes Types\n",
    "\n",
    "With `scikit-learn`, we can implement Naive Bayes models in Python. There are three types of Naive Bayes models, all of which we'll review in the following sections.\n",
    "\n",
    "### Gaussian\n",
    "\n",
    "The Gaussian Naive Bayes Model is used in classification and assumes that features will follow a normal distribution. \n",
    "\n",
    "We begin an example by importing the needed modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always, we need predictor and target variables, so we assign those:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = np.array([[-3,7],[1,5], [1,2], [-2,0], [2,3], [-4,0], [-1,1], [1,1], [-2,2], [2,7], [-4,1], [-2,7]])\n",
    "y = np.array([3, 3, 3, 3, 4, 3, 3, 4, 3, 4, 4, 4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can initialize the Gaussian Classifier:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = GaussianNB()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can train the model using the training sets:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GaussianNB(priors=None)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3 4]\n"
     ]
    }
   ],
   "source": [
    "predicted = model.predict([[1,2],[3,4]])\n",
    "print(predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multinomial\n",
    "\n",
    "MultinomialNB implements the multinomial Naive Bayes algorithm and is one of the two classic Naive Bayes variants used in text classification. This classifier is suitable for classification with discrete features (such as word counts for text classification). \n",
    "\n",
    "The multinomial distribution normally requires integer feature counts. However, in practice, fractional counts may also work.\n",
    "\n",
    "First, we need some data, so we import numpy and \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "x = np.random.randint(5, size=(6, 100))\n",
    "y = np.array([1, 2, 3, 4, 5, 6])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's build the Multinomial Naive Bayes model: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "clf = MultinomialNB()\n",
    "clf.fit(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try an example:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3]\n"
     ]
    }
   ],
   "source": [
    "print(clf.predict(x[2:3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bernoulli\n",
    "\n",
    "Like MultinomialNB, this classifier is suitable for discrete data. BernoulliNB implements the Naive Bayes training and classification algorithms for data that is distributed according to multivariate Bernoulli distributions, meaning there may be multiple features but each one is assumed to be a binary value. \n",
    "\n",
    "The decision rule for Bernoulli Naive Bayes is based on\n",
    "\n",
    "![alt text](https://github.com/lesley2958/ml-bayes/blob/master/bernoulli.png?raw=true \"Logo Title Text 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "x = np.random.randint(2, size=(6, 100))\n",
    "y = np.array([1, 2, 3, 4, 4, 5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.naive_bayes import BernoulliNB\n",
    "clf = BernoulliNB()\n",
    "clf.fit(x, y)\n",
    "print(clf.predict(x[2:3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tips for Improvement\n",
    "\n",
    "If continuous features don't have a normal distribution (recall the assumption of normal distribution), you can use different methods to convert it to a normal distribution.\n",
    "\n",
    "As we mentioned before, if the test data set has a zero frequency issue, you can apply smoothing techniques â€œLaplace Correctionâ€ to predict the class.\n",
    "\n",
    "As usual, you can remove correlated features since the correlated features would be voted twice in the model and it can lead to over inflating importance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Joint Models\n",
    "\n",
    "If you have input data x and want to classify the data into labels y. A <b>generative model</b> learns the joint probability distribution `p(x,y)` and a discriminative model learns the <b>conditional probability distribution</b> `p(y|x)`.\n",
    "\n",
    "Here's an simple example of this form:\n",
    "\n",
    "```\n",
    "(1,0), (1,0), (2,0), (2, 1)\n",
    "```\n",
    "\n",
    "`p(x,y)` is\n",
    "\n",
    "```\n",
    "      y=0   y=1\n",
    "     -----------\n",
    "x=1 | 1/2   0\n",
    "x=2 | 1/4   1/4\n",
    "```\n",
    "\n",
    "Meanwhile,\n",
    "\n",
    "```\n",
    "p(y|x) is\n",
    "```\n",
    "\n",
    "```\n",
    "      y=0   y=1\n",
    "     -----------\n",
    "x=1 | 1     0\n",
    "x=2 | 1/2   1/2\n",
    "```\n",
    "\n",
    "Notice that if you add all 4 probabilities in the first chart, they add up to 1, but if you do the same for the second chart, they add up to 2. This is because the probabilities in chart 2 are read row by row. Hence, `1+0=1` in the first row and `1/2+1/2 = 1` in the second.\n",
    "\n",
    "The distribution `p(y|x)` is the natural distribution for classifying a given example `x` into a class `y`, which is why algorithms that model this directly are called discriminative algorithms. \n",
    "\n",
    "Generative algorithms model `p(x, y)`, which can be tranformed into `p(y|x)` by applying Bayes rule and then used for classification. However, the distribution `p(x, y)` can also be used for other purposes. For example you could use \n",
    "6`p(x,y)` to generate likely `(x, y)` pairs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}