Bandit Feedback - Organic best of vs Bandit best of.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bandit Feedback - Organic Best of vs Bandit Best of\n",
    "\n",
    "In this notebook we use the bandit signal for the first time, in the simplest possible way.  We compare how a recommender system that always makes the most popular organic product as a recommender with a recommender system that always makes the most popular bandit product.\n",
    "\n",
    "We see that there can be differences in behaviour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym, recogym\n",
    "from recogym import env_1_args\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "plt.rcParams['figure.figsize'] = [6, 3]\n",
    "\n",
    "ABTestNumberOfUsers=100\n",
    "NumberOfProducts=10\n",
    "NumberOfSamples = 20\n",
    "env_1_args['phi_var']=0.0\n",
    "env_1_args['number_of_flips']=10\n",
    "env_1_args['sigma_mu_organic'] = 2\n",
    "env_1_args['sigma_omega']=0\n",
    "env_1_args['random_seed'] = 42\n",
    "env_1_args['num_products'] = NumberOfProducts\n",
    "env_1_args['K'] = 5\n",
    "env_1_args['number_of_flips'] = 3\n",
    "env = gym.make('reco-gym-v1')\n",
    "env.init_gym(env_1_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from recogym.agents import OrganicUserEventCounterAgent, organic_user_count_args\n",
    "from recogym import Configuration\n",
    "\n",
    "organic_counter_agent = OrganicUserEventCounterAgent(Configuration({\n",
    "            **organic_user_count_args,\n",
    "            **env_1_args,\n",
    "            'select_randomly': True,\n",
    "        }))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "data = env.generate_logs(ABTestNumberOfUsers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the data the following columns are present:\n",
    "* **`t`**—_`Time`_. Currently, _`Time`_ reflects the order of _`Events`_ but it does **not** represent the notion of the time in a physical sense.\n",
    "* **`u`**—_`User`_.\n",
    "* **`z`**—_`Event Type`_. There are two types of _`Events`_: _Organic_ and _Bandit_.\n",
    "* **`v`**—_`View`_. The column shows which _`Product`_ was shown to a _`User`_ in an _Organic_ _`Event`_.\n",
    "* **`a`**—_`Action`_. Currently, _`Action`_ is a _`Product`_ that was provided to a _`User`_ during a _Bandit_ _`Event`_.\n",
    "* **`c`**—_`Click`_. This is a _Reward_ for an _`Action`_ provided by the _`Agent`_.\n",
    "* **`ps`**—Probability of selecting a particular _`Action`_.\n",
    "\n",
    "**Note #1:** _`Time`_, _`User`_, _`Views`_, and _`Actions`_ **all** them start with _`0`_.\n",
    "\n",
    "**Note #2:** For any _`User`_, _Organic_ _`Event`_ _**always**_ precedes a _Bandit_ _`Event`_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Analysis\n",
    "Let's analyse these histograms:\n",
    "* _Actions per Product_\n",
    "* _Clicks per Product_\n",
    "* _Views per Product_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "_, ax = plt.subplots()\n",
    "ax.set_title('Histogram of Actions per Product')\n",
    "\n",
    "ax.hist(data[data['z'] == 'bandit']['a'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is evident that all _`Actions`_ are _*evenly*_ distributed.\n",
    "That is an expected behaviour because *all* _`Actions`_ are randomly applied for all _Bandit_ _`Events`_.  This is not typical behaviour (usually actions are personalised not random) and we will relax this assumption later in the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "clicks = np.zeros(NumberOfProducts)\n",
    "bandits = data[data['z'] == 'bandit']\n",
    "for product_id in range(NumberOfProducts):\n",
    "    actions = bandits[bandits['a'] == product_id]\n",
    "    clicks[product_id] = np.sum(actions[actions['c'] == 1]['c'])\n",
    "    \n",
    "print(\"Clicks: \", clicks)\n",
    "\n",
    "_, ax = plt.subplots()\n",
    "ax.set_title('Histogram of Clicks per Product')\n",
    "\n",
    "ax.bar(range(NumberOfProducts), clicks)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can estimate a click through rate for each recommendation by dividing the number of times we obtained a click by the number of impressions.  This produces the following \"bandit best of\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "from scipy.stats.distributions import beta\n",
    "\n",
    "clicks = np.zeros(NumberOfProducts)\n",
    "impressions = np.zeros(NumberOfProducts)\n",
    "lower_errors = np.zeros(NumberOfProducts)\n",
    "upper_errors = np.zeros(NumberOfProducts)\n",
    "bandits = data[data['z'] == 'bandit']\n",
    "for product_id in range(NumberOfProducts):\n",
    "    actions = bandits[bandits['a'] == product_id]\n",
    "    clicks[product_id] = np.sum(actions[actions['c'] == 1]['c'])\n",
    "    impressions[product_id] = sum(actions['a']==product_id)\n",
    "\n",
    "    # Compute expected value according to a beta distribution\n",
    "    expected_value = beta.ppf(0.500, clicks[product_id] + 1, impressions[product_id] - clicks[product_id] + 1)\n",
    "    # Compute a confidence interval\n",
    "    lower_bound = beta.ppf(0.025, clicks[product_id] + 1, impressions[product_id] - clicks[product_id] + 1)\n",
    "    upper_bound = beta.ppf(0.975, clicks[product_id] + 1, impressions[product_id] - clicks[product_id] + 1)\n",
    "    # Save difference as error\n",
    "    lower_errors[product_id] = expected_value - lower_bound\n",
    "    upper_errors[product_id] = upper_bound - expected_value\n",
    "    \n",
    "\n",
    "_, ax = plt.subplots()\n",
    "ax.set_title('Click through rate (non personalised)')\n",
    "\n",
    "ax.errorbar(range(NumberOfProducts),\n",
    "            clicks/impressions,\n",
    "            yerr = (lower_errors, upper_errors),\n",
    "            fmt = 'o',\n",
    "            ecolor = 'darkred',\n",
    "            capsize = 4)\n",
    "\n",
    "plt.ylim([0,0.025])\n",
    "plt.xlabel('Item ID')\n",
    "plt.ylabel('Click-through rate')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The non-personalised click through rate can be uses as a non-personalised agent.  This will be our first likelihood based agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "from recogym.agents.abstract import Agent\n",
    "\n",
    "class GreedySingleActionAgent(Agent):\n",
    "    def __init__(self, preferred_action, config = Configuration({'num_products': 10})):\n",
    "        super(GreedySingleActionAgent, self).__init__(config)\n",
    "        self.preferred_action = preferred_action\n",
    "        \n",
    "    def act(self, observation, reward, done):\n",
    "        prob = np.zeros(self.config.num_products)\n",
    "        prob[self.preferred_action] = 1.\n",
    "        return {\n",
    "            **super().act(observation, reward, done),\n",
    "            **{\n",
    "                'a': self.preferred_action,\n",
    "                'ps': prob[self.preferred_action],\n",
    "                'ps-a': prob,\n",
    "            },\n",
    "        }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "# Initialise bandit\n",
    "top_ctr_item = np.argmax(clicks/impressions)\n",
    "print('The Greedy-Bandit agent will always take action {0}'.format(top_ctr_item))\n",
    "greedy_bandit = GreedySingleActionAgent(top_ctr_item, Configuration(env_1_args))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ads that in a non-personalised sense attracts the most clicks is not in general the same as a product that is often viewed organically, to see this we can plot the organic views:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "organics = data[data['z'] == 'organic']\n",
    "\n",
    "views = np.zeros(NumberOfProducts)\n",
    "for product_id in range(NumberOfProducts):\n",
    "    views[product_id] = organics[organics['v'] == product_id].shape[0]\n",
    "\n",
    "print(\"Views: \", views)\n",
    "\n",
    "_, ax = plt.subplots()\n",
    "ax.set_title('Histogram of Views per Product')\n",
    "\n",
    "ax.bar(range(NumberOfProducts), views)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "# Plot number of organic views on x-axis and CTR on y-axis\n",
    "CTR = np.array(clicks)/np.array(impressions)\n",
    "\n",
    "\n",
    "_, ax = plt.subplots()\n",
    "ax.set_title('CTR for number of organic views')\n",
    "\n",
    "ax.scatter(views, CTR, marker = '.', label = 'CTR')\n",
    "\n",
    "z = np.polyfit(views, CTR, 1)\n",
    "p = np.poly1d(z)\n",
    "ax.plot(views, p(views), color = 'darkgreen', label = 'Trend')\n",
    "\n",
    "plt.xlabel('Number of organic views')\n",
    "plt.ylabel('Click-through rate')\n",
    "\n",
    "plt.gca().set_ylim([0.007,0.016])\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While there may be a relationship between popular products and high click through rate it is certainly not guaranteed.  This fact underlies the need to use bandit feedback.  Here we will develop our final organic agent as an organic best of to hammer home this point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "# Initialise bandit\n",
    "top_viewed_item = np.argmax(views)\n",
    "print('The organic-best-of agent will always take action {0}'.format(top_viewed_item))\n",
    "greedy_organic = GreedySingleActionAgent(top_viewed_item, Configuration(env_1_args))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "from recogym.evaluate_agent import verify_agents, plot_verify_agents\n",
    "\n",
    "print('Starting A/B-testing at {0}'.format(datetime.now()))\n",
    "result = verify_agents(env,\n",
    "                       10000,\n",
    "                       {'Greedy-Bandit': greedy_bandit,\n",
    "                        'Greedy-Organic': greedy_organic})\n",
    "print('Finished A/B-testing at {0}'.format(datetime.now()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "fig = plot_verify_agents(result)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}