Bandit Feedback - Embedding-based LogReg with Bandit Feedback.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SVD-based Agent with LogReg on Bandit Feedback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Up until now, we have primarily focused on:\n",
    "- The differences between organic and bandit feedback, and how to properly exploit these signals for model evaluation\n",
    "- Building a model either entirely off of either organic, or bandit feedback\n",
    "- Feature engineering methods in combination with Logistic Regression to properly predict clicks\n",
    "\n",
    "However, we haven't yet looked into combining the organic and the bandit signal to get the best of both worlds.\n",
    "In this notebook, we show a simple example that combines an embedding-based approach (SVD) on the organic signal, with a simple Logistic Regression-based model that predicts whether a given user will interact with a recommendation, should we show it.\n",
    "In conclusion, we show that properly combining these different types of feedback yields a significant gain in model accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from numpy.random.mtrand import RandomState\n",
    "from recogym import Configuration\n",
    "from recogym.agents import Agent\n",
    "from scipy.special import logsumexp\n",
    "import pandas as pd\n",
    "from scipy.stats.distributions import beta\n",
    "from copy import deepcopy\n",
    "from recogym.agents import OrganicUserEventCounterAgent, organic_user_count_args\n",
    "\n",
    "from recogym.envs.observation import Observation\n",
    "from recogym.agents import RandomAgent, random_args\n",
    "from recogym import verify_agents, verify_agents_IPS\n",
    "from recogym.evaluate_agent import plot_verify_agents, verify_agents_recall_at_k\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "import gym\n",
    "import numpy as np\n",
    "from recogym import env_1_args, Configuration\n",
    "\n",
    "# Set style for pretty plots\n",
    "plt.style.use('ggplot')\n",
    "\n",
    "P = 10 # Number of Products\n",
    "U = 10 # Number of Users\n",
    "\n",
    "# You can overwrite environment arguments here:\n",
    "env_1_args['random_seed'] = 42\n",
    "env_1_args['num_products']= P\n",
    "env_1_args['phi_var']=0.0\n",
    "env_1_args['number_of_flips']=8\n",
    "env_1_args['sigma_mu_organic'] = 0.0\n",
    "env_1_args['sigma_omega']=0\n",
    "#env_1_args['normalize_beta']=True\n",
    "# Initialize the gym for the first time by calling .make() and .init_gym()\n",
    "env = gym.make('reco-gym-v1')\n",
    "env.init_gym(env_1_args)\n",
    "\n",
    "env.reset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Organic Users: 0it [00:00, ?it/s]\n",
      "Users: 100%|██████████| 10/10 [00:00<00:00, 14.30it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>t</th>\n",
       "      <th>u</th>\n",
       "      <th>z</th>\n",
       "      <th>v</th>\n",
       "      <th>a</th>\n",
       "      <th>c</th>\n",
       "      <th>ps</th>\n",
       "      <th>ps-a</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>organic</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>organic</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>organic</td>\n",
       "      <td>6</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>organic</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0</td>\n",
       "      <td>organic</td>\n",
       "      <td>6</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "      <td>organic</td>\n",
       "      <td>6</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6.0</td>\n",
       "      <td>0</td>\n",
       "      <td>organic</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>10.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>11.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>12.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>13.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>14.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>15.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>18.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>19.0</td>\n",
       "      <td>0</td>\n",
       "      <td>bandit</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>()</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       t  u        z     v     a    c   ps  ps-a\n",
       "0    0.0  0  organic     1  <NA>  NaN  NaN  None\n",
       "1    1.0  0  organic     1  <NA>  NaN  NaN  None\n",
       "2    2.0  0  organic     6  <NA>  NaN  NaN  None\n",
       "3    3.0  0  organic     1  <NA>  NaN  NaN  None\n",
       "4    4.0  0  organic     6  <NA>  NaN  NaN  None\n",
       "5    5.0  0  organic     6  <NA>  NaN  NaN  None\n",
       "6    6.0  0  organic     1  <NA>  NaN  NaN  None\n",
       "7    7.0  0   bandit  <NA>     7  0.0  0.1    ()\n",
       "8    8.0  0   bandit  <NA>     2  0.0  0.1    ()\n",
       "9    9.0  0   bandit  <NA>     0  0.0  0.1    ()\n",
       "10  10.0  0   bandit  <NA>     9  0.0  0.1    ()\n",
       "11  11.0  0   bandit  <NA>     7  0.0  0.1    ()\n",
       "12  12.0  0   bandit  <NA>     6  0.0  0.1    ()\n",
       "13  13.0  0   bandit  <NA>     2  0.0  0.1    ()\n",
       "14  14.0  0   bandit  <NA>     2  0.0  0.1    ()\n",
       "15  15.0  0   bandit  <NA>     2  0.0  0.1    ()\n",
       "16  16.0  0   bandit  <NA>     6  0.0  0.1    ()\n",
       "17  17.0  0   bandit  <NA>     9  0.0  0.1    ()\n",
       "18  18.0  0   bandit  <NA>     6  0.0  0.1    ()\n",
       "19  19.0  0   bandit  <NA>     3  0.0  0.1    ()"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate RecSys logs for U users\n",
    "reco_log = env.generate_logs(U)\n",
    "reco_log.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The critical part of our agent is a Logistic Regression model, that predicts whether a user will click a given recommendation, based on some features.\n",
    "In what follows, we present an overview of these simple features.\n",
    "\n",
    "- The first `K` features are the latent components representing the user state. As proposed in the well-known SVD++ approach and aligned with the previous notebooks, we decompose the item co-count matrix and represent a user with the average embedding of the items in her history.\n",
    "- Following this, we embed the specific item we want to recommend as a one-hot-encoded sparse vector of length `P`.\n",
    "- In order to further exploit the organic signal: we include a count of organic views for the given item in the training set, and the result of the dot-product between the user- and item-embedding.\n",
    "\n",
    "As discussed previously, we need cross-interaction between the user- and item-features to ensure a personalised approach.\n",
    "With this in mind, we take the Kronecker-product between the first `K` and last `P+2` features in order to obtain our final `KP+2` features.\n",
    "\n",
    "We make our agent greedy, i.e. it will always take the action it deems to have the highest probability of generating a click, without any exploratory behaviour for lesser actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "def create_embeddings(reco_log, U, P, K = 5):\n",
    "    organic_reco_log = reco_log[reco_log['z']=='organic']\n",
    "    counts = np.zeros((U,P))\n",
    "\n",
    "    for u in range(U):\n",
    "        counts[u,:] = to_categorical(np.array(organic_reco_log[organic_reco_log['u']==u].v,dtype=int),P).sum(0)\n",
    "\n",
    "    counts_above_zero = 1.*(counts>0) # above zero counts only\n",
    "    co_counts = np.matmul(counts_above_zero.T,counts_above_zero)\n",
    "    \n",
    "    w , v = np.linalg.eig(co_counts)\n",
    "    idx = np.argsort(w)[::-1] \n",
    "    v = np.real(v[:,idx])\n",
    "    w = np.real(w[idx])\n",
    "\n",
    "    wdash = np.zeros_like(w)\n",
    "\n",
    "    wdash[0:K] = w[0:K]\n",
    "    embeddings = np.matmul(v,np.sqrt(np.diag(wdash)))[:,0:K] # keep the non-zero components\n",
    "    return embeddings\n",
    "\n",
    "class SVDLogRegAgent(Agent):\n",
    "    def __init__(self, config, reco_log, U = U, P = P, K = 5, greedy = False):\n",
    "        super(SVDLogRegAgent, self).__init__(config)\n",
    "        self.rng = RandomState(self.config.random_seed)\n",
    "        self.organic_views = np.zeros(self.config.num_products)\n",
    "        assert(P >= K)\n",
    "        assert(reco_log.v.max() < P)\n",
    "        self.K = K\n",
    "        self.history_length = 0\n",
    "        self.greedy = greedy\n",
    "        self.item_popularities = defaultdict(int)\n",
    "        self.train(reco_log, U, P)\n",
    "        \n",
    "    def train(self, reco_log, U, P):\n",
    "        # Generate item embeddings by SVD on item co-count matrix\n",
    "        self.embeddings = create_embeddings(reco_log, U, P, self.K)\n",
    "        \n",
    "        # Compute item popularities\n",
    "        for row in reco_log.loc[reco_log['z'] == 'organic'].itertuples():\n",
    "            self.item_popularities[int(row.v)] += 1\n",
    "        \n",
    "        # Generate feature-matrix X to train on\n",
    "        X = []\n",
    "        y = []\n",
    "        uid = 0\n",
    "        uemb = np.zeros(self.K)\n",
    "        ucnt = 0\n",
    "        for row in reco_log.itertuples():\n",
    "            # Do we have a new user?\n",
    "            if row.u != uid:\n",
    "                # Reset the embedding\n",
    "                uid = row.u\n",
    "                uemb = np.zeros(self.K)\n",
    "                ucnt = 0\n",
    "            # Is this an organic event?\n",
    "            if row.z == 'organic':\n",
    "                # Update the user embedding\n",
    "                uemb += self.embeddings[int(row.v),:]\n",
    "                ucnt += 1\n",
    "            elif row.z == 'bandit':\n",
    "                # Pad the user embedding with zeros\n",
    "                #features = np.hstack((uemb,np.zeros(P)))\n",
    "                # One-hot encode the item in the last P columns\n",
    "                #features[self.K + int(row.a)] = 1.\n",
    "                aemb = np.zeros(P + 2)\n",
    "                aemb[int(row.a)] = 1.\n",
    "                aemb[-2] = self.item_popularities[int(row.a)]\n",
    "                dot = np.matmul(self.embeddings[int(row.a),:],uemb/ucnt)\n",
    "                aemb[-1] = dot\n",
    "                features = np.kron(uemb/ucnt, aemb)\n",
    "                X.append(features)\n",
    "                y.append(row.c)\n",
    "        X = np.asarray(X)\n",
    "        y = np.asarray(y)\n",
    "        \n",
    "        # Train Logistic Regression model\n",
    "        self.model = LogisticRegression(solver = 'liblinear',\n",
    "                                        max_iter = 200).fit(X,y)\n",
    "    \n",
    "    def observe(self, observation):\n",
    "        for session in observation.sessions():\n",
    "            self.user_embedding += self.embeddings[session['v'],:]\n",
    "            self.history_length += 1\n",
    "\n",
    "    def act(self, observation, reward, done):\n",
    "        \"\"\"Act method returns an Action based on current observation and past history\"\"\"\n",
    "        self.observe(observation)\n",
    "\n",
    "        # Generate a prediction for every possible action\n",
    "        X = []\n",
    "        for a in range(self.embeddings.shape[0]):\n",
    "            # Pad the user embedding with zeros\n",
    "            #features = np.hstack((self.user_embedding,np.zeros(self.embeddings.shape[0])))\n",
    "            # One-hot encode the item in the last P columns\n",
    "            #features[self.K + a] = 1.\n",
    "            aemb = np.zeros(self.embeddings.shape[0] + 2)\n",
    "            aemb[int(a)] = 1.\n",
    "            aemb[-2] = self.item_popularities[int(a)]\n",
    "            dot = np.matmul(self.embeddings[int(a),:],(self.user_embedding/self.history_length))\n",
    "            aemb[-1] = dot\n",
    "            features = np.kron((self.user_embedding/self.history_length), aemb)\n",
    "            X.append(features)\n",
    "        prob = self.model.predict_log_proba(np.asarray(X))[:,1]\n",
    "        prob = np.exp(prob-logsumexp(prob))\n",
    "        \n",
    "        if self.greedy:\n",
    "            a = np.argmax(prob)\n",
    "            prob = np.zeros(prob.shape)\n",
    "            prob[a] = 1.\n",
    "        \n",
    "        # Sample action according to probability distribution\n",
    "        action = self.rng.choice(self.config.num_products, p = prob)\n",
    "        return {\n",
    "            **super().act(observation, reward, done),\n",
    "            **{\n",
    "                'a': action,\n",
    "                'ps': prob[action],\n",
    "                'ps-a': prob,\n",
    "            },\n",
    "        }\n",
    "\n",
    "    def reset(self):\n",
    "        self.user_embedding = np.zeros(self.K)\n",
    "        self.history_length = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "organic_counter_agent = OrganicUserEventCounterAgent(Configuration({\n",
    "            **organic_user_count_args,\n",
    "            **env_1_args,\n",
    "            'select_randomly': True,\n",
    "        }))\n",
    "\n",
    "random_args['num_products'] = P\n",
    "agent_rand = RandomAgent(Configuration({**env_1_args, **random_args,}))\n",
    "\n",
    "SVDLR = SVDLogRegAgent(Configuration(env_1_args), reco_log, U, P, 8, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A/B-Test Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We evaluate our new model by simulating an A/B-test against two baselines: the random agent and the popularity based agent.\n",
    "As we have seen before, although the popularity baseline is naive and simple, it is not always that trivial to beat in a first try."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Organic Users: 0it [00:00, ?it/s]\n",
      "Users: 100%|██████████| 10/10 [00:00<00:00, 52.04it/s]\n",
      "Organic Users: 0it [00:00, ?it/s]\n",
      "Users: 100%|██████████| 10/10 [00:00<00:00, 35.56it/s]\n",
      "Organic Users: 0it [00:00, ?it/s]\n",
      "Users: 100%|██████████| 10/10 [00:00<00:00, 13.53it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Agent</th>\n",
       "      <th>0.025</th>\n",
       "      <th>0.500</th>\n",
       "      <th>0.975</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Random</td>\n",
       "      <td>0.005209</td>\n",
       "      <td>0.010941</td>\n",
       "      <td>0.019806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Popularity</td>\n",
       "      <td>0.007115</td>\n",
       "      <td>0.013778</td>\n",
       "      <td>0.023631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>SVD-LR</td>\n",
       "      <td>0.005209</td>\n",
       "      <td>0.010941</td>\n",
       "      <td>0.019806</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Agent     0.025     0.500     0.975\n",
       "0      Random  0.005209  0.010941  0.019806\n",
       "1  Popularity  0.007115  0.013778  0.023631\n",
       "2      SVD-LR  0.005209  0.010941  0.019806"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_AB = verify_agents(env, U, {' Random': agent_rand, 'Popularity': organic_counter_agent, 'SVD-LR': SVDLR})\n",
    "result_AB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAExCAYAAABBKLRzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deVxU190/8M9srDOoM2xBcMPgAkYNY2JwAyFPXRIfkjSamhq3tElcEmifPkRD6xJtaY3a6GCrqaFW8RUTY6xtn8SEIJpCIaASo6YRROMCiCwaEJSZuef3hz9vHQGR5c6Ift6vF694zz3n3O+Fk/nOvefMHZUQQoCIiKiTqV0dABER3ZuYYIiISBFMMEREpAgmGCIiUgQTDBERKYIJhoiIFMEEQ3c1lUqFbdu2uTqMNvv666/xyCOPwMPDA3369HFpLNHR0XjxxRcdyhYtWoSAgACoVCr8+c9/BgCsX78ewcHBUKvVWLp0qfMDpXuPoC6vsrJS/OIXvxBhYWHC3d1d+Pn5iTFjxogtW7YIq9UqANz2p3fv3kIIIcaNGyeXabVa0bt3b7FgwQJRU1Nz2+Pv27evxb4/+OCDOzqH2NhYMXPmzCblZWVloqGhoa2/kjbbunWr6Mz/HSZMmCBiY2PFqVOnREVFRaf1e8PNfyudTif8/f3F+PHjxR/+8AfR2NjoULeqqkpcvnxZ3s7NzRUAxO7du0VZWZmor68X58+fF2q1WlgsFlFaWipqa2s7Pea2Onv2rAAg9u3bd8dtcnNzhVqtFmazWbnA7lBoaKhYsmSJq8NwKa1Tsxl1urNnz2L06NHQarVYvnw5hg8fDp1Oh5ycHLz11lt46KGHUFZWJtfPycnBM888g0OHDuGBBx4AAGg0Gnn/9OnTsXr1athsNhw9ehRz585FVVUVtm/f3mosN/d5Q48ePTp0foGBgR1q7ypFRUWYOXNmh65ehBCw2WzQ6XTN7r/xt7Lb7bhw4QI+//xzLF68GFu3bsVnn30GLy8vAIDRaGwSm1qtxn//93/LZQcPHoQkSZgyZUqTv2FbWK3WFuN1ho0bN+KVV17B9u3bUVhYiGHDhrksFgKvYLq6J554QgQEBIhLly412dfY2Cjq6uocym5cbZw9e7ZJ/XHjxom5c+c6lP3sZz8TRqPxtjHcrs8bLl++LGbNmiUCAgKEm5ubCA4OFomJiUIIIWbOnNnkyufGu1YAYuvWrXI/AMS6devE1KlThZeXlwgJCREffPCBuHTpkpg+fbrQ6/Wib9++YufOnQ7HX7x4sRg4cKDw9PQUwcHB4qWXXpJ/Z81dgd18NbVu3ToxYMAA4e7uLvr37y9WrFghrFZrs+d56tSpJn3deBf773//W0yaNEl4e3sLb29v8cQTT4iioiK5bVpamtBoNCIzM1MMGzZM6HQ68X//93/NHqe5v5UQQnz11VdCq9WKpUuXNlu3ud/1kiVLmpSdOnVKCCHEp59+KqKiooSHh4cICgoSs2bNEpWVlXLfM2fOFLGxsWLdunWid+/eQqVSifr6elFeXi5mzpwpfH19hV6vF1FRUWL//v1yuxu/808//VSMGTNGeHp6ikGDBjmc760x3bjSbsmlS5eEl5eXOHLkiHj55ZfFyy+/3KROSUmJePzxx4W7u7sIDg4WFoulye+ysbFRLFmyRPTp00e4u7uLwYMHiz/+8Y8O/QAQqamp4sc//rHQ6/WiZ8+e4te//rXD77y532ljY6NITEwUPXv2FG5ubiIwMFBMmzbttufVlTHBdGFVVVVCrVaLN998847btCXBFBUViYEDB4rAwMB293nDwoULxUMPPSRyc3PFd999J7Kzs8WmTZuEENdfGMaMGSOmTp0qysrKRFlZmbh27ZoQovkEExAQIP785z+LoqIi8corrwgPDw8xYcIEkZaWJoqKisSCBQuEl5eXwwvhm2++KQ4cOCBOnTolMjIyxIABA8QLL7wghBDi2rVrwmKxCADy8W8knyVLlohevXqJXbt2iZKSEvGPf/xDhISEiOTk5GbP02azibKyMhEcHCySkpJEWVmZqK2tFfX19aJXr15i/PjxoqCgQBQUFIjo6GgRGhoqn2taWppQqVRixIgRIjMzU5w8ebLF22stJRghrr/pCA8Pb7bupUuXxO9//3uh0Wjkc62trRUffvihACAOHTokysrKhM1mE59//rnw9PQU69atEydOnBBffvmliI6OFmPHjhWSJAkhricYg8Eg4uPjRWFhoThy5Iioq6sTgwYNEk8//bTIz88XRUVFYsWKFcLNzU0cP37cYcw89NBD4uOPPxYnTpwQs2bNEgaDQVRXVwshhDh06JAAID788ENRVlbW6q1Gi8UiHn74YSHE9VtlBoPB4Q2WJEli6NCh4pFHHhF5eXni8OHDYuLEicLHx8fhdzlz5kwxZMgQsXfvXlFSUiLee+890a1bN/GnP/1JrgNA+Pv7i02bNoni4mJ5/GRkZAghrv+/2adPH/Hzn/9c/j3bbDaxevVq0bNnT7Fv3z7x3XffiS+//FKsXbv2tufVlTHBdGF5eXny/4B3qrUEo9Vqhbe3t3B3d5ffea1bt+6O+vTy8pLfnd/4OX/+vBBCiClTpjQ7x3JDS3MwzSWY1157Td6uqKgQAMSCBQvksurqagFA/O1vf2vxeLt27RJubm7CbrcLIZqfg7ly5Yrw9PQUH3/8sUP5li1bRLdu3VrsWwghevfu7ZD4//SnPwlPT09x8eJFuay8vFx4eHiILVu2CCGuJxgA4sCBA7ftW4jbJ5ikpCTh6enZYt0bV0o3a25cjBs3TiQlJTnU++677wQAcfjwYSHE9Rfjbt26OczZpKWliZ49eza5youJiZH/djeOd/PYLS8vFwDEJ598IoRo+xzM0KFDHcbqgAEDxDvvvCNvf/rppwKAw1VjVVWV8PT0lH8/JSUlQqVSiW+++cah72XLlomhQ4fK2wDEwoULHeoMHDhQvP766/J2c3Mwr776qoiJiZET9L2OczBdmFDgOaVPPfUUfv3rX+PKlStITU1FVVUV5s+ff0dt9+7d22TOJCAgAAAwb948PPPMMygoKEBsbCwmTJiAH/zgB1Cr276QcejQofK//fz8oNFo8NBDD8llPXr0gJubGyoqKuSyXbt24fe//z2Ki4vx/fffQ5IkNDY2ory8HEFBQc0e59ixY2hoaMAzzzwDlUoll9vtdly9ehUXL16En5/fHcV87NgxDB48GL6+vnJZQEAABgwYgGPHjjnUHTFixB312RIhhEO87ZWfn4/c3FxYLJYm+4qKiuT5jUGDBkGv1zu0Ky8vR/fu3R3aXLt2DZ6eng5lN8+RBAQEQKPR4MKFC22ONS8vD9988w2mT58ul82cORMbN26UV9AdP34cvr6+6N+/v1zHaDRiwIAB8nZBQQGEEDCbzQ7922w2h7nKW2MHgKCgoFZjnz17Nh5//HH0798fjz/+OB5//HE8+eSTcHNza9sJdxFMMF3Ygw8+CLVajePHj+Ppp5/ulD59fHzk/wE3bdqE0aNH4ze/+Q3eeOONVtv26dMHwcHBze77wQ9+gDNnzmDv3r3IysrCj3/8YwwZMgSff/55k/9xW9PcJPKtZSqVCpIkAbj+4vPss89i0aJFWLVqFXr06IHc3FzMnDkTjY2NLR7nRvsPPvgAYWFhTfbfOnneGTQaDTw8PDrUx7Fjx9CvX78OxyJJEpKSkjBjxowm+25+I+Ht7d2k3aBBg/DRRx81aXdj4cENzb2w3vi9t8XGjRvR2Ngov6EBridaSZIcJvtbS7w3jp2Tk9Mk1lvb3hr7zWOuJcOGDcOpU6fw2WefYd++fXjttdfwy1/+Erm5ufDx8bn9SXZB/BxMF2Y0GjFx4kRYLBZcvny5yX6r1YorV6506BjLli3DypUrcf78+Q71A1yP90c/+hE2btyIf/zjH9i/fz+OHz8O4Pr/rHa7vcPHaM4///lP+Pr6YsWKFXj00UcRFhaGc+fOOdS58WJxcwzh4eHw8PBASUkJ+vfv3+SnLYkxPDwcx48fR2VlpVx24cIFfPvtt4iIiOjgGf7HkSNHsHfvXjz77LMd7stsNuPYsWPNnvvNVyzNtSspKZHfrNz809LVYnOa+5s05/Lly9ixYwdSU1NRWFgo/3z11VcYO3YsNm7cCAAYPHgwLl68iJMnT8pta2pqcOLECXk7MjISAHDmzJkmsYeGht5x7Dfiby52vV6Pp556CuvWrUNBQQG++eYb7N+/v019dxVMMF3chg0boNPpEBkZie3bt+P48eMoLi7Gtm3bYDabUVRU1KH+Y2NjMXDgQCxfvrzVuhcvXkR5ebnDT319PQDgjTfewK5du/Dtt9+iqKgI6enp0Ov16NWrFwCgb9++OHjwIE6ePInKykpYrdYOxX2zAQMG4OLFi9i8eTNKSkrwl7/8BRs2bHCo07dvXwDAnj17cPHiRdTV1UGv12Px4sVYvHgxUlNT8e233+LYsWN47733kJSU1KYYpk+fDj8/P0ybNg2HDh3CwYMH8dxzz6Fnz56YNm1au86roaEB5eXlOH/+PA4fPoxVq1YhOjoajzzyCP7nf/6nXX3ebPny5fjrX/+Kn/3sZygsLMTJkyfxySefYO7cuWhoaGix3fPPP4++ffti8uTJ+PTTT3H69Gnk5eXhN7/5DXbv3n3Hx/f19YVer8enn36K8vJy1NTUNFtv27ZtUKvVmD17NiIiIhx+nn/+eaSnp+PKlSuIi4vD0KFDMWPGDOTn5+Orr77CjBkzoNVq5auT/v37Y86cOfjJT36CrVu3ori4GF999RXeffdd/Pa3v23T769v377Izs7GmTNnUFlZCUmSsGrVKqSnp+PYsWM4deoU3n33XWg0mmavkO8Jrp0Cos5QUVEhfv7zn4sHH3xQ/qDl2LFjxdatW5tMtLZ1mbIQQqSnpwutVuswOdpcn839rFq1SgghxPLly0V4eLjw9vYWPj4+YuzYseKLL76Q+zh58qQYM2aM8Pb2bnWZ8s3bQgih0WhEWlqaQ5m7u7vDBG9ycrLw9/cXXl5eYuLEiWL79u0Oy3GFEOK1114Tfn5+TZYpv/POO2Lo0KHC3d1ddO/eXTzyyCNiw4YNzf4ubrh1kl+I68uUJ06cKC+AmDx5crPLlO/ErR+K9fPza/GDlu2d5BdCiAMHDojY2Fih1+uFl5eXGDhwoHjttdfkcXVjmfKtKisrxcsvvyyCgoKETqcTQUFBIj4+Xhw6dOi2x7v1b7llyxbRp08fodFoWlymPHToUPHcc881u+/ixYtCq9XKY6GkpETExcU5LFMeMWKEwyIRm80mfvvb34oBAwYInU4nTCaTGDt2rHj//fflOs2Nw1sXquTn54vhw4cLDw8Peaz98Y9/FA8//LAwGAzC29tbmM1msXv37mZjvxeohOA3WhLR/am2thbBwcFYsWIFFi5c6Opw7jmc5Cei+8aePXug1WoxaNAgVFRUYNmyZVCpVJg6daqrQ7snMcEQ0X2jvr4ey5cvx+nTp+Ht7Y3IyEj885//dFh9Rp2Ht8iIiEgRXEVGRESKYIIhIiJFcA7mJqWlpe1q5+vr6/ABOqLOxPFFSuvIGLvdh2d5BUNERIpggiEiIkUwwRARkSKYYIiISBFMMEREpAgmGCIiUgQTDBERKYIJhoiIFMEPWhLdZaQ92yH+9p68feu3vKuefA7qKdNBdLdjgiG6y6inTAf+fwKxr1oMnU4HKWGZi6MiajveIiMiIkUwwRARkSKYYIiISBFMMEREpAgmGCIiUgQTDBERKYIJhoiIFOG0z8EUFhYiLS0NkiQhNjYW8fHxDvutVissFgtKSkpgMBiQkJAAf39/HDlyBOnp6bDZbNBqtZgxYwYiIiJw7do1rFmzBhcuXIBarUZkZCSef/55AEBWVha2bt0Ko9EIAJgwYQJiY2OddapERAQnJRhJkrB582YkJyfDZDJh0aJFMJvNCA4OlutkZmbC29sb69evR3Z2NtLT05GYmAiDwYCkpCQYjUacOXMGK1euxMaNGwEATz75JCIiImCz2bB8+XIcPnwYw4cPBwBERUVh7ty5zjg9IiJqhlNukRUXFyMwMBABAQHQarWIiopCfn6+Q52CggJER0cDAEaOHImjR49CCIG+ffvKVyIhISFobGyE1WqFu7s7IiIiAABarRZ9+/ZFVVWVM06HiIjugFOuYKqrq2EymeRtk8mEoqKiFutoNBp4eXmhtrYWPj4+cp28vDz069cPOp3Ooe2VK1dw8OBBTJo0yaHuN998gwceeAAzZ86Er69vk7gyMjKQkZEBAEhJSWm2zp3QarXtbkt0O9U6HVQqFccXKUqp17Au8yyys2fPIj09HW+88YZDud1ux9tvv42JEyciICAAABAZGYlRo0ZBp9Phs88+Q2pqKpYsWdKkz7i4OMTFxcnblZWV7YrN19e33W2JbsdutUKn03F8kaI68hoWFBTU4j6n3CIzGo0Ot6+qqqrk217N1bHb7aivr4fBYJDrv/XWW5g/fz4CAwMd2m3cuBGBgYGYPHmyXGYwGOSrnNjYWJSUlChyXkRE1DKnJJjQ0FCUlZWhoqICNpsNOTk5MJvNDnUiIyORlZUFAMjNzUV4eDhUKhWuXLmClJQUTJ8+HQMHDnRo895776G+vh6zZs1yKK+pqZH/XVBQ4LCYgIiInMMpt8g0Gg3mzJmDlStXQpIkxMTEICQkBDt27EBoaCjMZjPGjx8Pi8WChQsXQq/XIyEhAQDwySefoLy8HDt37sTOnTsBAMnJybDZbNi1axd69uyJpKQkAP9Zjvzxxx+joKAAGo0Ger0e8+bNc8ZpEhHRTVRCCOHqIO4WpaWl7WrHORhSCr8PhpyhS8/BEBHR/YcJhoiIFMEEQ0REimCCISIiRTDBEBGRIphgiIhIEUwwRESkCCYYIiJSBBMMEREpggmGiIgUwQRDRESKYIIhIiJFMMEQEZEimGCIiEgRTDBERKQIJhgiIlIEEwwRESmCCYaIiBShddaBCgsLkZaWBkmSEBsbi/j4eIf9VqsVFosFJSUlMBgMSEhIgL+/P44cOYL09HTYbDZotVrMmDEDERERAICSkhKkpqaisbERw4cPx+zZs6FSqVBXV4e1a9fi4sWL8PPzQ2JiIvR6vbNOlYiI4KQrGEmSsHnzZixevBhr165FdnY2zp0751AnMzMT3t7eWL9+PSZPnoz09HQAgMFgQFJSElavXo358+dj/fr1cpt33nkHL730EtatW4fy8nIUFhYCAHbv3o0hQ4Zg3bp1GDJkCHbv3u2M0yQiops4JcEUFxcjMDAQAQEB0Gq1iIqKQn5+vkOdgoICREdHAwBGjhyJo0ePQgiBvn37wmg0AgBCQkLQ2NgIq9WKmpoaNDQ0ICwsDCqVCmPHjpX7zM/Px7hx4wAA48aNa3IsIiJSnlNukVVXV8NkMsnbJpMJRUVFLdbRaDTw8vJCbW0tfHx85Dp5eXno168fdDpds31WV1cDAC5fvowePXoAALp3747Lly83G1dGRgYyMjIAACkpKfD19W3X+Wm12na3Jbqdap0OKpWK44sUpdRrmNPmYDrq7NmzSE9PxxtvvNGmdiqVCiqVqtl9cXFxiIuLk7crKyvbFZuvr2+72xLdjt1qhU6n4/giRXXkNSwoKKjFfU5JMEajEVVVVfJ2VVWVfNvr1jomkwl2ux319fUwGAxy/bfeegvz589HYGBgq31269YNNTU16NGjB2pqahyugjqDtGc7xN/ek7cv3LJf9eRzUE+Z3qnHJCLqLM56DXNKggkNDUVZWRkqKipgNBqRk5ODV1991aFOZGQksrKyEBYWhtzcXISHh0OlUuHKlStISUnB9OnTMXDgQLl+jx494OnpiRMnTuDBBx/EgQMHMGHCBACA2WzG/v37ER8fj/3792PEiBGdej7qKdOB///Lt69aDJ1OBylhWaceg4hIKc56DXNKgtFoNJgzZw5WrlwJSZIQExODkJAQ7NixA6GhoTCbzRg/fjwsFgsWLlwIvV6PhIQEAMAnn3yC8vJy7Ny5Ezt37gQAJCcno1u3bnjxxRexYcMGNDY2YtiwYRg+fDgAID4+HmvXrkVmZqa8TJmIiJxLJYQQrg7iblFaWtrmNryCISVxfJHSOjrGbjcHw0/yExGRIphgiIhIEUwwRESkCCYYIiJSBBMMEREpggmGiIgUwQRDRESKYIIhIiJFMMEQEZEimGCIiEgRTDBERKQIJhgiIlIEEwwRESmCCYaIiBTBBENERIpggiEiIkUwwRARkSKYYIiISBFaZx2osLAQaWlpkCQJsbGxiI+Pd9hvtVphsVhQUlICg8GAhIQE+Pv7o7a2FmvWrEFxcTGio6Mxd+5cAEBDQwN+9atfye2rq6sxZswYzJo1C1lZWdi6dSuMRiMAYMKECYiNjXXWqRIREZyUYCRJwubNm5GcnAyTyYRFixbBbDYjODhYrpOZmQlvb2+sX78e2dnZSE9PR2JiInQ6HaZNm4YzZ87g7Nmzcn1PT0+sWrVK3k5KSsIjjzwib0dFRcnJiIiInM8pt8iKi4sRGBiIgIAAaLVaREVFIT8/36FOQUEBoqOjAQAjR47E0aNHIYSAh4cHBg4cCDc3txb7Ly0txffff49BgwYpeRpERNQGTrmCqa6uhslkkrdNJhOKioparKPRaODl5YXa2lr4+Pi02n9OTg4ee+wxqFQquSwvLw/ffPMNHnjgAcycORO+vr5N2mVkZCAjIwMAkJKS0mydVs9Np4NKpWpXW6LWcHyR0pQcY06bg1FSdnY2Fi5cKG9HRkZi1KhR0Ol0+Oyzz5CamoolS5Y0aRcXF4e4uDh5u7Kyss3Htlut0Ol07WpL1BqOL1JaR8dYUFBQi/uccovMaDSiqqpK3q6qqpIn4JurY7fbUV9fD4PB0Grfp0+fhiRJ6Nevn1xmMBig0+kAALGxsSgpKemM0yAiojZwSoIJDQ1FWVkZKioqYLPZkJOTA7PZ7FAnMjISWVlZAIDc3FyEh4c73PJqSXZ2NkaNGuVQVlNTI/+7oKDAYTEBERE5h1NukWk0GsyZMwcrV66EJEmIiYlBSEgIduzYgdDQUJjNZowfPx4WiwULFy6EXq9HQkKC3H7+/Pmor6+HzWZDfn4+kpOT5aTxr3/9C4sWLXI43scff4yCggJoNBro9XrMmzfPGadJREQ3UQkhhKuDuFuUlpa2uY191WLodDpICcsUiIjudxxfpLSOjjGXz8EQEdH9hwmGiIgUwQRDRESKYIIhIiJFMMEQEZEimGCIiEgRTDBERKQIJhgiIlIEEwwRESmCCYaIiBTBBENERIpggiEiIkUwwRARkSKYYIiISBFMMEREpAgmGCIiUgQTDBERKYIJhoiIFKHtSGObzYaMjAxMmDCh1bqFhYVIS0uDJEmIjY1FfHy8w36r1QqLxYKSkhIYDAYkJCTA398ftbW1WLNmDYqLixEdHY25c+fKbZYuXYqamhq4ubkBAJKTk9GtW7cW+yIiIue5owTz9ddf4/Tp0wgMDMSIESNgt9uxd+9e/PWvf4Ver281wUiShM2bNyM5ORkmkwmLFi2C2WxGcHCwXCczMxPe3t5Yv349srOzkZ6ejsTEROh0OkybNg1nzpzB2bNnm/T96quvIjQ01KGspb6IiMh5Wk0wu3fvxocffoiQkBCcPXsWP/jBD3Ds2DHodDq89NJLePjhh1s9SHFxMQIDAxEQEAAAiIqKQn5+vkOCKSgowLPPPgsAGDlyJN59910IIeDh4YGBAweivLz8jk+qpb5UKtUd90FERB3TaoLJyMjAsmXL0K9fP5w4cQK//OUv8cILL2Dy5Ml3fJDq6mqYTCZ522QyoaioqMU6Go0GXl5eqK2thY+Pz2373rBhA9RqNR599FE888wzUKlUd9xXRkYGMjIyAAApKSnw9fW943OS49bpoFKp2tWWqDUcX6Q0JcdYqwmmtrYW/fr1AwCEhYVBp9Nh0qRJnR5Ie7z66qswGo1oaGjA6tWrceDAAYwbN+6O28fFxSEuLk7erqysbHMMdqsVOp2uXW2JWsPxRUrr6BgLCgpqcd8drSITQkCSJEiSBJ1OBwDytiRJrbY3Go2oqqqSt6uqqmA0GlusY7fbUV9fD4PB0Gq/AODp6YnRo0ejuLi43X0REVHnavUK5urVq3juueccym7d3rFjx237CA0NRVlZGSoqKmA0GpGTk4NXX33VoU5kZCSysrIQFhaG3NxchIeH33bOxG6348qVK/Dx8YHNZsPBgwcxZMiQdvVFRESdr9UEY7FYOnwQjUaDOXPmYOXKlZAkCTExMQgJCcGOHTsQGhoKs9mM8ePHw2KxYOHChdDr9UhISJDbz58/H/X19bDZbMjPz0dycjJ8fX2xcuVK2O12SJKEIUOGyLe7btcXERE5h0oIIW5X4Te/+Q0WLVrkrHhcqrS0tM1t7KsWQ6fTQUpYpkBEdL/j+CKldXSMdWgO5t///ne7DkpERPc3PiqGiIgU0eocTGNjY6vzMAsWLOi0gIiI6N7QaoJRqVTyJ/CJiIjuVKsJRqfTyY9dISLnkXKzgJJvYbVZgaS5UD01A+qR0a4Oi+iOtToH08oiMyJSgJSbBbE1FbBZrxdUX4TYmno96RB1Ea0mmAcffBDbtm1rdl96ejpOnDjR6UER3e/ER1uBxmuOhY3XrpcTdRGtJhitVovBgwc3uy88PBy7du3q9KCI7nvVLTwXqqVyortQqwnm9OnTGDZsWLP7hgwZglOnTnV6UET3PWMLT7ZtqZzoLtRqgmloaIDNZmt2n91uR0NDQ6cHRXS/Uz01A3Bzdyx0c79eTtRFtJpgevbsia+++qrZfV999RV69uzZ6UER3e/UI6OhmjEf0F5/ejmMflDNmM9VZNSltLpMefLkydi0aRMkScKIESOgVqshSRLy8/OxefNmvPDCC86Ik+i+ox4ZDfsXn/JZZNRltZpgRo8ejUuXLiE1NRVWqxU+Pj74/vvvodPpMHXqVIwePdoZcRIRURfTaoIBgCeeeALjx4/HiRMnUFdXB71ej7CwMHh5eSkdHxERdVF3lGAAwMvLq8XVZERERLfi00KfXJ4AABidSURBVJSJiEgRTDBERKQIJhgiIlLEHc/BdFRhYSHS0tIgSRJiY2MRHx/vsN9qtcJisaCkpAQGgwEJCQnw9/dHbW0t1qxZg+LiYkRHR2Pu3LkAgGvXrmHNmjW4cOEC1Go1IiMj8fzzzwMAsrKysHXrVhiNRgDAhAkTEBsb66xTJSIiOCnBSJKEzZs3Izk5GSaTCYsWLYLZbEZwcLBcJzMzE97e3li/fj2ys7ORnp6OxMRE6HQ6TJs2DWfOnMHZs2cd+n3yyScREREBm82G5cuX4/Dhwxg+fDgAICoqSk5GRETkfE65RVZcXIzAwEAEBARAq9UiKioK+fn5DnUKCgoQHR0NABg5ciSOHj0KIQQ8PDwwcOBAuLm5OdR3d3dHREQEgOsP5Ozbty+qqqqccTpERHQHnHIFU11dDZPJJG+bTCYUFRW1WEej0cDLywu1tbXw8fFptf8rV67g4MGDmDRpklyWl5eHb775Bg888ABmzpwJX9+mDwnMyMhARkYGACAlJaXZOq2em04HlUrVrrZEreH4IqUpOcacNgejFLvdjrfffhsTJ06Uv9o5MjISo0aNgk6nw2effYbU1FQsWbKkSdu4uDjExcXJ25WVbX8Uut1qhU6na1dbotZwfJHSOjrGgoKCWtznlFtkRqPR4fZVVVWVPAHfXB273Y76+noYDIZW+964cSMCAwMxefJkucxgMECnu/6QwNjYWJSUlHTGaRARURs4JcGEhoairKwMFRUVsNlsyMnJgdlsdqgTGRmJrKwsAEBubi7Cw8OhUqlu2+97772H+vp6zJo1y6G8pqZG/ndBQYHDYgIiInIOp9wi02g0mDNnDlauXAlJkhATE4OQkBDs2LEDoaGhMJvNGD9+PCwWCxYuXAi9Xo+EhAS5/fz581FfXw+bzYb8/HwkJyfD09MTu3btQs+ePZGUlATgP8uRP/74YxQUFECj0UCv12PevHnOOE0iIrqJSgghXB3E3aK0tLTNbeyrFvNx6qQYji9SWkfHmMvnYIiI6P7DBENERIpggiEiIkUwwRARkSKYYIiISBFMMEREpAgmGCIiUgQTTAdIuVlAybewHjsMe9Lc69tERF2E0q9hTDDtJOVmQWxNBWzW6wXVFyG2pjLJEFGX4IzXMCaYdhIfbQUarzkWNl67Xk5EdJdzxmsYE0x7VbfwaOuWyomI7iZOeA1jgmkvYwtfztNSORHR3cQJr2FMMO2kemoG4ObuWOjmfr2ciOgu54zXsC7/jZauoh4ZDQmA2LL++iSZ0Q+qp2ZAPTLa1aEREbXKGa9hTDAdoB4ZDfsXn/Jx6kTUJSn9GsZbZEREpAgmGCIiUoTTbpEVFhYiLS0NkiQhNjYW8fHxDvutVissFgtKSkpgMBiQkJAAf39/1NbWYs2aNSguLkZ0dDTmzp0rtykpKUFqaioaGxsxfPhwzJ49GyqVCnV1dVi7di0uXrwIPz8/JCYmQq/XO+tUiYgITrqCkSQJmzdvxuLFi7F27VpkZ2fj3LlzDnUyMzPh7e2N9evXY/LkyUhPTwcA6HQ6TJs2DTNmNF3Z8M477+Cll17CunXrUF5ejsLCQgDA7t27MWTIEKxbtw5DhgzB7t27lT9JIiJy4JQEU1xcjMDAQAQEBECr1SIqKgr5+fkOdQoKChAdHQ0AGDlyJI4ePQohBDw8PDBw4EC4ubk51K+pqUFDQwPCwsKgUqkwduxYuc/8/HyMGzcOADBu3LgmxyIiIuU5JcFUV1fDZDLJ2yaTCdXV1S3W0Wg08PLyQm1tbbv6vHz5Mnr06AEA6N69Oy5fvtxp50JERHfmnl+mrFKpoFKpmt2XkZGBjIwMAEBKSgp8fdv+CdZqnQ4qlapdbYlaw/FFSlNyjDklwRiNRlRVVcnbVVVVMBqNzdYxmUyw2+2or6+HwWBoV5/dunVDTU0NevTogZqaGvj4+DTbR1xcHOLi4uTtysq2P4PHbrVCp9O1qy1Razi+SGkdHWNBQUEt7nPKLbLQ0FCUlZWhoqICNpsNOTk5MJvNDnUiIyORlZUFAMjNzUV4eHiLVx4A0KNHD3h6euLEiRMQQuDAgQNyn2azGfv37wcA7N+/HyNGjFDmxIiIqEVOuYLRaDSYM2cOVq5cCUmSEBMTg5CQEOzYsQOhoaEwm80YP348LBYLFi5cCL1ej4SEBLn9/PnzUV9fD5vNhvz8fCQnJyM4OBgvvvgiNmzYgMbGRgwbNgzDhw8HAMTHx2Pt2rXIzMyUlykTEZFzqYQQwtVB3C1KS0vb3Ma+ajEfFUOK4fgipXV0jLn8FhkREd1/mGCIiEgRTDBERKQIJhgiIlIEEwwRESmCCYaIiBTBBENERIpggiEiIkUwwRARkSKYYIiISBFMMEREpAgmGCIiUgQTDBERKYIJhoiIFMEEQ0REimCCISIiRTDBEBGRIphgiIhIEVpnHaiwsBBpaWmQJAmxsbGIj4932G+1WmGxWFBSUgKDwYCEhAT4+/sDAD766CNkZmZCrVZj9uzZGDZsGEpLS7F27Vq5fUVFBaZOnYrJkyfj/fffx+effw4fHx8AwI9+9CM8/PDDzjpVIiKCkxKMJEnYvHkzkpOTYTKZsGjRIpjNZgQHB8t1MjMz4e3tjfXr1yM7Oxvp6elITEzEuXPnkJOTgzVr1qCmpgZvvvkm3n77bQQFBWHVqlVy/y+99BIeeeQRub/JkydjypQpzjg9IiJqhlNukRUXFyMwMBABAQHQarWIiopCfn6+Q52CggJER0cDAEaOHImjR49CCIH8/HxERUVBp9PB398fgYGBKC4udmj79ddfIzAwEH5+fs44HSIiugNOuYKprq6GyWSSt00mE4qKilqso9Fo4OXlhdraWlRXV+PBBx+U6xmNRlRXVzu0zc7OxqhRoxzK9u7diwMHDqBfv3544YUXoNfrm8SVkZGBjIwMAEBKSgp8fX3bfm46HVQqVbvaErWG44uUpuQYc9ocjFJsNhsOHjyI6dOny2X/9V//hR/+8IcAgB07duAvf/kL5s2b16RtXFwc4uLi5O3Kyso2H99utUKn07WrLVFrOL5IaR0dY0FBQS3uc8otMqPRiKqqKnm7qqoKRqOxxTp2ux319fUwGAxN2lZXVzu0PXz4MPr27Yvu3bvLZd27d4darYZarUZsbCxOnjyp1KkREVELnJJgQkNDUVZWhoqKCthsNuTk5MBsNjvUiYyMRFZWFgAgNzcX4eHhUKlUMJvNyMnJgdVqRUVFBcrKytC/f3+5XXO3x2pqauR/f/nllwgJCVHu5IiIqFlOuUWm0WgwZ84crFy5EpIkISYmBiEhIdixYwdCQ0NhNpsxfvx4WCwWLFy4EHq9HgkJCQCAkJAQPPbYY/jZz34GtVqNuXPnQq2+nhevXr2KI0eO4Kc//anD8bZt24bTp09DpVLBz8+vyX6iu5m0ZzvE396Tt60A8JP/rIhUPfkc1FOmN21IdJdRCSGEq4O4W5SWlra5jX3VYuh0OkgJyxSIiAjw9fXlHAwppqOvYS6fgyEiovsPEwwRESmCCYaIiBTBBENERIpggiEiIkUwwRARkSKYYIiISBFMMEREpAgmGCIiUgQTDBERKYIJhoiIFMEEQ0REimCCISIiRTDBEBGRIphgiIhIEUwwRESkCCYYIiJSBBMMEREpQuusAxUWFiItLQ2SJCE2Nhbx8fEO+61WKywWC0pKSmAwGJCQkAB/f38AwEcffYTMzEyo1WrMnj0bw4YNAwDMnz8fHh4eUKvV0Gg0SElJAQDU1dVh7dq1uHjxIvz8/JCYmAi9Xu+sUyUiIjgpwUiShM2bNyM5ORkmkwmLFi2C2WxGcHCwXCczMxPe3t5Yv349srOzkZ6ejsTERJw7dw45OTlYs2YNampq8Oabb+Ltt9+GWn394mvJkiXw8fFxON7u3bsxZMgQxMfHY/fu3di9ezd+/OMfd9757NkO8bf35G0rAPxkirytevI5qKdM77TjERF1Jme9hjklwRQXFyMwMBABAQEAgKioKOTn5zskmIKCAjz77LMAgJEjR+Ldd9+FEAL5+fmIioqCTqeDv78/AgMDUVxcjLCwsBaPl5+fj6VLlwIAxo0bh6VLl3ZqglFPmQ7c9Mv39fVFZWVlp/VPRKQkZ72GOSXBVFdXw2QyydsmkwlFRUUt1tFoNPDy8kJtbS2qq6vx4IMPyvWMRiOqq6vl7ZUrVwIAHn/8ccTFxQEALl++jB49egAAunfvjsuXLzcbV0ZGBjIyMgAAKSkp8PX1bdf5abXadrclag3HFylNqTHmtDkYJbz55pswGo24fPkyVqxYgaCgIAwePNihjkqlgkqlarZ9XFycnJQAtDuD8wqGlMTxRUrryBgLCgpqcZ9TVpEZjUZUVVXJ21VVVTAajS3WsdvtqK+vh8FgaNK2urpabnvjv926dcOIESNQXFwsb9fU1AAAampqmszREBGR8pySYEJDQ1FWVoaKigrYbDbk5OTAbDY71ImMjERWVhYAIDc3F+Hh4VCpVDCbzcjJyYHVakVFRQXKysrQv39/XL16FQ0NDQCAq1ev4siRI+jVqxcAwGw2Y//+/QCA/fv3Y8SIEc44TSIiuolKCCGccaBDhw5hy5YtkCQJMTExePrpp7Fjxw6EhobCbDajsbERFosFp06dgl6vR0JCgrwoYNeuXdi3bx/UajVmzZqF4cOH48KFC3jrrbcAXL/iGT16NJ5++mkAQG1tLdauXYvKyso2LVMuLS1t17nxFgYpieOLlKbULTKnJZiugAmG7kYcX6S0Lj0HQ0RE9x8mGCIiUgRvkRERkSJ4BdMJXn/9dVeHQPcwji9SmlJjjAmGiIgUwQRDRESKYILpBDc/boaos3F8kdKUGmOc5CciIkXwCoaIiBTBBENERIpggiEiIkV06e+DcRVJknDo0CFUVFRAkiS5/IknnnBhVEREdxdewbTDb3/7W2RlZaGurg4NDQ3yD1Fn+ctf/oKzZ8+6Ogy6T0iShC+++KLT++UVTDtUVVXJXxVApISePXti06ZNsNvtiI6OxujRo+Hl5eXqsKiLq6+vx969e1FdXQ2z2YyHHnoIn3zyCf7+97+jd+/eGDNmTKcej8uU22Hbtm0YMmQIhg4d6upQ6B5XWlqKffv2ITs7GwMGDEBsbCwiIiJcHRZ1Ub/73e/g7e2NsLAwfP311/j+++8hhMDs2bPRp0+fTj8er2DaISwsDG+99RYkSYJWq4UQAiqVClu2bHF1aHQPkSQJ58+fx/nz52EwGNC7d2/8/e9/R0ZGBhISElwdHnVBFy5cwOrVqwEAsbGx+OlPf4oNGzbAzc1NkeMxwbTDli1bsGLFCvTq1QsqlcrV4dA96M9//jMOHTqEiIgIPP300+jfv7+877XXXnNhZNSVabX/eclXq9UwmUyKJReAt8jaZcmSJViyZAnUaq6RIGXs27cPjz32GDw8PJrsq6+v53wMtcu0adPkMSWEQGNjI9zd3RW7C8MrmHbw9/fHsmXLMGzYMOh0Ormcy5Sps3zxxReIiYlxKFu+fDl+9atfMblQu+3YscOpx+Nb8Hbw9/dHREQEbDYblylTp2psbERdXR1qa2tRV1cn/1RUVKC6utrV4dE97JVXXun0PnkF0w7PPvssAODq1asA0OxtDKL2yMjIwD/+8Q/U1NQgKSlJLvfy8sKECRNcGBlR2zHBtMOZM2dgsVhQV1cHADAYDFiwYAFCQkJcHBl1dZMmTcKkSZPw8ccfY+LEia4Oh6hDmGDaYdOmTXjhhRfkzyMcO3YMGzduxIoVK1wcGXV1R48eRUREBIxGI/Ly8prsf/TRR10QFd0r/v73vzdbLoSQ78h0JiaYdrh27ZrDh93Cw8Nx7do1F0ZE94rjx48jIiICBw8ebHY/Ewx1xO3miidNmtTpx2OCaQd/f3/s3LkTY8eOBXB9xY+/v7+Lo6J7wdSpUyFJEoYNG4aoqChXh0P3mBvzx87CVWTt8Morr+D777/H6tWrsXr1anz//feKrMCg+5NarcaePXtcHQbdJ25eTNLZeAXTDnq9HnPmzHF1GHQPGzJkCPbs2YOoqCiHVYp6vd6FUdG9SMnP2jPBtEFKSsptHw2j5DsBur/k5OQAAPbu3SuXqVQqWCwWV4VE96iHH35Ysb75qJg2OH78OAAgLy8Ply5dkh9tnZ2djW7dumHWrFkujI6I6M5cuXIFZWVlAICgoCDFng7BK5g2GDx4MIDrXwaVkpIil5vNZrz++uuuCovuUWfOnMG5c+dgtVrlsnHjxrkwIurqrFYrNm3ahPz8fPj7+0MIgcrKSowYMQI//elPHR6G2RmYYNrh2rVruHDhAgICAgAAFRUVXKZMneqDDz7A8ePHce7cOQwfPhyHDx/GwIEDmWCoQ3bt2gW73Y4//OEP8PT0BHB96fLmzZuxc+dOPPfcc516PCaYdpg5cyaWLl2KgIAA+R3AT37yE1eHRfeQ3NxcrFq1CklJSZg3bx4uXbqE9evXuzos6uK+/PJL/PrXv4a7u7tc5unpiRdffBFvvPEGE8zdYNiwYVi3bh3Onz8P4PrX2978VGWijnJzc4NarYZarUZ9fT26deuGqqoqV4dFXZxKpXJILjco9TxFJph2KikpwcWLF2G32/Hdd98B4P1x6jyhoaG4cuUKYmNj8frrr8PDwwNhYWGuDou6OJVKJT9D8VZKfL8VV5G1w/r163HhwgX06dPH4Y/Cz8aQEioqKtDQ0IDevXu7OhTq4ubPnw+VStXsZ1+UWAbPBNMOiYmJWLNmDb8umTpdSUnJbff369fPSZEQdRxvkbVDSEgILl26hB49erg6FLrHbN269bb7lyxZ4qRI6F6UmJiIMWPGYNSoUfIqWCXxCqYdli1bhtOnT6N///4O68b5SX4iupudPn0aOTk5+Ne//gW9Xo9Ro0YhKioKRqNRkeMxwbTDjU/03+rGBzGJOmr//v3NlnMhCXWWEydOICcnB3l5eQgMDMSoUaMQFxfXqcdggiG6C7377rvyvxsbG3H06FH07dsXP//5z10YFd2Ljh07hi1btuDcuXPYvn17p/bNOZh2OHHiBNLS0nDu3DnYbDZIkgQPDw9s2bLF1aHRPeLWFYlXrlzB73//exdFQ/ea4uJiZGdnIy8vD/7+/oiLi8Njjz3W6cdhgmmHd999FwkJCVizZg1SUlKwf/9++cFxREpwd3dHRUWFq8OgLm779u3y/EtUVBTefPNNmEwmxY7HBNNOgYGBkCQJarUaMTEx+N///V9Mnz7d1WHRPeLmr4aQJAnnz59X5B0m3V/c3Nwwb948DBo0CMD1ub68vDz4+vpi6tSpnf59Q0ww7eDu7g6bzYY+ffpg27Zt6N69u6Jf2kP3nylTpsj/VqvV8PPzU/SdJt0f8vPzMWHCBADXFytt374ds2fPxunTp7Fx48ZOn+PjVya3w4IFCyBJEubMmQN3d3dUVVVx8pU61eDBgxEUFIT6+nrU1dVBo9G4OiS6B0iSJF+l5OTkIDY2FiNHjsRzzz2H8vLyTj8eE0w7+Pn5wc3NDV5eXnj22WfxzDPPyN9ASNQZPv/8cyxatAh5eXnIzc3FG2+8gczMTFeHRV2cJEmw2+0AgKNHjyIiIsJhX2fjLbI2qKysxIcffoiamhqMGDECo0aNwvvvv48DBw5g1KhRrg6P7iF79uzB7373OxgMBgBAbW0tkpOTMX78eBdHRl3ZqFGjsHTpUhgMBri5uclzMeXl5Yp8qyUTTBukpqZi0KBBePTRR1FYWIhFixahd+/eeOutt9C9e3dXh0f3EIPBIH8hFHD9OztuJBui9nr66acRERGBS5cu4aGHHnJYSDJ79uxOPx4/aNkGv/jFL7Bq1Sp5++WXX8aGDRsUecw13d8sFgvOnDkDs9kMlUqFgoIC9OrVS36i8hNPPOHiCIlaxyuYNrr5uxQMBgPq6+vl7c5e4kf3r4CAAIeHEZrNZgDXv96WqKvgFUwbOPu7FIiuXr0KQLlvHCRSEhMM0V3ozJkzsFgs8hWzwWDAggULEBIS4uLIiO4cb5ER3YU2bdqEF154QV5GeuzYMWzcuBErVqxwcWREd46z00R3oWvXrjl8RiE8PBzXrl1zYUREbccrGKK7kL+/P3bu3ImxY8cCAL744gv4+/u7OCqituEcDNFdqK6uDu+//z6+/fZbAMCgQYPwwx/+kCsVqUthgiG6izQ2NuKzzz5DeXk5evXqhZiYGIev5SbqSjgHQ3QXSU1NxcmTJ9GrVy8cPnwY27Ztc3VIRO3Gt0ZEd5Fz585h9erVAIDx48dj8eLFLo6IqP14BUN0F7n5dhgf0U9dHedgiO4i06ZNkz+1L4RAY2Mj3N3dIYSASqXCli1bXBwh0Z1jgiEiIkXwFhkRESmCCYaIiBTBBENERIpggiEiIkX8P1YaHzEKprMDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plot_verify_agents(result_AB)\n",
    "plt.ylabel('CTR')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On average, we see a 14% improvement in click-through rate when comparing the popularity baseline with our new SVD-LR model.\n",
    "This difference might seem small, but you can imagine that a 14% increase in a million-dollar business has quite a big impact.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: numba in ./venv/lib/python3.7/site-packages (0.49.0)\n",
      "Requirement already satisfied: llvmlite<=0.33.0.dev0,>=0.31.0.dev0 in ./venv/lib/python3.7/site-packages (from numba) (0.32.0)\n",
      "Requirement already satisfied: setuptools in ./venv/lib/python3.7/site-packages (from numba) (46.1.3)\n",
      "Requirement already satisfied: numpy>=1.15 in ./venv/lib/python3.7/site-packages (from numba) (1.18.4)\n"
     ]
    }
   ],
   "source": [
    "!pip install numba\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}