eval-report.ml10m.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ce485e34",
   "metadata": {
    "papermill": {
     "duration": 0.004895,
     "end_time": "2024-12-08T00:44:51.816723",
     "exception": false,
     "start_time": "2024-12-08T00:44:51.811828",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Evaluation Analysis for Recommender Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17c9a5e6",
   "metadata": {
    "papermill": {
     "duration": 0.004453,
     "end_time": "2024-12-08T00:44:51.830312",
     "exception": false,
     "start_time": "2024-12-08T00:44:51.825859",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this section we will analyze and compare the generated recommendations and predictions from a predefined list of algorithms with the goal of assessing the performance of each algorithm with respect to a metric. In other words, we would rank the algorithms for each metric considered with respect to performance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f14bb433",
   "metadata": {
    "papermill": {
     "duration": 0.004494,
     "end_time": "2024-12-08T00:44:51.839173",
     "exception": false,
     "start_time": "2024-12-08T00:44:51.834679",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b8308de",
   "metadata": {
    "papermill": {
     "duration": 0.004141,
     "end_time": "2024-12-08T00:44:51.847518",
     "exception": false,
     "start_time": "2024-12-08T00:44:51.843377",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Below are the list of packages required to successfully run the analysis. They are divided into partitions to signify their specific task.<br>\n",
    "We need the pathlib package for working with files and folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "77d2e6e9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:44:51.857181Z",
     "iopub.status.busy": "2024-12-08T00:44:51.856960Z",
     "iopub.status.idle": "2024-12-08T00:44:51.861809Z",
     "shell.execute_reply": "2024-12-08T00:44:51.861435Z"
    },
    "papermill": {
     "duration": 0.010912,
     "end_time": "2024-12-08T00:44:51.862573",
     "exception": false,
     "start_time": "2024-12-08T00:44:51.851661",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8607dbba",
   "metadata": {
    "papermill": {
     "duration": 0.004389,
     "end_time": "2024-12-08T00:44:51.871221",
     "exception": false,
     "start_time": "2024-12-08T00:44:51.866832",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We would use the pandas for analyzing and manipulating our data while seaborn and matplotlib are used for data visualization. statsmodels.graphics.gofplots and scipy.stats.shapiro are used for normality check. Scipy.stats.friedmanchisquare is a non-parametric test used to determine the statistical significance in metric results and the wilcoxon test is used for pairwise comparison of sample data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "54c9b1a0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:44:51.880772Z",
     "iopub.status.busy": "2024-12-08T00:44:51.880571Z",
     "iopub.status.idle": "2024-12-08T00:44:59.295291Z",
     "shell.execute_reply": "2024-12-08T00:44:59.294745Z"
    },
    "papermill": {
     "duration": 7.420896,
     "end_time": "2024-12-08T00:44:59.296358",
     "exception": false,
     "start_time": "2024-12-08T00:44:51.875462",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from statsmodels.graphics.gofplots import qqplot\n",
    "from scipy.stats import shapiro\n",
    "from scipy.stats import friedmanchisquare, wilcoxon\n",
    "from itertools import combinations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09d4299f",
   "metadata": {
    "papermill": {
     "duration": 0.004625,
     "end_time": "2024-12-08T00:44:59.306339",
     "exception": false,
     "start_time": "2024-12-08T00:44:59.301714",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Logging to show what's happening in LensKit routines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fd84093f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:44:59.316164Z",
     "iopub.status.busy": "2024-12-08T00:44:59.315835Z",
     "iopub.status.idle": "2024-12-08T00:44:59.335724Z",
     "shell.execute_reply": "2024-12-08T00:44:59.335322Z"
    },
    "papermill": {
     "duration": 0.025512,
     "end_time": "2024-12-08T00:44:59.336404",
     "exception": false,
     "start_time": "2024-12-08T00:44:59.310892",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.util.log notebook logging configured\n"
     ]
    }
   ],
   "source": [
    "from lenskit import util\n",
    "util.log_to_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c7c3c1f0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:44:59.346554Z",
     "iopub.status.busy": "2024-12-08T00:44:59.346331Z",
     "iopub.status.idle": "2024-12-08T00:44:59.348698Z",
     "shell.execute_reply": "2024-12-08T00:44:59.348297Z"
    },
    "papermill": {
     "duration": 0.008243,
     "end_time": "2024-12-08T00:44:59.349345",
     "exception": false,
     "start_time": "2024-12-08T00:44:59.341102",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "_log = logging.getLogger('eval-report')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af5b4ebc",
   "metadata": {
    "papermill": {
     "duration": 0.004614,
     "end_time": "2024-12-08T00:44:59.358292",
     "exception": false,
     "start_time": "2024-12-08T00:44:59.353678",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We will use lenskit for training, running, and evaluating recommender algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "803d7eb5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:44:59.368619Z",
     "iopub.status.busy": "2024-12-08T00:44:59.368411Z",
     "iopub.status.idle": "2024-12-08T00:44:59.455174Z",
     "shell.execute_reply": "2024-12-08T00:44:59.454721Z"
    },
    "papermill": {
     "duration": 0.093408,
     "end_time": "2024-12-08T00:44:59.456151",
     "exception": false,
     "start_time": "2024-12-08T00:44:59.362743",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from lenskit import topn\n",
    "from lenskit.metrics.predict import rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62cde0fa",
   "metadata": {
    "papermill": {
     "duration": 0.004978,
     "end_time": "2024-12-08T00:44:59.466541",
     "exception": false,
     "start_time": "2024-12-08T00:44:59.461563",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b8bdd0f",
   "metadata": {
    "papermill": {
     "duration": 0.004365,
     "end_time": "2024-12-08T00:44:59.475556",
     "exception": false,
     "start_time": "2024-12-08T00:44:59.471191",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We specify the dataset we will use for our analysis and the main directory from where we read the recommendation and prediction files. From the main directory we find all the directories associated with the dataset and then read the recommendation and predictions files from those directories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "66d37091",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:44:59.485185Z",
     "iopub.status.busy": "2024-12-08T00:44:59.484955Z",
     "iopub.status.idle": "2024-12-08T00:44:59.487328Z",
     "shell.execute_reply": "2024-12-08T00:44:59.486928Z"
    },
    "papermill": {
     "duration": 0.007993,
     "end_time": "2024-12-08T00:44:59.487991",
     "exception": false,
     "start_time": "2024-12-08T00:44:59.479998",
     "status": "completed"
    },
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "dataset = \"ml100k\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9c987eb1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:44:59.497770Z",
     "iopub.status.busy": "2024-12-08T00:44:59.497421Z",
     "iopub.status.idle": "2024-12-08T00:44:59.499665Z",
     "shell.execute_reply": "2024-12-08T00:44:59.499286Z"
    },
    "papermill": {
     "duration": 0.008098,
     "end_time": "2024-12-08T00:44:59.500395",
     "exception": false,
     "start_time": "2024-12-08T00:44:59.492297",
     "status": "completed"
    },
    "tags": [
     "injected-parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "dataset = \"ml10m\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b7f9f66b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:44:59.510014Z",
     "iopub.status.busy": "2024-12-08T00:44:59.509799Z",
     "iopub.status.idle": "2024-12-08T00:44:59.512042Z",
     "shell.execute_reply": "2024-12-08T00:44:59.511651Z"
    },
    "papermill": {
     "duration": 0.007897,
     "end_time": "2024-12-08T00:44:59.512692",
     "exception": false,
     "start_time": "2024-12-08T00:44:59.504795",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "output_root = Path(\"runs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8688a4d0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:44:59.524289Z",
     "iopub.status.busy": "2024-12-08T00:44:59.524013Z",
     "iopub.status.idle": "2024-12-08T00:44:59.526895Z",
     "shell.execute_reply": "2024-12-08T00:44:59.526505Z"
    },
    "papermill": {
     "duration": 0.008916,
     "end_time": "2024-12-08T00:44:59.527665",
     "exception": false,
     "start_time": "2024-12-08T00:44:59.518749",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "dirs = [fld for fld in output_root.glob(f'{dataset}-*')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "56d609aa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:44:59.537648Z",
     "iopub.status.busy": "2024-12-08T00:44:59.537443Z",
     "iopub.status.idle": "2024-12-08T00:45:14.100655Z",
     "shell.execute_reply": "2024-12-08T00:45:14.100097Z"
    },
    "papermill": {
     "duration": 14.569375,
     "end_time": "2024-12-08T00:45:14.101517",
     "exception": false,
     "start_time": "2024-12-08T00:44:59.532142",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 48910800 entries, 0 to 48910799\n",
      "Data columns (total 5 columns):\n",
      " #   Column     Dtype   \n",
      "---  ------     -----   \n",
      " 0   item       int64   \n",
      " 1   score      float64 \n",
      " 2   user       int64   \n",
      " 3   rank       int64   \n",
      " 4   algorithm  category\n",
      "dtypes: category(1), float64(1), int64(3)\n",
      "memory usage: 1.5 GB\n"
     ]
    }
   ],
   "source": [
    "recs = []\n",
    "for fld in dirs:\n",
    "    for file in fld.glob(\"recs-*\"):\n",
    "        rec = pd.read_csv(file)\n",
    "        rec[\"algorithm\"] = fld.name.split(\"-\")[1]\n",
    "        recs.append(rec)\n",
    "\n",
    "recs = pd.concat(recs, ignore_index=True)\n",
    "recs = recs.astype({'algorithm': 'category'})\n",
    "recs.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "feaa4493",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:45:14.113722Z",
     "iopub.status.busy": "2024-12-08T00:45:14.113474Z",
     "iopub.status.idle": "2024-12-08T00:45:14.272453Z",
     "shell.execute_reply": "2024-12-08T00:45:14.272082Z"
    },
    "papermill": {
     "duration": 0.165396,
     "end_time": "2024-12-08T00:45:14.273134",
     "exception": false,
     "start_time": "2024-12-08T00:45:14.107738",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['UUE', 'IALS', 'Pop', 'III', 'ALS', 'Bias', 'IIE']\n",
       "Categories (7, object): ['ALS', 'Bias', 'IALS', 'IIE', 'III', 'Pop', 'UUE']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rec_algos = recs['algorithm'].unique()\n",
    "rec_algos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b378a77c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:45:14.284056Z",
     "iopub.status.busy": "2024-12-08T00:45:14.283819Z",
     "iopub.status.idle": "2024-12-08T00:45:15.256895Z",
     "shell.execute_reply": "2024-12-08T00:45:15.256442Z"
    },
    "papermill": {
     "duration": 0.979146,
     "end_time": "2024-12-08T00:45:15.257687",
     "exception": false,
     "start_time": "2024-12-08T00:45:14.278541",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2096340 entries, 0 to 2096339\n",
      "Data columns (total 6 columns):\n",
      " #   Column      Dtype   \n",
      "---  ------      -----   \n",
      " 0   user        int64   \n",
      " 1   item        int64   \n",
      " 2   rating      float64 \n",
      " 3   timestamp   int64   \n",
      " 4   prediction  float64 \n",
      " 5   algorithm   category\n",
      "dtypes: category(1), float64(2), int64(3)\n",
      "memory usage: 82.0 MB\n"
     ]
    }
   ],
   "source": [
    "preds = []\n",
    "for fld in dirs:\n",
    "    for file in fld.glob(\"pred-*\"):\n",
    "        pred = pd.read_csv(file)\n",
    "        pred[\"algorithm\"] = fld.name.split(\"-\")[1]\n",
    "        preds.append(pred)\n",
    "\n",
    "preds = pd.concat(preds, ignore_index=True)\n",
    "preds = preds.astype({'algorithm': 'category'})\n",
    "preds.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca476ae1",
   "metadata": {
    "papermill": {
     "duration": 0.004902,
     "end_time": "2024-12-08T00:45:15.268393",
     "exception": false,
     "start_time": "2024-12-08T00:45:15.263491",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We need to load the test data so that we have the ground truths for computing accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "25294727",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:45:15.279815Z",
     "iopub.status.busy": "2024-12-08T00:45:15.279582Z",
     "iopub.status.idle": "2024-12-08T00:45:15.282164Z",
     "shell.execute_reply": "2024-12-08T00:45:15.281760Z"
    },
    "papermill": {
     "duration": 0.009144,
     "end_time": "2024-12-08T00:45:15.282823",
     "exception": false,
     "start_time": "2024-12-08T00:45:15.273679",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "split_root = Path(\"data-split\")\n",
    "split_dir = split_root / dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "88fc0d4e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:45:15.294378Z",
     "iopub.status.busy": "2024-12-08T00:45:15.294163Z",
     "iopub.status.idle": "2024-12-08T00:45:15.474573Z",
     "shell.execute_reply": "2024-12-08T00:45:15.474147Z"
    },
    "papermill": {
     "duration": 0.187415,
     "end_time": "2024-12-08T00:45:15.475431",
     "exception": false,
     "start_time": "2024-12-08T00:45:15.288016",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>item</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>part</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>3</td>\n",
       "      <td>5505</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1136075848</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>3</td>\n",
       "      <td>1597</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1133571226</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>3</td>\n",
       "      <td>5952</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1133570984</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>3</td>\n",
       "      <td>151</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1133571026</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>3</td>\n",
       "      <td>7153</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1133571171</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user  item  rating   timestamp    part\n",
       "index                                        \n",
       "62        3  5505     2.0  1136075848  test-3\n",
       "54        3  1597     4.5  1133571226  test-3\n",
       "64        3  5952     3.5  1133570984  test-3\n",
       "43        3   151     4.5  1133571026  test-3\n",
       "68        3  7153     4.0  1133571171  test-3"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = []\n",
    "for file in split_dir.glob(\"test-*.parquet\"):\n",
    "    test.append(pd.read_parquet(file,).assign(part=file.stem.replace('.parquet', '')))\n",
    "\n",
    "test = pd.concat(test).rename(columns={\n",
    "    'user_id': 'user',\n",
    "    'item_id': 'item',\n",
    "})\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5770c372",
   "metadata": {
    "papermill": {
     "duration": 0.00493,
     "end_time": "2024-12-08T00:45:15.486393",
     "exception": false,
     "start_time": "2024-12-08T00:45:15.481463",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Top-N Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e347325",
   "metadata": {
    "papermill": {
     "duration": 0.005385,
     "end_time": "2024-12-08T00:45:15.496948",
     "exception": false,
     "start_time": "2024-12-08T00:45:15.491563",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The topn.RecListAnalysis class computes top-N metrics for recommendation list and takes care of making sure that the recommendations and ground truths are properly matched. Refer to the documentation for detailed explanation of the purpose for the RecListAnalysis class and how the analysis is done - https://lkpy.lenskit.org/en/stable/evaluation/topn-metrics.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "198eda4d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:45:15.507930Z",
     "iopub.status.busy": "2024-12-08T00:45:15.507701Z",
     "iopub.status.idle": "2024-12-08T00:45:53.738955Z",
     "shell.execute_reply": "2024-12-08T00:45:53.738436Z"
    },
    "papermill": {
     "duration": 38.237817,
     "end_time": "2024-12-08T00:45:53.739805",
     "exception": false,
     "start_time": "2024-12-08T00:45:15.501988",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn analyzing 48910800 recommendations (349390 truth rows)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn using rec key columns ['algorithm', 'user']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn using truth key columns ['user']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn numbering truth lists\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn numbering rec lists\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn collecting metric results\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn measured 489108 lists in 30.23s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn filling in missing user info (489108 initial rows)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/mde48/micromamba/envs/lkdemo/lib/python3.10/site-packages/lenskit/topn.py:152: FutureWarning: In a future version of pandas, a length 1 tuple will be returned when iterating over a groupby with a grouper equal to a list of length 1. Don't supply a list with a single grouper to avoid this warning.\n",
      "  for key, df in res.groupby(ug_cols):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>nrecs</th>\n",
       "      <th>precision</th>\n",
       "      <th>recip_rank</th>\n",
       "      <th>ndcg</th>\n",
       "      <th>ntruth</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th>user</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">ALS</th>\n",
       "      <th>6</th>\n",
       "      <td>100.0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.010309</td>\n",
       "      <td>0.042542</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>100.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>100.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>100.0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.025641</td>\n",
       "      <td>0.053122</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>100.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                nrecs  precision  recip_rank      ndcg  ntruth\n",
       "algorithm user                                                \n",
       "ALS       6     100.0       0.01    0.010309  0.042542       5\n",
       "          8     100.0       0.00    0.000000  0.000000       5\n",
       "          12    100.0       0.00    0.000000  0.000000       5\n",
       "          16    100.0       0.01    0.025641  0.053122       5\n",
       "          22    100.0       0.00    0.000000  0.000000       5"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rla = topn.RecListAnalysis()\n",
    "\n",
    "rla.add_metric(topn.precision)\n",
    "rla.add_metric(topn.recip_rank)\n",
    "rla.add_metric(topn.ndcg)\n",
    "results = rla.compute(recs, test.drop(columns=['rating']), include_missing=True)\n",
    "results = results.fillna(0)\n",
    "results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75dbe033",
   "metadata": {
    "papermill": {
     "duration": 0.005579,
     "end_time": "2024-12-08T00:45:53.751822",
     "exception": false,
     "start_time": "2024-12-08T00:45:53.746243",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We will reshape the 'results' dataframe by stacking the columns to index and then use the bar chart to visualize the performance of our algorithms with respect to the precision, reciprocal rank and ndcg metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4c7ffaf7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:45:53.764521Z",
     "iopub.status.busy": "2024-12-08T00:45:53.764270Z",
     "iopub.status.idle": "2024-12-08T00:45:53.841652Z",
     "shell.execute_reply": "2024-12-08T00:45:53.841187Z"
    },
    "papermill": {
     "duration": 0.08474,
     "end_time": "2024-12-08T00:45:53.842389",
     "exception": false,
     "start_time": "2024-12-08T00:45:53.757649",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>algorithm</th>\n",
       "      <th>user</th>\n",
       "      <th>metric</th>\n",
       "      <th>val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ALS</td>\n",
       "      <td>6</td>\n",
       "      <td>precision</td>\n",
       "      <td>0.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ALS</td>\n",
       "      <td>6</td>\n",
       "      <td>recip_rank</td>\n",
       "      <td>0.010309</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ALS</td>\n",
       "      <td>6</td>\n",
       "      <td>ndcg</td>\n",
       "      <td>0.042542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ALS</td>\n",
       "      <td>8</td>\n",
       "      <td>precision</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ALS</td>\n",
       "      <td>8</td>\n",
       "      <td>recip_rank</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  algorithm  user      metric       val\n",
       "0       ALS     6   precision  0.010000\n",
       "1       ALS     6  recip_rank  0.010309\n",
       "2       ALS     6        ndcg  0.042542\n",
       "3       ALS     8   precision  0.000000\n",
       "4       ALS     8  recip_rank  0.000000"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pltData = (results.drop(columns=['nrecs', 'ntruth']).stack()).reset_index()\n",
    "pltData.columns = ['algorithm', 'user', 'metric', 'val']\n",
    "pltData.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66c6e8ba",
   "metadata": {
    "papermill": {
     "duration": 0.005499,
     "end_time": "2024-12-08T00:45:53.854340",
     "exception": false,
     "start_time": "2024-12-08T00:45:53.848841",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We need to determine if the differences we observe in the performances of the algorithms for the various metrics are statistically significant. To achieve this, we will need to use either a parametric or non-parametric statistical test for comparing the differences. We will consider a parametric test - repeated ANOVA measure cause our sample groups are correlated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bd651349",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:45:53.867047Z",
     "iopub.status.busy": "2024-12-08T00:45:53.866798Z",
     "iopub.status.idle": "2024-12-08T00:46:06.660715Z",
     "shell.execute_reply": "2024-12-08T00:46:06.660194Z"
    },
    "papermill": {
     "duration": 12.801524,
     "end_time": "2024-12-08T00:46:06.661556",
     "exception": false,
     "start_time": "2024-12-08T00:45:53.860032",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABEAAAAEiCAYAAAD5+6e5AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABZnUlEQVR4nO3de1hVZfr/8c+Wo4GQooGmIh5S8ZAKJmCkToqHNJ10RCvUxkOMlikdSZ3Umsg0RfPcWGiTiIWmNZRi3zwlWRI4/Sa/5jg6mAMZVpA0guL6/eHXPW43KCCbzd68X9e1rlzPftaz7mfDdW+697PWMhmGYQgAAAAAAMCJ1bN3AAAAAAAAALZGAQQAAAAAADg9CiAAAAAAAMDpUQABAAAAAABOjwIIAAAAAABwehRAAAAAAACA06MAAgAAAAAAnB4FEAAAAAAA4PQogAAAAAAAAKdHAQT4P3PnzlW3bt3sHYZdmEwmvf/++9XeF0Dd5aw5tVWrVkpMTLR3GBVCvgZQGY6at/v27asZM2bYOww4CJNhGIa9gwBspW/fvurWrVuF/lg9d+6ciouL5efnZ/vAapm8vDw1bNhQHh4e1doXgHMhp0o//PCDvLy8dMstt9g7lBsymUzaunWrRowYYe9QANhJXcjblZkj4GrvAAB7MwxDpaWl8vb2lre3t73DqZSSkhK5u7vf9DgBAQE26Qug7qmtOfXChQtyc3O76XGaNGlSDdGUr7riBICKqq15G7AFLoFBrdC3b189/vjjmjFjhho2bCh/f3+tXbtWRUVFeuSRR9SgQQO1adNGH330kcVx33zzjYYMGSJvb2/5+/srJiZG+fn5kqQJEyZoz549Wrp0qUwmk0wmk06ePKndu3fLZDJpx44dCg0NlYeHh/bt21fmsr8333xTnTp1koeHh5o2barHHnvMZu/BlfOvWbNGLVq00C233KLf/e53+vnnn819JkyYoBEjRighIUHNmjXTHXfcIUk6ffq0oqOj1bBhQ/n5+Wn48OE6efJkhedy9TLpkpISPfbYY2ratKk8PT3VqlUrJSQklNlXkr7++mv95je/Uf369eXn56cpU6bo3LlzVjEvWrRITZs2lZ+fn6ZNm6YLFy5U35sHwAI59b859c0331Tr1q3l4eEhwzBUUFCgKVOm6LbbbpOPj49+85vf6PDhwxbHbt++XaGhofL09FTjxo31wAMPmF+79hIYk8mkVatWafDgwapfv76CgoL07rvvVijGkydPymQyafPmzerbt688PT31l7/8RWfPntXYsWPVvHlz3XLLLerSpYuSk5Mtju3bt6+mT5+uZ555Ro0aNVJAQIDmzp173fPNnz9f/v7+ys7OrlB8AGoOefu/efvtt99Wq1at5OvrqzFjxuiXX34x9ykqKtK4cePk7e2tpk2b6rXXXrMap7i4WM8884xatGghDw8PtWvXTuvWrTO/vn37drVr107169dXv379tH79eplMJou/ueG8KICg1li/fr0aN26sL774Qo8//rj+8Ic/6He/+50iIiL01VdfaeDAgYqJidGvv/4qScrNzVWfPn3UrVs3HTp0SB9//LG+//57jR49WpK0dOlShYeHa/LkycrNzVVubq5atGhhPt8zzzyjhIQEHTlyRF27drWKZ9WqVZo2bZqmTJmir7/+Wtu3b1fbtm3Ljf+dd94xV87L2955553rvgf/+Mc/tHnzZn3wwQf6+OOPlZ2drWnTpln0+eSTT3TkyBGlp6frww8/1K+//qp+/frJ29tbe/fu1f79++Xt7a1BgwappKSk0nNZtmyZtm/frs2bN+vo0aP6y1/+olatWpXZ99dff9WgQYPUsGFDffnll3r33Xe1a9cuqw/HTz/9VMePH9enn36q9evXKykpSUlJSdd9LwDcHHLqf3Nqamqq+X/677vvPuXl5SktLU2ZmZnq0aOH7r33Xv3444+SpL/+9a964IEHdN999ykrK0uffPKJQkNDr3ueOXPmaOTIkTp8+LAefvhhjR07VkeOHLnuMVd79tlnNX36dB05ckQDBw7U+fPnFRISog8//FD/7//9P02ZMkUxMTE6ePCgxXHr16+Xl5eXDh48qFdffVXz589Xenq61fiGYeiJJ57QunXrtH//foe8xh+oC8jb0vHjx/X+++/rww8/1Icffqg9e/bolVdeMb/+9NNP69NPP9XWrVu1c+dO7d69W5mZmRZjjBs3Tps2bdKyZct05MgRrV692ryy5eTJkxo1apRGjBih7OxsPfroo5o1a9YNfjJwKgZQC/Tp08e4++67zfsXL140vLy8jJiYGHNbbm6uIcnIyMgwDMMw5syZY0RFRVmMc+rUKUOScfToUfO4TzzxhEWfTz/91JBkvP/++xbtL7zwgnHnnXea95s1a2bMmjWrwnMoLCw0jh07dt2tsLCw3ONfeOEFw8XFxTh16pS57aOPPjLq1atn5ObmGoZhGOPHjzf8/f2N4uJic59169YZ7du3Ny5dumRuKy4uNurXr2/s2LGjQnORZGzdutUwDMN4/PHHjd/85jcW45XXd+3atUbDhg2Nc+fOmV//61//atSrV8/Iy8szxxwYGGhcvHjR3Od3v/udER0dXW48AG4OOfXy+d3c3IwzZ86Y2z755BPDx8fHOH/+vEXfNm3aGGvWrDEMwzDCw8ONhx56qNxxAwMDjSVLlpj3JRmxsbEWfXr16mX84Q9/uOEcT5w4YUgyEhMTb9h3yJAhxpNPPmnev/ZnbBiG0bNnT+PZZ5+1iO3dd981Hn74YaNDhw4Wny8Aahfy9uXz33LLLRZ9nn76aaNXr16GYRjGL7/8Yri7uxubNm0yv3727Fmjfv365jkePXrUkGSkp6eXeY5nn33W6Ny5s0XbrFmzDEnGTz/9VOG5wnFxDxDUGldXnl1cXOTn56cuXbqY2/z9/SVJZ86ckSRlZmbq008/LfNaxePHj5svDynP9b7RO3PmjP7973/r3nvvrXD8DRo0UIMGDSrcvywtW7ZU8+bNzfvh4eG6dOmSjh49ar73RpcuXSzu+5GZmal//OMfVuc+f/68jh8/Xum5TJgwQQMGDFD79u01aNAgDR06VFFRUWX2PXLkiO688055eXmZ23r37m2O+crPrFOnTnJxcTH3adq0qb7++usKxQOgasipUmBgoMU9OzIzM3Xu3DmrG/z95z//0fHjxyVJ2dnZmjx5cqXOEx4ebrVfmctMrn3vSktL9corryglJUWnT59WcXGxiouLLXKtJKtvbJs2bWr+eV4xc+ZMeXh46PPPP1fjxo0rHBOAmkfevnyZ4dVjXJ3Xjh8/rpKSEouc26hRI7Vv3968n52dLRcXF/Xp06fM8Y8ePaqePXtatN111103FTMcCwUQ1BrX3vTNZDJZtJlMJknSpUuXzP8dNmyYFixYYDVW06ZNb3i+a/+QvFr9+vUrFPPV3nnnHT366KPX7bNmzRo99NBDFR7zypyv/FeyjvvSpUsKCQkpc0lhkyZNVK9e5a5069Gjh06cOKGPPvpIu3bt0ujRo9W/f3+99957Vn0Nw7CIrazYpbJ/tld+jgBsg5xadr5s2rSpdu/ebdX31ltvrXKsZSkvN5bl2jhfe+01LVmyRImJierSpYu8vLw0Y8YM82WNV1Qktw4YMEDJycnasWNHpT5/ANQ88vb185pRgYeX3ijusv52rci4cB4UQOCwevToodTUVLVq1UqurmX/Kru7u6u0tLTSYzdo0ECtWrXSJ598on79+lXomPvvv1+9evW6bp8rlfvy5OTk6N///reaNWsmScrIyFC9evWuW8Hv0aOHUlJSzDf0K0tl5+Lj46Po6GhFR0dr1KhRGjRokH788Uc1atTIol9wcLDWr1+voqIi84foZ599dsOYAdQ+zphTr9WjRw/l5eXJ1dW13Hsbde3aVZ988okeeeSRCo/7+eefa9y4cRb73bt3r1RsV9u3b5+GDx+uhx9+WNLl/8k5duyYOnbsWOmx7r//fg0bNkwPPvigXFxcNGbMmCrHBaB2qQt5+2pt27aVm5ubPv/8c7Vs2VKS9NNPP+nbb781r/jo0qWLLl26pD179qh///5WY3To0EFpaWkWbYcOHapyTHA8FEDgsKZNm6Y33nhDY8eO1dNPP63GjRvrH//4hzZt2qQ33nhDLi4uatWqlQ4ePKiTJ0/K29vb6n/gr2fu3LmKjY3VbbfdpsGDB+uXX37RZ599pscff7zM/tWx7M/T01Pjx4/XokWLVFhYqOnTp2v06NHXffTsQw89pIULF2r48OGaP3++mjdvrpycHG3ZskVPP/20mjdvXqm5LFmyRE2bNlW3bt1Ur149vfvuuwoICDB/O3rtuV944QWNHz9ec+fO1Q8//KDHH39cMTExN/UBB6DmOWNOvVb//v0VHh6uESNGaMGCBWrfvr3+/e9/Ky0tTSNGjFBoaKheeOEF3XvvvWrTpo3GjBmjixcv6qOPPtIzzzxT7rjvvvuuQkNDdffdd+udd97RF198YfHEgcpq27atUlNTdeDAATVs2FCLFy9WXl5elQogkvTb3/5Wb7/9tmJiYuTq6qpRo0ZVOTYAtUddyNtX8/b21sSJE/X000/Lz89P/v7+mjVrlsVq51atWmn8+PH6/e9/r2XLlunOO+/Uv/71L505c0ajR4/Wo48+qsWLF+vZZ5/VxIkTlZ2dbb4xf2VW7sFx8RQYOKxmzZrps88+U2lpqQYOHKjOnTvriSeekK+vrzkRPvXUU3JxcVFwcLCaNGminJycCo8/fvx4JSYmauXKlerUqZOGDh2qY8eO2Wo6ki7/0fvAAw9oyJAhioqKUufOnbVy5crrHnPLLbdo7969atmypR544AF17NhRv//97/Wf//zHvCKkMnPx9vbWggULFBoaqp49e+rkyZNKS0sr81KaW265RTt27NCPP/6onj17atSoUbr33nu1fPnym38zANQoZ8yp1zKZTEpLS9M999yj3//+97rjjjs0ZswYnTx50ly07du3r959911t375d3bp1029+8xurp69ca968edq0aZO6du2q9evX65133lFwcHCV45wzZ4569OihgQMHqm/fvgoICNCIESOqPJ4kjRo1SuvXr1dMTIy2bNlyU2MBqB3qQt6+1sKFC3XPPffo/vvvV//+/XX33XcrJCTEos+qVas0atQoTZ06VR06dNDkyZNVVFQkSQoKCtJ7772nLVu2qGvXrlq1apX5KTAeHh41Ph/UPJPBRU9ArTB37ly9//77lbpxHgDAvkwmk7Zu3XrTBQoAgH386U9/0urVq3Xq1Cl7h4IawCUwAAAAAIA6YeXKlerZs6f8/Pz02WefaeHChXrsscfsHRZqCJfAAAAA2MDLL78sb2/vMrfBgwfbOzwAqJOOHTum4cOHKzg4WC+++KKefPJJzZ07195hoYZwCQwAAIAN/Pjjj/rxxx/LfK1+/fq6/fbbazgiAADqNgogAAAAAADA6XEJDAAAAFBDVq5cqaCgIHl6eiokJET79u0rt+/+/fvVu3dv+fn5qX79+urQoYOWLFli1S81NVXBwcHy8PBQcHCwtm7dasspAIDDogACAAAA1ICUlBTNmDFDs2bNUlZWliIjIzV48OByH03q5eWlxx57THv37tWRI0c0e/ZszZ49W2vXrjX3ycjIUHR0tGJiYnT48GHFxMRo9OjRN3x8MwDURVwCU0WGYeiXX35RgwYNZDKZ7B0OADglci0AZ9KrVy/16NFDq1atMrd17NhRI0aMUEJCQoXGeOCBB+Tl5aW3335bkhQdHa3CwkJ99NFH5j6DBg1Sw4YNlZycfMPxyLMA6hJWgFTRL7/8Il9fX/3yyy/2DgUAnBa5FoCzKCkpUWZmpqKioizao6KidODAgQqNkZWVpQMHDqhPnz7mtoyMDKsxBw4cWO6YxcXFKiwsNG+nT58mzwKoMyiAAAAAADaWn5+v0tJS+fv7W7T7+/srLy/vusc2b95cHh4eCg0N1bRp0zRp0iTza3l5eZUaMyEhQb6+vuatRYsWVZwRADgeCiAAAABADbn2MhPDMG546cm+fft06NAhrV69WomJiVaXtlRmzPj4eBUUFJi3U6dOVWEWAOCYXO0dAAAAAODsGjduLBcXF6uVGWfOnLFawXGtoKAgSVKXLl30/fffa+7cuRo7dqwkKSAgoFJjenh4yMPDo6rTAACHxgoQAAAAwMbc3d0VEhKi9PR0i/b09HRFRERUeBzDMFRcXGzeDw8Ptxpz586dlRoTAOoKVoAAAAAANSAuLk4xMTEKDQ1VeHi41q5dq5ycHMXGxkq6fHnK6dOntWHDBknSihUr1LJlS3Xo0EGStH//fi1atEiPP/64ecwnnnhC99xzjxYsWKDhw4dr27Zt2rVrl/bv31/zEwSAWo4CCAAAAFADoqOjdfbsWc2fP1+5ubnq3Lmz0tLSFBgYKEnKzc1VTk6Ouf+lS5cUHx+vEydOyNXVVW3atNErr7yiRx991NwnIiJCmzZt0uzZszVnzhy1adNGKSkp6tWrV43PDwBqO5NhGIa9g3BEhYWF8vX1VUFBgXx8fOwdDgA4JXItANgWeRZAXcIKEDgdwzBUVFRk3vfy8rrh3dUBALAXPrcAlIXcAFQ/CiBwOkVFRRo+fLh5f9u2bfL29rZjRAAAlI/PLQBlITcA1Y+nwAAAAAAAAKfHChDUSiFPb6jysaaLJfK9ar/vnE0yXN2rNFbmwnFVjgMAAACo6t+1/E0LVD8KIAAAADepNhTu+R8cAACuj0tgAAAAAACA02MFCJyO4eKmgq5jLfYBAAAAR8LftED1owAC52MyVfn6SAAAAKBW4G9aoNrZ/RKYlStXKigoSJ6engoJCdG+ffuu23/Pnj0KCQmRp6enWrdurdWrV1u8vmXLFoWGhurWW2+Vl5eXunXrprfffvumzwsAzq4yeXH//v3q3bu3/Pz8VL9+fXXo0EFLliyx6peamqrg4GB5eHgoODhYW7duteUUAId05VveKxvf8gIAYBt2LYCkpKRoxowZmjVrlrKyshQZGanBgwcrJyenzP4nTpzQkCFDFBkZqaysLD3//POaPn26UlNTzX0aNWqkWbNmKSMjQ3/729/0yCOP6JFHHtGOHTuqfF4AcHaVzYteXl567LHHtHfvXh05ckSzZ8/W7NmztXbtWnOfjIwMRUdHKyYmRocPH1ZMTIxGjx6tgwcP1tS0AMfwf9/yXtlkMtk7IgAAnJLJMAzDXifv1auXevTooVWrVpnbOnbsqBEjRighIcGq/7PPPqvt27fryJEj5rbY2FgdPnxYGRkZ5Z6nR48euu+++/Tiiy9W6bxlKSwslK+vrwoKCuTj41OhY1BxN3M3/erEHfVRV1RHXnzggQfk5eVlXnUXHR2twsJCffTRR+Y+gwYNUsOGDZWcnFyhMcm1cBS14XOLzyxUBXnW9sgPQO1htxUgJSUlyszMVFRUlEV7VFSUDhw4UOYxGRkZVv0HDhyoQ4cO6cKFC1b9DcPQJ598oqNHj+qee+6p8nklqbi4WIWFhRYbADiDqubFq2VlZenAgQPq06ePua28nE2uBQAAgD3YrQCSn5+v0tJS+fv7W7T7+/srLy+vzGPy8vLK7H/x4kXl5+eb2woKCuTt7S13d3fdd999ev311zVgwIAqn1eSEhIS5Ovra95atGhRqfkCQG1V1bwoSc2bN5eHh4dCQ0M1bdo0TZo0yfxaeTmbXAsAAAB7sPtNUE3XXOdqGIZV2436X9veoEEDZWdn68svv9Sf/vQnxcXFaffu3Td13vj4eBUUFJi3U6dOXXdeAOBoKpsXJWnfvn06dOiQVq9ercTERKtLW8i1AAAAqC3s9hjcxo0by8XFxeqbwDNnzlh9Y3hFQEBAmf1dXV3l5+dnbqtXr57atm0rSerWrZuOHDmihIQE9e3bt0rnlSQPDw95eHhUao4A4AiqmhclKSgoSJLUpUsXff/995o7d67Gjh0rqfycTa4FAACAPdhtBYi7u7tCQkKUnp5u0Z6enq6IiIgyjwkPD7fqv3PnToWGhsrNrfxHxhmGoeLi4iqfFwCcWXXlxatzrVR+zibXAgAAwB7stgJEkuLi4hQTE6PQ0FCFh4dr7dq1ysnJUWxsrKTLS6FPnz6tDRsu3zk5NjZWy5cvV1xcnCZPnqyMjAytW7fOYsl1QkKCQkND1aZNG5WUlCgtLU0bNmyweLLBjc4LAHVNZfPxihUr1LJlS3Xo0EGStH//fi1atEiPP/64ecwnnnhC99xzjxYsWKDhw4dr27Zt2rVrl/bv31/zEwQAAECdZ9cCSHR0tM6ePav58+crNzdXnTt3VlpamgIDAyVJubm5ysnJMfcPCgpSWlqaZs6cqRUrVqhZs2ZatmyZRo4cae5TVFSkqVOn6rvvvlP9+vXVoUMH/eUvf1F0dHSFzwsAdU1l8/GlS5cUHx+vEydOyNXVVW3atNErr7yiRx991NwnIiJCmzZt0uzZszVnzhy1adNGKSkp6tWrV43PDwAAADAZV+4iikrhmem2VRuely7xzHTA3si1cBS14XOLzyxUBXnW9sgPQO1h96fAAAAAAAAA2BoFEAAAAAAA4PQogAAAAAAAAKdHAQQAAAAAADg9CiAAAABADVm5cqWCgoLk6empkJAQ7du3r9y+W7Zs0YABA9SkSRP5+PgoPDxcO3bssOiTlJQkk8lktZ0/f97WUwEAh0MBBAAAVJphGDp37px546FywI2lpKRoxowZmjVrlrKyshQZGanBgwdbPGb8anv37tWAAQOUlpamzMxM9evXT8OGDVNWVpZFPx8fH+Xm5lpsnp6eNTElAHAorvYOAAAAOJ6ioiINHz7cvL9t2zZ5e3vbMSKg9lu8eLEmTpyoSZMmSZISExO1Y8cOrVq1SgkJCVb9ExMTLfZffvllbdu2TR988IG6d+9ubjeZTAoICLBp7IAjMwxDRUVF5n0vLy+ZTCY7RgR7YQUIAAAAYGMlJSXKzMxUVFSURXtUVJQOHDhQoTEuXbqkX375RY0aNbJoP3funAIDA9W8eXMNHTrUaoUIUNddKdpf2a4uhqBuoQACAAAA2Fh+fr5KS0vl7+9v0e7v76+8vLwKjfHaa6+pqKhIo0ePNrd16NBBSUlJ2r59u5KTk+Xp6anevXvr2LFjZY5RXFyswsJCiw0A6gougQEAoI4KeXpDlY81XSyR71X7fedskuHqXulxMheOq3IMgCO6dtm9YRgVWoqfnJysuXPnatu2bbrtttvM7WFhYQoLCzPv9+7dWz169NDrr7+uZcuWWY2TkJCgefPm3cQMAPup6udWdX1mSXxuOTpWgAAAAAA21rhxY7m4uFit9jhz5ozVqpBrpaSkaOLEidq8ebP69+9/3b716tVTz549y10BEh8fr4KCAvN26tSpyk0EABwYBRAAAADAxtzd3RUSEqL09HSL9vT0dEVERJR7XHJysiZMmKCNGzfqvvvuu+F5DMNQdna2mjZtWubrHh4e8vHxsdgAoK7gEhgAAFBphoubCrqOtdgHcH1xcXGKiYlRaGiowsPDtXbtWuXk5Cg2NlbS5dUZp0+f1oYNl5f5Jycna9y4cVq6dKnCwsLMq0fq168vX9/LC/rnzZunsLAwtWvXToWFhVq2bJmys7O1YsUK+0wSqIX4zMIVFEAAAEDlmUxVvn4aqKuio6N19uxZzZ8/X7m5uercubPS0tIUGBgoScrNzVVOTo65/5o1a3Tx4kVNmzZN06ZNM7ePHz9eSUlJkqSff/5ZU6ZMUV5ennx9fdW9e3ft3btXd911V43ODajV+MzC/6EAAgAAANSQqVOnaurUqWW+dqWoccXu3btvON6SJUu0ZMmSaogMAJwf9wABAAAAAABOjwIIAAAAAABwehRAAAAAAACA06MAAgAAAAAAnB4FEAAAAAAA4PTsXgBZuXKlgoKC5OnpqZCQEO3bt++6/ffs2aOQkBB5enqqdevWWr16tcXrb7zxhiIjI9WwYUM1bNhQ/fv31xdffGHRZ+7cuTKZTBZbQEBAtc8NABxJZfLxli1bNGDAADVp0kQ+Pj4KDw/Xjh07LPokJSVZ5VqTyaTz58/beioAAACAFbsWQFJSUjRjxgzNmjVLWVlZioyM1ODBgy2ef361EydOaMiQIYqMjFRWVpaef/55TZ8+XampqeY+u3fv1tixY/Xpp58qIyNDLVu2VFRUlE6fPm0xVqdOnZSbm2vevv76a5vOFQBqs8rm471792rAgAFKS0tTZmam+vXrp2HDhikrK8uin4+Pj0Wuzc3NlaenZ01MCQAAALDgas+TL168WBMnTtSkSZMkSYmJidqxY4dWrVqlhIQEq/6rV69Wy5YtlZiYKEnq2LGjDh06pEWLFmnkyJGSpHfeecfimDfeeEPvvfeePvnkE40bN87c7urqyqoPAPg/lc3HV/LwFS+//LK2bdumDz74QN27dze3s8IOAAAAtYXdVoCUlJQoMzNTUVFRFu1RUVE6cOBAmcdkZGRY9R84cKAOHTqkCxculHnMr7/+qgsXLqhRo0YW7ceOHVOzZs0UFBSkMWPG6J///Od14y0uLlZhYaHFBgDOoCr5+FqXLl3SL7/8YpVrz507p8DAQDVv3lxDhw61WiFyLXItAAAAbMVuBZD8/HyVlpbK39/fot3f3195eXllHpOXl1dm/4sXLyo/P7/MY5577jndfvvt6t+/v7mtV69e2rBhg3bs2KE33nhDeXl5ioiI0NmzZ8uNNyEhQb6+vuatRYsWFZ0qANRqVcnH13rttddUVFSk0aNHm9s6dOigpKQkbd++XcnJyfL09FTv3r117Nixcsch1wIAAMBW7H4TVJPJZLFvGIZV2436l9UuSa+++qqSk5O1ZcsWi2vOBw8erJEjR6pLly7q37+//vrXv0qS1q9fX+554+PjVVBQYN5OnTp148kBgAOpbD6+Ijk5WXPnzlVKSopuu+02c3tYWJgefvhh3XnnnYqMjNTmzZt1xx136PXXXy93LHItAAAAbMVu9wBp3LixXFxcrL5dPHPmjNW3kFcEBASU2d/V1VV+fn4W7YsWLdLLL7+sXbt2qWvXrteNxcvLS126dLnut5IeHh7y8PC47jgA4Iiqko+vSElJ0cSJE/Xuu+9arLQrS7169dSzZ09yLQAAAOzCbitA3N3dFRISovT0dIv29PR0RURElHlMeHi4Vf+dO3cqNDRUbm5u5raFCxfqxRdf1Mcff6zQ0NAbxlJcXKwjR46oadOmVZgJADi2quRj6fLKjwkTJmjjxo267777bngewzCUnZ1NrgUAAIBd2PUpMHFxcYqJiVFoaKjCw8O1du1a5eTkKDY2VtLlpdCnT5/Whg0bJEmxsbFavny54uLiNHnyZGVkZGjdunVKTk42j/nqq69qzpw52rhxo1q1amX+RtPb21ve3t6SpKeeekrDhg1Ty5YtdebMGb300ksqLCzU+PHja/gdAIDaobL5ODk5WePGjdPSpUsVFhZmzrX169eXr6+vJGnevHkKCwtTu3btVFhYqGXLlik7O1srVqywzyQBAABQp9m1ABIdHa2zZ89q/vz5ys3NVefOnZWWlqbAwEBJUm5urnJycsz9g4KClJaWppkzZ2rFihVq1qyZli1bZn4EriStXLlSJSUlGjVqlMW5XnjhBc2dO1eS9N1332ns2LHKz89XkyZNFBYWps8//9x8XgCoayqbj9esWaOLFy9q2rRpmjZtmrl9/PjxSkpKkiT9/PPPmjJlivLy8uTr66vu3btr7969uuuuu2p0bgAAAIAkmYwrdxFFpRQWFsrX11cFBQXy8fGxdzhOJ+TpDfYOQZKUuXCcvUOwK8MwVFRUZN738vKq0E0xgepCrrWt2pBrnSXP8l7CUZFnbY/8UH14L3Gz7LoCBEDtVlRUpOHDh5v3t23bZr6UDAAAAAAcCQUQwMndTKXcdLFEvlft952zSYare5XGoloOAAAAwJ7s9hQYAAAAAACAmkIBBAAAAAAAOD0ugQFQLsPFTQVdx1rsAwAAAIAjogACoHwmU5Xv+QEAAAAAtQmXwAAAAAAAAKdHAQQAAAAAADg9CiAAAABADVm5cqWCgoLk6empkJAQ7du3r9y+W7Zs0YABA9SkSRP5+PgoPDxcO3bssOqXmpqq4OBgeXh4KDg4WFu3brXlFADAYVEAAQAAAGpASkqKZsyYoVmzZikrK0uRkZEaPHiwcnJyyuy/d+9eDRgwQGlpacrMzFS/fv00bNgwZWVlmftkZGQoOjpaMTExOnz4sGJiYjR69GgdPHiwpqYFAA6DAggAAABQAxYvXqyJEydq0qRJ6tixoxITE9WiRQutWrWqzP6JiYl65pln1LNnT7Vr104vv/yy2rVrpw8++MCiz4ABAxQfH68OHTooPj5e9957rxITE2toVgDgOCiAAAAAADZWUlKizMxMRUVFWbRHRUXpwIEDFRrj0qVL+uWXX9SoUSNzW0ZGhtWYAwcOLHfM4uJiFRYWWmwAUFdQAAEAAABsLD8/X6WlpfL397do9/f3V15eXoXGeO2111RUVKTRo0eb2/Ly8io1ZkJCgnx9fc1bixYtKjkTAHBcFEAAAACAGmIymSz2DcOwaitLcnKy5s6dq5SUFN12221VHjM+Pl4FBQXm7dSpU5WcAQA4Lld7BwAAAAA4u8aNG8vFxcVqZcaZM2esVnBcKyUlRRMnTtS7776r/v37W7wWEBBQqTE9PDzk4eFRhRkAgONjBQgAAABgY+7u7goJCVF6erpFe3p6uiIiIso9Ljk5WRMmTNDGjRt13333Wb0eHh5uNebOnTuvOyYA1FWsAAEAAABqQFxcnGJiYhQaGqrw8HCtXbtWOTk5io2NlXT58pTTp09rw4YNki4XP8aNG6elS5cqLCzMvNKjfv368vX1lSQ98cQTuueee7RgwQINHz5c27Zt065du7R//377TBIAajFWgAAAAAA1IDo6WomJiZo/f766deumvXv3Ki0tTYGBgZKk3Nxc5eTkmPuvWbNGFy9e1LRp09S0aVPz9sQTT5j7REREaNOmTXrrrbfUtWtXJSUlKSUlRb169arx+QFAbccKEAAAAKCGTJ06VVOnTi3ztaSkJIv93bt3V2jMUaNGadSoUTcZGQA4P1aAAAAAAAAAp2f3AsjKlSsVFBQkT09PhYSEaN++fdftv2fPHoWEhMjT01OtW7fW6tWrLV5/4403FBkZqYYNG6phw4bq37+/vvjii5s+LwA4u8rkxS1btmjAgAFq0qSJfHx8FB4erh07dlj1S01NVXBwsDw8PBQcHKytW7facgoAAABAuexaAElJSdGMGTM0a9YsZWVlKTIyUoMHD7a49vFqJ06c0JAhQxQZGamsrCw9//zzmj59ulJTU819du/erbFjx+rTTz9VRkaGWrZsqaioKJ0+fbrK5wUAZ1fZvLh3714NGDBAaWlpyszMVL9+/TRs2DBlZWWZ+2RkZCg6OloxMTE6fPiwYmJiNHr0aB08eLCmpgUAAACYmQzDMOx18l69eqlHjx5atWqVua1jx44aMWKEEhISrPo/++yz2r59u44cOWJui42N1eHDh5WRkVHmOUpLS9WwYUMtX75c48aNq9J5y1JYWChfX18VFBTIx8enQseg4kKe3mDvECRJmQvH2TuEm8Z7iYqojrzYqVMnRUdH649//KOkyzf7Kyws1EcffWTuM2jQIDVs2FDJyckVGpNca1u1IT84S27gvYSjIs/aHvmh+vBe4mbZbQVISUmJMjMzFRUVZdEeFRWlAwcOlHlMRkaGVf+BAwfq0KFDunDhQpnH/Prrr7pw4YIaNWpU5fMCgDOrjrx46dIl/fLLL+ZcK5Wfs8m1AAAAsAe7PQUmPz9fpaWl8vf3t2j39/c3P+P8Wnl5eWX2v3jxovLz89W0aVOrY5577jndfvvt6t+/f5XPK0nFxcUqLi427xcWFl5/ggDgIKqaF6/22muvqaioSKNHjza3lZezybUAAACwB7vfBNVkMlnsG4Zh1Xaj/mW1S9Krr76q5ORkbdmyRZ6enjd13oSEBPn6+pq3Fi1alNsXABxRZfPiFcnJyZo7d65SUlJ022233dSY5FoAAADYit0KII0bN5aLi4vVN4Fnzpyx+sbwioCAgDL7u7q6ys/Pz6J90aJFevnll7Vz50517dr1ps4rSfHx8SooKDBvp06dqtA8AaC2q2pelC7fPHXixInavHmzeaXdFeXlbHItAAAA7MFuBRB3d3eFhIQoPT3doj09PV0RERFlHhMeHm7Vf+fOnQoNDZWbm5u5beHChXrxxRf18ccfKzQ09KbPK0keHh7y8fGx2ADAGVQ1LyYnJ2vChAnauHGj7rvvPqvXy8vZ5FoAAADYg93uASJJcXFxiomJUWhoqMLDw7V27Vrl5OQoNjZW0uVvAk+fPq0NGy7f7Tc2NlbLly9XXFycJk+erIyMDK1bt87iaQKvvvqq5syZo40bN6pVq1bmbx+9vb3l7e1dofMCQF1T2XycnJyscePGaenSpQoLCzPn2vr168vX11eS9MQTT+iee+7RggULNHz4cG3btk27du3S/v377TNJAAAA1Gl2LYBER0fr7Nmzmj9/vnJzc9W5c2elpaUpMDBQkpSbm6ucnBxz/6CgIKWlpWnmzJlasWKFmjVrpmXLlmnkyJHmPitXrlRJSYlGjRplca4XXnhBc+fOrdB5AaCuqWw+XrNmjS5evKhp06Zp2rRp5vbx48crKSlJkhQREaFNmzZp9uzZmjNnjtq0aaOUlBT16tWrRucGAAAASHYugEjS1KlTNXXq1DJfu/JH9NX69Omjr776qtzxTp48edPnBYC6qDL5ePfu3RUac9SoUVYFaQAAAMAe7P4UGAAAAAAAAFujAAIAAAAAAJweBRAAAAAAAOD0KIAAAAAAAACnRwEEAAAAAAA4PQogAAAAAADA6dn9MbgAAACo3QzDUFFRkXnfy8tLJpPJjhEBAFB5FSqALFu2rMIDTp8+vcrBAEBV8Ic5ANhWUVGRhg8fbt7ftm2bvL297RgRAACVV6ECyJIlSyo0mMlkogACoMbxhzkAAACAG6lQAeTEiRO2jgNAHRfy9IYqH2u6WCLfq/b7ztkkw9W90uNkLhxX5RgAwBFUNddWV56VyLUAAPvhJqgAAAAAAMDpVakA8t1332nlypV67rnnFBcXZ7EBAAAAKNvKlSsVFBQkT09PhYSEaN++feX2zc3N1YMPPqj27durXr16mjFjhlWfpKQkmUwmq+38+fM2nAUAWzAMQ+fOnTNvhmHYOySnU+mnwHzyySe6//77FRQUpKNHj6pz5846efKkDMNQjx49bBEjAFyX4eKmgq5jLfYBANWHPFs9UlJSNGPGDK1cuVK9e/fWmjVrNHjwYH3zzTdq2bKlVf/i4mI1adJEs2bNuu49+Xx8fHT06FGLNk9Pz2qPH4BtcV8726t0ASQ+Pl5PPvmk5s+frwYNGig1NVW33XabHnroIQ0aNMgWMQLA9ZlMVb4WHQBQAeTZarF48WJNnDhRkyZNkiQlJiZqx44dWrVqlRISEqz6t2rVSkuXLpUkvfnmm+WOazKZFBAQYJugAVRKbbivncT9lspT6Utgjhw5ovHjx0uSXF1d9Z///Efe3t6aP3++FixYUO0BAgAAAI6upKREmZmZioqKsmiPiorSgQMHbmrsc+fOKTAwUM2bN9fQoUOVlZV1U+MBgLOqdAHEy8tLxcXFkqRmzZrp+PHj5tfy8/OrLzIAAADASeTn56u0tFT+/v4W7f7+/srLy6vyuB06dFBSUpK2b9+u5ORkeXp6qnfv3jp27FiZ/YuLi1VYWGixAUBdUelLYMLCwvTZZ58pODhY9913n5588kl9/fXX2rJli8LCwmwRIwAAAOAUTCaTxb5hGFZtlREWFmbxN3jv3r3Vo0cPvf7661q2bJlV/4SEBM2bN6/K5wNgO9xvyfYqXQBZvHixzp07J0maO3euzp07p5SUFLVt2/a6N2cCAAAA6qrGjRvLxcXFarXHmTNnrFaF3Ix69eqpZ8+e5a4AiY+Pt3hyY2FhoVq0aFFt5wdwE7jfks1VugDy4osv6uGHH5ZhGLrlllu0cuVKW8QFAAAAOA13d3eFhIQoPT1dv/3tb83t6enpFk99uFmGYSg7O1tdunQp83UPDw95eHhU2/kAwJFUugBy9uxZ3XffffLz89OYMWMUExOjbt262SA0AAAAwHnExcUpJiZGoaGhCg8P19q1a5WTk6PY2FhJl1dnnD59Whs2/PcpEtnZ2ZIu3+j0hx9+UHZ2ttzd3RUcHCxJmjdvnsLCwtSuXTsVFhZq2bJlys7O1ooVK2p8fgBQ21X6Jqjbt29XXl6eXnjhBWVmZiokJETBwcF6+eWXdfLkyUoHsHLlSgUFBcnT01MhISHat2/fdfvv2bNHISEh8vT0VOvWrbV69WqL1//+979r5MiRatWqlUwmkxITE63GmDt3rkwmk8XGo8MA1HWVyce5ubl68MEH1b59e9WrV08zZsyw6pOUlGSVa00mk86fP2/DWQBA7RUdHa3ExETNnz9f3bp10969e5WWlqbAwEBJl3NrTk6OxTHdu3dX9+7dlZmZqY0bN6p79+4aMmSI+fWff/5ZU6ZMUceOHRUVFaXTp09r7969uuuuu2p0bgDgCCpdAJGkW2+9VVOmTNHu3bv1r3/9S4888ojefvtttW3btlLjpKSkaMaMGZo1a5aysrIUGRmpwYMHWyX+K06cOKEhQ4YoMjJSWVlZev755zV9+nSlpqaa+/z6669q3bq1XnnllesWNTp16qTc3Fzz9vXXX1cqdgBwJpXNx8XFxWrSpIlmzZqlO++8s9xxfXx8LHJtbm6uPD09bTUNAKj1pk6dqpMnT6q4uFiZmZm65557zK8lJSVp9+7dFv0Nw7Darv7SccmSJfrXv/6l4uJinTlzRjt27FB4eHgNzQYAHEulL4G52oULF3To0CEdPHhQJ0+erPQNnBYvXqyJEydq0qRJkqTExETt2LFDq1atUkJCglX/1atXq2XLluZVHR07dtShQ4e0aNEijRw5UpLUs2dP9ezZU5L03HPPlXtuV1dXVn0AwP+pbD5u1aqVli5dKkl68803yx2XFXYAAACoLaq0AuTTTz/V5MmT5e/vr/Hjx6tBgwb64IMPdOrUqQqPUVJSoszMTEVFRVm0R0VF6cCBA2Uek5GRYdV/4MCBOnTokC5cuFCpORw7dkzNmjVTUFCQxowZo3/+85+VOh4AnEVV8nFFnTt3ToGBgWrevLmGDh2qrKysmxoPAAAAqKpKrwBp3ry5zp49q4EDB2rNmjUaNmxYlZYz5+fnq7S01GrViL+/v9Xjwa7Iy8srs//FixeVn5+vpk2bVujcvXr10oYNG3THHXfo+++/10svvaSIiAj9/e9/l5+fX5nHFBcXq7i42LxfWFhYoXMBQG1XlXxcER06dFBSUpK6dOmiwsJCLV26VL1799bhw4fVrl27Mo8h1wIAAMBWKl0A+eMf/6jf/e53atiwYbUEYDKZLPYNw7Bqu1H/stqvZ/DgweZ/d+nSReHh4WrTpo3Wr19v8Vz0qyUkJGjevHkVPgcAOJrK5uMbCQsLU1hYmHm/d+/e6tGjh15//XUtW7aszGPItQAAALCVSl8CM2XKlGopfjRu3FguLi5W3y6eOXOm3HuJBAQElNnf1dW13JUbFeHl5aUuXbro2LFj5faJj49XQUGBeavM5T4AUJtVJR9XRb169dSzZ09yLQAAAOyiSvcAqQ7u7u4KCQlRenq6RXt6eroiIiLKPCY8PNyq/86dOxUaGio3N7cqx1JcXKwjR45c9xIaDw8P+fj4WGwA4Ayqko+rwjAMZWdnk2sBAABgFzf1FJibFRcXp5iYGIWGhio8PFxr165VTk6OYmNjJV3+JvD06dPasGGDJCk2NlbLly9XXFycJk+erIyMDK1bt07JycnmMUtKSvTNN9+Y/3369GllZ2fL29vb/Jjep556SsOGDVPLli115swZvfTSSyosLNT48eNr+B0AgNqhsvlYkrKzsyVdvtHpDz/8oOzsbLm7uys4OFiSNG/ePIWFhaldu3YqLCzUsmXLlJ2drRUrVtT4/AAAAAC7FkCio6N19uxZzZ8/X7m5uercubPS0tIUGBgoScrNzVVOTo65f1BQkNLS0jRz5kytWLFCzZo107Jly8yPwJWkf//73+revbt5f9GiRVq0aJH69Oljfq76d999p7Fjxyo/P19NmjRRWFiYPv/8c/N5AaCuqWw+lmSRazMzM7Vx40YFBgbq5MmTkqSff/5ZU6ZMUV5ennx9fdW9e3ft3btXd911V43NCwAAALjCrgUQSZo6daqmTp1a5mtJSUlWbX369NFXX31V7nitWrUy3xi1PJs2bapUjABQF1Q2H98o1y5ZskRLliypjtAAAACAm2a3e4AAAAAAAADUFAogAAAAAADA6VEAAQAAAAAATo8CCAAAAAAAcHoUQAAAAAAAgNOjAAIAAAAAAJweBRAAAAAAAOD0KIAAAAAAAACnRwEEAAAAAAA4PQogAAAAAADA6VEAAQAAAAAATo8CCAAAAAAAcHoUQAAAAAAAgNOjAAIAAAAAAJweBRAAAACghqxcuVJBQUHy9PRUSEiI9u3bV27f3NxcPfjgg2rfvr3q1aunGTNmlNkvNTVVwcHB8vDwUHBwsLZu3Wqj6AHAsVEAAQAAAGpASkqKZsyYoVmzZikrK0uRkZEaPHiwcnJyyuxfXFysJk2aaNasWbrzzjvL7JORkaHo6GjFxMTo8OHDiomJ0ejRo3Xw4EFbTgUAHBIFEAAAAKAGLF68WBMnTtSkSZPUsWNHJSYmqkWLFlq1alWZ/Vu1aqWlS5dq3Lhx8vX1LbNPYmKiBgwYoPj4eHXo0EHx8fG69957lZiYaMOZAIBjogACAAAA2FhJSYkyMzMVFRVl0R4VFaUDBw5UedyMjAyrMQcOHHhTYwKAs3K1dwAAAACAs8vPz1dpaan8/f0t2v39/ZWXl1flcfPy8io1ZnFxsYqLi837hYWFVT43ADgaVoAAAAAANcRkMlnsG4Zh1WbLMRMSEuTr62veWrRocVPnBgBHQgEEAAAAsLHGjRvLxcXFamXGmTNnrFZwVEZAQEClxoyPj1dBQYF5O3XqVJXPDQCOxu4FkMo8CkyS9uzZo5CQEHl6eqp169ZavXq1xet///vfNXLkSLVq1Uomk6ncG0BV9rwA4Ox4NCMA2I67u7tCQkKUnp5u0Z6enq6IiIgqjxseHm415s6dO8sd08PDQz4+PhYbANQVdi2AVPZRYCdOnNCQIUMUGRmprKwsPf/885o+fbpSU1PNfX799Ve1bt1ar7zyigICAqrlvADg7Hg0IwDYXlxcnP785z/rzTff1JEjRzRz5kzl5OQoNjZW0uXVGePGjbM4Jjs7W9nZ2Tp37px++OEHZWdn65tvvjG//sQTT2jnzp1asGCB/vd//1cLFizQrl27yi1MA0BdZtcCSGUfBbZ69Wq1bNlSiYmJ6tixoyZNmqTf//73WrRokblPz549tXDhQo0ZM0YeHh7Vcl4AcHY8mhEAbC86OlqJiYmaP3++unXrpr179yotLU2BgYGSLq+uu7bw3L17d3Xv3l2ZmZnauHGjunfvriFDhphfj4iI0KZNm/TWW2+pa9euSkpKUkpKinr16lWjcwMAR2C3p8BceRTYc889Z9F+vUeBlfeYr3Xr1unChQtyc3OzyXkl7pgNwHlVNS/eSEZGhmbOnGnRNnDgwOsWQMi1AJzd1KlTNXXq1DJfS0pKsmozDOOGY44aNUqjRo262dAAwOnZbQVIVR4FVt5jvi5evKj8/HybnVfijtkAnFdteTSjRK4FAACA7dj9JqiVfRRYWf3Laq/u83LHbADOzt6PZpTItQAAALAdu10CU5VHgZX3mC9XV1f5+fnZ7LzS5Ttml3dPEQBwZLXl0YwSuRYAAAC2Y7cVIFV5FFh5j/kKDQ2t0P0/qnpeAHBmteXRjAAAAIAt2W0FiHT5UWAxMTEKDQ1VeHi41q5da/UosNOnT2vDhg2SpNjYWC1fvlxxcXGaPHmyMjIytG7dOiUnJ5vHLCkpMT8arKSkRKdPn1Z2dra8vb3Vtm3bCp0XAOqayuZj6fKjGSVZPJrR3d1dwcHBki4/mvGee+7RggULNHz4cG3btk27du3S/v37a3x+AAAAgF0LINHR0Tp79qzmz5+v3Nxcde7c+bqPAgsKClJaWppmzpypFStWqFmzZlq2bJlGjhxp7vPvf/9b3bt3N+8vWrRIixYtUp8+fbR79+4KnRcA6prK5mNJFrn2yuMZAwMDdfLkSUn/fTTj7NmzNWfOHLVp04ZHMwIAAMBu7FoAkSr/KLA+ffroq6++Kne8Vq1aVehxYdc7LwDURTyaEQAAAM7M7k+BAQAAAAAAsDUKIAAAAAAAwOlRAAEAAAAAAE6PAggAAAAAAHB6FEAAAAAAAIDTowACAAAAAACcHgUQAAAAAADg9CiAAAAAAAAAp0cBBAAAAAAAOD1XewcAAAD+yzAMFRUVmfe9vLxkMpnsGBEAAIBzoAACAEAtUlRUpOHDh5v3t23bJm9vbztGBAAA4By4BAYAAAAAADg9VoAAAGADIU9vqNJxposl8r1qv++cTTJc3as0VubCcVU6DgAAwBmxAgQAAAAAADg9CiAAAAAAAMDpcQkMAAC1iOHipoKuYy32AQAAcPMogAAAUJuYTFW+5wcAAADKxyUwAAAAQA1ZuXKlgoKC5OnpqZCQEO3bt++6/ffs2aOQkBB5enqqdevWWr16tcXrSUlJMplMVtv58+dtOQ0AcEgUQAAAAIAakJKSohkzZmjWrFnKyspSZGSkBg8erJycnDL7nzhxQkOGDFFkZKSysrL0/PPPa/r06UpNTbXo5+Pjo9zcXIvN09OzJqYEAA6FS2AAAACAGrB48WJNnDhRkyZNkiQlJiZqx44dWrVqlRISEqz6r169Wi1btlRiYqIkqWPHjjp06JAWLVqkkSNHmvuZTCYFBATUyBwAwJHZfQVIdS8DlKTU1FQFBwfLw8NDwcHB2rp1q8Xrc+fOtVomyIcGgLqOZdkAYDslJSXKzMxUVFSURXtUVJQOHDhQ5jEZGRlW/QcOHKhDhw7pwoUL5rZz584pMDBQzZs319ChQ5WVlVVuHMXFxSosLLTYAKCusGsBxBbLADMyMhQdHa2YmBgdPnxYMTExGj16tA4ePGgxVqdOnSyWCX799dc2nSsA1GYsywYA28rPz1dpaan8/f0t2v39/ZWXl1fmMXl5eWX2v3jxovLz8yVJHTp0UFJSkrZv367k5GR5enqqd+/eOnbsWJljJiQkyNfX17y1aNGiGmYHAI7BrgWQq5cBduzYUYmJiWrRooVWrVpVZv+rlwF27NhRkyZN0u9//3stWrTI3CcxMVEDBgxQfHy8OnTooPj4eN17773mpYNXuLq6KiAgwLw1adLEllMFgFrNFvlY+u+y7Ks3ezIMQ+fOnTNvhmHYNR4AdY/JZLLYNwzDqu1G/a9uDwsL08MPP6w777xTkZGR2rx5s+644w69/vrrZY4XHx+vgoIC83bq1KmbmQ4AOBS7FUBstQywvD7Xjnns2DE1a9ZMQUFBGjNmjP75z39eN16WCwJwVrVlWbZk+1xbVFSk4cOHm7eioqJqHR8AytO4cWO5uLhYrfY4c+aM1SqPKwICAsrs7+rqKj8/vzKPqVevnnr27FnuChAPDw/5+PhYbHURBXGgbrLbTVBtsQywadOm5fa5esxevXppw4YNuuOOO/T999/rpZdeUkREhP7+97+X+2GSkJCgefPmVWWqAFCr2SofX1mW3aVLFxUWFmrp0qXq3bu3Dh8+rHbt2pU5bkVybcjTGyoxO0umiyXyvWq/75xNMlzdKz1O5sJxVY4BQN3k7u6ukJAQpaen67e//a25PT09XcOHDy/zmPDwcH3wwQcWbTt37lRoaKjc3NzKPMYwDGVnZ6tLly7VF7wTulIQv2Lbtm3y9va2Y0QAaoLdb4Ja3csAKzLm4MGDNXLkSHXp0kX9+/fXX//6V0nS+vXryz0vywUBODt7L8uWyLUAnFtcXJz+/Oc/680339SRI0c0c+ZM5eTkKDY2VtLlHDhu3H8LrLGxsfrXv/6luLg4HTlyRG+++abWrVunp556ytxn3rx52rFjh/75z38qOztbEydOVHZ2tnlMAMB/2W0FiK2WAZbXp7wxJcnLy0tdunQpd6mgdHm5oIeHx3Xn5Chu5tvT6sK3p0DtUVuWZUvOlWsB4FrR0dE6e/as5s+fr9zcXHXu3FlpaWkKDAyUJOXm5lrcfDooKEhpaWmaOXOmVqxYoWbNmmnZsmUWj8D9+eefNWXKFOXl5cnX11fdu3fX3r17ddddd9X4/GoaKwIBVJbdCiC2WgYYHh6u9PR0zZw506JPREREubEUFxfryJEjioyMvJkp1YrCgkQiBlA5dWlZtuHipoKuYy32AaAmTZ06VVOnTi3ztaSkJKu2Pn366Kuvvip3vCVLlmjJkiXVFR4AODW7FUCky8sAY2JiFBoaqvDwcK1du9ZqGeDp06e1YcPlwkJsbKyWL1+uuLg4TZ48WRkZGVq3bp2Sk5PNYz7xxBO65557tGDBAg0fPlzbtm3Trl27tH//fnOfp556SsOGDVPLli115swZvfTSSyosLNT48eNr9g0AgFrCFvl43rx5CgsLU7t27VRYWKhly5YpOztbK1assMscJUkmU5W+4QMAOBcK4kDdZNcCiC2WAUZERGjTpk2aPXu25syZozZt2iglJUW9evUy9/nuu+80duxY5efnq0mTJgoLC9Pnn39uPi8A1DUsywYAXK02rGy26apmCuJAnWTXAohU/csAJWnUqFEaNWpUua9v2rSpUjECQF3AsmwAAAA4M7s/BQYAAAAAAMDWKIAAAAAAAACnRwEEAAAAAAA4PQogAAAAAADA6VEAAQAAAAAATo8CCAAAAAAAcHoUQAAAAAAAgNOjAAIAAAAAAJweBRAAAAAAAOD0KIAAAAAAAACn52rvAACgLjAMQ0VFReZ9Ly8vmUwmO0YEAAAA1C0UQICbEPL0BnuHoMyF4+wdAiqgqKhIw4cPN+9v27ZN3t7edowIAAAAqFu4BAYAAAAAADg9VoAAQCVUddWP6WKJfK/a7ztnkwxX9yqNxaofAAAAoPJYAQIAAAAAAJweK0AAoAYYLm4q6DrWYh8AAABAzaEAAgA1wWSq8iUvAAAAtsSN/VFXcAkMAAAAAABweqwAAQAAAACgDqmrq35YAQIAAAAAAJye3VeArFy5UgsXLlRubq46deqkxMRERUZGltt/z549iouL09///nc1a9ZMzzzzjGJjYy36pKamas6cOTp+/LjatGmjP/3pT/rtb397U+cFAGdnr3wM3Ehd/ZYKzolcCwD2Y9cVICkpKZoxY4ZmzZqlrKwsRUZGavDgwcrJySmz/4kTJzRkyBBFRkYqKytLzz//vKZPn67U1FRzn4yMDEVHRysmJkaHDx9WTEyMRo8erYMHD1b5vADg7OyVjwGgLiHXAoB92XUFyOLFizVx4kRNmjRJkpSYmKgdO3Zo1apVSkhIsOq/evVqtWzZUomJiZKkjh076tChQ1q0aJFGjhxpHmPAgAGKj4+XJMXHx2vPnj1KTExUcnJylc4LAM7OXvkYAK7lzCt+yLUAYF92WwFSUlKizMxMRUVFWbRHRUXpwIEDZR6TkZFh1X/gwIE6dOiQLly4cN0+V8asynkBwJnZKx8DQF1CrgUA+7PbCpD8/HyVlpbK39/fot3f3195eXllHpOXl1dm/4sXLyo/P19NmzYtt8+VMatyXkkqLi5WcXGxeb+goECSVFhYaG4rLf5PucfXpKtjKkttiNMRYpQcI05HiFFyjDhvFKNUe+Ns0KCBTCZTlcazVz4ui6PkWkf+XbmaI8QoOUacjhCj5Bhx1uYYnSHXViTPSrX753CFI8QoOUacjhCj5Bhx1oYYJceIs7r/pq0Iu98E9drJGYZx3QmX1f/a9oqMWdnzJiQkaN68eVbtLVq0KPcYe/F9PfbGnezMEWKUHCNOR4hRcow4HSFGqew4CwoK5OPjc1Pj2isfX81Rcq0j/67UNo4Qo+QYcTpCjJJjxFlejM6Qax0lz0qO/btS2zhCnI4Qo0Sc1clWf9Nej90KII0bN5aLi4tVdfrMmTNWVewrAgICyuzv6uoqPz+/6/a5MmZVzitdvp4yLi7OvH/p0iX9+OOP8vPzq7YKVWFhoVq0aKFTp07Z9Id+MxwhRok4q5MjxCg5Rpy2jLFBgwZVPtZe+bgs5NrLHCFGyTHidIQYJceI0xFilMi1NxqTPPtfjhCnI8QoOUacjhCj5Bhx1tY8WxF2K4C4u7srJCRE6enpFo/pSk9P1/Dhw8s8Jjw8XB988IFF286dOxUaGio3Nzdzn/T0dM2cOdOiT0RERJXPK0keHh7y8PCwaLv11lsrNtlK8vHxqbW/7Fc4QowScVYnR4hRcow4a1uM9srHZSHXWnKEGCXHiNMRYpQcI05HiFGqfXHWllxLnrXmCHE6QoySY8TpCDFKjhGnI8RoxbCjTZs2GW5ubsa6deuMb775xpgxY4bh5eVlnDx50jAMw3juueeMmJgYc/9//vOfxi233GLMnDnT+Oabb4x169YZbm5uxnvvvWfu89lnnxkuLi7GK6+8Yhw5csR45ZVXDFdXV+Pzzz+v8HntpaCgwJBkFBQU2DWO63GEGA2DOKuTI8RoGI4RZ22O0V752B5q88/hCkeI0TAcI05HiNEwHCNOR4jRMGp3nHUl19bmn8HVHCFOR4jRMBwjTkeI0TAcI05HiLE8di2AGIZhrFixwggMDDTc3d2NHj16GHv27DG/Nn78eKNPnz4W/Xfv3m10797dcHd3N1q1amWsWrXKasx3333XaN++veHm5mZ06NDBSE1NrdR57cURfpEcIUbDIM7q5AgxGoZjxFnbY7RXPq5ptf3nYBiOEaNhOEacjhCjYThGnI4Qo2HU/jjrQq6t7T+DKxwhTkeI0TAcI05HiNEwHCNOR4ixPHYvgOC/zp8/b7zwwgvG+fPn7R1KuRwhRsMgzurkCDEahmPE6Qgx1gWO8HNwhBgNwzHidIQYDcMx4nSEGA3DceJ0Zo7yM3CEOB0hRsNwjDgdIUbDcIw4HSHG8pgM4/9uJQ0AAAAAAOCk6tk7AAAAAAAAAFujAAIAAAAAAJweBRBc18mTJ2UymZSdnW3vUADAaZFrAcD2yLUAKIDUkAMHDsjFxUWDBg2yaL9RIi4tLVVCQoI6dOig+vXrq1GjRgoLC9Nbb71VLXFNmDBBJpPJvPn5+WnQoEH629/+Jklq0aKFcnNz1blz52o5X1VjHDFihEVbee+nZP/3tCxXz+Ha+Vz7M7iylTW3mnCjWK/9Wdg6livvh5ubm1q3bq2nnnpKRUVFNRbD9fTt21czZsywan///fdlMpkkSXPnzlW3bt2s+vz8888ymUzavXu3pP/+3pa1ff755zachXMh195cjI6cax0pz14bI7m2fOTZ2qe25lmJXEuutUaurZi6lGtd7R1AXfHmm2/q8ccf15///Gfl5OSoZcuWFTpu7ty5Wrt2rZYvX67Q0FAVFhbq0KFD+umnn6ottkGDBpmTZF5enmbPnq2hQ4cqJydHLi4uCggIqLZzVZeqvp9SzbynlXX1z+AKDw8PO0VTu1x5by5cuKB9+/Zp0qRJKioq0qpVq+wdmk3s2rVLnTp1smjz8/OzUzSOh1xbvZwp15Jnr68u5Vry7M2pzXlWIteSa2s3cq39cy0rQGpAUVGRNm/erD/84Q8aOnSokpKSKnzsBx98oKlTp+p3v/udgoKCdOedd2rixImKi4urtvg8PDwUEBCggIAAdevWTc8++6xOnTqlH374warqXFpaqokTJyooKEj169dX+/bttXTpUovxdu/erbvuukteXl669dZb1bt3b/3rX/+qtnhv5v2UauY9rayrfwZXtoYNG9otntrkynvTokULPfjgg3rooYf0/vvvq7i4WNOnT9dtt90mT09P3X333fryyy/Nx+3evVsmk0l//etfdeedd8rT01O9evXS119/bcfZ3Jifn5/V74Kbm5u9w3II5Fpy7fWQZ6+vLuVa8mzV1fY8K5FrybW1G7nW/rmWAkgNSElJUfv27dW+fXs9/PDDeuutt1TRpw8HBATof/7nf/TDDz/YOMrLzp07p3feeUdt27Yts0J36dIlNW/eXJs3b9Y333yjP/7xj3r++ee1efNmSdLFixc1YsQI9enTR3/729+UkZGhKVOmmJdOVYebeT+lmn9PUb3q16+vCxcu6JlnnlFqaqrWr1+vr776Sm3bttXAgQP1448/WvR/+umntWjRIn355Ze67bbbdP/99+vChQt2ih62RK4l16L6kGtRFkfKsxK5FrUfubbmUQCpAevWrdPDDz8s6fKyp3PnzumTTz6p0LGLFy/WDz/8oICAAHXt2lWxsbH66KOPqjW+Dz/8UN7e3vL29laDBg20fft2paSkqF49618PNzc3zZs3Tz179lRQUJAeeughTZgwwfxBUVhYqIKCAg0dOlRt2rRRx44dNX78+Eot5buRm3k/pZp5Tyvr6p/Ble3FF1+0a0y10RdffKGNGzeqX79+WrVqlRYuXKjBgwcrODhYb7zxhurXr69169ZZHPPCCy9owIAB6tKli9avX6/vv/9eW7dutdMMbiwiIsLqd6G0tNTeYTkEci259nrIsxXn7LmWPFt1tT3PSuRacq3jINfaBwUQGzt69Ki++OILjRkzRpLk6uqq6OhovfnmmxU6Pjg4WP/v//0/ff7553rkkUf0/fffa9iwYZo0aVK1xdivXz9lZ2crOztbBw8eVFRUlAYPHlzu8r7Vq1crNDRUTZo0kbe3t9544w3l5ORIkho1aqQJEyZo4MCBGjZsmJYuXarc3Nxqi/Vm30+pZt7Tyrr6Z3BlmzZtmt3iqU2ufJB6enoqPDxc99xzjx5//HFduHBBvXv3Nvdzc3PTXXfdpSNHjlgcHx4ebv53o0aN1L59e6s+tUlKSorV74KLi4u9w6r1yLXk2hshz15fXcq15NmqcYQ8K5FrybW1G7nW/rmWm6Da2Lp163Tx4kXdfvvt5jbDMOTm5lbhmxPVq1dPPXv2VM+ePTVz5kz95S9/UUxMjGbNmqWgoKCbjtHLy0tt27Y174eEhMjX11dvvPGGVfLcvHmzZs6cqddee03h4eFq0KCBFi5cqIMHD5r7vPXWW5o+fbo+/vhjpaSkaPbs2UpPT1dYWNhNx3qj97Oi1xja+j2trGt/BvivK1VxNzc3NWvWTG5ubjp8+LAkWS1BNQyjQstSq2vpqo+PjwoKCqzaf/75Z/n4+NywjyT5+vpatLdo0YLfhSog15Jrb4Q8e321NdeSZ2sPR8izErmWXFu7kWvtjxUgNnTx4kVt2LBBr732mkXl6/DhwwoMDNQ777xTpXGDg4MlyWaPTDKZTKpXr57+85//WL22b98+RUREaOrUqerevbvatm2r48ePW/Xr3r274uPjdeDAAXXu3FkbN2686bhs9X5Ktn9PUXVXPkgDAwPNN05q27at3N3dtX//fnO/Cxcu6NChQ+rYsaPF8Vc/buunn37St99+qw4dOlRLbB06dNChQ4es2r/88ku1b9/e3Oe7775TXl6eVZ969erVyg8GR0OuJdfi5tXWXEuerR0cNc9K5FrULuRa+2MFiA19+OGH+umnnzRx4kSritioUaO0bt06DR06VNLlJXDXCg4O1oMPPqjevXsrIiJCAQEBOnHihOLj43XHHXdU2//EFRcXm3+Rf/rpJy1fvlznzp3TsGHDrPq2bdtWGzZs0I4dOxQUFKS3335bX375pbm6fOLECa1du1b333+/mjVrpqNHj+rbb7/VuHHjbjrOiryfjz32mLnNnu9pZV39M7jC1dVVjRs3tks8tZ2Xl5f+8Ic/6Omnn1ajRo3UsmVLvfrqq/r11181ceJEi77z58+Xn5+f/P39NWvWLDVu3Ljanvk+depULV++XNOmTdOUKVNUv359paena926dXr77bclSVFRUerYsaPGjBmjP/3pT2rWrJn+9re/6amnnlJsbKwaNGhgMebZs2etfhduvfVWeXp6VkvMzohcS66tCPJs5dWGXEuerR0cJc9K5FpyreMh19YwAzYzdOhQY8iQIWW+lpmZaUgy/7es7cSJE8batWuNfv36GU2aNDHc3d2Nli1bGhMmTDBOnjxZLTGOHz/e4pwNGjQwevbsabz33nuGYRjGiRMnDElGVlaWYRiGcf78eWPChAmGr6+vceuttxp/+MMfjOeee8648847DcMwjLy8PGPEiBFG06ZNDXd3dyMwMND44x//aJSWlt5UjMOHD6/w+3klZnu9p9ebw7X/vrJfVqzt27e3WTw3E+vV+zUZy7X+85//GI8//rjRuHFjw8PDw+jdu7fxxRdfmF//9NNPDUnGBx98YHTq1Mlwd3c3evbsaWRnZ1drjIcOHTIGDhxo3HbbbYaPj48RGhpqJCcnW/TJzc01HnnkESMwMNCoX7++0aFDB2P+/PnG+fPnzX2u93t77XiwRK4l114d/7X/vrJfm/LstTGSa6+PPGt/jpBnDYNcS661Rq6tuLqSa02GUYnnLAGAg9i9e7f69eunn376Sbfeequ9wwEAp0SuBQDbI9dWH+4BAgAAAAAAnB4FEAAAAAAA4PS4BAYAAAAAADg9VoAAAAAAAACnRwEEAAAAAAA4PQogAAAAAADA6VEAAQAAAAAATo8CCAAAAAAAcHoUQIDrOHnypEwmk7Kzs2vNufr27asZM2bYPB4AqAnkWQCwPXItcBkFEKCWaNGihXJzc9W5c2dJ0u7du2UymfTzzz/bNzAAcBLkWQCwPXItajNXewcAQCopKZG7u7sCAgLsHQoAOCXyLADYHrkWtR0rQFDnffzxx7r77rt16623ys/PT0OHDtXx48fL7b99+3a1a9dO9evXV79+/bR+/XqrqnZqaqo6deokDw8PtWrVSq+99prFGK1atdJLL72kCRMmyNfXV5MnT7ZYLnjy5En169dPktSwYUOZTCZNmDDBfPylS5f0zDPPqFGjRgoICNDcuXMtxjeZTFqzZo2GDh2qW265RR07dlRGRob+8Y9/qG/fvvLy8lJ4ePh15wkA1YU8CwC2R64FKsAA6rj33nvPSE1NNb799lsjKyvLGDZsmNGlSxejtLTUOHHihCHJyMrKMgzDME6cOGG4ubkZTz31lPG///u/RnJysnH77bcbkoyffvrJMAzDOHTokFGvXj1j/vz5xtGjR4233nrLqF+/vvHWW2+ZzxkYGGj4+PgYCxcuNI4dO2YcO3bM4lwXL140UlNTDUnG0aNHjdzcXOPnn382DMMw+vTpY/j4+Bhz5841vv32W2P9+vWGyWQydu7caR5fknH77bcbKSkpxtGjR40RI0YYrVq1Mn7zm98YH3/8sfHNN98YYWFhxqBBg2rqbQZQh5FnAcD2yLXAjVEAAa5x5swZQ5Lx9ddfW31YPPvss0bnzp0t+s+aNcviw+LBBx80BgwYYNHn6aefNoKDg837gYGBxogRIyz6XHuuTz/91GLcK/r06WPcfffdFm09e/Y0nn32WfO+JGP27Nnm/YyMDEOSsW7dOnNbcnKy4enpeeM3BACqGXkWAGyPXAtY4xIY1HnHjx/Xgw8+qNatW8vHx0dBQUGSpJycHKu+R48eVc+ePS3a7rrrLov9I0eOqHfv3hZtvXv31rFjx1RaWmpuCw0NrXLMXbt2tdhv2rSpzpw5U24ff39/SVKXLl0s2s6fP6/CwsIqxwEAFUGeJc8CsD1yLbkWN8ZNUFHnDRs2TC1atNAbb7yhZs2a6dKlS+rcubNKSkqs+hqGIZPJZNVW2T6S5OXlVeWY3dzcLPZNJpMuXbpUbp8r8ZTVdu1xAFDdyLPkWQC2R64l1+LGKICgTjt79qyOHDmiNWvWKDIyUpK0f//+cvt36NBBaWlpFm2HDh2y2A8ODrYa48CBA7rjjjvk4uJS4djc3d0lyaLCDgCOhjwLALZHrgUqhktgUKc1bNhQfn5+Wrt2rf7xj3/of/7nfxQXF1du/0cffVT/+7//q2effVbffvutNm/erKSkJEn/rT4/+eST+uSTT/Tiiy/q22+/1fr167V8+XI99dRTlYotMDBQJpNJH374oX744QedO3euyvMEAHshzwKA7ZFrgYqhAII6rV69etq0aZMyMzPVuXNnzZw5UwsXLiy3f1BQkN577z1t2bJFXbt21apVqzRr1ixJkoeHhySpR48e2rx5szZt2qTOnTvrj3/8o+bPn2/xyK+KuP322zVv3jw999xz8vf312OPPVbleQKAvZBnAcD2yLVAxZiMsi7kAlBhf/rTn7R69WqdOnXK3qEAgFMizwKA7ZFrURdwDxCgklauXKmePXvKz89Pn332mRYuXEglGwCqEXkWAGyPXIu6iAIIUEnHjh3TSy+9pB9//FEtW7bUk08+qfj4eHuHBQBOgzwLALZHrkVdxCUwAAAAAADA6XETVAAAAAAA4PQogAAAAAAAAKdHAQQAAAAAADg9CiAAAAAAAMDpUQABAAAAAABOjwIIAAAAAABwehRAAAAAAACA06MAAgAAAAAAnB4FEAAAAAAA4PT+P67M6+vJvxiNAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1091.11x300 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = sns.catplot(x = \"algorithm\", y = \"val\", data = pltData, kind=\"bar\", col = \"metric\", aspect=1.2, height=3, sharey=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e32b381",
   "metadata": {
    "papermill": {
     "duration": 0.006186,
     "end_time": "2024-12-08T00:46:06.675326",
     "exception": false,
     "start_time": "2024-12-08T00:46:06.669140",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Prediction RMSE\n",
    "\n",
    "We will also look at the prediction RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "fb282b1c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:46:06.688517Z",
     "iopub.status.busy": "2024-12-08T00:46:06.688244Z",
     "iopub.status.idle": "2024-12-08T00:48:49.996229Z",
     "shell.execute_reply": "2024-12-08T00:48:49.995575Z"
    },
    "papermill": {
     "duration": 163.316001,
     "end_time": "2024-12-08T00:48:49.997338",
     "exception": false,
     "start_time": "2024-12-08T00:46:06.681337",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "preds = preds.rename(columns={'score': 'prediction'})\n",
    "user_rmse = preds.groupby(['algorithm', 'user']).apply(lambda df: rmse(df['prediction'], df['rating']))\n",
    "user_rmse = user_rmse.reset_index(name='RMSE')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "73116a6d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:48:50.012507Z",
     "iopub.status.busy": "2024-12-08T00:48:50.012240Z",
     "iopub.status.idle": "2024-12-08T00:48:53.106904Z",
     "shell.execute_reply": "2024-12-08T00:48:53.106484Z"
    },
    "papermill": {
     "duration": 3.102275,
     "end_time": "2024-12-08T00:48:53.107590",
     "exception": false,
     "start_time": "2024-12-08T00:48:50.005315",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x1555302d3fd0>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAHpCAYAAACFlZVCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAsPElEQVR4nO3df1RVdb7/8ddR8MBVwCQFLFS8GiqM6QUbqfyVE6aj6V3ee2uyFMu6XE0nGUZD+zqp4zDrai1zWZqFmjn+Wh0rS/PqHUWbxEkMq2uKNheDYTiZlZDcPAju7x8uz3jkh4DAhg/Px1p71edzPp+z33uv7Xqxf5xzHJZlWQIAAMZqY3cBAACgcRH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAM1+rC3rIslZSUiK8XAAC0Fq0u7H/44QeFhITohx9+sLsUAACaRKsLewAAWhvCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOD+7CwAANA3LslRaWuptt2/fXg6Hw8aK0FQIewBoJUpLSzV+/Hhv+91331WHDh1srAhNhcv4AAAYrtmEfXp6uhwOh5555pkaxx04cEBxcXEKCAhQz549tXr16qYpEACAFqpZXMY/cuSI1qxZo/79+9c4Li8vT2PGjNGTTz6pjRs36qOPPtL06dPVuXNnTZw4sYmqBYCGE/frDU22Lkd5mUKuaQ//f1tk+bVr9PUeXTq50deBmtl+Zn/hwgVNmjRJr732mm655ZYax65evVrdunXT8uXL1bdvX02bNk2PP/64li1b1kTVAgDQ8tge9jNmzNDPf/5z/exnP7vh2KysLCUmJvr0jRo1StnZ2bp06VKVczwej0pKSnwWAABaE1sv42/ZskWffPKJjhw5UqvxbrdbYWFhPn1hYWEqLy/XuXPnFBERUWlOenq6Fi5c2CD1AkBLZrX1V3H/X/i00TrYdmZfUFCgX/7yl9q4caMCAgJqPe/6z4RallVl/1VpaWkqLi72LgUFBfUvGgBaModDll877yI+Y99q2HZmf/ToUZ09e1ZxcXHevoqKCh08eFArV66Ux+NR27ZtfeaEh4fL7Xb79J09e1Z+fn4KDQ2tcj1Op1NOp7PhNwAAgBbCtrAfOXKkPv/8c5++qVOnqk+fPpo7d26loJekhIQEvffeez59e/bsUXx8vPz9uRwFAEBVbAv7oKAgxcbG+vS1b99eoaGh3v60tDQVFhZqw4YrH01JTk7WypUrlZKSoieffFJZWVnKyMjQ5s2bm7x+AABaCtufxq9JUVGR8vPzve2oqCjt2rVLmZmZGjBggBYvXqwVK1bwGXsAAGrgsK4+4dZKlJSUKCQkRMXFxQoODra7HACtXFN+qY5d+FId+zXrM3sAAHDzCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGszXsV61apf79+ys4OFjBwcFKSEjQBx98UO34zMxMORyOSsvJkyebsGoAAFoWPztXfvvtt+v3v/+9evXqJUl64403NH78eOXk5CgmJqbaebm5uQoODva2O3fu3Oi1AgDQUtka9uPGjfNpL1myRKtWrdLhw4drDPsuXbqoY8eOjVwdAABmaDb37CsqKrRlyxaVlpYqISGhxrEDBw5URESERo4cqf3799c41uPxqKSkxGcBAKA1sT3sP//8c3Xo0EFOp1PJycl6++231a9fvyrHRkREaM2aNXK5XNq+fbuio6M1cuRIHTx4sNr3T09PV0hIiHeJjIxsrE0BAKBZcliWZdlZQFlZmfLz83X+/Hm5XC69/vrrOnDgQLWBf71x48bJ4XBox44dVb7u8Xjk8Xi87ZKSEkVGRqq4uNjnvj8A2CHu1xvsLqHRHV062e4SWj1b79lLUrt27bwP6MXHx+vIkSN66aWX9Oqrr9Zq/uDBg7Vx48ZqX3c6nXI6nQ1SKwAALZHtl/GvZ1mWz5n4jeTk5CgiIqIRKwIAoGWz9cx+3rx5Gj16tCIjI/XDDz9oy5YtyszM1O7duyVJaWlpKiws1IYNVy5zLV++XD169FBMTIzKysq0ceNGuVwuuVwuOzcDAIBmzdaw//rrr/XYY4+pqKhIISEh6t+/v3bv3q37779fklRUVKT8/Hzv+LKyMqWmpqqwsFCBgYGKiYnRzp07NWbMGLs2AQCAZs/2B/SaWklJiUJCQnhAD0CzwAN6aArN7p49AABoWIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYztawX7Vqlfr376/g4GAFBwcrISFBH3zwQY1zDhw4oLi4OAUEBKhnz55avXp1E1ULAEDLZGvY33777fr973+v7OxsZWdn67777tP48eN1/PjxKsfn5eVpzJgxGjJkiHJycjRv3jzNmjVLLperiSsHAKDlcFiWZdldxLU6deqkpUuX6oknnqj02ty5c7Vjxw6dOHHC25ecnKxPP/1UWVlZVb6fx+ORx+PxtktKShQZGani4mIFBwc3/AYAQB3E/XqD3SU0uqNLJ9tdQqvXbO7ZV1RUaMuWLSotLVVCQkKVY7KyspSYmOjTN2rUKGVnZ+vSpUtVzklPT1dISIh3iYyMbPDaAQBozmwP+88//1wdOnSQ0+lUcnKy3n77bfXr16/KsW63W2FhYT59YWFhKi8v17lz56qck5aWpuLiYu9SUFDQ4NsAAEBz5md3AdHR0Tp27JjOnz8vl8ulKVOm6MCBA9UGvsPh8GlfvQtxff9VTqdTTqezYYsGAKAFsT3s27Vrp169ekmS4uPjdeTIEb300kt69dVXK40NDw+X2+326Tt79qz8/PwUGhraJPUC1eHeK4DmyvbL+NezLMvngbprJSQkaO/evT59e/bsUXx8vPz9/ZuiPAAAWhxbw37evHn68MMPdebMGX3++eeaP3++MjMzNWnSJElX7rdPnvz3M4nk5GR99dVXSklJ0YkTJ7R27VplZGQoNTXVrk0AAKDZs/Uy/tdff63HHntMRUVFCgkJUf/+/bV7927df//9kqSioiLl5+d7x0dFRWnXrl2aPXu2Xn75ZXXt2lUrVqzQxIkT7doEAACaPVvDPiMjo8bX169fX6lv2LBh+uSTTxqpIgAAzNPs7tkDAICGRdgDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwnK1hn56erkGDBikoKEhdunTRhAkTlJubW+OczMxMORyOSsvJkyebqGoAAFoWW8P+wIEDmjFjhg4fPqy9e/eqvLxciYmJKi0tveHc3NxcFRUVeZfevXs3QcUAALQ8fnaufPfu3T7tdevWqUuXLjp69KiGDh1a49wuXbqoY8eON1yHx+ORx+PxtktKSupVKwAALVWzumdfXFwsSerUqdMNxw4cOFAREREaOXKk9u/fX+249PR0hYSEeJfIyMgGqxcAgJag2YS9ZVlKSUnRvffeq9jY2GrHRUREaM2aNXK5XNq+fbuio6M1cuRIHTx4sMrxaWlpKi4u9i4FBQWNtQkAADRLtl7Gv9bTTz+tzz77TH/6059qHBcdHa3o6GhvOyEhQQUFBVq2bFmVl/6dTqecTmeD1wsAQEvRLM7sZ86cqR07dmj//v26/fbb6zx/8ODBOn36dCNUBgBAy2frmb1lWZo5c6befvttZWZmKioqql7vk5OTo4iIiAauDgAAM9ga9jNmzNCmTZv07rvvKigoSG63W5IUEhKiwMBASVfuuRcWFmrDhg2SpOXLl6tHjx6KiYlRWVmZNm7cKJfLJZfLZdt2AADQnNka9qtWrZIkDR8+3Kd/3bp1SkpKkiQVFRUpPz/f+1pZWZlSU1NVWFiowMBAxcTEaOfOnRozZkxTlQ0AQIti+2X8G1m/fr1Pe86cOZozZ04jVQQAgHmaxQN6AACg8RD2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADFensP/4449VUVHhbV//e/Qej0fbtm1rmMoAAECDqFPYJyQk6Ntvv/W2Q0JC9L//+7/e9vnz5/WLX/yi4aoDAAA3rU5hf/2Z/PXt6voAAIB9GvyevcPhaOi3BAAAN4EH9AAAMJxfXSd88cUXcrvdkq5csj958qQuXLggSTp37lzDVgcAAG5ancN+5MiRPvflx44dK+nK5XvLsriMDwBAM1OnsM/Ly2usOgAAQCOpU9h37969seoAAACNpE4P6H333Xf661//6tN3/PhxTZ06Vf/2b/+mTZs2NWhxAADg5tUp7GfMmKEXX3zR2z579qyGDBmiI0eOyOPxKCkpSW+++WaDFwkAAOqvTmF/+PBhPfjgg972hg0b1KlTJx07dkzvvvuufve73+nll19u8CIBAED91Sns3W63oqKivO19+/bpn//5n+Xnd+XW/4MPPqjTp083bIUAAOCm1Cnsg4ODdf78eW/7448/1uDBg71th8Mhj8fTYMUBAICbV6ewv+uuu7RixQpdvnxZb731ln744Qfdd9993tdPnTqlyMjIBi8SAADUX50+erd48WL97Gc/08aNG1VeXq558+bplltu8b6+ZcsWDRs2rMGLBAAA9VensB8wYIBOnDihQ4cOKTw8XD/96U99Xn/44YfVr1+/Bi0QAADcnDp/XW7nzp01fvz4Kl/7+c9/ftMFAQCAhlWnsN+wYUOtxk2ePLlexQAAgIZXp7BPSkpShw4d5Ofn5/NjONdyOByEPQAAzUidwr5v3776+uuv9eijj+rxxx9X//79G6suAADQQOr00bvjx49r586d+vHHHzV06FDFx8dr1apVKikpaaz6AADATapT2EvST3/6U7366qsqKirSrFmztG3bNkVERGjSpEl8oQ4AAM1QncP+qsDAQE2ePFkLFy7UXXfdpS1btuj//u//GrI2AADQAOoV9oWFhfrd736n3r176+GHH9agQYN0/Phxny/YqY309HQNGjRIQUFB6tKliyZMmKDc3Nwbzjtw4IDi4uIUEBCgnj17avXq1fXZDAAAWoU6hf22bds0evRo9e7dW0eOHNELL7yggoIC/ed//qf69OlT55UfOHBAM2bM0OHDh7V3716Vl5crMTFRpaWl1c7Jy8vTmDFjNGTIEOXk5GjevHmaNWuWXC5XndcPAEBr4LCq+wxdFdq0aaNu3bpp0qRJCgsLq3bcrFmz6lXMN998oy5duujAgQMaOnRolWPmzp2rHTt26MSJE96+5ORkffrpp8rKyrrhOkpKShQSEqLi4mIFBwfXq06gKnG/rt33ULRkR5fysdqGxnGDplCnj95169ZNDodDmzZtqnaMw+God9gXFxdLkjp16lTtmKysLCUmJvr0jRo1ShkZGbp06ZL8/f19XvN4PD4PDvLJAQBAa1OnsD9z5swNxxQWFtarEMuylJKSonvvvVexsbHVjnO73ZWuKoSFham8vFznzp1TRESEz2vp6elauHBhvWoCAMAE9X4a/3put1uzZs1Sr1696jX/6aef1meffabNmzffcKzD4fBpX70TcX2/JKWlpam4uNi7FBQU1Ks+AABaqjqF/fnz5zVp0iR17txZXbt29f62/YIFC9SzZ09lZWVp7dq1dS5i5syZ2rFjh/bv36/bb7+9xrHh4eFyu90+fWfPnpWfn59CQ0MrjXc6nQoODvZZAABoTep0GX/evHk6ePCgpkyZot27d2v27NnavXu3Ll68qA8++KDOv2VvWZZmzpypt99+W5mZmYqKirrhnISEBL333ns+fXv27FF8fHyl+/UAAKCOZ/Y7d+7UunXrtGzZMu3YsUOWZemOO+7Qvn376hz0kjRjxgxt3LhRmzZtUlBQkNxut9xut3788UfvmLS0NJ8f1klOTtZXX32llJQUnThxQmvXrlVGRoZSU1PrvH4AAFqDOoX93/72N/Xr10+S1LNnTwUEBGjatGn1XvmqVatUXFys4cOHKyIiwrts3brVO6aoqEj5+fnedlRUlHbt2qXMzEwNGDBAixcv1ooVKzRx4sR61wEAgMnqdBn/8uXLPpfK27Ztq/bt29d75bX5iP/69esr9Q0bNkyffPJJvdcLAEBrUqewtyxLSUlJcjqdkqSLFy8qOTm5UuBv37694SoEAAA3pU5hP2XKFJ/2o48+2qDFAACAhlensF+3bl1j1QEAABpJg32pDgAAaJ4IewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAw9ka9gcPHtS4cePUtWtXORwOvfPOOzWOz8zMlMPhqLScPHmyaQoGAKAF8rNz5aWlpbrzzjs1depUTZw4sdbzcnNzFRwc7G137ty5McoDAMAItob96NGjNXr06DrP69Klizp27NjwBQEAYKAWec9+4MCBioiI0MiRI7V///4ax3o8HpWUlPgsAAC0Ji0q7CMiIrRmzRq5XC5t375d0dHRGjlypA4ePFjtnPT0dIWEhHiXyMjIJqwYAAD72XoZv66io6MVHR3tbSckJKigoEDLli3T0KFDq5yTlpamlJQUb7ukpITABwC0Ki3qzL4qgwcP1unTp6t93el0Kjg42GcBAKA1afFhn5OTo4iICLvLAACg2bL1Mv6FCxf05Zdfett5eXk6duyYOnXqpG7duiktLU2FhYXasGGDJGn58uXq0aOHYmJiVFZWpo0bN8rlcsnlctm1CQAANHu2hn12drZGjBjhbV+9tz5lyhStX79eRUVFys/P975eVlam1NRUFRYWKjAwUDExMdq5c6fGjBnT5LUDANBS2Br2w4cPl2VZ1b6+fv16n/acOXM0Z86cRq4KAACztPh79gAAoGaEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGM7WsD948KDGjRunrl27yuFw6J133rnhnAMHDiguLk4BAQHq2bOnVq9e3fiFAgDQgtka9qWlpbrzzju1cuXKWo3Py8vTmDFjNGTIEOXk5GjevHmaNWuWXC5XI1cKAEDL5WfnykePHq3Ro0fXevzq1avVrVs3LV++XJLUt29fZWdna9myZZo4cWKVczwejzwej7ddUlJyUzUDANDStKh79llZWUpMTPTpGzVqlLKzs3Xp0qUq56SnpyskJMS7REZGNkWpAAA0Gy0q7N1ut8LCwnz6wsLCVF5ernPnzlU5Jy0tTcXFxd6loKCgKUoFAKDZsPUyfn04HA6ftmVZVfZf5XQ65XQ6G70uAACaqxZ1Zh8eHi632+3Td/bsWfn5+Sk0NNSmqgAAaN5aVNgnJCRo7969Pn179uxRfHy8/P39baoKAIDmzdawv3Dhgo4dO6Zjx45JuvLRumPHjik/P1/SlfvtkydP9o5PTk7WV199pZSUFJ04cUJr165VRkaGUlNT7SgfAIAWwdZ79tnZ2RoxYoS3nZKSIkmaMmWK1q9fr6KiIm/wS1JUVJR27dql2bNn6+WXX1bXrl21YsWKaj92BwAAbA774cOHex+wq8r69esr9Q0bNkyffPJJI1YFAIBZWtQ9ewAAUHeEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwLe4nbmGvuF9vsLuERnd06eQbDwKAFoQzewAADMeZvY0sy1Jpaam33b59ezkcDhsrAgCYiLC3UWlpqcaPH+9tv/vuu+rQoYONFQEtG39Am4Xbhg2HsK9CUx1gjvIyhVzTHv7/tsjya9fo6+WeNEzFH9BA1bhnDwCA4TizB9DoTL9aJnHFDM0bYW8jq62/ivv/wqcNAEBDI+zt5HA02VkH0BrwBzRQNcIegDn4AxqoEg/oAQBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcLaH/SuvvKKoqCgFBAQoLi5OH374YbVjMzMz5XA4Ki0nT55swooBAGhZbA37rVu36plnntH8+fOVk5OjIUOGaPTo0crPz69xXm5uroqKirxL7969m6hiAABaHlvD/sUXX9QTTzyhadOmqW/fvlq+fLkiIyO1atWqGud16dJF4eHh3qVt27bVjvV4PCopKfFZAABoTWwL+7KyMh09elSJiYk+/YmJiTp06FCNcwcOHKiIiAiNHDlS+/fvr3Fsenq6QkJCvEtkZORN1w4AQEtiW9ifO3dOFRUVCgsL8+kPCwuT2+2uck5ERITWrFkjl8ul7du3Kzo6WiNHjtTBgwerXU9aWpqKi4u9S0FBQYNuBwAAzZ2f3QU4HA6ftmVZlfquio6OVnR0tLedkJCggoICLVu2TEOHDq1yjtPplNPpbLiCAQBoYWw7s7/11lvVtm3bSmfxZ8+erXS2X5PBgwfr9OnTDV0eAADGsC3s27Vrp7i4OO3du9enf+/evbr77rtr/T45OTmKiIho6PIAADCGrZfxU1JS9Nhjjyk+Pl4JCQlas2aN8vPzlZycLOnK/fbCwkJt2LBBkrR8+XL16NFDMTExKisr08aNG+VyueRyuezcDAAAmjVbw/6hhx7St99+q0WLFqmoqEixsbHatWuXunfvLkkqKiry+cx9WVmZUlNTVVhYqMDAQMXExGjnzp0aM2aMXZsAAECzZ/sDetOnT9f06dOrfG39+vU+7Tlz5mjOnDlNUBUAAOaw/etyAQBA4yLsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGM72sH/llVcUFRWlgIAAxcXF6cMPP6xx/IEDBxQXF6eAgAD17NlTq1evbqJKAQBomWwN+61bt+qZZ57R/PnzlZOToyFDhmj06NHKz8+vcnxeXp7GjBmjIUOGKCcnR/PmzdOsWbPkcrmauHIAAFoOW8P+xRdf1BNPPKFp06apb9++Wr58uSIjI7Vq1aoqx69evVrdunXT8uXL1bdvX02bNk2PP/64li1b1sSVAwDQcvjZteKysjIdPXpUzz77rE9/YmKiDh06VOWcrKwsJSYm+vSNGjVKGRkZunTpkvz9/SvN8Xg88ng83nZxcbEkqaSkpNraKjw/1no7WqKatv1GTN83Uv33D/umeuyb6rFvqse+kYKCguRwOG5+RZZNCgsLLUnWRx995NO/ZMkS64477qhyTu/eva0lS5b49H300UeWJOtvf/tblXN+85vfWJJYWFhYWFha3FJcXNwgmWvbmf1V1//FYllWjX/FVDW+qv6r0tLSlJKS4m1fvnxZ3333nUJDQxvmr6WbVFJSosjISBUUFCg4ONjucpoV9k312DfVY9/UjP1Tvea4b4KCghrkfWwL+1tvvVVt27aV2+326T979qzCwsKqnBMeHl7leD8/P4WGhlY5x+l0yul0+vR17Nix/oU3kuDg4GZzcDU37JvqsW+qx76pGfuneibuG9se0GvXrp3i4uK0d+9en/69e/fq7rvvrnJOQkJCpfF79uxRfHx8lffrAQCAzU/jp6Sk6PXXX9fatWt14sQJzZ49W/n5+UpOTpZ05RL85MmTveOTk5P11VdfKSUlRSdOnNDatWuVkZGh1NRUuzYBAIBmz9Z79g899JC+/fZbLVq0SEVFRYqNjdWuXbvUvXt3SVJRUZHPZ+6joqK0a9cuzZ49Wy+//LK6du2qFStWaOLEiXZtwk1zOp36zW9+U+lWA9g3NWHfVI99UzP2T/VM3jcO6+oTbgAAwEi2f10uAABoXIQ9AACGI+wBADAcYY9m48yZM3I4HDp27JjdpQCAUQj7RnTo0CG1bdtWDzzwgE//jUKtoqJC6enp6tOnjwIDA9WpUycNHjxY69ata4KqG09SUpIcDod3CQ0N1QMPPKDPPvtMkhQZGen9VEZrk5SUpAkTJvj0VXf8SK33GLrWtfvs+v13/bF2dalqX5ruRvvp+uPOFMOHD9czzzxTqf+dd97xfnvq888/rwEDBlQac/78eTkcDmVmZkr6+7+3qpbDhw834lY0HNu/Ltdka9eu1cyZM/X6668rPz9f3bp1q9W8559/XmvWrNHKlSsVHx+vkpISZWdn6/vvv2/kihvfAw884A0ct9ut5557TmPHjlV+fr7atm2r8PBwmytsPup7/EhmH0O1de2xdpWJH6lC0/nv//5vxcTE+PRV9+2tzQ1h30hKS0u1bds2HTlyRG63W+vXr9eCBQtqNfe9997T9OnT9a//+q/evjvvvLOxSm1STqfTG+jh4eGaO3euhg4dqm+++UalpaWKiopSTk6OBgwYoIqKCj311FPat2+f3G63unXrpunTp+uXv/yl9/0yMzM1Z84cHT9+XP7+/oqJidGmTZu839XQUt3M8SOZfQzV1rXHGtAQQkNDW+wxxWX8RrJ161ZFR0crOjpajz76qNatW6fafqVBeHi49u3bp2+++aaRq7TXhQsX9Ic//EG9evWq8q/jy5cv6/bbb9e2bdv0xRdfaMGCBZo3b562bdsmSSovL9eECRM0bNgwffbZZ8rKytJTTz3VLH7g6GbdzPEjtZ5jCEDtEPaNJCMjQ48++qikK5cTL1y4oD/+8Y+1mvviiy/qm2++UXh4uPr376/k5GR98MEHjVluk3n//ffVoUMHdejQQUFBQdqxY4e2bt2qNm0qH4r+/v5auHChBg0apKioKE2aNElJSUnesC8pKVFxcbHGjh2rf/zHf1Tfvn01ZcqUOl3ubq5u5viRzD6GauvaY+3qsnjxYrvLQgt29913VzqmKioq7C6rVgj7RpCbm6uPP/5YDz/8sCTJz89PDz30kNauXVur+f369dP//M//6PDhw5o6daq+/vprjRs3TtOmTWvMspvEiBEjdOzYMR07dkx//vOflZiYqNGjR+urr76qcvzq1asVHx+vzp07q0OHDnrttde8X6HcqVMnJSUladSoURo3bpxeeuklFRUVNeXmNIqbPX4ks4+h2rr2WLu6zJgxw+6y0IJt3bq10jHVtm1bu8uqFe7ZN4KMjAyVl5frtttu8/ZZliV/f/9aPyDVpk0bDRo0SIMGDdLs2bO1ceNGPfbYY5o/f76ioqIaq/RG1759e/Xq1cvbjouLU0hIiF577bVKQbRt2zbNnj1bL7zwghISEhQUFKSlS5fqz3/+s3fMunXrNGvWLO3evVtbt27Vc889p71792rw4MFNtk0N7UbHzy233FKr9zH1GKqt6481tC7BwcEqLi6u1H/+/Hnvz9fWNEaSQkJCfPojIyNb7DHFmX0DKy8v14YNG/TCCy/4/PX36aefqnv37vrDH/5Qr/ft16+fpCsPbpnE4XCoTZs2+vHHHyu99uGHH+ruu+/W9OnTNXDgQPXq1Ut/+ctfKo0bOHCg0tLSdOjQIcXGxmrTpk1NUXqjaKzjRzL3GAKq0qdPH2VnZ1fqP3LkiKKjo71j/vrXv8rtdlca06ZNmxYb7FXhzL6Bvf/++/r+++/1xBNPVPqr8F/+5V+UkZGhsWPHSrpyufZ6/fr10yOPPKJ77rlHd999t8LDw5WXl6e0tDTdcccd6tOnT5NsR2PxeDzef1jff/+9Vq5cqQsXLmjcuHGVxvbq1UsbNmzQf/3XfykqKkpvvvmmjhw54j0rzcvL05o1a/Tggw+qa9euys3N1alTp3x+Frmlqc3x8/TTT3v7WuMxVFvXHmtX+fn56dZbb7WpIjSl6dOna+XKlZoxY4aeeuopBQYGau/evcrIyNCbb74pSUpMTFTfvn318MMPa8mSJeratas+++wzpaamKjk5WUFBQT7v+e2331Y6pjp27KiAgIAm2656s9Cgxo4da40ZM6bK144ePWpJ8v63qiUvL89as2aNNWLECKtz585Wu3btrG7dullJSUnWmTNnmnhrGtaUKVN8tjUoKMgaNGiQ9dZbb1mWZVl5eXmWJCsnJ8eyLMu6ePGilZSUZIWEhFgdO3a0/uM//sN69tlnrTvvvNOyLMtyu93WhAkTrIiICKtdu3ZW9+7drQULFlgVFRU2bWH9TZkyxRo/fnytj5+r+6q1HUPXurrPrv//q+2q9k10dLQ9xdroRvvp2rZpsrOzrVGjRlldunSxgoODrfj4eGvz5s0+Y4qKiqypU6da3bt3twIDA60+ffpYixYtsi5evOgdU9O/t+vfr7niJ24BADAc9+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsgVbgzJkzcjgcOnbsWLNZ1/Dhw/XMM880ej0ACHsADSwyMlJFRUWKjY2VJGVmZsrhcHh/SQxA0+OHcAA0mLKyMrVr107h4eF2lwLgGpzZA4bYvXu37r33XnXs2FGhoaEaO3ZslT8JfNWOHTvUu3dvBQYGasSIEXrjjTcqnYG7XC7FxMTI6XSqR48eeuGFF3zeo0ePHvrtb3+rpKQkhYSE6Mknn/S5jH/mzBmNGDFCknTLLbfI4XAoKSnJO//y5cuaM2eOOnXqpPDwcD3//PM+7+9wOPTqq69q7Nix+od/+Af17dtXWVlZ+vLLLzV8+HC1b99eCQkJNW4nAPGrd4Ap3nrrLcvlclmnTp2ycnJyrHHjxlk/+clPrIqKikq/KJiXl2f5+/tbqamp1smTJ63Nmzdbt912myXJ+v777y3LuvKLYW3atLEWLVpk5ebmWuvWrbMCAwOtdevWedfZvXt3Kzg42Fq6dKl1+vRp6/Tp0z7rKi8vt1wulyXJys3NtYqKiqzz589blmVZw4YNs4KDg63nn3/eOnXqlPXGG29YDofD2rNnj/f9JVm33XabtXXrVis3N9eaMGGC1aNHD+u+++6zdu/ebX3xxRfW4MGDrQceeKCpdjPQIhH2gKHOnj1rSbI+//zzSmE/d+5cKzY21mf8/PnzfcL+kUcese6//36fMb/+9a+tfv36edvdu3e3JkyY4DPm+nXt37/f532vGjZsmHXvvff69A0aNMiaO3euty3Jeu6557ztrKwsS5KVkZHh7du8ebMVEBBw4x0CtGJcxgcM8Ze//EWPPPKIevbsqeDgYEVFRUmS8vPzK43Nzc3VoEGDfPruuusun/aJEyd0zz33+PTdc889On36tCoqKrx98fHx9a65f//+Pu2IiAidPXu22jFhYWGSpJ/85Cc+fRcvXlRJSUm96wBMxwN6gCHGjRunyMhIvfbaa+ratasuX76s2NhYlZWVVRprWZYcDkelvrqOkaT27dvXu2Z/f3+ftsPh0OXLl6sdc7Weqvqunwfg7wh7wADffvutTpw4oVdffVVDhgyRJP3pT3+qdnyfPn20a9cun77s7Gyfdr9+/Sq9x6FDh3THHXeobdu2ta6tXbt2kuRzNQBA0+IyPmCAW265RaGhoVqzZo2+/PJL7du3TykpKdWO//d//3edPHlSc+fO1alTp7Rt2zatX79e0t/PlH/1q1/pj3/8oxYvXqxTp07pjTfe0MqVK5Wamlqn2rp37y6Hw6H3339f33zzjS5cuFDv7QRQP4Q9YIA2bdpoy5YtOnr0qGJjYzV79mwtXbq02vFRUVF66623tH37dvXv31+rVq3S/PnzJUlOp1OS9E//9E/atm2btmzZotjYWC1YsECLFi3y+ehcbdx2221auHChnn32WYWFhenpp5+u93YCqB+HVdVNOACtzpIlS7R69WoVFBTYXQqABsY9e6CVeuWVVzRo0CCFhobqo48+0tKlSznrBgxF2AOt1OnTp/Xb3/5W3333nbp166Zf/epXSktLs7ssAI2Ay/gAABiOB/QAADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhvv/+jsn1AMGdH4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 511.111x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.catplot(x='algorithm', y='RMSE', data=user_rmse, kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffbd7ba0",
   "metadata": {
    "papermill": {
     "duration": 0.006749,
     "end_time": "2024-12-08T00:48:53.122048",
     "exception": false,
     "start_time": "2024-12-08T00:48:53.115299",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Save Metrics\n",
    "\n",
    "We'll now save the top-N recommendation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1b776675",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:48:53.135340Z",
     "iopub.status.busy": "2024-12-08T00:48:53.135100Z",
     "iopub.status.idle": "2024-12-08T00:48:53.170762Z",
     "shell.execute_reply": "2024-12-08T00:48:53.170329Z"
    },
    "papermill": {
     "duration": 0.043267,
     "end_time": "2024-12-08T00:48:53.171570",
     "exception": false,
     "start_time": "2024-12-08T00:48:53.128303",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RecipRank</th>\n",
       "      <th>NDCG</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ALS</th>\n",
       "      <td>0.050287</td>\n",
       "      <td>0.038741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bias</th>\n",
       "      <td>0.069619</td>\n",
       "      <td>0.043852</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IALS</th>\n",
       "      <td>0.275229</td>\n",
       "      <td>0.285604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IIE</th>\n",
       "      <td>0.007768</td>\n",
       "      <td>0.012593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>III</th>\n",
       "      <td>0.312870</td>\n",
       "      <td>0.249497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pop</th>\n",
       "      <td>0.175094</td>\n",
       "      <td>0.153006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UUE</th>\n",
       "      <td>0.004210</td>\n",
       "      <td>0.009815</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           RecipRank      NDCG\n",
       "algorithm                     \n",
       "ALS         0.050287  0.038741\n",
       "Bias        0.069619  0.043852\n",
       "IALS        0.275229  0.285604\n",
       "IIE         0.007768  0.012593\n",
       "III         0.312870  0.249497\n",
       "Pop         0.175094  0.153006\n",
       "UUE         0.004210  0.009815"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "saved = results[['recip_rank', 'ndcg']].rename(columns={'ndcg': 'NDCG', 'recip_rank': 'RecipRank'})\n",
    "saved = saved.groupby('algorithm').agg('mean')\n",
    "saved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c4083a95",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-08T00:48:53.185410Z",
     "iopub.status.busy": "2024-12-08T00:48:53.185175Z",
     "iopub.status.idle": "2024-12-08T00:48:53.189686Z",
     "shell.execute_reply": "2024-12-08T00:48:53.189297Z"
    },
    "papermill": {
     "duration": 0.012224,
     "end_time": "2024-12-08T00:48:53.190478",
     "exception": false,
     "start_time": "2024-12-08T00:48:53.178254",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "saved.to_json(f'eval-metrics.{dataset}.json', orient='index')"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md",
   "notebook_metadata_filter": "split_at_heading"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 246.682476,
   "end_time": "2024-12-08T00:48:53.613512",
   "environment_variables": {},
   "exception": null,
   "input_path": "eval-report.ipynb",
   "output_path": "eval-report.ml10m.ipynb",
   "parameters": {
    "dataset": "ml10m"
   },
   "start_time": "2024-12-08T00:44:46.931036",
   "version": "2.6.0"
  },
  "split_at_heading": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}