eval-report.mlsmall.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ce485e34",
   "metadata": {
    "papermill": {
     "duration": 0.005875,
     "end_time": "2024-12-07T21:06:43.540731",
     "exception": false,
     "start_time": "2024-12-07T21:06:43.534856",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Evaluation Analysis for Recommender Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17c9a5e6",
   "metadata": {
    "papermill": {
     "duration": 0.004569,
     "end_time": "2024-12-07T21:06:43.550437",
     "exception": false,
     "start_time": "2024-12-07T21:06:43.545868",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this section we will analyze and compare the generated recommendations and predictions from a predefined list of algorithms with the goal of assessing the performance of each algorithm with respect to a metric. In other words, we would rank the algorithms for each metric considered with respect to performance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f14bb433",
   "metadata": {
    "papermill": {
     "duration": 0.004672,
     "end_time": "2024-12-07T21:06:43.559524",
     "exception": false,
     "start_time": "2024-12-07T21:06:43.554852",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b8308de",
   "metadata": {
    "papermill": {
     "duration": 0.004371,
     "end_time": "2024-12-07T21:06:43.568165",
     "exception": false,
     "start_time": "2024-12-07T21:06:43.563794",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Below are the list of packages required to successfully run the analysis. They are divided into partitions to signify their specific task.<br>\n",
    "We need the pathlib package for working with files and folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "77d2e6e9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:43.578564Z",
     "iopub.status.busy": "2024-12-07T21:06:43.578332Z",
     "iopub.status.idle": "2024-12-07T21:06:43.583110Z",
     "shell.execute_reply": "2024-12-07T21:06:43.582736Z"
    },
    "papermill": {
     "duration": 0.011399,
     "end_time": "2024-12-07T21:06:43.583786",
     "exception": false,
     "start_time": "2024-12-07T21:06:43.572387",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8607dbba",
   "metadata": {
    "papermill": {
     "duration": 0.004435,
     "end_time": "2024-12-07T21:06:43.592658",
     "exception": false,
     "start_time": "2024-12-07T21:06:43.588223",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We would use the pandas for analyzing and manipulating our data while seaborn and matplotlib are used for data visualization. statsmodels.graphics.gofplots and scipy.stats.shapiro are used for normality check. Scipy.stats.friedmanchisquare is a non-parametric test used to determine the statistical significance in metric results and the wilcoxon test is used for pairwise comparison of sample data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "54c9b1a0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:43.602442Z",
     "iopub.status.busy": "2024-12-07T21:06:43.602231Z",
     "iopub.status.idle": "2024-12-07T21:06:44.771763Z",
     "shell.execute_reply": "2024-12-07T21:06:44.771206Z"
    },
    "papermill": {
     "duration": 1.175877,
     "end_time": "2024-12-07T21:06:44.772886",
     "exception": false,
     "start_time": "2024-12-07T21:06:43.597009",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from statsmodels.graphics.gofplots import qqplot\n",
    "from scipy.stats import shapiro\n",
    "from scipy.stats import friedmanchisquare, wilcoxon\n",
    "from itertools import combinations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09d4299f",
   "metadata": {
    "papermill": {
     "duration": 0.004565,
     "end_time": "2024-12-07T21:06:44.782787",
     "exception": false,
     "start_time": "2024-12-07T21:06:44.778222",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Logging to show what's happening in LensKit routines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fd84093f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:44.792732Z",
     "iopub.status.busy": "2024-12-07T21:06:44.792414Z",
     "iopub.status.idle": "2024-12-07T21:06:44.813178Z",
     "shell.execute_reply": "2024-12-07T21:06:44.812785Z"
    },
    "papermill": {
     "duration": 0.02673,
     "end_time": "2024-12-07T21:06:44.813919",
     "exception": false,
     "start_time": "2024-12-07T21:06:44.787189",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.util.log notebook logging configured\n"
     ]
    }
   ],
   "source": [
    "from lenskit import util\n",
    "util.log_to_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c7c3c1f0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:44.824645Z",
     "iopub.status.busy": "2024-12-07T21:06:44.824416Z",
     "iopub.status.idle": "2024-12-07T21:06:44.826749Z",
     "shell.execute_reply": "2024-12-07T21:06:44.826361Z"
    },
    "papermill": {
     "duration": 0.008552,
     "end_time": "2024-12-07T21:06:44.827379",
     "exception": false,
     "start_time": "2024-12-07T21:06:44.818827",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "_log = logging.getLogger('eval-report')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af5b4ebc",
   "metadata": {
    "papermill": {
     "duration": 0.004704,
     "end_time": "2024-12-07T21:06:44.836985",
     "exception": false,
     "start_time": "2024-12-07T21:06:44.832281",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We will use lenskit for training, running, and evaluating recommender algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "803d7eb5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:44.846491Z",
     "iopub.status.busy": "2024-12-07T21:06:44.846080Z",
     "iopub.status.idle": "2024-12-07T21:06:44.912350Z",
     "shell.execute_reply": "2024-12-07T21:06:44.911911Z"
    },
    "papermill": {
     "duration": 0.072194,
     "end_time": "2024-12-07T21:06:44.913447",
     "exception": false,
     "start_time": "2024-12-07T21:06:44.841253",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from lenskit import topn\n",
    "from lenskit.metrics.predict import rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62cde0fa",
   "metadata": {
    "papermill": {
     "duration": 0.00484,
     "end_time": "2024-12-07T21:06:44.923515",
     "exception": false,
     "start_time": "2024-12-07T21:06:44.918675",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b8bdd0f",
   "metadata": {
    "papermill": {
     "duration": 0.004633,
     "end_time": "2024-12-07T21:06:44.932657",
     "exception": false,
     "start_time": "2024-12-07T21:06:44.928024",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We specify the dataset we will use for our analysis and the main directory from where we read the recommendation and prediction files. From the main directory we find all the directories associated with the dataset and then read the recommendation and predictions files from those directories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "66d37091",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:44.942639Z",
     "iopub.status.busy": "2024-12-07T21:06:44.942396Z",
     "iopub.status.idle": "2024-12-07T21:06:44.944753Z",
     "shell.execute_reply": "2024-12-07T21:06:44.944364Z"
    },
    "papermill": {
     "duration": 0.008034,
     "end_time": "2024-12-07T21:06:44.945439",
     "exception": false,
     "start_time": "2024-12-07T21:06:44.937405",
     "status": "completed"
    },
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "dataset = \"ml100k\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a4ad7a04",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:44.955233Z",
     "iopub.status.busy": "2024-12-07T21:06:44.954902Z",
     "iopub.status.idle": "2024-12-07T21:06:44.957075Z",
     "shell.execute_reply": "2024-12-07T21:06:44.956698Z"
    },
    "papermill": {
     "duration": 0.007877,
     "end_time": "2024-12-07T21:06:44.957755",
     "exception": false,
     "start_time": "2024-12-07T21:06:44.949878",
     "status": "completed"
    },
    "tags": [
     "injected-parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "dataset = \"mlsmall\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b7f9f66b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:44.968136Z",
     "iopub.status.busy": "2024-12-07T21:06:44.967930Z",
     "iopub.status.idle": "2024-12-07T21:06:44.970158Z",
     "shell.execute_reply": "2024-12-07T21:06:44.969782Z"
    },
    "papermill": {
     "duration": 0.008517,
     "end_time": "2024-12-07T21:06:44.970875",
     "exception": false,
     "start_time": "2024-12-07T21:06:44.962358",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "output_root = Path(\"runs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8688a4d0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:44.980158Z",
     "iopub.status.busy": "2024-12-07T21:06:44.979952Z",
     "iopub.status.idle": "2024-12-07T21:06:44.982832Z",
     "shell.execute_reply": "2024-12-07T21:06:44.982411Z"
    },
    "papermill": {
     "duration": 0.008403,
     "end_time": "2024-12-07T21:06:44.983534",
     "exception": false,
     "start_time": "2024-12-07T21:06:44.975131",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "dirs = [fld for fld in output_root.glob(f'{dataset}-*')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "56d609aa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:44.993377Z",
     "iopub.status.busy": "2024-12-07T21:06:44.993175Z",
     "iopub.status.idle": "2024-12-07T21:06:45.279166Z",
     "shell.execute_reply": "2024-12-07T21:06:45.278702Z"
    },
    "papermill": {
     "duration": 0.292238,
     "end_time": "2024-12-07T21:06:45.280265",
     "exception": false,
     "start_time": "2024-12-07T21:06:44.988027",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 426900 entries, 0 to 426899\n",
      "Data columns (total 5 columns):\n",
      " #   Column     Non-Null Count   Dtype   \n",
      "---  ------     --------------   -----   \n",
      " 0   item       426900 non-null  int64   \n",
      " 1   score      426900 non-null  float64 \n",
      " 2   user       426900 non-null  int64   \n",
      " 3   rank       426900 non-null  int64   \n",
      " 4   algorithm  426900 non-null  category\n",
      "dtypes: category(1), float64(1), int64(3)\n",
      "memory usage: 13.4 MB\n"
     ]
    }
   ],
   "source": [
    "recs = []\n",
    "for fld in dirs:\n",
    "    for file in fld.glob(\"recs-*\"):\n",
    "        rec = pd.read_csv(file)\n",
    "        rec[\"algorithm\"] = fld.name.split(\"-\")[1]\n",
    "        recs.append(rec)\n",
    "\n",
    "recs = pd.concat(recs, ignore_index=True)\n",
    "recs = recs.astype({'algorithm': 'category'})\n",
    "recs.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "feaa4493",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:45.292290Z",
     "iopub.status.busy": "2024-12-07T21:06:45.292042Z",
     "iopub.status.idle": "2024-12-07T21:06:45.298846Z",
     "shell.execute_reply": "2024-12-07T21:06:45.298467Z"
    },
    "papermill": {
     "duration": 0.013033,
     "end_time": "2024-12-07T21:06:45.299554",
     "exception": false,
     "start_time": "2024-12-07T21:06:45.286521",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Pop', 'ALS', 'UUE', 'IIE', 'Bias', 'III', 'IALS']\n",
       "Categories (7, object): ['ALS', 'Bias', 'IALS', 'IIE', 'III', 'Pop', 'UUE']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rec_algos = recs['algorithm'].unique()\n",
    "rec_algos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b378a77c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:45.310034Z",
     "iopub.status.busy": "2024-12-07T21:06:45.309816Z",
     "iopub.status.idle": "2024-12-07T21:06:45.404771Z",
     "shell.execute_reply": "2024-12-07T21:06:45.404343Z"
    },
    "papermill": {
     "duration": 0.101069,
     "end_time": "2024-12-07T21:06:45.405452",
     "exception": false,
     "start_time": "2024-12-07T21:06:45.304383",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 18300 entries, 0 to 18299\n",
      "Data columns (total 6 columns):\n",
      " #   Column      Non-Null Count  Dtype   \n",
      "---  ------      --------------  -----   \n",
      " 0   user        18300 non-null  int64   \n",
      " 1   item        18300 non-null  int64   \n",
      " 2   rating      18300 non-null  float64 \n",
      " 3   timestamp   18300 non-null  int64   \n",
      " 4   prediction  18022 non-null  float64 \n",
      " 5   algorithm   18300 non-null  category\n",
      "dtypes: category(1), float64(2), int64(3)\n",
      "memory usage: 733.1 KB\n"
     ]
    }
   ],
   "source": [
    "preds = []\n",
    "for fld in dirs:\n",
    "    for file in fld.glob(\"pred-*\"):\n",
    "        pred = pd.read_csv(file)\n",
    "        pred[\"algorithm\"] = fld.name.split(\"-\")[1]\n",
    "        preds.append(pred)\n",
    "\n",
    "preds = pd.concat(preds, ignore_index=True)\n",
    "preds = preds.astype({'algorithm': 'category'})\n",
    "preds.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca476ae1",
   "metadata": {
    "papermill": {
     "duration": 0.005296,
     "end_time": "2024-12-07T21:06:45.416191",
     "exception": false,
     "start_time": "2024-12-07T21:06:45.410895",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We need to load the test data so that we have the ground truths for computing accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "25294727",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:45.426697Z",
     "iopub.status.busy": "2024-12-07T21:06:45.426480Z",
     "iopub.status.idle": "2024-12-07T21:06:45.428777Z",
     "shell.execute_reply": "2024-12-07T21:06:45.428390Z"
    },
    "papermill": {
     "duration": 0.008549,
     "end_time": "2024-12-07T21:06:45.429552",
     "exception": false,
     "start_time": "2024-12-07T21:06:45.421003",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "split_root = Path(\"data-split\")\n",
    "split_dir = split_root / dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "88fc0d4e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:45.440188Z",
     "iopub.status.busy": "2024-12-07T21:06:45.439829Z",
     "iopub.status.idle": "2024-12-07T21:06:45.502789Z",
     "shell.execute_reply": "2024-12-07T21:06:45.502400Z"
    },
    "papermill": {
     "duration": 0.068996,
     "end_time": "2024-12-07T21:06:45.503521",
     "exception": false,
     "start_time": "2024-12-07T21:06:45.434525",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>item</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>part</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>249</th>\n",
       "      <td>2</td>\n",
       "      <td>86345</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1445715166</td>\n",
       "      <td>test-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>238</th>\n",
       "      <td>2</td>\n",
       "      <td>46970</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1445715013</td>\n",
       "      <td>test-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>234</th>\n",
       "      <td>2</td>\n",
       "      <td>1704</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1445715228</td>\n",
       "      <td>test-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>233</th>\n",
       "      <td>2</td>\n",
       "      <td>333</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1445715029</td>\n",
       "      <td>test-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235</th>\n",
       "      <td>2</td>\n",
       "      <td>3578</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1445714885</td>\n",
       "      <td>test-1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user   item  rating   timestamp    part\n",
       "index                                         \n",
       "249       2  86345     4.0  1445715166  test-1\n",
       "238       2  46970     4.0  1445715013  test-1\n",
       "234       2   1704     4.5  1445715228  test-1\n",
       "233       2    333     4.0  1445715029  test-1\n",
       "235       2   3578     4.0  1445714885  test-1"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = []\n",
    "for file in split_dir.glob(\"test-*.parquet\"):\n",
    "    test.append(pd.read_parquet(file,).assign(part=file.stem.replace('.parquet', '')))\n",
    "\n",
    "test = pd.concat(test).rename(columns={\n",
    "    'user_id': 'user',\n",
    "    'item_id': 'item',\n",
    "})\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5770c372",
   "metadata": {
    "papermill": {
     "duration": 0.005015,
     "end_time": "2024-12-07T21:06:45.513720",
     "exception": false,
     "start_time": "2024-12-07T21:06:45.508705",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Top-N Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e347325",
   "metadata": {
    "papermill": {
     "duration": 0.004781,
     "end_time": "2024-12-07T21:06:45.524011",
     "exception": false,
     "start_time": "2024-12-07T21:06:45.519230",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The topn.RecListAnalysis class computes top-N metrics for recommendation list and takes care of making sure that the recommendations and ground truths are properly matched. Refer to the documentation for detailed explanation of the purpose for the RecListAnalysis class and how the analysis is done - https://lkpy.lenskit.org/en/stable/evaluation/topn-metrics.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "198eda4d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:45.534364Z",
     "iopub.status.busy": "2024-12-07T21:06:45.534145Z",
     "iopub.status.idle": "2024-12-07T21:06:45.781259Z",
     "shell.execute_reply": "2024-12-07T21:06:45.780859Z"
    },
    "papermill": {
     "duration": 0.253116,
     "end_time": "2024-12-07T21:06:45.782009",
     "exception": false,
     "start_time": "2024-12-07T21:06:45.528893",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn analyzing 426900 recommendations (3050 truth rows)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn using rec key columns ['algorithm', 'user']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn using truth key columns ['user']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn numbering truth lists\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn numbering rec lists\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn collecting metric results\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn measured 4269 lists in  152ms\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn filling in missing user info (4269 initial rows)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/mde48/micromamba/envs/lkdemo/lib/python3.10/site-packages/lenskit/topn.py:152: FutureWarning: In a future version of pandas, a length 1 tuple will be returned when iterating over a groupby with a grouper equal to a list of length 1. Don't supply a list with a single grouper to avoid this warning.\n",
      "  for key, df in res.groupby(ug_cols):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>nrecs</th>\n",
       "      <th>precision</th>\n",
       "      <th>recip_rank</th>\n",
       "      <th>ndcg</th>\n",
       "      <th>ntruth</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th>user</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">ALS</th>\n",
       "      <th>14</th>\n",
       "      <td>100.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>100.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.090909</td>\n",
       "      <td>0.191715</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>100.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>100.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>100.0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.177148</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                nrecs  precision  recip_rank      ndcg  ntruth\n",
       "algorithm user                                                \n",
       "ALS       14    100.0       0.00    0.000000  0.000000       5\n",
       "          17    100.0       0.03    0.090909  0.191715       5\n",
       "          24    100.0       0.00    0.000000  0.000000       5\n",
       "          26    100.0       0.00    0.000000  0.000000       5\n",
       "          30    100.0       0.01    0.333333  0.177148       5"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rla = topn.RecListAnalysis()\n",
    "\n",
    "rla.add_metric(topn.precision)\n",
    "rla.add_metric(topn.recip_rank)\n",
    "rla.add_metric(topn.ndcg)\n",
    "results = rla.compute(recs, test.drop(columns=['rating']), include_missing=True)\n",
    "results = results.fillna(0)\n",
    "results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75dbe033",
   "metadata": {
    "papermill": {
     "duration": 0.0059,
     "end_time": "2024-12-07T21:06:45.794464",
     "exception": false,
     "start_time": "2024-12-07T21:06:45.788564",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We will reshape the 'results' dataframe by stacking the columns to index and then use the bar chart to visualize the performance of our algorithms with respect to the precision, reciprocal rank and ndcg metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4c7ffaf7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:45.806666Z",
     "iopub.status.busy": "2024-12-07T21:06:45.806423Z",
     "iopub.status.idle": "2024-12-07T21:06:45.815274Z",
     "shell.execute_reply": "2024-12-07T21:06:45.814880Z"
    },
    "papermill": {
     "duration": 0.01555,
     "end_time": "2024-12-07T21:06:45.815913",
     "exception": false,
     "start_time": "2024-12-07T21:06:45.800363",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>algorithm</th>\n",
       "      <th>user</th>\n",
       "      <th>metric</th>\n",
       "      <th>val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ALS</td>\n",
       "      <td>14</td>\n",
       "      <td>precision</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ALS</td>\n",
       "      <td>14</td>\n",
       "      <td>recip_rank</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ALS</td>\n",
       "      <td>14</td>\n",
       "      <td>ndcg</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ALS</td>\n",
       "      <td>17</td>\n",
       "      <td>precision</td>\n",
       "      <td>0.030000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ALS</td>\n",
       "      <td>17</td>\n",
       "      <td>recip_rank</td>\n",
       "      <td>0.090909</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  algorithm  user      metric       val\n",
       "0       ALS    14   precision  0.000000\n",
       "1       ALS    14  recip_rank  0.000000\n",
       "2       ALS    14        ndcg  0.000000\n",
       "3       ALS    17   precision  0.030000\n",
       "4       ALS    17  recip_rank  0.090909"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pltData = (results.drop(columns=['nrecs', 'ntruth']).stack()).reset_index()\n",
    "pltData.columns = ['algorithm', 'user', 'metric', 'val']\n",
    "pltData.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66c6e8ba",
   "metadata": {
    "papermill": {
     "duration": 0.005539,
     "end_time": "2024-12-07T21:06:45.827342",
     "exception": false,
     "start_time": "2024-12-07T21:06:45.821803",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We need to determine if the differences we observe in the performances of the algorithms for the various metrics are statistically significant. To achieve this, we will need to use either a parametric or non-parametric statistical test for comparing the differences. We will consider a parametric test - repeated ANOVA measure cause our sample groups are correlated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bd651349",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:45.839319Z",
     "iopub.status.busy": "2024-12-07T21:06:45.839105Z",
     "iopub.status.idle": "2024-12-07T21:06:46.862258Z",
     "shell.execute_reply": "2024-12-07T21:06:46.861804Z"
    },
    "papermill": {
     "duration": 1.030172,
     "end_time": "2024-12-07T21:06:46.863032",
     "exception": false,
     "start_time": "2024-12-07T21:06:45.832860",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABEAAAAEiCAYAAAD5+6e5AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABXUElEQVR4nO3de1xVVf7/8feRq4mQogEmAuYF8EpACkbqpJimaaNJNqKWlzEsL3QlalJrItMUnRQvY6EzhVhmZcOk2C8vJdlI4DTlmONIOA5kWklYguL+/eHXMx4PKCBwOIfX8/HYj3Gvs/ban3UO84E+Z+29TYZhGAIAAAAAAHBgzWwdAAAAAAAAQH2jAAIAAAAAABweBRAAAAAAAODwKIAAAAAAAACHRwEEAAAAAAA4PAogAAAAAADA4VEAAQAAAAAADo8CCAAAAAAAcHgUQAAAAAAAgMOjAAL8n7lz56p37962DsMmTCaT3nnnnTrvC6DpctScGhgYqNTUVFuHUS3kawA1Ya95e8CAAZo9e7atw4CdMBmGYdg6CKC+DBgwQL17967WH6ulpaUqKyuTt7d3/QfWyBQXF6tVq1Zyc3Or074AHAs5Vfruu+/UokULXXfddbYO5apMJpM2b96sUaNG2ToUADbSFPJ2TeYIONs6AMDWDMNQRUWFPDw85OHhYetwaqS8vFyurq7XPI6vr2+99AXQ9DTWnHr27Fm5uLhc8zht27atg2iqVldxAkB1Nda8DdQHLoFBozBgwAA9/PDDmj17tlq1aiUfHx+tXr1ap0+f1v3336+WLVvqpptu0l//+leL47766isNGzZMHh4e8vHxUXx8vE6cOCFJmjRpknbu3KmlS5fKZDLJZDKpoKBAO3bskMlk0tatWxURESE3Nzft3r270mV/r776qrp16yY3Nzf5+fnpoYceqrf34OL5V61aJX9/f1133XW655579OOPP5r7TJo0SaNGjVJKSoratWunLl26SJKOHTumuLg4tWrVSt7e3ho5cqQKCgqqPZdLl0mXl5froYcekp+fn9zd3RUYGKiUlJRK+0rSF198oV/96ldq3ry5vL29NW3aNJWWllrFvGjRIvn5+cnb21szZszQ2bNn6+7NA2CBnPq/nPrqq6+qY8eOcnNzk2EYOnXqlKZNm6YbbrhBnp6e+tWvfqX9+/dbHPvee+8pIiJC7u7uatOmjX7961+bX7v8EhiTyaS0tDQNHTpUzZs3V1BQkN58881qxVhQUCCTyaSNGzdqwIABcnd315///GedPHlS48aNU/v27XXdddepR48eysjIsDh2wIABmjlzph5//HG1bt1avr6+mjt37hXPN3/+fPn4+Cg/P79a8QFoOOTt/+XtP/3pTwoMDJSXl5fuvfde/fTTT+Y+p0+f1oQJE+Th4SE/Pz+9/PLLVuOUlZXp8ccfl7+/v9zc3NS5c2etXbvW/Pp7772nzp07q3nz5ho4cKDWrVsnk8lk8Tc3HBcFEDQa69atU5s2bfTZZ5/p4Ycf1oMPPqh77rlH0dHR+vzzzzVkyBDFx8fr559/liQVFRWpf//+6t27t/bt26cPPvhA3377rcaOHStJWrp0qaKiojR16lQVFRWpqKhI/v7+5vM9/vjjSklJ0YEDB9SzZ0+reNLS0jRjxgxNmzZNX3zxhd577z116tSpyvhff/11c+W8qu3111+/4nvwr3/9Sxs3btSWLVv0wQcfKD8/XzNmzLDo8+GHH+rAgQPKzs7W+++/r59//lkDBw6Uh4eHdu3apY8//lgeHh664447VF5eXuO5LFu2TO+99542btyogwcP6s9//rMCAwMr7fvzzz/rjjvuUKtWrfS3v/1Nb775prZv3271y/Gjjz7S4cOH9dFHH2ndunVKT09Xenr6Fd8LANeGnPq/nLpp0ybzf/TfeeedKi4uVlZWlnJzc3XzzTfr9ttv1/fffy9J+stf/qJf//rXuvPOO5WXl6cPP/xQERERVzzPM888o9GjR2v//v0aP368xo0bpwMHDlzxmEs98cQTmjlzpg4cOKAhQ4bozJkzCg8P1/vvv69//OMfmjZtmuLj47V3716L49atW6cWLVpo7969eumllzR//nxlZ2dbjW8YhmbNmqW1a9fq448/tstr/IGmgLwtHT58WO+8847ef/99vf/++9q5c6defPFF8+uPPfaYPvroI23evFnbtm3Tjh07lJubazHGhAkTtGHDBi1btkwHDhzQypUrzStbCgoKNGbMGI0aNUr5+fn67W9/q+Tk5Kt8MnAoBtAI9O/f37j11lvN++fOnTNatGhhxMfHm9uKiooMSUZOTo5hGIbxzDPPGLGxsRbjHD161JBkHDx40DzurFmzLPp89NFHhiTjnXfesWh/9tlnjV69epn327VrZyQnJ1d7DiUlJcahQ4euuJWUlFR5/LPPPms4OTkZR48eNbf99a9/NZo1a2YUFRUZhmEYEydONHx8fIyysjJzn7Vr1xpdu3Y1zp8/b24rKyszmjdvbmzdurVac5FkbN682TAMw3j44YeNX/3qVxbjVdV39erVRqtWrYzS0lLz63/5y1+MZs2aGcXFxeaYAwICjHPnzpn73HPPPUZcXFyV8QC4NuTUC+d3cXExjh8/bm778MMPDU9PT+PMmTMWfW+66SZj1apVhmEYRlRUlPGb3/ymynEDAgKMJUuWmPclGdOnT7fo06dPH+PBBx+86hyPHDliSDJSU1Ov2nfYsGHGI488Yt6//DM2DMOIjIw0nnjiCYvY3nzzTWP8+PFGcHCwxe8XAI0LefvC+a+77jqLPo899pjRp08fwzAM46effjJcXV2NDRs2mF8/efKk0bx5c/McDx48aEgysrOzKz3HE088YXTv3t2iLTk52ZBk/PDDD9WeK+wX9wBBo3Fp5dnJyUne3t7q0aOHuc3Hx0eSdPz4cUlSbm6uPvroo0qvVTx8+LD58pCqXOkbvePHj+u///2vbr/99mrH37JlS7Vs2bLa/SvToUMHtW/f3rwfFRWl8+fP6+DBg+Z7b/To0cPivh+5ubn617/+ZXXuM2fO6PDhwzWey6RJkzR48GB17dpVd9xxh4YPH67Y2NhK+x44cEC9evVSixYtzG39+vUzx3zxM+vWrZucnJzMffz8/PTFF19UKx4AtUNOlQICAizu2ZGbm6vS0lKrG/z98ssvOnz4sCQpPz9fU6dOrdF5oqKirPZrcpnJ5e9dRUWFXnzxRWVmZurYsWMqKytTWVmZRa6VZPWNrZ+fn/nzvGjOnDlyc3PTp59+qjZt2lQ7JgANj7x94TLDS8e4NK8dPnxY5eXlFjm3devW6tq1q3k/Pz9fTk5O6t+/f6XjHzx4UJGRkRZtt9xyyzXFDPtCAQSNxuU3fTOZTBZtJpNJknT+/Hnz/44YMUILFiywGsvPz++q57v8D8lLNW/evFoxX+r111/Xb3/72yv2WbVqlX7zm99Ue8yLc774v5J13OfPn1d4eHilSwrbtm2rZs1qdqXbzTffrCNHjuivf/2rtm/frrFjx2rQoEF66623rPoahmERW2WxS5V/thc/RwD1g5xaeb708/PTjh07rPpef/31tY61MlXlxspcHufLL7+sJUuWKDU1VT169FCLFi00e/Zs82WNF1Untw4ePFgZGRnaunVrjX7/AGh45O0r5zWjGg8vvVrclf3tWp1x4TgogMBu3Xzzzdq0aZMCAwPl7Fz5j7Krq6sqKipqPHbLli0VGBioDz/8UAMHDqzWMXfddZf69OlzxT4XK/dVKSws1H//+1+1a9dOkpSTk6NmzZpdsYJ/8803KzMz03xDv8rUdC6enp6Ki4tTXFycxowZozvuuEPff/+9WrdubdEvNDRU69at0+nTp82/RD/55JOrxgyg8XHEnHq5m2++WcXFxXJ2dq7y3kY9e/bUhx9+qPvvv7/a43766aeaMGGCxX5YWFiNYrvU7t27NXLkSI0fP17Shf/IOXTokEJCQmo81l133aURI0bovvvuk5OTk+69995axwWgcWkKeftSnTp1kouLiz799FN16NBBkvTDDz/o66+/Nq/46NGjh86fP6+dO3dq0KBBVmMEBwcrKyvLom3fvn21jgn2hwII7NaMGTO0Zs0ajRs3To899pjatGmjf/3rX9qwYYPWrFkjJycnBQYGau/evSooKJCHh4fVf8Bfydy5czV9+nTdcMMNGjp0qH766Sd98sknevjhhyvtXxfL/tzd3TVx4kQtWrRIJSUlmjlzpsaOHXvFR8/+5je/0cKFCzVy5EjNnz9f7du3V2Fhod5++2099thjat++fY3msmTJEvn5+al3795q1qyZ3nzzTfn6+pq/Hb383M8++6wmTpyouXPn6rvvvtPDDz+s+Pj4a/oFB6DhOWJOvdygQYMUFRWlUaNGacGCBeratav++9//KisrS6NGjVJERISeffZZ3X777brpppt077336ty5c/rrX/+qxx9/vMpx33zzTUVEROjWW2/V66+/rs8++8ziiQM11alTJ23atEl79uxRq1attHjxYhUXF9eqACJJd999t/70pz8pPj5ezs7OGjNmTK1jA9B4NIW8fSkPDw9NnjxZjz32mLy9veXj46Pk5GSL1c6BgYGaOHGiHnjgAS1btky9evXSN998o+PHj2vs2LH67W9/q8WLF+uJJ57Q5MmTlZ+fb74xf01W7sF+8RQY2K127drpk08+UUVFhYYMGaLu3btr1qxZ8vLyMifCRx99VE5OTgoNDVXbtm1VWFhY7fEnTpyo1NRUrVixQt26ddPw4cN16NCh+pqOpAt/9P7617/WsGHDFBsbq+7du2vFihVXPOa6667Trl271KFDB/36179WSEiIHnjgAf3yyy/mFSE1mYuHh4cWLFigiIgIRUZGqqCgQFlZWZVeSnPddddp69at+v777xUZGakxY8bo9ttv1yuvvHLtbwaABuWIOfVyJpNJWVlZuu222/TAAw+oS5cuuvfee1VQUGAu2g4YMEBvvvmm3nvvPfXu3Vu/+tWvrJ6+crl58+Zpw4YN6tmzp9atW6fXX39doaGhtY7zmWee0c0336whQ4ZowIAB8vX11ahRo2o9niSNGTNG69atU3x8vN5+++1rGgtA49AU8vblFi5cqNtuu0133XWXBg0apFtvvVXh4eEWfdLS0jRmzBglJCQoODhYU6dO1enTpyVJQUFBeuutt/T222+rZ8+eSktLMz8Fxs3NrcHng4ZnMrjoCWgU5s6dq3feeadGN84DANiWyWTS5s2br7lAAQCwjd///vdauXKljh49autQ0AC4BAYAAAAA0CSsWLFCkZGR8vb21ieffKKFCxfqoYcesnVYaCBcAgMAAFAPXnjhBXl4eFS6DR061NbhAUCTdOjQIY0cOVKhoaF67rnn9Mgjj2ju3Lm2DgsNhEtgAAAA6sH333+v77//vtLXmjdvrhtvvLGBIwIAoGmjAAIAAAAAABwel8AAAAAAAACHRwEEAAAAAAA4PAogtWQYhkpKSsQVRABQf8i1AFC/yLMAmhIKILX0008/ycvLSz/99JOtQwEAh0WuBYD6RZ4F0JRQAAEAAAAAAA6PAggAAAAAAHB4FEAAAAAAAIDDowACAAAAAAAcHgUQAAAAAADg8CiAAAAAAAAAh0cBBAAAAAAAODxnWwcA1DXDMHT69GnzfosWLWQymWwYEQAAVeP3FoDKkBuAukcBBA7n9OnTGjlypHn/3XfflYeHhw0jAgCgavzeAlAZcgNQ97gEBgAAAAAAODxWgAAAAAAAHBaXE+EiCiAAAAAAAIfF5US4yOaXwKxYsUJBQUFyd3dXeHi4du/efcX+O3fuVHh4uNzd3dWxY0etXLnS4vU1a9YoJiZGrVq1UqtWrTRo0CB99tlnFn3mzp0rk8lksfn6+tb53AAAAAAAQONg0wJIZmamZs+ereTkZOXl5SkmJkZDhw5VYWFhpf2PHDmiYcOGKSYmRnl5eXrqqac0c+ZMbdq0ydxnx44dGjdunD766CPl5OSoQ4cOio2N1bFjxyzG6tatm4qKiszbF198Ua9zBQAAAAAAtmPTS2AWL16syZMna8qUKZKk1NRUbd26VWlpaUpJSbHqv3LlSnXo0EGpqamSpJCQEO3bt0+LFi3S6NGjJUmvv/66xTFr1qzRW2+9pQ8//FATJkwwtzs7O7PqAwAAAACAJsJmK0DKy8uVm5ur2NhYi/bY2Fjt2bOn0mNycnKs+g8ZMkT79u3T2bNnKz3m559/1tmzZ9W6dWuL9kOHDqldu3YKCgrSvffeq3//+99XjLesrEwlJSUWGwAAAAAAsA82K4CcOHFCFRUV8vHxsWj38fFRcXFxpccUFxdX2v/cuXM6ceJEpcc8+eSTuvHGGzVo0CBzW58+fbR+/Xpt3bpVa9asUXFxsaKjo3Xy5Mkq401JSZGXl5d58/f3r+5UAcAu1OSeTB9//LH69esnb29vNW/eXMHBwVqyZIlVv02bNik0NFRubm4KDQ3V5s2b63MKAAAAQJVsfhPUyx8/ZBjGFR9JVFn/ytol6aWXXlJGRobefvttubu7m9uHDh2q0aNHq0ePHho0aJD+8pe/SJLWrVtX5XmTkpJ06tQp83b06NGrTw4A7ERN78nUokULPfTQQ9q1a5cOHDigp59+Wk8//bRWr15t7pOTk6O4uDjFx8dr//79io+P19ixY7V3796GmhYAAABgZrN7gLRp00ZOTk5Wqz2OHz9utcrjIl9f30r7Ozs7y9vb26J90aJFeuGFF7R9+3b17NnzirG0aNFCPXr00KFDh6rs4+bmJjc3tyuOg7oT/tj6Wh9rOlcur0v2BzyzQYaza63Gyl044eqdAAdQ03syhYWFKSwszLwfGBiot99+W7t379a0adPMYwwePFhJSUmSLhSSd+7cqdTUVGVkZDTArAAAAID/sdkKEFdXV4WHhys7O9uiPTs7W9HR0ZUeExUVZdV/27ZtioiIkIuLi7lt4cKFeu655/TBBx8oIiLiqrGUlZXpwIED8vPzq8VMAMC+1eaeTJfLy8vTnj171L9/f3NbVfdtutKY3G8JAAAA9cWml8AkJibqj3/8o1599VUdOHBAc+bMUWFhoaZPny7pwreFlz65Zfr06frmm2+UmJioAwcO6NVXX9XatWv16KOPmvu89NJLevrpp/Xqq68qMDBQxcXFKi4uVmlpqbnPo48+qp07d+rIkSPau3evxowZo5KSEk2cOLHhJg8AjURt7sl0Ufv27eXm5qaIiAjNmDHDvIJEqvq+TVcak/stAQAAoL7Y9DG4cXFxOnnypObPn6+ioiJ1795dWVlZCggIkCQVFRVZXH8eFBSkrKwszZkzR8uXL1e7du20bNky8yNwpQs38SsvL9eYMWMszvXss89q7ty5kqT//Oc/GjdunE6cOKG2bduqb9+++vTTT83nBYCmqKb3ZJKk3bt3q7S0VJ9++qmefPJJderUSePGjav1mElJSUpMTDTvl5SUUAQBANi12l7azWXdQN2zaQFEkhISEpSQkFDpa+np6VZt/fv31+eff17leAUFBVc954YNG6obHgA4vNrck+mioKAgSVKPHj307bffau7cueYCSFX3bbrSmNxvCQAAAPXF5k+BAQDYVm3uyVQZwzBUVlZm3q/qvk01GRMAAACoKzZfAQIAsL3ExETFx8crIiJCUVFRWr16tdU9mY4dO6b16y8s412+fLk6dOig4OBgSdLHH3+sRYsW6eGHHzaPOWvWLN12221asGCBRo4cqXfffVfbt2/Xxx9/3PATBAAAdo/LiXCtKIAAAGp8T6bz588rKSlJR44ckbOzs2666Sa9+OKL+u1vf2vuEx0drQ0bNujpp5/WM888o5tuukmZmZnq06dPg88PAAAAoAACAJBUs3syPfzwwxarPaoyZswYq5tSA46ott9KSnX3zSTfSgIAcGXcAwQAAAAAADg8CiAAAAAAAMDhcQkMHI7h5KJTPcdZ7AMAAAAAmjYKIHA8JlOt7+oMAAAAAHBMXAIDAAAAAAAcHgUQAAAAAADg8CiAAAAAAA1kxYoVCgoKkru7u8LDw7V79+4q+7799tsaPHiw2rZtK09PT0VFRWnr1q1W/TZt2qTQ0FC5ubkpNDRUmzdvrs8pAIDdogACAAAANIDMzEzNnj1bycnJysvLU0xMjIYOHarCwsJK++/atUuDBw9WVlaWcnNzNXDgQI0YMUJ5eXnmPjk5OYqLi1N8fLz279+v+Ph4jR07Vnv37m2oaQGA3aAAAgAAADSAxYsXa/LkyZoyZYpCQkKUmpoqf39/paWlVdo/NTVVjz/+uCIjI9W5c2e98MIL6ty5s7Zs2WLRZ/DgwUpKSlJwcLCSkpJ0++23KzU1tYFmBQD2gwIIAAAAUM/Ky8uVm5ur2NhYi/bY2Fjt2bOnWmOcP39eP/30k1q3bm1uy8nJsRpzyJAh1R4TjZfh5KJTPceZN8PJxdYhAXaPx+ACAAAA9ezEiROqqKiQj4+PRbuPj4+Ki4urNcbLL7+s06dPa+zYsea24uLiGo1ZVlamsrIy835JSUl1p4CGZjLJcHa1dRSAQ2EFCAAAANBATCaTxb5hGFZtlcnIyNDcuXOVmZmpG264odZjpqSkyMvLy7z5+/vXcAYAYL8ogACokmEYKi0tNW+GYdg6JAAA7FKbNm3k5ORktTLj+PHjVis4LpeZmanJkydr48aNGjRokMVrvr6+NRozKSlJp06dMm9Hjx6txWwAwD5xCQyAKp0+fVojR44077/77rvy8PCwYUQAANgnV1dXhYeHKzs7W3fffbe5PTs72+J37eUyMjL0wAMPKCMjQ3feeafV61FRUcrOztacOXPMbdu2bVN0dHSl47m5ucnNze0aZgLYn4v3U7l0H00TBRAAAACgASQmJio+Pl4RERGKiorS6tWrVVhYqOnTp0u6sDrj2LFjWr9+vaQLxY8JEyZo6dKl6tu3r3mlR/PmzeXl5SVJmjVrlm677TYtWLBAI0eO1Lvvvqvt27fr448/ts0kgcaI+6ng/3AJDAAAANAA4uLilJqaqvnz56t3797atWuXsrKyFBAQIEkqKipSYWGhuf+qVat07tw5zZgxQ35+fuZt1qxZ5j7R0dHasGGDXnvtNfXs2VPp6enKzMxUnz59Gnx+ANDYsQIEAAAAaCAJCQlKSEio9LX09HSL/R07dlRrzDFjxmjMmDHXGBkAOD5WgAAAAAAAAIdHAQQAAAAAADg8LoEBAACwIZ5OAABAw2AFCABAkrRixQoFBQXJ3d1d4eHh2r17d5V93377bQ0ePFht27aVp6enoqKitHXrVos+6enpMplMVtuZM2fqeyqAffm/pxNc3GQy2ToiAAAcEgUQAIAyMzM1e/ZsJScnKy8vTzExMRo6dKjF0wgutWvXLg0ePFhZWVnKzc3VwIEDNWLECOXl5Vn08/T0VFFRkcXm7u7eEFMCAAAALHAJDABAixcv1uTJkzVlyhRJUmpqqrZu3aq0tDSlpKRY9U9NTbXYf+GFF/Tuu+9qy5YtCgsLM7ebTCb5+vrWa+wAAABAddh8BUhNllxL0s6dOxUeHi53d3d17NhRK1eutHh9zZo1iomJUatWrdSqVSsNGjRIn3322TWfFwAcVXl5uXJzcxUbG2vRHhsbqz179lRrjPPnz+unn35S69atLdpLS0sVEBCg9u3ba/jw4VYrRC5XVlamkpISiw0AAACoCzZdAXJxyfWKFSvUr18/rVq1SkOHDtVXX32lDh06WPU/cuSIhg0bpqlTp+rPf/6zPvnkEyUkJKht27YaPXq0pAvPSx83bpyio6Pl7u6ul156SbGxsfryyy9144031uq8gD0Lf2x9rY81nSuX1yX7A57ZcOH69FrIXTih1nGgfp04cUIVFRXy8fGxaPfx8VFxcXG1xnj55Zd1+vRpjR071twWHBys9PR09ejRQyUlJVq6dKn69eun/fv3q3PnzpWOk5KSonnz5tV+MgAAAEAVbLoC5NIl1yEhIUpNTZW/v7/S0tIq7b9y5Up16NBBqampCgkJ0ZQpU/TAAw9o0aJF5j6vv/66EhIS1Lt3bwUHB2vNmjU6f/68Pvzww1qfFwCaAtNlN140DMOqrTIZGRmaO3euMjMzdcMNN5jb+/btq/Hjx6tXr16KiYnRxo0b1aVLF/3hD3+ocqykpCSdOnXKvB09erT2EwIAAAAuYbMCSG2WXOfk5Fj1HzJkiPbt26ezZ89WeszPP/+ss2fPmpdl13apN8uyATiqNm3ayMnJyWq1x/Hjx61WhVwuMzNTkydP1saNGzVo0KAr9m3WrJkiIyN16NChKvu4ubnJ09PTYgMAAADqgs0KILVZcl1cXFxp/3PnzunEiROVHvPkk0/qxhtvNP9hXtul3ikpKfLy8jJv/v7+V50jANgDV1dXhYeHKzs726I9Oztb0dHRVR6XkZGhSZMm6Y033tCdd9551fMYhqH8/Hz5+fldc8ywPcMwVFpaat4Mw7B1SAAAAFdk86fA1HTJdWX9K2uXpJdeekkZGRnasWOH1WMXa3repKQkJSYmmvdLSkooggBwGImJiYqPj1dERISioqK0evVqFRYWavr06ZIu5MBjx45p/foL95TJyMjQhAkTtHTpUvXt29dcQG7evLm8vC7cOWbevHnq27evOnfurJKSEi1btkz5+flavny5bSaJOnX69GmNHDnSvP/uu+/Kw8PDhhEBAABcmc0KILVZcu3r61tpf2dnZ3l7e1u0L1q0SC+88IK2b9+unj17XtN5pQvLst3c3Ko1NwCwN3FxcTp58qTmz5+voqIide/eXVlZWQoICJAkFRUVqbCw0Nx/1apVOnfunGbMmKEZM2aY2ydOnKj09HRJ0o8//qhp06apuLhYXl5eCgsL065du3TLLbc06NwAAAAAyYYFkEuXXN99993m9uzsbItvlC4VFRWlLVu2WLRt27ZNERERcnFxMbctXLhQzz//vLZu3aqIiIhrPi8ANAUJCQlKSEio9LWLRY2LduzYcdXxlixZoiVLltRBZAAAAMC1s+klMDVdcj19+nS98sorSkxM1NSpU5WTk6O1a9cqIyPDPOZLL72kZ555Rm+88YYCAwPNKz08PDzMS3Ovdl4AAAAAAOBYbFoAqemS66CgIGVlZWnOnDlavny52rVrp2XLlmn06NHmPitWrFB5ebnGjBljca5nn31Wc+fOrdZ5AQAAAACAY7H5TVBrsuRakvr376/PP/+8yvEKCgqu+bwAAAAAAMCx2LwAAqDxMpxcdKrnOIt9AAAAALBHFEAAVM1kkuHsausoAAAAAOCaNbN1AAAAAAAAAPWNAggAAAAAAHB4FEAAAAAAAIDDowACAAAAAAAcHgUQAAAAAADg8CiAAAAAAAAAh0cBBAAAAAAAODwKIAAAAAAAwOE52zoAAABgG+GPra/1saZz5fK6ZH/AMxtkOLvWeJzchRNqHQMAAEBNsAIEAAAAAAA4PAogAAAAAADA4VEAAQAAAAAADo8CCAAAAAAAcHgUQAAAAIAGsmLFCgUFBcnd3V3h4eHavXt3lX2Liop03333qWvXrmrWrJlmz55t1Sc9PV0mk8lqO3PmTD3OAgDsEwUQAAAAoAFkZmZq9uzZSk5OVl5enmJiYjR06FAVFhZW2r+srExt27ZVcnKyevXqVeW4np6eKioqstjc3d3raxoAYLcogAAAJNXsW8m3335bgwcPVtu2beXp6amoqCht3brVqt+mTZsUGhoqNzc3hYaGavPmzfU5BQBo1BYvXqzJkydrypQpCgkJUWpqqvz9/ZWWllZp/8DAQC1dulQTJkyQl5dXpX0kyWQyydfX12IDAFijAAIAqPG3krt27dLgwYOVlZWl3NxcDRw4UCNGjFBeXp65T05OjuLi4hQfH6/9+/crPj5eY8eO1d69extqWgDqiGEYKi0tNW+GYdg6JLtTXl6u3NxcxcbGWrTHxsZqz5491zR2aWmpAgIC1L59ew0fPtwiF1+urKxMJSUlFhsANBUUQAAANf5WMjU1VY8//rgiIyPVuXNnvfDCC+rcubO2bNli0Wfw4MFKSkpScHCwkpKSdPvttys1NbWBZgWgrpw+fVojR440b6dPn7Z1SHbnxIkTqqiokI+Pj0W7j4+PiouLaz1ucHCw0tPT9d577ykjI0Pu7u7q16+fDh06VGn/lJQUeXl5mTd/f/9anxtA3aLYXP8ogABAE1cX30qeP39eP/30k1q3bm1uy8nJsRpzyJAhVxyTbyYBODqTyWSxbxiGVVtN9O3bV+PHj1evXr0UExOjjRs3qkuXLvrDH/5Qaf+kpCSdOnXKvB09erTW5wZQtyg21z9nWwcAANfKMAyLXxAtWrS4pj8mm5q6+Fby5Zdf1unTpzV27FhzW3FxcY3HTElJ0bx582oQPQDYhzZt2sjJyckqBx4/ftwqV16LZs2aKTIyssoVIG5ubnJzc6uz8wGAPWEFCAC7R7W8btT2W8mMjAzNnTtXmZmZuuGGG65pTL6ZBOCoXF1dFR4eruzsbIv27OxsRUdH19l5DMNQfn6+/Pz86mxMAHAUrAABgCbuWr6VzMzM1OTJk/Xmm29q0KBBFq/5+vrWeEy+mQTgyBITExUfH6+IiAhFRUVp9erVKiws1PTp0yVdKAIfO3ZM69evNx+Tn58v6cKNTr/77jvl5+fL1dVVoaGhkqR58+apb9++6ty5s0pKSrRs2TLl5+dr+fLlDT4/AGjsKIAAQBN36beSd999t7k9OztbI0eOrPK4jIwMPfDAA8rIyNCdd95p9XpUVJSys7M1Z84cc9u2bdvq9JtOALAncXFxOnnypObPn6+ioiJ1795dWVlZCggIkCQVFRVZPX0rLCzM/O/c3Fy98cYbCggIUEFBgSTpxx9/1LRp01RcXCwvLy+FhYVp165duuWWWxpsXgBgLyiAAABq/K1kRkaGJkyYoKVLl6pv377mlR7NmzeXl5eXJGnWrFm67bbbtGDBAo0cOVLvvvuutm/fro8//tg2k0SdMpxcdKrnOIt9AFeXkJCghISESl9LT0+3arvaUyCWLFmiJUuW1EVoAODwuAcIAEBxcXFKTU3V/Pnz1bt3b+3ateuK30quWrVK586d04wZM+Tn52feZs2aZe4THR2tDRs26LXXXlPPnj2Vnp6uzMxM9enTp8Hnh3pgMslwdjVv4sbDAACgkbN5AWTFihUKCgqSu7u7wsPDtXv37iv237lzp8LDw+Xu7q6OHTtq5cqVFq9/+eWXGj16tAIDA2UymZSammo1xty5c2UymSw2X1/fupwWANidhIQEFRQUqKysTLm5ubrtttvMr6Wnp2vHjh3m/R07dsgwDKvt8m8vx4wZo3/+858qLy/XgQMH9Otf/7qBZgMAAABYsmkBJDMzU7Nnz1ZycrLy8vIUExOjoUOHWl37eNGRI0c0bNgwxcTEKC8vT0899ZRmzpypTZs2mfv8/PPP6tixo1588cUrFjW6deumoqIi8/bFF1/U+fwAAAAAAEDjYNN7gCxevFiTJ0/WlClTJEmpqanaunWr0tLSlJKSYtV/5cqV6tChg3lVR0hIiPbt26dFixZp9OjRkqTIyEhFRkZKkp588skqz+3s7MyqDwAAAAAAmgibrQApLy9Xbm6uYmNjLdpjY2O1Z8+eSo/Jycmx6j9kyBDt27dPZ8+erdH5Dx06pHbt2ikoKEj33nuv/v3vf1+xf1lZmUpKSiw2AAAAAABgH2y2AuTEiROqqKiQj4+PRbuPj4/5aQKXKy4urrT/uXPndOLECfn5+VXr3H369NH69evVpUsXffvtt3r++ecVHR2tL7/8Ut7e3pUek5KSonnz5lVrfAA1F/7Y+lofazpXLq9L9gc8s+HCTRlrKHfhhFrHAAAAADSGv2kl/q6tis1vgmq67K7xhmFYtV2tf2XtVzJ06FCNHj1aPXr00KBBg/SXv/xFkrRu3boqj0lKStKpU6fM29GjR6t9PgAAAAAAYFs2WwHSpk0bOTk5Wa32OH78uNUqj4t8fX0r7e/s7Fzlyo3qaNGihXr06KFDhw5V2cfNzU1ubm61PgcAAAAAALCdahVAli1bVu0BZ86cWa1+rq6uCg8PV3Z2tu6++25ze3Z2tkaOHFnpMVFRUdqyZYtF27Zt2xQRESEXF5dqx3i5srIyHThwQDExMbUeAwAAoLGr7dJslmUDABxBtQogS5YsqdZgJpOp2gUQSUpMTFR8fLwiIiIUFRWl1atXq7CwUNOnT5d04bKTY8eOaf36C7+sp0+frldeeUWJiYmaOnWqcnJytHbtWmVkZJjHLC8v11dffWX+97Fjx5Sfny8PDw916tRJkvToo49qxIgR6tChg44fP67nn39eJSUlmjhxYrVjBwAAAAAA9qNaBZAjR47Uy8nj4uJ08uRJzZ8/X0VFRerevbuysrIUEBAgSSoqKlJhYaG5f1BQkLKysjRnzhwtX75c7dq107Jly8yPwJWk//73vwoLCzPvL1q0SIsWLVL//v21Y8cOSdJ//vMfjRs3TidOnFDbtm3Vt29fffrpp+bzAgAAAAAAx2Kze4BclJCQoISEhEpfS09Pt2rr37+/Pv/88yrHCwwMNN8YtSobNmyoUYwAAAAAAMC+1aoA8p///EfvvfeeCgsLVV5ebvHa4sWL6yQwAAAAAACAulLjAsiHH36ou+66S0FBQTp48KC6d++ugoICGYahm2++uT5iBAAAAAAAuCbNanpAUlKSHnnkEf3jH/+Qu7u7Nm3apKNHj6p///6655576iNGAAAAAACAa1LjFSAHDhwwP3XF2dlZv/zyizw8PDR//nyNHDlSDz74YJ0HCQBXYji56FTPcRb7AAAAAHCpGhdAWrRoobKyMklSu3btdPjwYXXr1k2SdOLEibqNDgCqw2SS4exq6ygAAACAWuNLvfpX4wJI37599cknnyg0NFR33nmnHnnkEX3xxRd6++231bdv3/qIEQAAAAAAx8aXevWuxgWQxYsXq7S0VJI0d+5clZaWKjMzU506ddKSJUvqPEAAAAAAAIBrVeMCyHPPPafx48fLMAxdd911WrFiRX3EBQAAAAAAUGdq/BSYkydP6s4771T79u31yCOPKD8/vx7CAgAAAAAAqDs1LoC89957Ki4u1rPPPqvc3FyFh4crNDRUL7zwggoKCuohRAAAAAAAgGtT4wKIJF1//fWaNm2aduzYoW+++Ub333+//vSnP6lTp051HR8AoIGsWLFCQUFBcnd3V3h4uHbv3l1l36KiIt13333q2rWrmjVrptmzZ1v1SU9Pl8lkstrOnDlTj7MAAAAAKlerAshFZ8+e1b59+7R3714VFBTIx8enruICADSgzMxMzZ49W8nJycrLy1NMTIyGDh2qwsLCSvuXlZWpbdu2Sk5OVq9evaoc19PTU0VFRRabu7t7fU0DAAAAqFKtCiAfffSRpk6dKh8fH02cOFEtW7bUli1bdPTo0bqODwDQABYvXqzJkydrypQpCgkJUWpqqvz9/ZWWllZp/8DAQC1dulQTJkyQl5dXleOaTCb5+vpabAAAAIAt1PgpMO3bt9fJkyc1ZMgQrVq1SiNGjODbPACwY+Xl5crNzdWTTz5p0R4bG6s9e/Zc09ilpaUKCAhQRUWFevfureeee05hYWFV9i8rK1NZWZl5v6Sk5JrODwAAAFxU4wLI7373O91zzz1q1apVfcQDAGhgJ06cUEVFhdVljD4+PiouLq71uMHBwUpPT1ePHj1UUlKipUuXql+/ftq/f786d+5c6TEpKSmaN29erc8JoH4YTi461XOcxT4AAPamxpfATJs2jeIHADggk8lksW8YhlVbTfTt21fjx49Xr169FBMTo40bN6pLly76wx/+UOUxSUlJOnXqlHnj0kqgkTCZZDi7mjddQ24AAMBWarwCBADgWNq0aSMnJyer1R7Hjx+v05tbN2vWTJGRkTp06FCVfdzc3OTm5lZn5wQAAAAuuqanwAAA7J+rq6vCw8OVnZ1t0Z6dna3o6Og6O49hGMrPz5efn1+djQkAAABUFytAAABKTExUfHy8IiIiFBUVpdWrV6uwsFDTp0+XdOHSlGPHjmn9+vXmY/Lz8yVduNHpd999p/z8fLm6uio0NFSSNG/ePPXt21edO3dWSUmJli1bpvz8fC1fvrzB5wcAAACwAgQAoLi4OKWmpmr+/Pnq3bu3du3apaysLAUEBEiSioqKVFhYaHFMWFiYwsLClJubqzfeeENhYWEaNmyY+fUff/xR06ZNU0hIiGJjY3Xs2DHt2rVLt9xyS4PODQAakxUrVigoKEju7u4KDw/X7t27q+xbVFSk++67T127dlWzZs00e/bsSvtt2rRJoaGhcnNzU2hoqDZv3lxP0QOAfaMAAgCQJCUkJKigoEBlZWXKzc3VbbfdZn4tPT1dO3bssOhvGIbVVlBQYH59yZIl+uabb1RWVqbjx49r69atioqKaqDZAEDjk5mZqdmzZys5OVl5eXmKiYnR0KFDrQrMF5WVlalt27ZKTk5Wr169Ku2Tk5OjuLg4xcfHa//+/YqPj9fYsWO1d+/e+pwKANglCiAAAABAA1i8eLEmT56sKVOmKCQkRKmpqfL391daWlql/QMDA7V06VJNmDBBXl5elfZJTU3V4MGDlZSUpODgYCUlJen2229XampqPc4EAOwTBRAAAACgnpWXlys3N1exsbEW7bGxsdqzZ0+tx83JybEac8iQIVWOWVZWppKSEosNAJoKCiAAAABAPTtx4oQqKiqsHi/u4+Nj9RjymiguLq7RmCkpKfLy8jJv/v7+tT43ANgbCiAAAABAAzGZTBb7hmFYtdXnmElJSTp16pR5O3r06DWdGwDsCY/BBQAAAOpZmzZt5OTkZLUy4/jx41YrOGrC19e3RmO6ubnJzc2t1ucDAHvGChAAAACgnrm6uio8PFzZ2dkW7dnZ2YqOjq71uFFRUVZjbtu27ZrGBABHZfMCSE2ehS5JO3fuVHh4uNzd3dWxY0etXLnS4vUvv/xSo0ePVmBgoEwmU5V3wK7peQEAAIBrkZiYqD/+8Y969dVXdeDAAc2ZM0eFhYWaPn26pAuXp0yYMMHimPz8fOXn56u0tFTfffed8vPz9dVXX5lfnzVrlrZt26YFCxbon//8pxYsWKDt27dr9uzZDTk1ALALNi2A1PRZ6EeOHNGwYcMUExOjvLw8PfXUU5o5c6Y2bdpk7vPzzz+rY8eOevHFF+Xr61sn5wUAAACuVVxcnFJTUzV//nz17t1bu3btUlZWlgICAiRJRUVFVn+PhoWFKSwsTLm5uXrjjTcUFhamYcOGmV+Pjo7Whg0b9Nprr6lnz55KT09XZmam+vTp06BzAwB7YNN7gFz6LHTpwnPMt27dqrS0NKWkpFj1X7lypTp06GBe1RESEqJ9+/Zp0aJFGj16tCQpMjJSkZGRkqQnn3yyTs4LAAAA1IWEhAQlJCRU+lp6erpVm2EYVx1zzJgxGjNmzLWGBgAOz2YrQGrzLPSqnnO+b98+nT17tt7OCwAAAAAA7JvNVoDU5lnoVT3n/Ny5czpx4oT8/Pzq5bySVFZWprKyMvN+SUnJVc8FAAAAAAAaB5vfBLWmz0KvrH9l7XV93pSUFHl5eZk3f3//Gp0PAAAAAADYjs0KILV5FnpVzzl3dnaWt7d3vZ1XunBX7lOnTpm3o0ePVut8AAAAAADA9mxWAKnNs9Cres55RESEXFxc6u28kuTm5iZPT0+LDQAAAAAA2AebPgUmMTFR8fHxioiIUFRUlFavXm31LPRjx45p/fr1kqTp06frlVdeUWJioqZOnaqcnBytXbtWGRkZ5jHLy8vNz0YvLy/XsWPHlJ+fLw8PD3Xq1Kla5wUAAAAAAI7FpgWQuLg4nTx5UvPnz1dRUZG6d+9+xWehBwUFKSsrS3PmzNHy5cvVrl07LVu2zPwIXEn673//q7CwMPP+okWLtGjRIvXv3187duyo1nkBAAAAAIBjsWkBRKr5s9D79++vzz//vMrxAgMDq/W89CudFwAAAAAAOBabPwUGAAAAAACgvlEAAQAAAAAADo8CCAAAAAAAcHgUQAAAkqQVK1YoKChI7u7uCg8P1+7du6vsW1RUpPvuu09du3ZVs2bNNHv27Er7bdq0SaGhoXJzc1NoaKg2b95cT9EDAAAAV0YBBACgzMxMzZ49W8nJycrLy1NMTIyGDh1q8SSuS5WVlalt27ZKTk5Wr169Ku2Tk5OjuLg4xcfHa//+/YqPj9fYsWO1d+/e+pwKAAAAUCkKIAAALV68WJMnT9aUKVMUEhKi1NRU+fv7Ky0trdL+gYGBWrp0qSZMmCAvL69K+6Smpmrw4MFKSkpScHCwkpKSdPvttys1NbUeZwIAAABUjgIIADRx5eXlys3NVWxsrEV7bGys9uzZU+txc3JyrMYcMmTINY0JAAAA1JazrQMAANjWiRMnVFFRIR8fH4t2Hx8fFRcX13rc4uLiGo9ZVlamsrIy835JSUmtzw8AAABcihUgAABJkslkstg3DMOqrb7HTElJkZeXl3nz9/e/pvPbI8MwVFpaat4Mw7B1SAAAAA6BAggANHFt2rSRk5OT1cqM48ePW63gqAlfX98aj5mUlKRTp06Zt6NHj9b6/Pbq9OnTGjlypHk7ffq0rUMCAABwCBRAAKCJc3V1VXh4uLKzsy3as7OzFR0dXetxo6KirMbctm3bFcd0c3OTp6enxQYAAADUBe4BgmozDMPim8gWLVpc8/J4AI1DYmKi4uPjFRERoaioKK1evVqFhYWaPn26pAsrM44dO6b169ebj8nPz5cklZaW6rvvvlN+fr5cXV0VGhoqSZo1a5Zuu+02LViwQCNHjtS7776r7du36+OPP27w+QEAAAAUQFBtF5dlX/Tuu+/Kw8PDhhEBqCtxcXE6efKk5s+fr6KiInXv3l1ZWVkKCAiQJBUVFamwsNDimLCwMPO/c3Nz9cYbbyggIEAFBQWSpOjoaG3YsEFPP/20nnnmGd10003KzMxUnz59GmxeAAAAwEUUQAAAkqSEhAQlJCRU+lp6erpVW3VuzjlmzBiNGTPmWkMDAAAArhn3AAEAAAAAAA6PAggAAAAAAHB4XAIDAAAAoEnh5v5A00QBBAAAAECTws39gaaJAkgTFP7Y+qt3qoTpXLm8Ltkf8MwGGc6utRord+GEWh0HAAAAAEBtcA8QAAAAAADg8FgB0khwHSIAAAAAAPWHAkgjwXWIAOBYuNwQAACgceESGAAAAAAA4PAogAAAAAANZMWKFQoKCpK7u7vCw8O1e/fuK/bfuXOnwsPD5e7uro4dO2rlypUWr6enp8tkMlltZ86cqc9pAIBd4hIYAAAAoAFkZmZq9uzZWrFihfr166dVq1Zp6NCh+uqrr9ShQwer/keOHNGwYcM0depU/fnPf9Ynn3yihIQEtW3bVqNHjzb38/T01MGDBy2OdXd3r/f52FptLzWU6u5yQy41BOwLBRBUm+HkolM9x1nsAwAAoHoWL16syZMna8qUKZKk1NRUbd26VWlpaUpJSbHqv3LlSnXo0EGpqamSpJCQEO3bt0+LFi2yKICYTCb5+vo2yBwAwJ5xCQyqz2SS4exq3sRTagAAAKqlvLxcubm5io2NtWiPjY3Vnj17Kj0mJyfHqv+QIUO0b98+nT171txWWlqqgIAAtW/fXsOHD1deXl7dT6CaDMNQaWmpeTMMw2axAMDlbL4CZMWKFVq4cKGKiorUrVs3paamKiYmpsr+O3fuVGJior788ku1a9dOjz/+uKZPn27RZ9OmTXrmmWd0+PBh3XTTTfr973+vu+++2/z63LlzNW/ePItjfHx8VFxcfE1zaQzL8CSW4gEAADQ2J06cUEVFhXx8fCzar/Q3aHFxcaX9z507pxMnTsjPz0/BwcFKT09Xjx49VFJSoqVLl6pfv37av3+/OnfubDVmWVmZysrKzPslJSV1MLv/4cmGABozm64AuXgdZHJysvLy8hQTE6OhQ4eqsLCw0v4Xr4OMiYlRXl6ennrqKc2cOVObNm0y98nJyVFcXJzi4+O1f/9+xcfHa+zYsdq7d6/FWN26dVNRUZF5++KLL+p1rgAA2+ObSQC2ZrpsBa1hGFZtV+t/aXvfvn01fvx49erVSzExMdq4caO6dOmiP/zhD5WOl5KSIi8vL/Pm7+9/LdMBALti0wLIpddBhoSEKDU1Vf7+/kpLS6u0/6XXQYaEhGjKlCl64IEHtGjRInOf1NRUDR48WElJSQoODlZSUpJuv/1287WTFzk7O8vX19e8tW3btj6nCgBoBC5+M3lxO336tK1DAtBEtGnTRk5OTlarPY4fP261yuMiX1/fSvs7OzvL29u70mOaNWumyMhIHTp0qNLXk5KSdOrUKfN29OjRWswGAOyTzQog9XUdZFV9Lh/z0KFDateunYKCgnTvvffq3//+97VOCQAAAKiUq6urwsPDlZ2dbdGenZ2t6OjoSo+Jioqy6r9t2zZFRETIxaXym9EbhqH8/Hz5+flV+rqbm5s8PT0tNgBoKmx2D5D6ug6yqj6XjtmnTx+tX79eXbp00bfffqvnn39e0dHR+vLLL6usptf39ZIAAABwbImJiYqPj1dERISioqK0evVqFRYWmu9nl5SUpGPHjmn9+gv3lZs+fbpeeeUVJSYmaurUqcrJydHatWuVkZFhHnPevHnq27evOnfurJKSEi1btkz5+flavny5TeZoL3i6IdA02fwmqHV9HWR1xhw6dKj53z169FBUVJRuuukmrVu3TomJiZWeNyUlxerGqQAAAEB1xcXF6eTJk5o/f76KiorUvXt3ZWVlKSAgQJJUVFRkcS+8oKAgZWVlac6cOVq+fLnatWunZcuWWTwC98cff9S0adNUXFwsLy8vhYWFadeuXbrlllsafH525f+ebgigabFZAaS+roOsqk9VY0pSixYt1KNHjyqvlZQuVOQvLY6UlJRw0ygAAADUSEJCghISEip9LT093aqtf//++vzzz6scb8mSJVqyZEldhQcADs1mBZBLr4O89BG12dnZFo/OulRUVJS2bNli0Xb5dZAXr5WcM2eORZ+qrq2ULlzecuDAgSs+ftfNzU1ubm7VmlttsAwPgK3V9WPJ09PTdf/991sd98svv8jd3b1e5uAI+H0AoDEIf2x9rY4znSuX1yX7A57ZUOuVFrkLJ9TqOACoik2fApOYmKg//vGPevXVV3XgwAHNmTPH6jrICRP+l/imT5+ub775RomJiTpw4IBeffVVrV27Vo8++qi5z6xZs7Rt2zYtWLBA//znP7VgwQJt375ds2fPNvd59NFHtXPnTh05ckR79+7VmDFjVFJSookTJzbY3K383zK8i5uucBkQANS1+ngsuSR5enpaPHK8qKiI4sfV8PsAAACgXtj0HiD1cR1kdHS0NmzYoKefflrPPPOMbrrpJmVmZqpPnz7mPv/5z380btw4nThxQm3btlXfvn316aefms8LAE3NpY8lly48Unzr1q1KS0tTSkqKVf9LH0suSSEhIdq3b58WLVpkkZNNJpN8fX0bZA4AAADAldj8Jqh1fR2kJI0ZM0Zjxoyp8vUNGzbUKEYAcGQXH0v+5JNPWrTX5rHka9eu1dmzZ82XJZaWliogIEAVFRXq3bu3nnvuOYWFhdXPRAAAAIArsHkBBABgW/X1WPLg4GClp6erR48eKikp0dKlS9WvXz/t379fnTt3rnTc6jxyvLbXpUt1d20616UDAADYH5veAwQA0HjU9WPJ+/btq/Hjx6tXr16KiYnRxo0b1aVLF/3hD3+ocsyUlBR5eXmZN562BQAAgLpCAQQAmrj6eiz55Zo1a6bIyMirPnL81KlT5u3o0aM1nA0AAABQOS6BAYAmrr4eS345wzCUn5+vHj16VBlLfT9yHABQv3iUN4DGjBUgAIB6eSz5vHnztHXrVv373/9Wfn6+Jk+erPz8fPOYAAAHxKO8ATRirAABANTLY8l//PFHTZs2TcXFxfLy8lJYWJh27dqlW265pcHnBwAAAFAAAQBIqvvHki9ZskRLliypq/AAAACAa8IlMAAAAAAAwOFRAAEAAAAAAA6PAggAAAAAAHB43AMEANBk8HhGAACAposCCACg6fi/xzMCAACg6eESGAAAAAAA4PAogAAAAAAAAIdHAQQAAAAAADg8CiAAAAAAAMDhUQABAAAAAAAOjwIIAAAAAABweBRAAAAAAACAw6MAAgAAAAAAHB4FEAAAAAAA4PAogAAAAAAAAIdHAQQAAAAAADg8CiAAAAAAAMDhUQABAAAAAAAOz9nWAQBNlWEYOn36tHm/RYsWMplMNowIAAAAABwXBRDARk6fPq2RI0ea99999115eHjYMCIAAAAAcFw2vwRmxYoVCgoKkru7u8LDw7V79+4r9t+5c6fCw8Pl7u6ujh07auXKlVZ9Nm3apNDQULm5uSk0NFSbN2++5vMCgKOzVT4GgKaEXAtHYxiGSktLzZthGLYOCaiSTQsgmZmZmj17tpKTk5WXl6eYmBgNHTpUhYWFlfY/cuSIhg0bppiYGOXl5empp57SzJkztWnTJnOfnJwcxcXFKT4+Xvv371d8fLzGjh2rvXv31vq8AODobJWPAaApIdfCEV1c1Xxxu/QSb6CxseklMIsXL9bkyZM1ZcoUSVJqaqq2bt2qtLQ0paSkWPVfuXKlOnTooNTUVElSSEiI9u3bp0WLFmn06NHmMQYPHqykpCRJUlJSknbu3KnU1FRlZGTU6rxAVcIfW1/rY03nyuV1yf6AZzbIcHat8Ti5CyfUOgY0nMZ+zxdb5WMAaErItQDsXWP/m/ZqbFYAKS8vV25urp588kmL9tjYWO3Zs6fSY3JychQbG2vRNmTIEK1du1Znz56Vi4uLcnJyNGfOHKs+F39x1Oa8AHCtGvM9X2yVjwGgKSHXojHjSz1UV2P+m7Y6bFYAOXHihCoqKuTj42PR7uPjo+Li4kqPKS4urrT/uXPndOLECfn5+VXZ5+KYtTmvJJWVlamsrMy8f+rUKUlSSUmJua2i7Jcqj29Il8ZUmcYQpz3EKNVvnKZz5Tp37pzFWEZFRY3H4b2sO1eLUZJue7p236aZzpXL85LPO+bx12r1x4Ek7Xp+nFVby5Yta119t1U+roy95Nrq/KzYQ5z2EKNU+//f1aXK/n93KXt5L+0lzsbKEXJtdfKsZB8/K/YQo2QfcdrD37QXx7Y1e/i5lK4epyP+TVsdNn8KzOWTMwzjihOurP/l7dUZs6bnTUlJ0bx586za/f39qzzGVrz+MN3WIVyVPcQoNXCcu3bV6jDey7pjD5+3VHmcp06dkqen57VEZLN8fCl7ybX28PMs2Uec9hCjZB9x2kOMkv3EWRlHyLX2kmcl+/hZsYcYJfv4G4f3sm41WJyN8G/aK7FZAaRNmzZycnKyqk4fP37cqop9ka+vb6X9nZ2d5e3tfcU+F8eszXmlC9dTJiYmmvfPnz+v77//Xt7e3nVWoSopKZG/v7+OHj1arx/6tbCHGCXirEv2EKNkH3HWZ4wtW7as9bG2yseVIddeYA8xSvYRpz3EKNlHnPYQo0SuvdqY5Nn/sYc47SFGyT7itIcYJfuIs7Hm2eqwWQHE1dVV4eHhys7O1t13321uz87Otrim6FJRUVHasmWLRdu2bdsUEREhFxcXc5/s7GyLayG3bdum6OjoWp9Xktzc3OTm5mbRdv3111dvsjXk6enZaH/YL7KHGCXirEv2EKNkH3E2thhtlY8rQ661ZA8xSvYRpz3EKNlHnPYQo9T44mwsuZY8a80e4rSHGCX7iNMeYpTsI057iNGKYUMbNmwwXFxcjLVr1xpfffWVMXv2bKNFixZGQUGBYRiG8eSTTxrx8fHm/v/+97+N6667zpgzZ47x1VdfGWvXrjVcXFyMt956y9znk08+MZycnIwXX3zROHDggPHiiy8azs7Oxqefflrt89rKqVOnDEnGqVOnbBrHldhDjIZBnHXJHmI0DPuIszHHaKt8bAuN+XO4yB5iNAz7iNMeYjQM+4jTHmI0jMYdZ1PJtY35M7iUPcRpDzEahn3EaQ8xGoZ9xGkPMVbFpgUQwzCM5cuXGwEBAYarq6tx8803Gzt37jS/NnHiRKN///4W/Xfs2GGEhYUZrq6uRmBgoJGWlmY15ptvvml07drVcHFxMYKDg41NmzbV6Ly2Yg8/SPYQo2EQZ12yhxgNwz7ibOwx2iofN7TG/jkYhn3EaBj2Eac9xGgY9hGnPcRoGI0/zqaQaxv7Z3CRPcRpDzEahn3EaQ8xGoZ9xGkPMVbF5gUQ/M+ZM2eMZ5991jhz5oytQ6mSPcRoGMRZl+whRsOwjzjtIcamwB4+B3uI0TDsI057iNEw7CNOe4jRMOwnTkdmL5+BPcRpDzEahn3EaQ8xGoZ9xGkPMVbFZBj/dytpAAAAAAAAB9XM1gEAAAAAAADUNwogAAAAAADA4VEAwRUVFBTIZDIpPz/f1qEAgMMi1wJA/SPXAqAA0kD27NkjJycn3XHHHRbtV0vEFRUVSklJUXBwsJo3b67WrVurb9++eu211+okrkmTJslkMpk3b29v3XHHHfr73/8uSfL391dRUZG6d+9eJ+erbYyjRo2yaKvq/ZRs/55W5tI5XD6fyz+Di1tlc2sIV4v18s+ivmO5+H64uLioY8eOevTRR3X69OkGi+FKBgwYoNmzZ1u1v/POOzKZTJKkuXPnqnfv3lZ9fvzxR5lMJu3YsUPS/35uK9s+/fTTepyFYyHXXluM9pxr7SnPXh4jubZq5NnGp7HmWYlcS661Rq6tnqaUa51tHUBT8eqrr+rhhx/WH//4RxUWFqpDhw7VOm7u3LlavXq1XnnlFUVERKikpET79u3TDz/8UGex3XHHHeYkWVxcrKefflrDhw9XYWGhnJyc5OvrW2fnqiu1fT+lhnlPa+rSz+AiNzc3G0XTuFx8b86ePavdu3drypQpOn36tNLS0mwdWr3Yvn27unXrZtHm7e1to2jsD7m2bjlSriXPXllTyrXk2WvTmPOsRK4l1zZu5Frb51pWgDSA06dPa+PGjXrwwQc1fPhwpaenV/vYLVu2KCEhQffcc4+CgoLUq1cvTZ48WYmJiXUWn5ubm3x9feXr66vevXvriSee0NGjR/Xdd99ZVZ0rKio0efJkBQUFqXnz5uratauWLl1qMd6OHTt0yy23qEWLFrr++uvVr18/ffPNN3UW77W8n1LDvKc1delncHFr1aqVzeJpTC6+N/7+/rrvvvv0m9/8Ru+8847Kyso0c+ZM3XDDDXJ3d9ett96qv/3tb+bjduzYIZPJpL/85S/q1auX3N3d1adPH33xxRc2nM3VeXt7W/0suLi42Dosu0CuJddeCXn2yppSriXP1l5jz7MSuZZc27iRa22faymANIDMzEx17dpVXbt21fjx4/Xaa6+puk8f9vX11f/7f/9P3333XT1HeUFpaalef/11derUqdIK3fnz59W+fXtt3LhRX331lX73u9/pqaee0saNGyVJ586d06hRo9S/f3/9/e9/V05OjqZNm2ZeOlUXruX9lBr+PUXdat68uc6ePavHH39cmzZt0rp16/T555+rU6dOGjJkiL7//nuL/o899pgWLVqkv/3tb7rhhht011136ezZszaKHvWJXEuuRd0h16Iy9pRnJXItGj9ybcOjANIA1q5dq/Hjx0u6sOyptLRUH374YbWOXbx4sb777jv5+vqqZ8+emj59uv7617/WaXzvv/++PDw85OHhoZYtW+q9995TZmammjWz/vFwcXHRvHnzFBkZqaCgIP3mN7/RpEmTzL8oSkpKdOrUKQ0fPlw33XSTQkJCNHHixBot5buaa3k/pYZ5T2vq0s/g4vbcc8/ZNKbG6LPPPtMbb7yhgQMHKi0tTQsXLtTQoUMVGhqqNWvWqHnz5lq7dq3FMc8++6wGDx6sHj16aN26dfr222+1efNmG83g6qKjo61+FioqKmwdll0g15Jrr4Q8W32OnmvJs7XX2POsRK4l19oPcq1tUACpZwcPHtRnn32me++9V5Lk7OysuLg4vfrqq9U6PjQ0VP/4xz/06aef6v7779e3336rESNGaMqUKXUW48CBA5Wfn6/8/Hzt3btXsbGxGjp0aJXL+1auXKmIiAi1bdtWHh4eWrNmjQoLCyVJrVu31qRJkzRkyBCNGDFCS5cuVVFRUZ3Feq3vp9Qw72lNXfoZXNxmzJhhs3gak4u/SN3d3RUVFaXbbrtNDz/8sM6ePat+/fqZ+7m4uOiWW27RgQMHLI6Piooy/7t169bq2rWrVZ/GJDMz0+pnwcnJydZhNXrkWnLt1ZBnr6wp5VrybO3YQ56VyLXk2saNXGv7XMtNUOvZ2rVrde7cOd14443mNsMw5OLiUu2bEzVr1kyRkZGKjIzUnDlz9Oc//1nx8fFKTk5WUFDQNcfYokULderUybwfHh4uLy8vrVmzxip5bty4UXPmzNHLL7+sqKgotWzZUgsXLtTevXvNfV577TXNnDlTH3zwgTIzM/X0008rOztbffv2veZYr/Z+Vvcaw/p+T2vq8s8A/3OxKu7i4qJ27drJxcVF+/fvlySrJaiGYVRrWWpdLV319PTUqVOnrNp//PFHeXp6XrWPJHl5eVm0+/v787NQC+Racu3VkGevrLHmWvJs42EPeVYi15JrGzdyre2xAqQenTt3TuvXr9fLL79sUfnav3+/AgIC9Prrr9dq3NDQUEmqt0cmmUwmNWvWTL/88ovVa7t371Z0dLQSEhIUFhamTp066fDhw1b9wsLClJSUpD179qh79+564403rjmu+no/pfp/T1F7F3+RBgQEmG+c1KlTJ7m6uurjjz829zt79qz27dunkJAQi+MvfdzWDz/8oK+//lrBwcF1EltwcLD27dtn1f63v/1NXbt2Nff5z3/+o+LiYqs+zZo1a5S/GOwNuZZci2vXWHMtebZxsNc8K5Fr0biQa22PFSD16P3339cPP/ygyZMnW1XExowZo7Vr12r48OGSLiyBu1xoaKjuu+8+9evXT9HR0fL19dWRI0eUlJSkLl261Nl/xJWVlZl/kH/44Qe98sorKi0t1YgRI6z6durUSevXr9fWrVsVFBSkP/3pT/rb3/5mri4fOXJEq1ev1l133aV27drp4MGD+vrrrzVhwoRrjrM67+dDDz1kbrPle1pTl34GFzk7O6tNmzY2iaexa9GihR588EE99thjat26tTp06KCXXnpJP//8syZPnmzRd/78+fL29paPj4+Sk5PVpk2bOnvme0JCgl555RXNmDFD06ZNU/PmzZWdna21a9fqT3/6kyQpNjZWISEhuvfee/X73/9e7dq109///nc9+uijmj59ulq2bGkx5smTJ61+Fq6//nq5u7vXScyOiFxLrq0O8mzNNYZcS55tHOwlz0rkWnKt/SHXNjAD9Wb48OHGsGHDKn0tNzfXkGT+38q2I0eOGKtXrzYGDhxotG3b1nB1dTU6dOhgTJo0ySgoKKiTGCdOnGhxzpYtWxqRkZHGW2+9ZRiGYRw5csSQZOTl5RmGYRhnzpwxJk2aZHh5eRnXX3+98eCDDxpPPvmk0atXL8MwDKO4uNgYNWqU4efnZ7i6uhoBAQHG7373O6OiouKaYhw5cmS138+LMdvqPb3SHC7/98X9ymLt2rVrvcVzLbFeut+QsVzul19+MR5++GGjTZs2hpubm9GvXz/js88+M7/+0UcfGZKMLVu2GN26dTNcXV2NyMhIIz8/v05j3LdvnzFkyBDjhhtuMDw9PY2IiAgjIyPDok9RUZFx//33GwEBAUbz5s2N4OBgY/78+caZM2fMfa70c3v5eLBEriXXXhr/5f++uN+Y8uzlMZJrr4w8a3v2kGcNg1xLrrVGrq2+ppJrTYZRg+csAYCd2LFjhwYOHKgffvhB119/va3DAQCHRK4FgPpHrq073AMEAAAAAAA4PAogAAAAAADA4XEJDAAAAAAAcHisAAEAAAAAAA6PAggAAAAAAHB4FEAAAAAAAIDDowACAAAAAAAcHgUQAAAAAADg8CiAAFdQUFAgk8mk/Pz8RnOuAQMGaPbs2fUeDwA0BPIsANQ/ci1wAQUQoJHw9/dXUVGRunfvLknasWOHTCaTfvzxR9sGBgAOgjwLAPWPXIvGzNnWAQCQysvL5erqKl9fX1uHAgAOiTwLAPWPXIvGjhUgaPI++OAD3Xrrrbr++uvl7e2t4cOH6/Dhw1X2f++999S5c2c1b95cAwcO1Lp166yq2ps2bVK3bt3k5uamwMBAvfzyyxZjBAYG6vnnn9ekSZPk5eWlqVOnWiwXLCgo0MCBAyVJrVq1kslk0qRJk8zHnz9/Xo8//rhat24tX19fzZ0712J8k8mkVatWafjw4bruuusUEhKinJwc/etf/9KAAQPUokULRUVFXXGeAFBXyLMAUP/ItUA1GEAT99ZbbxmbNm0yvv76ayMvL88YMWKE0aNHD6OiosI4cuSIIcnIy8szDMMwjhw5Yri4uBiPPvqo8c9//tPIyMgwbrzxRkOS8cMPPxiGYRj79u0zmjVrZsyfP984ePCg8dprrxnNmzc3XnvtNfM5AwICDE9PT2PhwoXGoUOHjEOHDlmc69y5c8amTZsMScbBgweNoqIi48cffzQMwzD69+9veHp6GnPnzjW+/vprY926dYbJZDK2bdtmHl+SceONNxqZmZnGwYMHjVGjRhmBgYHGr371K+ODDz4wvvrqK6Nv377GHXfc0VBvM4AmjDwLAPWPXAtcHQUQ4DLHjx83JBlffPGF1S+LJ554wujevbtF/+TkZItfFvfdd58xePBgiz6PPfaYERoaat4PCAgwRo0aZdHn8nN99NFHFuNe1L9/f+PWW2+1aIuMjDSeeOIJ874k4+mnnzbv5+TkGJKMtWvXmtsyMjIMd3f3q78hAFDHyLMAUP/ItYA1LoFBk3f48GHdd9996tixozw9PRUUFCRJKiwstOp78OBBRUZGWrTdcsstFvsHDhxQv379LNr69eunQ4cOqaKiwtwWERFR65h79uxpse/n56fjx49X2cfHx0eS1KNHD4u2M2fOqKSkpNZxAEB1kGfJswDqH7mWXIur4yaoaPJGjBghf39/rVmzRu3atdP58+fVvXt3lZeXW/U1DEMmk8mqraZ9JKlFixa1jtnFxcVi32Qy6fz581X2uRhPZW2XHwcAdY08S54FUP/IteRaXB0FEDRpJ0+e1IEDB7Rq1SrFxMRIkj7++OMq+wcHBysrK8uibd++fRb7oaGhVmPs2bNHXbp0kZOTU7Vjc3V1lSSLCjsA2BvyLADUP3ItUD1cAoMmrVWrVvL29tbq1av1r3/9S//v//0/JSYmVtn/t7/9rf75z3/qiSee0Ndff62NGzcqPT1d0v+qz4888og+/PBDPffcc/r666+1bt06vfLKK3r00UdrFFtAQIBMJpPef/99fffddyotLa31PAHAVsizAFD/yLVA9VAAQZPWrFkzbdiwQbm5uerevbvmzJmjhQsXVtk/KChIb731lt5++2317NlTaWlpSk5OliS5ublJkm6++WZt3LhRGzZsUPfu3fW73/1O8+fPt3jkV3XceOONmjdvnp588kn5+PjooYceqvU8AcBWyLMAUP/ItUD1mIzKLuQCUG2///3vtXLlSh09etTWoQCAQyLPAkD9I9eiKeAeIEANrVixQpGRkfL29tYnn3yihQsXUskGgDpEngWA+keuRVNEAQSooUOHDun555/X999/rw4dOuiRRx5RUlKSrcMCAIdBngWA+keuRVPEJTAAAAAAAMDhcRNUAAAAAADg8CiAAAAAAAAAh0cBBAAAAAAAODwKIAAAAAAAwOFRAAEAAAAAAA6PAggAAAAAAHB4FEAAAAAAAIDDowACAAAAAAAcHgUQAAAAAADg8P4/Eg3lbmmpkl0AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1091.11x300 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = sns.catplot(x = \"algorithm\", y = \"val\", data = pltData, kind=\"bar\", col = \"metric\", aspect=1.2, height=3, sharey=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e32b381",
   "metadata": {
    "papermill": {
     "duration": 0.006261,
     "end_time": "2024-12-07T21:06:46.876715",
     "exception": false,
     "start_time": "2024-12-07T21:06:46.870454",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Prediction RMSE\n",
    "\n",
    "We will also look at the prediction RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "fb282b1c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:46.889592Z",
     "iopub.status.busy": "2024-12-07T21:06:46.889347Z",
     "iopub.status.idle": "2024-12-07T21:06:48.300633Z",
     "shell.execute_reply": "2024-12-07T21:06:48.300233Z"
    },
    "papermill": {
     "duration": 1.41872,
     "end_time": "2024-12-07T21:06:48.301495",
     "exception": false,
     "start_time": "2024-12-07T21:06:46.882775",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "preds = preds.rename(columns={'score': 'prediction'})\n",
    "user_rmse = preds.groupby(['algorithm', 'user']).apply(lambda df: rmse(df['prediction'], df['rating']))\n",
    "user_rmse = user_rmse.reset_index(name='RMSE')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "73116a6d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:48.317857Z",
     "iopub.status.busy": "2024-12-07T21:06:48.317635Z",
     "iopub.status.idle": "2024-12-07T21:06:48.586581Z",
     "shell.execute_reply": "2024-12-07T21:06:48.586182Z"
    },
    "papermill": {
     "duration": 0.278443,
     "end_time": "2024-12-07T21:06:48.587338",
     "exception": false,
     "start_time": "2024-12-07T21:06:48.308895",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x1555303bfee0>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAHpCAYAAACFlZVCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAf7UlEQVR4nO3df5CU9X3A8c/y66DAnUBURA/EEvkhQVPAiD/BGJUKQmfahviLs7GWghohJoha4o8YMsPoJB0aIhZRTBBuNAnWRBoaiyZBIlCpFhUwhUIsJ1blTq5yDsfTPxy3HodyB3e3d19er5md5Hnuud3PPvOsb/bZvd1clmVZAADJalfoAQCA5iX2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBIXJuOfZZlUVVVFT4qAAA+WZuO/XvvvRclJSXx3nvvFXoUAGi12nTsAYBDE3sASJzYA0DixB4AEif2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJE7sASBxYg8AietQ6AEAaD5ZlkV1dXV+uWvXrpHL5Qo4EYUg9gAJq66ujgkTJuSXly9fHt26dSvgRBSC0/gAkDixB4DEiT0AJE7sASBxYg8AiRN7AEic2ANA4sQeABIn9gCQOLEHgMSJPQAkTuwBIHG+CAegBQ3/xuIWvb3cvg+i5GPLo/9uaWQdOrXIba+fe02L3A6H5pk9ACRO7AEgcWIPAIkTewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJE7sASBxYg8AiRN7AEicL8IBSFjWvmNUDvtKnWWOPmIPkLJcrsW+5Y7Wy2l8AEic2ANA4sQeABIn9gCQOLEHgMS1mtjPmTMncrlc3HzzzYUeBQCS0ipiv3bt2liwYEEMGzas0KMAQHIKHvs9e/bElVdeGQ8++GD06NHjU7etqamJqqqqOhcA4NMVPPbTpk2Lyy67LC666KJDbjtnzpwoKSnJX0pLS1tgQgBo2woa+6VLl8a//du/xZw5cxq0/axZs6KysjJ/2bFjRzNPCABtX8E+LnfHjh3xta99LX75y19G586dG/Q7RUVFUVRU1MyTAUBaChb79evXx65du2L48OH5dbW1tfHcc8/FvHnzoqamJtq3b1+o8QAgGQWL/Re/+MV4+eWX66y79tprY9CgQTFz5kyhB4AmUrDYd+/ePYYOHVpnXdeuXaNXr1711gMAh6/g78YHAJpXq/o++1WrVhV6BABIjmf2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJE7sASBxYg8AiRN7AEic2ANA4sQeABIn9gCQOLEHgMSJPQAkTuwBIHFiDwCJE3sASJzYA0DixB4AEif2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJE7sASBxYg8AiRN7AEic2ANA4sQeABIn9gCQOLEHgMSJPQAkTuwBIHFiDwCJE3sASJzYA0DixB4AEif2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJE7sASBxYg8AiRN7AEic2ANA4joUegCgvizLorq6Or/ctWvXyOVyBZwIaMvEHlqh6urqmDBhQn55+fLl0a1btwJOBLRlTuMDQOLEHgASJ/YAkDixB4DEeYMeNMDwbyxu0dvL7fsgSj62PPrvlkbWoVOL3Pb6ude0yO0ALcczewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJM6f3kErlLXvGJXDvlJnGeBwiT20Rrlci/1dPZA+p/EBIHFiDwCJE3sASJzYA0DiChr7+fPnx7Bhw6K4uDiKi4tj1KhR8fTTTxdyJABITkFjf9JJJ8V3v/vdWLduXaxbty4uvPDCmDBhQmzcuLGQYwFAUgr6p3fjx4+vs3zvvffG/PnzY82aNXHaaafV276mpiZqamryy1VVVc0+IwC0da3mNfva2tpYunRpVFdXx6hRow66zZw5c6KkpCR/KS0tbeEpAaDtKXjsX3755ejWrVsUFRXFlClT4qc//WkMGTLkoNvOmjUrKisr85cdO3a08LQA0PYU/BP0Bg4cGBs2bIjdu3fHE088EZMnT45nn332oMEvKiqKoqKiAkwJAG1XwWPfqVOnGDBgQEREjBgxItauXRvf//7344EHHijwZACQhoKfxj9QlmV13oQHAByZgj6zv+2222Ls2LFRWloa7733XixdujRWrVoVK1asKORYAJCUgsb+zTffjKuvvjp27twZJSUlMWzYsFixYkV86UtfKuRYAJCUgsZ+4cKFhbx5ADgqtLrX7AGApiX2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJE7sASBxYg8AiRN7AEic2ANA4sQeABIn9gCQOLEHgMSJPQAkTuwBIHFiDwCJE3sASJzYA0DixB4AEif2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJE7sASBxYg8AiRN7AEic2ANA4sQeABIn9gCQuEbF/oUXXoja2tr8cpZldX5eU1MT5eXlTTMZANAkGhX7UaNGxdtvv51fLikpif/8z//ML+/evTu+8pWvNN10AMARa1TsD3wmf+DyJ60DAAqnyV+zz+VyTX2VAMAR8AY9AEhch8b+wiuvvBIVFRUR8eEp+9deey327NkTERH/8z//07TTAQBHrNGx/+IXv1jndflx48ZFxIen77MscxofAFqZRsV+69atzTUHANBMGhX7fv36NdccAEAzadQb9N555534wx/+UGfdxo0b49prr42//Mu/jCVLljTpcADAkWtU7KdNmxb3339/fnnXrl1x3nnnxdq1a6OmpibKysri0UcfbfIhAYDD16jYr1mzJi6//PL88uLFi6Nnz56xYcOGWL58eXznO9+Jf/iHf2jyIQGAw9eo2FdUVET//v3zy88880z82Z/9WXTo8OFL/5dffnls2bKlaScEAI5Io2JfXFwcu3fvzi+/8MILcdZZZ+WXc7lc1NTUNNlwAMCRa1TszzzzzPj7v//72L9/fzz++OPx3nvvxYUXXpj/+ebNm6O0tLTJhwQADl+j/vTunnvuiYsuuih+9KMfxb59++K2226LHj165H++dOnSuOCCC5p8SADg8DUq9meccUa8+uqrsXr16ujdu3d84QtfqPPzSZMmxZAhQ5p0QADgyDT643KPPfbYmDBhwkF/dtlllx3xQABA02pU7BcvXtyg7a655prDGgYAaHqNin1ZWVl069YtOnToUOfLcD4ul8uJPQC0Io2K/eDBg+PNN9+Mq666Kv7qr/4qhg0b1lxzAQBNpFF/erdx48b4+c9/Hu+//36cf/75MWLEiJg/f35UVVU113wAwBFqVOwjIr7whS/EAw88EDt37oybbropysvL44QTTogrr7zSB+oAQCvU6Nh/pEuXLnHNNdfEXXfdFWeeeWYsXbo0/vd//7cpZwMAmsBhxf6NN96I73znO/HZz342Jk2aFCNHjoyNGzfW+YAdAKB1aNQb9MrLy2PRokXx7LPPxiWXXBL33XdfXHbZZdG+ffvmmg8AOEKNiv2kSZOib9++MX369Dj++ONj27ZtB/1K25tuuqnJBgQAjkyjYt+3b9/I5XKxZMmST9wml8uJPQC0Io2K/bZt2w65zRtvvHG4swAAzeCw341/oIqKirjppptiwIABTXWVAEATaFTsd+/eHVdeeWUce+yx0adPn/x328+ePTtOOeWUeP755+Ohhx5qrlkBgMPQqNP4t912Wzz33HMxefLkWLFiRUyfPj1WrFgRe/fujaefftp32QNAK9So2P/85z+PRYsWxUUXXRRTp06NAQMGxKmnnhrf+973mmk8AOBINeo0/n//93/HkCFDIiLilFNOic6dO8d1113XLIMBAE2jUbHfv39/dOzYMb/cvn376Nq1a5MPBQA0nUadxs+yLMrKyqKoqCgiIvbu3RtTpkypF/yf/OQnTTchAHBEGhX7yZMn11m+6qqrmnQYAKDpNSr2ixYtaq45AIBm0mQfqgMAtE5iDwCJK2js58yZEyNHjozu3bvHcccdFxMnToxNmzYVciQASE5BY//ss8/GtGnTYs2aNbFy5crYt29fXHzxxVFdXV3IsQAgKY16g15TW7FiRZ3lRYsWxXHHHRfr16+P888/v972NTU1UVNTk1+uqqpq9hkBoK1rVa/ZV1ZWRkREz549D/rzOXPmRElJSf5SWlrakuMBQJvUamKfZVnMmDEjzj333Bg6dOhBt5k1a1ZUVlbmLzt27GjhKQGg7SnoafyPu+GGG+Kll16K3/zmN5+4TVFRUf7T+wCAhmkVsb/xxhvjySefjOeeey5OOumkQo8DAEkpaOyzLIsbb7wxfvrTn8aqVauif//+hRwHAJJU0NhPmzYtlixZEsuXL4/u3btHRUVFRESUlJREly5dCjkaACSjoG/Qmz9/flRWVsbo0aPjhBNOyF+WLVtWyLEAICkFP40PADSvVvOndwBA8xB7AEic2ANA4sQeABIn9gCQOLEHgMSJPQAkTuwBIHFiDwCJE3sASJzYA0DixB4AEif2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJE7sASBxYg8AiRN7AEic2ANA4sQeABIn9gCQOLEHgMSJPQAkTuwBIHFiDwCJE3sASJzYA0DixB4AEif2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJE7sASBxYg8AiRN7AEic2ANA4sQeABIn9gCQOLEHgMSJPQAkTuwBIHFiDwCJE3sASJzYA0DixB4AEif2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJE7sASBxYg8AiRN7AEic2ANA4sQeABIn9gCQOLEHgMSJPQAkTuwBIHFiDwCJK2jsn3vuuRg/fnz06dMncrlc/OxnPyvkOACQpILGvrq6Ok4//fSYN29eIccAgKR1KOSNjx07NsaOHVvIEQAgeQWNfWPV1NRETU1NfrmqqqqA0wBA29Cm3qA3Z86cKCkpyV9KS0sLPRIAtHptKvazZs2KysrK/GXHjh2FHgkAWr02dRq/qKgoioqKCj0GALQpbeqZPQDQeAV9Zr9nz554/fXX88tbt26NDRs2RM+ePaNv374FnAwA0lHQ2K9bty7GjBmTX54xY0ZEREyePDkefvjhAk0FAGkpaOxHjx4dWZYVcgQASJ7X7AEgcWIPAIlrU396BwBNLcuyqK6uzi937do1crlcASdqemJPizoaHlRA21JdXR0TJkzILy9fvjy6detWwImantjToo6GBxVAa+M1ewBInGf2Tcgpamg5Hm/QcGLfhJyihpbj8Za24d9Y3GK3ldv3QZR8bHn03y2NrEOnFrnt9XOvaZHbEXuOmgdVRMs9sABaE6/ZA0DiPLMHmoyzRNA6JR37lvwPT8TR8boPAG1P0rEHgEPJ2neMymFfqbOcGrGnRR0NDyqgjcnlWvQloEIQe1rWUfCgAmhtxL4JedYKLcfjDRpO7JuSZ63QcjzeoMH8nT0AJE7sASBxYg8AiRN7AEic2ANA4sQeABIn9gCQOLEHgMSJPQAkTuwBIHFiDwCJE3sASJzYA0DixB4AEif2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJE7sASBxYg8AiRN7AEic2ANA4sQeABIn9gCQOLEHgMSJPQAkTuwBIHFiDwCJE3sASJzYA0DixB4AEif2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBInNgDQOLEHgASJ/YAkDixB4DEiT0AJE7sASBxYg8AiRN7AEic2ANA4sQeABIn9gCQOLEHgMSJPQAkTuwBIHFiDwCJE3sASJzYA0DixB4AEif2AJA4sQeAxIk9ACSu4LH/wQ9+EP3794/OnTvH8OHD49e//nWhRwKApBQ09suWLYubb745br/99njxxRfjvPPOi7Fjx8b27dsLORYAJKWgsb///vvjq1/9alx33XUxePDg+N73vhelpaUxf/78Qo4FAEnpUKgb/uCDD2L9+vVx66231ll/8cUXx+rVqw/6OzU1NVFTU5NfrqysjIiIqqqqg25fW/N+E03b+n3SPmgI++nQ7KOGsZ8OzT5qmKNlPzVkH3Xv3j1yudyR3VBWIG+88UYWEdlvf/vbOuvvvffe7NRTTz3o73zrW9/KIsLFxcXFxeWouVRWVh5xcwv2zP4jB/5rJcuyT/wXzKxZs2LGjBn55f3798c777wTvXr1OvJ/9TSRqqqqKC0tjR07dkRxcXGhx2mV7KOGsZ8OzT5qGPvp0FrzPurevfsRX0fBYv+Zz3wm2rdvHxUVFXXW79q1K44//viD/k5RUVEUFRXVWXfMMcc014hHpLi4uNUdMK2NfdQw9tOh2UcNYz8dWqr7qGBv0OvUqVMMHz48Vq5cWWf9ypUr4+yzzy7QVACQnoKexp8xY0ZcffXVMWLEiBg1alQsWLAgtm/fHlOmTCnkWACQlILG/stf/nK8/fbbcffdd8fOnTtj6NCh8Ytf/CL69etXyLGOSFFRUXzrW9+q93ID/88+ahj76dDso4axnw4t9X2Uy7IsK/QQAEDzKfjH5QIAzUvsASBxYg8AiRN7mt22bdsil8vFhg0bCj0KwFFJ7Bth9erV0b59+7j00kvrrD9UzGpra2POnDkxaNCg6NKlS/Ts2TPOOuusWLRoUQtM3fzKysoil8vlL7169YpLL700XnrppYiIKC0tzf+1xdGqrKwsJk6cWGfdJx1PEY6pA318/x24Lw88/j66HGy/Hi0Otb8OPBZTNHr06Lj55pvrrf/Zz36W/8TVO++8M84444x62+zevTtyuVysWrUqIv7/8Xiwy5o1a5rxXjSdgn9cblvy0EMPxY033hj/+I//GNu3b4++ffs26PfuvPPOWLBgQcybNy9GjBgRVVVVsW7dunj33XebeeKWc+mll+ZDU1FREXfccUeMGzcutm/fHu3bt4/evXsXeMLW53CPp4ij45hqjI8ffx9J9U+oKJx/+Zd/idNOO63Oul69ehVomsYR+waqrq6O8vLyWLt2bVRUVMTDDz8cs2fPbtDv/tM//VNMnTo1/uIv/iK/7vTTT2+uUQuiqKgoH/TevXvHzJkz4/zzz4+33norqquro3///vHiiy/GGWecEbW1tXH99dfHM888ExUVFdG3b9+YOnVqfO1rX8tf36pVq+Kb3/xmbNy4MTp27BinnXZaLFmypE1/BsPHHcnxFHF0HFON8fHjD5pLr1692uxx5jR+Ay1btiwGDhwYAwcOjKuuuioWLVoUDf2Igt69e8czzzwTb731VjNP2Trs2bMnfvzjH8eAAQMO+q/e/fv3x0knnRTl5eXxyiuvxOzZs+O2226L8vLyiIjYt29fTJw4MS644IJ46aWX4vnnn4/rr7++1XzZUVM4kuMp4ug7poAjI/YNtHDhwrjqqqsi4sNThnv27Ilf/epXDfrd+++/P956663o3bt3DBs2LKZMmRJPP/10c47b4p566qno1q1bdOvWLbp37x5PPvlkLFu2LNq1q3+IdezYMe66664YOXJk9O/fP6688sooKyvLx76qqioqKytj3Lhx8cd//McxePDgmDx5cqNOc7d2R3I8RRwdx1RjfPz4++hyzz33FHosEnP22WfXO85qa2sLPVaDiH0DbNq0KV544YWYNGlSRER06NAhvvzlL8dDDz3UoN8fMmRI/Md//EesWbMmrr322njzzTdj/Pjxcd111zXn2C1qzJgxsWHDhtiwYUP87ne/i4svvjjGjh0b//Vf/3XQ7X/4wx/GiBEj4thjj41u3brFgw8+GNu3b4+IiJ49e0ZZWVlccsklMX78+Pj+978fO3fubMm706yO9HiKODqOqcb4+PH30WXatGmFHovELFu2rN5x1r59+0KP1SBes2+AhQsXxr59++LEE0/Mr8uyLDp27NjgN0S1a9cuRo4cGSNHjozp06fHj370o7j66qvj9ttvj/79+zfX6C2ma9euMWDAgPzy8OHDo6SkJB588MF6ASovL4/p06fHfffdF6NGjYru3bvH3Llz43e/+11+m0WLFsVNN90UK1asiGXLlsUdd9wRK1eujLPOOqvF7lNzOdTx1KNHjwZdT+rHVGMcePxBcXFxVFZW1lu/e/fu/FfYfto2ERElJSV11peWlrbZ48wz+0PYt29fLF68OO677746/5r793//9+jXr1/8+Mc/PqzrHTJkSER8+EatFOVyuWjXrl28//779X7261//Os4+++yYOnVqfP7zn48BAwbE73//+3rbff7zn49Zs2bF6tWrY+jQobFkyZKWGL1ZNdfxFJH+MQWNMWjQoFi3bl299WvXro2BAwfmt/nDH/4QFRUV9bZp165dmw37wXhmfwhPPfVUvPvuu/HVr3613r/y/vzP/zwWLlwY48aNi4gPT88eaMiQIXHFFVfEOeecE2effXb07t07tm7dGrNmzYpTTz01Bg0a1CL3o7nV1NTkHzDvvvtuzJs3L/bs2RPjx4+vt+2AAQNi8eLF8c///M/Rv3//ePTRR2Pt2rX5Z6Nbt26NBQsWxOWXXx59+vSJTZs2xebNm+Oaa65p0fvUHBpyPN1www35dUfzMdUYHz/+PtKhQ4f4zGc+U6CJKLSpU6fGvHnzYtq0aXH99ddHly5dYuXKlbFw4cJ49NFHIyLi4osvjsGDB8ekSZPi3nvvjT59+sRLL70Ut9xyS0yZMiW6d+9e5zrffvvtesfZMcccE507d26x+3XYMj7VuHHjsj/90z896M/Wr1+fRUT+fw922bp1a7ZgwYJszJgx2bHHHpt16tQp69u3b1ZWVpZt27athe9N85g8eXKd+9y9e/ds5MiR2eOPP55lWZZt3bo1i4jsxRdfzLIsy/bu3ZuVlZVlJSUl2THHHJP97d/+bXbrrbdmp59+epZlWVZRUZFNnDgxO+GEE7JOnTpl/fr1y2bPnp3V1tYW6B4eucmTJ2cTJkxo8PH00T47Wo+pA320/w78/x8tH2w/DRw4sDDDtgKH2l8fX07ZunXrsksuuSQ77rjjsuLi4mzEiBHZY489VmebnTt3Ztdee23Wr1+/rEuXLtmgQYOyu+++O9u7d29+m097PB54fa2Vr7gFgMR5zR4AEif2AJA4sQeAxIk9ACRO7AEgcWIPAIkTewBInNgDQOLEHhKybdu2yOVysWHDhlZzW6NHj46bb7652ecBPpnYA4eltLQ0du7cGUOHDo2IiFWrVkUul8t/YxjQevgiHKDRPvjgg+jUqVP07t270KMADeCZPbQxK1asiHPPPTeOOeaY6NWrV4wbN+6gXxH8kSeffDI++9nPRpcuXWLMmDHxyCOP1HsG/sQTT8Rpp50WRUVFcfLJJ8d9991X5zpOPvnk+Pa3vx1lZWVRUlISf/3Xf13nNP62bdtizJgxERHRo0ePyOVyUVZWlv/9/fv3xze/+c3o2bNn9O7dO+68884615/L5eKBBx6IcePGxR/90R/F4MGD4/nnn4/XX389Ro8eHV27do1Ro0Z96v0EPkWhv4kHaJzHH388e+KJJ7LNmzdnL774YjZ+/Pjsc5/7XFZbW1vvGwa3bt2adezYMbvllluy1157LXvssceyE088MYuI7N13382y7MNvBmvXrl129913Z5s2bcoWLVqUdenSJVu0aFH+Nvv165cVFxdnc+fOzbZs2ZJt2bKlzm3t27cve+KJJ7KIyDZt2pTt3Lkz2717d5ZlWXbBBRdkxcXF2Z133plt3rw5e+SRR7JcLpf98pe/zF9/RGQnnnhitmzZsmzTpk3ZxIkTs5NPPjm78MILsxUrVmSvvPJKdtZZZ2WXXnppS+1mSIrYQxu3a9euLCKyl19+uV7sZ86cmQ0dOrTO9rfffnud2F9xxRXZl770pTrbfOMb38iGDBmSX+7Xr182ceLEOtsceFv/+q//Wud6P3LBBRdk5557bp11I0eOzGbOnJlfjojsjjvuyC8///zzWURkCxcuzK977LHHss6dOx96hwD1OI0Pbczvf//7uOKKK+KUU06J4uLi6N+/f0REbN++vd62mzZtipEjR9ZZd+aZZ9ZZfvXVV+Occ86ps+6cc86JLVu2RG1tbX7diBEjDnvmYcOG1Vk+4YQTYteuXZ+4zfHHHx8REZ/73OfqrNu7d29UVVUd9hxwtPIGPWhjxo8fH6WlpfHggw9Gnz59Yv/+/TF06ND44IMP6m2bZVnkcrl66xq7TURE165dD3vmjh071lnO5XKxf//+T9zmo3kOtu7A3wMOTeyhDXn77bfj1VdfjQceeCDOO++8iIj4zW9+84nbDxo0KH7xi1/UWbdu3bo6y0OGDKl3HatXr45TTz012rdv3+DZOnXqFBFR52wA0Do4jQ9tSI8ePaJXr16xYMGCeP311+OZZ56JGTNmfOL2f/M3fxOvvfZazJw5MzZv3hzl5eXx8MMPR8T/P1P++te/Hr/61a/innvuic2bN8cjjzwS8+bNi1tuuaVRs/Xr1y9yuVw89dRT8dZbb8WePXsO+34CTUvsoQ1p165dLF26NNavXx9Dhw6N6dOnx9y5cz9x+/79+8fjjz8eP/nJT2LYsGExf/78uP322yMioqioKCIi/uRP/iTKy8tj6dKlMXTo0Jg9e3bcfffddf50riFOPPHEuOuuu+LWW2+N448/Pm644YbDvp9A08plB3txDkjWvffeGz/84Q9jx44dhR4FaCFes4fE/eAHP4iRI0dGr1694re//W3MnTvXs244yog9JG7Lli3x7W9/O955553o27dvfP3rX49Zs2YVeiygBTmNDwCJ8wY9AEic2ANA4sQeABIn9gCQOLEHgMSJPQAkTuwBIHFiDwCJ+z/7To/H0JbrUAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 511.111x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.catplot(x='algorithm', y='RMSE', data=user_rmse, kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffbd7ba0",
   "metadata": {
    "papermill": {
     "duration": 0.006178,
     "end_time": "2024-12-07T21:06:48.600051",
     "exception": false,
     "start_time": "2024-12-07T21:06:48.593873",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Save Metrics\n",
    "\n",
    "We'll now save the top-N recommendation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1b776675",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:48.613636Z",
     "iopub.status.busy": "2024-12-07T21:06:48.613363Z",
     "iopub.status.idle": "2024-12-07T21:06:48.620865Z",
     "shell.execute_reply": "2024-12-07T21:06:48.620488Z"
    },
    "papermill": {
     "duration": 0.015262,
     "end_time": "2024-12-07T21:06:48.621667",
     "exception": false,
     "start_time": "2024-12-07T21:06:48.606405",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RecipRank</th>\n",
       "      <th>NDCG</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ALS</th>\n",
       "      <td>0.050323</td>\n",
       "      <td>0.042460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bias</th>\n",
       "      <td>0.078126</td>\n",
       "      <td>0.054602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IALS</th>\n",
       "      <td>0.218723</td>\n",
       "      <td>0.219548</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IIE</th>\n",
       "      <td>0.000719</td>\n",
       "      <td>0.001999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>III</th>\n",
       "      <td>0.290649</td>\n",
       "      <td>0.215907</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pop</th>\n",
       "      <td>0.178859</td>\n",
       "      <td>0.136401</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UUE</th>\n",
       "      <td>0.000137</td>\n",
       "      <td>0.000327</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           RecipRank      NDCG\n",
       "algorithm                     \n",
       "ALS         0.050323  0.042460\n",
       "Bias        0.078126  0.054602\n",
       "IALS        0.218723  0.219548\n",
       "IIE         0.000719  0.001999\n",
       "III         0.290649  0.215907\n",
       "Pop         0.178859  0.136401\n",
       "UUE         0.000137  0.000327"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "saved = results[['recip_rank', 'ndcg']].rename(columns={'ndcg': 'NDCG', 'recip_rank': 'RecipRank'})\n",
    "saved = saved.groupby('algorithm').agg('mean')\n",
    "saved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c4083a95",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T21:06:48.634960Z",
     "iopub.status.busy": "2024-12-07T21:06:48.634746Z",
     "iopub.status.idle": "2024-12-07T21:06:48.638676Z",
     "shell.execute_reply": "2024-12-07T21:06:48.638280Z"
    },
    "papermill": {
     "duration": 0.011441,
     "end_time": "2024-12-07T21:06:48.639365",
     "exception": false,
     "start_time": "2024-12-07T21:06:48.627924",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "saved.to_json(f'eval-metrics.{dataset}.json', orient='index')"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md",
   "notebook_metadata_filter": "split_at_heading"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 6.569449,
   "end_time": "2024-12-07T21:06:49.061547",
   "environment_variables": {},
   "exception": null,
   "input_path": "eval-report.ipynb",
   "output_path": "eval-report.mlsmall.ipynb",
   "parameters": {
    "dataset": "mlsmall"
   },
   "start_time": "2024-12-07T21:06:42.492098",
   "version": "2.6.0"
  },
  "split_at_heading": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}