eval-report.ml1m.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ce485e34",
   "metadata": {
    "papermill": {
     "duration": 0.005331,
     "end_time": "2024-12-07T20:56:54.898878",
     "exception": false,
     "start_time": "2024-12-07T20:56:54.893547",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Evaluation Analysis for Recommender Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17c9a5e6",
   "metadata": {
    "papermill": {
     "duration": 0.004349,
     "end_time": "2024-12-07T20:56:54.908492",
     "exception": false,
     "start_time": "2024-12-07T20:56:54.904143",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this section we will analyze and compare the generated recommendations and predictions from a predefined list of algorithms with the goal of assessing the performance of each algorithm with respect to a metric. In other words, we would rank the algorithms for each metric considered with respect to performance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f14bb433",
   "metadata": {
    "papermill": {
     "duration": 0.004561,
     "end_time": "2024-12-07T20:56:54.917348",
     "exception": false,
     "start_time": "2024-12-07T20:56:54.912787",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b8308de",
   "metadata": {
    "papermill": {
     "duration": 0.004256,
     "end_time": "2024-12-07T20:56:54.925850",
     "exception": false,
     "start_time": "2024-12-07T20:56:54.921594",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Below are the list of packages required to successfully run the analysis. They are divided into partitions to signify their specific task.<br>\n",
    "We need the pathlib package for working with files and folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "77d2e6e9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:56:54.935778Z",
     "iopub.status.busy": "2024-12-07T20:56:54.935563Z",
     "iopub.status.idle": "2024-12-07T20:56:54.940295Z",
     "shell.execute_reply": "2024-12-07T20:56:54.939925Z"
    },
    "papermill": {
     "duration": 0.010964,
     "end_time": "2024-12-07T20:56:54.941054",
     "exception": false,
     "start_time": "2024-12-07T20:56:54.930090",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8607dbba",
   "metadata": {
    "papermill": {
     "duration": 0.004603,
     "end_time": "2024-12-07T20:56:54.950133",
     "exception": false,
     "start_time": "2024-12-07T20:56:54.945530",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We would use the pandas for analyzing and manipulating our data while seaborn and matplotlib are used for data visualization. statsmodels.graphics.gofplots and scipy.stats.shapiro are used for normality check. Scipy.stats.friedmanchisquare is a non-parametric test used to determine the statistical significance in metric results and the wilcoxon test is used for pairwise comparison of sample data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "54c9b1a0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:56:54.959677Z",
     "iopub.status.busy": "2024-12-07T20:56:54.959466Z",
     "iopub.status.idle": "2024-12-07T20:57:04.611953Z",
     "shell.execute_reply": "2024-12-07T20:57:04.611498Z"
    },
    "papermill": {
     "duration": 9.658426,
     "end_time": "2024-12-07T20:57:04.612971",
     "exception": false,
     "start_time": "2024-12-07T20:56:54.954545",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from statsmodels.graphics.gofplots import qqplot\n",
    "from scipy.stats import shapiro\n",
    "from scipy.stats import friedmanchisquare, wilcoxon\n",
    "from itertools import combinations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09d4299f",
   "metadata": {
    "papermill": {
     "duration": 0.004873,
     "end_time": "2024-12-07T20:57:04.623511",
     "exception": false,
     "start_time": "2024-12-07T20:57:04.618638",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Logging to show what's happening in LensKit routines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fd84093f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:04.633265Z",
     "iopub.status.busy": "2024-12-07T20:57:04.632962Z",
     "iopub.status.idle": "2024-12-07T20:57:04.653267Z",
     "shell.execute_reply": "2024-12-07T20:57:04.652878Z"
    },
    "papermill": {
     "duration": 0.025967,
     "end_time": "2024-12-07T20:57:04.654009",
     "exception": false,
     "start_time": "2024-12-07T20:57:04.628042",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.util.log notebook logging configured\n"
     ]
    }
   ],
   "source": [
    "from lenskit import util\n",
    "util.log_to_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c7c3c1f0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:04.664288Z",
     "iopub.status.busy": "2024-12-07T20:57:04.664070Z",
     "iopub.status.idle": "2024-12-07T20:57:04.666400Z",
     "shell.execute_reply": "2024-12-07T20:57:04.666014Z"
    },
    "papermill": {
     "duration": 0.008278,
     "end_time": "2024-12-07T20:57:04.667095",
     "exception": false,
     "start_time": "2024-12-07T20:57:04.658817",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "_log = logging.getLogger('eval-report')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af5b4ebc",
   "metadata": {
    "papermill": {
     "duration": 0.004419,
     "end_time": "2024-12-07T20:57:04.675965",
     "exception": false,
     "start_time": "2024-12-07T20:57:04.671546",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We will use lenskit for training, running, and evaluating recommender algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "803d7eb5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:04.685285Z",
     "iopub.status.busy": "2024-12-07T20:57:04.685077Z",
     "iopub.status.idle": "2024-12-07T20:57:04.716838Z",
     "shell.execute_reply": "2024-12-07T20:57:04.716469Z"
    },
    "papermill": {
     "duration": 0.037343,
     "end_time": "2024-12-07T20:57:04.717649",
     "exception": false,
     "start_time": "2024-12-07T20:57:04.680306",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from lenskit import topn\n",
    "from lenskit.metrics.predict import rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62cde0fa",
   "metadata": {
    "papermill": {
     "duration": 0.004695,
     "end_time": "2024-12-07T20:57:04.727496",
     "exception": false,
     "start_time": "2024-12-07T20:57:04.722801",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b8bdd0f",
   "metadata": {
    "papermill": {
     "duration": 0.004283,
     "end_time": "2024-12-07T20:57:04.736801",
     "exception": false,
     "start_time": "2024-12-07T20:57:04.732518",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We specify the dataset we will use for our analysis and the main directory from where we read the recommendation and prediction files. From the main directory we find all the directories associated with the dataset and then read the recommendation and predictions files from those directories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "66d37091",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:04.746260Z",
     "iopub.status.busy": "2024-12-07T20:57:04.745841Z",
     "iopub.status.idle": "2024-12-07T20:57:04.748070Z",
     "shell.execute_reply": "2024-12-07T20:57:04.747701Z"
    },
    "papermill": {
     "duration": 0.007778,
     "end_time": "2024-12-07T20:57:04.748764",
     "exception": false,
     "start_time": "2024-12-07T20:57:04.740986",
     "status": "completed"
    },
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "dataset = \"ml100k\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "236227c7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:04.758138Z",
     "iopub.status.busy": "2024-12-07T20:57:04.757930Z",
     "iopub.status.idle": "2024-12-07T20:57:04.760085Z",
     "shell.execute_reply": "2024-12-07T20:57:04.759707Z"
    },
    "papermill": {
     "duration": 0.007581,
     "end_time": "2024-12-07T20:57:04.760700",
     "exception": false,
     "start_time": "2024-12-07T20:57:04.753119",
     "status": "completed"
    },
    "tags": [
     "injected-parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "dataset = \"ml1m\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b7f9f66b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:04.769888Z",
     "iopub.status.busy": "2024-12-07T20:57:04.769642Z",
     "iopub.status.idle": "2024-12-07T20:57:04.771877Z",
     "shell.execute_reply": "2024-12-07T20:57:04.771492Z"
    },
    "papermill": {
     "duration": 0.007681,
     "end_time": "2024-12-07T20:57:04.772490",
     "exception": false,
     "start_time": "2024-12-07T20:57:04.764809",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "output_root = Path(\"runs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8688a4d0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:04.781935Z",
     "iopub.status.busy": "2024-12-07T20:57:04.781729Z",
     "iopub.status.idle": "2024-12-07T20:57:04.784678Z",
     "shell.execute_reply": "2024-12-07T20:57:04.784284Z"
    },
    "papermill": {
     "duration": 0.008546,
     "end_time": "2024-12-07T20:57:04.785360",
     "exception": false,
     "start_time": "2024-12-07T20:57:04.776814",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "dirs = [fld for fld in output_root.glob(f'{dataset}-*')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "56d609aa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:04.794970Z",
     "iopub.status.busy": "2024-12-07T20:57:04.794774Z",
     "iopub.status.idle": "2024-12-07T20:57:06.326153Z",
     "shell.execute_reply": "2024-12-07T20:57:06.325696Z"
    },
    "papermill": {
     "duration": 1.537043,
     "end_time": "2024-12-07T20:57:06.326954",
     "exception": false,
     "start_time": "2024-12-07T20:57:04.789911",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4228000 entries, 0 to 4227999\n",
      "Data columns (total 5 columns):\n",
      " #   Column     Dtype   \n",
      "---  ------     -----   \n",
      " 0   item       int64   \n",
      " 1   score      float64 \n",
      " 2   user       int64   \n",
      " 3   rank       int64   \n",
      " 4   algorithm  category\n",
      "dtypes: category(1), float64(1), int64(3)\n",
      "memory usage: 133.1 MB\n"
     ]
    }
   ],
   "source": [
    "recs = []\n",
    "for fld in dirs:\n",
    "    for file in fld.glob(\"recs-*\"):\n",
    "        rec = pd.read_csv(file)\n",
    "        rec[\"algorithm\"] = fld.name.split(\"-\")[1]\n",
    "        recs.append(rec)\n",
    "\n",
    "recs = pd.concat(recs, ignore_index=True)\n",
    "recs = recs.astype({'algorithm': 'category'})\n",
    "recs.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "feaa4493",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:06.339402Z",
     "iopub.status.busy": "2024-12-07T20:57:06.339170Z",
     "iopub.status.idle": "2024-12-07T20:57:06.357887Z",
     "shell.execute_reply": "2024-12-07T20:57:06.357494Z"
    },
    "papermill": {
     "duration": 0.025147,
     "end_time": "2024-12-07T20:57:06.358673",
     "exception": false,
     "start_time": "2024-12-07T20:57:06.333526",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Bias', 'IIE', 'Pop', 'UUE', 'III', 'IALS', 'ALS']\n",
       "Categories (7, object): ['ALS', 'Bias', 'IALS', 'IIE', 'III', 'Pop', 'UUE']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rec_algos = recs['algorithm'].unique()\n",
    "rec_algos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b378a77c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:06.369162Z",
     "iopub.status.busy": "2024-12-07T20:57:06.368757Z",
     "iopub.status.idle": "2024-12-07T20:57:06.533541Z",
     "shell.execute_reply": "2024-12-07T20:57:06.533095Z"
    },
    "papermill": {
     "duration": 0.17107,
     "end_time": "2024-12-07T20:57:06.534406",
     "exception": false,
     "start_time": "2024-12-07T20:57:06.363336",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 181200 entries, 0 to 181199\n",
      "Data columns (total 6 columns):\n",
      " #   Column      Non-Null Count   Dtype   \n",
      "---  ------      --------------   -----   \n",
      " 0   user        181200 non-null  int64   \n",
      " 1   item        181200 non-null  int64   \n",
      " 2   rating      181200 non-null  float64 \n",
      " 3   timestamp   181200 non-null  int64   \n",
      " 4   prediction  181154 non-null  float64 \n",
      " 5   algorithm   181200 non-null  category\n",
      "dtypes: category(1), float64(2), int64(3)\n",
      "memory usage: 7.1 MB\n"
     ]
    }
   ],
   "source": [
    "preds = []\n",
    "for fld in dirs:\n",
    "    for file in fld.glob(\"pred-*\"):\n",
    "        pred = pd.read_csv(file)\n",
    "        pred[\"algorithm\"] = fld.name.split(\"-\")[1]\n",
    "        preds.append(pred)\n",
    "\n",
    "preds = pd.concat(preds, ignore_index=True)\n",
    "preds = preds.astype({'algorithm': 'category'})\n",
    "preds.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca476ae1",
   "metadata": {
    "papermill": {
     "duration": 0.005022,
     "end_time": "2024-12-07T20:57:06.545138",
     "exception": false,
     "start_time": "2024-12-07T20:57:06.540116",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We need to load the test data so that we have the ground truths for computing accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "25294727",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:06.555727Z",
     "iopub.status.busy": "2024-12-07T20:57:06.555485Z",
     "iopub.status.idle": "2024-12-07T20:57:06.557942Z",
     "shell.execute_reply": "2024-12-07T20:57:06.557561Z"
    },
    "papermill": {
     "duration": 0.00845,
     "end_time": "2024-12-07T20:57:06.558601",
     "exception": false,
     "start_time": "2024-12-07T20:57:06.550151",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "split_root = Path(\"data-split\")\n",
    "split_dir = split_root / dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "88fc0d4e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:06.570467Z",
     "iopub.status.busy": "2024-12-07T20:57:06.570256Z",
     "iopub.status.idle": "2024-12-07T20:57:06.661372Z",
     "shell.execute_reply": "2024-12-07T20:57:06.660959Z"
    },
    "papermill": {
     "duration": 0.098943,
     "end_time": "2024-12-07T20:57:06.662210",
     "exception": false,
     "start_time": "2024-12-07T20:57:06.563267",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>item</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>part</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>364</th>\n",
       "      <td>5</td>\n",
       "      <td>321</td>\n",
       "      <td>3.0</td>\n",
       "      <td>978245863</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>360</th>\n",
       "      <td>5</td>\n",
       "      <td>1719</td>\n",
       "      <td>3.0</td>\n",
       "      <td>978244205</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>5</td>\n",
       "      <td>1554</td>\n",
       "      <td>3.0</td>\n",
       "      <td>978244540</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>372</th>\n",
       "      <td>5</td>\n",
       "      <td>3105</td>\n",
       "      <td>2.0</td>\n",
       "      <td>978246576</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>386</th>\n",
       "      <td>5</td>\n",
       "      <td>2028</td>\n",
       "      <td>2.0</td>\n",
       "      <td>978244053</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user  item  rating  timestamp    part\n",
       "index                                       \n",
       "364       5   321     3.0  978245863  test-3\n",
       "360       5  1719     3.0  978244205  test-3\n",
       "319       5  1554     3.0  978244540  test-3\n",
       "372       5  3105     2.0  978246576  test-3\n",
       "386       5  2028     2.0  978244053  test-3"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = []\n",
    "for file in split_dir.glob(\"test-*.parquet\"):\n",
    "    test.append(pd.read_parquet(file,).assign(part=file.stem.replace('.parquet', '')))\n",
    "\n",
    "test = pd.concat(test).rename(columns={\n",
    "    'user_id': 'user',\n",
    "    'item_id': 'item',\n",
    "})\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5770c372",
   "metadata": {
    "papermill": {
     "duration": 0.005176,
     "end_time": "2024-12-07T20:57:06.673346",
     "exception": false,
     "start_time": "2024-12-07T20:57:06.668170",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Top-N Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e347325",
   "metadata": {
    "papermill": {
     "duration": 0.004889,
     "end_time": "2024-12-07T20:57:06.683195",
     "exception": false,
     "start_time": "2024-12-07T20:57:06.678306",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The topn.RecListAnalysis class computes top-N metrics for recommendation list and takes care of making sure that the recommendations and ground truths are properly matched. Refer to the documentation for detailed explanation of the purpose for the RecListAnalysis class and how the analysis is done - https://lkpy.lenskit.org/en/stable/evaluation/topn-metrics.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "198eda4d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:06.693902Z",
     "iopub.status.busy": "2024-12-07T20:57:06.693668Z",
     "iopub.status.idle": "2024-12-07T20:57:09.205914Z",
     "shell.execute_reply": "2024-12-07T20:57:09.205493Z"
    },
    "papermill": {
     "duration": 2.518668,
     "end_time": "2024-12-07T20:57:09.206719",
     "exception": false,
     "start_time": "2024-12-07T20:57:06.688051",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn analyzing 4228000 recommendations (30200 truth rows)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn using rec key columns ['algorithm', 'user']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn using truth key columns ['user']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn numbering truth lists\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn numbering rec lists\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn collecting metric results\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn measured 42280 lists in 1.84s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn filling in missing user info (42280 initial rows)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/mde48/micromamba/envs/lkdemo/lib/python3.10/site-packages/lenskit/topn.py:152: FutureWarning: In a future version of pandas, a length 1 tuple will be returned when iterating over a groupby with a grouper equal to a list of length 1. Don't supply a list with a single grouper to avoid this warning.\n",
      "  for key, df in res.groupby(ug_cols):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>nrecs</th>\n",
       "      <th>precision</th>\n",
       "      <th>recip_rank</th>\n",
       "      <th>ndcg</th>\n",
       "      <th>ntruth</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th>user</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">ALS</th>\n",
       "      <th>3</th>\n",
       "      <td>100</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>100</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>100</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.018868</td>\n",
       "      <td>0.049018</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>100</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.029412</td>\n",
       "      <td>0.055189</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>100</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.014706</td>\n",
       "      <td>0.046123</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                nrecs  precision  recip_rank      ndcg  ntruth\n",
       "algorithm user                                                \n",
       "ALS       3       100       0.00    0.000000  0.000000       5\n",
       "          4       100       0.00    0.000000  0.000000       5\n",
       "          7       100       0.01    0.018868  0.049018       5\n",
       "          9       100       0.01    0.029412  0.055189       5\n",
       "          11      100       0.01    0.014706  0.046123       5"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rla = topn.RecListAnalysis()\n",
    "\n",
    "rla.add_metric(topn.precision)\n",
    "rla.add_metric(topn.recip_rank)\n",
    "rla.add_metric(topn.ndcg)\n",
    "results = rla.compute(recs, test.drop(columns=['rating']), include_missing=True)\n",
    "results = results.fillna(0)\n",
    "results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75dbe033",
   "metadata": {
    "papermill": {
     "duration": 0.005611,
     "end_time": "2024-12-07T20:57:09.219330",
     "exception": false,
     "start_time": "2024-12-07T20:57:09.213719",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We will reshape the 'results' dataframe by stacking the columns to index and then use the bar chart to visualize the performance of our algorithms with respect to the precision, reciprocal rank and ndcg metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4c7ffaf7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:09.231266Z",
     "iopub.status.busy": "2024-12-07T20:57:09.231027Z",
     "iopub.status.idle": "2024-12-07T20:57:09.246156Z",
     "shell.execute_reply": "2024-12-07T20:57:09.245773Z"
    },
    "papermill": {
     "duration": 0.022042,
     "end_time": "2024-12-07T20:57:09.246853",
     "exception": false,
     "start_time": "2024-12-07T20:57:09.224811",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>algorithm</th>\n",
       "      <th>user</th>\n",
       "      <th>metric</th>\n",
       "      <th>val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ALS</td>\n",
       "      <td>3</td>\n",
       "      <td>precision</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ALS</td>\n",
       "      <td>3</td>\n",
       "      <td>recip_rank</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ALS</td>\n",
       "      <td>3</td>\n",
       "      <td>ndcg</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ALS</td>\n",
       "      <td>4</td>\n",
       "      <td>precision</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ALS</td>\n",
       "      <td>4</td>\n",
       "      <td>recip_rank</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  algorithm  user      metric  val\n",
       "0       ALS     3   precision  0.0\n",
       "1       ALS     3  recip_rank  0.0\n",
       "2       ALS     3        ndcg  0.0\n",
       "3       ALS     4   precision  0.0\n",
       "4       ALS     4  recip_rank  0.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pltData = (results.drop(columns=['nrecs', 'ntruth']).stack()).reset_index()\n",
    "pltData.columns = ['algorithm', 'user', 'metric', 'val']\n",
    "pltData.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66c6e8ba",
   "metadata": {
    "papermill": {
     "duration": 0.005416,
     "end_time": "2024-12-07T20:57:09.257877",
     "exception": false,
     "start_time": "2024-12-07T20:57:09.252461",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We need to determine if the differences we observe in the performances of the algorithms for the various metrics are statistically significant. To achieve this, we will need to use either a parametric or non-parametric statistical test for comparing the differences. We will consider a parametric test - repeated ANOVA measure cause our sample groups are correlated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bd651349",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:09.269799Z",
     "iopub.status.busy": "2024-12-07T20:57:09.269574Z",
     "iopub.status.idle": "2024-12-07T20:57:11.423044Z",
     "shell.execute_reply": "2024-12-07T20:57:11.422613Z"
    },
    "papermill": {
     "duration": 2.160256,
     "end_time": "2024-12-07T20:57:11.423853",
     "exception": false,
     "start_time": "2024-12-07T20:57:09.263597",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABEAAAAEiCAYAAAD5+6e5AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABX1klEQVR4nO3de1xVVd7H8e+Rq4mQooEWAmajkncgBTN1UkzT0RlNuqGWlyGaDMkuRE1KTzGlKToJXkZDp0IsK61hUvLxVpIlgdMzMWaOhDkwhpUkJSju5w+HMx4PICBwOIfP+/Xar9zrrL3Wb5/D63fox9p7mwzDMAQAAAAAAODA2tg6AAAAAAAAgKZGAQQAAAAAADg8CiAAAAAAAMDhUQABAAAAAAAOjwIIAAAAAABweBRAAAAAAACAw6MAAgAAAAAAHB4FEAAAAAAA4PAogAAAAAAAAIdHAQT4jwULFmjAgAG2DsMmTCaT3nnnnUbvC6D1ctScGhAQoOTkZFuHUSfkawD1Ya95e8SIEYqNjbV1GLATJsMwDFsHATSVESNGaMCAAXX6ZfX06dMqLy+Xt7d30wfWwhQXF6tDhw5yc3Nr1L4AHAs5Vfr222/Vrl07XXXVVbYO5bJMJpPefvttTZo0ydahALCR1pC363OOgLOtAwBszTAMVVZWysPDQx4eHrYOp14qKirk6up6xeP4+vo2SV8ArU9Lzalnz56Vi4vLFY/TuXPnRoimZo0VJwDUVUvN20BT4BIYtAgjRozQQw89pNjYWHXo0EE+Pj5avXq1ysrKdN9996l9+/a6/vrr9de//tXiuC+++ELjxo2Th4eHfHx8FBUVpZKSEknSjBkztHv3bi1btkwmk0kmk0kFBQXatWuXTCaTtm3bppCQELm5uWnv3r3VLvtbt26dbrzxRrm5ualLly763e9+12TvQdX8q1atkp+fn6666irdcccd+uGHH8x9ZsyYoUmTJikpKUldu3bVL37xC0nS8ePHFRkZqQ4dOsjb21sTJ05UQUFBnc/l4mXSFRUV+t3vfqcuXbrI3d1dAQEBSkpKqravJH3++ef65S9/qbZt28rb21tz5szR6dOnrWJevHixunTpIm9vbz344IM6e/Zs4715ACyQU/+bU9etW6fu3bvLzc1NhmHo1KlTmjNnjq655hp5enrql7/8pQ4ePGhx7NatWxUSEiJ3d3d16tRJv/nNb8yvXXoJjMlkUmpqqsaOHau2bdsqMDBQb7zxRp1iLCgokMlk0qZNmzRixAi5u7vr1Vdf1cmTJ3XXXXfpuuuu01VXXaW+ffsqPT3d4tgRI0Zo7ty5euyxx9SxY0f5+vpqwYIFtc6XmJgoHx8f5eXl1Sk+AM2HvP3fvP3nP/9ZAQEB8vLy0p133qkff/zR3KesrEzTpk2Th4eHunTpopdeeslqnPLycj322GPy8/OTm5ubbrjhBq1du9b8+tatW3XDDTeobdu2GjlypNavXy+TyWTxOzccFwUQtBjr169Xp06d9Mknn+ihhx7SAw88oDvuuEPh4eH67LPPNGbMGEVFRemnn36SJBUVFWn48OEaMGCADhw4oPfff1///ve/NXXqVEnSsmXLFBYWptmzZ6uoqEhFRUXy8/Mzz/fYY48pKSlJ+fn56tevn1U8qampevDBBzVnzhx9/vnn2rp1q3r06FFj/K+99pq5cl7T9tprr9X6Hnz11VfatGmT3n33Xb3//vvKy8vTgw8+aNFnx44dys/PV1ZWlt577z399NNPGjlypDw8PLRnzx59+OGH8vDw0G233aaKiop6n8vy5cu1detWbdq0SYcOHdKrr76qgICAavv+9NNPuu2229ShQwd9+umneuONN/TBBx9YfTnu3LlTR44c0c6dO7V+/XqlpaUpLS2t1vcCwJUhp/43p27evNn8P/233367iouLlZmZqZycHA0aNEi33nqrvvvuO0nSX/7yF/3mN7/R7bffrtzcXO3YsUMhISG1zvP0009r8uTJOnjwoO69917dddddys/Pr/WYiz3++OOaO3eu8vPzNWbMGJ05c0bBwcF677339H//93+aM2eOoqKitH//fovj1q9fr3bt2mn//v168cUXlZiYqKysLKvxDcPQww8/rLVr1+rDDz+0y2v8gdaAvC0dOXJE77zzjt577z2999572r17t/7whz+YX3/00Ue1c+dOvf3229q+fbt27dqlnJwcizGmTZumjRs3avny5crPz9fKlSvNK1sKCgo0ZcoUTZo0SXl5efrtb3+rhISEy3wycCgG0AIMHz7cuPnmm837586dM9q1a2dERUWZ24qKigxJRnZ2tmEYhvH0008bERERFuMcO3bMkGQcOnTIPO7DDz9s0Wfnzp2GJOOdd96xaH/mmWeM/v37m/e7du1qJCQk1PkcSktLjcOHD9e6lZaW1nj8M888Yzg5ORnHjh0zt/31r3812rRpYxQVFRmGYRjTp083fHx8jPLycnOftWvXGj179jTOnz9vbisvLzfatm1rbNu2rU7nIsl4++23DcMwjIceesj45S9/aTFeTX1Xr15tdOjQwTh9+rT59b/85S9GmzZtjOLiYnPM/v7+xrlz58x97rjjDiMyMrLGeABcGXLqhfldXFyMEydOmNt27NhheHp6GmfOnLHoe/311xurVq0yDMMwwsLCjHvuuafGcf39/Y2lS5ea9yUZ0dHRFn0GDx5sPPDAA5c9x6NHjxqSjOTk5Mv2HTdunPHII4+Y9y/9jA3DMEJDQ43HH3/cIrY33njDuPfee41evXpZfL8AaFnI2xfmv+qqqyz6PProo8bgwYMNwzCMH3/80XB1dTU2btxofv3kyZNG27Ztzed46NAhQ5KRlZVV7RyPP/640adPH4u2hIQEQ5Lx/fff1/lcYb+4BwhajIsrz05OTvL29lbfvn3NbT4+PpKkEydOSJJycnK0c+fOaq9VPHLkiPnykJrU9he9EydO6F//+pduvfXWOsffvn17tW/fvs79q9OtWzddd9115v2wsDCdP39ehw4dMt97o2/fvhb3/cjJydFXX31lNfeZM2d05MiRep/LjBkzNHr0aPXs2VO33Xabxo8fr4iIiGr75ufnq3///mrXrp25bejQoeaYqz6zG2+8UU5OTuY+Xbp00eeff16neAA0DDlV8vf3t7hnR05Ojk6fPm11g7+ff/5ZR44ckSTl5eVp9uzZ9ZonLCzMar8+l5lc+t5VVlbqD3/4gzIyMnT8+HGVl5ervLzcItdKsvqLbZcuXcyfZ5V58+bJzc1NH3/8sTp16lTnmAA0P/L2hcsMLx7j4rx25MgRVVRUWOTcjh07qmfPnub9vLw8OTk5afjw4dWOf+jQIYWGhlq03XTTTVcUM+wLBRC0GJfe9M1kMlm0mUwmSdL58+fN/50wYYJeeOEFq7G6dOly2fku/UXyYm3btq1TzBd77bXX9Nvf/rbWPqtWrdI999xT5zGrzrnqv5J13OfPn1dwcHC1Swo7d+6sNm3qd6XboEGDdPToUf31r3/VBx98oKlTp2rUqFF68803rfoahmERW3WxS9V/tlWfI4CmQU6tPl926dJFu3btsup79dVXNzjW6tSUG6tzaZwvvfSSli5dquTkZPXt21ft2rVTbGys+bLGKnXJraNHj1Z6erq2bdtWr+8fAM2PvF17XjPq8PDSy8Vd3e+udRkXjoMCCOzWoEGDtHnzZgUEBMjZufofZVdXV1VWVtZ77Pbt2ysgIEA7duzQyJEj63TMr371Kw0ePLjWPlWV+5oUFhbqX//6l7p27SpJys7OVps2bWqt4A8aNEgZGRnmG/pVp77n4unpqcjISEVGRmrKlCm67bbb9N1336ljx44W/YKCgrR+/XqVlZWZv0Q/+uijy8YMoOVxxJx6qUGDBqm4uFjOzs413tuoX79+2rFjh+677746j/vxxx9r2rRpFvsDBw6sV2wX27t3ryZOnKh7771X0oX/yTl8+LB69+5d77F+9atfacKECbr77rvl5OSkO++8s8FxAWhZWkPevliPHj3k4uKijz/+WN26dZMkff/99/ryyy/NKz769u2r8+fPa/fu3Ro1apTVGL169VJmZqZF24EDBxocE+wPBRDYrQcffFBr1qzRXXfdpUcffVSdOnXSV199pY0bN2rNmjVycnJSQECA9u/fr4KCAnl4eFj9D3xtFixYoOjoaF1zzTUaO3asfvzxR3300Ud66KGHqu3fGMv+3N3dNX36dC1evFilpaWaO3eupk6dWuujZ++55x4tWrRIEydOVGJioq677joVFhbqrbfe0qOPPqrrrruuXueydOlSdenSRQMGDFCbNm30xhtvyNfX1/zX0UvnfuaZZzR9+nQtWLBA3377rR566CFFRUVd0RccgObniDn1UqNGjVJYWJgmTZqkF154QT179tS//vUvZWZmatKkSQoJCdEzzzyjW2+9Vddff73uvPNOnTt3Tn/961/12GOP1TjuG2+8oZCQEN1888167bXX9Mknn1g8caC+evTooc2bN2vfvn3q0KGDlixZouLi4gYVQCTp17/+tf785z8rKipKzs7OmjJlSoNjA9BytIa8fTEPDw/NnDlTjz76qLy9veXj46OEhASL1c4BAQGaPn267r//fi1fvlz9+/fX119/rRMnTmjq1Kn67W9/qyVLlujxxx/XzJkzlZeXZ74xf31W7sF+8RQY2K2uXbvqo48+UmVlpcaMGaM+ffro4YcflpeXlzkRzp8/X05OTgoKClLnzp1VWFhY5/GnT5+u5ORkpaSk6MYbb9T48eN1+PDhpjodSRd+6f3Nb36jcePGKSIiQn369FFKSkqtx1x11VXas2ePunXrpt/85jfq3bu37r//fv3888/mFSH1ORcPDw+98MILCgkJUWhoqAoKCpSZmVntpTRXXXWVtm3bpu+++06hoaGaMmWKbr31Vr388stX/mYAaFaOmFMvZTKZlJmZqVtuuUX333+/fvGLX+jOO+9UQUGBuWg7YsQIvfHGG9q6dasGDBigX/7yl1ZPX7nUwoULtXHjRvXr10/r16/Xa6+9pqCgoAbH+fTTT2vQoEEaM2aMRowYIV9fX02aNKnB40nSlClTtH79ekVFRemtt966orEAtAytIW9fatGiRbrlllv0q1/9SqNGjdLNN9+s4OBgiz6pqamaMmWKYmJi1KtXL82ePVtlZWWSpMDAQL355pt666231K9fP6WmppqfAuPm5tbs54PmZzK46AloERYsWKB33nmnXjfOAwDYlslk0ttvv33FBQoAgG0899xzWrlypY4dO2brUNAMuAQGAAAAANAqpKSkKDQ0VN7e3vroo4+0aNEi/e53v7N1WGgmXAIDAADQBJ5//nl5eHhUu40dO9bW4QFAq3T48GFNnDhRQUFBevbZZ/XII49owYIFtg4LzYRLYAAAAJrAd999p++++67a19q2batrr722mSMCAKB1owACAAAAAAAcHpfAAAAAAAAAh2fzAkhKSooCAwPl7u6u4OBg7d27t9b+u3fvVnBwsNzd3dW9e3etXLnS4vW33npLISEhuvrqq9WuXTsNGDBAf/7zn694XgAAAAAAYL9sWgDJyMhQbGysEhISlJubq2HDhmns2LE1Pp/66NGjGjdunIYNG6bc3Fw9+eSTmjt3rjZv3mzu07FjRyUkJCg7O1t/+9vfdN999+m+++7Ttm3bGjxvdQzDUGlpqbiCCACaDrkWAJoWeRZAa2LTe4AMHjxYgwYNUmpqqrmtd+/emjRpkpKSkqz6P/7449q6davy8/PNbdHR0Tp48KCys7NrnGfQoEG6/fbb9eyzzzZo3uqUlpbKy8tLp06dkqenZ52OAQDUD7kWAJoWeRZAa2KzFSAVFRXKyclRRESERXtERIT27dtX7THZ2dlW/ceMGaMDBw7o7NmzVv0Nw9COHTt06NAh3XLLLQ2eV5LKy8tVWlpqsQEAAAAAAPtgswJISUmJKisr5ePjY9Hu4+Oj4uLiao8pLi6utv+5c+dUUlJibjt16pQ8PDzk6uqq22+/XX/84x81evToBs8rSUlJSfLy8jJvfn5+9TpfAAAAAABgOza/CarJZLLYNwzDqu1y/S9tb9++vfLy8vTpp5/queeeU1xcnHbt2nVF88bHx+vUqVPm7dixY7WeFwAAAAAAaDmcbTVxp06d5OTkZLXq4sSJE1arM6r4+vpW29/Z2Vne3t7mtjZt2qhHjx6SpAEDBig/P19JSUkaMWJEg+aVJDc3N7m5udXrHAEAAAAAQMtgsxUgrq6uCg4OVlZWlkV7VlaWwsPDqz0mLCzMqv/27dsVEhIiFxeXGucyDEPl5eUNnhcAAAAAANg3m60AkaS4uDhFRUUpJCREYWFhWr16tQoLCxUdHS3pwmUnx48f14YNGyRdeOLLyy+/rLi4OM2ePVvZ2dlau3at0tPTzWMmJSUpJCRE119/vSoqKpSZmakNGzZYPPHlcvMCAAAAAADHYtMCSGRkpE6ePKnExEQVFRWpT58+yszMlL+/vySpqKhIhYWF5v6BgYHKzMzUvHnztGLFCnXt2lXLly/X5MmTzX3KysoUExOjb775Rm3btlWvXr306quvKjIyss7zAgAAAAAAx2Iyqu4iinrhmektl2EYKisrM++3a9eu1hvcAmi5yLVoDfjegi2RZ1sucgPQ+Gy6AgRoCmVlZZo4caJ5f8uWLfLw8LBhRAAA1IzvLQDVITcAjc/mj8EFAAAAAABoahRAAAAAAACAw+MSGLRIwY9uaPCxpnMV8rpof8TTG2U4uzZorJxF0xocBwAAAACg5WAFCAAAAAAAcHisAAEAAACAJtLQlc2sagYaHytAAACSpJSUFAUGBsrd3V3BwcHau3dvjX3feustjR49Wp07d5anp6fCwsK0bds2iz5paWkymUxW25kzZ5r6VAAAAAArrAABACgjI0OxsbFKSUnR0KFDtWrVKo0dO1ZffPGFunXrZtV/z549Gj16tJ5//nldffXVeuWVVzRhwgTt379fAwcONPfz9PTUoUOHLI51d3dv8vMBmltLuHcVf+EFAKB2FEDgcAwnF53qd5fFPoDaLVmyRDNnztSsWbMkScnJydq2bZtSU1OVlJRk1T85Odli//nnn9eWLVv07rvvWhRATCaTfH19mzR2AAAAoC64BAaOx2SS4exq3mQy2ToioEWrqKhQTk6OIiIiLNojIiK0b9++Oo1x/vx5/fjjj+rYsaNF++nTp+Xv76/rrrtO48ePV25ubqPFDQAAANQHK0AAoJUrKSlRZWWlfHx8LNp9fHxUXFxcpzFeeukllZWVaerUqea2Xr16KS0tTX379lVpaamWLVumoUOH6uDBg7rhhhuqHae8vFzl5eXm/dLS0gacEQAA9o9VzUDjowACAJB04XKVixmGYdVWnfT0dC1YsEBbtmzRNddcY24fMmSIhgwZYt4fOnSoBg0apD/+8Y9avnx5tWMlJSVp4cKFDTwDAAAcyH9WNQNoPFwCAwCtXKdOneTk5GS12uPEiRNWq0IulZGRoZkzZ2rTpk0aNWpUrX3btGmj0NBQHT58uMY+8fHxOnXqlHk7duxY3U8EAAAAqAUFEABo5VxdXRUcHKysrCyL9qysLIWHh9d4XHp6umbMmKHXX39dt99++2XnMQxDeXl56tKlS4193Nzc5OnpabEBAAAAjYFLYAAAiouLU1RUlEJCQhQWFqbVq1ersLBQ0dHRki6szDh+/Lg2bLjwqM/09HRNmzZNy5Yt05AhQ8yrR9q2bSsvrwsP9Fy4cKGGDBmiG264QaWlpVq+fLny8vK0YsUK25wkAAAAWjUKIAAARUZG6uTJk0pMTFRRUZH69OmjzMxM+fv7S5KKiopUWFho7r9q1SqdO3dODz74oB588EFz+/Tp05WWliZJ+uGHHzRnzhwVFxfLy8tLAwcO1J49e3TTTTc167kBAAAAEgUQAMB/xMTEKCYmptrXqooaVXbt2nXZ8ZYuXaqlS5c2QmSAY+NJDwAANA8KIAAAALbEkx4AAGgW3AQVAAAAAAA4PAogAAAAAADA4VEAAQAAAAAADo8CCAAAAAAAcHgUQAAAAAAAgMOzeQEkJSVFgYGBcnd3V3BwsPbu3Vtr/927dys4OFju7u7q3r27Vq5cafH6mjVrNGzYMHXo0EEdOnTQqFGj9Mknn1j0WbBggUwmk8Xm6+vb6OcGAAAAAABaBpsWQDIyMhQbG6uEhATl5uZq2LBhGjt2rAoLC6vtf/ToUY0bN07Dhg1Tbm6unnzySc2dO1ebN28299m1a5fuuusu7dy5U9nZ2erWrZsiIiJ0/Phxi7FuvPFGFRUVmbfPP/+8Sc8VAAAAAADYjrMtJ1+yZIlmzpypWbNmSZKSk5O1bds2paamKikpyar/ypUr1a1bNyUnJ0uSevfurQMHDmjx4sWaPHmyJOm1116zOGbNmjV68803tWPHDk2bNs3c7uzszKoPAAAAAABaCZutAKmoqFBOTo4iIiIs2iMiIrRv375qj8nOzrbqP2bMGB04cEBnz56t9piffvpJZ8+eVceOHS3aDx8+rK5duyowMFB33nmn/vnPf17B2QAAAACXV5/Lv9966y2NHj1anTt3lqenp8LCwrRt2zarfps3b1ZQUJDc3NwUFBSkt99+uylPAQDsls0KICUlJaqsrJSPj49Fu4+Pj4qLi6s9pri4uNr+586dU0lJSbXHPPHEE7r22ms1atQoc9vgwYO1YcMGbdu2TWvWrFFxcbHCw8N18uTJGuMtLy9XaWmpxQYAAADUVX0v/96zZ49Gjx6tzMxM5eTkaOTIkZowYYJyc3PNfbKzsxUZGamoqCgdPHhQUVFRmjp1qvbv399cpwUAdsPmN0E1mUwW+4ZhWLVdrn917ZL04osvKj09XW+99Zbc3d3N7WPHjtXkyZPVt29fjRo1Sn/5y18kSevXr69x3qSkJHl5eZk3Pz+/y58cAAAA8B8XX/7du3dvJScny8/PT6mpqdX2T05O1mOPPabQ0FDdcMMNev7553XDDTfo3XfftegzevRoxcfHq1evXoqPj9ett95qvmQcAPBfNiuAdOrUSU5OTlarPU6cOGG1yqOKr69vtf2dnZ3l7e1t0b548WI9//zz2r59u/r161drLO3atVPfvn11+PDhGvvEx8fr1KlT5u3YsWO1jgkAAABUacjl35c6f/68fvzxR4tLu2u6RLyuYwJAa2KzAoirq6uCg4OVlZVl0Z6VlaXw8PBqjwkLC7Pqv337doWEhMjFxcXctmjRIj377LN6//33FRISctlYysvLlZ+fry5dutTYx83NTZ6enhYbAAAAUBcNufz7Ui+99JLKyso0depUc1tNl4jXNCaXdQNozWx6CUxcXJz+9Kc/ad26dcrPz9e8efNUWFio6OhoSRdWXVz85Jbo6Gh9/fXXiouLU35+vtatW6e1a9dq/vz55j4vvviinnrqKa1bt04BAQEqLi5WcXGxTp8+be4zf/587d69W0ePHtX+/fs1ZcoUlZaWavr06c138gAA2DHDMHT69GnzVnVJKoDa1ffy7yrp6elasGCBMjIydM011zR4TC7rBtCa2fQxuJGRkTp58qQSExNVVFSkPn36KDMzU/7+/pKkoqIii5tCBQYGKjMzU/PmzdOKFSvUtWtXLV++3PwIXOnCnbUrKio0ZcoUi7meeeYZLViwQJL0zTff6K677lJJSYk6d+6sIUOG6OOPPzbPCwAAaldWVqaJEyea97ds2SIPDw8bRgS0bA25/LtKRkaGZs6cqTfeeMPixv5SzZeI1zRmfHy84uLizPulpaUUQeDwDMNQWVmZeb9du3Z1KjzC8di0ACJJMTExiomJqfa1tLQ0q7bhw4frs88+q3G8goKCy865cePGuoYHAAAAXLGLL//+9a9/bW7PysqyKCZeKj09Xffff7/S09N1++23W71edYn4vHnzzG3bt2+v8ZJyNzc3ubm5XcGZAPaHoj2q2LwAAgAAALQGcXFxioqKUkhIiMLCwrR69Wqry7+PHz+uDRs2SLpQ/Jg2bZqWLVumIUOGmFd6tG3bVl5eXpKkhx9+WLfccoteeOEFTZw4UVu2bNEHH3ygDz/80DYnCQAtmM0fgwsAAAC0BpGRkUpOTlZiYqIGDBigPXv21Hr596pVq3Tu3Dk9+OCD6tKli3l7+OGHzX3Cw8O1ceNGvfLKK+rXr5/S0tKUkZGhwYMHN/v5AUBLxwoQAAAAoJnU5/LvXbt21WnMKVOmWN3/DgBgjQIIAAAAAKDFC350Q4OOM52rkNdF+yOe3ijD2bVBY+Usmnb5TmixuAQGAAAAAAA4PFaAAADQSjX0L2lS4/01jb+kAQCA5sIKEAAAAAAA4PAogAAAAAAAAIdHAQQAAAAAADg87gECAAAAAHBYhpOLTvW7y2IfrRMrQAAAkqSUlBQFBgbK3d1dwcHB2rt3b41933rrLY0ePVqdO3eWp6enwsLCtG3bNqt+mzdvVlBQkNzc3BQUFKS33367KU8BAADAmskkw9nVvMlksnVEsBEKIAAAZWRkKDY2VgkJCcrNzdWwYcM0duxYFRYWVtt/z549Gj16tDIzM5WTk6ORI0dqwoQJys3NNffJzs5WZGSkoqKidPDgQUVFRWnq1Knav39/c50WAAAAYMYlMAAALVmyRDNnztSsWbMkScnJydq2bZtSU1OVlJRk1T85Odli//nnn9eWLVv07rvvauDAgeY+o0ePVnx8vCQpPj5eu3fvVnJystLT05v2hNDkWE4MAADsDStAAKCVq6ioUE5OjiIiIizaIyIitG/fvjqNcf78ef3444/q2LGjuS07O9tqzDFjxtR5TLRwLCcGAAB2hhUgANDKlZSUqLKyUj4+PhbtPj4+Ki4urtMYL730ksrKyjR16lRzW3Fxcb3HLC8vV3l5uXm/tLS0TvMDAAAAl8MKEACAJMl0yV/wDcOwaqtOenq6FixYoIyMDF1zzTVXNGZSUpK8vLzMm5+fXz3OAAAAAKgZBRAAaOU6deokJycnq5UZJ06csFrBcamMjAzNnDlTmzZt0qhRoyxe8/X1rfeY8fHxOnXqlHk7duxYPc8GAAAAqB4FEABo5VxdXRUcHKysrCyL9qysLIWHh9d4XHp6umbMmKHXX39dt99+u9XrYWFhVmNu37691jHd3Nzk6elpsQEAAACNgXuAAAAUFxenqKgohYSEKCwsTKtXr1ZhYaGio6MlXViZcfz4cW3YsEHSheLHtGnTtGzZMg0ZMsS80qNt27by8vKSJD388MO65ZZb9MILL2jixInasmWLPvjgA3344Ye2OUkAAAC0aqwAAQAoMjJSycnJSkxM1IABA7Rnzx5lZmbK399fklRUVKTCwkJz/1WrVuncuXN68MEH1aVLF/P28MMPm/uEh4dr48aNeuWVV9SvXz+lpaUpIyNDgwcPbvbzAwAAAFgBAgCQJMXExCgmJqba19LS0iz2d+3aVacxp0yZoilTplxhZAAAAMCVYwUIAAAAAABweBRAAAAAAACAw7N5ASQlJUWBgYFyd3dXcHCw9u7dW2v/3bt3Kzg4WO7u7urevbtWrlxp8fqaNWs0bNgwdejQQR06dNCoUaP0ySefXPG8QGtkGIZOnz5t3gzDsHVIAAAAANAgNi2AZGRkKDY2VgkJCcrNzdWwYcM0duxYixvtXezo0aMaN26chg0bptzcXD355JOaO3euNm/ebO6za9cu3XXXXdq5c6eys7PVrVs3RURE6Pjx4w2eF2itysrKNHHiRPNWVlZm65AAAAAAoEFsWgBZsmSJZs6cqVmzZql3795KTk6Wn5+fUlNTq+2/cuVKdevWTcnJyerdu7dmzZql+++/X4sXLzb3ee211xQTE6MBAwaoV69eWrNmjc6fP68dO3Y0eF4AAAAAAGDfbFYAqaioUE5OjiIiIizaIyIitG/fvmqPyc7Otuo/ZswYHThwQGfPnq32mJ9++klnz55Vx44dGzwvAAAAAACwbzZ7DG5JSYkqKyvl4+Nj0e7j46Pi4uJqjykuLq62/7lz51RSUqIuXbpYHfPEE0/o2muv1ahRoxo8rySVl5ervLzcvF9aWlr7CQItRPCjGxp8rOlchbwu2h/x9EYZzq4NGitn0bQGxwEAAAAAV8rmN0E1mUwW+4ZhWLVdrn917ZL04osvKj09XW+99Zbc3d2vaN6kpCR5eXmZNz8/vxr7AgAAAACAlsVmBZBOnTrJycnJatXFiRMnrFZnVPH19a22v7Ozs7y9vS3aFy9erOeff17bt29Xv379rmheSYqPj9epU6fM27Fjx+p0ngAAAAAAwPZsVgBxdXVVcHCwsrKyLNqzsrIUHh5e7TFhYWFW/bdv366QkBC5uLiY2xYtWqRnn31W77//vkJCQq54Xklyc3OTp6enxQYAAAAAAOyDze4BIklxcXGKiopSSEiIwsLCtHr1ahUWFio6OlrShVUXx48f14YNF+5hEB0drZdffllxcXGaPXu2srOztXbtWqWnp5vHfPHFF/X000/r9ddfV0BAgHmlh4eHhzw8POo0L4ALDCcXnep3l8U+AAAAANgjmxZAIiMjdfLkSSUmJqqoqEh9+vRRZmam/P39JUlFRUUqLCw09w8MDFRmZqbmzZunFStWqGvXrlq+fLkmT55s7pOSkqKKigpNmTLFYq5nnnlGCxYsqNO8AP7DZGrwTU8BAAAAoCWxaQFEkmJiYhQTE1Pta2lpaVZtw4cP12effVbjeAUFBVc8LwAAAAAAcCw2fwoMAAAA0FqkpKQoMDBQ7u7uCg4O1t69e2vsW1RUpLvvvls9e/ZUmzZtFBsba9UnLS1NJpPJajtz5kwTngUA2CcKIAAAAEAzyMjIUGxsrBISEpSbm6thw4Zp7NixFpd8X6y8vFydO3dWQkKC+vfvX+O4np6eKioqstjc3d2b6jQAwG5RAAEAAACawZIlSzRz5kzNmjVLvXv3VnJysvz8/JSamlpt/4CAAC1btkzTpk2Tl5dXjeOaTCb5+vpabAAAaxRAAAAAgCZWUVGhnJwcRUREWLRHRERo3759VzT26dOn5e/vr+uuu07jx49Xbm7uFY0HAI6KAggAAADQxEpKSlRZWSkfHx+Ldh8fHxUXFzd43F69eiktLU1bt25Venq63N3dNXToUB0+fLja/uXl5SotLbXYAKC1oAACAAAANBOTyWSxbxiGVVt9DBkyRPfee6/69++vYcOGadOmTfrFL36hP/7xj9X2T0pKkpeXl3nz8/Nr8NwAYG8ogAAAAABNrFOnTnJycrJa7XHixAmrVSFXok2bNgoNDa1xBUh8fLxOnTpl3o4dO9ZocwNAS0cBBAAAAGhirq6uCg4OVlZWlkV7VlaWwsPDG20ewzCUl5enLl26VPu6m5ubPD09LTYAaC2cbR0AAAAA0BrExcUpKipKISEhCgsL0+rVq1VYWKjo6GhJF1ZnHD9+XBs2bDAfk5eXJ+nCjU6//fZb5eXlydXVVUFBQZKkhQsXasiQIbrhhhtUWlqq5cuXKy8vTytWrGj28wOAlo4CCAAAAGplGIbKysrM++3atbui+1a0VpGRkTp58qQSExNVVFSkPn36KDMzU/7+/pKkoqIiFRYWWhwzcOBA879zcnL0+uuvy9/fXwUFBZKkH374QXPmzFFxcbG8vLw0cOBA7dmzRzfddFOznRcA2AsKIAAAAKhVWVmZJk6caN7fsmWLPDw8bBiR/YqJiVFMTEy1r6WlpVm1GYZR63hLly7V0qVLGyM0AHB43AMEAAAAAAA4PAogAAAAAADA4VEAAQAAAAAADo8CCABAkpSSkqLAwEC5u7srODhYe/furbFvUVGR7r77bvXs2VNt2rRRbGysVZ+0tDSZTCar7cyZM014FgAAAED1KIAAAJSRkaHY2FglJCQoNzdXw4YN09ixY62eRlClvLxcnTt3VkJCgvr371/juJ6enioqKrLY3N3dm+o0AAAAgBrxFBgAgJYsWaKZM2dq1qxZkqTk5GRt27ZNqampSkpKsuofEBCgZcuWSZLWrVtX47gmk0m+vr5NEzSAegt+dEODjjOdq5DXRfsjnt4ow9m1QWPlLJrWoOMAALhSrAABgFauoqJCOTk5ioiIsGiPiIjQvn37rmjs06dPy9/fX9ddd53Gjx+v3NzcWvuXl5ertLTUYgMAAGgNDMPQ6dOnzdvlHoON+qMAAgCtXElJiSorK+Xj42PR7uPjo+Li4gaP26tXL6WlpWnr1q1KT0+Xu7u7hg4dqsOHD9d4TFJSkry8vMybn59fg+cHAACwJ2VlZZo4caJ5Kysrs3VIDqdOl8AsX768zgPOnTu3wcEAAGzHZDJZ7BuGYdVWH0OGDNGQIUPM+0OHDtWgQYP0xz/+scbvlfj4eMXFxZn3S0tLKYIAAACgUdSpALJ06dI6DWYymSiAAICd6dSpk5ycnKxWe5w4ccJqVciVaNOmjUJDQ2tdAeLm5iY3N7dGmxMAAACoUqcCyNGjR5s6DgBoMMMwLJYItmvX7opWLrQ2rq6uCg4OVlZWln7961+b27OysjRx4sRGm8cwDOXl5alv376NNiYAAEBL0tCbTUvccLo58BQYAHav6nrJKlu2bJGHh4cNI7I/cXFxioqKUkhIiMLCwrR69WoVFhYqOjpa0oVLU44fP64NG/77pZ6Xlyfpwo1Ov/32W+Xl5cnV1VVBQUGSpIULF2rIkCG64YYbVFpaquXLlysvL08rVqxo9vMDAAAAGlQA+eabb7R161YVFhaqoqLC4rUlS5Y0SmAAgOYTGRmpkydPKjExUUVFRerTp48yMzPl7+8vSSoqKlJhYaHFMQMHDjT/OycnR6+//rr8/f1VUFAgSfrhhx80Z84cFRcXy8vLSwMHDtSePXt00003Ndt5AWgchpOLTvW7y2IfAAB7U++nwOzYsUM9e/ZUSkqKXnrpJe3cuVOvvPKK1q1bZ/5rYH2kpKQoMDBQ7u7uCg4O1t69e2vtv3v3bgUHB8vd3V3du3fXypUrLV7/+9//rsmTJysgIEAmk0nJyclWYyxYsEAmk8li8/X1rXfsAOBIYmJiVFBQoPLycuXk5OiWW24xv5aWlqZdu3ZZ9DcMw2qrKn5IF+4f9fXXX6u8vFwnTpzQtm3bFBYW1kxnA6BRmUwynF3Nm7jMEABgh+q9AiQ+Pl6PPPKIEhMT1b59e23evFnXXHON7rnnHt122231GisjI0OxsbFKSUnR0KFDtWrVKo0dO1ZffPGFunXrZtX/6NGjGjdunGbPnq1XX31VH330kWJiYtS5c2dNnjxZkvTTTz+pe/fuuuOOOzRv3rwa577xxhv1wQcfmPednJzqFTuAxtUSrpfkWkkAAADAcdW7AJKfn6/09PQLBzs76+eff5aHh4cSExM1ceJEPfDAA3Uea8mSJZo5c6ZmzZolSUpOTta2bduUmpqqpKQkq/4rV65Ut27dzKs6evfurQMHDmjx4sXmAkhoaKhCQ0MlSU888USNczs7O7PqAwAAAADQInC5YdOr9yUw7dq1U3l5uSSpa9euOnLkiPm1kpKSOo9TUVGhnJwcRUREWLRHRERo37591R6TnZ1t1X/MmDE6cOCAzp49W+e5Jenw4cPq2rWrAgMDdeedd+qf//xnrf3Ly8tVWlpqsQEAAAAA0Ci43LDJ1bsAMmTIEH300UeSpNtvv12PPPKInnvuOd1///0aMmRInccpKSlRZWWlfHx8LNp9fHxUXFxc7THFxcXV9j937ly9ii+DBw/Whg0btG3bNq1Zs0bFxcUKDw/XyZMnazwmKSlJXl5e5s3Pz6/O8wEAAAAAANuq9yUwS5Ys0enTpyVduJno6dOnlZGRoR49emjp0qX1DsB0SVXLMAyrtsv1r669NmPHjjX/u2/fvgoLC9P111+v9evXKy4urtpj4uPjLV4rLS2lCAK0ECwXBAAAAHA59S6APPvss7r33ntlGIauuuoqpaSkNGjiTp06ycnJyWq1x4kTJ6xWeVTx9fWttr+zs7O8vb0bFId04bKevn376vDhwzX2cXNzk5ubW4PnANCE/rNcEAAAAABqUu9LYE6ePKnbb79d1113nR555JEGPfpWklxdXRUcHKysrCyL9qysLIWHh1d7TFhYmFX/7du3KyQkRC4uDf+Lb3l5ufLz89WlS5cGjwEAAAAAAFquehdAtm7dquLiYj3zzDPKyclRcHCwgoKC9Pzzz6ugoKBeY8XFxelPf/qT1q1bp/z8fM2bN0+FhYWKjo6WdOGyk2nT/vtYyujoaH399deKi4tTfn6+1q1bp7Vr12r+/PnmPhUVFcrLy1NeXp4qKip0/Phx5eXl6auvvjL3mT9/vnbv3q2jR49q//79mjJlikpLSzV9+vT6vh0AAAAAAMAO1PsSGEm6+uqrNWfOHM2ZM0fffPON0tPTtW7dOv3+97/XuXPn6jxOZGSkTp48qcTERBUVFalPnz7KzMyUv7+/JKmoqEiFhYXm/oGBgcrMzNS8efO0YsUKde3aVcuXLzc/AleS/vWvf2ngwIHm/cWLF2vx4sUaPny4du3aJUn65ptvdNddd6mkpESdO3fWkCFD9PHHH5vnBQAAAAAAjqVBBZAqZ8+e1YEDB7R//34VFBTUeO+O2sTExCgmJqba19LS0qzahg8frs8++6zG8QICAsw3Rq3Jxo0b6xUjAAAAAACwb/W+BEaSdu7cqdmzZ8vHx0fTp09X+/bt9e677+rYsWONHR8AAAAAAMAVq/cKkOuuu04nT57UmDFjtGrVKk2YMEHu7u5NERsAAAAAAECjqHcB5Pe//73uuOMOdejQoSniAQAAAAAAaHT1LoDMmTOnKeIAAAAAAABoMg26BwgAAAAAAIA9oQACAAAAAAAcHgUQAAAAAADg8CiAAAAAAM0kJSVFgYGBcnd3V3BwsPbu3Vtj36KiIt19993q2bOn2rRpo9jY2Gr7bd68WUFBQXJzc1NQUJDefvvtJooeAOwbBRAAAACgGWRkZCg2NlYJCQnKzc3VsGHDNHbsWBUWFlbbv7y8XJ07d1ZCQoL69+9fbZ/s7GxFRkYqKipKBw8eVFRUlKZOnar9+/c35akAgF2iAAIAAAA0gyVLlmjmzJmaNWuWevfureTkZPn5+Sk1NbXa/gEBAVq2bJmmTZsmLy+vavskJydr9OjRio+PV69evRQfH69bb71VycnJTXgmAGCfKIAAAAAATayiokI5OTmKiIiwaI+IiNC+ffsaPG52drbVmGPGjLmiMQHAUTnbOgAAAADA0ZWUlKiyslI+Pj4W7T4+PiouLm7wuMXFxfUas7y8XOXl5eb90tLSBs8NAPaGFSAAAABAMzGZTBb7hmFYtTXlmElJSfLy8jJvfn5+VzQ3ANgTCiAAAABAE+vUqZOcnJysVmacOHHCagVHffj6+tZrzPj4eJ06dcq8HTt2rMFzA4C9oQACAAAANDFXV1cFBwcrKyvLoj0rK0vh4eENHjcsLMxqzO3bt9c4ppubmzw9PS02AGgtuAcIAAAA0Azi4uIUFRWlkJAQhYWFafXq1SosLFR0dLSkC6szjh8/rg0bNpiPycvLkySdPn1a3377rfLy8uTq6qqgoCBJ0sMPP6xbbrlFL7zwgiZOnKgtW7bogw8+0Icfftjs5wcALR0rQAAAkqSUlBQFBgbK3d1dwcHB2rt3b419i4qKdPfdd6tnz55q06aNYmNjq+23efNmBQUFyc3NTUFBQXr77bebKHoAaPkiIyOVnJysxMREDRgwQHv27FFmZqb8/f0lXcithYWFFscMHDhQAwcOVE5Ojl5//XUNHDhQ48aNM78eHh6ujRs36pVXXlG/fv2UlpamjIwMDR48uFnPDQDsAQUQAIAyMjIUGxurhIQE5ebmatiwYRo7dqzVL+JVysvL1blzZyUkJKh///7V9snOzlZkZKSioqJ08OBBRUVFaerUqdq/f39TngoAtGgxMTEqKChQeXm5cnJydMstt5hfS0tL065duyz6G4ZhtRUUFFj0mTJliv7xj3+ooqJC+fn5+s1vftMMZwIA9ocCCABAS5Ys0cyZMzVr1iz17t1bycnJ8vPzU2pqarX9AwICtGzZMk2bNk1eXl7V9klOTtbo0aMVHx+vXr16KT4+XrfeequSk5Ob8EwAAACA6lEAAYBWrqKiQjk5OYqIiLBoj4iI0L59+xo8bnZ2ttWYY8aMuaIxAQAAgIbiJqgA0MqVlJSosrLS6pGJPj4+Vo9WrI/i4uJ6j1leXq7y8nLzfmlpaYPnBwAAAC7GChAAgCTJZDJZ7BuGYdXW1GMmJSXJy8vLvPn5+V3R/AAAAEAVCiAA0Mp16tRJTk5OViszTpw4YbWCoz58fX3rPWZ8fLxOnTpl3o4dO9bg+QEAAICL2bwAUp/HLkrS7t27FRwcLHd3d3Xv3l0rV660eP3vf/+7Jk+erICAAJlMphpvtlffeQHAUbm6uio4OFhZWVkW7VlZWQoPD2/wuGFhYVZjbt++vdYx3dzc5OnpabEBAAAAjcGmBZD6Pnbx6NGjGjdunIYNG6bc3Fw9+eSTmjt3rjZv3mzu89NPP6l79+76wx/+IF9f30aZFwAcXVxcnP70pz9p3bp1ys/P17x581RYWKjo6GhJF1ZmTJs2zeKYvLw85eXl6fTp0/r222+Vl5enL774wvz6ww8/rO3bt+uFF17QP/7xD73wwgv64IMPFBsb25ynBgAAAEiy8U1QL37sonThkYnbtm1TamqqkpKSrPqvXLlS3bp1M6/q6N27tw4cOKDFixdr8uTJkqTQ0FCFhoZKkp544olGmRcAHF1kZKROnjypxMREFRUVqU+fPsrMzJS/v78kqaioyKpIPHDgQPO/c3Jy9Prrr8vf318FBQWSpPDwcG3cuFFPPfWUnn76aV1//fXKyMjQ4MGDm+28AAAAgCo2K4BUPXbx0iJFbY9drOmRimvXrtXZs2fl4uLSJPNKPJkAgOOLiYlRTExMta+lpaVZtRmGcdkxp0yZoilTplxpaAAAAMAVs9klMA157GJNj1Q8d+6cSkpKmmxeiScTAAAAAABgz2x+E9T6PiKxuv7VtTf2vDyZAAAAAAAA+2WzS2Aa8tjFmh6p6OzsLG9v7yabV7rwZAI3N7c6zQEAAAAAAFoWm60AachjF2t6pGJISEid7v/R0HkBAAAAAIB9s+lTYOLi4hQVFaWQkBCFhYVp9erVVo9dPH78uDZs2CBJio6O1ssvv6y4uDjNnj1b2dnZWrt2rdLT081jVlRUmB/DWFFRoePHjysvL08eHh7q0aNHneYFAAAAAACOxaYFkPo+djEwMFCZmZmaN2+eVqxYoa5du2r58uXmR+BK0r/+9S+LRzMuXrxYixcv1vDhw7Vr1646zQsAAAAAAByLTQsgUv0fuzh8+HB99tlnNY4XEBBQp0cz1jYvAAAAAABwLDZ/CgwAAAAAAEBTowACAAAAAAAcHgUQAAAAAADg8CiAAAAAAAAAh0cBBAAAAAAAODwKIAAAAAAAwOFRAAEAAAAAAA7P2dYB4ALDMFRWVmbeb9eunUwmkw0jAgAAAADAcVAAaSHKyso0ceJE8/6WLVvk4eFhw4isUaQBAAAAANgrCiCoM3so0gAAAAAAUB0KII0o+NENDT7WdK5CXhftj3h6owxn1waNlbNoWoPjAAAAAADAEXETVAAAAAAA4PBYAdIKNXSlCqtUAAAAAAD2igIIAAAAgEbBTfMBtGQUQFoIw8lFp/rdZbEPx8YvCAAAwNFw03wALRn3AGkpTCYZzq7mTS3wf4SrijRVG0WaK1P1C0LVdnExBAAAOKaUlBQFBgbK3d1dwcHB2rt3b639d+/ereDgYLm7u6t79+5auXKlxetpaWkymUxW25kzZ5ryNADALlEAQd3ZQZEGAACgpcrIyFBsbKwSEhKUm5urYcOGaezYsSosLKy2/9GjRzVu3DgNGzZMubm5evLJJzV37lxt3rzZop+np6eKioosNnd39+Y4JbtlGIZOnz5t3gzDsHVIAJoBl8AAAFoNLj0DYEtLlizRzJkzNWvWLElScnKytm3bptTUVCUlJVn1X7lypbp166bk5GRJUu/evXXgwAEtXrxYkydPNvczmUzy9fVt1Fgd/ab5XKoDtE4UQIAr0NBfDqTG+wWBJ+oAdccvvABspaKiQjk5OXriiScs2iMiIrRv375qj8nOzlZERIRF25gxY7R27VqdPXtWLi4XLkc+ffq0/P39VVlZqQEDBujZZ5/VwIEDm+ZEAMCOcQkMAAAA0MRKSkpUWVkpHx8fi3YfHx8VFxdXe0xxcXG1/c+dO6eSkhJJUq9evZSWlqatW7cqPT1d7u7uGjp0qA4fPlztmOXl5SotLbXYAKC1YAUIADQDe7j0IiUlRYsWLVJRUZFuvPFGJScna9iwYTX23717t+Li4vT3v/9dXbt21WOPPabo6Gjz62lpabrvvvusjvv555+5Nh1Aq3Vp7jcMo9bvg+r6X9w+ZMgQDRkyxPz60KFDNWjQIP3xj3/U8uXLrcZLSkrSwoULGxz/5TTnkw1ZiQugviiAADbCo49bl5Z+6UXVjflSUlI0dOhQrVq1SmPHjtUXX3yhbt26WfWvujHf7Nmz9eqrr+qjjz5STEyMOnfubHFduqenpw4dOmRxLMUPAK1Rp06d5OTkZLXa48SJE1arPKr4+vpW29/Z2Vne3t7VHtOmTRuFhobWuAIkPj5ecXFx5v3S0lL5+fnV51Rq95+b5gNAS0QBBLAVfkFAC9Iabswn8Rc/ALbj6uqq4OBgZWVl6de//rW5PSsry6JAfrGwsDC9++67Fm3bt29XSEiI+f4flzIMQ3l5eerbt2+1r7u5ucnNza2BZwEA9s3mBZDGXnItSZs3b9bTTz+tI0eO6Prrr9dzzz1n8UWzYMECq6V/tV1/CQBVHPGu+C3pxnzl5eUqLy8373NtOgBHEhcXp6ioKIWEhCgsLEyrV69WYWGh+XfZ+Ph4HT9+XBs2XPiuiY6O1ssvv6y4uDjNnj1b2dnZWrt2rdLT081jLly4UEOGDNENN9yg0tJSLV++XHl5eVqxYoVNztFesBIXaJ1sehPUpngWenZ2tiIjIxUVFaWDBw8qKipKU6dO1f79+y3GuvHGGy2elf7555836bkCQEvVUm7MJ124Nt3Ly8u8NeqybACwscjISCUnJysxMVEDBgzQnj17lJmZKX9/f0lSUVGRxe/BgYGByszM1K5du8xF5OXLl1ustPvhhx80Z84c9e7dWxERETp+/Lj27Nmjm266qdnPz678ZyVu1aYWdl8uAE3DpitAmmLJdXJyskaPHq34+HhJFyrpu3fvVnJyskW13NnZudGXZQOAPbP1jfmkZrg23Q7Yww1zATRcTEyMYmJiqn0tLS3Nqm348OH67LPPahxv6dKlWrp0aWOFBwAOzWYFkKZacp2dna158+ZZ9akqmlQ5fPiwunbtKjc3Nw0ePFjPP/+8unfvXmO8LMsGcCVa8lLblnJjPolr06WWf8NcAAAuRuEe9sRml8A01ZLrmvpcPObgwYO1YcMGbdu2TWvWrFFxcbHCw8N18uTJGuNlWTaAK9KCl9pefGO+i2VlZSk8PLzaY8LCwqz61/XGfF26dGmcwBugqhBVtbWkQhQAAPaoqnBftV1cDAFaGpveA0Rq/CXXdRlz7Nixmjx5svr27atRo0bpL3/5iyRp/fr1Nc4bHx+vU6dOmbdjx45d5swAwH7ExcXpT3/6k9atW6f8/HzNmzfP6sZ806b99+ar0dHR+vrrrxUXF6f8/HytW7dOa9eu1fz58819Fi5cqG3btumf//yn8vLyNHPmTOXl5VnduLpZteBCFAAAAJqWzS6Baaol1zX1qWlM6cIyrb59+7IsG0CrFRkZqZMnTyoxMVFFRUXq06dPnW7MN2/ePK1YsUJdu3at8cZ8xcXF8vLy0sCBA1vVjfkc8YlBAADHxCPm0VrYrADSVM9Cr1qWffF9QLZv317jMm7pwv098vPza338LgA4Om7MBwAAAEdm00tgmmLJ9cMPP6zt27frhRde0D/+8Q+98MIL+uCDDxQbG2vuM3/+fO3evVtHjx7V/v37NWXKFJWWlmr69OnNdu4AAAAAAKD52PQxuE2x5Do8PFwbN27UU089paefflrXX3+9MjIyNHjwYHOfb775RnfddZdKSkrUuXNnDRkyRB9//LF5XgAAAADA5bXkJ92h8dn7U39sWgCRGn/JtSRNmTJFU6ZMqfH1jRs31itGAACaC79IAgDsyn9uMI7WoeqpP1W2bNkiDw8PG0ZUPzYvgAAAgIvwiyQAAECToAACAAAAAEAr0lqfVmfTm6ACAAAAAAA0BwogAAAAAADA4XEJDAAAAAAAuCx7v1k7BRAAAOCQ7P1RfQAAtDh2frN2CiAAAMAh2fuj+gAAQOPiHiAAAAAAAMDhUQABAAAAAAAOj0tgAABAixb86IYGHWc6VyGvi/ZHPL2xwdct5yya1qDjAABAy8EKEAAAAAAA4PAogAAAAAAAAIfHJTAAAMAhGU4uOtXvLot9AADQelEAAQAAjslkavA9P2CfDMNQWVmZeb9du3YymUw2jAgA0JJQAAEAAIBDKCsr08SJE837W7ZskYeHhw0jAgC0JBRAAAAA0GI09Kk/UuM9+Yen/gCAY+ImqAAAAAAAwOFRAAEAAAAAAA6PS2AAAADgEHjyDwCgNhRAAAAA4Bh48g8AoBZcAgMAAAAAABweBRAAAAAAAODwKIAAAAAAAACHZ/MCSEpKigIDA+Xu7q7g4GDt3bu31v67d+9WcHCw3N3d1b17d61cudKqz+bNmxUUFCQ3NzcFBQXp7bffvuJ5AcDR2SofA0BrQq4FANuxaQEkIyNDsbGxSkhIUG5uroYNG6axY8eqsLCw2v5Hjx7VuHHjNGzYMOXm5urJJ5/U3LlztXnzZnOf7OxsRUZGKioqSgcPHlRUVJSmTp2q/fv3N3heAHB0tsrHANCakGsBwLZsWgBZsmSJZs6cqVmzZql3795KTk6Wn5+fUlNTq+2/cuVKdevWTcnJyerdu7dmzZql+++/X4sXLzb3SU5O1ujRoxUfH69evXopPj5et956q5KTkxs8LwA4OlvlYwBoTci1AGBbNiuAVFRUKCcnRxERERbtERER2rdvX7XHZGdnW/UfM2aMDhw4oLNnz9bap2rMhswLAI7MVvkYAFoTci0A2J6zrSYuKSlRZWWlfHx8LNp9fHxUXFxc7THFxcXV9j937pxKSkrUpUuXGvtUjdmQeSWpvLxc5eXl5v1Tp05JkkpLS81tleU/13h8c7o4puq0hDjtIUbJPuK0hxgl+4jzcjFKLTfO9u3by2QyNWg8W+Xj6thLrrXnn5WL2UOMkn3EaQ8xSvYRZ0uO0RFybV3yrNSyP4cq9hCjZB9x2kOMkn3E2RJilOwjzsb+nbYubFYAqXLpyRmGUesJV9f/0va6jFnfeZOSkrRw4UKrdj8/vxqPsRWvP0bbOoTLsocYJfuI0x5ilOwjTnuIUao+zlOnTsnT0/OKxrVVPr6YveRae/5ZaWnsIUbJPuK0hxgl+4izphgdIdfaS56V7PtnpaWxhzjtIUaJOBtTU/1OWxubFUA6deokJycnq+r0iRMnrKrYVXx9favt7+zsLG9v71r7VI3ZkHklKT4+XnFxceb98+fP67vvvpO3t3ejVahKS0vl5+enY8eONemHfiXsIUaJOBuTPcQo2UecTRlj+/btG3ysrfJxdci1F9hDjJJ9xGkPMUr2Eac9xCiRay83Jnn2v+whTnuIUbKPOO0hRsk+4mypebYubFYAcXV1VXBwsLKysvTrX//a3J6VlaWJEydWe0xYWJjeffddi7bt27crJCRELi4u5j5ZWVmaN2+eRZ/w8PAGzytJbm5ucnNzs2i7+uqr63ay9eTp6dlif9ir2EOMEnE2JnuIUbKPOFtajLbKx9Uh11qyhxgl+4jTHmKU7CNOe4hRanlxtpRcS561Zg9x2kOMkn3EaQ8xSvYRpz3EaMWwoY0bNxouLi7G2rVrjS+++MKIjY012rVrZxQUFBiGYRhPPPGEERUVZe7/z3/+07jqqquMefPmGV988YWxdu1aw8XFxXjzzTfNfT766CPDycnJ+MMf/mDk5+cbf/jDHwxnZ2fj448/rvO8tnLq1ClDknHq1CmbxlEbe4jRMIizMdlDjIZhH3G25BhtlY9toSV/DlXsIUbDsI847SFGw7CPOO0hRsNo2XG2llzbkj+Di9lDnPYQo2HYR5z2EKNh2Eec9hBjTWxaADEMw1ixYoXh7+9vuLq6GoMGDTJ2795tfm369OnG8OHDLfrv2rXLGDhwoOHq6moEBAQYqampVmO+8cYbRs+ePQ0XFxejV69exubNm+s1r63Yww+SPcRoGMTZmOwhRsOwjzhbeoy2ysfNraV/DoZhHzEahn3EaQ8xGoZ9xGkPMRpGy4+zNeTalv4ZVLGHOO0hRsOwjzjtIUbDsI847SHGmti8AIL/OnPmjPHMM88YZ86csXUoNbKHGA2DOBuTPcRoGPYRpz3E2BrYw+dgDzEahn3EaQ8xGoZ9xGkPMRqG/cTpyOzlM7CHOO0hRsOwjzjtIUbDsI847SHGmpgM4z+3kgYAAAAAAHBQbWwdAAAAAAAAQFOjAAIAAAAAABweBRDUqqCgQCaTSXl5ebYOBQAcFrkWAJoeuRYABZBmsm/fPjk5Oem2226zaL9cIq6srFRSUpJ69eqltm3bqmPHjhoyZIheeeWVRolrxowZMplM5s3b21u33Xab/va3v0mS/Pz8VFRUpD59+jTKfA2NcdKkSRZtNb2fku3f0+pcfA6Xns+ln0HVVt25NYfLxXrpZ9HUsVS9Hy4uLurevbvmz5+vsrKyZouhNiNGjFBsbKxV+zvvvCOTySRJWrBggQYMGGDV54cffpDJZNKuXbsk/ffntrrt448/bsKzcCzk2iuL0Z5zrT3l2UtjJNfWjDzb8rTUPCuRa8m11si1ddOacq2zrQNoLdatW6eHHnpIf/rTn1RYWKhu3brV6bgFCxZo9erVevnllxUSEqLS0lIdOHBA33//faPFdtttt5mTZHFxsZ566imNHz9ehYWFcnJykq+vb6PN1Vga+n5KzfOe1tfFn0EVNzc3G0XTslS9N2fPntXevXs1a9YslZWVKTU11dahNYkPPvhAN954o0Wbt7e3jaKxP+TaxuVIuZY8W7vWlGvJs1emJedZiVxLrm3ZyLW2z7WsAGkGZWVl2rRpkx544AGNHz9eaWlpdT723XffVUxMjO644w4FBgaqf//+mjlzpuLi4hotPjc3N/n6+srX11cDBgzQ448/rmPHjunbb7+1qjpXVlZq5syZCgwMVNu2bdWzZ08tW7bMYrxdu3bppptuUrt27XT11Vdr6NCh+vrrrxst3it5P6XmeU/r6+LPoGrr0KGDzeJpSareGz8/P919992655579M4776i8vFxz587VNddcI3d3d91888369NNPzcft2rVLJpNJf/nLX9S/f3+5u7tr8ODB+vzzz214Npfn7e1t9bPg4uJi67DsArmWXFsb8mztWlOuJc82XEvPsxK5llzbspFrbZ9rKYA0g4yMDPXs2VM9e/bUvffeq1deeUV1ffqwr6+v/vd//1fffvttE0d5wenTp/Xaa6+pR48e1Vbozp8/r+uuu06bNm3SF198od///vd68skntWnTJknSuXPnNGnSJA0fPlx/+9vflJ2drTlz5piXTjWGK3k/peZ/T9G42rZtq7Nnz+qxxx7T5s2btX79en322Wfq0aOHxowZo++++86i/6OPPqrFixfr008/1TXXXKNf/epXOnv2rI2iR1Mi15Jr0XjItaiOPeVZiVyLlo9c2/wogDSDtWvX6t5775V0YdnT6dOntWPHjjodu2TJEn377bfy9fVVv379FB0drb/+9a+NGt97770nDw8PeXh4qH379tq6dasyMjLUpo31j4eLi4sWLlyo0NBQBQYG6p577tGMGTPMXxSlpaU6deqUxo8fr+uvv169e/fW9OnT67WU73Ku5P2Umuc9ra+LP4Oq7dlnn7VpTC3RJ598otdff10jR45UamqqFi1apLFjxyooKEhr1qxR27ZttXbtWotjnnnmGY0ePVp9+/bV+vXr9e9//1tvv/22jc7g8sLDw61+FiorK20dll0g15Jra0OerTtHz7Xk2YZr6XlWIteSa+0HudY2KIA0sUOHDumTTz7RnXfeKUlydnZWZGSk1q1bV6fjg4KC9H//93/6+OOPdd999+nf//63JkyYoFmzZjVajCNHjlReXp7y8vK0f/9+RUREaOzYsTUu71u5cqVCQkLUuXNneXh4aM2aNSosLJQkdezYUTNmzNCYMWM0YcIELVu2TEVFRY0W65W+n1LzvKf1dfFnULU9+OCDNounJan6InV3d1dYWJhuueUWPfTQQzp79qyGDh1q7ufi4qKbbrpJ+fn5FseHhYWZ/92xY0f17NnTqk9LkpGRYfWz4OTkZOuwWjxyLbn2csiztWtNuZY82zD2kGclci25tmUj19o+13IT1Ca2du1anTt3Ttdee625zTAMubi41PnmRG3atFFoaKhCQ0M1b948vfrqq4qKilJCQoICAwOvOMZ27dqpR48e5v3g4GB5eXlpzZo1Vslz06ZNmjdvnl566SWFhYWpffv2WrRokfbv32/u88orr2ju3Ll6//33lZGRoaeeekpZWVkaMmTIFcd6ufezrtcYNvV7Wl+Xfgb4r6qquIuLi7p27SoXFxcdPHhQkqyWoBqGUadlqY21dNXT01OnTp2yav/hhx/k6el52T6S5OXlZdHu5+fHz0IDkGvJtZdDnq1dS8215NmWwx7yrESuJde2bORa22MFSBM6d+6cNmzYoJdeesmi8nXw4EH5+/vrtddea9C4QUFBktRkj0wymUxq06aNfv75Z6vX9u7dq/DwcMXExGjgwIHq0aOHjhw5YtVv4MCBio+P1759+9SnTx+9/vrrVxxXU72fUtO/p2i4qi9Sf39/842TevToIVdXV3344YfmfmfPntWBAwfUu3dvi+MvftzW999/ry+//FK9evVqlNh69eqlAwcOWLV/+umn6tmzp7nPN998o+LiYqs+bdq0aZFfDPaGXEuuxZVrqbmWPNsy2Guelci1aFnItbbHCpAm9N577+n777/XzJkzrSpiU6ZM0dq1azV+/HhJF5bAXSooKEh33323hg4dqvDwcPn6+uro0aOKj4/XL37xi0b7n7jy8nLzD/L333+vl19+WadPn9aECROs+vbo0UMbNmzQtm3bFBgYqD//+c/69NNPzdXlo0ePavXq1frVr36lrl276tChQ/ryyy81bdq0K46zLu/n7373O3ObLd/T+rr4M6ji7OysTp062SSelq5du3Z64IEH9Oijj6pjx47q1q2bXnzxRf3000+aOXOmRd/ExER5e3vLx8dHCQkJ6tSpU6M98z0mJkYvv/yyHnzwQc2ZM0dt27ZVVlaW1q5dqz//+c+SpIiICPXu3Vt33nmnnnvuOXXt2lV/+9vfNH/+fEVHR6t9+/YWY548edLqZ+Hqq6+Wu7t7o8TsiMi15Nq6IM/WX0vIteTZlsFe8qxEriXX2h9ybTMz0GTGjx9vjBs3rtrXcnJyDEnm/1a3HT161Fi9erUxcuRIo3Pnzoarq6vRrVs3Y8aMGUZBQUGjxDh9+nSLOdu3b2+EhoYab775pmEYhnH06FFDkpGbm2sYhmGcOXPGmDFjhuHl5WVcffXVxgMPPGA88cQTRv/+/Q3DMIzi4mJj0qRJRpcuXQxXV1fD39/f+P3vf29UVlZeUYwTJ06s8/tZFbOt3tPazuHSf1ftVxdrz549myyeK4n14v3mjOVSP//8s/HQQw8ZnTp1Mtzc3IyhQ4can3zyifn1nTt3GpKMd99917jxxhsNV1dXIzQ01MjLy2vUGA8cOGCMGTPGuOaaawxPT08jJCTESE9Pt+hTVFRk3HfffYa/v7/Rtm1bo1evXkZiYqJx5swZc5/afm4vHQ+WyLXk2ovjv/TfVfstKc9eGiO5tnbkWduzhzxrGORacq01cm3dtZZcazKMejxnCQDsxK5duzRy5Eh9//33uvrqq20dDgA4JHItADQ9cm3j4R4gAAAAAADA4VEAAQAAAAAADo9LYAAAAAAAgMNjBQgAAAAAAHB4FEAAAAAAAIDDowACAAAAAAAcHgUQAAAAAADg8CiAAAAAAAAAh0cBBKhFQUGBTCaT8vLyWsxcI0aMUGxsbJPHAwDNgTwLAE2PXAtcQAEEaCH8/PxUVFSkPn36SJJ27dolk8mkH374wbaBAYCDIM8CQNMj16Ilc7Z1AACkiooKubq6ytfX19ahAIBDIs8CQNMj16KlYwUIWr33339fN998s66++mp5e3tr/PjxOnLkSI39t27dqhtuuEFt27bVyJEjtX79equq9ubNm3XjjTfKzc1NAQEBeumllyzGCAgI0P/8z/9oxowZ8vLy0uzZsy2WCxYUFGjkyJGSpA4dOshkMmnGjBnm48+fP6/HHntMHTt2lK+vrxYsWGAxvslk0qpVqzR+/HhdddVV6t27t7Kzs/XVV19pxIgRateuncLCwmo9TwBoLORZAGh65FqgDgyglXvzzTeNzZs3G19++aWRm5trTJgwwejbt69RWVlpHD161JBk5ObmGoZhGEePHjVcXFyM+fPnG//4xz+M9PR049prrzUkGd9//71hGIZx4MABo02bNkZiYqJx6NAh45VXXjHatm1rvPLKK+Y5/f39DU9PT2PRokXG4cOHjcOHD1vMde7cOWPz5s2GJOPQoUNGUVGR8cMPPxiGYRjDhw83PD09jQULFhhffvmlsX79esNkMhnbt283jy/JuPbaa42MjAzj0KFDxqRJk4yAgADjl7/8pfH+++8bX3zxhTFkyBDjtttua663GUArRp4FgKZHrgUujwIIcIkTJ04YkozPP//c6svi8ccfN/r06WPRPyEhweLL4u677zZGjx5t0efRRx81goKCzPv+/v7GpEmTLPpcOtfOnTstxq0yfPhw4+abb7ZoCw0NNR5//HHzviTjqaeeMu9nZ2cbkoy1a9ea29LT0w13d/fLvyEA0MjIswDQ9Mi1gDUugUGrd+TIEd19993q3r27PD09FRgYKEkqLCy06nvo0CGFhoZatN10000W+/n5+Ro6dKhF29ChQ3X48GFVVlaa20JCQhocc79+/Sz2u3TpohMnTtTYx8fHR5LUt29fi7YzZ86otLS0wXEAQF2QZ8mzAJoeuZZci8vjJqho9SZMmCA/Pz+tWbNGXbt21fnz59WnTx9VVFRY9TUMQyaTyaqtvn0kqV27dg2O2cXFxWLfZDLp/PnzNfapiqe6tkuPA4DGRp4lzwJoeuRaci0ujwIIWrWTJ08qPz9fq1at0rBhwyRJH374YY39e/XqpczMTIu2AwcOWOwHBQVZjbFv3z794he/kJOTU51jc3V1lSSLCjsA2BvyLAA0PXItUDdcAoNWrUOHDvL29tbq1av11Vdf6X//938VFxdXY//f/va3+sc//qHHH39cX375pTZt2qS0tDRJ/60+P/LII9qxY4eeffZZffnll1q/fr1efvllzZ8/v16x+fv7y2Qy6b333tO3336r06dPN/g8AcBWyLMA0PTItUDdUABBq9amTRtt3LhROTk56tOnj+bNm6dFixbV2D8wMFBvvvmm3nrrLfXr10+pqalKSEiQJLm5uUmSBg0apE2bNmnjxo3q06ePfv/73ysxMdHikV91ce2112rhwoV64okn5OPjo9/97ncNPk8AsBXyLAA0PXItUDcmo7oLuQDU2XPPPaeVK1fq2LFjtg4FABwSeRYAmh65Fq0B9wAB6iklJUWhoaHy9vbWRx99pEWLFlHJBoBGRJ4FgKZHrkVrRAEEqKfDhw/rf/7nf/Tdd9+pW7dueuSRRxQfH2/rsADAYZBnAaDpkWvRGnEJDAAAAAAAcHjcBBUAAAAAADg8CiAAAAAAAMDhUQABAAAAAAAOjwIIAAAAAABweBRAAAAAAACAw6MAAgAAAAAAHB4FEAAAAAAA4PAogAAAAAAAAIdHAQQAAAAAADi8/we0LDNmD351tAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1091.11x300 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = sns.catplot(x = \"algorithm\", y = \"val\", data = pltData, kind=\"bar\", col = \"metric\", aspect=1.2, height=3, sharey=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e32b381",
   "metadata": {
    "papermill": {
     "duration": 0.006328,
     "end_time": "2024-12-07T20:57:11.437646",
     "exception": false,
     "start_time": "2024-12-07T20:57:11.431318",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Prediction RMSE\n",
    "\n",
    "We will also look at the prediction RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "fb282b1c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:11.450393Z",
     "iopub.status.busy": "2024-12-07T20:57:11.450158Z",
     "iopub.status.idle": "2024-12-07T20:57:25.574447Z",
     "shell.execute_reply": "2024-12-07T20:57:25.573808Z"
    },
    "papermill": {
     "duration": 14.132042,
     "end_time": "2024-12-07T20:57:25.575465",
     "exception": false,
     "start_time": "2024-12-07T20:57:11.443423",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "preds = preds.rename(columns={'score': 'prediction'})\n",
    "user_rmse = preds.groupby(['algorithm', 'user']).apply(lambda df: rmse(df['prediction'], df['rating']))\n",
    "user_rmse = user_rmse.reset_index(name='RMSE')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "73116a6d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:25.590676Z",
     "iopub.status.busy": "2024-12-07T20:57:25.590400Z",
     "iopub.status.idle": "2024-12-07T20:57:26.095624Z",
     "shell.execute_reply": "2024-12-07T20:57:26.095208Z"
    },
    "papermill": {
     "duration": 0.512965,
     "end_time": "2024-12-07T20:57:26.096379",
     "exception": false,
     "start_time": "2024-12-07T20:57:25.583414",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x1554f5cb80d0>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAHpCAYAAACFlZVCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAshUlEQVR4nO3df1TVdZ7H8dcV8MIqoJICFiiuhQhjumAjlb+yME3TPe5uTT8Uy2kZLUcZstC2qRyHOUfrkMeSLMTI8dfpWjZpruwk2qROYri1pWizKA4DmZWQbEHgd//oeKcrFwTl8oUPz8c5n1Ofz/187/f9/Z6v58X3x73XYVmWJQAAYKxudhcAAAB8i7AHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGC4Lhf2lmWpurpafL0AAKCr6HJh/8033yg0NFTffPON3aUAANAuulzYAwDQ1RD2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADD+dtdAACgfViWpZqaGne/R48ecjgcNlaE9kLYA0AXUVNTo2nTprn727ZtU8+ePW2sCO2Fy/gAABiOsAcAwHCEPQAAhiPsAQAwHA/oAYCNEh/Nb7d1OerrFPqj/rj/2CTLv7vP13to+UyfrwPN48weAADDEfYAABiOsAcAwHCEPQAAhuMBPQDoIiy/AFUN+5lHH10DYQ8AXYXD0S5P36Pj4TI+AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhuswYZ+VlSWHw6EFCxY0O2/Pnj1KTExUYGCgBg0apJycnPYpEACATqpDhP3Bgwe1Zs0aDRs2rNl5paWlmjx5skaPHq3i4mItXrxY8+fPl8vlaqdKAQDofGwP+3Pnzunee+/Vyy+/rN69ezc7NycnR9HR0crOzlZcXJzmzJmjBx54QCtWrGhymdraWlVXV3s0AAC6EtvDft68ebrjjjt06623XnLu/v37lZKS4jE2ceJEFRUV6fvvv/e6TFZWlkJDQ90tKiqqTeoGAKCzsDXsN23apA8//FBZWVktml9ZWanw8HCPsfDwcNXX1+vMmTNel8nMzFRVVZW7nTp16orrBgCgM7HtJ25PnTqlX/7yl9q1a5cCAwNbvJzD4fDoW5bldfwCp9Mpp9N5+YUCANDJ2Rb2hw4d0unTp5WYmOgea2ho0N69e7Vq1SrV1tbKz8/PY5mIiAhVVlZ6jJ0+fVr+/v4KCwtrl7oBAOhsbAv7CRMm6OOPP/YYmz17toYMGaLHHnusUdBLUnJysv7whz94jO3atUtJSUkKCAjwab0AAHRWtoV9cHCwEhISPMZ69OihsLAw93hmZqbKy8uVn58vSUpLS9OqVauUnp6un//859q/f79yc3O1cePGdq8fAIDOwvan8ZtTUVGhsrIydz8mJkY7duxQYWGhhg8frqVLl2rlypWaMWOGjVUCANCxOawLT7h1EdXV1QoNDVVVVZVCQkLsLgdAF5f4aL7dJfjcoeUz7S6hy+vQZ/YAAODKEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIazNexXr16tYcOGKSQkRCEhIUpOTtY777zT5PzCwkI5HI5G7ejRo+1YNQAAnYu/nSu/5ppr9Lvf/U6DBw+WJL366quaNm2aiouLFR8f3+RyJSUlCgkJcff79u3r81oBAOisbA37qVOnevSXLVum1atX68CBA82Gfb9+/dSrVy8fVwcAgBk6zD37hoYGbdq0STU1NUpOTm527ogRIxQZGakJEyZo9+7dzc6tra1VdXW1RwMAoCuxPew//vhj9ezZU06nU2lpaXrjjTc0dOhQr3MjIyO1Zs0auVwubd26VbGxsZowYYL27t3b5PtnZWUpNDTU3aKiony1KQAAdEgOy7IsOwuoq6tTWVmZzp49K5fLpVdeeUV79uxpMvAvNnXqVDkcDr311lteX6+trVVtba27X11draioKFVVVXnc9wcAOyQ+mm93CT53aPlMu0vo8my9Zy9J3bt3dz+gl5SUpIMHD+r555/XSy+91KLlR40apfXr1zf5utPplNPpbJNaAQDojGy/jH8xy7I8zsQvpbi4WJGRkT6sCACAzs3WM/vFixdr0qRJioqK0jfffKNNmzapsLBQO3fulCRlZmaqvLxc+fk/XObKzs7WwIEDFR8fr7q6Oq1fv14ul0sul8vOzQAAoEOzNew///xz3X///aqoqFBoaKiGDRumnTt36rbbbpMkVVRUqKyszD2/rq5OGRkZKi8vV1BQkOLj47V9+3ZNnjzZrk0AAKDDs/0BvfZWXV2t0NBQHtAD0CHwgB7aQ4e7Zw8AANoWYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABjO3+4CALSOZVmqqalx93v06CGHw2FjRQA6OsIe6GRqamo0bdo0d3/btm3q2bOnjRUB6Oi4jA8AgOEIewAADMdlfKCNJD6a3y7rcdTXKfRH/XH/sUmWf/d2Wfeh5TPbZT0A2hZn9gAAGI6wBwDAcLaG/erVqzVs2DCFhIQoJCREycnJeuedd5pdZs+ePUpMTFRgYKAGDRqknJycdqoW6BgsvwBVDfuZu1l+AXaXBKCDszXsr7nmGv3ud79TUVGRioqKdMstt2jatGn65JNPvM4vLS3V5MmTNXr0aBUXF2vx4sWaP3++XC5XO1cO2MjhkOXf3d3EZ+wBXIKtD+hNnTrVo79s2TKtXr1aBw4cUHx8fKP5OTk5io6OVnZ2tiQpLi5ORUVFWrFihWbMmNEeJQMA0Ol0mHv2DQ0N2rRpk2pqapScnOx1zv79+5WSkuIxNnHiRBUVFen777/3ukxtba2qq6s9GgAAXYntYf/xxx+rZ8+ecjqdSktL0xtvvKGhQ4d6nVtZWanw8HCPsfDwcNXX1+vMmTNel8nKylJoaKi7RUVFtfk2AADQkdke9rGxsTp8+LAOHDigX/ziF5o1a5Y+/fTTJudf/B3glmV5Hb8gMzNTVVVV7nbq1Km2Kx4AgE7A9i/V6d69uwYPHixJSkpK0sGDB/X888/rpZdeajQ3IiJClZWVHmOnT5+Wv7+/wsLCvL6/0+mU0+ls+8IBAOgkbD+zv5hlWaqtrfX6WnJysgoKCjzGdu3apaSkJAUE8PEjAAC8sTXsFy9erPfee08nTpzQxx9/rCVLlqiwsFD33nuvpB8uwc+c+fev50xLS9PJkyeVnp6uI0eOaO3atcrNzVVGRoZdmwAAQIdn62X8zz//XPfff78qKioUGhqqYcOGaefOnbrtttskSRUVFSorK3PPj4mJ0Y4dO7Rw4UK98MIL6t+/v1auXMnH7gAAaIatYZ+bm9vs6+vWrWs0NnbsWH344Yc+qggAAPN0uHv2AACgbRH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADCcrWGflZWlkSNHKjg4WP369dP06dNVUlLS7DKFhYVyOByN2tGjR9upagAAOhdbw37Pnj2aN2+eDhw4oIKCAtXX1yslJUU1NTWXXLakpEQVFRXudu2117ZDxQAAdD7+dq58586dHv28vDz169dPhw4d0pgxY5pdtl+/furVq5cPqwMAwAwd6p59VVWVJKlPnz6XnDtixAhFRkZqwoQJ2r17d5PzamtrVV1d7dEAAOhKOkzYW5al9PR03XzzzUpISGhyXmRkpNasWSOXy6WtW7cqNjZWEyZM0N69e73Oz8rKUmhoqLtFRUX5ahMAAOiQbL2M/2MPP/ywPvroI/3pT39qdl5sbKxiY2Pd/eTkZJ06dUorVqzweuk/MzNT6enp7n51dTWBDwDoUjrEmf0jjzyit956S7t379Y111zT6uVHjRql48ePe33N6XQqJCTEowEA0JXYemZvWZYeeeQRvfHGGyosLFRMTMxlvU9xcbEiIyPbuDoAAMxga9jPmzdPGzZs0LZt2xQcHKzKykpJUmhoqIKCgiT9cBm+vLxc+fn5kqTs7GwNHDhQ8fHxqqur0/r16+VyueRyuWzbDgAAOjJbw3716tWSpHHjxnmM5+XlKTU1VZJUUVGhsrIy92t1dXXKyMhQeXm5goKCFB8fr+3bt2vy5MntVTYAAJ2K7ZfxL2XdunUe/UWLFmnRokU+qggAAPN0iAf0AACA7xD2AAAYjrAHAMBwhD0AAIZrVdh/8MEHamhocPcvfsCutrZWW7ZsaZvKAABAm2hV2CcnJ+vLL79090NDQ/W///u/7v7Zs2f1s5/9rO2qAwAAV6xVYX/xmby3j8615ON0AACg/bT5PXuHw9HWbwkAAK4AD+gBAGC4Vn+D3qeffur+DnvLsnT06FGdO3dOknTmzJm2rQ4AAFyxVof9hAkTPO7LT5kyRdIPl+8ty+IyPgAAHUyrwr60tNRXdQAAAB9pVdgPGDDAV3UAAAAfadUDel999ZX++te/eox98sknmj17tv7t3/5NGzZsaNPiAADAlWtV2M+bN0/PPfecu3/69GmNHj1aBw8eVG1trVJTU/Xaa6+1eZEAAODytSrsDxw4oDvvvNPdz8/PV58+fXT48GFt27ZNv/3tb/XCCy+0eZEAAODytSrsKysrFRMT4+6/++67+ud//mf5+/9w6//OO+/U8ePH27ZCAABwRVoV9iEhITp79qy7/8EHH2jUqFHuvsPhUG1tbZsVBwAArlyrwv6GG27QypUrdf78eb3++uv65ptvdMstt7hfP3bsmKKiotq8SAAAcPla9dG7pUuX6tZbb9X69etVX1+vxYsXq3fv3u7XN23apLFjx7Z5kQAA4PK1KuyHDx+uI0eOaN++fYqIiNBPf/pTj9fvvvtuDR06tE0LBAAAV6bVX5fbt29fTZs2zetrd9xxxxUXBAAA2larwj4/P79F82bOnHlZxQAAgLbXqrBPTU1Vz5495e/v7/FjOD/mcDgIewAAOpBWhX1cXJw+//xz3XfffXrggQc0bNgwX9UFAADaSKs+evfJJ59o+/bt+vbbbzVmzBglJSVp9erVqq6u9lV9AADgCrUq7CXppz/9qV566SVVVFRo/vz52rJliyIjI3XvvffyhToAAHRArQ77C4KCgjRz5kw9/fTTuuGGG7Rp0yb93//9X1vWBgAA2sBlhX15ebl++9vf6tprr9Xdd9+tkSNH6pNPPvH4gh0AANAxtOoBvS1btigvL0979uzRxIkT9eyzz+qOO+6Qn5+fr+oDAABXqFVhf/fddys6OloLFy5UeHi4Tpw44fUnbefPn99mBQIAgCvTqrCPjo6Ww+HQhg0bmpzjcDgIewAAOpBWhf2JEycuOae8vPxyawEAAD5w2U/jX6yyslLz58/X4MGD2+otAQBAG2hV2J89e1b33nuv+vbtq/79+7t/2/7JJ5/UoEGDtH//fq1du9ZXtQIAgMvQqsv4ixcv1t69ezVr1izt3LlTCxcu1M6dO/Xdd9/pnXfe4bfsAQDogFoV9tu3b1deXp5uvfVWzZ07V4MHD9Z1112n7OxsH5UHAACuVKsu4//tb3/T0KFDJUmDBg1SYGCg5syZc9krz8rK0siRIxUcHKx+/fpp+vTpKikpueRye/bsUWJiogIDAzVo0CDl5ORcdg0AAJiuVWF//vx5BQQEuPt+fn7q0aPHZa98z549mjdvng4cOKCCggLV19crJSVFNTU1TS5TWlqqyZMna/To0SouLtbixYs1f/58uVyuy64DAACTteoyvmVZSk1NldPplCR99913SktLaxT4W7dubdH77dy506Ofl5enfv366dChQxozZozXZXJychQdHe2+dRAXF6eioiKtWLFCM2bMaM3mAADQJbQq7GfNmuXRv++++9q0mKqqKklSnz59mpyzf/9+paSkeIxNnDhRubm5+v777z2uPEhSbW2tx6/x8XO8AICuplVhn5eX56s6ZFmW0tPTdfPNNyshIaHJeZWVlQoPD/cYCw8PV319vc6cOaPIyEiP17KysvT000/7pGYAADqDNvtSnSv18MMP66OPPtLGjRsvOdfhcHj0LcvyOi5JmZmZqqqqcrdTp061TcEAAHQSrTqz95VHHnlEb731lvbu3atrrrmm2bkRERGqrKz0GDt9+rT8/f0VFhbWaL7T6XQ/YwAAQFdk65m9ZVl6+OGHtXXrVr377ruKiYm55DLJyckqKCjwGNu1a5eSkpIa3a8HAAA2h/28efO0fv16bdiwQcHBwaqsrFRlZaW+/fZb95zMzEzNnDnT3U9LS9PJkyeVnp6uI0eOaO3atcrNzVVGRoYdmwAAQIdna9ivXr1aVVVVGjdunCIjI91t8+bN7jkVFRUqKytz92NiYrRjxw4VFhZq+PDhWrp0qVauXMnH7gAAaIKt9+wvPFjXnHXr1jUaGzt2rD788EMfVAQAgHk6zNP4AADANwh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABjO1rDfu3evpk6dqv79+8vhcOjNN99sdn5hYaEcDkejdvTo0fYpGACATsjfzpXX1NTo+uuv1+zZszVjxowWL1dSUqKQkBB3v2/fvr4oDwAAI9ga9pMmTdKkSZNavVy/fv3Uq1evti8IAAADdcp79iNGjFBkZKQmTJig3bt3Nzu3trZW1dXVHg0AgK6kU4V9ZGSk1qxZI5fLpa1btyo2NlYTJkzQ3r17m1wmKytLoaGh7hYVFdWOFQMAYD9bL+O3VmxsrGJjY9395ORknTp1SitWrNCYMWO8LpOZman09HR3v7q6msAHAHQpnerM3ptRo0bp+PHjTb7udDoVEhLi0QAA6Eo6fdgXFxcrMjLS7jIAAOiwbL2Mf+7cOX322WfufmlpqQ4fPqw+ffooOjpamZmZKi8vV35+viQpOztbAwcOVHx8vOrq6rR+/Xq5XC65XC67NgEAgA7P1rAvKirS+PHj3f0L99ZnzZqldevWqaKiQmVlZe7X6+rqlJGRofLycgUFBSk+Pl7bt2/X5MmT2712AAA6C1vDfty4cbIsq8nX161b59FftGiRFi1a5OOqAAAwS6e/Zw8AAJpH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcLaG/d69ezV16lT1799fDodDb7755iWX2bNnjxITExUYGKhBgwYpJyfH94UCANCJ2Rr2NTU1uv7667Vq1aoWzS8tLdXkyZM1evRoFRcXa/HixZo/f75cLpePKwUAoPPyt3PlkyZN0qRJk1o8PycnR9HR0crOzpYkxcXFqaioSCtWrNCMGTN8VCUAAJ1bp7pnv3//fqWkpHiMTZw4UUVFRfr++++9LlNbW6vq6mqPBgBAV9Kpwr6yslLh4eEeY+Hh4aqvr9eZM2e8LpOVlaXQ0FB3i4qKao9SAQDoMDpV2EuSw+Hw6FuW5XX8gszMTFVVVbnbqVOnfF4jAAAdia337FsrIiJClZWVHmOnT5+Wv7+/wsLCvC7jdDrldDrbozwAADqkTnVmn5ycrIKCAo+xXbt2KSkpSQEBATZVBQBAx2Zr2J87d06HDx/W4cOHJf3w0brDhw+rrKxM0g+X4GfOnOmen5aWppMnTyo9PV1HjhzR2rVrlZubq4yMDDvKBwCgU7D1Mn5RUZHGjx/v7qenp0uSZs2apXXr1qmiosId/JIUExOjHTt2aOHChXrhhRfUv39/rVy5ko/dAQDQDFvDfty4ce4H7LxZt25do7GxY8fqww8/9GFVAACYpVPdswcAAK1H2AMAYLhO9dE7dB2WZammpsbd79GjR5PfpQAAaB5hjw6ppqZG06ZNc/e3bdumnj172lgRAJOZfoJB2AMAujzTTzAIe7RK4qP57bIeR32dQn/UH/cfm2T5d2+XdR9aPvPSkwCgEyHsbWT6ZSMAuBLtdXIh2XeC0V4nF4S9jUy/bAS0N/6ABrwj7NEhWX4Bqhr2M48+cCn8AQ14R9h7Yfp96U5xT9rhaLd79ABg+gkGYQ/A50z/A1rqJH9Eo2mGn2DwDXoAABiOM3sbmX7ZCADQMRD2djL8shHQ3vgDGvCOsAdgDv6ABrzinj0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAw9ke9i+++KJiYmIUGBioxMREvffee03OLSwslMPhaNSOHj3ajhUDANC52Br2mzdv1oIFC7RkyRIVFxdr9OjRmjRpksrKyppdrqSkRBUVFe527bXXtlPFAAB0PraG/XPPPacHH3xQc+bMUVxcnLKzsxUVFaXVq1c3u1y/fv0UERHhbn5+fu1UMQAAnY9tYV9XV6dDhw4pJSXFYzwlJUX79u1rdtkRI0YoMjJSEyZM0O7du5udW1tbq+rqao8GAEBXYlvYnzlzRg0NDQoPD/cYDw8PV2VlpddlIiMjtWbNGrlcLm3dulWxsbGaMGGC9u7d2+R6srKyFBoa6m5RUVFtuh0AAHR0/nYX4HA4PPqWZTUauyA2NlaxsbHufnJysk6dOqUVK1ZozJgxXpfJzMxUenq6u19dXU3gAwC6FNvO7K+66ir5+fk1Oos/ffp0o7P95owaNUrHjx9v8nWn06mQkBCPBgBAV2Jb2Hfv3l2JiYkqKCjwGC8oKNCNN97Y4vcpLi5WZGRkW5cHAIAxbL2Mn56ervvvv19JSUlKTk7WmjVrVFZWprS0NEk/XIIvLy9Xfn6+JCk7O1sDBw5UfHy86urqtH79erlcLrlcLjs3AwCADs3WsL/rrrv05Zdf6plnnlFFRYUSEhK0Y8cODRgwQJJUUVHh8Zn7uro6ZWRkqLy8XEFBQYqPj9f27ds1efJkuzYBAIAOz/YH9ObOnau5c+d6fW3dunUe/UWLFmnRokXtUBUAAOaw/etyAQCAbxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHCEPQAAhiPsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7AEAMBxhDwCA4Qh7AAAMR9gDAGA4wh4AAMMR9gAAGI6wBwDAcIQ9AACGI+wBADCc7WH/4osvKiYmRoGBgUpMTNR7773X7Pw9e/YoMTFRgYGBGjRokHJyctqpUgAAOidbw37z5s1asGCBlixZouLiYo0ePVqTJk1SWVmZ1/mlpaWaPHmyRo8ereLiYi1evFjz58+Xy+Vq58oBAOg8bA375557Tg8++KDmzJmjuLg4ZWdnKyoqSqtXr/Y6PycnR9HR0crOzlZcXJzmzJmjBx54QCtWrGjnygEA6Dz87VpxXV2dDh06pMcff9xjPCUlRfv27fO6zP79+5WSkuIxNnHiROXm5ur7779XQEBAo2Vqa2tVW1vr7ldVVUmSqqurm6ytofbbFm9HZ9Tctl+K6ftGuvz9w75pGvumaeybprFvpODgYDkcjitfkWWT8vJyS5L1/vvve4wvW7bMuu6667wuc+2111rLli3zGHv//fctSdbf/vY3r8v8+te/tiTRaDQajdbpWlVVVZtkrm1n9hdc/BeLZVnN/hXjbb638QsyMzOVnp7u7p8/f15fffWVwsLC2uavpStUXV2tqKgonTp1SiEhIXaX06Gwb5rGvmka+6Z57J+mdcR9Exwc3CbvY1vYX3XVVfLz81NlZaXH+OnTpxUeHu51mYiICK/z/f39FRYW5nUZp9Mpp9PpMdarV6/LL9xHQkJCOszB1dGwb5rGvmka+6Z57J+mmbhvbHtAr3v37kpMTFRBQYHHeEFBgW688UavyyQnJzeav2vXLiUlJXm9Xw8AAGx+Gj89PV2vvPKK1q5dqyNHjmjhwoUqKytTWlqapB8uwc+cOdM9Py0tTSdPnlR6erqOHDmitWvXKjc3VxkZGXZtAgAAHZ6t9+zvuusuffnll3rmmWdUUVGhhIQE7dixQwMGDJAkVVRUeHzmPiYmRjt27NDChQv1wgsvqH///lq5cqVmzJhh1yZcMafTqV//+teNbjWAfdMc9k3T2DfNY/80zeR947AuPOEGAACMZPvX5QIAAN8i7AEAMBxhDwCA4Qh7dBgnTpyQw+HQ4cOH7S4FAIxC2PvQvn375Ofnp9tvv91j/FKh1tDQoKysLA0ZMkRBQUHq06ePRo0apby8vHao2ndSU1PlcDjcLSwsTLfffrs++ugjSVJUVJT7UxldTWpqqqZPn+4x1tTxI3XdY+jHfrzPLt5/Fx9rF5q3fWm6S+2ni487U4wbN04LFixoNP7mm2+6vz31qaee0vDhwxvNOXv2rBwOhwoLCyX9/d+bt3bgwAEfbkXbsf3rck22du1aPfLII3rllVdUVlam6OjoFi331FNPac2aNVq1apWSkpJUXV2toqIiff311z6u2Pduv/12d+BUVlbqiSee0JQpU1RWViY/Pz9FRETYXGHHcbnHj2T2MdRSPz7WLjDxI1VoP//1X/+l+Ph4j7Gmvr21oyHsfaSmpkZbtmzRwYMHVVlZqXXr1unJJ59s0bJ/+MMfNHfuXP3rv/6re+z666/3Vantyul0ugM9IiJCjz32mMaMGaMvvvhCNTU1iomJUXFxsYYPH66GhgY99NBDevfdd1VZWano6GjNnTtXv/zlL93vV1hYqEWLFumTTz5RQECA4uPjtWHDBvd3NXRWV3L8SGYfQy3142MNaAthYWGd9pjiMr6PbN68WbGxsYqNjdV9992nvLw8tfQrDSIiIvTuu+/qiy++8HGV9jp37px+//vfa/DgwV7/Oj5//ryuueYabdmyRZ9++qmefPJJLV68WFu2bJEk1dfXa/r06Ro7dqw++ugj7d+/Xw899FCH+IGjK3Ulx4/UdY4hAC1D2PtIbm6u7rvvPkk/XE48d+6c/vjHP7Zo2eeee05ffPGFIiIiNGzYMKWlpemdd97xZbnt5u2331bPnj3Vs2dPBQcH66233tLmzZvVrVvjQzEgIEBPP/20Ro4cqZiYGN17771KTU11h311dbWqqqo0ZcoU/eM//qPi4uI0a9asVl3u7qiu5PiRzD6GWurHx9qFtnTpUrvLQid24403NjqmGhoa7C6rRQh7HygpKdEHH3ygu+++W5Lk7++vu+66S2vXrm3R8kOHDtX//M//6MCBA5o9e7Y+//xzTZ06VXPmzPFl2e1i/PjxOnz4sA4fPqw///nPSklJ0aRJk3Ty5Emv83NycpSUlKS+ffuqZ8+eevnll91fodynTx+lpqZq4sSJmjp1qp5//nlVVFS05+b4xJUeP5LZx1BL/fhYu9DmzZtnd1noxDZv3tzomPLz87O7rBbhnr0P5Obmqr6+XldffbV7zLIsBQQEtPgBqW7dumnkyJEaOXKkFi5cqPXr1+v+++/XkiVLFBMT46vSfa5Hjx4aPHiwu5+YmKjQ0FC9/PLLjYJoy5YtWrhwoZ599lklJycrODhYy5cv15///Gf3nLy8PM2fP187d+7U5s2b9cQTT6igoECjRo1qt21qa5c6fnr37t2i9zH1GGqpi481dC0hISGqqqpqNH727Fn3z9c2N0eSQkNDPcajoqI67THFmX0bq6+vV35+vp599lmPv/7++7//WwMGDNDvf//7y3rfoUOHSvrhwS2TOBwOdevWTd9++22j19577z3deOONmjt3rkaMGKHBgwfrL3/5S6N5I0aMUGZmpvbt26eEhARt2LChPUr3CV8dP5K5xxDgzZAhQ1RUVNRo/ODBg4qNjXXP+etf/6rKyspGc7p169Zpg90bzuzb2Ntvv62vv/5aDz74YKO/Cv/lX/5Fubm5mjJliqQfLtdebOjQobrnnnt000036cYbb1RERIRKS0uVmZmp6667TkOGDGmX7fCV2tpa9z+sr7/+WqtWrdK5c+c0derURnMHDx6s/Px8/ed//qdiYmL02muv6eDBg+6z0tLSUq1Zs0Z33nmn+vfvr5KSEh07dszjZ5E7m5YcPw8//LB7rCseQy3142PtAn9/f1111VU2VYT2NHfuXK1atUrz5s3TQw89pKCgIBUUFCg3N1evvfaaJCklJUVxcXG6++67tWzZMvXv318fffSRMjIylJaWpuDgYI/3/PLLLxsdU7169VJgYGC7bddls9CmpkyZYk2ePNnra4cOHbIkuf/rrZWWllpr1qyxxo8fb/Xt29fq3r27FR0dbaWmplonTpxo561pW7NmzfLY1uDgYGvkyJHW66+/blmWZZWWllqSrOLiYsuyLOu7776zUlNTrdDQUKtXr17WL37xC+vxxx+3rr/+esuyLKuystKaPn26FRkZaXXv3t0aMGCA9eSTT1oNDQ02beHlmzVrljVt2rQWHz8X9lVXO4Z+7MI+u/j/L/S97ZvY2Fh7irXRpfbTj/umKSoqsiZOnGj169fPCgkJsZKSkqyNGzd6zKmoqLBmz55tDRgwwAoKCrKGDBliPfPMM9Z3333nntPcv7eL36+j4iduAQAwHPfsAQAwHGEPAIDhCHsAAAxH2AMAYDjCHgAAwxH2AAAYjrAHAMBwhD0AAIYj7IEu4MSJE3I4HDp8+HCHWde4ceO0YMECn9cDgLAH0MaioqJUUVGhhIQESVJhYaEcDof7l8QAtD9+CAdAm6mrq1P37t0VERFhdykAfoQze8AQO3fu1M0336xevXopLCxMU6ZM8fqTwBe89dZbuvbaaxUUFKTx48fr1VdfbXQG7nK5FB8fL6fTqYEDB+rZZ5/1eI+BAwfqN7/5jVJTUxUaGqqf//znHpfxT5w4ofHjx0uSevfuLYfDodTUVPfy58+f16JFi9SnTx9FREToqaee8nh/h8Ohl156SVOmTNE//MM/KC4uTvv379dnn32mcePGqUePHkpOTm52OwGIX70DTPH6669bLpfLOnbsmFVcXGxNnTrV+slPfmI1NDQ0+kXB0tJSKyAgwMrIyLCOHj1qbdy40br66qstSdbXX39tWdYPvxjWrVs365lnnrFKSkqsvLw8KygoyMrLy3Ovc8CAAVZISIi1fPly6/jx49bx48c91lVfX2+5XC5LklVSUmJVVFRYZ8+etSzLssaOHWuFhIRYTz31lHXs2DHr1VdftRwOh7Vr1y73+0uyrr76amvz5s1WSUmJNX36dGvgwIHWLbfcYu3cudP69NNPrVGjRlm33357e+1moFMi7AFDnT592pJkffzxx43C/rHHHrMSEhI85i9ZssQj7O+55x7rtttu85jz6KOPWkOHDnX3BwwYYE2fPt1jzsXr2r17t8f7XjB27Fjr5ptv9hgbOXKk9dhjj7n7kqwnnnjC3d+/f78lycrNzXWPbdy40QoMDLz0DgG6MC7jA4b4y1/+onvuuUeDBg1SSEiIYmJiJEllZWWN5paUlGjkyJEeYzfccINH/8iRI7rppps8xm666SYdP35cDQ0N7rGkpKTLrnnYsGEe/cjISJ0+fbrJOeHh4ZKkn/zkJx5j3333naqrqy+7DsB0PKAHGGLq1KmKiorSyy+/rP79++v8+fNKSEhQXV1do7mWZcnhcDQaa+0cSerRo8dl1xwQEODRdzgcOn/+fJNzLtTjbezi5QD8HWEPGODLL7/UkSNH9NJLL2n06NGSpD/96U9Nzh8yZIh27NjhMVZUVOTRHzp0aKP32Ldvn6677jr5+fm1uLbu3btLksfVAADti8v4gAF69+6tsLAwrVmzRp999pneffddpaenNzn/3//933X06FE99thjOnbsmLZs2aJ169ZJ+vuZ8q9+9Sv98Y9/1NKlS3Xs2DG9+uqrWrVqlTIyMlpV24ABA+RwOPT222/riy++0Llz5y57OwFcHsIeMEC3bt20adMmHTp0SAkJCVq4cKGWL1/e5PyYmBi9/vrr2rp1q4YNG6bVq1dryZIlkiSn0ylJ+qd/+idt2bJFmzZtUkJCgp588kk988wzHh+da4mrr75aTz/9tB5//HGFh4fr4YcfvuztBHB5HJa3m3AAupxly5YpJydHp06dsrsUAG2Me/ZAF/Xiiy9q5MiRCgsL0/vvv6/ly5dz1g0YirAHuqjjx4/rN7/5jb766itFR0frV7/6lTIzM+0uC4APcBkfAADD8YAeAACGI+wBADAcYQ8AgOEIewAADEfYAwBgOMIeAADDEfYAABiOsAcAwHD/D9UNTonf+DgLAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 511.111x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.catplot(x='algorithm', y='RMSE', data=user_rmse, kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffbd7ba0",
   "metadata": {
    "papermill": {
     "duration": 0.006196,
     "end_time": "2024-12-07T20:57:26.109691",
     "exception": false,
     "start_time": "2024-12-07T20:57:26.103495",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Save Metrics\n",
    "\n",
    "We'll now save the top-N recommendation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1b776675",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:26.123099Z",
     "iopub.status.busy": "2024-12-07T20:57:26.122847Z",
     "iopub.status.idle": "2024-12-07T20:57:26.133048Z",
     "shell.execute_reply": "2024-12-07T20:57:26.132671Z"
    },
    "papermill": {
     "duration": 0.017916,
     "end_time": "2024-12-07T20:57:26.133739",
     "exception": false,
     "start_time": "2024-12-07T20:57:26.115823",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RecipRank</th>\n",
       "      <th>NDCG</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ALS</th>\n",
       "      <td>0.053269</td>\n",
       "      <td>0.053823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bias</th>\n",
       "      <td>0.043377</td>\n",
       "      <td>0.050045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IALS</th>\n",
       "      <td>0.213257</td>\n",
       "      <td>0.231143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IIE</th>\n",
       "      <td>0.025549</td>\n",
       "      <td>0.038885</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>III</th>\n",
       "      <td>0.255053</td>\n",
       "      <td>0.192291</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pop</th>\n",
       "      <td>0.152608</td>\n",
       "      <td>0.130418</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UUE</th>\n",
       "      <td>0.016278</td>\n",
       "      <td>0.031407</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           RecipRank      NDCG\n",
       "algorithm                     \n",
       "ALS         0.053269  0.053823\n",
       "Bias        0.043377  0.050045\n",
       "IALS        0.213257  0.231143\n",
       "IIE         0.025549  0.038885\n",
       "III         0.255053  0.192291\n",
       "Pop         0.152608  0.130418\n",
       "UUE         0.016278  0.031407"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "saved = results[['recip_rank', 'ndcg']].rename(columns={'ndcg': 'NDCG', 'recip_rank': 'RecipRank'})\n",
    "saved = saved.groupby('algorithm').agg('mean')\n",
    "saved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c4083a95",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:57:26.147128Z",
     "iopub.status.busy": "2024-12-07T20:57:26.146897Z",
     "iopub.status.idle": "2024-12-07T20:57:26.151492Z",
     "shell.execute_reply": "2024-12-07T20:57:26.151106Z"
    },
    "papermill": {
     "duration": 0.012185,
     "end_time": "2024-12-07T20:57:26.152259",
     "exception": false,
     "start_time": "2024-12-07T20:57:26.140074",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "saved.to_json(f'eval-metrics.{dataset}.json', orient='index')"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md",
   "notebook_metadata_filter": "split_at_heading"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 35.897875,
   "end_time": "2024-12-07T20:57:26.574634",
   "environment_variables": {},
   "exception": null,
   "input_path": "eval-report.ipynb",
   "output_path": "eval-report.ml1m.ipynb",
   "parameters": {
    "dataset": "ml1m"
   },
   "start_time": "2024-12-07T20:56:50.676759",
   "version": "2.6.0"
  },
  "split_at_heading": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}