eval-report.ml100k.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ce485e34",
   "metadata": {
    "papermill": {
     "duration": 0.010755,
     "end_time": "2024-12-07T20:27:59.211184",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.200429",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Evaluation Analysis for Recommender Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17c9a5e6",
   "metadata": {
    "papermill": {
     "duration": 0.001507,
     "end_time": "2024-12-07T20:27:59.214439",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.212932",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this section we will analyze and compare the generated recommendations and predictions from a predefined list of algorithms with the goal of assessing the performance of each algorithm with respect to a metric. In other words, we would rank the algorithms for each metric considered with respect to performance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f14bb433",
   "metadata": {
    "papermill": {
     "duration": 0.00146,
     "end_time": "2024-12-07T20:27:59.217379",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.215919",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b8308de",
   "metadata": {
    "papermill": {
     "duration": 0.001432,
     "end_time": "2024-12-07T20:27:59.220294",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.218862",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Below are the list of packages required to successfully run the analysis. They are divided into partitions to signify their specific task.<br>\n",
    "We need the pathlib package for working with files and folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "77d2e6e9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.224508Z",
     "iopub.status.busy": "2024-12-07T20:27:59.224317Z",
     "iopub.status.idle": "2024-12-07T20:27:59.228141Z",
     "shell.execute_reply": "2024-12-07T20:27:59.227828Z"
    },
    "papermill": {
     "duration": 0.006983,
     "end_time": "2024-12-07T20:27:59.228735",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.221752",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8607dbba",
   "metadata": {
    "papermill": {
     "duration": 0.00153,
     "end_time": "2024-12-07T20:27:59.231772",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.230242",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We would use the pandas for analyzing and manipulating our data while seaborn and matplotlib are used for data visualization. statsmodels.graphics.gofplots and scipy.stats.shapiro are used for normality check. Scipy.stats.friedmanchisquare is a non-parametric test used to determine the statistical significance in metric results and the wilcoxon test is used for pairwise comparison of sample data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "54c9b1a0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.235688Z",
     "iopub.status.busy": "2024-12-07T20:27:59.235307Z",
     "iopub.status.idle": "2024-12-07T20:27:59.661466Z",
     "shell.execute_reply": "2024-12-07T20:27:59.660925Z"
    },
    "papermill": {
     "duration": 0.429167,
     "end_time": "2024-12-07T20:27:59.662454",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.233287",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from statsmodels.graphics.gofplots import qqplot\n",
    "from scipy.stats import shapiro\n",
    "from scipy.stats import friedmanchisquare, wilcoxon\n",
    "from itertools import combinations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09d4299f",
   "metadata": {
    "papermill": {
     "duration": 0.001645,
     "end_time": "2024-12-07T20:27:59.674913",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.673268",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Logging to show what's happening in LensKit routines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fd84093f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.678727Z",
     "iopub.status.busy": "2024-12-07T20:27:59.678599Z",
     "iopub.status.idle": "2024-12-07T20:27:59.685941Z",
     "shell.execute_reply": "2024-12-07T20:27:59.685691Z"
    },
    "papermill": {
     "duration": 0.009795,
     "end_time": "2024-12-07T20:27:59.686298",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.676503",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.util.log notebook logging configured\n"
     ]
    }
   ],
   "source": [
    "from lenskit import util\n",
    "util.log_to_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c7c3c1f0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.689983Z",
     "iopub.status.busy": "2024-12-07T20:27:59.689905Z",
     "iopub.status.idle": "2024-12-07T20:27:59.691556Z",
     "shell.execute_reply": "2024-12-07T20:27:59.691315Z"
    },
    "papermill": {
     "duration": 0.003886,
     "end_time": "2024-12-07T20:27:59.691865",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.687979",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "_log = logging.getLogger('eval-report')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af5b4ebc",
   "metadata": {
    "papermill": {
     "duration": 0.001571,
     "end_time": "2024-12-07T20:27:59.695064",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.693493",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We will use lenskit for training, running, and evaluating recommender algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "803d7eb5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.698511Z",
     "iopub.status.busy": "2024-12-07T20:27:59.698438Z",
     "iopub.status.idle": "2024-12-07T20:27:59.724213Z",
     "shell.execute_reply": "2024-12-07T20:27:59.723780Z"
    },
    "papermill": {
     "duration": 0.028108,
     "end_time": "2024-12-07T20:27:59.724775",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.696667",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from lenskit import topn\n",
    "from lenskit.metrics.predict import rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62cde0fa",
   "metadata": {
    "papermill": {
     "duration": 0.001677,
     "end_time": "2024-12-07T20:27:59.728336",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.726659",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b8bdd0f",
   "metadata": {
    "papermill": {
     "duration": 0.001631,
     "end_time": "2024-12-07T20:27:59.731611",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.729980",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We specify the dataset we will use for our analysis and the main directory from where we read the recommendation and prediction files. From the main directory we find all the directories associated with the dataset and then read the recommendation and predictions files from those directories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "66d37091",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.735267Z",
     "iopub.status.busy": "2024-12-07T20:27:59.735173Z",
     "iopub.status.idle": "2024-12-07T20:27:59.737008Z",
     "shell.execute_reply": "2024-12-07T20:27:59.736670Z"
    },
    "papermill": {
     "duration": 0.004416,
     "end_time": "2024-12-07T20:27:59.737626",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.733210",
     "status": "completed"
    },
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "dataset = \"ml100k\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f5422058",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.741223Z",
     "iopub.status.busy": "2024-12-07T20:27:59.741146Z",
     "iopub.status.idle": "2024-12-07T20:27:59.742625Z",
     "shell.execute_reply": "2024-12-07T20:27:59.742390Z"
    },
    "papermill": {
     "duration": 0.003923,
     "end_time": "2024-12-07T20:27:59.743189",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.739266",
     "status": "completed"
    },
    "tags": [
     "injected-parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "dataset = \"ml100k\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b7f9f66b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.746936Z",
     "iopub.status.busy": "2024-12-07T20:27:59.746863Z",
     "iopub.status.idle": "2024-12-07T20:27:59.748358Z",
     "shell.execute_reply": "2024-12-07T20:27:59.748122Z"
    },
    "papermill": {
     "duration": 0.00412,
     "end_time": "2024-12-07T20:27:59.748944",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.744824",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "output_root = Path(\"runs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8688a4d0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.752681Z",
     "iopub.status.busy": "2024-12-07T20:27:59.752593Z",
     "iopub.status.idle": "2024-12-07T20:27:59.754352Z",
     "shell.execute_reply": "2024-12-07T20:27:59.754108Z"
    },
    "papermill": {
     "duration": 0.004303,
     "end_time": "2024-12-07T20:27:59.754928",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.750625",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "dirs = [fld for fld in output_root.glob(f'{dataset}-*')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "56d609aa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.758835Z",
     "iopub.status.busy": "2024-12-07T20:27:59.758762Z",
     "iopub.status.idle": "2024-12-07T20:27:59.895323Z",
     "shell.execute_reply": "2024-12-07T20:27:59.894876Z"
    },
    "papermill": {
     "duration": 0.139239,
     "end_time": "2024-12-07T20:27:59.895922",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.756683",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 660100 entries, 0 to 660099\n",
      "Data columns (total 5 columns):\n",
      " #   Column     Non-Null Count   Dtype   \n",
      "---  ------     --------------   -----   \n",
      " 0   item       660100 non-null  int64   \n",
      " 1   score      660100 non-null  float64 \n",
      " 2   user       660100 non-null  int64   \n",
      " 3   rank       660100 non-null  int64   \n",
      " 4   algorithm  660100 non-null  category\n",
      "dtypes: category(1), float64(1), int64(3)\n",
      "memory usage: 20.8 MB\n"
     ]
    }
   ],
   "source": [
    "recs = []\n",
    "for fld in dirs:\n",
    "    for file in fld.glob(\"recs-*\"):\n",
    "        rec = pd.read_csv(file)\n",
    "        rec[\"algorithm\"] = fld.name.split(\"-\")[1]\n",
    "        recs.append(rec)\n",
    "\n",
    "recs = pd.concat(recs, ignore_index=True)\n",
    "recs = recs.astype({'algorithm': 'category'})\n",
    "recs.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "feaa4493",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.907731Z",
     "iopub.status.busy": "2024-12-07T20:27:59.907641Z",
     "iopub.status.idle": "2024-12-07T20:27:59.912005Z",
     "shell.execute_reply": "2024-12-07T20:27:59.911762Z"
    },
    "papermill": {
     "duration": 0.014601,
     "end_time": "2024-12-07T20:27:59.912592",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.897991",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['UUE', 'ALS', 'IIE', 'Bias', 'III', 'Pop', 'IALS']\n",
       "Categories (7, object): ['ALS', 'Bias', 'IALS', 'IIE', 'III', 'Pop', 'UUE']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rec_algos = recs['algorithm'].unique()\n",
    "rec_algos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b378a77c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.916792Z",
     "iopub.status.busy": "2024-12-07T20:27:59.916706Z",
     "iopub.status.idle": "2024-12-07T20:27:59.937674Z",
     "shell.execute_reply": "2024-12-07T20:27:59.937311Z"
    },
    "papermill": {
     "duration": 0.023891,
     "end_time": "2024-12-07T20:27:59.938265",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.914374",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 18860 entries, 0 to 18859\n",
      "Data columns (total 6 columns):\n",
      " #   Column      Non-Null Count  Dtype   \n",
      "---  ------      --------------  -----   \n",
      " 0   user        18860 non-null  int64   \n",
      " 1   item        18860 non-null  int64   \n",
      " 2   rating      18860 non-null  float64 \n",
      " 3   timestamp   18860 non-null  int64   \n",
      " 4   prediction  18850 non-null  float64 \n",
      " 5   algorithm   18860 non-null  category\n",
      "dtypes: category(1), float64(2), int64(3)\n",
      "memory usage: 755.5 KB\n"
     ]
    }
   ],
   "source": [
    "preds = []\n",
    "for fld in dirs:\n",
    "    for file in fld.glob(\"pred-*\"):\n",
    "        pred = pd.read_csv(file)\n",
    "        pred[\"algorithm\"] = fld.name.split(\"-\")[1]\n",
    "        preds.append(pred)\n",
    "\n",
    "preds = pd.concat(preds, ignore_index=True)\n",
    "preds = preds.astype({'algorithm': 'category'})\n",
    "preds.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca476ae1",
   "metadata": {
    "papermill": {
     "duration": 0.001755,
     "end_time": "2024-12-07T20:27:59.941918",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.940163",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We need to load the test data so that we have the ground truths for computing accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "25294727",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.946073Z",
     "iopub.status.busy": "2024-12-07T20:27:59.945985Z",
     "iopub.status.idle": "2024-12-07T20:27:59.947498Z",
     "shell.execute_reply": "2024-12-07T20:27:59.947300Z"
    },
    "papermill": {
     "duration": 0.004335,
     "end_time": "2024-12-07T20:27:59.947976",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.943641",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "split_root = Path(\"data-split\")\n",
    "split_dir = split_root / dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "88fc0d4e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.951907Z",
     "iopub.status.busy": "2024-12-07T20:27:59.951840Z",
     "iopub.status.idle": "2024-12-07T20:27:59.975000Z",
     "shell.execute_reply": "2024-12-07T20:27:59.974718Z"
    },
    "papermill": {
     "duration": 0.025608,
     "end_time": "2024-12-07T20:27:59.975346",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.949738",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>item</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>part</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14425</th>\n",
       "      <td>1</td>\n",
       "      <td>255</td>\n",
       "      <td>2.0</td>\n",
       "      <td>885345822</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77073</th>\n",
       "      <td>1</td>\n",
       "      <td>124</td>\n",
       "      <td>5.0</td>\n",
       "      <td>875071484</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20952</th>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>4.0</td>\n",
       "      <td>875072404</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>933</th>\n",
       "      <td>1</td>\n",
       "      <td>155</td>\n",
       "      <td>2.0</td>\n",
       "      <td>878542201</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40519</th>\n",
       "      <td>1</td>\n",
       "      <td>109</td>\n",
       "      <td>5.0</td>\n",
       "      <td>874965739</td>\n",
       "      <td>test-3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user  item  rating  timestamp    part\n",
       "index                                       \n",
       "14425     1   255     2.0  885345822  test-3\n",
       "77073     1   124     5.0  875071484  test-3\n",
       "20952     1    22     4.0  875072404  test-3\n",
       "933       1   155     2.0  878542201  test-3\n",
       "40519     1   109     5.0  874965739  test-3"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = []\n",
    "for file in split_dir.glob(\"test-*.parquet\"):\n",
    "    test.append(pd.read_parquet(file,).assign(part=file.stem.replace('.parquet', '')))\n",
    "\n",
    "test = pd.concat(test).rename(columns={\n",
    "    'user_id': 'user',\n",
    "    'item_id': 'item',\n",
    "})\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5770c372",
   "metadata": {
    "papermill": {
     "duration": 0.001902,
     "end_time": "2024-12-07T20:27:59.979261",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.977359",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Top-N Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e347325",
   "metadata": {
    "papermill": {
     "duration": 0.001876,
     "end_time": "2024-12-07T20:27:59.983019",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.981143",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The topn.RecListAnalysis class computes top-N metrics for recommendation list and takes care of making sure that the recommendations and ground truths are properly matched. Refer to the documentation for detailed explanation of the purpose for the RecListAnalysis class and how the analysis is done - https://lkpy.lenskit.org/en/stable/evaluation/topn-metrics.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "198eda4d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:27:59.987111Z",
     "iopub.status.busy": "2024-12-07T20:27:59.987025Z",
     "iopub.status.idle": "2024-12-07T20:28:00.150720Z",
     "shell.execute_reply": "2024-12-07T20:28:00.150250Z"
    },
    "papermill": {
     "duration": 0.166406,
     "end_time": "2024-12-07T20:28:00.151284",
     "exception": false,
     "start_time": "2024-12-07T20:27:59.984878",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn analyzing 660100 recommendations (4715 truth rows)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn using rec key columns ['algorithm', 'user']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn using truth key columns ['user']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn numbering truth lists\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn numbering rec lists\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn collecting metric results\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn measured 6601 lists in  95ms\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn filling in missing user info (6601 initial rows)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/linuxbrew/.linuxbrew/envs/lkdemo/lib/python3.10/site-packages/lenskit/topn.py:152: FutureWarning: In a future version of pandas, a length 1 tuple will be returned when iterating over a groupby with a grouper equal to a list of length 1. Don't supply a list with a single grouper to avoid this warning.\n",
      "  for key, df in res.groupby(ug_cols):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>nrecs</th>\n",
       "      <th>precision</th>\n",
       "      <th>recip_rank</th>\n",
       "      <th>ndcg</th>\n",
       "      <th>ntruth</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th>user</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">ALS</th>\n",
       "      <th>1</th>\n",
       "      <td>100</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.031250</td>\n",
       "      <td>0.102606</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>100</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.170686</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>100</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.454573</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>100</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.071429</td>\n",
       "      <td>0.073745</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>100</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                nrecs  precision  recip_rank      ndcg  ntruth\n",
       "algorithm user                                                \n",
       "ALS       1       100       0.02    0.031250  0.102606       5\n",
       "          10      100       0.02    0.166667  0.170686       5\n",
       "          12      100       0.05    0.333333  0.454573       5\n",
       "          20      100       0.01    0.071429  0.073745       5\n",
       "          22      100       0.00    0.000000  0.000000       5"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rla = topn.RecListAnalysis()\n",
    "\n",
    "rla.add_metric(topn.precision)\n",
    "rla.add_metric(topn.recip_rank)\n",
    "rla.add_metric(topn.ndcg)\n",
    "results = rla.compute(recs, test.drop(columns=['rating']), include_missing=True)\n",
    "results = results.fillna(0)\n",
    "results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75dbe033",
   "metadata": {
    "papermill": {
     "duration": 0.009024,
     "end_time": "2024-12-07T20:28:00.162761",
     "exception": false,
     "start_time": "2024-12-07T20:28:00.153737",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We will reshape the 'results' dataframe by stacking the columns to index and then use the bar chart to visualize the performance of our algorithms with respect to the precision, reciprocal rank and ndcg metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4c7ffaf7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:28:00.167543Z",
     "iopub.status.busy": "2024-12-07T20:28:00.167414Z",
     "iopub.status.idle": "2024-12-07T20:28:00.172437Z",
     "shell.execute_reply": "2024-12-07T20:28:00.172254Z"
    },
    "papermill": {
     "duration": 0.008086,
     "end_time": "2024-12-07T20:28:00.172966",
     "exception": false,
     "start_time": "2024-12-07T20:28:00.164880",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>algorithm</th>\n",
       "      <th>user</th>\n",
       "      <th>metric</th>\n",
       "      <th>val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ALS</td>\n",
       "      <td>1</td>\n",
       "      <td>precision</td>\n",
       "      <td>0.020000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ALS</td>\n",
       "      <td>1</td>\n",
       "      <td>recip_rank</td>\n",
       "      <td>0.031250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ALS</td>\n",
       "      <td>1</td>\n",
       "      <td>ndcg</td>\n",
       "      <td>0.102606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ALS</td>\n",
       "      <td>10</td>\n",
       "      <td>precision</td>\n",
       "      <td>0.020000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ALS</td>\n",
       "      <td>10</td>\n",
       "      <td>recip_rank</td>\n",
       "      <td>0.166667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  algorithm  user      metric       val\n",
       "0       ALS     1   precision  0.020000\n",
       "1       ALS     1  recip_rank  0.031250\n",
       "2       ALS     1        ndcg  0.102606\n",
       "3       ALS    10   precision  0.020000\n",
       "4       ALS    10  recip_rank  0.166667"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pltData = (results.drop(columns=['nrecs', 'ntruth']).stack()).reset_index()\n",
    "pltData.columns = ['algorithm', 'user', 'metric', 'val']\n",
    "pltData.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66c6e8ba",
   "metadata": {
    "papermill": {
     "duration": 0.002177,
     "end_time": "2024-12-07T20:28:00.177337",
     "exception": false,
     "start_time": "2024-12-07T20:28:00.175160",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We need to determine if the differences we observe in the performances of the algorithms for the various metrics are statistically significant. To achieve this, we will need to use either a parametric or non-parametric statistical test for comparing the differences. We will consider a parametric test - repeated ANOVA measure cause our sample groups are correlated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bd651349",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:28:00.182121Z",
     "iopub.status.busy": "2024-12-07T20:28:00.182030Z",
     "iopub.status.idle": "2024-12-07T20:28:00.627164Z",
     "shell.execute_reply": "2024-12-07T20:28:00.626861Z"
    },
    "papermill": {
     "duration": 0.448265,
     "end_time": "2024-12-07T20:28:00.627812",
     "exception": false,
     "start_time": "2024-12-07T20:28:00.179547",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABDcAAAEiCAYAAADtbjlYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABNcUlEQVR4nO3dfVxUZf7/8feI3AWKigaa3KUlEJoKlmimVmKWhm0mWaEWZoRZSlqx6KbWxrqaohWopandKJaW2lJKfjM1zZKgbb+RmV8Nt4W8qSQoQfD8/vDnrCOg3A8HXs/H4zzqXHOdaz7XDI/PjJ+5zjkWwzAMAQAAAAAAmFQLewcAAAAAAABQGxQ3AAAAAACAqVHcAAAAAAAApkZxAwAAAAAAmBrFDQAAAAAAYGoUNwAAAAAAgKlR3AAAAAAAAKZGcQMAAAAAAJgaxQ0AAAAAAGBqFDfQLMyaNUs9e/a0dxh2YbFY9N5779V5XwDNV1PNqf7+/kpOTrZ3GFVCvgZQHWbN24MGDdKUKVPsHQZMwmIYhmHvIICaGDRokHr27FmlL6KFhYUqLi6Wp6dn/QfWyOTn56tt27Zydnau074AmhZyqnTs2DG5ubnpsssus3col2SxWPTuu+9q5MiR9g4FgJ00h7xdnTkCLe0dAFCfDMNQWVmZ3N3d5e7ubu9wqqWkpEROTk61Hsfb27te+gJofhprTj19+rQcHR1rPU6HDh3qIJrK1VWcAFBVjTVvA/WB01JQ7wYNGqTJkydrypQpatu2rby8vLRs2TIVFRXpgQceUKtWrdSlSxd98MEHNsd98803uu222+Tu7i4vLy9FR0fr+PHjkqTx48frk08+0aJFi2SxWGSxWHT48GFt375dFotFW7ZsUVhYmJydnbVz584Kl+KtWLFC11xzjZydndWxY0c9+uij9fYanHv+pUuXysfHR5dddpnuvvtu/frrr9Y+48eP18iRI5WUlKROnTrp6quvliT9+OOPioqKUtu2beXp6anIyEgdPny4ynM5f+lySUmJHn30UXXs2FEuLi7y9/dXUlJShX0l6euvv9ZNN90kV1dXeXp6auLEiSosLCwX8/z589WxY0d5enpq0qRJOn36dN29eABskFP/m1NXrFihK6+8Us7OzjIMQydPntTEiRN1+eWXq3Xr1rrpppv01Vdf2Ry7adMmhYWFycXFRe3bt9ef/vQn62MXnpZisViUmpqqYcOGydXVVQEBAXr77berFOPhw4dlsVi0bt06DRo0SC4uLnrjjTd04sQJjRkzRp07d9Zll12m7t27a82aNTbHDho0SI899piefPJJtWvXTt7e3po1a9ZFn2/OnDny8vJSdnZ2leID0HDI2//N26+//rr8/f3l4eGhe+65R7/99pu1T1FRkcaOHSt3d3d17NhRL7zwQrlxiouL9eSTT8rHx0fOzs666qqrtHz5cuvjmzZt0lVXXSVXV1cNHjxYq1atksVisfnOjaaL4gYaxKpVq9S+fXt9/vnnmjx5sh555BHdfffd6tevn7788ksNHTpU0dHR+v333yVJeXl5GjhwoHr27Kl9+/bpww8/1E8//aTRo0dLkhYtWqTw8HA99NBDysvLU15ennx8fKzP9+STTyopKUk5OTnq0aNHuXhSU1M1adIkTZw4UV9//bU2bdqkrl27Vhr/m2++aa14V7a9+eabF30Nvv/+e61bt06bN2/Whx9+qOzsbE2aNMmmz7Zt25STk6OMjAy9//77+v333zV48GC5u7trx44d2rVrl9zd3XXrrbeqpKSk2nNZvHixNm3apHXr1mn//v1644035O/vX2Hf33//Xbfeeqvatm2rL774Qm+//bY++uijch98H3/8sQ4ePKiPP/5Yq1at0sqVK7Vy5cqLvhYAaoec+t+cun79eus/6G+//Xbl5+crPT1dmZmZ6t27t26++Wb9/PPPkqR//OMf+tOf/qTbb79dWVlZ2rZtm8LCwi76PDNnztRdd92lr776Svfff7/GjBmjnJycix5zvqeeekqPPfaYcnJyNHToUJ06dUqhoaF6//339a9//UsTJ05UdHS09u7da3PcqlWr5Obmpr179+rvf/+75syZo4yMjHLjG4ahxx9/XMuXL9euXbtMeU490ByQt6WDBw/qvffe0/vvv6/3339fn3zyif72t79ZH58+fbo+/vhjvfvuu9q6dau2b9+uzMxMmzHGjh2rtWvXavHixcrJydGSJUusK1IOHz6sUaNGaeTIkcrOztbDDz+sxMTES7wzaFIMoJ4NHDjQuOGGG6z7paWlhpubmxEdHW1ty8vLMyQZe/bsMQzDMGbOnGlERETYjHPkyBFDkrF//37ruI8//rhNn48//tiQZLz33ns27c8884xx7bXXWvc7depkJCYmVnkOBQUFxoEDBy66FRQUVHr8M888Yzg4OBhHjhyxtn3wwQdGixYtjLy8PMMwDGPcuHGGl5eXUVxcbO2zfPlyo1u3bsaZM2esbcXFxYarq6uxZcuWKs1FkvHuu+8ahmEYkydPNm666Sab8Srru2zZMqNt27ZGYWGh9fF//OMfRosWLYz8/HxrzH5+fkZpaam1z913321ERUVVGg+A2iGnnn1+R0dH4+jRo9a2bdu2Ga1btzZOnTpl07dLly7G0qVLDcMwjPDwcOO+++6rdFw/Pz9j4cKF1n1JRmxsrE2f66+/3njkkUcuOcdDhw4Zkozk5ORL9r3tttuMJ554wrp/4XtsGIbRp08f46mnnrKJ7e233zbuv/9+IzAw0ObzBUDjQt4++/yXXXaZTZ/p06cb119/vWEYhvHbb78ZTk5Oxtq1a62PnzhxwnB1dbXOcf/+/YYkIyMjo8LneOqpp4yQkBCbtsTEREOS8csvv1R5rjAvrrmBBnF+xdjBwUGenp7q3r27tc3Ly0uSdPToUUlSZmamPv744wrPDTx48KD1lI3KXOyXuKNHj+o///mPbr755irH36pVK7Vq1arK/Svi6+urzp07W/fDw8N15swZ7d+/33qti+7du9tcZyMzM1Pff/99uec+deqUDh48WO25jB8/XkOGDFG3bt106623avjw4YqIiKiwb05Ojq699lq5ublZ2/r372+N+dx7ds0118jBwcHap2PHjvr666+rFA+AmiGnSn5+fjbXyMjMzFRhYWG5i+X98ccfOnjwoCQpOztbDz30ULWeJzw8vNx+dU79uPC1Kysr09/+9jelpaXpxx9/VHFxsYqLi21yraRyv7R27NjR+n6eM3XqVDk7O+uzzz5T+/btqxwTgIZH3j576t/5Y5yf1w4ePKiSkhKbnNuuXTt169bNup+dnS0HBwcNHDiwwvH379+vPn362LRdd911tYoZ5kJxAw3iwguoWSwWmzaLxSJJOnPmjPW/I0aM0Ny5c8uN1bFjx0s+34VfEs/n6upapZjP9+abb+rhhx++aJ+lS5fqvvvuq/KY5+Z87r9S+bjPnDmj0NDQCpf5dejQQS1aVO/Mst69e+vQoUP64IMP9NFHH2n06NG65ZZb9M4775TraxiGTWwVxS5V/N6eex8B1A9yasX5smPHjtq+fXu5vm3atKlxrBWpLDdW5MI4X3jhBS1cuFDJycnq3r273NzcNGXKFOuphudUJbcOGTJEa9as0ZYtW6r1+QOg4ZG3L57XjCrcwPNScVf03bUq46LpoLiBRql3795av369/P391bJlxX+mTk5OKisrq/bYrVq1kr+/v7Zt26bBgwdX6Zg77rhD119//UX7nKu4VyY3N1f/+c9/1KlTJ0nSnj171KJFi4tW3nv37q20tDTrxfEqUt25tG7dWlFRUYqKitKoUaN066236ueff1a7du1s+gUHB2vVqlUqKiqyfkB++umnl4wZQOPTFHPqhXr37q38/Hy1bNmy0msJ9ejRQ9u2bdMDDzxQ5XE/++wzjR071ma/V69e1YrtfDt37lRkZKTuv/9+SWf/AXPgwAEFBQVVe6w77rhDI0aM0L333isHBwfdc889NY4LQOPSHPL2+bp27SpHR0d99tln8vX1lST98ssv+u6776wrNbp3764zZ87ok08+0S233FJujMDAQKWnp9u07du3r8YxwXwobqBRmjRpkl555RWNGTNG06dPV/v27fX9999r7dq1euWVV+Tg4CB/f3/t3btXhw8flru7e7l/nF/MrFmzFBsbq8svv1zDhg3Tb7/9pk8//VSTJ0+usH9dLMVzcXHRuHHjNH/+fBUUFOixxx7T6NGjL3r71fvuu0/z5s1TZGSk5syZo86dOys3N1cbNmzQ9OnT1blz52rNZeHCherYsaN69uypFi1a6O2335a3t7f1V80Ln/uZZ57RuHHjNGvWLB07dkyTJ09WdHR0rT68ADS8pphTL3TLLbcoPDxcI0eO1Ny5c9WtWzf95z//UXp6ukaOHKmwsDA988wzuvnmm9WlSxfdc889Ki0t1QcffKAnn3yy0nHffvtthYWF6YYbbtCbb76pzz//3ObK/NXVtWtXrV+/Xrt371bbtm21YMEC5efn16i4IUl33nmnXn/9dUVHR6tly5YaNWpUjWMD0Hg0h7x9Pnd3d8XExGj69Ony9PSUl5eXEhMTbVYp+/v7a9y4cXrwwQe1ePFiXXvttfrhhx909OhRjR49Wg8//LAWLFigp556SjExMcrOzrZe5L46K+5gXtwtBY1Sp06d9Omnn6qsrExDhw5VSEiIHn/8cXl4eFiT3LRp0+Tg4KDg4GB16NBBubm5VR5/3LhxSk5OVkpKiq655hoNHz5cBw4cqK/pSDr7hfZPf/qTbrvtNkVERCgkJEQpKSkXPeayyy7Tjh075Ovrqz/96U8KCgrSgw8+qD/++MO6kqM6c3F3d9fcuXMVFhamPn366PDhw0pPT6/w9JbLLrtMW7Zs0c8//6w+ffpo1KhRuvnmm/XSSy/V/sUA0KCaYk69kMViUXp6um688UY9+OCDuvrqq3XPPffo8OHD1oLsoEGD9Pbbb2vTpk3q2bOnbrrppnJ3KbnQ7NmztXbtWvXo0UOrVq3Sm2++qeDg4BrHOXPmTPXu3VtDhw7VoEGD5O3trZEjR9Z4PEkaNWqUVq1apejoaG3YsKFWYwFoHJpD3r7QvHnzdOONN+qOO+7QLbfcohtuuEGhoaE2fVJTUzVq1CjFxcUpMDBQDz30kIqKiiRJAQEBeuedd7Rhwwb16NFDqamp1rulODs7N/h80PAsBiciAfVu1qxZeu+996p1EToAgH1ZLBa9++67tS4+AADs469//auWLFmiI0eO2DsUNABOSwEAAAAAmF5KSor69OkjT09Pffrpp5o3b54effRRe4eFBsJpKQAAANX0/PPPy93dvcJt2LBh9g4PAJqlAwcOKDIyUsHBwXr22Wf1xBNPaNasWfYOCw2E01IAAACq6eeff9bPP/9c4WOurq664oorGjgiAACaN4obAAAAAADA1DgtBQAAAAAAmBrFDQAAAAAAYGoUNypgGIYKCgrEGTsAUH/ItQBQ/8i1AJoLihsV+O233+Th4aHffvvN3qEAQJNFrgWA+keuBdBcUNwAAAAAAACmRnEDAAAAAACYGsUNAAAAAABgahQ3AAAAAACAqVHcAAAAAAAApkZxAwAAAAAAmBrFDQAAAAAAYGot7R0AUB2GYaioqMi67+bmJovFYseIAACoHJ9bACpCbgDqHsUNmEpRUZEiIyOt+xs3bpS7u7sdIwIAoHJ8bgGoCLkBqHuclgIAAADUgZSUFAUEBMjFxUWhoaHauXNnpX137dql/v37y9PTU66urgoMDNTChQvL9Vu/fr2Cg4Pl7Oys4OBgvfvuu/U5BQAwLYobAAAAQC2lpaVpypQpSkxMVFZWlgYMGKBhw4YpNze3wv5ubm569NFHtWPHDuXk5GjGjBmaMWOGli1bZu2zZ88eRUVFKTo6Wl999ZWio6M1evRo7d27t6GmBQCmQXEDAAAAqKUFCxYoJiZGEyZMUFBQkJKTk+Xj46PU1NQK+/fq1UtjxozRNddcI39/f91///0aOnSozWqP5ORkDRkyRAkJCQoMDFRCQoJuvvlmJScnN9CsAMA8KG4AAAAAtVBSUqLMzExFRETYtEdERGj37t1VGiMrK0u7d+/WwIEDrW179uwpN+bQoUMvOmZxcbEKCgpsNgBoDihuAAAAALVw/PhxlZWVycvLy6bdy8tL+fn5Fz22c+fOcnZ2VlhYmCZNmqQJEyZYH8vPz6/2mElJSfLw8LBuPj4+NZgRAJgPd0uBXYROX12j4yylJfI4b3/QzLUyWjrVaKzMeWNrdBwAAEBFLryVp2EYl7y9586dO1VYWKjPPvtMTz/9tLp27aoxY8bUeMyEhATFx8db9wsKCihwoMnj1rqQKG4AAAAAtdK+fXs5ODiUW1Fx9OjRcisvLhQQECBJ6t69u3766SfNmjXLWtzw9vau9pjOzs5ydnauyTQA0+LWupAobgAAAAC14uTkpNDQUGVkZOjOO++0tmdkZNj8g+tSDMNQcXGxdT88PFwZGRmaOnWqtW3r1q3q169f3QSOWmM1MtB4UNwAAAAAaik+Pl7R0dEKCwtTeHi4li1bptzcXMXGxko6e7rIjz/+qNWrz/5j+OWXX5avr68CAwMlSbt27dL8+fM1efJk65iPP/64brzxRs2dO1eRkZHauHGjPvroI+3atavhJwgAjRzFDQAAAKCWoqKidOLECc2ZM0d5eXkKCQlRenq6/Pz8JEl5eXnKzc219j9z5owSEhJ06NAhtWzZUl26dNHf/vY3Pfzww9Y+/fr109q1azVjxgzNnDlTXbp0UVpamq6//voGnx8ANHYUNwAAAIA6EBcXp7i4uAofW7lypc3+5MmTbVZpVGbUqFEaNWpUXYQHAE0axQ2YiuHgqJM9xtjsAwAAAACaN4obMBeLpcYXWwIAoCZqesFAqe4uGsgFAwEAuLgW9g4AAAAAAACgNli5AQAAAACwK1bJobZYuQEAAAAAAEyNlRsAAAAA0IC4SD5Q9yhuAAAAAEBD4iL5QJ3jtBQAAAAAAGBqFDcAAAAAAICpUdwAAAAAAACmRnEDAAAAAACYGsUNAAAAAABgahQ3AAAAAACAqXErWAAAAACAaRkOjjrZY4zNPpofu6/cSElJUUBAgFxcXBQaGqqdO3detP8nn3yi0NBQubi46Morr9SSJUtsHt+wYYPCwsLUpk0bubm5qWfPnnr99dfrcwoAAAAAAHuxWGS0dLJusljsHRHswK4rN9LS0jRlyhSlpKSof//+Wrp0qYYNG6ZvvvlGvr6+5fofOnRIt912mx566CG98cYb+vTTTxUXF6cOHTrorrvukiS1a9dOiYmJCgwMlJOTk95//3098MADuvzyyzV06NCGniIAAGjG+DURAICGYdeVGwsWLFBMTIwmTJigoKAgJScny8fHR6mpqRX2X7JkiXx9fZWcnKygoCBNmDBBDz74oObPn2/tM2jQIN15550KCgpSly5d9Pjjj6tHjx7atWtXQ00LAADgLH5NBACgQdituFFSUqLMzExFRETYtEdERGj37t0VHrNnz55y/YcOHap9+/bp9OnT5fobhqFt27Zp//79uvHGGyuNpbi4WAUFBTYbAAAAAAAwB7sVN44fP66ysjJ5eXnZtHt5eSk/P7/CY/Lz8yvsX1paquPHj1vbTp48KXd3dzk5Oen222/Xiy++qCFDhlQaS1JSkjw8PKybj49PLWYGAAAAAAAakt0vKGq5YHmmYRjl2i7V/8L2Vq1aKTs7W1988YX++te/Kj4+Xtu3b690zISEBJ08edK6HTlypAYzAQAAAAAA9mC3C4q2b99eDg4O5VZpHD16tNzqjHO8vb0r7N+yZUt5enpa21q0aKGuXbtKknr27KmcnBwlJSVp0KBBFY7r7OwsZ2fnWswGMB/DMFRUVGTdd3Nzu2hhEQAAAAAaK7ut3HByclJoaKgyMjJs2jMyMtSvX78KjwkPDy/Xf+vWrQoLC5OjY+VXHzcMQ8XFxbUPGmhCioqKFBkZad3OL3QAAIDqS0lJUUBAgFxcXBQaGqqdO3dW2nfDhg0aMmSIOnTooNatWys8PFxbtmyx6bNy5UpZLJZy26lTp+p7KgBgOnY9LSU+Pl6vvvqqVqxYoZycHE2dOlW5ubmKjY2VdPZ0kbFjx1r7x8bG6ocfflB8fLxycnK0YsUKLV++XNOmTbP2SUpKUkZGhv7v//5P3377rRYsWKDVq1fr/vvvb/D5AQAAoHlIS0vTlClTlJiYqKysLA0YMEDDhg1Tbm5uhf137NihIUOGKD09XZmZmRo8eLBGjBihrKwsm36tW7dWXl6ezebi4tIQUwIAU7HbaSmSFBUVpRMnTmjOnDnKy8tTSEiI0tPT5efnJ0nKy8uz+UAICAhQenq6pk6dqpdfflmdOnXS4sWLddddd1n7FBUVKS4uTv/+97/l6uqqwMBAvfHGG4qKimrw+QEAAKB5WLBggWJiYjRhwgRJUnJysrZs2aLU1FQlJSWV65+cnGyz//zzz2vjxo3avHmzevXqZW23WCzy9vau19gBoCmwa3FDkuLi4hQXF1fhYytXrizXNnDgQH355ZeVjvfcc8/pueeeq6vwAAAAgIsqKSlRZmamnn76aZv2iIgI7d69u0pjnDlzRr/99pvatWtn015YWCg/Pz+VlZWpZ8+eevbZZ22KHxcqLi62OR27oKCgGjMBAPOy+91SAACNQ3XOFd+1a5f69+8vT09P6yq5hQsXNmC0ANB4HD9+XGVlZeUuiu/l5VXuYviVeeGFF1RUVKTRo0db2wIDA7Vy5Upt2rRJa9askYuLi/r3768DBw5UOk5SUpI8PDysm4+PT80mBQAmY/eVGwAA+zt3rnhKSor69++vpUuXatiwYfrmm2/k6+tbrr+bm5seffRR9ejRQ25ubtq1a5cefvhhubm5aeLEiXaYAQDY34V3HTMMo0p3IluzZo1mzZqljRs36vLLL7e29+3bV3379rXu9+/fX71799aLL76oxYsXVzhWQkKC4uPjrfsFBQUUOAA0CxQ3AJMLnb66RsdZSkvkcd7+oJlrZbR0qtFYmfPGXroTGrXqniveq1cvm2XR/v7+2rBhg3bu3ElxA0Cz0759ezk4OJRbpXH06NFyqzkulJaWppiYGL399tu65ZZbLtq3RYsW6tOnz0VXbjg7O8vZ2bnqwQNAE8FpKQDQzJ07VzwiIsKmvTrnimdlZWn37t0aOHBgfYQIAI2ak5OTQkNDlZGRYdOekZGhfv36VXrcmjVrNH78eL311lu6/fbbL/k8hmEoOztbHTt2rHXMANDUsHIDAJq52pwr3rlzZx07dkylpaWaNWuWdeVHRbjIHYCmLD4+XtHR0QoLC1N4eLiWLVum3NxcxcbGSjp7usiPP/6o1avPrrhcs2aNxo4dq0WLFqlv377WfOvq6ioPj7NrK2fPnq2+ffvqqquuUkFBgRYvXqzs7Gy9/PLL9pkkADRiFDcAAJJqdq74zp07VVhYqM8++0xPP/20unbtqjFjxlTYNykpSbNnz66zeAGgMYmKitKJEyc0Z84c5eXlKSQkROnp6fLz85Mk5eXlKTc319p/6dKlKi0t1aRJkzRp0iRr+7hx46x3DPz11181ceJE5efny8PDQ7169dKOHTt03XXXNejcAMAMKG4AQDNXm3PFAwICJEndu3fXTz/9pFmzZlVa3OAidwCauri4OMXFxVX42LmCxTnbt2+/5HgLFy7kTlQAUEVccwMAmrmanit+IcMwbE47uZCzs7Nat25tswEAAAB1gZUbAIBqnyv+8ssvy9fXV4GBgZKkXbt2af78+Zo8ebLd5gAAAIDmi+IG0EwZDo462WOMzT6ar+qeK37mzBklJCTo0KFDatmypbp06aK//e1vevjhh+01BQAAADRjFDeA5spikdHSyd5RoBGpzrnikydPZpUGAAAAGg2uuQEAAAAAAEyN4gYAAAAAADA1TksBAAA2DMNQUVGRdd/NzU0Wi8WOEQEAAFwcxQ0AAGCjqKhIkZGR1v2NGzfK3d3djhEBAABcHKelAAAAAAAAU6O4AQAAAAAATI3iBgAAAAAAMDWKGwAAAAAAwNQobgAAAAAAAFOjuAEAAAAAAEyN4gYAAAAAADA1ihsAAAAAAMDUKG4AAAAAAABTo7gBAAAAAABMjeIGAAAAAAAwtZb2DgAAANS90Omra3yspbREHuftD5q5VkZLp2qPkzlvbI1jAAAAqA5WbgAAAAAAAFOjuAEAAADUgZSUFAUEBMjFxUWhoaHauXNnpX03bNigIUOGqEOHDmrdurXCw8O1ZcuWcv3Wr1+v4OBgOTs7Kzg4WO+++259TgEATIviBgAAAFBLaWlpmjJlihITE5WVlaUBAwZo2LBhys3NrbD/jh07NGTIEKWnpyszM1ODBw/WiBEjlJWVZe2zZ88eRUVFKTo6Wl999ZWio6M1evRo7d27t6GmBQCmQXEDQKNmGIYKCwutm2EY9g4JAIByFixYoJiYGE2YMEFBQUFKTk6Wj4+PUlNTK+yfnJysJ598Un369NFVV12l559/XldddZU2b95s02fIkCFKSEhQYGCgEhISdPPNNys5ObmBZgWgrvCdtv5xQVEAjVpRUZEiIyOt+xs3bpS7u7sdIwIAwFZJSYkyMzP19NNP27RHRERo9+7dVRrjzJkz+u2339SuXTtr2549ezR16lSbfkOHDqW4AZgQ32nrH8UNAAAAoBaOHz+usrIyeXl52bR7eXkpPz+/SmO88MILKioq0ujRo61t+fn51R6zuLhYxcXF1v2CgoIqPT8AmB2npQAAAAB1wGKx2OwbhlGurSJr1qzRrFmzlJaWpssvv7xWYyYlJcnDw8O6+fj4VGMGAGBeFDcAAACAWmjfvr0cHBzKrag4evRouZUXF0pLS1NMTIzWrVunW265xeYxb2/vao+ZkJCgkydPWrcjR45UczYAYE4UNwAAAIBacHJyUmhoqDIyMmzaMzIy1K9fv0qPW7NmjcaPH6+33npLt99+e7nHw8PDy425devWi47p7Oys1q1b22wA0BxwzQ0AAACgluLj4xUdHa2wsDCFh4dr2bJlys3NVWxsrKSzKyp+/PFHrV69WtLZwsbYsWO1aNEi9e3b17pCw9XVVR4eHpKkxx9/XDfeeKPmzp2ryMhIbdy4UR999JF27dpln0kCQCNGcQNAvQudvrrGx1pKS+Rx3v6gmWtltHSq9jiZ88bWOAYAaMoMw1BRUZF1383NrUrXiYCtqKgonThxQnPmzFFeXp5CQkKUnp4uPz8/SVJeXp5yc3Ot/ZcuXarS0lJNmjRJkyZNsraPGzdOK1eulCT169dPa9eu1YwZMzRz5kx16dJFaWlpuv766xt0bgBgBhQ3AACADcPBUSd7jLHZR9PF7QnrTlxcnOLi4ip87FzB4pzt27dXacxRo0Zp1KhRtYwMQF1oDD/YSfxoVxmKGwAAwJbFUuMvXAAAAPbABUUBAAAAAICpUdwAAAAAAACmRnEDAAAAAACYGtfcANCocWFDAAAAAJdCcQNA48aFDQEAAABcgt1PS0lJSVFAQIBcXFwUGhqqnTt3XrT/J598otDQULm4uOjKK6/UkiVLbB5/5ZVXNGDAALVt21Zt27bVLbfcos8//7w+pwAAAAAAQKXOrUY+t7Eaue7ZtbiRlpamKVOmKDExUVlZWRowYICGDRum3NzcCvsfOnRIt912mwYMGKCsrCz9+c9/1mOPPab169db+2zfvl1jxozRxx9/rD179sjX11cRERH68ccfG2paAAAAAAD81/9fjXxuk8Vi74iaHLsWNxYsWKCYmBhNmDBBQUFBSk5Olo+Pj1JTUyvsv2TJEvn6+io5OVlBQUGaMGGCHnzwQc2fP9/a580331RcXJx69uypwMBAvfLKKzpz5oy2bdvWUNMCAAAAAAANyG7X3CgpKVFmZqaefvppm/aIiAjt3r27wmP27NmjiIgIm7ahQ4dq+fLlOn36tBwdyy/t+f3333X69Gm1a9eu7oIHAABoZEKnr67RcZbSEnmctz9o5toaX+soc97YGh0HAEBt2a24cfz4cZWVlcnLy8um3cvLS/n5+RUek5+fX2H/0tJSHT9+XB07dix3zNNPP60rrrhCt9xyS6WxFBcXq7i42LpfUFBQnakAAAAAAAA7qlJxY/HixVUe8LHHHqtWAJYLzjUyDKNc26X6V9QuSX//+9+1Zs0abd++XS4uLpWOmZSUpNmzZ1cnbAAAAAAA0EhUqbixcOHCKg1msViqXNxo3769HBwcyq3SOHr0aLnVGed4e3tX2L9ly5by9PS0aZ8/f76ef/55ffTRR+rRo8dFY0lISFB8fLx1v6CgQD4+PlWaBwAAAAAAsK8qFTcOHTpU50/s5OSk0NBQZWRk6M4777S2Z2RkKDIyssJjwsPDtXnzZpu2rVu3KiwszOZ6G/PmzdNzzz2nLVu2KCws7JKxODs7y9nZuYYzAQAAAAAA9mTXu6XEx8fr1Vdf1YoVK5STk6OpU6cqNzdXsbGxks6uqBg79r8XpoqNjdUPP/yg+Ph45eTkaMWKFVq+fLmmTZtm7fP3v/9dM2bM0IoVK+Tv76/8/Hzl5+ersLCwwecHAGaSkpKigIAAubi4KDQ0VDt37qy074YNGzRkyBB16NBBrVu3Vnh4uLZs2dKA0QIAAAD/VaMLiv773//Wpk2blJubq5KSEpvHFixYUOVxoqKidOLECc2ZM0d5eXkKCQlRenq6/Pz8JEl5eXnKzc219g8ICFB6erqmTp2ql19+WZ06ddLixYt11113WfukpKSopKREo0aNsnmuZ555RrNmzarBbAGg6UtLS9OUKVOUkpKi/v37a+nSpRo2bJi++eYb+fr6luu/Y8cODRkyRM8//7zatGmj1157TSNGjNDevXvVq1cvO8wAAAAAzVm1ixvbtm3THXfcoYCAAO3fv18hISE6fPiwDMNQ7969qx1AXFyc4uLiKnxs5cqV5doGDhyoL7/8stLxDh8+XO0YAKC5W7BggWJiYjRhwgRJUnJysrZs2aLU1FQlJSWV65+cnGyz//zzz2vjxo3avHkzxQ0AAAA0uGqflpKQkKAnnnhC//rXv+Ti4qL169fryJEjGjhwoO6+++76iBEAUI9KSkqUmZmpiIgIm/aIiAjt3r27SmOcOXNGv/32m9q1a1dpn+LiYhUUFNhsAAAAQF2odnEjJydH48aNkyS1bNlSf/zxh9zd3TVnzhzNnTu3zgMEANSv48ePq6ysrNydqry8vMrdoaoyL7zwgoqKijR69OhK+yQlJcnDw8O6cVcqAAAA1JVqFzfc3NxUXFwsSerUqZMOHjxofez48eN1FxkAoEFZLBabfcMwyrVVZM2aNZo1a5bS0tJ0+eWXV9ovISFBJ0+etG5HjhypdcwAAACAVINrbvTt21effvqpgoODdfvtt+uJJ57Q119/rQ0bNqhv3771ESMAoB61b99eDg4O5VZpHD16tNxqjgulpaUpJiZGb7/9tm655ZaL9uW22wAAAKgv1V65sWDBAl1//fWSpFmzZmnIkCFKS0uTn5+fli9fXucBAgDql5OTk0JDQ5WRkWHTnpGRoX79+lV63Jo1azR+/Hi99dZbuv322+s7TAD1xHBw1MkeY6yb4eBo75BMqzq31M7Ly9O9996rbt26qUWLFpoyZUq5PitXrpTFYim3nTp1qh5nAQDmVO2VG88++6zuv/9+GYahyy67TCkpKfURFwCgAcXHxys6OlphYWEKDw/XsmXLlJubq9jYWElnTyn58ccftXr1aklnCxtjx47VokWL1LdvX+uqD1dXV3l4eNhtHgBqwGKR0dLJ3lGYXnVvqV1cXKwOHTooMTFRCxcurHTc1q1ba//+/TZtLi4udR4/AJhdtVdunDhxQrfffrs6d+6sJ554QtnZ2fUQFgCgIUVFRSk5OVlz5sxRz549tWPHDqWnp8vPz0/S2V8Yc3Nzrf2XLl2q0tJSTZo0SR07drRujz/+uL2mAAB2df4ttYOCgpScnCwfHx+lpqZW2N/f31+LFi3S2LFjL1oUtlgs8vb2ttkAAOVVu7ixadMm5efn65lnnlFmZqZCQ0MVHBys559/XocPH66HEAEADSEuLk6HDx9WcXGxMjMzdeONN1ofW7lypbZv327d3759uwzDKLetXLmy4QMHADuri1tqV6awsFB+fn7q3Lmzhg8frqysrFqNBwBNVbWLG5LUpk0bTZw4Udu3b9cPP/ygBx54QK+//rq6du1a1/EBAAAAjVpd3FK7IoGBgVq5cqU2bdqkNWvWyMXFRf3799eBAwcqPaa4uFgFBQU2GwA0BzUqbpxz+vRp7du3T3v37tXhw4cveVV9AAAAoKmq6S21K9O3b1/df//9uvbaazVgwACtW7dOV199tV588cVKj0lKSpKHh4d18/HxqfHzA4CZ1Ki48fHHH+uhhx6Sl5eXxo0bp1atWmnz5s06cuRIXccHAAAANGq1uaV2dbRo0UJ9+vS56MqNhIQEnTx50rrx/RxAc1Htu6V07txZJ06c0NChQ7V06VKNGDGCKzYDAACg2Tr/ltp33nmntT0jI0ORkZF19jyGYSg7O1vdu3evtI+zs7OcnZ3r7DkBwCyqXdz4y1/+orvvvltt27atj3gAAAAA06nuLbUlWe86WFhYqGPHjik7O1tOTk4KDg6WJM2ePVt9+/bVVVddpYKCAi1evFjZ2dl6+eWXG3x+ANDYVbu4MXHixPqIAwAAADCtqKgonThxQnPmzFFeXp5CQkIuekttSerVq5f1/zMzM/XWW2/Jz8/PegfCX3/9VRMnTlR+fr48PDzUq1cv7dixQ9ddd12DzQsAzKLaxQ0AAAAA5cXFxSkuLq7Cxyq6VbZhGBcdb+HChVq4cGFdhAYATV6t7pYCAAAAAABgbxQ3AAAAAACAqVHcAAAAAAAApkZxAwAAAAAAmBrFDQAAAAAAYGoUNwAAAAAAgKlR3AAAAAAAAKZGcQMAAAAAAJgaxQ0AAAAAAGBqFDcAAAAAAICpUdwAAAAAAACmRnEDAAAAAACYGsUNAAAAAABgahQ3AAAAAACAqVHcAAAAAAAApkZxAwAAAAAAmBrFDQAAAAAAYGoUNwAAAAAAgKlR3AAAAAAAAKZGcQMAAAAAAJgaxQ0AAAAAAGBqFDcAAAAAAICpUdwAAAAA6kBKSooCAgLk4uKi0NBQ7dy5s9K+eXl5uvfee9WtWze1aNFCU6ZMqbDf+vXrFRwcLGdnZwUHB+vdd9+tp+gBwNwobgAAAAC1lJaWpilTpigxMVFZWVkaMGCAhg0bptzc3Ar7FxcXq0OHDkpMTNS1115bYZ89e/YoKipK0dHR+uqrrxQdHa3Ro0dr79699TkVADAlihsAAABALS1YsEAxMTGaMGGCgoKClJycLB8fH6WmplbY39/fX4sWLdLYsWPl4eFRYZ/k5GQNGTJECQkJCgwMVEJCgm6++WYlJyfX40wAwJwobjQAwzBUWFho3QzDsHdIFTJLnAAAAI1JSUmJMjMzFRERYdMeERGh3bt313jcPXv2lBtz6NChFx2zuLhYBQUFNhsANAct7R1Ac1BUVKTIyEjr/saNG+Xu7m7HiCpmljgBAAAak+PHj6usrExeXl427V5eXsrPz6/xuPn5+dUeMykpSbNnz67xcwKAWbFyAwAAAKgDFovFZt8wjHJt9T1mQkKCTp48ad2OHDlSq+cHALNg5QYAAABQC+3bt5eDg0O5FRVHjx4tt/KiOry9vas9prOzs5ydnWv8nABgVqzcAAAAAGrByclJoaGhysjIsGnPyMhQv379ajxueHh4uTG3bt1aqzEBoKmye3GjOvcDl6RPPvlEoaGhcnFx0ZVXXqklS5bYPP6///u/uuuuu+Tv7y+LxcLVpAEAAFDv4uPj9eqrr2rFihXKycnR1KlTlZubq9jYWElnTxcZO3aszTHZ2dnKzs5WYWGhjh07puzsbH3zzTfWxx9//HFt3bpVc+fO1bfffqu5c+fqo48+0pQpUxpyagBgCnY9LeXc/cBTUlLUv39/LV26VMOGDdM333wjX1/fcv0PHTqk2267TQ899JDeeOMNffrpp4qLi1OHDh101113SZJ+//13XXnllbr77rs1derUOos1dPrqGh9rKS3R+Tf4GjRzrYyWTjUaK3Pe2Is+3hjivFSMAAAATU1UVJROnDihOXPmKC8vTyEhIUpPT5efn58kKS8vT7m5uTbH9OrVy/r/mZmZeuutt+Tn56fDhw9Lkvr166e1a9dqxowZmjlzprp06aK0tDRdf/31DTYvADALuxY3zr8fuHT2Xt5btmxRamqqkpKSyvVfsmSJfH19rasxgoKCtG/fPs2fP99a3OjTp4/69OkjSXr66acbZiIAAABo9uLi4hQXF1fhYytXrizXZhjGJcccNWqURo0aVdvQAKDJs9tpKTW5H3hl9/ret2+fTp8+XW+xAgAAAM2dYRgqLCy0blUpzgBAQ7FbcaMm9wOv7F7fpaWlOn78eI1jKS4uVkFBgc0G1AYf/jCj6lwDKS8vT/fee6+6deumFi1acP43ADQDRUVFioyMtG5FRUX2DgkArOx+QdHq3ru7ov4VtVdHUlKSPDw8rJuPj0+NxwIkPvxhPueugZSYmKisrCwNGDBAw4YNK3d++DnFxcXq0KGDEhMTde211zZwtAAAVI4fmYDmyW7FjZrcD7yye323bNlSnp6eNY4lISFBJ0+etG5Hjhyp8VgAYEbnXwMpKChIycnJ8vHxUWpqaoX9/f39tWjRIo0dO1YeHh4V9gEAwB74kQlonux2QdHz7wd+5513WtszMjIUGRlZ4THh4eHavHmzTdvWrVsVFhYmR0fHGsfi7OwsZ2fnGh9/KYaDo072GGOz3xiZJU4AdevcNZAuvAjzxa6BVBPFxcUqLi627nMKIAAAAOqKXU9Lqe79wGNjY/XDDz8oPj5eOTk5WrFihZYvX65p06ZZ+5SUlFjvGV5SUqIff/xR2dnZ+v777xt8flYWi4yWTtZNtTiFpl6ZJU4Adaom10Cqifo+BZBlyAAAAM2XXW8FW937gQcEBCg9PV1Tp07Vyy+/rE6dOmnx4sXW28BK0n/+8x+be4bPnz9f8+fP18CBA7V9+/YGmxsAmE11r4FUXQkJCYqPj7fuFxQU1GmB49wy5HM2btwod3f3OhsfAJqK0Omra3ScpbRE55+IOGjm2rM/iNVA5ryxl+4EANVg1+KGVP37gQ8cOFBffvllpeP5+/vzax3qRE0/+KW6+/Dngx8NoSbXQKqJ+j4FEAAAAM2X3YsbAAD7qsk1kAAAqE+N4UcmiR+aADOhuAEAUHx8vKKjoxUWFqbw8HAtW7as3DWQfvzxR61e/d8vm9nZ2ZKkwsJCHTt2TNnZ2XJyclJwcLA9pgAAAIBmjOIGAKDa10CSZHN9o8zMTL311lvy8/PT4cOHGzJ0AAAAgOIGAOCs6l4DiesbAQAAoLGguAEAtWQYhoqKiqz7bm5udXqXEQAAAAAXR3EDqAeGg6NO9hhjs4+mi1uQoqoohAEwM77fAGjMKG4A9cFiqfFVuYHmrDFcHb8+r4xPIQyAqZnk+w1FGKB5orgBAAAAoOkwSREGQN2iuAEAAAAAKIfTKWEmFDcA4P+r6SkRdXU6hFS/p0QAAABUB6dTwkxa2DsAAAAAAACA2qC4AQAAAAAATI3iBgAAAAAAMDWuuQEAQDVxfRYAFUlJSdG8efOUl5ena665RsnJyRowYECl/T/55BPFx8frf//3f9WpUyc9+eSTio2NtT6+cuVKPfDAA+WO++OPP+Ti4lIvc0DT0xhusy7xuYX6x8oNAAAAoJbS0tI0ZcoUJSYmKisrSwMGDNCwYcOUm5tbYf9Dhw7ptttu04ABA5SVlaU///nPeuyxx7R+/Xqbfq1bt1ZeXp7NRmEDAMpj5QYA1JLh4KiTPcbY7KPh8T4AsKcFCxYoJiZGEyZMkCQlJydry5YtSk1NVVJSUrn+S5Yska+vr5KTkyVJQUFB2rdvn+bPn6+77rrL2s9iscjb27tB5gAAZsbKDQCoLYtFRksn6ybu/24fvA8A7KSkpESZmZmKiIiwaY+IiNDu3bsrPGbPnj3l+g8dOlT79u3T6dOnrW2FhYXy8/NT586dNXz4cGVlZV00luLiYhUUFNhsANAcUNwAAAAAauH48eMqKyuTl5eXTbuXl5fy8/MrPCY/P7/C/qWlpTp+/LgkKTAwUCtXrtSmTZu0Zs0aubi4qH///jpw4EClsSQlJcnDw8O6+fj41HJ2AGAOnJYCAAAA1AHLBSvGDMMo13ap/ue39+3bV3379rU+3r9/f/Xu3VsvvviiFi9eXOGYCQkJio+Pt+4XFBRQ4ECNcconzITiBgAAAFAL7du3l4ODQ7lVGkePHi23OuMcb2/vCvu3bNlSnp6eFR7TokUL9enT56IrN5ydneXs7FzNGQCV+P+nfAJmwGkpAAA0kHO/gJ3b+AUMaBqcnJwUGhqqjIwMm/aMjAz169evwmPCw8PL9d+6davCwsLk6FhxbjAMQ9nZ2erYsWPdBA4ATQjFDQAAGgoXPQWarPj4eL366qtasWKFcnJyNHXqVOXm5io2NlbS2dNFxo4da+0fGxurH374QfHx8crJydGKFSu0fPlyTZs2zdpn9uzZ2rJli/7v//5P2dnZiomJUXZ2tnVMAMB/cVoKAAAAUEtRUVE6ceKE5syZo7y8PIWEhCg9PV1+fn6SpLy8POXm5lr7BwQEKD09XVOnTtXLL7+sTp06afHixTa3gf311181ceJE5efny8PDQ7169dKOHTt03XXXNfj8ADQPhmGoqKjIuu/m5nbRawc1JhQ3AACA6Zj5yxearri4OMXFxVX42MqVK8u1DRw4UF9++WWl4y1cuFALFy6sq/AA4JKKiooUGRlp3d+4caPc3d3tGFHVUdwAAACmY+YvXwAAoO5xzQ0AAAAAAGBqrNwAAAAAAKCJCJ2+usbHWkpL5HHe/qCZa2t0O+DMeWMv3amOsXIDAAAAAACYGis3AACA3dT016W6+mVJss+vSwAAoG6xcgMAAAAAAJgaKzcAAAAAAIAMB0ed7DHGZt8sKG4AAAAAAADJYqnxaZ72xmkpAAAAAADA1ChuAAAAAAAAU+O0FAAAYDpmPicYAADUPYobAADAfEx8TjAAAKh7nJYCAAAAAABMjeIGAAAAAAAwNYobAAAAAADA1ChuAAAAAAAAU+OCogAAAGj0DMNQUVGRdd/NzU0Wi8WOEQEAGhOKGwAAAGj0ioqKFBkZad3fuHGj3N3d7RgRAKAx4bQUAAAAAABgaqzcAAAAQIMInb66xsdaSkvkcd7+oJlrZbR0qtFYmfPG1jgOAEDjxMoNAAAAAABganYvbqSkpCggIEAuLi4KDQ3Vzp07L9r/k08+UWhoqFxcXHTllVdqyZIl5fqsX79ewcHBcnZ2VnBwsN599936Ch8Amoz6yMcAUFcMB0ed7DHGuhkOjvYOqRy+1wKA/di1uJGWlqYpU6YoMTFRWVlZGjBggIYNG6bc3NwK+x86dEi33XabBgwYoKysLP35z3/WY489pvXr11v77NmzR1FRUYqOjtZXX32l6OhojR49Wnv37m2oaQGA6dRHPgaAOmWxyGjpZN3UyO6UwvdaALAvuxY3FixYoJiYGE2YMEFBQUFKTk6Wj4+PUlNTK+y/ZMkS+fr6Kjk5WUFBQZowYYIefPBBzZ8/39onOTlZQ4YMUUJCggIDA5WQkKCbb75ZycnJDTQrADCf+sjHANCc8L0WAOzLbsWNkpISZWZmKiIiwqY9IiJCu3fvrvCYPXv2lOs/dOhQ7du3T6dPn75on8rGBIDmrr7yMQA0F3yvBQD7s9vdUo4fP66ysjJ5eXnZtHt5eSk/P7/CY/Lz8yvsX1paquPHj6tjx46V9qlsTEkqLi5WcXGxdf/kyZOSpIKCAmtbWfEfVZtYPTs/poo0hjgvFaNkjjjNEKNkjjjNEKPUeONs1aqVLPW4/Lq+8vGFzJJrzfy3cj4zxCiZI04zxCiZI87GEKNU97mW77XVZ+a/lfM1hjjNEKNkjjjNEKNkjjjt8Z3W7reCvXByhmFcdMIV9b+wvbpjJiUlafbs2eXafXx8Kg/cTjxejLV3CJdkhhglc8Rphhglc8RphhiliuM8efKkWrduXe/PXR/5+HxmybVm/ltpbMwQo2SOOM0Qo2TuOOsi1/K9turM/LfS2JghRskccZohRskccdrjO63dihvt27eXg4NDucrz0aNHy1Woz/H29q6wf8uWLeXp6XnRPpWNKUkJCQmKj4+37p85c0Y///yzPD0966yyVFBQIB8fHx05cqRB/pFSU2aI0wwxSuaI0wwxSuaIsz5jbNWqVZ2Od6H6yscXIteeZYYYJXPEaYYYJXPEaYYYpcaba5vT91r+VuqOGWKUiLMumSFGqf7irO/vtHYrbjg5OSk0NFQZGRm68847re0ZGRmKjIys8Jjw8HBt3rzZpm3r1q0KCwuTo6OjtU9GRoamTp1q06dfv36VxuLs7CxnZ2ebtjZt2lR3SlXSunXrRv2HfI4Z4jRDjJI54jRDjJI54jRDjBeqr3x8IXKtLTPEKJkjTjPEKJkjTjPEKDW+OJvj99rG9h5UxgxxmiFGiTjrkhlilMwT5zl2vVtKfHy8Xn31Va1YsUI5OTmaOnWqcnNzFRt7dglLQkKCxo4da+0fGxurH374QfHx8crJydGKFSu0fPlyTZs2zdrn8ccf19atWzV37lx9++23mjt3rj766CNNmTKloacHAKZRH/kYAJoTvtcCgH3Z9ZobUVFROnHihObMmaO8vDyFhIQoPT1dfn5+kqS8vDybe4MHBAQoPT1dU6dO1csvv6xOnTpp8eLFuuuuu6x9+vXrp7Vr12rGjBmaOXOmunTporS0NF1//fUNPj8AMIv6yMcA0JzwvRYA7MxAgzh16pTxzDPPGKdOnbJ3KBdlhjjNEKNhmCNOM8RoGOaI0wwxNgdmeB/MEKNhmCNOM8RoGOaI0wwxGoZ54mzKzPIemCFOM8RoGMRZl8wQo2GYJ84LWQzj/1+WGQAAAAAAwITses0NAAAAAACA2qK4AQAAAAAATI3iRjN2+PBhWSwWZWdn2zsUAGiyyLUAUP/ItQAobtSB3bt3y8HBQbfeeqtN+6WSbFlZmZKSkhQYGChXV1e1a9dOffv21WuvvVYncY0fP14Wi8W6eXp66tZbb9U///lPSZKPj4/1at72Mn78eI0cOdKmrbLXU7L/a1qR8+dw4XwufA/ObRXNraFcKt4L34/6jOPc6+Ho6Kgrr7xS06ZNU1FRUYM8/6UMGjSowlvtvffee7JYLJKkWbNmqWfPnuX6/Prrr7JYLNq+fbuk//7dVrR99tln9TiLpoVcW7sYzZxrybO1i4VcS66tqsaaZyVyLbm2PHJt1TSnPGvXW8E2FStWrNDkyZP16quvKjc3V76+vlU6btasWVq2bJleeuklhYWFqaCgQPv27dMvv/xSZ7Hdeuut1gSYn5+vGTNmaPjw4crNzZWDg4O8vb3r7LnqSk1fT6lhXtPqOv89OMfZ2dlO0TQu516b06dPa+fOnZowYYKKioqUmppq79DqxUcffaRrrrnGps3T09NO0ZgPubZuNaVcS569OHItubaqGnOelci15NrGrTnl2saaZ1m5UUtFRUVat26dHnnkEQ0fPlwrV66s8rGbN29WXFyc7r77bgUEBOjaa69VTEyM4uPj6yw+Z2dneXt7y9vbWz179tRTTz2lI0eO6NixY+WqxWVlZYqJiVFAQIBcXV3VrVs3LVq0yGa87du367rrrpObm5vatGmj/v3764cffqizeGvzekoN85pW1/nvwbmtbdu2dounMTn32vj4+Ojee+/Vfffdp/fee0/FxcV67LHHdPnll8vFxUU33HCDvvjiC+tx27dvl8Vi0T/+8Q9de+21cnFx0fXXX6+vv/7ajrO5NE9Pz3J/C46OjvYOyxTIteTaiyHPXhy5llxbFY09z0rkWnJt49accm1jzbMUN2opLS1N3bp1U7du3XT//ffrtddeU1Xvruvt7a3/+Z//0bFjx+o5yrMKCwv15ptvqmvXrhVW1s6cOaPOnTtr3bp1+uabb/SXv/xFf/7zn7Vu3TpJUmlpqUaOHKmBAwfqn//8p/bs2aOJEydalzPVhdq8nlLDv6aoW66urjp9+rSefPJJrV+/XqtWrdKXX36prl27aujQofr5559t+k+fPl3z58/XF198ocsvv1x33HGHTp8+bafoUZ/IteRa1B1yLSpipjwrkWvR+JFrGx7FjVpavny57r//fklnlyIVFhZq27ZtVTp2wYIFOnbsmLy9vdWjRw/Fxsbqgw8+qNP43n//fbm7u8vd3V2tWrXSpk2blJaWphYtyr/1jo6Omj17tvr06aOAgADdd999Gj9+vPVDoKCgQCdPntTw4cPVpUsXBQUFady4cdVaXncptXk9pYZ5Tavr/Pfg3Pbss8/aNabG6PPPP9dbb72lwYMHKzU1VfPmzdOwYcMUHBysV155Ra6urlq+fLnNMc8884yGDBmi7t27a9WqVfrpp5/07rvv2mkGl9avX79yfwtlZWX2DssUyLXk2oshz1YduRaVaex5ViLXkmvNo6nn2saaZylu1ML+/fv1+eef65577pEktWzZUlFRUVqxYkWVjg8ODta//vUvffbZZ3rggQf0008/acSIEZowYUKdxTh48GBlZ2crOztbe/fuVUREhIYNG1bpkrslS5YoLCxMHTp0kLu7u1555RXl5uZKktq1a6fx48dr6NChGjFihBYtWqS8vLw6i7W2r6fUMK9pdZ3/HpzbJk2aZLd4GpNzH5IuLi4KDw/XjTfeqMmTJ+v06dPq37+/tZ+jo6Ouu+465eTk2BwfHh5u/f927dqpW7du5fo0JmlpaeX+FhwcHOwdVqNHriXXXgp59uLIteTaSzFDnpXIteTaxq055drGmme5oGgtLF++XKWlpbriiiusbYZhyNHRscoX+mnRooX69OmjPn36aOrUqXrjjTcUHR2txMREBQQE1DpGNzc3de3a1bofGhoqDw8PvfLKK+US47p16zR16lS98MILCg8PV6tWrTRv3jzt3bvX2ue1117TY489pg8//FBpaWmaMWOGMjIy1Ldv31rHeqnXs6rn9NX3a1pdF74H+K9z1WxHR0d16tRJjo6O+uqrrySp3LJQwzCqtFS0rpaTtm7dWidPnizX/uuvv6p169aX7CNJHh4eNu0+Pj78LdQAuZZceynk2Ysj1+JSzJBnJXItubZxa6y5tjnlWVZu1FBpaalWr16tF154waZi9dVXX8nPz09vvvlmjcYNDg6WpHq7bZDFYlGLFi30xx9/lHts586d6tevn+Li4tSrVy917dpVBw8eLNevV69eSkhI0O7duxUSEqK33nqr1nHV1+sp1f9ripo79yHp5+dnvQhR165d5eTkpF27dln7nT59Wvv27VNQUJDN8effcuqXX37Rd999p8DAwDqJLTAwUPv27SvX/sUXX6hbt27WPv/+97+Vn59frk+LFi0aZdI3G3ItuRa1R67FxZg1z0rkWjQujTXXNqc8y8qNGnr//ff1yy+/KCYmplwla9SoUVq+fLmGDx8u6eyytAsFBwfr3nvvVf/+/dWvXz95e3vr0KFDSkhI0NVXX11nXxqKi4utf6S//PKLXnrpJRUWFmrEiBHl+nbt2lWrV6/Wli1bFBAQoNdff11ffPGFtSp86NAhLVu2THfccYc6deqk/fv367vvvtPYsWNrHWdVXs9HH33U2mbP17S6zn8PzmnZsqXat29vl3gaOzc3Nz3yyCOaPn262rVrJ19fX/3973/X77//rpiYGJu+c+bMkaenp7y8vJSYmKj27dvX2T3N4+Li9NJLL2nSpEmaOHGiXF1dlZGRoeXLl+v111+XJEVERCgoKEj33HOP/vrXv6pTp0765z//qWnTpik2NlatWrWyGfPEiRPl/hbatGkjFxeXOom5KSLXkmurgjxbfeRanGOWPCuRa8m15tMYcm2zyrMGamT48OHGbbfdVuFjmZmZhiTrfyvaDh06ZCxbtswYPHiw0aFDB8PJycnw9fU1xo8fbxw+fLhOYhw3bpzNc7Zq1cro06eP8c477xiGYRiHDh0yJBlZWVmGYRjGqVOnjPHjxxseHh5GmzZtjEceecR4+umnjWuvvdYwDMPIz883Ro4caXTs2NFwcnIy/Pz8jL/85S9GWVlZrWKMjIys8ut5LmZ7vaYXm8OF/39uv6JYu3XrVm/xXMql4j1/v6HiuNAff/xhTJ482Wjfvr3h7Oxs9O/f3/j888+tj3/88ceGJGPz5s3GNddcYzg5ORl9+vQxsrOz6zTGffv2GUOHDjUuv/xyo3Xr1kZYWJixZs0amz55eXnGAw88YPj5+Rmurq5GYGCgMWfOHOPUqVPWPhf7u71wPNgi15Jrz4//wv8/t0+erVosFyLXwjDMkWcNg1xLri2PXFt1zSXPWgyjGvcjAoBGYPv27Ro8eLB++eUXtWnTxt7hAECTRK4FgPpHrq07XHMDAAAAAACYGsUNAAAAAABgapyWAgAAAAAATI2VGwAAAAAAwNQobgAAAAAAAFOjuAEAAAAAAEyN4gYAAAAAADA1ihsAAAAAAMDUKG6g2Tp8+LAsFouys7MbzXMNGjRIU6ZMqfd4AKChkGsBoH6RZ4GzKG4ADcDHx0d5eXkKCQmRJG3fvl0Wi0W//vqrfQMDgCaEXAsA9Ys8i8aspb0DAJq6kpISOTk5ydvb296hAECTRa4FgPpFnkVjx8oNNGkffvihbrjhBrVp00aenp4aPny4Dh48WGn/TZs26aqrrpKrq6sGDx6sVatWlatGr1+/Xtdcc42cnZ3l7++vF154wWYMf39/Pffccxo/frw8PDz00EMP2SzhO3z4sAYPHixJatu2rSwWi8aPH289/syZM3ryySfVrl07eXt7a9asWTbjWywWLV26VMOHD9dll12moKAg7dmzR99//70GDRokNzc3hYeHX3SeAFCXyLUAUL/Is0AVGEAT9s477xjr1683vvvuOyMrK8sYMWKE0b17d6OsrMw4dOiQIcnIysoyDMMwDh06ZDg6OhrTpk0zvv32W2PNmjXGFVdcYUgyfvnlF8MwDGPfvn1GixYtjDlz5hj79+83XnvtNcPV1dV47bXXrM/p5+dntG7d2pg3b55x4MAB48CBAzbPVVpaaqxfv96QZOzfv9/Iy8szfv31V8MwDGPgwIFG69atjVmzZhnfffedsWrVKsNisRhbt261ji/JuOKKK4y0tDRj//79xsiRIw1/f3/jpptuMj788EPjm2++Mfr27WvceuutDfUyA2jmyLUAUL/Is8ClUdxAs3L06FFDkvH111+X+yB46qmnjJCQEJv+iYmJNh8E9957rzFkyBCbPtOnTzeCg4Ot+35+fsbIkSNt+lz4XB9//LHNuOcMHDjQuOGGG2za+vTpYzz11FPWfUnGjBkzrPt79uwxJBnLly+3tq1Zs8ZwcXG59AsCAPWAXAsA9Ys8C5THaSlo0g4ePKh7771XV155pVq3bq2AgABJUm5ubrm++/fvV58+fWzarrvuOpv9nJwc9e/f36atf//+OnDggMrKyqxtYWFhNY65R48eNvsdO3bU0aNHK+3j5eUlSerevbtN26lTp1RQUFDjOACgqsi15FoA9Ys8S57FpXFBUTRpI0aMkI+Pj1555RV16tRJZ86cUUhIiEpKSsr1NQxDFoulXFt1+0iSm5tbjWN2dHS02bdYLDpz5kylfc7FU1HbhccBQH0g15JrAdQv8ix5FpdGcQNN1okTJ5STk6OlS5dqwIABkqRdu3ZV2j8wMFDp6ek2bfv27bPZDw4OLjfG7t27dfXVV8vBwaHKsTk5OUmSTWUcAMyIXAsA9Ys8C1QNp6WgyWrbtq08PT21bNkyff/99/qf//kfxcfHV9r/4Ycf1rfffqunnnpK3333ndatW6eVK1dK+m/V+IknntC2bdv07LPP6rvvvtOqVav00ksvadq0adWKzc/PTxaLRe+//76OHTumwsLCGs8TAOyJXAsA9Ys8C1QNxQ00WS1atNDatWuVmZmpkJAQTZ06VfPmzau0f0BAgN555x1t2LBBPXr0UGpqqhITEyVJzs7OkqTevXtr3bp1Wrt2rUJCQvSXv/xFc+bMsbntVVVcccUVmj17tp5++ml5eXnp0UcfrfE8AcCeyLUAUL/Is0DVWIyKTq4CIEn661//qiVLlujIkSP2DgUAmixyLQDUL/IsmgOuuQGcJyUlRX369JGnp6c+/fRTzZs3jwo0ANQxci0A1C/yLJojihvAeQ4cOKDnnntOP//8s3x9ffXEE08oISHB3mEBQJNCrgWA+kWeRXPEaSkAAAAAAMDUuKAoAAAAAAAwNYobAAAAAADA1ChuAAAAAAAAU6O4AQAAAAAATI3iBgAAAAAAMDWKGwAAAAAAwNQobgAAAAAAAFOjuAEAAAAAAEyN4gYAAAAAADC1/wfJQJzr5bRNlAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1091.11x300 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = sns.catplot(x = \"algorithm\", y = \"val\", data = pltData, kind=\"bar\", col = \"metric\", aspect=1.2, height=3, sharey=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e32b381",
   "metadata": {
    "papermill": {
     "duration": 0.009048,
     "end_time": "2024-12-07T20:28:00.639554",
     "exception": false,
     "start_time": "2024-12-07T20:28:00.630506",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Prediction RMSE\n",
    "\n",
    "We will also look at the prediction RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "fb282b1c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:28:00.645088Z",
     "iopub.status.busy": "2024-12-07T20:28:00.644945Z",
     "iopub.status.idle": "2024-12-07T20:28:01.281302Z",
     "shell.execute_reply": "2024-12-07T20:28:01.280969Z"
    },
    "papermill": {
     "duration": 0.640179,
     "end_time": "2024-12-07T20:28:01.282144",
     "exception": false,
     "start_time": "2024-12-07T20:28:00.641965",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "preds = preds.rename(columns={'score': 'prediction'})\n",
    "user_rmse = preds.groupby(['algorithm', 'user']).apply(lambda df: rmse(df['prediction'], df['rating']))\n",
    "user_rmse = user_rmse.reset_index(name='RMSE')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "73116a6d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:28:01.294901Z",
     "iopub.status.busy": "2024-12-07T20:28:01.294800Z",
     "iopub.status.idle": "2024-12-07T20:28:01.393139Z",
     "shell.execute_reply": "2024-12-07T20:28:01.392852Z"
    },
    "papermill": {
     "duration": 0.109008,
     "end_time": "2024-12-07T20:28:01.393690",
     "exception": false,
     "start_time": "2024-12-07T20:28:01.284682",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x7f00b84867d0>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAHpCAYAAACFlZVCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAk2klEQVR4nO3df1RUZeLH8c8AAmaCiQZoiJipGFktaIKZaYqamu7ZVlsrxGSLxXTFLEVaU7M4u64etzVRUzHLH5C62y9WY2srDduUg1urplYaahCJCWoKX+B+//A42zhYogMDD+/XOXPqXp4781xv9ube+WWzLMsSAAAwloe7JwAAAOoWsQcAwHDEHgAAwxF7AAAMR+wBADAcsQcAwHDEHgAAwzW52FuWpbKyMvHxAgCApqLJxf7UqVPy9/fXqVOn3D0VAADqRZOLPQAATQ2xBwDAcMQeAADDEXsAAAxH7AEAMByxBwDAcMQeAADDEXsAAAxH7AEAMByxBwDAcMQeAADDEXsAAAxH7AEAMByxBwDAcMQeAADDEXsAAAxH7AEAMJyXuycANCWWZenMmTP25RYtWshms7lxRgCaAmIP1KMzZ85o5MiR9uXXX39d1157rRtnBKApIPYAgEaLq2WXh9gDABotrpZdHl6gBwCA4Yg9AACGI/YAABiO2AMAYDhiDwCA4Yg9AACGI/YAABiO99nDGJFPrnH3FH6WrbJC/j9avvsPG2R5ebttPj8nb36cu6cAwAU4swcAwHCc2RuAj4sEUBe4WlY33HHFjNgbgI+LBFyLX6BhGmIPABfhF2iYhufsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMPxPnugHlmezVTa4zcOywBQ14j9z+DjIl2vSX+5is3WoI8N0NjwC/TlIfYAgMaLX6AvC7EHUO8a+hWzxna1TGriV8zws3iBHgAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOH4UB0D8HGRAICfQuxNwMdFAgB+ArEHgItwtQymIfYAcDGulsEwvEAPAADDEXsAAAxH7AEAMByxBwDAcMQeAADDEXsAAAxH7AEAMByxBwDAcG6P/ZIlSxQWFiZfX19FRkZq27ZtPzl+7dq1uvXWW3XNNdcoODhY48ePV0lJST3NFgCAxsetsc/MzNSUKVOUmpqq/Px89e3bV0OHDlVBQUGN47dv3664uDhNmDBBe/bs0WuvvaadO3cqISGhnmcOAEDj4dbYL1y4UBMmTFBCQoLCw8O1aNEihYSEKD09vcbxH3/8sTp27KjJkycrLCxMd955px577DHt2rXrko9RXl6usrIyhxsAAE2J22JfUVGhvLw8xcbGOqyPjY1Vbm5ujdvExMTo6NGjys7OlmVZ+vbbb7Vx40YNGzbsko+TlpYmf39/+y0kJMSl+wEAQEPnttgfP35cVVVVCgwMdFgfGBiooqKiGreJiYnR2rVrNWbMGHl7eysoKEitWrXSX//610s+TkpKikpLS+23I0eOuHQ/AABo6Nz+Aj2bzeawbFmW07oL9u7dq8mTJ2vWrFnKy8vTli1bdOjQISUmJl7y/n18fOTn5+dwAwCgKXHbV9y2adNGnp6eTmfxxcXFTmf7F6SlpalPnz568sknJUk9evRQixYt1LdvX82bN0/BwcF1Pm8AABobt53Ze3t7KzIyUjk5OQ7rc3JyFBMTU+M2P/zwgzw8HKfs6ekp6fwVAQAA4Mytl/GnTp2qFStWaNWqVdq3b5+Sk5NVUFBgvyyfkpKiuLg4+/gRI0Zo8+bNSk9P11dffaWPPvpIkydPVq9evdSuXTt37QYAAA2a2y7jS9KYMWNUUlKiuXPnqrCwUBEREcrOzlZoaKgkqbCw0OE99/Hx8Tp16pQWL16sJ554Qq1atdKAAQP0xz/+0V27AABAg+fW2EtSUlKSkpKSavzZ6tWrndZNmjRJkyZNquNZAQBgDre/Gh8AANQtYg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOGIPQAAhiP2AAAYjtgDAGA4Yg8AgOHcHvslS5YoLCxMvr6+ioyM1LZt235yfHl5uVJTUxUaGiofHx/deOONWrVqVT3NFgCAxsfLnQ+emZmpKVOmaMmSJerTp4+WLVumoUOHau/everQoUON24wePVrffvutVq5cqc6dO6u4uFiVlZX1PHMAABoPt8Z+4cKFmjBhghISEiRJixYt0tatW5Wenq60tDSn8Vu2bNEHH3ygr776Sq1bt5YkdezYsT6nDABAo+O2y/gVFRXKy8tTbGysw/rY2Fjl5ubWuM0bb7yhqKgo/elPf1L79u3VpUsXTZs2TWfPnr3k45SXl6usrMzhBgBAU+K2M/vjx4+rqqpKgYGBDusDAwNVVFRU4zZfffWVtm/fLl9fX/3tb3/T8ePHlZSUpBMnTlzyefu0tDTNmTPH5fMHAKCxcPsL9Gw2m8OyZVlO6y6orq6WzWbT2rVr1atXL917771auHChVq9efcmz+5SUFJWWltpvR44ccfk+AADQkLntzL5Nmzby9PR0OosvLi52Otu/IDg4WO3bt5e/v799XXh4uCzL0tGjR3XTTTc5bePj4yMfHx/XTh4AgEbEbWf23t7eioyMVE5OjsP6nJwcxcTE1LhNnz599M033+j06dP2dQcOHJCHh4duuOGGOp0vAACNlVsv40+dOlUrVqzQqlWrtG/fPiUnJ6ugoECJiYmSzl+Cj4uLs48fO3asAgICNH78eO3du1cffvihnnzyST3yyCNq3ry5u3YDAIAGza1vvRszZoxKSko0d+5cFRYWKiIiQtnZ2QoNDZUkFRYWqqCgwD7+2muvVU5OjiZNmqSoqCgFBARo9OjRmjdvnrt2AQCABs+tsZekpKQkJSUl1fiz1atXO63r1q2b06V/AABwaW5/NT4AAKhbxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwtYr9J598oqqqKvuyZVkOPy8vL1dWVpZrZgYAAFyiVrGPjo5WSUmJfdnf319fffWVffnkyZP6zW9+47rZAQCAq1ar2F98Jn/x8qXWAQAA93H5c/Y2m83VdwkAAK4CL9ADAMBwXrXdYO/evSoqKpJ0/pL9559/rtOnT0uSjh8/7trZAQCAq1br2N9zzz0Oz8sPHz5c0vnL95ZlcRkfAIAGplaxP3ToUF3NAwAA1JFaxT40NLSu5gEAAOpIrV6gd+LECR09etRh3Z49ezR+/HiNHj1a69atc+nkAADA1atV7CdOnKiFCxfal4uLi9W3b1/t3LlT5eXlio+P1yuvvOLySQIAgCtXq9h//PHHuu++++zLa9asUevWrbV79269/vrrev755/Xiiy+6fJIAAODK1Sr2RUVFCgsLsy+/9957+uUvfykvr/NP/d933306ePCga2cIAACuSq1i7+fnp5MnT9qXP/nkE/Xu3du+bLPZVF5e7rLJAQCAq1er2Pfq1UsvvPCCqqurtXHjRp06dUoDBgyw//zAgQMKCQlx+SQBAMCVq9Vb75599lkNHDhQr776qiorKzVz5kxdd9119p9v2LBB/fr1c/kkAQDAlatV7G+77Tbt27dPubm5CgoK0h133OHw8wceeEDdu3d36QQBAMDVqfXH5bZt21YjR46s8WfDhg276gkBAADXqlXs16xZc1nj4uLirmgyAADA9WoV+/j4eF177bXy8vJy+DKcH7PZbMQeAIAGpFaxDw8P17fffquHHnpIjzzyiHr06FFX8wIAAC5Sq7fe7dmzR2+//bbOnj2ru+66S1FRUUpPT1dZWVldzQ8AAFylWsVeku644w4tW7ZMhYWFmjx5srKyshQcHKwHH3yQD9QBAKABqnXsL2jevLni4uI0Z84c9erVSxs2bNAPP/zgyrkBAAAXuKLYHzt2TM8//7xuuukmPfDAA+rZs6f27Nnj8AE7AACgYajVC/SysrKUkZGhDz74QIMHD9aCBQs0bNgweXp61tX8AADAVapV7B944AF16NBBycnJCgwM1OHDh2v8StvJkye7bIIAAODq1Cr2HTp0kM1m07p16y45xmazEXsAABqQWsX+8OHDPzvm2LFjVzoXAABQB6741fgXKyoq0uTJk9W5c2dX3SUAAHCBWsX+5MmTevDBB9W2bVu1a9fO/t32s2bNUqdOnbRjxw6tWrWqruYKAACuQK0u48+cOVMffvihxo0bpy1btig5OVlbtmzRuXPn9I9//IPvsgcAoAGqVezffvttZWRkaODAgUpKSlLnzp3VpUsXLVq0qI6mBwAArlatLuN/88036t69uySpU6dO8vX1VUJCQp1MDAAAuEatYl9dXa1mzZrZlz09PdWiRQuXTwoAALhOrS7jW5al+Ph4+fj4SJLOnTunxMREp+Bv3rzZdTMEAABXpVaxHzdunMPyQw895NLJAAAA16tV7DMyMupqHgAAoI647EN1AABAw0TsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMO5PfZLlixRWFiYfH19FRkZqW3btl3Wdh999JG8vLx022231e0EAQBo5Nwa+8zMTE2ZMkWpqanKz89X3759NXToUBUUFPzkdqWlpYqLi9M999xTTzMFAKDxcmvsFy5cqAkTJighIUHh4eFatGiRQkJClJ6e/pPbPfbYYxo7dqyio6PraaYAADRebot9RUWF8vLyFBsb67A+NjZWubm5l9wuIyNDX375pZ555pnLepzy8nKVlZU53AAAaErcFvvjx4+rqqpKgYGBDusDAwNVVFRU4zYHDx7UjBkztHbtWnl5Xd53+KSlpcnf399+CwkJueq5AwDQmLj9BXo2m81h2bIsp3WSVFVVpbFjx2rOnDnq0qXLZd9/SkqKSktL7bcjR45c9ZwBAGhMavUVt67Upk0beXp6Op3FFxcXO53tS9KpU6e0a9cu5efn6/HHH5ckVVdXy7IseXl56Z133tGAAQOctvPx8ZGPj0/d7AQAAI2A287svb29FRkZqZycHIf1OTk5iomJcRrv5+enzz77TLt377bfEhMT1bVrV+3evVt33HFHfU0dAIBGxW1n9pI0depUPfzww4qKilJ0dLSWL1+ugoICJSYmSjp/Cf7YsWNas2aNPDw8FBER4bD99ddfL19fX6f1AADgf9wa+zFjxqikpERz585VYWGhIiIilJ2drdDQUElSYWHhz77nHgAA/DS3xl6SkpKSlJSUVOPPVq9e/ZPbzp49W7Nnz3b9pAAAMIjbX40PAADqFrEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMO5PfZLlixRWFiYfH19FRkZqW3btl1y7ObNmzVo0CC1bdtWfn5+io6O1tatW+txtgAAND5ujX1mZqamTJmi1NRU5efnq2/fvho6dKgKCgpqHP/hhx9q0KBBys7OVl5envr3768RI0YoPz+/nmcOAEDj4eXOB1+4cKEmTJighIQESdKiRYu0detWpaenKy0tzWn8okWLHJaff/55vf7663rzzTd1++231/gY5eXlKi8vty+XlZW5bgcAAGgE3HZmX1FRoby8PMXGxjqsj42NVW5u7mXdR3V1tU6dOqXWrVtfckxaWpr8/f3tt5CQkKuaNwAAjY3bYn/8+HFVVVUpMDDQYX1gYKCKioou6z4WLFigM2fOaPTo0Zcck5KSotLSUvvtyJEjVzVvAAAaG7dexpckm83msGxZltO6mqxfv16zZ8/W66+/ruuvv/6S43x8fOTj43PV8wQAoLFyW+zbtGkjT09Pp7P44uJip7P9i2VmZmrChAl67bXXNHDgwLqcJgAAjZ7bLuN7e3srMjJSOTk5DutzcnIUExNzye3Wr1+v+Ph4rVu3TsOGDavraQIA0Oi59TL+1KlT9fDDDysqKkrR0dFavny5CgoKlJiYKOn88+3Hjh3TmjVrJJ0PfVxcnP7yl7+od+/e9qsCzZs3l7+/v9v2AwCAhsytsR8zZoxKSko0d+5cFRYWKiIiQtnZ2QoNDZUkFRYWOrznftmyZaqsrNTEiRM1ceJE+/px48Zp9erV9T19AAAaBbe/QC8pKUlJSUk1/uzigL///vt1PyEAAAzj9o/LBQAAdYvYAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOLfHfsmSJQoLC5Ovr68iIyO1bdu2nxz/wQcfKDIyUr6+vurUqZOWLl1aTzMFAKBxcmvsMzMzNWXKFKWmpio/P199+/bV0KFDVVBQUOP4Q4cO6d5771Xfvn2Vn5+vmTNnavLkydq0aVM9zxwAgMbDrbFfuHChJkyYoISEBIWHh2vRokUKCQlRenp6jeOXLl2qDh06aNGiRQoPD1dCQoIeeeQR/fnPf67nmQMA0Hh4ueuBKyoqlJeXpxkzZjisj42NVW5ubo3b7NixQ7GxsQ7rBg8erJUrV+r//u//1KxZM6dtysvLVV5ebl8uLS2VJJWVlV3WPKvKz17WOFy+y/2zry2OletxrBqPujhWHKe6UZtj1bJlS9lstqt+TLfF/vjx46qqqlJgYKDD+sDAQBUVFdW4TVFRUY3jKysrdfz4cQUHBzttk5aWpjlz5jitDwkJuYrZ42r4/zXR3VPAZeJYNR4cq8ajNseqtLRUfn5+V/2Ybov9BRf/xmJZ1k/+FlPT+JrWX5CSkqKpU6fal6urq3XixAkFBAS45LelhqKsrEwhISE6cuSIS/7DQN3hWDUOHKfGw+Rj1bJlS5fcj9ti36ZNG3l6ejqdxRcXFzudvV8QFBRU43gvLy8FBATUuI2Pj498fHwc1rVq1erKJ97A+fn5Gfcfu6k4Vo0Dx6nx4FhdmtteoOft7a3IyEjl5OQ4rM/JyVFMTEyN20RHRzuNf+eddxQVFVXj8/UAAMDNr8afOnWqVqxYoVWrVmnfvn1KTk5WQUGBEhPPP5+RkpKiuLg4+/jExER9/fXXmjp1qvbt26dVq1Zp5cqVmjZtmrt2AQCABs+tz9mPGTNGJSUlmjt3rgoLCxUREaHs7GyFhoZKkgoLCx3ecx8WFqbs7GwlJyfrxRdfVLt27fTCCy/oV7/6lbt2ocHw8fHRM8884/SUBRoejlXjwHFqPDhWP89mXXiFGwAAMJLbPy4XAADULWIPAIDhiD0AAIYj9kAdOnz4sGw2m3bv3u3uqQBowoh9I5CbmytPT08NGTLEYf3PhaSqqkppaWnq1q2bmjdvrtatW6t3797KyMioh1k3DfHx8bLZbPZbQECAhgwZok8//VTS+Y9lvvBOEzRM8fHxGjVqlNO/X1j+8fG9cLv47yJc6+6779aUKVOc1v/973+3f/Lp7NmzddtttzmNOXnypGw2m95//31J//v/ZE23jz/+uA73omFx+8fl4uetWrVKkyZN0ooVK1RQUKAOHTpc1nazZ8/W8uXLtXjxYkVFRamsrEy7du3S999/X8czblqGDBli/wWqqKhITz/9tIYPH66CggJ5enoqKCjIzTPE1fjx8b2At3g1Pv/85z918803O6y71CevmojYN3BnzpxRVlaWdu7cqaKiIq1evVqzZs26rG3ffPNNJSUl6de//rV93a233lpXU22yfHx87EEPCgrS9OnTddddd+m7777TmTNnFBYWpvz8fN12222qqqrSo48+qvfee09FRUXq0KGDkpKS9Pvf/95+f++//76eeuop7dmzR82aNdPNN9+sdevW2T9/AvXrx8cXjVdAQECTPo5cxm/gMjMz1bVrV3Xt2lUPPfSQMjIydLkfjRAUFKT33ntP3333XR3PEhecPn1aa9euVefOnWs8a6iurtYNN9ygrKws7d27V7NmzdLMmTOVlZUlSaqsrNSoUaPUr18/ffrpp9qxY4ceffRRo760CUD948y+gVu5cqUeeughSecvJ54+fVrvvvuuBg4c+LPbLly4UPfff7+CgoJ08803KyYmRiNHjtTQoUPretpNyltvvaVrr71W0vkrMcHBwXrrrbfk4eH8u3SzZs0cvnI5LCxMubm5ysrK0ujRo1VWVqbS0lINHz5cN954oyQpPDy8fnYENfrx8b1g+vTp+sMf/uCmGeFKxMTEOP2dLC0tlaenp5tmVL+IfQO2f/9+ffLJJ9q8ebMkycvLS2PGjNGqVasuK/bdu3fXf//7X+Xl5Wn79u368MMPNWLECMXHx2vFihV1Pf0mo3///kpPT5cknThxQkuWLNHQoUP1ySef1Dh+6dKlWrFihb7++mudPXtWFRUV9hcatW7dWvHx8Ro8eLAGDRqkgQMHavTo0QoODq6v3cFFfnx8L2jdurWbZoMrlZmZ6fSLc1MJvUTsG7SVK1eqsrJS7du3t6+zLEvNmjW77BfZeXh4qGfPnurZs6eSk5P16quv6uGHH1ZqaqrCwsLqaupNSosWLdS5c2f7cmRkpPz9/fXSSy8pISHBYWxWVpaSk5O1YMECRUdHq2XLlpo/f77+/e9/28dkZGRo8uTJ2rJlizIzM/X0008rJydHvXv3rrd9wv9cfHxR9/z8/FRaWuq0/uTJk/avsP2pMZLk7+/vsD4kJKRJH0ees2+gKisrtWbNGi1YsEC7d++23/7zn/8oNDRUa9euvaL77d69u6Tzl5tRN2w2mzw8PHT27Fmnn23btk0xMTFKSkrS7bffrs6dO+vLL790Gnf77bcrJSVFubm5ioiI0Lp16+pj6kCD0K1bN+3atctp/c6dO9W1a1f7mKNHj6qoqMhpjIeHR5MOe004s2+g3nrrLX3//feaMGGC02+o999/v1auXKnhw4dLOn+5/2Ldu3fX2LFj1adPH8XExCgoKEiHDh1SSkqKunTpom7dutXLfjQF5eXl9v/hfP/991q8eLFOnz6tESNGOI3t3Lmz1qxZo61btyosLEyvvPKKdu7cab/KcujQIS1fvlz33Xef2rVrp/379+vAgQMOX/WM+vXj43uBl5eX2rRp46YZmS8pKUmLFy/WxIkT9eijj6p58+bKycnRypUr9corr0iSYmNjFR4ergceeEDPPfec2rVrp08//VTTpk1TYmKiWrZs6XCfJSUlTsexVatW8vX1rbf9cisLDdLw4cOte++9t8af5eXlWZLs/6zpdujQIWv58uVW//79rbZt21re3t5Whw4drPj4eOvw4cP1vDfmGjdunMOfe8uWLa2ePXtaGzdutCzLsg4dOmRJsvLz8y3Lsqxz585Z8fHxlr+/v9WqVSvrd7/7nTVjxgzr1ltvtSzLsoqKiqxRo0ZZwcHBlre3txUaGmrNmjXLqqqqctMemm/cuHHWyJEjnf79wnJNf7+6du3qnsk2Ibt27bIGDx5sXX/99Zafn58VFRVlrV+/3mFMYWGhNX78eCs0NNRq3ry51a1bN2vu3LnWuXPn7GMu/B2s6Xbx/ZmMr7gFAMBwPGcPAIDhiD0AAIYj9gAAGI7YAwBgOGIPAIDhiD0AAIYj9gAAGI7YAwBgOGIPNAGHDx+WzWbT7t27G8xj3X333ZoyZUqdzwcAsQfgYiEhISosLFRERIQk6f3335fNZrN/GxmA+scX4QBwmYqKCnl7eysoKMjdUwHwI5zZA4bYsmWL7rzzTrVq1UoBAQEaPnx4jV+fe8Ebb7yhm266Sc2bN1f//v318ssvO52Bb9q0STfffLN8fHzUsWNHLViwwOE+OnbsqHnz5ik+Pl7+/v767W9/63AZ//Dhw+rfv78k6brrrpPNZlN8fLx9++rqaj311FNq3bq1goKCNHv2bIf7t9lsWrZsmYYPH65rrrlG4eHh2rFjh7744gvdfffdatGihaKjo39yPwGIb70DTLFx40Zr06ZN1oEDB6z8/HxrxIgR1i233GJVVVU5ffveoUOHrGbNmlnTpk2zPv/8c2v9+vVW+/btLUnW999/b1nW+W8d8/DwsObOnWvt37/fysjIsJo3b25lZGTYHzM0NNTy8/Oz5s+fbx08eNA6ePCgw2NVVlZamzZtsiRZ+/fvtwoLC62TJ09almVZ/fr1s/z8/KzZs2dbBw4csF5++WXLZrNZ77zzjv3+JVnt27e3MjMzrf3791ujRo2yOnbsaA0YMMDasmWLtXfvXqt3797WkCFD6uuPGWiUiD1gqOLiYkuS9dlnnznFfvr06VZERITD+NTUVIfYjx071ho0aJDDmCeffNLq3r27fTk0NNQaNWqUw5iLH+tf//qXw/1e0K9fP+vOO+90WNezZ09r+vTp9mVJ1tNPP21f3rFjhyXJWrlypX3d+vXrLV9f35//AwGaMC7jA4b48ssvNXbsWHXq1El+fn4KCwuTJBUUFDiN3b9/v3r27OmwrlevXg7L+/btU58+fRzW9enTRwcPHlRVVZV9XVRU1BXPuUePHg7LwcHBKi4uvuSYwMBASdItt9zisO7cuXMqKyu74nkApuMFeoAhRowYoZCQEL300ktq166dqqurFRERoYqKCqexlmXJZrM5ravtGElq0aLFFc+5WbNmDss2m03V1dWXHHNhPjWtu3g7AP9D7AEDlJSUaN++fVq2bJn69u0rSdq+ffslx3fr1k3Z2dkO63bt2uWw3L17d6f7yM3NVZcuXeTp6XnZc/P29pYkh6sBAOoXl/EBA1x33XUKCAjQ8uXL9cUXX+i9997T1KlTLzn+scce0+eff67p06frwIEDysrK0urVqyX970z5iSee0Lvvvqtnn31WBw4c0Msvv6zFixdr2rRptZpbaGiobDab3nrrLX333Xc6ffr0Fe8ngCtD7AEDeHh4aMOGDcrLy1NERISSk5M1f/78S44PCwvTxo0btXnzZvXo0UPp6elKTU2VJPn4+EiSfvGLXygrK0sbNmxQRESEZs2apblz5zq8de5ytG/fXnPmzNGMGTMUGBioxx9//Ir3E8CVsVk1PQkHoMl57rnntHTpUh05csTdUwHgYjxnDzRRS5YsUc+ePRUQEKCPPvpI8+fP56wbMBSxB5qogwcPat68eTpx4oQ6dOigJ554QikpKe6eFoA6wGV8AAAMxwv0AAAwHLEHAMBwxB4AAMMRewAADEfsAQAwHLEHAMBwxB4AAMMRewAADPf/lzplZDDwn3YAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 511.111x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.catplot(x='algorithm', y='RMSE', data=user_rmse, kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffbd7ba0",
   "metadata": {
    "papermill": {
     "duration": 0.002494,
     "end_time": "2024-12-07T20:28:01.398925",
     "exception": false,
     "start_time": "2024-12-07T20:28:01.396431",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Save Metrics\n",
    "\n",
    "We'll now save the top-N recommendation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1b776675",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:28:01.404277Z",
     "iopub.status.busy": "2024-12-07T20:28:01.404177Z",
     "iopub.status.idle": "2024-12-07T20:28:01.408419Z",
     "shell.execute_reply": "2024-12-07T20:28:01.408226Z"
    },
    "papermill": {
     "duration": 0.007596,
     "end_time": "2024-12-07T20:28:01.408994",
     "exception": false,
     "start_time": "2024-12-07T20:28:01.401398",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RecipRank</th>\n",
       "      <th>NDCG</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ALS</th>\n",
       "      <td>0.084256</td>\n",
       "      <td>0.079510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bias</th>\n",
       "      <td>0.070918</td>\n",
       "      <td>0.076594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IALS</th>\n",
       "      <td>0.260084</td>\n",
       "      <td>0.295100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IIE</th>\n",
       "      <td>0.040787</td>\n",
       "      <td>0.056837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>III</th>\n",
       "      <td>0.355172</td>\n",
       "      <td>0.284440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pop</th>\n",
       "      <td>0.217630</td>\n",
       "      <td>0.192362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UUE</th>\n",
       "      <td>0.013723</td>\n",
       "      <td>0.036543</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           RecipRank      NDCG\n",
       "algorithm                     \n",
       "ALS         0.084256  0.079510\n",
       "Bias        0.070918  0.076594\n",
       "IALS        0.260084  0.295100\n",
       "IIE         0.040787  0.056837\n",
       "III         0.355172  0.284440\n",
       "Pop         0.217630  0.192362\n",
       "UUE         0.013723  0.036543"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "saved = results[['recip_rank', 'ndcg']].rename(columns={'ndcg': 'NDCG', 'recip_rank': 'RecipRank'})\n",
    "saved = saved.groupby('algorithm').agg('mean')\n",
    "saved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c4083a95",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T20:28:01.414531Z",
     "iopub.status.busy": "2024-12-07T20:28:01.414437Z",
     "iopub.status.idle": "2024-12-07T20:28:01.416439Z",
     "shell.execute_reply": "2024-12-07T20:28:01.416144Z"
    },
    "papermill": {
     "duration": 0.005367,
     "end_time": "2024-12-07T20:28:01.416927",
     "exception": false,
     "start_time": "2024-12-07T20:28:01.411560",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "saved.to_json(f'eval-metrics.{dataset}.json', orient='index')"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md",
   "notebook_metadata_filter": "split_at_heading"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 3.100807,
   "end_time": "2024-12-07T20:28:01.633418",
   "environment_variables": {},
   "exception": null,
   "input_path": "eval-report.ipynb",
   "output_path": "eval-report.ml100k.ipynb",
   "parameters": {
    "dataset": "ml100k"
   },
   "start_time": "2024-12-07T20:27:58.532611",
   "version": "2.6.0"
  },
  "split_at_heading": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}