Chapter_05_TCP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5g8JAwEGo-p-"
   },
   "source": [
    "# Classical full Transductive Conformal Prediction for binary classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ohs_6mZOtAdr"
   },
   "source": [
    "\n",
    "Conformal Prediction is a machine learning framework that allows to quantify prediction uncertainty to produce reliable probabilisic predictions. Conformal Prediction can be used on top of any underlying model and works with any data distribution and any dataset size. \n",
    "\n",
    "The goal of this notebook is to help you understand Conformal Prediction for classification problem. We will build a conformal classifier by hand step-by-step explaining the required background on the way.\n",
    "\n",
    "We will be working with the classical Transductive (Full) Conformal Prediction framework."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:49.387797Z",
     "start_time": "2023-05-18T12:27:10.394961Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Ig67InlUpbFR",
    "outputId": "67c4769d-6837-464e-81f7-bc6098f079b1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error processing line 1 of /opt/anaconda3/envs/py39/lib/python3.9/site-packages/protobuf-3.19.6-py3.9-nspkg.pth:\n",
      "\n",
      "  Traceback (most recent call last):\n",
      "    File \"/opt/anaconda3/envs/py39/lib/python3.9/site.py\", line 169, in addpackage\n",
      "      exec(line)\n",
      "    File \"<string>\", line 1, in <module>\n",
      "    File \"<frozen importlib._bootstrap>\", line 562, in module_from_spec\n",
      "  AttributeError: 'NoneType' object has no attribute 'loader'\n",
      "\n",
      "Remainder of file ignored\n",
      "Collecting nonconformist\n",
      "  Downloading nonconformist-2.1.0.tar.gz (12 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: numpy in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from nonconformist) (1.23.5)\n",
      "Requirement already satisfied: scikit-learn in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from nonconformist) (1.2.2)\n",
      "Requirement already satisfied: scipy in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from nonconformist) (1.10.1)\n",
      "Requirement already satisfied: pandas in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from nonconformist) (1.5.3)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from pandas->nonconformist) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from pandas->nonconformist) (2023.3)\n",
      "Requirement already satisfied: joblib>=1.1.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from scikit-learn->nonconformist) (1.2.0)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from scikit-learn->nonconformist) (3.1.0)\n",
      "Requirement already satisfied: six>=1.5 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas->nonconformist) (1.16.0)\n",
      "Building wheels for collected packages: nonconformist\n",
      "  Building wheel for nonconformist (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for nonconformist: filename=nonconformist-2.1.0-py3-none-any.whl size=17818 sha256=07797c115f32330f3362e6d995c44ed4b5f003fa49309f7340148a73ead08e29\n",
      "  Stored in directory: /Users/vm/Library/Caches/pip/wheels/ce/2f/2f/dde1c910878402b5a67e999c3c3b0758d257466ebf093b6cc7\n",
      "Successfully built nonconformist\n",
      "Installing collected packages: nonconformist\n",
      "Successfully installed nonconformist-2.1.0\n",
      "Error processing line 1 of /opt/anaconda3/envs/py39/lib/python3.9/site-packages/protobuf-3.19.6-py3.9-nspkg.pth:\n",
      "\n",
      "  Traceback (most recent call last):\n",
      "    File \"/opt/anaconda3/envs/py39/lib/python3.9/site.py\", line 169, in addpackage\n",
      "      exec(line)\n",
      "    File \"<string>\", line 1, in <module>\n",
      "    File \"<frozen importlib._bootstrap>\", line 562, in module_from_spec\n",
      "  AttributeError: 'NoneType' object has no attribute 'loader'\n",
      "\n",
      "Remainder of file ignored\n",
      "Collecting openml\n",
      "  Downloading openml-0.13.1.tar.gz (127 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m127.6/127.6 kB\u001b[0m \u001b[31m1.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting liac-arff>=2.4.0\n",
      "  Using cached liac-arff-2.5.0.tar.gz (13 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting xmltodict\n",
      "  Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)\n",
      "Requirement already satisfied: requests in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from openml) (2.28.2)\n",
      "Requirement already satisfied: scikit-learn>=0.18 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from openml) (1.2.2)\n",
      "Requirement already satisfied: python-dateutil in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from openml) (2.8.2)\n",
      "Requirement already satisfied: pandas>=1.0.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from openml) (1.5.3)\n",
      "Requirement already satisfied: scipy>=0.13.3 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from openml) (1.10.1)\n",
      "Requirement already satisfied: numpy>=1.6.2 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from openml) (1.23.5)\n",
      "Collecting minio\n",
      "  Downloading minio-7.1.14-py3-none-any.whl (77 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.2/77.2 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: pyarrow in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from openml) (11.0.0)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from pandas>=1.0.0->openml) (2023.3)\n",
      "Requirement already satisfied: six>=1.5 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from python-dateutil->openml) (1.16.0)\n",
      "Requirement already satisfied: joblib>=1.1.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from scikit-learn>=0.18->openml) (1.2.0)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from scikit-learn>=0.18->openml) (3.1.0)\n",
      "Requirement already satisfied: urllib3 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from minio->openml) (1.26.15)\n",
      "Requirement already satisfied: certifi in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from minio->openml) (2022.6.15)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from requests->openml) (3.4)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from requests->openml) (2.1.1)\n",
      "Building wheels for collected packages: openml, liac-arff\n",
      "  Building wheel for openml (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for openml: filename=openml-0.13.1-py3-none-any.whl size=142784 sha256=ac8c37b6edb7c214a645e56c3b1492302e2b8658e6159d22f2733482936e3bf7\n",
      "  Stored in directory: /Users/vm/Library/Caches/pip/wheels/45/9a/36/e6701b6204a871ced537238ce7b5b2924a7408604bd301ad34\n",
      "  Building wheel for liac-arff (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for liac-arff: filename=liac_arff-2.5.0-py3-none-any.whl size=11717 sha256=9fd753ef489163feda16da496f4f2fc21d3d4fd014a07afad77184ceef1fbb23\n",
      "  Stored in directory: /Users/vm/Library/Caches/pip/wheels/08/82/8b/5c514221984e88c059b94e36a71d4722e590acaae04deab22e\n",
      "Successfully built openml liac-arff\n",
      "Installing collected packages: xmltodict, minio, liac-arff, openml\n",
      "Successfully installed liac-arff-2.5.0 minio-7.1.14 openml-0.13.1 xmltodict-0.13.0\n",
      "Error processing line 1 of /opt/anaconda3/envs/py39/lib/python3.9/site-packages/protobuf-3.19.6-py3.9-nspkg.pth:\n",
      "\n",
      "  Traceback (most recent call last):\n",
      "    File \"/opt/anaconda3/envs/py39/lib/python3.9/site.py\", line 169, in addpackage\n",
      "      exec(line)\n",
      "    File \"<string>\", line 1, in <module>\n",
      "    File \"<frozen importlib._bootstrap>\", line 562, in module_from_spec\n",
      "  AttributeError: 'NoneType' object has no attribute 'loader'\n",
      "\n",
      "Remainder of file ignored\n",
      "Requirement already satisfied: dtype_diet in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (0.0.1)\n",
      "Requirement already satisfied: pandas>=1.0.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from dtype_diet) (1.5.3)\n",
      "Requirement already satisfied: tabulate in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from dtype_diet) (0.9.0)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from pandas>=1.0.0->dtype_diet) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from pandas>=1.0.0->dtype_diet) (2023.3)\n",
      "Requirement already satisfied: numpy>=1.20.3 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from pandas>=1.0.0->dtype_diet) (1.23.5)\n",
      "Requirement already satisfied: six>=1.5 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas>=1.0.0->dtype_diet) (1.16.0)\n",
      "Error processing line 1 of /opt/anaconda3/envs/py39/lib/python3.9/site-packages/protobuf-3.19.6-py3.9-nspkg.pth:\n",
      "\n",
      "  Traceback (most recent call last):\n",
      "    File \"/opt/anaconda3/envs/py39/lib/python3.9/site.py\", line 169, in addpackage\n",
      "      exec(line)\n",
      "    File \"<string>\", line 1, in <module>\n",
      "    File \"<frozen importlib._bootstrap>\", line 562, in module_from_spec\n",
      "  AttributeError: 'NoneType' object has no attribute 'loader'\n",
      "\n",
      "Remainder of file ignored\n",
      "Requirement already satisfied: xgboost in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (1.7.1)\n",
      "Requirement already satisfied: numpy in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from xgboost) (1.23.5)\n",
      "Requirement already satisfied: scipy in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from xgboost) (1.10.1)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error processing line 1 of /opt/anaconda3/envs/py39/lib/python3.9/site-packages/protobuf-3.19.6-py3.9-nspkg.pth:\n",
      "\n",
      "  Traceback (most recent call last):\n",
      "    File \"/opt/anaconda3/envs/py39/lib/python3.9/site.py\", line 169, in addpackage\n",
      "      exec(line)\n",
      "    File \"<string>\", line 1, in <module>\n",
      "    File \"<frozen importlib._bootstrap>\", line 562, in module_from_spec\n",
      "  AttributeError: 'NoneType' object has no attribute 'loader'\n",
      "\n",
      "Remainder of file ignored\n",
      "Requirement already satisfied: xgboost in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (1.7.1)\n",
      "Collecting xgboost\n",
      "  Downloading xgboost-1.7.5-py3-none-macosx_10_15_x86_64.macosx_11_0_x86_64.macosx_12_0_x86_64.whl (1.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m0m\n",
      "\u001b[?25hRequirement already satisfied: numpy in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from xgboost) (1.23.5)\n",
      "Requirement already satisfied: scipy in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from xgboost) (1.10.1)\n",
      "Installing collected packages: xgboost\n",
      "  Attempting uninstall: xgboost\n",
      "    Found existing installation: xgboost 1.7.1\n",
      "    Uninstalling xgboost-1.7.1:\n",
      "      Successfully uninstalled xgboost-1.7.1\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "verstack 3.3.3 requires category-encoders<=2.5.1,>=2.4.0, but you have category-encoders 2.6.0 which is incompatible.\n",
      "verstack 3.3.3 requires lightgbm<=3.3.2,>=3.3.0, but you have lightgbm 3.3.5 which is incompatible.\n",
      "verstack 3.3.3 requires optuna<=3.0.4,>=2.10.0, but you have optuna 3.1.0 which is incompatible.\n",
      "verstack 3.3.3 requires plotly<=5.11.0,>=5.3.1, but you have plotly 5.14.0 which is incompatible.\n",
      "verstack 3.3.3 requires scikit-learn<=1.1.3,>=0.23.2, but you have scikit-learn 1.2.2 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed xgboost-1.7.5\n",
      "Error processing line 1 of /opt/anaconda3/envs/py39/lib/python3.9/site-packages/protobuf-3.19.6-py3.9-nspkg.pth:\n",
      "\n",
      "  Traceback (most recent call last):\n",
      "    File \"/opt/anaconda3/envs/py39/lib/python3.9/site.py\", line 169, in addpackage\n",
      "      exec(line)\n",
      "    File \"<string>\", line 1, in <module>\n",
      "    File \"<frozen importlib._bootstrap>\", line 562, in module_from_spec\n",
      "  AttributeError: 'NoneType' object has no attribute 'loader'\n",
      "\n",
      "Remainder of file ignored\n",
      "Requirement already satisfied: catboost in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (1.1.1)\n",
      "Requirement already satisfied: numpy>=1.16.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from catboost) (1.23.5)\n",
      "Requirement already satisfied: six in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from catboost) (1.16.0)\n",
      "Requirement already satisfied: pandas>=0.24.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from catboost) (1.5.3)\n",
      "Requirement already satisfied: graphviz in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from catboost) (0.20.1)\n",
      "Requirement already satisfied: plotly in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from catboost) (5.14.0)\n",
      "Requirement already satisfied: scipy in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from catboost) (1.10.1)\n",
      "Requirement already satisfied: matplotlib in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from catboost) (3.7.1)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from pandas>=0.24.0->catboost) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from pandas>=0.24.0->catboost) (2023.3)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from matplotlib->catboost) (4.39.3)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from matplotlib->catboost) (1.4.4)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from matplotlib->catboost) (23.0)\n",
      "Requirement already satisfied: pyparsing>=2.3.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from matplotlib->catboost) (3.0.9)\n",
      "Requirement already satisfied: importlib-resources>=3.2.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from matplotlib->catboost) (5.12.0)\n",
      "Requirement already satisfied: cycler>=0.10 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from matplotlib->catboost) (0.11.0)\n",
      "Requirement already satisfied: pillow>=6.2.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from matplotlib->catboost) (9.4.0)\n",
      "Requirement already satisfied: contourpy>=1.0.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from matplotlib->catboost) (1.0.7)\n",
      "Requirement already satisfied: tenacity>=6.2.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from plotly->catboost) (8.2.2)\n",
      "Requirement already satisfied: zipp>=3.1.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from importlib-resources>=3.2.0->matplotlib->catboost) (3.15.0)\n",
      "Error processing line 1 of /opt/anaconda3/envs/py39/lib/python3.9/site-packages/protobuf-3.19.6-py3.9-nspkg.pth:\n",
      "\n",
      "  Traceback (most recent call last):\n",
      "    File \"/opt/anaconda3/envs/py39/lib/python3.9/site.py\", line 169, in addpackage\n",
      "      exec(line)\n",
      "    File \"<string>\", line 1, in <module>\n",
      "    File \"<frozen importlib._bootstrap>\", line 562, in module_from_spec\n",
      "  AttributeError: 'NoneType' object has no attribute 'loader'\n",
      "\n",
      "Remainder of file ignored\n",
      "Requirement already satisfied: ipywidgets in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (8.0.6)\n",
      "Requirement already satisfied: widgetsnbextension~=4.0.7 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipywidgets) (4.0.7)\n",
      "Requirement already satisfied: jupyterlab-widgets~=3.0.7 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipywidgets) (3.0.7)\n",
      "Requirement already satisfied: traitlets>=4.3.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipywidgets) (5.9.0)\n",
      "Requirement already satisfied: ipykernel>=4.5.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipywidgets) (6.22.0)\n",
      "Requirement already satisfied: ipython>=6.1.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipywidgets) (8.12.0)\n",
      "Requirement already satisfied: tornado>=6.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (6.2)\n",
      "Requirement already satisfied: nest-asyncio in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (1.5.6)\n",
      "Requirement already satisfied: debugpy>=1.6.5 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (1.6.6)\n",
      "Requirement already satisfied: psutil in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (5.9.4)\n",
      "Requirement already satisfied: pyzmq>=20 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (25.0.2)\n",
      "Requirement already satisfied: appnope in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (0.1.3)\n",
      "Requirement already satisfied: comm>=0.1.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (0.1.3)\n",
      "Requirement already satisfied: jupyter-client>=6.1.12 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (7.4.9)\n",
      "Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (5.3.0)\n",
      "Requirement already satisfied: matplotlib-inline>=0.1 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (0.1.6)\n",
      "Requirement already satisfied: packaging in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (23.0)\n",
      "Requirement already satisfied: stack-data in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.6.2)\n",
      "Requirement already satisfied: jedi>=0.16 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.18.2)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (3.0.38)\n",
      "Requirement already satisfied: typing-extensions in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (4.5.0)\n",
      "Requirement already satisfied: pickleshare in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.7.5)\n",
      "Requirement already satisfied: decorator in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (5.1.1)\n",
      "Requirement already satisfied: pexpect>4.3 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (4.8.0)\n",
      "Requirement already satisfied: pygments>=2.4.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (2.14.0)\n",
      "Requirement already satisfied: backcall in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.2.0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.3)\n",
      "Requirement already satisfied: entrypoints in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets) (0.4)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets) (2.8.2)\n",
      "Requirement already satisfied: platformdirs>=2.5 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from jupyter-core!=5.0.*,>=4.12->ipykernel>=4.5.1->ipywidgets) (3.2.0)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0)\n",
      "Requirement already satisfied: wcwidth in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython>=6.1.0->ipywidgets) (0.2.6)\n",
      "Requirement already satisfied: asttokens>=2.1.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.2.1)\n",
      "Requirement already satisfied: pure-eval in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (0.2.2)\n",
      "Requirement already satisfied: executing>=1.2.0 in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (1.2.0)\n",
      "Requirement already satisfied: six in /opt/anaconda3/envs/py39/lib/python3.9/site-packages (from asttokens>=2.1.0->stack-data->ipython>=6.1.0->ipywidgets) (1.16.0)\n",
      "Error processing line 1 of /opt/anaconda3/envs/py39/lib/python3.9/site-packages/protobuf-3.19.6-py3.9-nspkg.pth:\n",
      "\n",
      "  Traceback (most recent call last):\n",
      "    File \"/opt/anaconda3/envs/py39/lib/python3.9/site.py\", line 169, in addpackage\n",
      "      exec(line)\n",
      "    File \"<string>\", line 1, in <module>\n",
      "    File \"<frozen importlib._bootstrap>\", line 562, in module_from_spec\n",
      "  AttributeError: 'NoneType' object has no attribute 'loader'\n",
      "\n",
      "Remainder of file ignored\n",
      "Error processing line 1 of /opt/anaconda3/envs/py39/lib/python3.9/site-packages/protobuf-3.19.6-py3.9-nspkg.pth:\n",
      "\n",
      "  Traceback (most recent call last):\n",
      "    File \"/opt/anaconda3/envs/py39/lib/python3.9/site.py\", line 169, in addpackage\n",
      "      exec(line)\n",
      "    File \"<string>\", line 1, in <module>\n",
      "    File \"<frozen importlib._bootstrap>\", line 562, in module_from_spec\n",
      "  AttributeError: 'NoneType' object has no attribute 'loader'\n",
      "\n",
      "Remainder of file ignored\n",
      "Enabling notebook extension jupyter-js-widgets/extension...\n",
      "      - Validating: \u001b[32mOK\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "#install OpenML\n",
    "!pip install nonconformist\n",
    "!pip install openml\n",
    "!pip install dtype_diet\n",
    "\n",
    "#install XGBoost\n",
    "!pip install xgboost\n",
    "!pip install --upgrade xgboost\n",
    "\n",
    "#install CatBoost\n",
    "!pip install catboost\n",
    "!pip install ipywidgets\n",
    "!jupyter nbextension enable --py widgetsnbextension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:53.197724Z",
     "start_time": "2023-05-18T12:27:49.393381Z"
    },
    "id": "Kc-JdD9IsWgm"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "from scipy.stats import rankdata\n",
    "\n",
    "import openml\n",
    "\n",
    "from sklearn.metrics import log_loss, accuracy_score, roc_auc_score, confusion_matrix\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.svm import SVC, LinearSVC\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from  sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n",
    "from sklearn.preprocessing import label_binarize\n",
    "\n",
    "import plotly.express as px\n",
    "\n",
    "from lightgbm import LGBMClassifier\n",
    "from xgboost import XGBClassifier\n",
    "from catboost import CatBoostClassifier\n",
    "\n",
    "from nonconformist.cp import TcpClassifier\n",
    "from nonconformist.nc import NcFactory\n",
    "from nonconformist.nc import InverseProbabilityErrFunc, MarginErrFunc\n",
    "\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uV4g10IOtW-6"
   },
   "source": [
    "\n",
    "# Load dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:54.911417Z",
     "start_time": "2023-05-18T12:27:53.199057Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XK2pin99tR1u",
    "outputId": "b8516228-9827-4598-ceea-fd0ee637364f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    did             name  version uploader  status format  MajorityClassSize  \\\n",
      "2     2           anneal        1        1  active   ARFF              684.0   \n",
      "3     3         kr-vs-kp        1        1  active   ARFF             1669.0   \n",
      "4     4            labor        1        1  active   ARFF               37.0   \n",
      "5     5       arrhythmia        1        1  active   ARFF              245.0   \n",
      "6     6           letter        1        1  active   ARFF              813.0   \n",
      "7     7        audiology        1        1  active   ARFF               57.0   \n",
      "8     8  liver-disorders        1        1  active   ARFF                NaN   \n",
      "9     9            autos        1        1  active   ARFF               67.0   \n",
      "10   10            lymph        1        1  active   ARFF               81.0   \n",
      "11   11    balance-scale        1        1  active   ARFF              288.0   \n",
      "\n",
      "    MaxNominalAttDistinctValues  MinorityClassSize  NumberOfClasses  \\\n",
      "2                           7.0                8.0              5.0   \n",
      "3                           3.0             1527.0              2.0   \n",
      "4                           3.0               20.0              2.0   \n",
      "5                          13.0                2.0             13.0   \n",
      "6                          26.0              734.0             26.0   \n",
      "7                          24.0                1.0             24.0   \n",
      "8                           NaN                NaN              0.0   \n",
      "9                          22.0                3.0              6.0   \n",
      "10                          8.0                2.0              4.0   \n",
      "11                          3.0               49.0              3.0   \n",
      "\n",
      "    NumberOfFeatures  NumberOfInstances  NumberOfInstancesWithMissingValues  \\\n",
      "2               39.0              898.0                               898.0   \n",
      "3               37.0             3196.0                                 0.0   \n",
      "4               17.0               57.0                                56.0   \n",
      "5              280.0              452.0                               384.0   \n",
      "6               17.0            20000.0                                 0.0   \n",
      "7               70.0              226.0                               222.0   \n",
      "8                6.0              345.0                                 0.0   \n",
      "9               26.0              205.0                                46.0   \n",
      "10              19.0              148.0                                 0.0   \n",
      "11               5.0              625.0                                 0.0   \n",
      "\n",
      "    NumberOfMissingValues  NumberOfNumericFeatures  NumberOfSymbolicFeatures  \n",
      "2                 22175.0                      6.0                      33.0  \n",
      "3                     0.0                      0.0                      37.0  \n",
      "4                   326.0                      8.0                       9.0  \n",
      "5                   408.0                    206.0                      74.0  \n",
      "6                     0.0                     16.0                       1.0  \n",
      "7                   317.0                      0.0                      70.0  \n",
      "8                     0.0                      6.0                       0.0  \n",
      "9                    59.0                     15.0                      11.0  \n",
      "10                    0.0                      3.0                      16.0  \n",
      "11                    0.0                      4.0                       1.0  \n"
     ]
    }
   ],
   "source": [
    "# List datasets https://docs.openml.org/Python-API/\n",
    "datasets_df = openml.datasets.list_datasets(output_format=\"dataframe\")\n",
    "print(datasets_df.head(n=10))\n",
    "\n",
    "datasets_df.set_index('did', inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:54.923163Z",
     "start_time": "2023-05-18T12:27:54.912637Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 299
    },
    "id": "7vaAPOxQtcqM",
    "outputId": "52ff5255-f05f-4ab3-f336-54a2124332c4"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>version</th>\n",
       "      <th>uploader</th>\n",
       "      <th>status</th>\n",
       "      <th>format</th>\n",
       "      <th>MajorityClassSize</th>\n",
       "      <th>MaxNominalAttDistinctValues</th>\n",
       "      <th>MinorityClassSize</th>\n",
       "      <th>NumberOfClasses</th>\n",
       "      <th>NumberOfFeatures</th>\n",
       "      <th>NumberOfInstances</th>\n",
       "      <th>NumberOfInstancesWithMissingValues</th>\n",
       "      <th>NumberOfMissingValues</th>\n",
       "      <th>NumberOfNumericFeatures</th>\n",
       "      <th>NumberOfSymbolicFeatures</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>did</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>anneal</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>active</td>\n",
       "      <td>ARFF</td>\n",
       "      <td>684.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>898.0</td>\n",
       "      <td>898.0</td>\n",
       "      <td>22175.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>33.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>kr-vs-kp</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>active</td>\n",
       "      <td>ARFF</td>\n",
       "      <td>1669.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1527.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>3196.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>37.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>labor</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>active</td>\n",
       "      <td>ARFF</td>\n",
       "      <td>37.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>326.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>arrhythmia</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>active</td>\n",
       "      <td>ARFF</td>\n",
       "      <td>245.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>280.0</td>\n",
       "      <td>452.0</td>\n",
       "      <td>384.0</td>\n",
       "      <td>408.0</td>\n",
       "      <td>206.0</td>\n",
       "      <td>74.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>letter</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>active</td>\n",
       "      <td>ARFF</td>\n",
       "      <td>813.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>734.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>20000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           name  version uploader  status format  MajorityClassSize  \\\n",
       "did                                                                   \n",
       "2        anneal        1        1  active   ARFF              684.0   \n",
       "3      kr-vs-kp        1        1  active   ARFF             1669.0   \n",
       "4         labor        1        1  active   ARFF               37.0   \n",
       "5    arrhythmia        1        1  active   ARFF              245.0   \n",
       "6        letter        1        1  active   ARFF              813.0   \n",
       "\n",
       "     MaxNominalAttDistinctValues  MinorityClassSize  NumberOfClasses  \\\n",
       "did                                                                    \n",
       "2                            7.0                8.0              5.0   \n",
       "3                            3.0             1527.0              2.0   \n",
       "4                            3.0               20.0              2.0   \n",
       "5                           13.0                2.0             13.0   \n",
       "6                           26.0              734.0             26.0   \n",
       "\n",
       "     NumberOfFeatures  NumberOfInstances  NumberOfInstancesWithMissingValues  \\\n",
       "did                                                                            \n",
       "2                39.0              898.0                               898.0   \n",
       "3                37.0             3196.0                                 0.0   \n",
       "4                17.0               57.0                                56.0   \n",
       "5               280.0              452.0                               384.0   \n",
       "6                17.0            20000.0                                 0.0   \n",
       "\n",
       "     NumberOfMissingValues  NumberOfNumericFeatures  NumberOfSymbolicFeatures  \n",
       "did                                                                            \n",
       "2                  22175.0                      6.0                      33.0  \n",
       "3                      0.0                      0.0                      37.0  \n",
       "4                    326.0                      8.0                       9.0  \n",
       "5                    408.0                    206.0                      74.0  \n",
       "6                      0.0                     16.0                       1.0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:54.942803Z",
     "start_time": "2023-05-18T12:27:54.924616Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 531
    },
    "id": "AijKKhPmttli",
    "outputId": "f02a32c9-41c7-4a53-e7d4-cc79d4a5835c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>version</th>\n",
       "      <th>uploader</th>\n",
       "      <th>status</th>\n",
       "      <th>format</th>\n",
       "      <th>MajorityClassSize</th>\n",
       "      <th>MaxNominalAttDistinctValues</th>\n",
       "      <th>MinorityClassSize</th>\n",
       "      <th>NumberOfClasses</th>\n",
       "      <th>NumberOfFeatures</th>\n",
       "      <th>NumberOfInstances</th>\n",
       "      <th>NumberOfInstancesWithMissingValues</th>\n",
       "      <th>NumberOfMissingValues</th>\n",
       "      <th>NumberOfNumericFeatures</th>\n",
       "      <th>NumberOfSymbolicFeatures</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>did</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>kr-vs-kp</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>active</td>\n",
       "      <td>ARFF</td>\n",
       "      <td>1669.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1527.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>3196.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>37.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>labor</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>active</td>\n",
       "      <td>ARFF</td>\n",
       "      <td>37.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>326.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>breast-cancer</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>active</td>\n",
       "      <td>ARFF</td>\n",
       "      <td>201.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>286.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>breast-w</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>active</td>\n",
       "      <td>ARFF</td>\n",
       "      <td>458.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>241.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>699.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>mushroom</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>active</td>\n",
       "      <td>ARFF</td>\n",
       "      <td>4208.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>3916.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>8124.0</td>\n",
       "      <td>2480.0</td>\n",
       "      <td>2480.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>23.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45376</th>\n",
       "      <td>timing_attack_100_micro_seconds_large</td>\n",
       "      <td>1</td>\n",
       "      <td>2086</td>\n",
       "      <td>active</td>\n",
       "      <td>arff</td>\n",
       "      <td>50113.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49847.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>99960.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45377</th>\n",
       "      <td>timing_attack_1000_micro_seconds_large</td>\n",
       "      <td>1</td>\n",
       "      <td>2086</td>\n",
       "      <td>active</td>\n",
       "      <td>arff</td>\n",
       "      <td>50061.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49897.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>99958.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45378</th>\n",
       "      <td>timing_attack_30_micro_seconds_large</td>\n",
       "      <td>1</td>\n",
       "      <td>2086</td>\n",
       "      <td>active</td>\n",
       "      <td>arff</td>\n",
       "      <td>50012.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49944.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>99956.0</td>\n",
       "      <td>99956.0</td>\n",
       "      <td>3498457.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45379</th>\n",
       "      <td>timing_attack_300_micro_seconds_large</td>\n",
       "      <td>1</td>\n",
       "      <td>2086</td>\n",
       "      <td>active</td>\n",
       "      <td>arff</td>\n",
       "      <td>50273.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49677.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>99950.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45380</th>\n",
       "      <td>timing_attack_60_micro_seconds_large</td>\n",
       "      <td>1</td>\n",
       "      <td>2086</td>\n",
       "      <td>active</td>\n",
       "      <td>arff</td>\n",
       "      <td>50132.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49832.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>99964.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1392 rows × 15 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         name  version uploader  status  \\\n",
       "did                                                                       \n",
       "3                                    kr-vs-kp        1        1  active   \n",
       "4                                       labor        1        1  active   \n",
       "13                              breast-cancer        1        1  active   \n",
       "15                                   breast-w        1        1  active   \n",
       "24                                   mushroom        1        1  active   \n",
       "...                                       ...      ...      ...     ...   \n",
       "45376   timing_attack_100_micro_seconds_large        1     2086  active   \n",
       "45377  timing_attack_1000_micro_seconds_large        1     2086  active   \n",
       "45378    timing_attack_30_micro_seconds_large        1     2086  active   \n",
       "45379   timing_attack_300_micro_seconds_large        1     2086  active   \n",
       "45380    timing_attack_60_micro_seconds_large        1     2086  active   \n",
       "\n",
       "      format  MajorityClassSize  MaxNominalAttDistinctValues  \\\n",
       "did                                                            \n",
       "3       ARFF             1669.0                          3.0   \n",
       "4       ARFF               37.0                          3.0   \n",
       "13      ARFF              201.0                         11.0   \n",
       "15      ARFF              458.0                          2.0   \n",
       "24      ARFF             4208.0                         12.0   \n",
       "...      ...                ...                          ...   \n",
       "45376   arff            50113.0                          NaN   \n",
       "45377   arff            50061.0                          NaN   \n",
       "45378   arff            50012.0                          NaN   \n",
       "45379   arff            50273.0                          NaN   \n",
       "45380   arff            50132.0                          NaN   \n",
       "\n",
       "       MinorityClassSize  NumberOfClasses  NumberOfFeatures  \\\n",
       "did                                                           \n",
       "3                 1527.0              2.0              37.0   \n",
       "4                   20.0              2.0              17.0   \n",
       "13                  85.0              2.0              10.0   \n",
       "15                 241.0              2.0              10.0   \n",
       "24                3916.0              2.0              23.0   \n",
       "...                  ...              ...               ...   \n",
       "45376            49847.0              2.0              95.0   \n",
       "45377            49897.0              2.0              95.0   \n",
       "45378            49944.0              2.0             130.0   \n",
       "45379            49677.0              2.0              95.0   \n",
       "45380            49832.0              2.0              95.0   \n",
       "\n",
       "       NumberOfInstances  NumberOfInstancesWithMissingValues  \\\n",
       "did                                                            \n",
       "3                 3196.0                                 0.0   \n",
       "4                   57.0                                56.0   \n",
       "13                 286.0                                 9.0   \n",
       "15                 699.0                                16.0   \n",
       "24                8124.0                              2480.0   \n",
       "...                  ...                                 ...   \n",
       "45376            99960.0                                 0.0   \n",
       "45377            99958.0                                 0.0   \n",
       "45378            99956.0                             99956.0   \n",
       "45379            99950.0                                 0.0   \n",
       "45380            99964.0                                 0.0   \n",
       "\n",
       "       NumberOfMissingValues  NumberOfNumericFeatures  \\\n",
       "did                                                     \n",
       "3                        0.0                      0.0   \n",
       "4                      326.0                      8.0   \n",
       "13                       9.0                      0.0   \n",
       "15                      16.0                      9.0   \n",
       "24                    2480.0                      0.0   \n",
       "...                      ...                      ...   \n",
       "45376                    0.0                     94.0   \n",
       "45377                    0.0                     94.0   \n",
       "45378              3498457.0                    129.0   \n",
       "45379                    0.0                     94.0   \n",
       "45380                    0.0                     94.0   \n",
       "\n",
       "       NumberOfSymbolicFeatures  \n",
       "did                              \n",
       "3                          37.0  \n",
       "4                           9.0  \n",
       "13                         10.0  \n",
       "15                          1.0  \n",
       "24                         23.0  \n",
       "...                         ...  \n",
       "45376                       0.0  \n",
       "45377                       0.0  \n",
       "45378                       0.0  \n",
       "45379                       0.0  \n",
       "45380                       0.0  \n",
       "\n",
       "[1392 rows x 15 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets_df.query('NumberOfClasses == 2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:54.949072Z",
     "start_time": "2023-05-18T12:27:54.944546Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rBVC3K7cuPZ9",
    "outputId": "e6e9527d-207a-4ca1-c8d6-7d2695264bbb"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name                                  credit-g\n",
       "version                                      1\n",
       "uploader                                     1\n",
       "status                                  active\n",
       "format                                    ARFF\n",
       "MajorityClassSize                        700.0\n",
       "MaxNominalAttDistinctValues               10.0\n",
       "MinorityClassSize                        300.0\n",
       "NumberOfClasses                            2.0\n",
       "NumberOfFeatures                          21.0\n",
       "NumberOfInstances                       1000.0\n",
       "NumberOfInstancesWithMissingValues         0.0\n",
       "NumberOfMissingValues                      0.0\n",
       "NumberOfNumericFeatures                    7.0\n",
       "NumberOfSymbolicFeatures                  14.0\n",
       "Name: 31, dtype: object"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# output information about selected dataset\n",
    "datasets_df.loc[31]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.046216Z",
     "start_time": "2023-05-18T12:27:54.950478Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wANKvmG-u_NI",
    "outputId": "f4ac7438-f844-4836-ed40-e88a16c1ddb8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is dataset 'credit-g', the target feature is 'class'\n",
      "URL: https://api.openml.org/data/v1/download/31/credit-g.arff\n",
      "**Author**: Dr. Hans Hofmann  \n",
      "**Source**: [UCI](https://archive.ics.uci.edu/ml/datasets/statlog+(german+credit+data)) - 1994    \n",
      "**Please cite**: [UCI](https://archive.ics.uci.edu/ml/citation_policy.html)\n",
      "\n",
      "**German Credit dataset**  \n",
      "This dataset classifies people described by a set of attributes as good or bad credit risks.\n",
      "\n",
      "This dataset comes with a cost matrix: \n",
      "``` \n",
      "Good  Bad (predicted)  \n",
      "Good   0    1   (actual)  \n",
      "Bad    5    0  \n",
      "```\n",
      "\n",
      "It is worse to class a customer as good when they are \n"
     ]
    }
   ],
   "source": [
    "# German credit dataset https://www.openml.org/d/31\n",
    "dataset = openml.datasets.get_dataset(31)\n",
    "\n",
    "# Print a summary\n",
    "print(\n",
    "    f\"This is dataset '{dataset.name}', the target feature is \"\n",
    "    f\"'{dataset.default_target_attribute}'\"\n",
    ")\n",
    "print(f\"URL: {dataset.url}\")\n",
    "print(dataset.description[:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.440514Z",
     "start_time": "2023-05-18T12:27:57.049396Z"
    },
    "id": "ZQDjL66ovJCk"
   },
   "outputs": [],
   "source": [
    "# openml API\n",
    "X, y, categorical_indicator, attribute_names = dataset.get_data(\n",
    "    dataset_format=\"array\", target=dataset.default_target_attribute\n",
    ")\n",
    "df = pd.DataFrame(X, columns=attribute_names)\n",
    "df[\"class\"] = y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.456319Z",
     "start_time": "2023-05-18T12:27:57.441705Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "id": "g9iGKJn4vOT9",
    "outputId": "8e33ebd3-8943-472d-f517-93a60a8cfc56"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>checking_status</th>\n",
       "      <th>duration</th>\n",
       "      <th>credit_history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>credit_amount</th>\n",
       "      <th>savings_status</th>\n",
       "      <th>employment</th>\n",
       "      <th>installment_commitment</th>\n",
       "      <th>personal_status</th>\n",
       "      <th>other_parties</th>\n",
       "      <th>...</th>\n",
       "      <th>property_magnitude</th>\n",
       "      <th>age</th>\n",
       "      <th>other_payment_plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>existing_credits</th>\n",
       "      <th>job</th>\n",
       "      <th>num_dependents</th>\n",
       "      <th>own_telephone</th>\n",
       "      <th>foreign_worker</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1169.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5951.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2096.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7882.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4870.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   checking_status  duration  credit_history  purpose  credit_amount  \\\n",
       "0              0.0       6.0             4.0      3.0         1169.0   \n",
       "1              1.0      48.0             2.0      3.0         5951.0   \n",
       "2              3.0      12.0             4.0      6.0         2096.0   \n",
       "3              0.0      42.0             2.0      2.0         7882.0   \n",
       "4              0.0      24.0             3.0      0.0         4870.0   \n",
       "\n",
       "   savings_status  employment  installment_commitment  personal_status  \\\n",
       "0             4.0         4.0                     4.0              2.0   \n",
       "1             0.0         2.0                     2.0              1.0   \n",
       "2             0.0         3.0                     2.0              2.0   \n",
       "3             0.0         3.0                     2.0              2.0   \n",
       "4             0.0         2.0                     3.0              2.0   \n",
       "\n",
       "   other_parties  ...  property_magnitude   age  other_payment_plans  housing  \\\n",
       "0            0.0  ...                 0.0  67.0                  2.0      1.0   \n",
       "1            0.0  ...                 0.0  22.0                  2.0      1.0   \n",
       "2            0.0  ...                 0.0  49.0                  2.0      1.0   \n",
       "3            2.0  ...                 1.0  45.0                  2.0      2.0   \n",
       "4            0.0  ...                 3.0  53.0                  2.0      2.0   \n",
       "\n",
       "   existing_credits  job  num_dependents  own_telephone  foreign_worker  class  \n",
       "0               2.0  2.0             1.0            1.0             0.0      0  \n",
       "1               1.0  2.0             1.0            0.0             0.0      1  \n",
       "2               1.0  1.0             2.0            0.0             0.0      0  \n",
       "3               1.0  2.0             2.0            0.0             0.0      0  \n",
       "4               2.0  2.0             2.0            0.0             0.0      1  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.466396Z",
     "start_time": "2023-05-18T12:27:57.461669Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Vt5W5pL0vZxk",
    "outputId": "f46c6331-55c1-4536-8baf-f587fa201f4a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    700\n",
       "1    300\n",
       "Name: class, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['class'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mRQ5oestwZXo"
   },
   "source": [
    "## Split into training and test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.470807Z",
     "start_time": "2023-05-18T12:27:57.467639Z"
    },
    "id": "VKqATqO5wyUC"
   },
   "outputs": [],
   "source": [
    "X = df.drop('class', axis = 1)\n",
    "y = df['class']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.485527Z",
     "start_time": "2023-05-18T12:27:57.472086Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 268
    },
    "id": "GHcOiGT9xEAU",
    "outputId": "e3de3f39-3856-4fd6-e086-61806bcfe54a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>checking_status</th>\n",
       "      <th>duration</th>\n",
       "      <th>credit_history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>credit_amount</th>\n",
       "      <th>savings_status</th>\n",
       "      <th>employment</th>\n",
       "      <th>installment_commitment</th>\n",
       "      <th>personal_status</th>\n",
       "      <th>other_parties</th>\n",
       "      <th>residence_since</th>\n",
       "      <th>property_magnitude</th>\n",
       "      <th>age</th>\n",
       "      <th>other_payment_plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>existing_credits</th>\n",
       "      <th>job</th>\n",
       "      <th>num_dependents</th>\n",
       "      <th>own_telephone</th>\n",
       "      <th>foreign_worker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1169.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5951.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2096.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7882.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4870.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   checking_status  duration  credit_history  purpose  credit_amount  \\\n",
       "0              0.0       6.0             4.0      3.0         1169.0   \n",
       "1              1.0      48.0             2.0      3.0         5951.0   \n",
       "2              3.0      12.0             4.0      6.0         2096.0   \n",
       "3              0.0      42.0             2.0      2.0         7882.0   \n",
       "4              0.0      24.0             3.0      0.0         4870.0   \n",
       "\n",
       "   savings_status  employment  installment_commitment  personal_status  \\\n",
       "0             4.0         4.0                     4.0              2.0   \n",
       "1             0.0         2.0                     2.0              1.0   \n",
       "2             0.0         3.0                     2.0              2.0   \n",
       "3             0.0         3.0                     2.0              2.0   \n",
       "4             0.0         2.0                     3.0              2.0   \n",
       "\n",
       "   other_parties  residence_since  property_magnitude   age  \\\n",
       "0            0.0              4.0                 0.0  67.0   \n",
       "1            0.0              2.0                 0.0  22.0   \n",
       "2            0.0              3.0                 0.0  49.0   \n",
       "3            2.0              4.0                 1.0  45.0   \n",
       "4            0.0              4.0                 3.0  53.0   \n",
       "\n",
       "   other_payment_plans  housing  existing_credits  job  num_dependents  \\\n",
       "0                  2.0      1.0               2.0  2.0             1.0   \n",
       "1                  2.0      1.0               1.0  2.0             1.0   \n",
       "2                  2.0      1.0               1.0  1.0             2.0   \n",
       "3                  2.0      2.0               1.0  2.0             2.0   \n",
       "4                  2.0      2.0               2.0  2.0             2.0   \n",
       "\n",
       "   own_telephone  foreign_worker  \n",
       "0            1.0             0.0  \n",
       "1            0.0             0.0  \n",
       "2            0.0             0.0  \n",
       "3            0.0             0.0  \n",
       "4            0.0             0.0  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.489911Z",
     "start_time": "2023-05-18T12:27:57.486938Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fHxKB3j9xE9Q",
    "outputId": "1ac03d31-b495-4b9b-b6a9-02864b12f978"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    1\n",
       "2    0\n",
       "3    0\n",
       "4    1\n",
       "Name: class, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.496442Z",
     "start_time": "2023-05-18T12:27:57.491056Z"
    },
    "id": "i_yvdFRLxa32"
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VflFBDa7zLKY"
   },
   "source": [
    "## Train Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.528256Z",
     "start_time": "2023-05-18T12:27:57.497565Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 74
    },
    "id": "NbFFrBm7zdAW",
    "outputId": "bc0246c0-669e-4a7a-c37a-3a69fff9264f"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a logistic regression model\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.537349Z",
     "start_time": "2023-05-18T12:27:57.530712Z"
    },
    "id": "pZfEuSWfzhES"
   },
   "outputs": [],
   "source": [
    "# make point predictions and obtain class scores\n",
    "y_pred_test = model.predict(X_test)\n",
    "y_pred_score_test = model.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.544901Z",
     "start_time": "2023-05-18T12:27:57.539229Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YiZb5DSAzq_S",
    "outputId": "1f58c598-32fe-48a2-cce6-e22cddfff562"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification accuracy: 0.78\n"
     ]
    }
   ],
   "source": [
    "print('Classification accuracy: {}'.format(accuracy_score(y_test, y_pred_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.552606Z",
     "start_time": "2023-05-18T12:27:57.546851Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aLCDuZky0YUE",
    "outputId": "9f8ec84b-ae9f-4a8b-b071-03d2cced0920"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROC AUC: 0.7923809523809523\n"
     ]
    }
   ],
   "source": [
    "print('ROC AUC: {}'.format(roc_auc_score(y_test, y_pred_score_test[:,1]))) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.557867Z",
     "start_time": "2023-05-18T12:27:57.554294Z"
    },
    "id": "u3ZDLLEk1vqy"
   },
   "outputs": [],
   "source": [
    "# predict classification scores on the training set\n",
    "y_pred_score_train = model.predict_proba(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.564061Z",
     "start_time": "2023-05-18T12:27:57.559400Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NwykNdnc1xV2",
    "outputId": "adce954c-f043-43da-ed12-ec9217133243"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.55639106, 0.44360894],\n",
       "       [0.82854616, 0.17145384],\n",
       "       [0.51564753, 0.48435247],\n",
       "       ...,\n",
       "       [0.86327331, 0.13672669],\n",
       "       [0.79113368, 0.20886632],\n",
       "       [0.87150336, 0.12849664]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_score_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.571393Z",
     "start_time": "2023-05-18T12:27:57.565992Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7jMkQnDg10I9",
    "outputId": "32b9750a-9a9c-4bbc-f79a-3b15a7908a5d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "828    1\n",
       "997    0\n",
       "148    0\n",
       "735    0\n",
       "130    0\n",
       "      ..\n",
       "492    0\n",
       "545    1\n",
       "298    0\n",
       "417    0\n",
       "749    0\n",
       "Name: class, Length: 800, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.577212Z",
     "start_time": "2023-05-18T12:27:57.573043Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kTiaxcZY2PKW",
    "outputId": "321b3fbf-beb9-4252-cd7a-942ccebbfb0b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "checking_status              1.0\n",
       "duration                    18.0\n",
       "credit_history               2.0\n",
       "purpose                      9.0\n",
       "credit_amount             1913.0\n",
       "savings_status               3.0\n",
       "employment                   1.0\n",
       "installment_commitment       3.0\n",
       "personal_status              3.0\n",
       "other_parties                0.0\n",
       "residence_since              3.0\n",
       "property_magnitude           0.0\n",
       "age                         36.0\n",
       "other_payment_plans          0.0\n",
       "housing                      1.0\n",
       "existing_credits             1.0\n",
       "job                          2.0\n",
       "num_dependents               1.0\n",
       "own_telephone                1.0\n",
       "foreign_worker               0.0\n",
       "Name: 30, dtype: float32"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = X_test.iloc[0]\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.582084Z",
     "start_time": "2023-05-18T12:27:57.578487Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "euf5A1I71-rc",
    "outputId": "0cdaab28-1a7f-4e4c-dadb-4772fd7755f7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = y_test.iloc[0]\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.587689Z",
     "start_time": "2023-05-18T12:27:57.584202Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0QGV7-cT9Upb",
    "outputId": "2d3e45e6-f402-4d87-bb85-ccb879853781"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.79837622, 0.20162378])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_score_test[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PHl251J_3QVR"
   },
   "source": [
    "# Define non-conformity measure\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.592336Z",
     "start_time": "2023-05-18T12:27:57.589763Z"
    },
    "id": "p84vDmiG4OnD"
   },
   "outputs": [],
   "source": [
    "# We use inverse probability nonconformity measure (also known as hinge loss). The function below calculates the probability of not predicting the correct class by simply looking up\n",
    "# probability score predicted by the underlying model for the correct class and subtracting it from 1.\n",
    "# For each correct output in y, inverse probablity nonconformity (hinge loss) score  is defined as math:: 1 - \\hat{P}(y_i | x)\n",
    "# This measure is also known as a hinge loss and is based simply on the probability estimate provided by the underlying point classification model for the correct class label y_i \n",
    "\n",
    "def InverseProbabilityNC(predicted_score, y):\n",
    "  prob = np.zeros(y.size, dtype=np.float32)\n",
    "  for i, y_ in enumerate(y):\n",
    "    if y_ >= predicted_score.shape[1]:\n",
    "      prob[i] = 0\n",
    "    else:\n",
    "      prob[i] = predicted_score[i, int(y_)]\n",
    "  return 1 - prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.616222Z",
     "start_time": "2023-05-18T12:27:57.593829Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 483
    },
    "id": "om2hMa-N1HEG",
    "outputId": "871ad7c8-d5fc-4563-86ed-42ae69290f89"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>checking_status</th>\n",
       "      <th>duration</th>\n",
       "      <th>credit_history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>credit_amount</th>\n",
       "      <th>savings_status</th>\n",
       "      <th>employment</th>\n",
       "      <th>installment_commitment</th>\n",
       "      <th>personal_status</th>\n",
       "      <th>other_parties</th>\n",
       "      <th>residence_since</th>\n",
       "      <th>property_magnitude</th>\n",
       "      <th>age</th>\n",
       "      <th>other_payment_plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>existing_credits</th>\n",
       "      <th>job</th>\n",
       "      <th>num_dependents</th>\n",
       "      <th>own_telephone</th>\n",
       "      <th>foreign_worker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>828</th>\n",
       "      <td>0.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8335.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>47.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>3.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>804.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>0.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5371.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>735</th>\n",
       "      <td>1.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3990.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>1.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8487.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>492</th>\n",
       "      <td>3.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1237.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>545</th>\n",
       "      <td>0.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1333.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>298</th>\n",
       "      <td>3.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2515.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>417</th>\n",
       "      <td>0.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>8471.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>749</th>\n",
       "      <td>3.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3029.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>800 rows × 20 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     checking_status  duration  credit_history  purpose  credit_amount  \\\n",
       "828              0.0      36.0             2.0      1.0         8335.0   \n",
       "997              3.0      12.0             2.0      3.0          804.0   \n",
       "148              0.0      36.0             4.0      2.0         5371.0   \n",
       "735              1.0      36.0             1.0      4.0         3990.0   \n",
       "130              1.0      48.0             2.0      0.0         8487.0   \n",
       "..               ...       ...             ...      ...            ...   \n",
       "492              3.0       6.0             4.0      3.0         1237.0   \n",
       "545              0.0      24.0             3.0      0.0         1333.0   \n",
       "298              3.0      18.0             2.0      2.0         2515.0   \n",
       "417              0.0      18.0             3.0      6.0         8471.0   \n",
       "749              3.0      15.0             2.0      1.0         3029.0   \n",
       "\n",
       "     savings_status  employment  installment_commitment  personal_status  \\\n",
       "828             4.0         4.0                     3.0              2.0   \n",
       "997             0.0         4.0                     4.0              2.0   \n",
       "148             0.0         2.0                     3.0              2.0   \n",
       "735             4.0         1.0                     3.0              1.0   \n",
       "130             4.0         3.0                     1.0              1.0   \n",
       "..              ...         ...                     ...              ...   \n",
       "492             1.0         2.0                     1.0              1.0   \n",
       "545             0.0         0.0                     4.0              2.0   \n",
       "298             0.0         2.0                     3.0              2.0   \n",
       "417             4.0         2.0                     1.0              1.0   \n",
       "749             0.0         3.0                     2.0              2.0   \n",
       "\n",
       "     other_parties  residence_since  property_magnitude   age  \\\n",
       "828            0.0              4.0                 3.0  47.0   \n",
       "997            0.0              4.0                 2.0  38.0   \n",
       "148            2.0              2.0                 1.0  28.0   \n",
       "735            0.0              2.0                 3.0  29.0   \n",
       "130            0.0              2.0                 2.0  24.0   \n",
       "..             ...              ...                 ...   ...   \n",
       "492            0.0              1.0                 1.0  27.0   \n",
       "545            0.0              2.0                 0.0  43.0   \n",
       "298            0.0              4.0                 0.0  43.0   \n",
       "417            0.0              2.0                 2.0  23.0   \n",
       "749            0.0              2.0                 2.0  33.0   \n",
       "\n",
       "     other_payment_plans  housing  existing_credits  job  num_dependents  \\\n",
       "828                  2.0      2.0               1.0  2.0             1.0   \n",
       "997                  2.0      1.0               1.0  2.0             1.0   \n",
       "148                  2.0      1.0               2.0  2.0             1.0   \n",
       "735                  0.0      1.0               1.0  0.0             1.0   \n",
       "130                  2.0      1.0               1.0  2.0             1.0   \n",
       "..                   ...      ...               ...  ...             ...   \n",
       "492                  2.0      1.0               2.0  2.0             1.0   \n",
       "545                  2.0      2.0               2.0  2.0             2.0   \n",
       "298                  2.0      1.0               1.0  2.0             1.0   \n",
       "417                  2.0      0.0               2.0  2.0             1.0   \n",
       "749                  2.0      1.0               1.0  2.0             1.0   \n",
       "\n",
       "     own_telephone  foreign_worker  \n",
       "828            0.0             0.0  \n",
       "997            0.0             0.0  \n",
       "148            0.0             0.0  \n",
       "735            0.0             0.0  \n",
       "130            0.0             0.0  \n",
       "..             ...             ...  \n",
       "492            0.0             0.0  \n",
       "545            0.0             0.0  \n",
       "298            1.0             0.0  \n",
       "417            1.0             0.0  \n",
       "749            0.0             0.0  \n",
       "\n",
       "[800 rows x 20 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.630349Z",
     "start_time": "2023-05-18T12:27:57.617775Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 144
    },
    "id": "h5nyrrfa02wP",
    "outputId": "d2839398-d27a-4413-ee41-cd7655ac5d9d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>checking_status</th>\n",
       "      <th>duration</th>\n",
       "      <th>credit_history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>credit_amount</th>\n",
       "      <th>savings_status</th>\n",
       "      <th>employment</th>\n",
       "      <th>installment_commitment</th>\n",
       "      <th>personal_status</th>\n",
       "      <th>other_parties</th>\n",
       "      <th>residence_since</th>\n",
       "      <th>property_magnitude</th>\n",
       "      <th>age</th>\n",
       "      <th>other_payment_plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>existing_credits</th>\n",
       "      <th>job</th>\n",
       "      <th>num_dependents</th>\n",
       "      <th>own_telephone</th>\n",
       "      <th>foreign_worker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>1.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1913.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    checking_status  duration  credit_history  purpose  credit_amount  \\\n",
       "30              1.0      18.0             2.0      9.0         1913.0   \n",
       "\n",
       "    savings_status  employment  installment_commitment  personal_status  \\\n",
       "30             3.0         1.0                     3.0              3.0   \n",
       "\n",
       "    other_parties  residence_since  property_magnitude   age  \\\n",
       "30            0.0              3.0                 0.0  36.0   \n",
       "\n",
       "    other_payment_plans  housing  existing_credits  job  num_dependents  \\\n",
       "30                  0.0      1.0               1.0  2.0             1.0   \n",
       "\n",
       "    own_telephone  foreign_worker  \n",
       "30            1.0             0.0  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let us see how Transductive Conformal Prediction works using the first point from the test dataset\n",
    "pd.DataFrame(X_test.iloc[0]).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.657618Z",
     "start_time": "2023-05-18T12:27:57.632794Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 483
    },
    "id": "6MbNfG4z1k35",
    "outputId": "db467413-b26c-4599-c4c1-f6958258b9c6"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>checking_status</th>\n",
       "      <th>duration</th>\n",
       "      <th>credit_history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>credit_amount</th>\n",
       "      <th>savings_status</th>\n",
       "      <th>employment</th>\n",
       "      <th>installment_commitment</th>\n",
       "      <th>personal_status</th>\n",
       "      <th>other_parties</th>\n",
       "      <th>residence_since</th>\n",
       "      <th>property_magnitude</th>\n",
       "      <th>age</th>\n",
       "      <th>other_payment_plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>existing_credits</th>\n",
       "      <th>job</th>\n",
       "      <th>num_dependents</th>\n",
       "      <th>own_telephone</th>\n",
       "      <th>foreign_worker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>828</th>\n",
       "      <td>0.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8335.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>47.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>3.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>804.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>0.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5371.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>735</th>\n",
       "      <td>1.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3990.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>1.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8487.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>545</th>\n",
       "      <td>0.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1333.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>298</th>\n",
       "      <td>3.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2515.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>417</th>\n",
       "      <td>0.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>8471.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>749</th>\n",
       "      <td>3.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3029.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>1.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1913.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>801 rows × 20 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     checking_status  duration  credit_history  purpose  credit_amount  \\\n",
       "828              0.0      36.0             2.0      1.0         8335.0   \n",
       "997              3.0      12.0             2.0      3.0          804.0   \n",
       "148              0.0      36.0             4.0      2.0         5371.0   \n",
       "735              1.0      36.0             1.0      4.0         3990.0   \n",
       "130              1.0      48.0             2.0      0.0         8487.0   \n",
       "..               ...       ...             ...      ...            ...   \n",
       "545              0.0      24.0             3.0      0.0         1333.0   \n",
       "298              3.0      18.0             2.0      2.0         2515.0   \n",
       "417              0.0      18.0             3.0      6.0         8471.0   \n",
       "749              3.0      15.0             2.0      1.0         3029.0   \n",
       "30               1.0      18.0             2.0      9.0         1913.0   \n",
       "\n",
       "     savings_status  employment  installment_commitment  personal_status  \\\n",
       "828             4.0         4.0                     3.0              2.0   \n",
       "997             0.0         4.0                     4.0              2.0   \n",
       "148             0.0         2.0                     3.0              2.0   \n",
       "735             4.0         1.0                     3.0              1.0   \n",
       "130             4.0         3.0                     1.0              1.0   \n",
       "..              ...         ...                     ...              ...   \n",
       "545             0.0         0.0                     4.0              2.0   \n",
       "298             0.0         2.0                     3.0              2.0   \n",
       "417             4.0         2.0                     1.0              1.0   \n",
       "749             0.0         3.0                     2.0              2.0   \n",
       "30              3.0         1.0                     3.0              3.0   \n",
       "\n",
       "     other_parties  residence_since  property_magnitude   age  \\\n",
       "828            0.0              4.0                 3.0  47.0   \n",
       "997            0.0              4.0                 2.0  38.0   \n",
       "148            2.0              2.0                 1.0  28.0   \n",
       "735            0.0              2.0                 3.0  29.0   \n",
       "130            0.0              2.0                 2.0  24.0   \n",
       "..             ...              ...                 ...   ...   \n",
       "545            0.0              2.0                 0.0  43.0   \n",
       "298            0.0              4.0                 0.0  43.0   \n",
       "417            0.0              2.0                 2.0  23.0   \n",
       "749            0.0              2.0                 2.0  33.0   \n",
       "30             0.0              3.0                 0.0  36.0   \n",
       "\n",
       "     other_payment_plans  housing  existing_credits  job  num_dependents  \\\n",
       "828                  2.0      2.0               1.0  2.0             1.0   \n",
       "997                  2.0      1.0               1.0  2.0             1.0   \n",
       "148                  2.0      1.0               2.0  2.0             1.0   \n",
       "735                  0.0      1.0               1.0  0.0             1.0   \n",
       "130                  2.0      1.0               1.0  2.0             1.0   \n",
       "..                   ...      ...               ...  ...             ...   \n",
       "545                  2.0      2.0               2.0  2.0             2.0   \n",
       "298                  2.0      1.0               1.0  2.0             1.0   \n",
       "417                  2.0      0.0               2.0  2.0             1.0   \n",
       "749                  2.0      1.0               1.0  2.0             1.0   \n",
       "30                   0.0      1.0               1.0  2.0             1.0   \n",
       "\n",
       "     own_telephone  foreign_worker  \n",
       "828            0.0             0.0  \n",
       "997            0.0             0.0  \n",
       "148            0.0             0.0  \n",
       "735            0.0             0.0  \n",
       "130            0.0             0.0  \n",
       "..             ...             ...  \n",
       "545            0.0             0.0  \n",
       "298            1.0             0.0  \n",
       "417            1.0             0.0  \n",
       "749            0.0             0.0  \n",
       "30             1.0             0.0  \n",
       "\n",
       "[801 rows x 20 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here, we concatenate the first point from the test dataset to the training data set \n",
    "X_train_plus_test = pd.concat([X_train,pd.DataFrame(X_test.iloc[0]).T])\n",
    "X_train_plus_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.663005Z",
     "start_time": "2023-05-18T12:27:57.659250Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XoCWsHht2KVt",
    "outputId": "5dcc8bc2-9267-49b0-a195-a10c997c6989"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.55639106, 0.44360894],\n",
       "       [0.82854616, 0.17145384],\n",
       "       [0.51564753, 0.48435247],\n",
       "       ...,\n",
       "       [0.86327331, 0.13672669],\n",
       "       [0.79113368, 0.20886632],\n",
       "       [0.87150336, 0.12849664]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_score_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.666804Z",
     "start_time": "2023-05-18T12:27:57.664286Z"
    },
    "id": "Jndqiprk3D3H"
   },
   "outputs": [],
   "source": [
    "# we create two datasets for potential labels for the first point from the test dataset\n",
    "y_train_plus_test_0 = np.append(y_train, 0)\n",
    "y_train_plus_test_1 = np.append(y_train, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.689119Z",
     "start_time": "2023-05-18T12:27:57.668220Z"
    },
    "id": "ta9wE2Bz2vZ4"
   },
   "outputs": [],
   "source": [
    "# We fit the classifier twice using both potential labels 0 & 1. Here we fit the classifier using the potential label 0 and predict class scores for the new object if it had label 0\n",
    "model.fit(X_train_plus_test, y_train_plus_test_0)\n",
    "y_pred_score_train_plus_test_0 = model.predict_proba(X_train_plus_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.716300Z",
     "start_time": "2023-05-18T12:27:57.690497Z"
    },
    "id": "x7kKxJAQ3zFE"
   },
   "outputs": [],
   "source": [
    "# We fit the classifier twice using both potential labels 0 & 1. Here we fit the classifier using the potential label 1 and predict class scores for the new object if it had label 1\n",
    "model.fit(X_train_plus_test, y_train_plus_test_1)\n",
    "y_pred_score_train_plus_test_1 = model.predict_proba(X_train_plus_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.720635Z",
     "start_time": "2023-05-18T12:27:57.717443Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "oxkvWH4q3UqT",
    "outputId": "5eb7fcbc-5ece-46a6-d0aa-950199944fdc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.55647673, 0.44352327],\n",
       "       [0.83101381, 0.16898619],\n",
       "       [0.51542769, 0.48457231],\n",
       "       ...,\n",
       "       [0.79119243, 0.20880757],\n",
       "       [0.87240832, 0.12759168],\n",
       "       [0.80173622, 0.19826378]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_score_train_plus_test_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.725284Z",
     "start_time": "2023-05-18T12:27:57.722248Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vF8w9eT92Lb8",
    "outputId": "0b38d277-d121-4e05-d6fb-172de0a008ad"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.55699827, 0.44300173],\n",
       "       [0.83167967, 0.16832033],\n",
       "       [0.51420573, 0.48579427],\n",
       "       ...,\n",
       "       [0.79335568, 0.20664432],\n",
       "       [0.87496455, 0.12503545],\n",
       "       [0.78278472, 0.21721528]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_score_train_plus_test_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.729455Z",
     "start_time": "2023-05-18T12:27:57.726451Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F8bvEmA94Jim",
    "outputId": "50c9f7db-969c-4201-b1fa-ec5ad7dc5f44"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.80173622, 0.19826378])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_score_train_plus_test_0[-1,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.735194Z",
     "start_time": "2023-05-18T12:27:57.731096Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "b6tT8t84TDhY",
    "outputId": "c555b834-8a71-4a45-ac39-9b3c49d6a14e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.78278472, 0.21721528])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_score_train_plus_test_1[-1,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.748206Z",
     "start_time": "2023-05-18T12:27:57.745088Z"
    },
    "id": "dY1Ievhp17Sp"
   },
   "outputs": [],
   "source": [
    "# we calculate nonconformity scores for both situations - potential labels 0 & 1 \n",
    "non_conformity_scores_train_plus_test_0 = InverseProbabilityNC(y_pred_score_train_plus_test_0, y_train_plus_test_0)\n",
    "non_conformity_scores_train_plus_test_1 = InverseProbabilityNC(y_pred_score_train_plus_test_1, y_train_plus_test_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.752850Z",
     "start_time": "2023-05-18T12:27:57.749595Z"
    },
    "id": "til732fL4eCp"
   },
   "outputs": [],
   "source": [
    "# extract nonconformity scores for the training set and the new test point\n",
    "non_conformity_scores_y_train_class_0 = non_conformity_scores_train_plus_test_0[:-1]\n",
    "non_conformity_scores_y_train_class_1 = non_conformity_scores_train_plus_test_1[:-1]\n",
    "\n",
    "non_conformity_scores_y_test_class_0 = non_conformity_scores_train_plus_test_0[-1]\n",
    "non_conformity_scores_y_test_class_1 = non_conformity_scores_train_plus_test_1[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.758719Z",
     "start_time": "2023-05-18T12:27:57.754566Z"
    },
    "id": "mwgfZ9pp5PM_"
   },
   "outputs": [],
   "source": [
    "# function computes p-values for the test set using classical definition as per Vovk's book by essentially comparing the nonconformity score of the new test point\n",
    "# with the nonconformity scores of the points in the training set and computing ratio (with test point added to the same 'bag' as training points) of the # of training points\n",
    "# that have nonconformity scores greater than equal to that of the new test point\n",
    "  \n",
    "def compute_p_value(non_conformity_scores_train, non_conformity_scores_test):\n",
    "  p_values = []\n",
    "  if not isinstance(non_conformity_scores_test, list):\n",
    "    non_conformity_scores_test = [non_conformity_scores_test]\n",
    "\n",
    "  for i, test_point_conformity_score in enumerate(non_conformity_scores_test):\n",
    "    p_value = (np.sum(non_conformity_scores_train >= test_point_conformity_score) + 1) / (len(non_conformity_scores_train)+1)\n",
    "    p_values.append(p_value)  \n",
    "  return np.array(p_values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.763438Z",
     "start_time": "2023-05-18T12:27:57.761064Z"
    },
    "id": "1ZKkarcA5e2m"
   },
   "outputs": [],
   "source": [
    "# compute p-value of the test point twice, each time assigning potential label 0 and 1\n",
    "p_value_class_0 = compute_p_value(non_conformity_scores_y_train_class_0, non_conformity_scores_y_test_class_0)\n",
    "p_value_class_1 = compute_p_value(non_conformity_scores_y_train_class_1, non_conformity_scores_y_test_class_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.769251Z",
     "start_time": "2023-05-18T12:27:57.765307Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "S8ydKZdH7z6h",
    "outputId": "8524771d-10d6-430d-946c-8a00c7d33c63"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "p-value for class 0: [0.5980025], p-value for class 1: [0.0536829]\n"
     ]
    }
   ],
   "source": [
    "print('p-value for class 0: {}, p-value for class 1: {}'.format(p_value_class_0,p_value_class_1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:57.774210Z",
     "start_time": "2023-05-18T12:27:57.771294Z"
    },
    "id": "1EmGYyC0FiUi"
   },
   "outputs": [],
   "source": [
    "# Alternative way to compute p-values using ranks of nonconformity scores\n",
    "\n",
    "# def compute_p_value(non_conformity_scores_train, non_conformity_scores_test):\n",
    "#   p_values = []\n",
    "#   if not isinstance(non_conformity_scores_test, list):\n",
    "#     non_conformity_scores_test = [non_conformity_scores_test]\n",
    "\n",
    "#   # Step 1 - sort non_conformity_scores_train in descending order\n",
    "#   A_sorted = np.sort(non_conformity_scores_train)[::-1]\n",
    "#   # Step 2 - calculate the rank R of the test instance's nonconformity score, the rank is position in the sorted sequence where nonconformity score alpha for the test object will be inserted\n",
    "#   for i,_ in enumerate(non_conformity_scores_test):\n",
    "#     A_sorted_append = np.append(A_sorted, non_conformity_scores_test[i])\n",
    "#     R = rankdata(A_sorted_append)[-1]\n",
    "#   # Step 3 - p-value is computed using rank R as: p_value = R / (n+1), where n is the length of the training set\n",
    "#     p_value = R / (len(non_conformity_scores_train)+1)\n",
    "#     p_values.append(p_value)  \n",
    "#   return np.array(p_values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:58.021947Z",
     "start_time": "2023-05-18T12:27:57.776047Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 560
    },
    "id": "FEecsRfm-zyM",
    "outputId": "fb8f36a7-8a55-4b3b-99e4-a8c411a1d282"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAygAAAIgCAYAAABu0E0oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAxOAAAMTgF/d4wjAABXbUlEQVR4nO3deXhN5/7+8XsnIUFMjVAEEUIaScRwas5BtdoYqoaWIlRKFU17aqhS7eHk0GrFGFVDjdUqpQ51TlrzXBQt1ZKQmIpoKoaSyLB+f/jaP5FgR7OzF3m/rssla61nP+uzdx676+6zBothGIYAAAAAwAScHF0AAAAAANxEQAEAAABgGgQUAAAAAKZBQAEAAABgGgQUAAAAAKZBQAEAAABgGgQUAAAAAKZBQAHwUOvdu7d69OiRp30mJCTIYrEoLi5OkjRv3jx5eXnl6T5y2o+jjB07VmXLlpXFYtHGjRsdWos9Oep9muX3fDuLxaK1a9c6ugwABRABBcADqXnz5rJYLLJYLCpSpIiqVaum3r1768cff8zSbvLkyYqOjr5nf+np6TYfmFaqVElnzpxR1apV77f8bGbPni1vb2+77ye3jh8/rnfeeUczZ87UmTNn1LhxY4fVYk+OfJ+3/55zGgsAUJAQUAA8sN544w2dOXNGhw8f1pw5c5SWlqa//e1vWrVqlbVNyZIlVbJkyTzbZ2pqqpydnfXoo4/K2dk5z/rNSX7t527i4+NlGIaeffZZPfrooypcuLDDarGnvHifqamp97VvM/yeAcBMCCgAHljFihXTo48+qsqVK6t58+b67LPPFBYWpldffVVpaWmSsp/iNWnSJFWtWlWurq7y8vLSP//5T0lS9erVJUktWrSQxWJR7969Jd2YqRkyZIj69u2rEiVKaPDgwXc8JWfhwoWqVKmS3N3d1bdvX12/ft267fbTZW7tY+PGjerbt6+OHz9unRXauHFjjvtZsGCBqlevLldXVwUGBuq///2vddvGjRtlsVi0bt06+fv7q3jx4urQoYMuXLhw18/xww8/VKVKleTq6qqGDRtq165dkm6cutaiRQtJkpOTkywWS46v/+c//6mmTZtq2rRpKl++vMqUKaNhw4bJMAxrm9jYWD311FMqUqSIypYtq6FDhyo9Pd263dvbWxMmTFCXLl1UrFgxPfbYY1q/fn2W/Xz33Xdq0KCB3NzcVK5cOQ0YMCBP+r/T+/zzzz/18ssvq3Tp0nJ3d1enTp107tw5a5+9e/dW9+7dNXz4cJUpU0adO3e2/g6+/fZb+fv7q2jRonr++eeVkpKiadOmqUKFCipbtqzGjx+fq7Hw2GOPafr06Vk+jzlz5qhq1apZPuebnn322SyfjyTt2bNHLi4uOnfunFJTUxUWFqZKlSqpWLFiqlevXrbP+1Y5ncZ48/d+U0ZGhkaNGiUvLy8VL15czZs3108//WTdvnfvXjVt2lTFihVT6dKl9fe//13Jycl33CeAgouAAuCh8tprr+n06dPau3dvtm27d+/We++9pxkzZig2NlZffvmlNZjs3LlTkvTVV1/pzJkzmjx5svV1n3zyiapVq6a9e/dq8ODBOe43KSlJc+bM0erVq7VixQp98803Gjt2rE01N27cWBMmTJCXl5fOnDlzx1OMtm/frj59+igiIkI//fSTnnvuOXXo0EEJCQlZ2kVGRmrevHnasGGDDhw4oMjIyDvue/HixfrnP/+p999/X/v371dQUJBCQ0N16dIlvfDCC/ryyy8lyVrXnfz000/avXu31q9fr9mzZ2vSpElavXq1pBsHrs8++6xcXV21a9cuzZ8/XwsWLMhykC5J48ePV7t27bR//341a9ZMPXr0sIa8Q4cOqU2bNmrVqpX27dun//73v3rsscfypP87vc9//OMf2rRpk1auXKnNmzfr9OnT6tmzZ5Y+V65cqWvXrmnbtm2Kioqyrh83bpwWLFigb7/9VuvXr1f79u21b98+rV+/XuPHj9dbb72V5eD9pjuNhV69emnhwoVZ2i5cuFA9e/bMMTh27dpVy5YtU0ZGhnXdl19+qebNm6tcuXJKT09XjRo1tGrVKv34449q3769nn32WSUmJt7xd3wvo0eP1po1a/T5559r3759atKkiZ588kldunRJktSjRw81adJEBw4c0NatW9W9e/f73heAh5wBAA+gv//978bIkSOzrU9JSTEkGV988YVhGIbRq1cvo3v37oZhGMayZcuMGjVqGGlpadlel5aWZkgyNmzYkG0/zZs3z7IuPj7ekGTExsYahmEYc+fONSQZv/zyi7XNrFmzDA8PD+uyJOO77767Yx+zZs0yqlSpctf9vPDCC0aXLl2ytGnQoIExZMgQwzAMY8OGDYYk4/vvv7duHzt2rFGvXr1s7/fW1w8dOjTL5+Dl5WVMmzbNMAzD+O6774x7/afivffeM0qXLm1cu3bNuu6pp54yBg8ebBiGYfz3v/813NzcjKSkJOv2jz/+2ChTpox1uUqVKsarr75qXf7tt98MScaBAwcMwzCMsLAwo02bNjnuPy/6v/19Xrp0yXBxcTG++eYb67pffvnFkGQcPHjQMIwbY8vHx8fIyMiwtsnpd/DKK68YjzzyiJGSkmJdV7NmTWPKlCmGYdg2Fk6fPm04OzsbR44cMQzDMBISEgwnJyfra2535coVo2jRolnGnLe3tzFr1qwc29+saf78+dblW8fs3LlzjYoVK2Zp/9577xlNmjQxDMMwrl27ZhQpUsT6ed7k6+trLFy40DAMw3B3dzc2b958x/0DwE3MoAB4qBj/d7pLTv9XuVWrVrJYLKpWrZr69++vb775JsfTY25Xp06de7YpXry4/Pz8rMuPP/64kpKSlJSUlIvq7+7w4cNq2LBhlnWNGjXS4cOHs6wLDAy0/vzoo4/e9f+K396ni4uL6tevn63Pe/H19ZWbm1uO+z18+LB8fX31yCOPZKn7999/1x9//HHHuiVZ+zh48KCaN29+x/fwV/u/3bFjx5Senp7ls/Hz81OpUqWyfDa1a9eWk1P2/5Teuq9y5cpZT8u7dd358+dz3HdOKlSooFatWllnURYtWqQGDRpYZwBvV6xYMbVp00ZLliyRJO3atUunT59Wx44drW0++ugjBQUF6ZFHHpG7u7tiY2N18uRJm2u61dGjR3Xt2jU1bNhQ7u7u1j9Hjx7VsWPHJEmDBg3SU089pQ4dOig6Olq///77fe0LwMOPgALgofLrr79KUo53QSpZsqR++uknffzxxypcuLD69OmjZ5999p59Fi1a9J5t7nR9xq3bbw1DN6+RyQ1bwpQkFSpUKMt+MzMzc72v3Lp1n7fv937rlmRTH3nR//32eaexcfu+7vb52Kp3795atGiRDMPQwoUL1atXr7u2f+GFF7R8+XKlpaVpyZIlevLJJ60hbtGiRRozZowGDx6sDRs2aP/+/fL397/juHRycsr2mdza9sqVK5JuXAe1f/9+65/Dhw9r0KBBkm6c9rZ79241bNhQCxcuVM2aNRUbG5urzwBAwUBAAfBQmTp1qipVqqS6devmuL1w4cIKDQ3VlClTtGrVKq1atUqJiYlydnaWk5NTlnP2c+PSpUtZ/s/67t275eHhIQ8PD0mSp6enzp49a91+4MCBLK8vVKjQPfft5+dnvVbmph07dmSZucmtmjVrZukzPT1de/bs+Ut93s7Pz0+xsbFZZjN27NghT0/PLLMedxMYGHjHW0DnRf+3q1atmlxcXLJ8Nr/++quSk5Pz9LPJyZ3Gws0bHkyYMEEJCQl64YUX7tpPmzZtlJaWpu+++07Lli3L0n7nzp1q2bKlevXqpdq1a+vRRx/ViRMn7tiXp6enkpKSsoSSW8fwY489psKFC+vMmTOqXr16lj+3/g4CAgI0fPhw7dy5U48++qhWrFhh02cCoGBxcXQBAHC//vzzT509e1bXr1/XsWPHNGvWLC1btkwrVqyQi0v2r7fVq1fr+PHjCgkJUbFixbRkyRKVKVNGHh4eslgsqlSpktavX6/AwEAVLVpU7u7uNtfi5uamV199VRMnTtTvv/+u9957TwMHDrRuDwkJ0eTJk1WnTh0lJSVlu3C9SpUqOnfunPbs2SNvb+8cb40cERGhkJAQTZs2TU899ZQWLVqkffv26YsvvsjFp5bV66+/rr59+yo4OFh169ZVVFSUrl27lqcPt3zqqadUtWpV9e7dW2PHjtXJkyf13nvv6Y033rC5j7feekvBwcEaOXKkevbsqdTUVG3ZssV62tBf7f92xYsXV58+ffTGG2+oePHiKlasmAYMGKAnn3xS/v7+992vLXIaC4UKFZKbm5u6du2qESNG6Nlnn1WpUqXu2o+bm5vat2+vt956S+fOnVOHDh2s26pVq6Yvv/xSW7Zs0SOPPKJ33333rjM6f/vb3+Tk5KQxY8aoV69eWrNmjTZv3qyAgABJUokSJTRo0CC9+uqrun79uurWrauzZ89q1apV6t69u3x8fPTWW2+pS5cuqly5sn7++WedOHFCNWvWzIuPDMBDhhkUAA+sSZMmqXz58qpRo4b69OmjQoUKaffu3QoNDc2xfalSpbRkyRI1a9ZMQUFB2rVrl1avXm19/sT48eP12WefqXz58tbTUmzl4eGhsLAwhYaGqn379mrdurVGjBhh3T5hwgQVL15cDRo00Jtvvmm9vfFNISEh6tq1q1q1aiVPT09t27Yt2z4aN26sTz/9VJMmTVJAQIBWrFihr7/++i891K9bt2567733NGzYMNWuXVs//fST1qxZoxIlStx3n7dzcnKy3u3qb3/7m3r16qWwsDANGzbM5j78/f21atUq/e9//1Pt2rXVunVr64xVXvSfkwkTJqhZs2Zq166dQkJCVLFixWx30rKHu42FXr16KS0tTWFhYTb11bVrVx08eFDPPPNMlt9p//799cQTTyg0NFRPPvmkmjVrptq1a9+xnzJlymju3LlatGiRgoOD9eOPP+rVV1/N0ubDDz/UgAEDNGTIENWsWVPPP/+8Tp48KQ8PDzk7OysxMVHdunVTjRo1NGjQIL377rs2nWIJoOCxGLaeaAsAABxq+fLlevXVV3X69OkcZwkB4GHAtxsAACaXmpqq48eP6/3331d4eDjhBMBDjVO8AAAwuc8//1z+/v5yc3PT8OHDHV0OANgVp3gBAAAAMA1mUAAAAACYBgEFAAAAgGk8dFfZubq6ytPTM9/2l5qaKldX13zbHx4OjBvcD8YN7hdjB/eDcYP7kdO4OX/+vFJTU23u46ELKJ6enjp16lS+7S8mJkatW7fOt/3h4cC4wf1g3OB+MXZwPxg3uB85jRsvL69c9cEpXgAAAABMg4ACAAAAwDQIKAAAAABM46G7BgUAAOBODMOw/sG9ZWRkOLoEPAAsFoucnPJu3oOAAgAAHnqZmZlKTExUcnIy4cRGnp6eOnLkiKPLwAOiUKFCqly5cp70RUABAAAPvePHj8vJyUne3t4qVKiQo8t5IFy6dEklSpRwdBl4ABiGoaSkJJ04cSJP+iOgAACAh1pmZqZSUlLk6+srFxcOfWzl5OQkZ2dnR5eBB4SHh4f++OOPPOmLi+QBAMBD7eYpXRaLxcGVAA+vvPz3RUABAAAAYBrMcwIAgALp7eUH7NLvuI6BNrVLT0/X2LFjtXjxYjk7OysjI0MhISEaP3689u/fryFDhmjPnj12qfGmOXPm6P3331dmZqaeeOIJTZ8+ndPg4HDMoAAAADhAeHi4du/erR07dujnn3/WoUOH9OSTT+bZefz3Eh8fr1GjRmnr1q2Ki4vT2bNnNWfOnHzZN3A3BBQAAIB8FhcXp6VLl2ru3LkqXbq0pBsXpXfp0kU+Pj5Z2qanp6t169aqX7++atWqpe7du+vq1auSpJ07d6pevXoKDg5WQECAPv74Y0nS7Nmz5e/vr+DgYAUGBur777/PVsOyZcv03HPPqVy5crJYLOrfv78+//xzO79z4N6YwwMAAMhne/fula+vr8qUKXPPts7Ozlq8eLE8PDxkGIYGDBig6dOna8iQIRo3bpwGDx6sF198UZJ04cIFSdLgwYP1yy+/qEKFCkpLS1Nqamq2fk+cOKEqVapYl729vfPsNrHAX0FAAQAAMDHDMDRx4kR98803Sk9P18WLFxUSEiJJatGihSIjIxUXF6eWLVuqadOmkqSWLVsqLCxM7dq10zPPPKMaNWrk2Petd17iAZYwC07xAgAAyGd169ZVbGyskpKS7tl28eLF2rRpkzZv3qwDBw5oyJAhSklJkSS98cYbWr16tcqXL68RI0ZowIABkqTly5fr/fffV1pamkJDQ/XFF19k67dy5cpKSEiwLh8/fjzPngQO/BUEFAAAgHxWvXp1derUSeHh4UpOTpZ0YwZjwYIFOnr0aJa2Fy5ckIeHh4oXL67Lly9r3rx51m2HDx+Wj4+P+vbtqxEjRmjnzp1KT0/X0aNHVb9+fQ0ZMkSdO3fWrl27stXQqVMnrVixQufOnZNhGJoxY4a6du1qz7cN2IRTvAAAQIFk6+2A7eXTTz9VZGSkGjRoIBcXFxmGoZCQELVv314nT560tgsLC9PKlSvl7++vihUrqlmzZjp9+rQkaerUqdqwYYMKFy4sZ2dnTZgwQRkZGXrppZd04cIFubi4yNPTU3Pnzs22fx8fH40ePVpNmjRRZmamWrZsqfDw8Hx7/8CdEFAAAAAcoFChQho9erRGjx6dbVvz5s2tz0ApWbKk1q5dm2Mf06ZNy3H9li1bbKqhb9++6tu3r40VA/nD7qd4RUREyNvbWxaLRQcPHrSub9y4sYKDg623xbNYLPrpp58kSb1795aXl5d1+9ChQ+1dJgAAAAATsPsMSufOnTVs2DDrXSVu2r59u/XnZcuWafTo0QoKCrKuGz58uAYNGmTv8gAAAACYiN0Dys3b4N3Np59+yjmPAAAAABx/F6/Tp09r48aN6tGjR5b1UVFRCgoKUtu2bbV//37HFAcAAAAgXzn8Ivl58+apbdu2WZ6k+u9//1vly5eXk5OTVqxYoWeeeUaxsbFyd3fP9vqoqChFRUVZl5OTkxUTE5MvtUtSSkpKvu4PD7Z+P/aTJE2pOYVxg1zj+wb3i7EjeXp66tKlS3Jycvj/m3UI9+PHJUlXbnly/L0YhqGLFy/aqyQ8ZDIzM3Xt2jWlpqb+5e8bhwYUwzA0d+5cRUdHZ1lfsWJF68/PPfechg8frsOHD6tevXrZ+njzzTf15ptvWpe9vLzUunVr+xV9m5iYmHzdHx5s7kdvhGw3NzfGDXKN7xvcr4I+djIyMnTkyBGVKFFCzs7Oji7HMf4vmJUsWdLml1y8eDFX7VGwZWRkqEiRInJ1df3L3zcODSibNm3S9evX9eSTT2ZZf+rUKXl5eUmSdu7cqaSkJFWvXt0RJQIAgIfVqtft02+7yTY1S09P19ixY7V48WI5OzsrIyNDISEhGj9+vPbv368hQ4ZYbzVsDwkJCerdu7f27dsnX19fu+4LyA27B5SBAwdq5cqVOnv2rFq1aiV3d3fFxcVJkubMmaOXXnop23Rr7969de7cOTk7O6tIkSJaunQpCR4AADxUwsPD9ccff2jHjh0qXbq0MjMz9dVXX+mPP/7Il/2XKFFCkZGRunjxot5777182SdgC7sHlOjo6GyncN20cOHCHNff6WFEAAAAD4O4uDgtXbpUJ06cUOnSpSVJTk5O6tKliyTpxIkT1rbp6elq06aNkpKSdO3aNQUHB2vWrFkqWrSodu7cqYEDByojI0Pp6ekaOHCgXn31Vc2ePVtRUVEqXLiwMjIyNPvtt9Xglsc5SNIjjzyipk2bauPGjfn2vgFbOPwieQAAgIJm79698vX1zXKToDtxdnbW4sWL5eHhIcMwNGDAAE2fPl1DhgzRuHHjNHjwYL344ouSpAsXLkiSBg8erF9++UUVKlRQWlqaUvfts+v7AfISAQUAAMDEDMPQxIkT9c033yg9PV0XL160PmeuRYsWioyMVFxcnFq2bGl9MHbLli0VFhamdu3a6ZlnnlGNokUd+RaAXCmY99oDAABwoLp16yo2NlZJSUn3bLt48WJt2rRJmzdv1oEDBzRkyBClpKRIkt544w2tXr1a5cuX14gRIzRgwABJ0vLly/X+++8rLS1NoaGh+uK//7Xr+wHyEgEFAAAgn1WvXl2dOnVSeHi4kpOTJd2YKVmwYIGOHj2ape2FCxfk4eGh4sWL6/Lly5o3b5512+HDh+Xj46O+fftqxIgR2rlzp9LT03X06FHVr19fQ4YMUefOnbXrwIF8fHfAX8MpXgAAAA7w6aefKjIyUg0aNJCLi4sMw1BISIjat2+vkydPWtuFhYVp5cqV8vf3V8WKFdWsWTOdPn1akjR16lRt2LBBhQsXlrOzsyZMmKCMjAy99NJLunDhglxcXOTp6am5b72Vbf+pqamqVq2aUlNTdfHiRXl5ealnz54aN25cvn0GQE4IKAAAoGCy8Xkl9lKoUCGNHj1ao0ePzratefPm1ueSlCxZ8o53OJ02bVqO67ds2ZJ1xcGD2dq4urrq1KlTuawasD9O8QIAAABgGgQUAAAAAKbBKV528PZyLkS71biOgY4uAQAAAA8IZlAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAcwNvbW35+fkpPT7euq1+/vjZu3Oi4onJw9OhR1a1bV3Xq1NHcuXPttp/Q0FAdPXpUkjRv3jwdOXLEbvuyt507dyowMFB16tRRTEzMX+orPT1dY8aMkZ+fn2rVqiU/Pz/169dPycnJ2rhxo+rXr59HVd/ZnDlz5Ovrq2rVqqlfv35Zxqw9EFAAAAAcJDU1VXPmzHF0GXe1bNkyNWrUSPv27dNLL71k8+tyexC7Zs0aVatWTZL5Akpu38v8+fMVFhamffv2qXXr1n9pP+Hh4dq9e7d27Nihn3/+WYcOHdKTTz6pP/74I1c13a/4+HiNGjVKW7duVVxcnM6ePWv3MUtAAQAAcJDRo0frX//6l65evZpt27lz5/Tcc88pMDBQAQEBmjlzpnWbt7e3Ro8ercaNG6tq1aqKjIy0bjt9+rQ6d+6soKAgBQUFadSoUTf6+/33XPe3YMECTZw4UUuXLlVwcLAOHTqkuLg4tWrVSkFBQQoODtbXX39t7cdisWjChAlq3ry53n77bfXu3Vv9+/fXE088oSpVquj111/Xhg0bFBISIm9vb0VFRWWp4eDBg5o9e7b27NmjiIgIBQcHa82aNQoMDNSOHTusbT/55BO98MIL2T6z2NhYNWnSRLVr11ZgYKDeeecdSdL169c1dOhQBQYGqnbt2nr66aclSRkZGRoyZIgCAgIUEBCg1157TdevX5ck9e7dWxEREXr66adVu3ZtSdLChQvVoEED1a1bV3//+9918ODBbDW8//77WrJkiSZPnqzg4GAlJydrz549atSokYKCgvT4449r27ZtkqSEhASVKVNGY8aMUbNmzTR16tQsfcXFxWnp0qWaO3euSpcuLUlycnJSly5d5OPjk6Vtenq6Wrdurfr166tWrVrq3r27dVzt3LlT9erVU3BwsAICAvTxxx9LkmbPni1/f38FBwcrMDBQ33//fbb3s2zZMj333HMqV66cLBaL+vfvr88//zxbu7zEc1AAAECBVGt6Lbv0+/OAn21uW7duXYWEhGjixIkaOXJklm0RERHy8/PTihUrlJiYaD3AfPzxxyVJycnJ2r59u86fP6/q1avrpZdeUsWKFdWjRw+FhoZq2bJlkqTz589L584p4v335Venjs39derUSWFhYTp27JiuXLmijz76SJLUoEEDhYeHq1+/foqNjVXDhg1Vr149VapUSdKNWaGbp6n17t1bBw8e1Lp165SRkSFvb29dvnxZGzdu1JkzZ1SzZk3169dP7u7u1vf98ssva9GiRRoyZIjatm0r6Uboio6OVqNGjSRJ0dHRio6OzvZ5Tps2TW3atNGIESMkyTrLMG7cOB09elR79uyRq6vrjc9E0syZM/XDDz/ohx9+kLOzs9q3b6/Jkydr6NChkqStW7dq8+bNcnd317Zt2/TFF19o8+bNcnV11ZYtW9S9e3f9+OOPWWoYPny4fv31V9WvX1+DBg3S9evX1bFjR82aNUutW7fW1q1b1blzZ8XFxUmSkpKSVL16db377rvZ3s/evXvl6+urMmXK3GMkSc7Ozlq8eLE8PDxkGIYGDBig6dOna8iQIRo3bpwGDx6sF198UZJ04cIFSdLgwYP1yy+/qEKFCkpLS1Nqamq2fk+cOKEqVapYl729vXXixIl71vNXMIMCAADgQJGRkZo0aZKSkpKyrF+7dq0GDhwoSSpbtqw6duyodevWWbd3795dkuTp6SkfHx/Fx8frypUr2r59u/7xj39Y23l6et7ob+fOXPV3/PjxbLVevnxZ+/fvV3h4uCTJ19dXTZs21datW61t+vTpk+U1HTp0kKurq4oWLaqaNWsqNDRUTk5OqlixokqXLq1Tp07d8zPq0aOH1q9fr8TERG3atEkWi0XNmjXL1i4kJESzZ8/WyJEj9e2336pUqVKSpNWrV+uNN96Qq6tr1s9k7VqFh4fL1dVVLi4u6tu3r9auXWvt7/nnn7eGp5UrV+rHH39UgwYNFBwcrNdee03nz5+3zrjcyeHDh1W4cGHrqV5NmzZV2bJl9dNPP0mS3Nzc1K1bt3t+BvdiGIYmTpyoOnXqKCgoSN988432798vSWrRooUiIyM1ZswYbd261Tob07JlS4WFhWny5MmKj4/PEhRvZbFYsuzH3phBAQAABVJuZjrsycfHR926dctymtZNtx4Y3r7s5uZm/dnZ2dmm6yT+an83D07v1s/tB7m393s/dRcpUkS9evXS7NmztW/fPg0aNCjHdp06dVLjxo313Xffadq0aZo0aZLWrFlzx34Nw7D5vRiGoT59+mjMmDH3rPde+7h1P8WKFctxu3Rjhi02NlZJSUny8PC4634WL16sTZs2afPmzSpevLimTJmizZs3S5LeeOMNtW/fXuvWrdOIESMUEBCg6dOna/ny5frhhx+0ceNGhYaGKjIyUl27ds3Sb+XKlZWQkGBdPn78uCpXrpybjyDXCCh20uHUeEeXYCILHV0AAACmNmrUKPn7+6tQoULWda1atdLMmTM1evRonT9/XitWrLCetnUn7u7uatq0qSZOnGg9Ten8+fPylNSqYcNc93e7EiVKKDg4WPPnz9dLL72ko0ePatu2bZo2bVqu3/O99nPx4sUs6wYOHKhGjRopNTVV8+fPz/F1sbGx8vHxUVhYmB5//HE1btxYktS+fXtNmjRJDRo0sJ7i5enpqSeffFLz5s1Tly5d5OTkpDlz5qhVq1Y59t2uXTuFhYWpb9++qlSpkjIzM7V379573kXLz89PqampWr9+vVq2bKnt27crMTFRgYGB1lPN7uTmqXbh4eGaN2+eSpUqJcMwtHDhQjVp0iRL2wsXLsjDw0PFixfX5cuXNW/ePOt1KocPH1bNmjXl4+OjSpUqacSIEUpPT1dCQoLq16+v+vXr6/fff9euXbuyBZROnTqpadOmevfdd1W2bFnNmDEjW5u8RkABAABwME9PT0VERGS5DmHKlCnq37+/goKClJmZqZEjR1qvF7mbhQsX6rXXXlOtWrXk4uKiDh06aHSXLpoyfLj6T56c6/5u99lnn+mVV17RpEmTZLFYNHv2bOv1J3mlX79+Gjx4sD788EONHTtWoaGh8vLyUnBwsGrUqKGiRYvm+LqlS5fqs88+U+HChWUYhmbMmCFJeuuttzRy5EjVqVNHhQsXVoUKFbRmzRr169fPehtlSWrevLkiIiJy7DskJERjx47Vs88+q4yMDKWlpalNmzb3DCiFCxfWV199pYiICP35559yc3PT0qVLVaxYsXsGFEn69NNPFRkZqQYNGsjFxUWGYSgkJETt27fXyZMnre3CwsK0cuVK+fv7q2LFimrWrJlOnz4tSZo6dao2bNigwoULy9nZWRMmTFBGRoZeeuklXbhwQS4uLvL09MzxNtI+Pj4aPXq0mjRposzMTLVs2dJ6ip+9WIz8OJEsH3l5edl0LmNeiYmJyXb7uLeXH2AG5RYNIphBuenmBZlR1aJyddtBQMr5+wawRUEfOxkZGTpy5Ihq1KghZ2dnR5fjGDfvNhUQYPNLLl68qJIlS9qpoNy7cuWK/Pz8tGXLFlWtWtXR5eA2N/+dnThxItv3TW6Pz7lIHgAAAKY2Y8YM+fn5acCAAYSTAoBTvAAAAGBq/fv3V//+/R1dBvIJMygAAAAATIOAAgAAAMA0CCgAAAAATIOAAgAAAMA0CCgAAAAO4O3tLT8/vyxPUq9fv742btzouKJycPM5IXXq1MnxORl5JTQ0VEePHpUkzZs3T0eOHLHbvuxt586dCgwMVJ06dRQTE/OX+kpPT9eYMWPk5+enWrVqyc/PT/369VNycrI2btx4z+ew/FUJCQlq3ry5SpYsafd93URAAQAAcJDU1FTNmTPH0WXc1bJly9SoUSPt27dPL730ks2vuzV42WLNmjWqVq2aJPMFlNy+l/nz5yssLEz79u3L1TOIctpPeHi4du/erR07dujnn3/WoUOH9OSTT+qPP/7IVU33q0SJEoqMjNTixYvzZX8SAQUAAMBhRo8erX/961+6evVqtm3nzp3Tc889p8DAQAUEBGjmzJnWbd7e3ho9erQaN26sqlWrKjIy0rrt9OnT6ty5s4KCghQUFKRRo0bd6O/333Pd34IFCzRx4kQtXbpUwcHBOnTokOLi4tSqVSsFBQUpODhYX3/9tbUfi8WiCRMmqHnz5nr77bfVu3dv9e/fX0888YSqVKmi119/XRs2bFBISIi8vb0VFRWVpYaDBw9q9uzZ2rNnjyIiIhQcHKw1a9YoMDBQO3bssLb95JNP9MILL2T7zGJjY9WkSRPVrl1bgYGBeueddyRJ169f19ChQxUYGKjatWvr6aeflnTj4YJDhgxRQECAAgIC9Nprr+n69euSpN69eysiIkJPP/20ateuLUlauHChGjRooLp16+rvf/+7Dt58AOYt3n//fS1ZskSTJ09WcHCwkpOTtWfPHjVq1EhBQUF6/PHHtW3bNkk3ZifKlCmjMWPGqFmzZpo6dWqWvuLi4rR06VLNnTtXpUuXliQ5OTmpS5cu8vHxydI2PT1drVu3Vv369VWrVi11797dOq527typevXqKTg4WAEBAfr4448lSbNnz5a/v7+Cg4MVGBio77//Ptv7eeSRR9S0aVMVK1Ys2zZ74TkoAACgYKpVyz79/vyzzU3r1q2rkJAQTZw4USNHjsyyLSIiQn5+flqxYoUSExOtB5iPP/64JCk5OVnbt2/X+fPnVb16db300kuqWLGievToodDQUC1btkySdP78eencOUW8/7786tSxub9OnTopLCxMx44d05UrV/TRRx9Jkho0aKDw8HD169dPsbGxatiwoerVq6dKlSpJujErdPM0td69e+vgwYNat26dMjIy5O3trcuXL2vjxo06c+aMatasqX79+snd3d36vl9++WUtWrRIQ4YMUdu2bSXdCF3R0dFq1KiRJCk6OlrR0dHZPs9p06apTZs2GjFihCRZZxnGjRuno0ePas+ePXJ1db3xmUiaOXOmfvjhB/3www9ydnZW+/btNXnyZA0dOlSStHXrVm3evFnu7u7atm2bvvjiC23evFmurq7asmWLunfvrh9//DFLDcOHD9evv/6q+vXra9CgQbp+/bo6duyoWbNmqXXr1tq6das6d+6suLg4SVJSUpKqV6+ud999N9v72bt3r3x9fVWmTJl7jCTJ2dlZixcvloeHhwzD0IABAzR9+nQNGTJE48aN0+DBg/Xiiy9Kki5cuCBJGjx4sH755RdVqFBBaWlpSk1Nved+8gMzKAAAAA4UGRmpSZMmKSkpKcv6tWvXauDAgZKksmXLqmPHjlq3bp11e/fu3SVJnp6e8vHxUXx8vK5cuaLt27frH//4h7Wdp6fnjf527sxVf8ePH89W6+XLl7V//36Fh4dLknx9fdW0aVNt3brV2qZPnz5ZXtOhQwe5urqqaNGiqlmzpkJDQ+Xk5KSKFSuqdOnSOnXq1D0/ox49emj9+vVKTEzUpk2bZLFY1KxZs2ztQkJCNHv2bI0cOVLffvutSpUqJUlavXq13njjDbm6umb9TNauVXh4uFxdXeXi4qK+fftq7dq11v6ef/55a3hauXKlfvzxRzVo0EDBwcF67bXXdP78eeuMy50cPnxYhQsXtp7q1bRpU5UtW1Y//fSTJMnNzU3dunW752dwL4ZhaOLEiapTp46CgoL0zTffaP/+/ZKkFi1aKDIyUmPGjNHWrVutszEtW7ZUWFiYJk+erPj4+CxB0ZGYQQEAAAVTLmY67MnHx0fdunXLcprWTRaL5Y7Lbm5u1p+dnZ1tuk7ir/ZnGMY9+7n9IPf2fu+n7iJFiqhXr16aPXu29u3bp0GDBuXYrlOnTmrcuLG+++47TZs2TZMmTdKaNWvu2K9hGDa/F8Mw1KdPH40ZM+ae9d5rH7fup1ixYjlul27MsMXGxiopKUkeHh533c/ixYu1adMmbd68WcWLF9eUKVO0efNmSdIbb7yh9u3ba926dRoxYoQCAgI0ffp0LV++XD/88IM2btyo0NBQRUZGqmvXrrl6f/bADAoAAICDjRo1SosWLdJvv/1mXdeqVSvrdSLnz5/XihUr1LJly7v24+7urqZNm2rixInWdTdPZ2rVsGGu+7tdiRIlFBwcrPnz50u6cYevbdu2qUmTJrnqx5b9XLx4Mcu6gQMH6uOPP9amTZussz23i42NVdmyZRUWFqbx48dr586dkqT27dtr0qRJ1lOYbn4mTz75pObNm6fr168rPT1dc+bMUatWrXLsu127dlqwYIFOnjwpScrMzNSePXvu+V78/PyUmpqq9evXS5K2b9+uxMREBQYG3vO1N0+1Cw8PV3JysqQbgWfBggXWO57ddOHCBXl4eKh48eK6fPmy5s2bZ912+PBh+fj4qG/fvhoxYoR27typ9PR0HT16VPXr19eQIUPUuXNn7dq165415QdmUAAAABzM09NTERERWa5DmDJlivr376+goCBlZmZq5MiR1utF7mbhwoV67bXXVKtWLbm4uKhDhw4a3aWLpgwfrv6TJ+e6v9t99tlneuWVVzRp0iRZLBbNnj3bev1JXunXr58GDx6sDz/8UGPHjlVoaKi8vLwUHBysGjVqqGjRojm+bunSpfrss89UuHBhGYahGTNmSJLeeustjRw5UnXq1FHhwoVVoUIFrVmzRv369bPeRlmSmjdvroiIiBz7DgkJ0dixY/Xss88qIyNDaWlpatOmzT1vvVu4cGF99dVXioiI0J9//ik3NzctXbpUxYoVswalu/n0008VGRmpBg0ayMXFRYZhKCQkRO3bt7eGJUkKCwvTypUr5e/vr4oVK6pZs2Y6ffq0JGnq1KnasGGDChcuLGdnZ02YMEEZGRl66aWXdOHCBbm4uMjT0zPH20inpqaqWrVqSk1N1cWLF+Xl5aWePXtq3Lhx96z9flmMm3N1DwkvLy+bzmXMKzExMdluH/f28gPqcGp8vtVgdg0iFjq6BNOoNf3GBZlR1aJyddtBQMr5+wawRUEfOxkZGTpy5Ihq1KghZ2dnR5fjGDfvNhUQYPNLLl68qJIlS9qpoNy7cuWK/Pz8tGXLFlWtWtXR5eA2N/+dnThxItv3TW6PzznFCwAAAKY2Y8YM+fn5acCAAYSTAoBTvAAAAGBq/fv3V//+/R1dBvIJMygAAAAATIOAAgAAHmo3b+H6kF12Czy0CCgAAOCh5uTkJGdnZ6WkpDi6FOChlZaWdsfnueQW16AAAICHnqenp06fPq2KFSvKzc0tzw6kHhg3Z48yMmx+SWZmpjJy0R4Fl2EYOnfunEqVKqXjx4//5f4IKAAA4KFXunRpSdJvv/1WMA+6ExNv/O1i+6HftWvXVKRIETsVhIeNm5ubypYtmyd9EVAAAECBULp0aZUuXVqZmZkF73qUF1+88bcNTz6/ae3atXd8qjpwK4vFIienvLtyhIACAAAKlLw8kHpgXL9+4+9cPqiywD7YEg5VAP+FAgAAADArAgoAAAAA0yCgAAAAADANAgoAAAAA0yCgAAAAADANAgoAAAAA0yCgAAAAADANAgoAAAAA0yCgAAAAADANAgoAAAAA0yCgAAAAADANF0cXgIff28sPOLoE00i8lKqyJVwdXQYAAIBp2X0GJSIiQt7e3rJYLDp48KB1ffPmzeXj46Pg4GAFBwdr4sSJ1m1Xr15Vt27dVL16ddWoUUPLly+3d5kAAAAATMDuMyidO3fWsGHD1LRp02zbpkyZorZt22Zb/9FHH8nV1VVxcXGKj49Xo0aN1KJFC5UuXdre5QIAAABwILvPoISEhMjLyytXr1myZIkGDhwoSapatapCQkK0cuVKe5QHAAAAwEQcepH80KFDFRgYqBdeeEHHjh2zrj9x4oSqVKliXfb29taJEyccUSIAAACAfOSwi+QXLlyoSpUqyTAMRUdHq23btjp06JB1u8Visf5sGMYd+4mKilJUVJR1OTk5WTExMfYpOgcpKSnZ9lfHKU2JFUPzrQaza2wcvHejAuIzpejKlbQcxw1wL4wb3C/GDppcuSJJ2paLccC4wf3Ii3HjsIBSqVIlSTeCyKBBgzRkyBAlJSXJw8NDlStXVkJCgjw9PSVJx48fV2hozgf8b775pt58803rspeXl1q3bm3/N/B/YmJisu3v7eUH1OH0+HyrAQ8Ol7QrKuJRUW5ubvk6TvFwyOn7BrAFYwdyd5ekXI0Dxg3uR16MG4ec4pWenq5z585Zl7/66iuVK1dOHh4ekqQuXbooOjpakhQfH69Nmzapffv2jigVAAAAQD6y+wzKwIEDtXLlSp09e1atWrWSu7u7fvzxR7Vp00apqalycnJSmTJl9J///Mf6mqFDh6pPnz6qXr26nJycFB0drUceecTepQIAAABwMLsHlOjoaOtsyK327Nlzx9cUK1ZMS5YssWdZAAAAAEzIoXfxAgAAAIBbEVAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmAYBBQAAAIBpEFAAAAAAmIbdA0pERIS8vb1lsVh08OBB6/o+ffqoZs2aCg4OVkhIiPbv32/d1rt3b3l5eSk4OFjBwcEaOnSovcsEAAAAYAIu9t5B586dNWzYMDVt2jTL+g4dOmjmzJlycXHR6tWr9fzzz+vIkSPW7cOHD9egQYPsXR4AAAAAE7F7QAkJCclxffv27a0/N2zYUMePH1dmZqacnDjrDAAAACioTJEGJk+erNDQ0CzhJCoqSkFBQWrbtm2W078AAAAAPLzsPoNyL4sWLdKXX36pLVu2WNf9+9//Vvny5eXk5KQVK1bomWeeUWxsrNzd3bO9PioqSlFRUdbl5ORkxcTE5EvtkpSSkpJtf3Wc0pRYMTTfasCDI/3qBl25fEkpKdcUs3KJo8txPLdSjq7ggZLT9w1gC8YOmly5IknalotxwLjB/ciLcWMxDMPIo3ruytvbW6tXr1ZAQIB13ZIlS/TOO+9o3bp1qly58h1fW7NmTS1evFj16tW75368vLx06tSpPKnZFjExMWrdunWWdW8vP6AOp8bnWw14cHS7uEpFCjsrym+6Wjttd3Q5jtdusqMreKDk9H0D2IKxA9WqdePvn3+2+SWMG9yPnMZNbo/PHXaK15dffql33nlHa9euzRZObn0DO3fuVFJSkqpXr57fJQIAAADIZ3Y/xWvgwIFauXKlzp49q1atWsnd3V1xcXHq3r27Hn30UT377LPWtuvWrZOHh4d69+6tc+fOydnZWUWKFNHSpUtVsmRJe5cKAAAAwMHsHlCio6MVHR2dbX1aWtodX7N27Vp7lgQAAADApBx+kTxQ0Fy7nqHr6Zn6/vQfji7F4Ro4ugAAAGA6prjNMAAAAABIBBQAAAAAJkJAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApmFTQMnIyNDkyZM1aNAgSdLRo0e1fv16uxYGAAAAoOBxsaXRa6+9prS0NG3dulWS5OHhoa5du2r37t12LQ4AAABAwWJTQNm+fbv279+vOnXqSJJKlSql69ev27UwAAAAAAWPTad4ubm5ZVnOyMhQZmamXQoCAAAAUHDZFFCCgoL02WefyTAMJSQkaMCAAQoJCbF3bQAAAAAKGJsCSlRUlDZv3qwzZ86oQYMGyszM1AcffGDv2gAAAAAUMDZdg+Lu7q5PPvlEn3zyib3rAQAAAFCA2RRQFixYkOP6sLCwPC0GAAAAQMFmU0BZtWqV9eeUlBRt3bpVDRs2JKAAAAAAyFM2BZSlS5dmWY6Pj9eoUaPsUhAAAACAgsumi+RvV7VqVR0+fDivawEAAABQwNk0g7JmzRrrzxkZGfr+++9lsVjsVhQAAACAgsmmgPLhhx/+/xe4uKhatWpasmSJ3YoCAAAAUDDZFFA2bNhg7zoAAAAA4O4B5dChQ3d9sb+/f54WAwAAAKBgu2tAadOmzR23WSwWHTt2LM8LAgAAAFBw3TWgxMfH51cdAAAAAGDbNSjSjbt3HT9+XCkpKdZ1nOIFAAAAIC/ZFFD++9//6uWXX1ZSUpKKFSum5ORkVa5cmRkWAAAAAHnKpgc1jhw5Utu3b9djjz2mpKQkzZ8/X88//7y9awMAAABQwNgUUCwWi6pUqaL09HRJUo8ePbRjxw67FgYAAACg4LHpFK9ChQpJkry8vLRq1SpVqVJFp06dsmthAAAAAAoemwLK66+/rgsXLigyMlJdu3ZVcnKyJk+ebO/aAAAAABQwNgWUbt26SZLq1aun2NhYuxYEAAAAoOCy6RqUevXqafr06bpw4YK96wEAAABQgNkUUCZMmKBdu3apevXq6tq1q2JiYmQYhr1rAwAAAFDA2BRQmjdvrnnz5un48eN66qmnFBkZqSpVqti7NgAAAAAFjE0B5aarV6/qwoULSk5OVsmSJe1VEwAAAIACyqaAsmLFCrVr104BAQE6evSoPv30Ux04cMDetQEAAAAoYGy6i9f06dPVp08fLVu2TK6urvauCQAAAEABZVNA+e677+xdBwAAAADk7hqU+xERESFvb29ZLBYdPHjQuj4xMVFPP/20fH19FRAQoK1bt1q3Xb16Vd26dVP16tVVo0YNLV++3N5lAgAAADABuweUzp07a+vWrdnu+jV8+HA1bNhQsbGxmjt3rrp376709HRJ0kcffSRXV1fFxcUpJiZGAwYM4BksAAAAQAFg94ASEhIiLy+vbOu//PJLDRw4UJL0t7/9TeXKlbPOoixZssS6rWrVqgoJCdHKlSvtXSoAAAAAB7MpoHTr1k3bt2/Ps50mJSUpMzNTnp6e1nXe3t46ceKEJOnEiRNZZlxu3QYAAADg4WXTRfLNmzfXq6++KicnJw0aNEjdu3eXm5vbX9qxxWLJsnz7k+lv3X63p9ZHRUUpKirKupycnKyYmJi/VFtupKSkZNtfHac0JVYMzbca8OBIv7pBkpTp7MYYkfL13+rDIKfvG8AWjB00uXJFkrQtF+OAcYP7kRfjxqaA8sorr+iVV17Rpk2bFB0drZEjRyosLEwDBw68ryfKe3h4SJLOnz9vnUU5fvy4KleuLEmqXLmyEhISsmwLDc35YO7NN9/Um2++aV328vJS69atc13T/YqJicm2v7eXH1CH0+PzrQY8OFzSbvwHwikjRWVPr3FwNY7XIGKho0t4oOT0fQPYgrEDubtLUq7GAeMG9yMvxk2urkGpVauW6tSpo8KFC+vXX39Vs2bN9MEHH9zXjrt06aLo6GhJ0u7du3X27Fk1bdo027b4+Hht2rRJ7du3v6/9AAAAAHhw2BRQvv/+e/Xs2VPBwcG6cuWKduzYof/85z86dOiQNUjcycCBA+Xl5aVTp06pVatWql69uiTpgw8+0Pbt2+Xr66vevXtr4cKFcnG5MaEzdOhQXbt2TdWrV1fr1q0VHR2tRx555C++VQAAAABmZ9MpXn379tXrr7+uWbNmZbn2xN3dXSNHjrzra6Ojo3MMMeXKldO3336b42uKFSumJUuW2FIaAAAAgIeITTMoH3zwgcLDw7OEk//973+SblyfAgAAAAB5waaAktMsyYgRI/K8GAAAAAAF211P8YqLi9ORI0d06dIlrVnz/+84dPHiRV29etXuxQEAAAAoWO4aULZt26Z58+bp3Llz+vDDD63rS5QooQkTJti9OAAAAAAFy10DSq9evdSrVy/NmTNH4eHh+VUTAAAAgALqrgElPj5eVatWVaNGjXTo0KFs2/39/e1WGAAAAICC564B5bXXXtPq1avVpk2bbNssFouOHTtmt8IAAAAAFDx3DSirV6+WdGMmBQAAAADszaYHNUrS1atXderUKaWnp1vXcYoXAAAAgLxkU0CZOHGi3n33XZUuXVrOzs6SOMULAAAAQN6zKaBMnTpVhw8fVoUKFexdDwAAAIACzKYnyVeqVIlwAgAAAMDubJpBGT16tF5++WWFhobKzc3Nuj40NNRuhQEAAAAoeGwKKF9//bVWrVqlI0eOZLkGhYACAAAAIC/ZFFBWrlyphIQEFSlSxN71AAAAACjAbLoGpVq1aipUqJC9awEAAABQwNk0g+Lr66uWLVuqQ4cOWa5BGTBggN0KAwAAAFDw2BRQUlJSVK1aNR04cMC6zmKx2K0oAAAAAAXTPQNKRkaGypYtqw8++CA/6gEAAABQgN3zGhRnZ2ft2rUrP2oBAAAAUMDZdJF8u3bt9MEHHygxMVFXr161/gEAAACAvGTTNShDhgyRJL399tvWdRaLRRkZGfapCgAAAECBZFNAyczMtHcdAAAAAGBbQJGk06dPa+vWrbJYLGratKkqVKhgz7oAAAAAFEA2XYOycuVK1a5dW59//rkWL16s4OBgrVq1yt61AQAAAChgbJpBGT16tHbu3Knq1atLko4ePaouXbqoXbt2di0OAAAAQMFi0wxKRkaGNZxIUrVq1bguBQAAAECesymglC1bVnPmzJFhGJKk+fPnq0yZMnYtDAAAAEDBY1NAmTFjhmbNmqWiRYuqSJEimjFjhmbOnGnv2gAAAAAUMHe9BmX+/Pnq1auX0tLStHPnTl25ckWGYah48eL5VR8AAACAAuSuMyhTpkyRJL344ouSJHd3d8IJAAAAALu55128XnvtNf32228aNmxYtm3jx4+3S1EAAAAACqa7zqAsXbpUlSpVksViUbFixbL9AQAAAIC8dNcZFB8fHw0bNkzly5dXz54986smAAXE28sPOLoEUxnXMdDRJQAA4HA2PaixZ8+eOnPmjOLj45Wenm5dHxISYrfCAAAAABQ8NgWUf//73/rwww/l4+MjZ2dnSZLFYtGuXbvsWhyAh1uHU1zHdtPXXtmv8wMAoCCyKaB8+umniouL4+GMAAAAAOzKpgc1Pvroo4QTAAAAAHZn0wxK69atNXjwYHXv3l1ubm7W9f7+/nYrDAAAAEDBY1NAmTt3riRp+fLl1nUWi0XHjh2zT1UAAAAACiSbAkp8fLy96wAAAAAA2wKKJO3Zs0fr1q2TxWLRE088oXr16tmzLgAAAAAFkE0Xyc+aNUsdO3bUmTNn9Ntvv6ljx46aPXu2vWsDAAAAUMDYNIMydepU/fDDD/L09JQkjRw5Uk888YRefvlluxYHAAAAoGCxaQZFkjWc3PzZYrHYpSAAAAAABZdNAaV69eoaOXKkfvvtN505c0ajR49WtWrV7F0bAAAAgALGpoAyY8YMHT16VEFBQQoKCtKvv/6qGTNm2Ls2AAAAAAWMTdeglC1bVl988YW9awEAAABQwNk0g/Lee+8pKSnJuvz7779r9OjRdisKAAAAQMFkU0BZuXKlPDw8rMtlypTR119/ba+aAAAAABRQNgUUwzCyrUtLS8vzYgAAAAAUbDYFlBo1aigqKkqGYSgzM1MTJkyQn5+fvWsDAAAAUMDYFFAmT56s1atXq0iRIipWrJj+97//aerUqfauDQAAAEABY9NdvCpUqKD169frzz//lCQVK1bMrkUBAAAAKJhsCiiSdObMGcXHxys9Pd26LiQkxC5FAQAAACiYbAoo//73v/Xhhx/Kx8dHzs7OkiSLxaJdu3bZtTgAAAAABYtNAeXTTz9VXFycypQpY+96AAAAABRgNl0k/+ijjxJOAAAAANidTTMorVu31uDBg9W9e3e5ublZ1/v7+9/3jpOTk9W8eXPr8tWrV3Xs2DElJiaqY8eOOnHihEqUKCFJ6tWrl/7xj3/c974AAAAAPBhsCihz586VJC1fvty6zmKx6NixY/e941KlSmn//v3W5Y8++kibNm3SI488IkmaMmWK2rZte9/9AwAAAHjw2BRQ4uPj7V2H5s6dq3//+9923w8AAAAA87LpGhRJ2rNnjz744AONHz9eP/zwQ54WsWPHDiUlJWWZMRk6dKgCAwP1wgsv/KWZGgAAAAAPDothGMa9Gs2aNUv/+te/1LFjR0nSihUrNGrUKL388st5UkTfvn1VunRpjR8/XpJ08uRJVapUSYZhKDo6WtOnT9ehQ4dyfG1UVJSioqKsy8nJyfrqq6/ypC5bpKSkZLkuR5IuXktTkYzL+VYDHhwDY4dKkqb6TZVTRoqDq4GZXHMurpJFCt21TU7fN4AtGDto0q+fJGnbzJk2v4Zxg/uR07gJDw/XqVOnbO7DpoASFBSkdevWydPTU5J0/vx5PfHEE/rpp59yWXJ2f/75p8qXL69du3bJz88vxzZubm46ffq0PDw87tmfl5dXrj6AvyomJkatW7fOsu7t5QfU4dT4fKsBD45uF1dJkib7f6yyp9c4uBqYyddewzSuY+Bd2+T0fQPYgrED1ap14++ff7b5JYwb3I+cxk1uj89tPsXrZji5+bPFYrF5J3ezdOlSBQUFWcNJenq6zp07Z93+1VdfqVy5cjaFEwAAAAAPNpsukq9evbpGjhypgQMHymKxaObMmapWrVqeFDBnzhyFh4dbl1NTU9WmTRulpqbKyclJZcqU0X/+85882RcAAAAAc7MpoMyYMUMREREKCgqSxWJRq1atNGPGjDwpYMuWLVmWixUrpj179uRJ3wAAAAAeLDYFlLJly+qLL76wdy0AAAAACri7BpTNmzff9cUhISF5WgwAAACAgu2uAWXw4MHZ1lksFv322286c+aMMjIy7FYYAAAAgILnrgFl9+7dWZb/+OMPRUZGatGiRRo9erRdCwMAAABQ8Nh0m+GUlBSNGzdOjz32mDIzM3Xo0CG988479q4NAAAAQAFz14CSmZmpmTNnytfXVz///LN27typSZMmqUyZMvlVHwAAAIAC5K6neAUEBCg1NVXjxo1T3bp1de3aNR06dMi63d/f3+4FAgAKpreXH3B0CaYxrmOgo0sAgHxz14By9epVWSwWjRo1ShaLRYZhWLdZLBYdO3bM7gUCAAAAKDjuGlASEhLyqQwAAAAAsPEieQAAAADIDwQUAAAAAKZx11O8AABwiFWvq8OpPxxdhSl87TXM0SUAQL5iBgUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaXAXLwAwgQ6nxkurHrl7o8zG0qrX86cgAAAchBkUAAAAAKZBQAEAAABgGgQUAAAAAKZBQAEAAABgGgQUAAAAAKZBQAEAAABgGgQUAAAAAKZBQAEAAABgGgQUAAAAAKZBQAEAAABgGgQUAAAAAKZBQAEAAABgGgQUAAAAAKZBQAEAAABgGgQUAAAAAKZBQAEAAABgGgQUAAAAAKbh4ugCAAA3fB//x123X6+Yqe9P370NAAAPOmZQAAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJiGi6MLAAAAd/f28gN/6fV1nNL+ch9mMq5joKNLAGBHDp1B8fb2lp+fn4KDgxUcHKwlS5ZIkhITE/X000/L19dXAQEB2rp1qyPLBAAAAJBPHD6DsmzZMgUEBGRZN3z4cDVs2FD/+9//tHv3bnXu3FlHjx6Vi4vDywUAIF91ODX+L/eRWDFUHU7/9X7M4GuvYY4uAYCdmfKI/8svv1R8fLwk6W9/+5vKlSunrVu3qnnz5o4tDAAAAIBdOTygdO/eXZmZmWrQoIHGjRsnJycnZWZmytPT09rG29tbJ06ccGCVAAAAAPKDQwPK5s2bVblyZaWlpemdd95Rr169tHDhQlksliztDMO4Yx9RUVGKioqyLicnJysmJsZuNd8uJSUl2/7qOKUpsWJovtWAB0f61Q2SpExnN8YIco1xg/v1MI2dOk7HFRPzm6PLeOA0uXJFkrQtF8dIOR3jAPeSF+PGYtzt6D8fnTlzRjVq1NDly5dVrFgxJSQkWGdRHn/8cY0fP96mU7y8vLx06tQpO1f7/8XExKh169ZZ1r29/ECenDOMh0+3i6skSZP9P1bZ02scXA0eNIkVQxk3uC8P09j52msYd/G6H7Vq3fj7559tfklOxzjAveQ0bnJ7fO6wu3j9+eefSk5Oti5//vnnqlOnjiSpS5cuio6OliTt3r1bZ8+eVdOmTR1RJgAAAIB85LBTvM6dO6dOnTopIyNDhmHIx8dHCxYskCR98MEH6tmzp3x9fVW4cGEtXLiQO3gBAAAABYDDjvp9fHy0b9++HLeVK1dO3377bT5XBAAAAMDRHPqgRgAAAAC4FQEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYBgEFAAAAgGkQUAAAAACYhoujCwAAALBVh1PjpVWPOLoMc2g32dEVAHbBDAoAAAAA0yCgAAAAADANTvECAAAPlO/j/3B0Cabw9fIDkqRxHQMdXAmQt5hBAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApkFAAQAAAGAaBBQAAAAApuHi6AIAAACQex1OjZckfT/l3m0D/zgtSTowpafN/V+vGKrvc9Hekb72GiZJGtcx0MGVIC8wgwIAAADANAgoAAAAAEyDgAIAAADANAgoAAAAAEyDgAIAAADANAgoAAAAAEzDYQElJSVFHTp0UI0aNRQcHKynn35aCQkJkqTmzZvLx8dHwcHBCg4O1sSJEx1VJgAAAIB85NDnoPTr10/PPPOMLBaLpk2bpn79+unbb7+VJE2ZMkVt27Z1ZHkAAAAA8pnDZlDc3NwUGhoqi8UiSWrYsKGOHTvmqHIAAAAAmIBprkGZMmWK2rVrZ10eOnSoAgMD9cILLxBcAAAAgALCoad43TR27FjFxsZqxowZkqSFCxeqUqVKMgxD0dHRatu2rQ4dOpTja6OiohQVFWVdTk5OVkxMTL7ULd24lub2/dVxSlNixdB8qwEPjvSrGyRJmc5ujBHkGuMG94uxgwyXG//9yc04eJDGTR2n45KkmJjfHFwJcjo2zi2LYRhGHtVzXz766CN98cUXWrt2rUqVKpVjGzc3N50+fVoeHh737M/Ly0unTp3K4yrvLCYmRq1bt86y7u3lB9Th1Ph8qwEPjm4XV0mSJvt/rLKn1zi4GjxoEiuGMm5wXxg7CBx3478/B95ud4+W/9+DNG6+9homSRrXMdDBlSCnY+PcHp879BSvqKgoff755/ruu++s4SQ9PV3nzp2ztvnqq69Urlw5m8IJAAAAgAebw07xOnXqlAYPHiwfHx+1aNFCkuTq6qr169erTZs2Sk1NlZOTk8qUKaP//Oc/jioTAAAAQD5yWEDx8vLSnc4u27NnTz5XAwAAAMAMTHMXLwAAAAAwxV28AAAAgPt18+ZE309xcCEm8aDfNIAZFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBoEFAAAAACmQUABAAAAYBqmDSixsbFq3LixatSooccff1yHDh1ydEkAAAAA7My0AeWVV15Rv379dOTIEQ0bNkzh4eGOLgkAAACAnZkyoCQmJmrv3r3q0aOHJKlTp06Kj49XQkKCYwsDAAAAYFemDCgnT55UhQoV5OLiIkmyWCyqXLmyTpw44eDKAAAAANiTi6MLuBOLxZJl2TCMHNtFRUUpKirKunz27Fl5eXnZtbZbXblyRe7u7tnWL8y3CvBguTFWel55NcdxA9zNlSurGDe4L4wdyPn/fv/jN9j8EsbNg+zG73lhRP7vOadj4/Pnz+eqD4txpyN/B0pMTJSvr6+SkpLk4uIiwzBUvnx57dy5U97e3o4uLwsvLy+dOnXK0WXgAcO4wf1g3OB+MXZwPxg3uB95MW5MeYpX2bJlVadOHS1atEiS9NVXX8nb29t04QQAAABA3jLtKV6ffPKJevfurbFjx6pEiRKaP3++o0sCAAAAYGemDSg1a9bUjh07HF3GPb355puOLgEPIMYN7gfjBveLsYP7wbjB/ciLcWPKa1AAAAAAFEymvAYFAAAAQMFEQAEAAABgGgQUG8TGxqpx48aqUaOGHn/8cR06dCjHdnPmzJGvr6+qVaumfv36KT09PZ8rhZnYMm7Wr1+vBg0ayN/fXwEBARo5cuQdn/mDgsHW7xtJSklJkb+/v+rXr5+PFcKsbB07Bw4cUPPmzfXYY4+pZs2aWr58eT5XCjOxZdwYhqGhQ4eqVq1aCgoKUosWLRQXF+eAamEWERER8vb2lsVi0cGDB+/Y7r6PjQ3cU4sWLYy5c+cahmEYS5cuNRo2bJitzbFjx4zy5csbZ8+eNTIzM4127doZM2bMyOdKYSa2jJu9e/caR48eNQzDMK5du2Y0adLE+Oyzz/KzTJiMLePmpjfffNPo06ePUa9evXyqDmZmy9j5888/DR8fH2PLli2GYRhGWlqakZiYmJ9lwmRsGTdff/218fjjjxvXr183DMMw/vWvfxldunTJzzJhMps2bTJOnjxpVKlSxThw4ECObf7KsTEzKPeQmJiovXv3qkePHpKkTp06KT4+XgkJCVnaLVu2TM8995zKlSsni8Wi/v376/PPP3dAxTADW8dNnTp15OPjI0lyc3NTcHCwjh07lt/lwiRsHTeStGXLFsXGxqpnz575XCXMyNaxs3jxYjVq1EhNmzaVJLm4uMjT0zO/y4VJ5OY7JzU1VSkpKTIMQ5cuXZKXl1c+VwszCQkJuecY+CvHxgSUezh58qQqVKggF5cbd2S2WCyqXLmyTpw4kaXdiRMnVKVKFeuyt7d3tjYoOGwdN7c6e/asli1bptDQ0PwqEyZj67j5888/9cYbb+jjjz92RJkwIVvHzqFDh+Tm5qa2bdsqODhYYWFhOn/+vCNKhgnYOm7atWunFi1a6NFHH1X58uW1bt06jRkzxhEl4wHyV46NCSg2sFgsWZaNO1wjcGu7O7VBwWHruJGkS5cuqV27dho2bJjq1q1r79JgYraMm6FDh2rgwIGqWLFifpWFB4AtYyctLU0xMTH65JNPtG/fPlWqVEkDBw7MrxJhQraMm7179+rXX3/V6dOn9dtvv+mJJ57QoEGD8qtEPMDu99iYgHIPlSpV0qlTp6wX9RiGoZMnT6py5cpZ2lWuXDnLlOjx48eztUHBYeu4kaTLly/r6aefVvv27XkoVgFn67jZunWrxowZI29vb3Xt2lUHDhxQrVq1HFEyTMLWsVOlShW1aNFCFStWlMViUffu3bVr1y5HlAwTsHXczJs3Ty1atFCpUqXk5OSkXr16acOGDY4oGQ+Qv3JsTEC5h7Jly6pOnTpatGiRJOmrr76St7e3vL29s7Tr1KmTVqxYoXPnzskwDM2YMUNdu3Z1QMUwA1vHzZUrV/T000+rdevWGjVqlAMqhZnYOm5++uknJSQkKCEhQV988YUCAwP1888/O6BimIWtY+f555/X7t27denSJUnS//73P9WuXTu/y4VJ2DpufHx8tG7dOqWlpUmSVq1apYCAgPwuFw+Yv3Rs/Jcu4S8gfv31V6Nhw4aGr6+vUa9ePePgwYOGYRhGeHi4sXLlSmu7mTNnGtWqVTOqVq1qhIeHW+92gYLJlnETGRlpuLi4GLVr17b+iYyMdGTZcDBbv29u2rBhA3fxgmEYto+d+fPnG/7+/kZQUJDxzDPPGCdPnnRUyTABW8ZNSkqK8fLLLxs1a9Y0AgMDjaeeesqIj493YNVwtAEDBhgVK1Y0nJ2djXLlyhnVqlUzDCPvjo0thsHFEgAAAADMgVO8AAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJgGAQUAAACAaRBQAAAAAJjG/wPiNqyOBKcG+QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 960x640 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot distribution of nonconformity score for both situations\n",
    "\n",
    "plt.figure(figsize=(12, 8), dpi=80)\n",
    "plt.hist(np.sort(non_conformity_scores_y_train_class_0), label='Class 0', alpha=0.6)\n",
    "plt.hist(np.sort(non_conformity_scores_y_train_class_1), label='Class 1', alpha=0.6)\n",
    "# single line\n",
    "plt.vlines(x = non_conformity_scores_y_test_class_0, ymin = 0, ymax = 175, colors = 'green',\n",
    "           label = 'Nonconformity score for Class 0')\n",
    "\n",
    "plt.vlines(x = non_conformity_scores_y_test_class_1, ymin = 0, ymax = 175, colors = 'red',\n",
    "           label = 'Nonconformity score for Class 1')\n",
    "\n",
    "plt.grid(True)\n",
    "plt.ylabel('Nonconformity value')\n",
    "plt.title('Distribution of nonconformity values')\n",
    " \n",
    "plt.legend( loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:27:58.030178Z",
     "start_time": "2023-05-18T12:27:58.023954Z"
    },
    "id": "Dt1ey2lID8mE"
   },
   "outputs": [],
   "source": [
    "# Function computes prediction sets - here we essentially collect all the steps done above into one function\n",
    "# Function takes training set, test points, underlying classifier, significance level\n",
    "# Function returns for each test point, prediction sets and two sets of p-values for each potential label 0 & 1 \n",
    "\n",
    "def prediction_sets(X_train, y_train, X_test, y_test, model = LogisticRegression(), significance_level = 0.05):\n",
    "\n",
    "    prediction_sets = []\n",
    "    p_values_class_0 = []\n",
    "    p_values_class_1 = []\n",
    "\n",
    "    for i,y in enumerate(y_test):\n",
    "      # add test point to the training set \n",
    "      X_train_plus_test = pd.concat([X_train,pd.DataFrame(X_test.iloc[i]).T])\n",
    "      y_train_plus_test_0 = np.append(y_train, 0)\n",
    "      y_train_plus_test_1 = np.append(y_train, 1)\n",
    "\n",
    "      # train the underlying classifier twice - two models for each potential value of the lable - 0 and 1\n",
    "      model.fit(X_train_plus_test, y_train_plus_test_0)\n",
    "      y_pred_score_train_plus_test_0 = model.predict_proba(X_train_plus_test)\n",
    "      model.fit(X_train_plus_test, y_train_plus_test_1)\n",
    "      y_pred_score_train_plus_test_1 = model.predict_proba(X_train_plus_test)\n",
    "\n",
    "      #compute nonconformity scores\n",
    "      non_conformity_scores_train_plus_test_0 = InverseProbabilityNC(y_pred_score_train_plus_test_0, y_train_plus_test_0)\n",
    "      non_conformity_scores_train_plus_test_1 = InverseProbabilityNC(y_pred_score_train_plus_test_1, y_train_plus_test_1)\n",
    "\n",
    "      non_conformity_scores_y_train_class_0 = non_conformity_scores_train_plus_test_0[:-1]\n",
    "      non_conformity_scores_y_train_class_1 = non_conformity_scores_train_plus_test_1[:-1]\n",
    "\n",
    "      non_conformity_scores_y_test_class_0 = non_conformity_scores_train_plus_test_0[-1]\n",
    "      non_conformity_scores_y_test_class_1 = non_conformity_scores_train_plus_test_1[-1]\n",
    "\n",
    "      p_value_class_0 = compute_p_value(non_conformity_scores_y_train_class_0, non_conformity_scores_y_test_class_0)\n",
    "      p_value_class_1 = compute_p_value(non_conformity_scores_y_train_class_1, non_conformity_scores_y_test_class_1)\n",
    "\n",
    "      p_values_class_0.append( p_value_class_0)\n",
    "      p_values_class_1.append( p_value_class_1)\n",
    "\n",
    "      if (p_value_class_0 > significance_level) and (p_value_class_1 > significance_level):\n",
    "        prediction_sets.append([True,True])\n",
    "      elif (p_value_class_0 > significance_level) and (p_value_class_1 <= significance_level):\n",
    "        prediction_sets.append([True,False])\n",
    "      elif (p_value_class_0 <= significance_level) and (p_value_class_1 >significance_level):\n",
    "        prediction_sets.append([False,True])\n",
    "      else:  \n",
    "        prediction_sets.append([False,False])\n",
    "\n",
    "    return prediction_sets, p_values_class_0, p_values_class_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:28:06.210447Z",
     "start_time": "2023-05-18T12:27:58.031872Z"
    },
    "id": "78Amyif3JiNo"
   },
   "outputs": [],
   "source": [
    "predictions_test_sets, p_values_test_set_class_0, p_values_test_set_class_1 = prediction_sets(X_train, y_train, X_test, y_test, model = LogisticRegression(), significance_level = 0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:28:06.219558Z",
     "start_time": "2023-05-18T12:28:06.211948Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ja1nGfWSG695",
    "outputId": "7e0c307a-3c1d-47d6-958b-9b5d6e9a3eb2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [False, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [False, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [False, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [False, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [False, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, True],\n",
       " [True, False],\n",
       " [False, True],\n",
       " [True, False],\n",
       " [True, False],\n",
       " [True, True],\n",
       " [True, False]]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions_test_sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:28:06.223847Z",
     "start_time": "2023-05-18T12:28:06.221048Z"
    },
    "id": "s-Ljf11wO5e8"
   },
   "outputs": [],
   "source": [
    "# Let's now take open source library nonconformist https://github.com/donlnz/nonconformist and do the same with just a few lines of code\n",
    "\n",
    "nc = NcFactory.create_nc(model,err_func=InverseProbabilityErrFunc())\t# Create a default nonconformity function, this is the inverse probability (hinge loss) nonconformity measure we used above\n",
    "tcp = TcpClassifier(nc)\t\t\t# Create a transductive conformal classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:28:06.227760Z",
     "start_time": "2023-05-18T12:28:06.225176Z"
    },
    "id": "mkXq7DDcQbk-"
   },
   "outputs": [],
   "source": [
    "# Fit the TCP using the proper training set\n",
    "tcp.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:28:13.081824Z",
     "start_time": "2023-05-18T12:28:06.230651Z"
    },
    "id": "_WcENreXQb-e"
   },
   "outputs": [],
   "source": [
    "# Produce predictions for the test set, with confidence 95%\n",
    "prediction = tcp.predict(X_test.values, significance=0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:28:13.088587Z",
     "start_time": "2023-05-18T12:28:13.083058Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kQBK4EHJR0ui",
    "outputId": "969e52f3-6ce4-4893-a095-74a27dd6624b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[False, True] [False  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[False, True] [False  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[False, True] [False  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[False, True] [False  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[False, True] [False  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n",
      "[False, True] [False  True]\n",
      "[True, False] [ True False]\n",
      "[True, False] [ True False]\n",
      "[True, True] [ True  True]\n",
      "[True, False] [ True False]\n"
     ]
    }
   ],
   "source": [
    "# Here we output prediction sets we constructed manually by writing code from scratch and comparing it with the output from Nonconformist. The results are the same.\n",
    "for i in range(len(predictions_test_sets)):\n",
    "  print(predictions_test_sets[i], prediction[i, :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-18T12:28:13.091992Z",
     "start_time": "2023-05-18T12:28:13.090035Z"
    },
    "id": "y7a0WvR1Agrn"
   },
   "outputs": [],
   "source": [
    "assert (predictions_test_sets, prediction)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}