tutorials/plugins/tutorial_01_preprocessing_plugins.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bad8b783",
   "metadata": {},
   "source": [
    "# Preprocessing Plugins"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bf69d04",
   "metadata": {},
   "source": [
    "Preprocessing datasets is a common requirement for many machine learning estimators. The techniques include:\n",
    " - dimensionality reduction: the process of reducing the dimension of your feature set.\n",
    " - feature scaling: the process of normalizing the range or the shape of the features in the dataset. \n",
    "\n",
    "**AutoPrognosis** provides a set of default preprocessing plugins and can be extended with any number of other plugins."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd370aeb",
   "metadata": {},
   "source": [
    "### Plugins 101\n",
    "\n",
    "Every **AutoPrognosis plugin** must implement the **`Plugin`** interface provided by `autoprognosis/plugins/core/base_plugin.py`.\n",
    "\n",
    "Each **AutoPrognosis preprocessing plugin** must implement the **`PreprocessorPlugin`** interface provided by `autoprognosis/plugins/preprocessors/base.py`\n",
    "\n",
    "__Warning__ : If a plugin doesn't override all the abstract methods, it won't be loaded by the library.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "__API__ : Every preprocessing plugin must implement the following methods:\n",
    "- `name()` - a static method that returns the name of the plugin. e.g., linear_svm, pca, etc.\n",
    "\n",
    "- `subtype()` - a static method that returns the plugin's subtype. e.g., \"dimensionality_reduction\", \"feature_scaling\" etc. It will be used for filtering the plugin in the optimization process.\n",
    "    \n",
    "- `hyperparameter_space()` - a static method that returns the hyperparameters that can be tuned during the optimization. The method will return a list of `skopt.space.Dimension` derived objects.\n",
    "    \n",
    "- `_fit()` - internal implementation, called by the `fit` method.\n",
    "\n",
    "- `_transform()` - internal implementation, called by the `transform` method."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa40d733",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a8a4649",
   "metadata": {},
   "outputs": [],
   "source": [
    "# stdlib\n",
    "import sys\n",
    "import time\n",
    "import warnings\n",
    "\n",
    "# third party\n",
    "from IPython.display import HTML, display\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import metrics\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "import tabulate\n",
    "from tqdm import tqdm\n",
    "import xgboost as xgb\n",
    "\n",
    "if not sys.warnoptions:\n",
    "    warnings.simplefilter(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0696cf7d",
   "metadata": {},
   "source": [
    "### Loading the Preprocessing plugins\n",
    "\n",
    "Make sure that you have installed AutoPrognosis in your workspace.\n",
    "\n",
    "You can do that by running `pip install .` in the root of the repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03adfab0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# autoprognosis absolute\n",
    "from autoprognosis.plugins.preprocessors import PreprocessorPlugin, Preprocessors\n",
    "\n",
    "preprocessors = Preprocessors()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8c4e6cd",
   "metadata": {},
   "source": [
    "### List the existing plugins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adbb6734",
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessors.list_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f222242",
   "metadata": {},
   "source": [
    "### Adding a new Preprocessing plugin\n",
    "\n",
    "By default, AutoPrognosis automatically loads the preprocessing plugins with the pattern `autoprognosis/plugins/preprocessors/plugin_*`. \n",
    "\n",
    "Alternatively, you can call `Preprocessors().add(<name>, <PreprocessorPlugin derived class>)` at runtime.\n",
    "\n",
    "Next, we show how to add a custom preprocessing plugin:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2ed8c5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# third party\n",
    "from sklearn.feature_selection import SelectFpr, chi2\n",
    "\n",
    "custom_select_fpr = \"custom_select_fpr\"\n",
    "\n",
    "\n",
    "class NewPlugin(PreprocessorPlugin):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self._model = SelectFpr(chi2)\n",
    "\n",
    "    @staticmethod\n",
    "    def name():\n",
    "        return custom_select_fpr\n",
    "\n",
    "    @staticmethod\n",
    "    def hyperparameter_space(*args, **kwargs):\n",
    "        return []\n",
    "\n",
    "    @staticmethod\n",
    "    def subtype() -> str:\n",
    "        return \"dimensionality_reduction\"\n",
    "\n",
    "    def _fit(self, *args, **kwargs):\n",
    "        self._model.fit(*args, **kwargs)\n",
    "\n",
    "        return self\n",
    "\n",
    "    def _transform(self, *args, **kwargs):\n",
    "        return self._model.transform(*args, **kwargs)\n",
    "\n",
    "    def save(self) -> bytes:\n",
    "        raise NotImplemented(\"placeholder\")\n",
    "\n",
    "    @classmethod\n",
    "    def load(cls, buff: bytes) -> \"NewPlugin\":\n",
    "        raise NotImplemented(\"placeholder\")\n",
    "\n",
    "\n",
    "preprocessors.add(custom_select_fpr, NewPlugin)\n",
    "\n",
    "assert preprocessors.get(custom_select_fpr) is not None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4719f55d",
   "metadata": {},
   "source": [
    "### List the existing plugins\n",
    "\n",
    "Now we should see the new plugins loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "322c6ea4",
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessors.list()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c425590",
   "metadata": {},
   "source": [
    "## Benchmarks\n",
    "\n",
    "We test the preprocessing plugins using the [Wisconsin Breast Cancer dataset](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic))."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a4965f9",
   "metadata": {},
   "source": [
    "### Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02f1afc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = load_breast_cancer(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0b19b61",
   "metadata": {},
   "source": [
    "### Duration benchmarks\n",
    "\n",
    "__About__ : This step measures the preprocessing duration for each plugin on the dataset. The times are reported in milliseconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df405503",
   "metadata": {},
   "outputs": [],
   "source": [
    "duration = []\n",
    "\n",
    "plugins = preprocessors.list()\n",
    "\n",
    "for plugin in tqdm(plugins):\n",
    "    plugin_duration = [plugin]\n",
    "    ctx = preprocessors.get(plugin)\n",
    "\n",
    "    start = time.time() * 1000\n",
    "    ctx.fit_transform(X, y)\n",
    "\n",
    "    plugin_duration.append(round(time.time() * 1000 - start, 4))\n",
    "\n",
    "    duration.append(plugin_duration)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbb54d72",
   "metadata": {},
   "source": [
    "### Duration(ms) results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3291cf93",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(\n",
    "    HTML(\n",
    "        tabulate.tabulate(duration, headers=[\"Plugin\", \"Duration(ms)\"], tablefmt=\"html\")\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b6de0d4",
   "metadata": {},
   "source": [
    "### Prediction performance using feature processing and XGBoost\n",
    "\n",
    "__Steps__\n",
    " - We run each preprocessing plugin on the dataset.\n",
    " - We train an XGBoost classifier using the processed dataset and report the accuracy,  AUROC, and AURPC metrics on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e99d644",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_metrics(X_train, y_train, X_test, y_test):\n",
    "    xgb_clf = xgb.XGBClassifier(verbosity=0)\n",
    "    xgb_clf = xgb_clf.fit(X_train, y_train)\n",
    "\n",
    "    y_pred = xgb_clf.predict(X_test)\n",
    "\n",
    "    score = xgb_clf.score(X_test, y_test)\n",
    "\n",
    "    fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred)\n",
    "    auroc = metrics.auc(fpr, tpr)\n",
    "\n",
    "    prec, recall, thresholds = metrics.precision_recall_curve(y_test, y_pred)\n",
    "    aurpc = metrics.auc(recall, prec)\n",
    "\n",
    "    return round(score, 4), round(auroc, 4), round(aurpc, 4)\n",
    "\n",
    "\n",
    "metrics_headers = [\"Plugin\", \"Acc score\", \"AUROC\", \"AURPC\"]\n",
    "xgboost_test_score = []\n",
    "\n",
    "xgboost_test_score.append(\n",
    "    [\"original dataset\", *get_metrics(X_train, y_train, X_test, y_test)]\n",
    ")\n",
    "\n",
    "for plugin in plugins:\n",
    "    fproc = preprocessors.get(plugin)\n",
    "\n",
    "    X_train_preprocessed = fproc.fit_transform(X_train.copy(), y_train.copy())\n",
    "    X_test_preprocessed = fproc.transform(X_test.copy())\n",
    "\n",
    "    score, auroc, aurpc = get_metrics(\n",
    "        X_train_preprocessed, y_train, X_test_preprocessed, y_test\n",
    "    )\n",
    "\n",
    "    xgboost_test_score.append([plugin, score, auroc, aurpc])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "943c86c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(\n",
    "    HTML(\n",
    "        tabulate.tabulate(xgboost_test_score, headers=metrics_headers, tablefmt=\"html\")\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3697c0f4",
   "metadata": {},
   "source": [
    "# Congratulations!\n",
    "\n",
    "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the movement towards Machine learning and AI for medicine, you can do so in the following ways!\n",
    "\n",
    "### Star AutoPrognosis on GitHub\n",
    "\n",
    "The easiest way to help our community is just by starring the Repos! This helps raise awareness of the tools we're building.\n",
    "\n",
    "- [Star AutoPrognosis](https://github.com/vanderschaarlab/autoprognosis)\n",
    "- [Star HyperImpute](https://github.com/vanderschaarlab/hyperimpute)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab8c379c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}