notebook_examples/model_evaluation-with-ADSEvaluator.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "@notebook{model_evaluation-with-ADSEvaluator.ipynb,\n",
    "    title: Model Evaluation with ADSEvaluator,\n",
    "    summary: Train and evaluate different types of models: binary classification using an imbalanced dataset, multi-class classification using a synthetically generated dataset consisting of three equally distributed classes, and a regression using a synthetically generated dataset with positive targets.,\n",
    "    developed_on: generalml_p38_cpu_v1,\n",
    "    keywords: model evaluation, binary classification, regression, multi-class classification, imbalanced dataset, synthetic dataset,\n",
    "    license: Universal Permissive License v 1.0\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upgrade Oracle ADS to pick up latest features and maintain compatibility with Oracle Cloud Infrastructure.\n",
    "\n",
    "!pip install -U oracle-ads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oracle Data Science service sample notebook.\n",
    "\n",
    "Copyright (c) 2019, 2022 Oracle, Inc. All rights reserved. Licensed under the [Universal Permissive License v 1.0](https://oss.oracle.com/licenses/upl).\n",
    "\n",
    "---\n",
    "\n",
    "# <font color=\"red\">Model Evaluation with ADSEvaluator</font>\n",
    "<p style=\"margin-left:10%; margin-right:10%;\">by the <font color=\"teal\">Oracle Cloud Infrastructure Data Science Service.</font></p>\n",
    "\n",
    "---\n",
    "\n",
    "# Overview:\n",
    "\n",
    "This notebook will demonstrate the capabilities of the `ADSEvaluator`. It is a machine learning (ML) evaluation component of Oracle Cloud Infrastructure's Accelerated Data Science (ADS) package. You will learn how it can be used for the evaluation of any general class of supervised machine learning models, as well as comparison amongst models within the same class.   \n",
    "\n",
    "Specifically, the notebook will focus on binary classification using an imbalanced dataset, multi-class classification using a synthetically generated dataset consisting of three equally distributed classes and lastly a regression using a synthetically generated dataset with positive targets. The training is done using a standard library, and subsequently, the models would be evaluated using `ADSEvaluator`.\n",
    "\n",
    "Compatible conda pack: [General Machine Learning](https://docs.oracle.com/en-us/iaas/data-science/using/conda-gml-fam.htm) for CPU on Python 3.8 (version 1.0)\n",
    "\n",
    "## Contents:\n",
    "\n",
    "- <a href='#binary'>Binary Classification</a>\n",
    "    - <a href='#binary_data'>Data</a>\n",
    "    - <a href='#binary_train'>Train</a>\n",
    "    - <a href='#binary_adsmodel'>Convert to an `ADSModel`</a>\n",
    "    - <a href='#binary_evaluation'>Model Evaluation</a>\n",
    "- <a href='#multi'>Multiclass Classification</a>\n",
    "    - <a href='#multi_data'>Data</a>\n",
    "    - <a href='#multi_train'>Train</a>\n",
    "    - <a href='#multi_adsmodel'>Convert to an `ADSModel`</a>\n",
    "    - <a href='#multi_evaluation'>Model Evaluation</a>\n",
    "- <a href='#reg'>Regression</a>\n",
    "    - <a href='#reg_data'>Data</a>\n",
    "    - <a href='#reg_train'>Train</a>\n",
    "    - <a href='#reg_adsmodel'>Convert to an `ADSModel`</a>\n",
    "    - <a href='#reg_evaluation'>Model Evaluation</a>\n",
    "- <a href='#adsevaluator'>Working with `ADSEvaluator`</a>\n",
    "    - <a href='#adsevaluator_metrics'>Raw Metrics</a>\n",
    "    - <a href='#adsevaluator_admod'>Add and Delete Models</a>\n",
    "    - <a href='#adsevaluatoradmet'>Add and Delete Custom Metrics</a>\n",
    "    - <a href='#adsevaluator_cost'>Calculate Cost</a>\n",
    "- <a href='#ref'>References</a>\n",
    " \n",
    "---\n",
    "\n",
    "\n",
    "Datasets are provided as a convenience.  Datasets are considered third-party content and are not considered materials \n",
    "under your agreement with Oracle.\n",
    "    \n",
    "You can access the `oracle_fraud_dataset1` dataset license [here](https://oss.oracle.com/licenses/upl). \n",
    "    \n",
    "You can access the `wine` dataset license is available [here](https://github.com/scikit-learn/scikit-learn/blob/master/COPYING).\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ads.environment.ml_runtime\n",
    "import logging\n",
    "import matplotlib.font_manager\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import warnings\n",
    "\n",
    "from ads.common.data import ADSData\n",
    "from ads.common.model import ADSModel\n",
    "from ads.dataset.dataset_browser import DatasetBrowser\n",
    "from ads.dataset.factory import DatasetFactory\n",
    "from ads.evaluations.evaluator import ADSEvaluator\n",
    "from os.path import join\n",
    "from scipy import stats\n",
    "from sklearn import tree\n",
    "from sklearn.datasets import make_regression\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression, LinearRegression, Lasso\n",
    "from sklearn.metrics import fbeta_score\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "logging.basicConfig(format=\"%(levelname)s:%(message)s\", level=logging.ERROR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='binary'></a>\n",
    "# Binary Classification\n",
    "\n",
    "The next few cells will demonstrate one way to create and binary classification `ADSEvaluator` object. However, each of these cells is modular and can be interchanged with your favorite alternative and weaved back in.\n",
    "\n",
    "<a id='binary_data'></a>\n",
    "## Data\n",
    "\n",
    "For this example, you want to predict whether or not a given transaction may be fraudulent based on a variety of columns.\n",
    "\n",
    "The data are using are stored on Oracle Cloud Infrastructure (OCI) Object Storage. OCI Object Storage is a performant hot storage option for data files. You read the file directly at the URL listed below.\n",
    "\n",
    "You will use a `DatasetFactory` object from the Oracle Accelerated Data Science (ADS) library to pull the data from Object Storage. `DatasetFactory.open()` creates an `ADSDataset` type object, which can be used for a variety of visualizations. Here you pass in the target variable, `anomalous`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "attrition_path = join(\n",
    "    \"/\", \"opt\", \"notebooks\", \"ads-examples\", \"oracle_data\", \"oracle_fraud_dataset1.csv\"\n",
    ")\n",
    "binary_fk = DatasetFactory.open(attrition_path, target=\"anomalous\").sample(frac=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have data with a target, you will split the dataset into two separate datasets. 85% of the data will be for training and 15% for model evaluation. This will be done using the `.train_test_split()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = binary_fk.train_test_split(test_size=0.15)\n",
    "X_train = train.X.values\n",
    "y_train = train.y.values\n",
    "X_test = test.X.values\n",
    "y_test = test.y.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='binary_train'></a>\n",
    "## Train\n",
    "\n",
    "Sklearn is a well-known Python library for training various kinds of machine learning models. You use it to train two classifiers:\n",
    " - __Logistic regression__: Logistic regression is a statistical model that in its basic form uses a logistic function to model a binary dependent variable, although many more complex extensions exist. In regression analysis, logistic regression (or logit regression) is estimating the parameters of a logistic model (a form of binary regression)\n",
    " - __Random Forest__: Random forests are an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_clf = LogisticRegression(\n",
    "    random_state=0, solver=\"lbfgs\", multi_class=\"multinomial\"\n",
    ").fit(X_train, y_train)\n",
    "\n",
    "rf_clf = RandomForestClassifier(n_estimators=10, random_state=0).fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='binary_adsmodel'></a>\n",
    "## Convert to an `ADSModel`\n",
    "\n",
    "The `ADSModel` class in the ADS package, has a `.from_estimator()` method. It takes as input a fitted estimator and converts it into an `ADSModel` object. In the case of classification, you need to pass the class labels in the `classes` argument. The `ADSModel` object is used for evaluation in the `ADSEvaluator` object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_lr_model = ADSModel.from_estimator(lr_clf, classes=[0, 1])\n",
    "bin_rf_model = ADSModel.from_estimator(rf_clf, classes=[0, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='binary_evaluation'></a>\n",
    "## Model Evaluation\n",
    "\n",
    "To instantiate an `ADSEvaluator` object, two main parameters are:\n",
    " - __ADSData__: The `ADSData` object for the test set prepared earlier\n",
    " - __Models__: The `ADSModel` objects for the logistic regression and random forest. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator = ADSEvaluator(\n",
    "    test, models=[bin_lr_model, bin_rf_model], training_data=train\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <a id='plot'></a>\n",
    "The `ADSEvaluator` object has a `.show_in_notebook()` method, which can be used to visualize a variety of machine learning evaluation metrics. For binary classification you can view the following:\n",
    "- __gain_chart__: A plot of gain vs % baseline positives (true positive rate or recall vs predictive positive rate). [Read more](http://mlwiki.org/index.php/Cumulative_Gain_Chart).\n",
    "- __ks_statistics__: (or the Kolmogorov–Smirnov statistic) A nonparametric plot of the difference in the distributions of both labels. [Read more](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test).\n",
    "- __lift_chart__: A plot of lift vs % baseline positives (cumulative gain/total gain vs predictive positive rate). [Read more](https://en.wikipedia.org/wiki/Lift_(data_mining)).\n",
    "- __pr_curve__: A plot of precision vs recall (the proportion of positive class predictions that were correct vs the proportion of positive class objects that were correctly identified). [Read more](https://en.wikipedia.org/wiki/Precision_and_recall).\n",
    "- __normalized_confusion_matrix__: A matrix of the number of actual vs predicted values for each class, normalized by the number of true labels per class (rows). [Read more](https://en.wikipedia.org/wiki/Confusion_matrix).\n",
    "- __roc_curve__: A plot of true positive rate vs false positive rate (recall vs the proportion of negative class objects that were identified incorrectly). [Read more](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "bin_evaluator.show_in_notebook(perfect=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Note on the parameters:\n",
    " - If `perfect` is set to `True`, a plot of a perfect classifier for comparison in the Lift and Gain charts.\n",
    " - If `baseline` is set `True`, then the baseline will not be included for the comparison of various plots.\n",
    " - If `use_training_data` is set `True`, the training data will be used for evaluations.\n",
    " - If `plots` contains a list of plot types, then only those plot types will be displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator.show_in_notebook(\n",
    "    [\"gain_chart\", \"lift_chart\"], baseline=False, use_training_data=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <a id='met'></a>\n",
    "Further, you can compare various metrics using the `metrics` property of the `ADSEvaluator` object. For binary classification, the following metrics are available:\n",
    "- __accuracy__: Proportion of correctly classified examples.\n",
    "- __auc__: Area under the ROC curve. \n",
    "- __f1__: Harmonic mean of precision and recall.\n",
    "- __hamming_loss__: Proportion of incorrectly classified examples.\n",
    "- __precision__: Proportion of positive class predictions that were correct.\n",
    "- __recall__: Proportion of positive class examples that were correctly identified.\n",
    "\n",
    "The metrics in blue are the best for that row of models on testing data. The metrics in yellow are the best for that row of models on training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator.metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can select specific metrics, by extracting them from the Pandas DataFrames `test_evaluations` or `train_evaluations` and subsequently indexing the row `.loc['metric_name']`. For example, you can show only the __precision__ metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator.test_evaluations.loc[\"precision\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='multi'></a>\n",
    "# Multiclass Classification\n",
    "\n",
    "This example is similar to the <a href='#binary'>binary classification</a> example. However, instead of predicting one of two classes, you will be predicting one of three classes. This adds a level of complexity to the model but also the evaluation metrics.\n",
    "\n",
    "<a id='multi_data'></a>\n",
    "## Data\n",
    "\n",
    "Here you will use the wine dataset from sklearn. This dataset contains certain features about different wines, and your goal is to predict the type. You can load the data using the `DatasetBrowser` object as shown in the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "multi_ds = DatasetBrowser.sklearn().open(\"wine\").set_target(\"target\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have data with a target, you will split the dataset into two separate datasets. 85% of the data will be for training and 15% for model evaluation. This will be done using the `.train_test_split()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = multi_ds.train_test_split(test_size=0.15)\n",
    "X_train = train.X.values\n",
    "y_train = train.y.values\n",
    "X_test = test.X.values\n",
    "y_test = test.y.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='multi_train'></a>\n",
    "## Train\n",
    "\n",
    "Use sklearn to train two multi-class classifiers:\n",
    "- __Multinomial Logistic Regression__: Multinomial logistic regression is a classification method that generalizes logistic regression to multiclass problems, i.e. with more than two possible discrete outcomes. It is also known as softmax regression, because of the use of a softmax function. The logistic function used in binary logistic regression is a special case of the softmax function for two outcomes.\n",
    "- __Random Forest__: Random forests are an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_clf = LogisticRegression(\n",
    "    random_state=0, solver=\"lbfgs\", multi_class=\"multinomial\"\n",
    ").fit(X_train, y_train)\n",
    "\n",
    "rf_clf = RandomForestClassifier(n_estimators=10).fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='multi_adsmodel'></a>\n",
    "## Convert to an `ADSModel`\n",
    "\n",
    "Similar to the case of binary classification, you need to pass in the class labels using the `classes` argument. There are three classes in the dataset, __0__, __1__ and __2__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_model = ADSModel.from_estimator(lr_clf)\n",
    "rf_model = ADSModel.from_estimator(rf_clf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='multi_evaluation'></a>\n",
    "## Model Evaluation\n",
    "\n",
    "Similar to the binary classification problem, to instantiate an `ADSEvaluator` object you need the following objects:\n",
    "- __ADSData__: The `ADSData` object for the test set prepared earlier\n",
    "- __Models__: The `ADSModel` objects for the multinomial logistic regression and random forest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "multi_evaluator = ADSEvaluator(test, models=[lr_model, rf_model])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For multi-class classification you can view the following using `show_in_notebook()`:\n",
    "- __confusion_matrix__: A matrix of the number of actual vs predicted values for each class. [Read more](https://en.wikipedia.org/wiki/Confusion_matrix)\n",
    "- __f1_by_label__: Harmonic mean of precision_by_label and recall_by_label. Compute f1 for each, __3__ f1 scores in this examples. [Read more](https://en.wikipedia.org/wiki/F1_score)\n",
    "- __jaccard_by_label__: Computes the similarity for each label distribution. [Read more](https://en.wikipedia.org/wiki/Jaccard_index)\n",
    "- __pr_curve__: A plot of precision vs recall (the proportion of positive class predictions that were correct vs the proportion of positive class objects that were correctly identified). [Read more](https://en.wikipedia.org/wiki/Precision_and_recall).\n",
    "- __precision_by_label__: Consider one label as positive class and rest as negative. Compute precision for each, __3__ precision numbers in this example. [Read more](https://en.wikipedia.org/wiki/Precision_(statistics))\n",
    "- __recall_by_label__: Consider one label as positive class and rest as negative. Compute recall for each, __3__ recall numbers in this example. [Read more](https://en.wikipedia.org/wiki/Precision_and_recall)\n",
    "- __roc_curve__: A plot of true positive rate vs false positive rate (recall vs the proportion of negative class objects that were identified incorrectly). [Read more](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "multi_evaluator.show_in_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For multi-class classification, you can have the following metrics:\n",
    "- __accuracy__: Number of correctly classified examples divided by total examples\n",
    "- __f1_micro__: Global f1. Can be calculated by using the harmonic mean of __precision_micro__ and __recall_micro__.\n",
    "- __f1_weighted__: Weighted average of __f1_by_label__. Weights are proportional to the number of true instances for each label.\n",
    "- __hamming_loss__: 1 - accuracy\n",
    "- __precision_micro__: Global precision. Calculated by using global true positives and false positives.\n",
    "- __precision_weighted__: Weighted average of __precision_by_label__. Weights are proportional to the number of true instances for each label.\n",
    "- __recall_micro__: Global recall. Calculated by using global true positives and false negatives. \n",
    "- __recall_weighted__: Weighted average of __recall_by_label__. Weights are proportional to the number of true instances for each label.\n",
    "\n",
    "All of these metrics can be computed directly from the confusion matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "multi_evaluator.metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='reg'></a>\n",
    "# Regression\n",
    "\n",
    "In this section, you will see another example of building an `ADSEvaluator` object. However, this section is a regression problem instead of the previous classification problems.\n",
    "\n",
    "<a id='reg_data'></a>\n",
    "## Data\n",
    "\n",
    "The next cell will create a synthetic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_X, _y = make_regression(\n",
    "    n_samples=10000, n_features=10, n_informative=2, random_state=42\n",
    ")\n",
    "df = pd.DataFrame(_X, columns=[\"F{}\".format(x) for x in range(10)])\n",
    "df[\"target\"] = pd.Series(_y)\n",
    "reg_ds = DatasetFactory.open(df).set_target(\"target\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have data with a target, you will split the dataset into two separate datasets. 85% of the data will be for training and 15% for model evaluation. This will be done using the `.train_test_split()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = reg_ds.train_test_split(test_size=0.15)\n",
    "X_train = train.X.values\n",
    "y_train = train.y.values\n",
    "X_test = test.X.values\n",
    "y_test = test.y.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='reg_train'></a>\n",
    "## Train\n",
    "\n",
    "You will use sklearn to train two regression models:\n",
    "- __Linear Regression__: Linear regression is a linear approach to modeling the relationship between a scalar response and one or more predictor variables. Linear regression models try to minimize the sum of the squares of the residuals, in the loss function.\n",
    "- __Lasso__: Lasso (least absolute shrinkage and selection operator) is a regression analysis method that performs both variable selection and regularization in order to enhance the prediction accuracy and interpretability of the statistical model it produces. Mathematically, it applies the L1-penalty to the least-squares of the residuals loss function in linear regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lin_reg = LinearRegression().fit(X_train, y_train)\n",
    "\n",
    "lasso_reg = Lasso(alpha=0.1).fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='reg_adsmodel'></a>\n",
    "## Convert to an `ADSModel`\n",
    "\n",
    "In the case of regression, you only need to pass in the fitted estimators (regression models in this case)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lin_reg_model = ADSModel.from_estimator(lin_reg)\n",
    "lasso_reg_model = ADSModel.from_estimator(lasso_reg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='reg'></a>\n",
    "## Model Evaluation\n",
    "\n",
    "Like before, the `ADSEvaluator` object needs two main things for instantiation:\n",
    " - __ADSData__: The `ADSData` object for the test set prepared earlier\n",
    " - __Models__: The `ADSModel` objects for the linear regression and lasso regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg_evaluator = ADSEvaluator(test, models=[lin_reg_model, lasso_reg_model])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For regression you can view the following using `show_in_notebook()`:\n",
    "- __observed_v_predicted__: Plot of the observed, or actual values against your predicted values output by your models.\n",
    "- __residuals_qq__: Quantile-quantile plot between residuals and quantiles of a standard normal distribution. Should be very close to a straight line for a good model. \n",
    "- __residuals_vs_observed__: Plot of Residuals vs Observed values. This, too, should not carry a lot of structure in a good model.\n",
    "- __residuals_vs_predicted__: Plot of Residuals vs Predicted values. This should not carry a lot of structure in a good model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "reg_evaluator.show_in_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For regression, you can use the `.metrics` property to see the following:\n",
    "- __explained_variance_score__: Variance of the model's predictions. Mean of the squared difference between the  predicted values and the true mean of the data. [Read more](https://en.wikipedia.org/wiki/Explained_variation)\n",
    "- __mean_absolute_error__: Mean of the absolute difference between the true values and predicted values. [Read more](https://en.wikipedia.org/wiki/Mean_absolute_error)\n",
    "- __mean_residuals__: Mean of the difference between the true values and predicted values. [Read more](https://en.wikipedia.org/wiki/Errors_and_residuals)\n",
    "- __mean_squared_error__: Mean of the squared difference between the true values and predicted values. [Read More](https://en.wikipedia.org/wiki/Mean_squared_error)\n",
    "- __r2_score__: Also known as __coefficient of determination__, is the proportion of the variance in the dependent variable that is predictable from the independent variables. [Read more](https://en.wikipedia.org/wiki/Coefficient_of_determination)\n",
    "- __root_mean_squared_error__: Square root of __mean_squared_error__. [Read more](https://en.wikipedia.org/wiki/Root-mean-square_deviation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg_evaluator.metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='adsevaluator'></a>\n",
    "# Working with `ADSEvaluator`\n",
    "\n",
    "In this section, you will generate various types of `ADSEvaluator` objects and examine the associated plots and metrics. Let's dive into a few more advanced features of the `ADSEvaluator` class.\n",
    "\n",
    "<a id='adsevaluator_metrics'></a>\n",
    "# Raw Metrics\n",
    "\n",
    "Going back to the original binary classification problem, you will access the raw metrics from you `ADSEvaluator` object. The results are returned in JSON format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "bin_evaluator.raw_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='adsevaluator_admod'></a>\n",
    "## Add and Delete Models\n",
    "\n",
    "You can also add models later on for evaluation, by using the `.add_models([model_list])` method. For example, assume you just read a paper that suggested Decision Tree Classifiers might be better for capturing a part of your data. You wish to add that to an existing `ADSEvaluator` object. You need to create this model and then add it to the`ADSEvaluator` object. The following cells demonstrate how to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, _ = binary_fk.train_test_split(test_size=0.15)\n",
    "X_train = train.X.values\n",
    "y_train = train.y.values\n",
    "\n",
    "tree_mod = tree.DecisionTreeClassifier(max_depth=3).fit(X_train, y_train)\n",
    "\n",
    "bin_tree_model = ADSModel.from_estimator(tree_mod, classes=[0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator.add_models([bin_tree_model])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator.metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the metrics summary above, this model doesn't seem to have improved any of the metrics you are interested in. Therefore, you may want to remove this model to de-clutter the output. To do this, use the `del_models()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator.del_models([\"DecisionTreeClassifier\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator.metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='adsevaluator_admet'></a>\n",
    "## Add and Delete Custom Metrics\n",
    "\n",
    "Just as you can add and delete models, you can add and delete metrics. This is for those problems that require esoteric and specific metrics not yet supported by the `ADSEvaluator` class. For example, with your highly imbalanced dataset of fraudulent credit card purchases, you might find that the $F_2$ score is more relevant than the $F_1$ score. However, there is no standard library that has the $F_2$ metric, thus you will have to write a function to compute it. Once you define the function you would pass it into the `ADSEvaluator` object via `.add_metrics()` method. This function will get the true values and the predicted values from your model and put them into your `evaluator.metrics` output.\n",
    "\n",
    "`.add_metrics()` accepts multiple functions passed in as a list. To demonstrate this, you will pass in a metric that tells you the number of correct predictions.\n",
    "\n",
    "Due to a limitation with how Jupyter handles created functions if you intend to use any of these functions within `AutoML` you need to define them in a separate file and import them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def func1(y_true, y_pred):\n",
    "    return sum(y_true == y_pred)\n",
    "\n",
    "\n",
    "def func2(y_true, y_pred):\n",
    "    return fbeta_score(y_true, y_pred, beta=2)\n",
    "\n",
    "\n",
    "bin_evaluator.add_metrics([func1, func2], [\"Total True\", \"F2 Score\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator.metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These metrics have been interesting, but ultimately didn't add much, let's delete them from your `ADSEvaluator` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator.del_metrics([\"Total True\", \"F2 Score\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator.metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='adsevaluator_cost'></a>\n",
    "## Calculate Cost\n",
    "\n",
    "The `.calculate_cost()` method helps you to evaluate your binary classification model based on your own weighting of the problem. If true positives are really important and false positives are less important in some sort of medical diagnosis, for example, you can use this method to quantify that difference.\n",
    "\n",
    "The method requires these parameters: `tn_weight`, `fp_weight`, `fn_weight`, `tp_weight`, which of course represent the weightings of the 4 values of a binary confusion matrix. See some example distributions below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator.calculate_cost(0, 1, 1, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assume you wanted a 100-1 ratio of false positives to false negatives. Meaning you would much rather have 99 positive estimations that were wrong, than 1 negative estimation that is wrong. Here you can see that while logistic regression won out on pure accuracy. The random forest classifier was actually better for the cases you care about."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin_evaluator.calculate_cost(0, 1, 0.01, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <a id='ref'></a>\n",
    "# References\n",
    "\n",
    "- [ADS Library Documentation](https://accelerated-data-science.readthedocs.io/en/latest/index.html)\n",
    "- [Data Science YouTube Videos](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
    "- [OCI Data Science Documentation](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
    "- [Oracle Data & AI Blog](https://blogs.oracle.com/datascience/)\n",
    "- Pedregosa, Fabian, et al. [Scikit-learn: Machine learning in Python.](http://jmlr.csail.mit.edu/papers/v12/pedregosa11a.html) Journal of machine learning research 12.Oct (2011): 2825-2830.\n",
    "- Tibshirani, Robert. [Regression shrinkage and selection via the lasso.](https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/j.2517-6161.1996.tb02080.x) Journal of the Royal Statistical Society: Series B (Methodological) 58.1 (1996): 267-288."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.6 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}