15Jun2022 - API Python I - Colab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ePVVRJ_7FkEg"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/datarobot-community/DRU-MLOps/blob/master/15Jun2022 - API Python I - Colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tFS-eBOmsi-F"
   },
   "source": [
    "# What to expect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The class is broken into two sessions each covering one half-day block.\n",
    "    - Each session is comprised of two modules.\n",
    "    - Within each module, multiple topics will be covered.\n",
    "- While the specific timings of each module may differ, overall we will plan to follow this format:\n",
    "    - Motivation for the topic covered will first be given.\n",
    "    - We'll examine how this is done in Python using the `datarobot` Python package and other packages like `pandas`, `matplotlib`, etc. You'll write the code as it is presented to help you start to become fluent in the syntax.\n",
    "    - Then you'll practice what you've just learned on a different problem/context by programming yourself.\n",
    "        - You can decide whether you'd like to try the beginner, intermediate, or advanced option.\n",
    "        - Ideally, you try all three!\n",
    "    - We'll go over ways to solve the particular problem and discuss solutions.\n",
    "    - We'll then head into covering the next topic in the module.\n",
    "- Overall, we want you to be able to practice your learning immediately to check for understanding and get help from us as needed to get things working."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Agenda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Session A\n",
    "- [Module 1.1 - Preliminaries](#Module-1.1:-Preliminaries)\n",
    "- [Module 1.2 - Model interpretation basics](#Module-1.2:-Model-interpretation-basics)\n",
    "\n",
    "Session B\n",
    "- [Module 2.1 - Further model interpretation and understanding](#Module-2.1:-Further-model-interpretation-and-understanding)\n",
    "- [Module 2.2 - Advanced feature selection](#Module-2.2:-Advanced-feature-selection)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Session A will be focused on building a machine learning project in DataRobot via code**\n",
    "* **Session B will be mostly focused on customizing the functionality provided by DataRobot**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll see that we have some overall themes in this class. First, we'll go over how to connect Python to the DataRobot client. Then we'll see how to make a DataRobot project. Inside that project, we'll look into featurelists and blueprints before diving into building models and evaluating and understanding them with the use of some visualizations. These main ideas are also covered in the DataRobot Python cheatsheet that has been included in the course ZIP file. We don't go over everything on that sheet in this class but it can be a useful resource. Now that we've talked about the general concepts of the course I'd like to cover the main expectations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "By the end of this mission, you will be able to:\n",
    "\n",
    "- Connect to the DataRobot client using an API key\n",
    "- Create a project in DataRobot programmatically\n",
    "- Set the target feature for a DataRobot project\n",
    "- Download and review DataRobot created featurelists\n",
    "- Extract and customize the features of a featurelist\n",
    "- Review the properties of a DataRobot project Repository\n",
    "- Build a new model from a blueprint and custom featurelist\n",
    "- Start autopilot with the maximum number of workers\n",
    "\n",
    "\n",
    "- List all models trained during autopilot\n",
    "- Create a custom function in Python to extract Leaderboard results\n",
    "- Get training predictions for a model\n",
    "- Create a custom lift chart to aggregate predicted and actual results\n",
    "- Retrieve Feature Impact for top performing models\n",
    "- Build a model with reduced features based on Feature Impact\n",
    "- Make predictions on scoring data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall, the purpose of what is shown in this class are ways to access DataRobot in a programmatic way. These examples may not necessarily be the best use cases for you directly, but our goal is for you to be able to directly apply your learning here to your own projects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In order to connect to the DataRobot client, you will need and API token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now obtain an API token from the DataRobot GUI; we need to paste this token into the file *drconfig.yaml* which can be found in the folder containing the class material."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.yaml` file is basically a text file containing two lines:\n",
    "\n",
    "`token: \"YOUR_API_TOKEN\"`  \n",
    "`endpoint: \"YOUR_HOSTNAME\"`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we add our API token to this file (don't forget to save the file!), we are ready to upload class assets to the Colab environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VeC6ZQmYnIYl"
   },
   "source": [
    "## Let's upload the resources we will need into the Colab environment:\n",
    "\n",
    "1. Training dataset: **shot_logs_wed.csv**\n",
    "2. Scoring dataset: **shot_logs_sat.csv**\n",
    "3. Configuration file: **drconfig.yaml**\n",
    "4. Requirements file: **colab_requirements.txt**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-IDgpv8NMs7r"
   },
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "uploaded = files.upload()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4sN_vquZolv_"
   },
   "outputs": [],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will install the Python modules we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -r colab_requirements.txt -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SESSION A: DataRobot Modeling Basics with the API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to the DataRobot client using the API key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use Python with DataRobot you first need to establish a connection between your development environment and a DataRobot instance. The recommended way to do that is by creating a `.yaml` file with your credentials. Here, we have chosen the filename `drconfig.yaml`; we have already placed this file in the Colab environment after editing it to include our API token."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this class we will be using `endpoint: https://app.datarobot.com/api/v2`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the YAML file contents after updating\n",
    "# Not necessary but a nice check\n",
    "!cat drconfig.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the datarobot package\n",
    "import datarobot as dr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# connect to datarobot using your saved credentials\n",
    "dr.Client(config_path = 'drconfig.yaml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Another option (not needed if running the previous cell works)\n",
    "# Fill in the token argument with your created token\n",
    "dr.Client(token = , endpoint = \"https://app.datarobot.com/api/v2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore data using `pandas`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load pandas\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To make it easier to follow the difference between user-defined names and API-defined functionality, we will use shorter **camelCase** notation for the former and **snake_case** for the latter.\n",
    "\n",
    "\n",
    "### Therefore: `userDef` vs `api_defined`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load data from text file\n",
    "shotData = pd.read_csv('shot_logs_wed.csv')\n",
    "\n",
    "# view first few rows of DataFrame\n",
    "shotData.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What is the Unit of Analysis for this DataFrame?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Which feature would be a good target for us to use for modeling?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start preliminary modeling steps\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There a few different ways to get a project started with the Python package. One way this can be done is using the `datarobot.Project.create()` method. You can specify the `sourcedata` as a URL, as a file, or as a Python DataFrame as we will do here. Let's also import the `date` class from the `datetime` module to get today's date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create name of project\n",
    "from datetime import date\n",
    "shotProjectName = 'NBA Shots ' + date.today().strftime(format = \"%Y-%m-%d\")\n",
    "shotProjectName"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a project in DataRobot\n",
    "shotProject = ___.___.___(\n",
    "    sourcedata = ___,\n",
    "    project_name = ___\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go to <https://app.datarobot.com/manage-projects> and find the \"NBA Shots\" project that was just created. Click on it and then go to the **Data** tab. Many of the things you've done in the app, like starting a new project, are also available via the API! You can also keep the app open to check for different tasks being started programmatically."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question C\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# shotProject.get_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What error is caused by uncommenting and running the above code? Why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show available optimization metrics\n",
    "shotProject.___(feature_name = '___') #['available_metrics']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the data uploaded and we've examined potential metrics, we can get the second round of exploratory data analysis and preliminary modeling steps kicked off using the `.set_target()` method. We'll use manual mode for now with the `mode` argument below. We'll kick off autopilot at the end of this Session A. \n",
    "\n",
    "We also set the `positive_class` to `\"made\"` here so that we can interpret the model predictions to be in terms of the chance of a shot being made instead of the chance of a shot being missed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare for modeling in manual mode\n",
    "shotProject.___(\n",
    "    target = \"___\",\n",
    "    metric = \"___\",\n",
    "    positive_class = \"___\",\n",
    "    mode = \"___\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting help\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the API package documentation you saw before, remember to use `?` too. This will open up the docstring and signature for a function/method in its own pane in Jupyter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "?dr.Project.create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "?shotProject.set_target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also investigate the code for the function/method by using two question marks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "??dr.Project.create"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also access help in google Colab by writing a period after an object's name and then placing the cursor after the period"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dr.Project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### YOUR TASK (Exercise 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download and review DataRobot created featurelists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a list of all current featurelists for the `shotProject`\n",
    "shotFeaturelists = shotProject.___()\n",
    "shotFeaturelists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `Informative Features - Leakage Removed` has been created here. If you haven't noticed this already, there are two features that are detected as target leakage if the target is `SHOT_RESULT`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "QUESTION: Look in https://app.datarobot.com at the `shotProject` to identify them. Which two are they?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract the features of a featurelist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract name and features of each feature list as a dictionary\n",
    "featurelistsDict = {featurelist.name: featurelist.features \n",
    "                      for featurelist in shotFeaturelists}\n",
    "\n",
    "# Examine this dictionary\n",
    "featurelistsDict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be building a featurelist in this part of the class and so we'd like to be able to extract the features that correspond to a particular featurelist. We could choose a different attribute here to have as the value corresponding to the key of `name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "leakageRemovedFeatures = featurelistsDict.get(\n",
    "    \"Informative Features - Leakage Removed\"\n",
    ")\n",
    "leakageRemovedFeatures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a list in Python laying out the names of the features in the Informative Features - Leakage Removed featurelist in DataRobot.\n",
    "\n",
    "If the overall goal is to predict if a shot in a game will be made or missed, there are a couple other features in our data currently that we won't know at the time of prediction and are thus target leakage. Can you identify which ones they are?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "leakageRemovedFeatures.remove('___')\n",
    "leakageRemovedFeatures.remove('___')\n",
    "leakageRemovedFeatures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create new featurelist in DataRobot\n",
    "modFeatlist = shotProject.___(\n",
    "    name = 'Modified Informative Features - Leakage Removed', \n",
    "    features = ___\n",
    ")\n",
    "modFeatlist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### YOUR TASK (Exercise 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Module 1.2: Model interpretation basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "By the end of this mission, you will be able to:\n",
    "\n",
    "- ~~Connect to the DataRobot client using an API key~~\n",
    "- ~~Create a project in DataRobot programmatically~~\n",
    "- ~~Set the target feature for a DataRobot project~~\n",
    "- ~~Download and review DataRobot created featurelists~~\n",
    "- ~~Extract and customize the features of a featurelist~~\n",
    "- Review the properties of a DataRobot project Repository\n",
    "- Build a new model from a blueprint and custom featurelist\n",
    "- Start autopilot with the maximum number of workers\n",
    "\n",
    "\n",
    "- List all models trained during autopilot\n",
    "- Create a custom function in Python to extract Leaderboard results\n",
    "- Get training predictions for a model\n",
    "- Create a custom lift chart to aggregate predicted and actual results\n",
    "- Retrieve Feature Impact for top performing models\n",
    "- Build a model with reduced features based on Feature Impact\n",
    "- Make predictions on scoring data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review properties of the Repository"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the Repository contains candidate blueprints corresponding to what DataRobot has determined to be recipes that should create high performing models when combined with data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve all blueprints for the NBA shots project\n",
    "shotProjectBlueprints = shotProject.___()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# See what a few model algorithm names are for blueprints\n",
    "shotProjectBlueprints[4:8]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Review the structure of the Blueprint class\n",
    "?dr.Blueprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create DataFrame of some information about blueprints\n",
    "bpDF = pd.DataFrame(\n",
    "    [(blueprint.id, blueprint.model_type, blueprint.processes) \n",
    "     for blueprint in shotProjectBlueprints],\n",
    "    columns = [\"blueprint_id\", \"model_type\", \"processes\"]\n",
    ")\n",
    "bpDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter to focus only on the Regularized Logistic Regression blueprints\n",
    "rlrBpDF = bpDF[bpDF['model_type'].str.contains(\"Regularized Logistic\")]\n",
    "rlrBpDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the sake of class time, let's just go with the blueprint in the last row here since its preprocessing steps look relatively simple. We are almost to building a model with this blueprint and to do so we'll need to keep track of the `blueprint_id`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get blueprint_id from the last row\n",
    "# (iloc is \"index locator\")\n",
    "bpIdToBuild = rlrBpDF.iloc[___].___\n",
    "bpIdToBuild "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Review the documentation of the `.train()` method.\n",
    "?shotProject.train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that `featurelist_id` is an argument to `.train()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the id of the newly created Featurelist\n",
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build a model using the last blueprint given above\n",
    "# and the Modified Informative Features - Leakage Removed \n",
    "# featurelist we created\n",
    "# Note that this will take a few seconds to complete\n",
    "rlrModelJobId = shotProject.___(\n",
    "    trainable = ___,\n",
    "    featurelist_id = ___,\n",
    "    sample_pct = 64,\n",
    "    scoring_type = \"validation\"\n",
    ")\n",
    "rlrModelJobId"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the Job object from the job ID\n",
    "rlrModelJob = dr.models.Job.get(\n",
    "    project_id = shotProject.id, \n",
    "    job_id = ___\n",
    ")\n",
    "rlrModelJob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve model when finished building\n",
    "rlrModel = ___.get_result_when_complete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that what is returned from `.train()` is a model job id. On the Leaderboard, this corresponds to the number after the M. We've only built one model here, but if we went to the Leaderboard right now we would see the number above next to the M for this Regularized Logistic Regression model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### YOUR TASK (Exercise 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a model trained and validated, we can start to evaluate the fit of the model and understand which features are most impactful to the model. We'll start by looking at the Feature Impact for this model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Feature Impact data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Request Feature Impact be calculated and then download it\n",
    "# If Feature Impact has already been calculated for a model, this\n",
    "# function will just get it too\n",
    "# Note this may take a bit to compute\n",
    "fi = ___\n",
    "\n",
    "# Save Feature Impact in pandas DataFrame\n",
    "fiDF = pd.DataFrame(fi)\n",
    "\n",
    "# View the Feature Impact for this model\n",
    "fiDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also get `impactUnnormalized` here, which can be useful in seeing the absolute Feature Impact of a particular feature instead of relative to the top one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### YOUR TASK (Exercise 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kicking off autopilot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To help us prepare for the content in Session B, let's start autopilot on this NBA Shots project. We'll continue reviewing many of the model interpretation tools in Session B and explore how to look at the Leaderboard as a pandas DataFrame as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start autopilot using the Featurelist we created\n",
    "shotProject.___(\n",
    "    featurelist_id = ___\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set number of workers\n",
    "shotProject.set_worker_count(\n",
    "    worker_count = -1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "QUESTION: What do you think `worker_count = -1` specifies for the number of workers?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important note**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is best practice after you kick off modeling with Quick mode or Autopilot to also run the `.wait_for_autopilot()` method. This will ensure that other tasks that depend on modeling being finished aren't run until the modeling phase is done in the phase of autopilot chosen. Run the code below to ensure this is done for your project AFTER you've started the model building.\n",
    "\n",
    "Running `.wait_for_autopilot()` will also block any other Python commands from being run so use it cautiously. Oftentimes it's best to do it just before you take a break."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shotProject.wait_for_autopilot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### YOUR TASK (Exercise 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SESSION B: DataRobot Model Interpretation and Understanding with the API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "By the end of this mission, you will be able to:\n",
    "\n",
    "- ~~Connect to the DataRobot client using an API key~~\n",
    "- ~~Create a project in DataRobot programmatically~~\n",
    "- ~~Set the target feature for a DataRobot project~~\n",
    "- ~~Download and review DataRobot created featurelists~~\n",
    "- ~~Extract and customize the features of a featurelist~~\n",
    "- ~~Review the properties of a DataRobot project Repository~~\n",
    "- ~~Build a new model from a blueprint and custom featurelist~~\n",
    "- ~~Start autopilot with the maximum number of workers~~\n",
    "\n",
    "\n",
    "- List all models trained during autopilot\n",
    "- Create a custom function in Python to extract Leaderboard results\n",
    "- Get training predictions for a model\n",
    "- Create a custom lift chart to aggregate predicted and actual results\n",
    "- Retrieve Feature Impact for top performing models\n",
    "- Build a model with reduced features based on Feature Impact\n",
    "- Make predictions on scoring data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resuming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make sure we are back at where we left off at the end of Session A."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VeC6ZQmYnIYl"
   },
   "source": [
    "## Let's upload resources into the Colab environment:\n",
    "\n",
    "1. Training dataset: **shot_logs_wed.csv**\n",
    "2. Scoring dataset: **shot_logs_sat.csv**\n",
    "3. Configuration file: **drconfig.yaml**\n",
    "4. Requirements file: **colab_requirements.txt**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-IDgpv8NMs7r"
   },
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "uploaded = files.upload()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4sN_vquZolv_"
   },
   "outputs": [],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's install the Python modules we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -r colab_requirements.txt -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# re-import packages\n",
    "import datarobot as dr\n",
    "import pandas as pd\n",
    "from datetime import date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reconnect to dr\n",
    "dr.Client(config_path = 'drconfig.yaml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need a way to get the Project id from DataRobot to continue your progress. One way to retrieve the Project id is by finding your current project at https://app.datarobot.com under Manage Projects. If you examine the URL now, you can get the Project id."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`https://app.datarobot.com/projects/<PROJECT ID>/`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shotProject = dr.Project.get(\n",
    "    project_id = \"___\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Module 2.1: Further model interpretation and understanding "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore models that have been built in Autopilot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We concluded Session A by running autopilot to build out many different models for this NBA Shots project. Let's now start to investigate these models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get list of trained models\n",
    "shotProjectModels = shotProject.___()\n",
    "\n",
    "len(shotProjectModels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pick an arbitrary model, e.g., the 7th model (index 6)\n",
    "someModel = shotProjectModels[___]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What's the name of this model?\n",
    "someModel.___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What are the available metric scores for this model?\n",
    "someModel.___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What is the optimization metric used for this project?\n",
    "shotProject.___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# How do we get a specific model's metrics for the current project metric?\n",
    "someModel.___[shotProject.___]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What is the name of the featurelist for this model?\n",
    "someModel.___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What percentage of the data was used for training with this model?\n",
    "someModel.___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is fine for one model, but will be harder to analyze for comparing multiple models. Let's build a function to extract the Leaderboard as a DataFrame!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a custom function to extract Leaderboard results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def retrieveLeaderboard(project, metric = None):\n",
    "    \n",
    "    # Set default metric if not passed\n",
    "    if metric is None:\n",
    "        metric = project.metric\n",
    "        \n",
    "    # Create an empty list to store the leaderboard to start\n",
    "    leaderboard = []\n",
    "    \n",
    "    # Get all of the models trained so far as we did above\n",
    "    models = project.___()\n",
    "    \n",
    "    # Iterate over each of the models extracting different\n",
    "    # pieces of relevant information as we did above\n",
    "    for model in models:\n",
    "        \n",
    "        # Store the results for a specific metric\n",
    "        temp = model.metrics[metric]\n",
    "        \n",
    "        # Store the name of the model algorithm\n",
    "        temp[\"algorithm\"] = model.model_type\n",
    "        \n",
    "        # Store the id of the model\n",
    "        temp[\"model_id\"] = model.___\n",
    "        \n",
    "        # Store the feature list used to create the model\n",
    "        temp[\"featurelist\"] = model.___\n",
    "        \n",
    "        # Store what % was used for training\n",
    "        temp[\"sample_pct\"] = model.sample_pct\n",
    "        \n",
    "        # Append this list to leaderboard and move to next model\n",
    "        leaderboard.append(temp)\n",
    "        \n",
    "    # Store leaderboard list as a pandas DataFrame    \n",
    "    leaderboardDf = pd.DataFrame(leaderboard)[\n",
    "        [\"algorithm\", \"model_id\", \"featurelist\", \"sample_pct\", \n",
    "         \"validation\", \"crossValidation\", \"holdout\"]\n",
    "    ]\n",
    "    \n",
    "    # Return this leaderboard to explore further\n",
    "    return ___  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make the resulting table a little easier to read, we can specify a pandas option for 5 decimal point values and extend the maximum column width."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change pandas display format for floating point numbers\n",
    "pd.options.display.float_format = '{:,.5f}'.format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Increase maximum column width\n",
    "pd.set_option('max_colwidth', 70)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identify key pieces of information about the Leaderboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shotLb = ___(project = ___)\n",
    "shotLb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To sort based on a different metric than the default (for this project, LogLoss)\n",
    "retrieveLeaderboard(shotProject, '___')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the 80% and 100% models use different techniques to calculate approximations for validation and cross-validation scores, let's turn our focus only to models using 64% or smaller for training sample size and examine the top performing models there. For the sake of computation time throughout class, let's also remove Blender models going forward. This also gives us a chance to play a bit with pandas DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note the use of:\n",
    "# & for \"and\" (use | for \"or\") \n",
    "# ~ (the invert operator) for \"not\"\n",
    "shot64 = shotLb[\n",
    "    (shotLb.___ <= 64) &  ~(shotLb.___.str.contains(\"___\"))\n",
    "]\n",
    "shot64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can get the model ID of the top model and then retrieve it using the `.get()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top64ModelId = ___.___.___\n",
    "top64ModelId"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top64Model = ___.___.___(\n",
    "    project = ___,\n",
    "    model_id = ___\n",
    ")\n",
    "top64Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### YOUR TASK (Exercise 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a customized lift chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While there are functions available in the datarobot Python package to download the data to create many different model interpretation plots you see in DataRobot such as `get_lift_chart()`, `get_confusion_chart()`, etc., we can also create our own custom plots that can directly relate to our business use case. \n",
    "\n",
    "**We are interested in creating a customized lift chart showing the likelihood of making a shot versus distance from the hoop.**  \n",
    "\n",
    "To do so, we'll need to use the `.request_training_predictions()` method with `.get_result_when_complete()` and specify which subset of our historical data we'd like predictions for. We'll focus on downloading predictions from the validation partition but there are other options too."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download validation predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "?top64Model.request_training_predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainPredJob = top64Model.request_training_predictions(\n",
    "    data_subset = 'validationAndHoldout'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top64ModelPreds = ___.get_result_when_complete()\n",
    "top64ModelPreds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top64PredsDF = ___.___()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top64PredsDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have both the Validation and Holdout predictions for each row, let's focus on the subset of this data we are most interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validationPreds = top64PredsDF[top64PredsDF.partition_id != \"Holdout\"][[\"row_id\", \"class_made\"]]\n",
    "validationPreds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(validationPreds.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This value of 857 rows can be used as a check for when we filter `shotData` to focus only on the `row_id` values given above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Re)load actual shot results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load data again from text file (using row id as index)\n",
    "shotData = pd.read_csv('shot_logs_wed.csv')\n",
    "# Focus on columns of interest\n",
    "shotDataSmall = shotData[[\"SHOT_DIST\", \"FGM\"]]\n",
    "shotDataSmall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Join prediction and actual data together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to link our two tables together so that we can look at actual values (`SHOT_RESULT`) and how that compares to the predicted values for each row in the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predAndAct = pd.merge(\n",
    "    left = ___,\n",
    "    right = ___,\n",
    "    left_index = True,\n",
    "    right_on = \"row_id\"\n",
    ")\n",
    "predAndAct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sanity check - we verify that the above merged dataframe has the same number of rows as the dataframe containing predictions: 857 rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### YOUR TASK (Exercise 7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create binned version of shot distance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use our knowledge (maybe newfounded!) of the NBA basketball court and types of shots to help us understand how the percentage of made shots changes as we bin the values of `SHOT_DIST` into categories in our defined way. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we can create some bins and give them names corresponding to different shot distances. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shotLabels = [\"Layup 2\", \"Mid-range 2\", \"Long-range 2\", \"3\", \"4\"]\n",
    "cutRanges = [0, 4, 15, 23.75, 30, 94]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predAndAct[\"shotRangeCat\"] = pd.cut(\n",
    "    x = predAndAct[\"___\"],\n",
    "    bins = ___,\n",
    "    labels = ___\n",
    ")\n",
    "predAndAct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find average predicted and actual values across each bin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to compute the average predicted and actual values across each shot range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "binned = predAndAct.groupby(\"___\").agg(\n",
    "    predicted = (\"___\", \"___\"),\n",
    "    actual = (\"___\", \"mean\"),\n",
    "    count = (\"FGM\", \"count\")\n",
    ")\n",
    "binned"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create visualization of model interpretation results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Increase size of matplotlib plots\n",
    "plt.rcParams['figure.figsize'] = [10, 5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(\n",
    "    ___,\n",
    "    ___,\n",
    "    '+-'\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(\n",
    "    binned.index,\n",
    "    binned.predicted,\n",
    "    '+-'\n",
    ")\n",
    "plt.plot(\n",
    "    binned.index,\n",
    "    binned.___,\n",
    "    '___',\n",
    "    fillstyle = 'none'\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(\n",
    "    binned.index,\n",
    "    binned.predicted,\n",
    "    '+-'\n",
    ")\n",
    "plt.plot(\n",
    "    binned.index,\n",
    "    binned.___,\n",
    "    'o-',\n",
    "    fillstyle = 'none'\n",
    ")\n",
    "plt.legend(labels = ['___', '___']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### YOUR TASK (Exercise 8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Module 2.2: Advanced feature selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By the end of this mission, you will be able to:\n",
    "\n",
    "- ~~Connect to the DataRobot client using an API key~~\n",
    "- ~~Create a project in DataRobot programmatically~~\n",
    "- ~~Set the target feature for a DataRobot project~~\n",
    "- ~~Download and review DataRobot created featurelists~~\n",
    "- ~~Extract and customize the features of a featurelist~~\n",
    "- ~~Review the properties of a DataRobot project Repository~~\n",
    "- ~~Build a new model from a blueprint and custom featurelist~~\n",
    "- ~~Start autopilot with the maximum number of workers~~\n",
    "\n",
    "\n",
    "\n",
    "- ~~List all models trained during autopilot~~\n",
    "- ~~Create a custom function in Python to extract Leaderboard results~~\n",
    "- ~~Get training predictions for a model~~\n",
    "- ~~Create a custom lift chart to aggregate predicted and actual results~~\n",
    "- Retrieve Feature Impact for top performing models\n",
    "- Build a model with reduced features based on Feature Impact\n",
    "- Make predictions on scoring data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a new feature list by considering Feature Impact across the top performing models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A clever way to improve the performance of the models in the leaderboard would be to \n",
    "\n",
    "1. Obtain feature impact for the best performing models\n",
    "2. Generate a new feature list by combining the most impactful features across the best performing models\n",
    "3. Retrain the top performing with this new feature list\n",
    "\n",
    "That will be the goal of this module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first retrieve the top three models from those built with 64% training that are not blenders trained with our modified leakage removed featurelist. These will be sorted based on validation score. This could be done by filtering the `nonblendersDF`, but it's sometimes helpful to work directly with the model objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve Feature Impact for best performing models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get all models\n",
    "allModels = shotProject.get_models()\n",
    "\n",
    "# Focus on top 3 models\n",
    "filteredModels = [\n",
    "    model \n",
    "    for model in allModels \n",
    "    if (\"Blender\" not in model.model_type) &\n",
    "       (model.sample_pct < 65)\n",
    "][0:3]\n",
    "filteredModels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's retrieve the Feature Impact scores and rankings for these three models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create DataFrame to store results\n",
    "allImpact = pd.DataFrame()\n",
    "\n",
    "# Walk through a for loop iterating over each model\n",
    "for model in ___:\n",
    "    \n",
    "    # This can take a minute (for each)\n",
    "    featureImpact = model.___(max_wait = 600)\n",
    "    \n",
    "    # Ready to be converted to DF\n",
    "    df = pd.___(featureImpact)\n",
    "    # Track model name and ID for bookkeeping purposes\n",
    "    df['model_type'] = model.model_type\n",
    "    df['model_id'] = model.id\n",
    "    \n",
    "    # By sorting and re-indexing, the new index becomes our 'ranking'\n",
    "    df = df.sort_values(by = 'impactUnnormalized', ascending = False)\n",
    "    df = df.reset_index(drop = True)\n",
    "    df['rank'] = df.index.values\n",
    "    \n",
    "    # Add to our master list of all models' feature ranks\n",
    "    allImpact = pd.___([allImpact, df], ignore_index = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### YOUR TASK (Exercise 9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build a new model with the top 10 features from across the best performing models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Examine allImpact DataFrame\n",
    "allImpact"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's determine the top 10 features from the collection of models in the `allImpact` DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 1: Rank impact across models with a grouping feature and an aggregation function\n",
    "# Step 2: Sort the values with the most impactful at the top\n",
    "# Step 3: Grab only the top X features (here, we'll use 10)\n",
    "# Step 4: Pull just the values of the indexing column (the grouping feature)\n",
    "\n",
    "allImpact#.groupby('___').median()#.sort_values('___')#.head(___)#.index.valuesb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then let's extract  a list of these features from the `allImpact` DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Turn the array made by the steps above into a list\n",
    "top10Feats = list(\n",
    "    allImpact\n",
    "    .groupby('___').median()\n",
    "    .sort_values('___').head(___)\n",
    "    .index.values\n",
    ")\n",
    "top10Feats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create a featurelist from these features with the goal to see the performance of a newly created model on this featurelist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top10Aggregated = ___.___(\n",
    "    name = 'top 10 Aggregated', \n",
    "    features = ___\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how a newly created model based on the `bestModel` in `filteredModels` performs on this `top 10 Aggregated` featurelist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bestModel = filteredModels[0]\n",
    "bestModel.___(\n",
    "    featurelist_id = ___.___,\n",
    "    scoring_type = \"crossValidation\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the new model finishing building with cross-validation, we can go to the Leaderboard in the app to compare the model built with the **top 10 Aggregated** featurelist to the similar one built with **Modified Informative Features - Leakage Removed**. \n",
    "\n",
    "Alternatively, we can view the Leaderboard with our `retrieveLeaderboard()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "retrieveLeaderboard(___).head(___)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the validation and cross-validation scores are similar. If we are concerned with speed of prediction more than just model performance, we may want to consider using this newly created model to make predictions going forward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### YOUR TASK (Exercise 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make predictions on new data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `shot_logs_sat.csv` file that you uploaded to the Colab environment can be used for making predictions. First, we load the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasetFromPath = shotProject.___(\"shot_logs_sat.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go into https://app.datarobot.com to get the `model_id` from the URL of the newly created model from `bestModel` in `filteredModels` on the `top 10 Aggregated` featurelist. We'll make predictions using this model next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prodModel = dr.Model.get(\n",
    "    project = ___,\n",
    "    model_id = \"___\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we set the prediction threshold to whichever value we feel comfortable saying is a made shot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "___.set_prediction_threshold(___)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we request predictions and get the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictJob = prodModel.___(datasetFromPath.id)\n",
    "predictions = predictJob.get_result_when_complete()\n",
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### YOUR TASK (Exercise 11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning objectives\n",
    "\n",
    "By now, you should be able to:\n",
    "\n",
    "- Connect to the DataRobot client using an API key\n",
    "- Create a project in DataRobot programmatically\n",
    "- Set the target feature for a DataRobot project\n",
    "- Download and review DataRobot created featurelists\n",
    "- Extract and customize the features of a featurelist\n",
    "- Review the properties of a DataRobot project Repository\n",
    "- Build a new model from a blueprint and custom featurelist\n",
    "- Start autopilot with the maximum number of workers\n",
    "\n",
    "\n",
    "- List all models trained during autopilot\n",
    "- Create a custom function in Python to extract Leaderboard results\n",
    "- Get training predictions for a model\n",
    "- Create a custom lift chart to aggregate predicted and actual results\n",
    "- Retrieve Feature Impact for top performing models\n",
    "- Build a model with reduced features based on Feature Impact\n",
    "- Make predictions on scoring data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What's Next?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Final project_ : Extend the idea of feature selection by working through this example on FIRE (Feature Importance Rank Ensembling) from [this](https://github.com/datarobot-community/examples-for-data-scientists/blob/master/Feature%20Lists%20Manipulation/Python/FeatureSelection_using_Feature_Importance_Rank_Ensembling.ipynb) DataRobot Community Jupyter notebook on GitHub."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Course Survey*: on university.datarobot.com [here](https://university.datarobot.com/advanced-datarobot-with-python/548199) or directly on SurveyMonkey [here](https://www.surveymonkey.com/r/APIIPython). Thanks for your feedback!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus: For NBA Shots - separating out Long 2s from Corner 3s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For shots between 22 and 23.75, in order to separate long 2's from short 3's,\n",
    "# we need to know what type of shot it is. So we need to add PTS_TYPE to shotDataSmall.\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "#shot_data = pd.read_csv(\"shot_logs_wed.csv\")\n",
    "shotDataSmall = shotData[[\"SHOT_DIST\", \"FGM\", \"PTS_TYPE\"]]\n",
    "\n",
    "# Now we need to re-merge with the predictions to include PTS_TYPE\n",
    "predAndAct = pd.merge(\n",
    "    left = shotDataSmall,\n",
    "    right = validationPreds,\n",
    "    left_index = True,\n",
    "    right_on = \"row_id\"\n",
    ")\n",
    "\n",
    "# Now split long-range 2s based on PTS_TYPE and create the corner 3\n",
    "predAndAct[\"shortRangeCat\"] = np.select(\n",
    "    [\n",
    "        predAndAct[\"SHOT_DIST\" <= 4],\n",
    "        predAndAct[\"SHOT_DIST\" <= 15],\n",
    "        np.where((predAndAct[\"SHOT_DIST\" <= 23.75) & (predAndAct[\"PTS_TYPE\" == 2), True, False),\n",
    "        np.where((predAndAct[\"SHOT_DIST\" <= 23.75) & (predAndAct[\"PTS_TYPE\" == 3), True, False),\n",
    "        predAndAct[\"SHOT_DIST\" <= 30],\n",
    "        predAndAct[\"SHOT_DIST\" <= 94]\n",
    "    ],\n",
    "    [\n",
    "        \"a Layup 2\",\n",
    "        \"b Mid-range 2\",\n",
    "        \"c Long-range 2\",\n",
    "        \"d Corner 3\",\n",
    "        \"e Standard 3\",\n",
    "        \"f 4\"\n",
    "    ],\n",
    "    default = \"unknown\"\n",
    ")\n",
    "predAndAct.head(20)                             "
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "MLOps III - DRUM",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}