2-training-inference/2.1-aml-training-and-hyperdrive/2-scikit-learn-remote-training-on-aml-compute-plus-hyperdrive/binayclassification-employee-attrition-aml-compute-notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scikit-Learn binary classification model. Remote train via Azure ML Compute (AML Cluster) and optional HyperDrive (Hyper-parameter tunning with multiple child runs) \n",
    "_**This notebook showcases the creation of a ScikitLearn Binary classification model by remotely training on Azure ML Compute Target (AMLCompute Cluster).**_\n",
    "\n",
    "_**It shows multiple ways of remote training like using a single Estimator, a ScriptRunConfig and hyper-parameter tunning with HyperDrive with multiple child trainings**_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check library versions\n",
    "This is important when interacting with different executions between remote compute environments (cluster) and the instance/VM with the Jupyter Notebook.\n",
    "If not using the same versions you can have issues when creating .pkl files in the cluster and downloading them to load it in the Jupyter notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Azure SDK version: 1.0.76\n",
      "scikit-learn version is 0.20.3.\n",
      "joblib version is 0.14.0.\n",
      "pandas version is 0.23.4.\n"
     ]
    }
   ],
   "source": [
    "# Check versions\n",
    "import azureml.core\n",
    "import sklearn\n",
    "import joblib\n",
    "import pandas\n",
    "\n",
    "print(\"Azure SDK version:\", azureml.core.VERSION)\n",
    "print('scikit-learn version is {}.'.format(sklearn.__version__))\n",
    "print('joblib version is {}.'.format(joblib.__version__))\n",
    "print('pandas version is {}.'.format(pandas.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup and connect to AML Workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cesardl-automl-northcentralus-ws\n",
      "automlpmdemo\n",
      "northcentralus\n",
      "102a16c3-37d3-48a8-9237-4c9b1e8e80e0\n"
     ]
    }
   ],
   "source": [
    "from azureml.core import Workspace\n",
    "\n",
    "ws = Workspace.from_config()\n",
    "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep='\\n')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create An Experiment\n",
    "\n",
    "**Experiment** is a logical container in an Azure ML Workspace. It hosts run records which can include run metrics and output artifacts from your experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Experiment\n",
    "experiment_name = 'classif-attrition-amlcompute'\n",
    "experiment = Experiment(workspace=ws, name=experiment_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to AmlCompute\n",
    "\n",
    "Azure Machine Learning Compute is managed compute infrastructure that allows the user to easily create single to multi-node compute of the appropriate VM Family. It is created **within your workspace region** and is a resource that can be used by other users in your workspace. It autoscales by default to the max_nodes, when a job is submitted, and executes in a containerized environment packaging the dependencies as specified by the user. \n",
    "\n",
    "Since it is managed compute, job scheduling and cluster management are handled internally by Azure Machine Learning service. \n",
    "\n",
    "For more information on Azure Machine Learning Compute, please read [this article](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)\n",
    "\n",
    "**Note**: As with other Azure services, there are limits on certain resources (for eg. AmlCompute quota) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create project directory and copy the training script into the project directory\n",
    "\n",
    "Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script, and any additional files your training script depends on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./classif-attrition-amlcompute/train.py'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import shutil\n",
    "\n",
    "project_folder = './classif-attrition-amlcompute'\n",
    "os.makedirs(project_folder, exist_ok=True)\n",
    "\n",
    "# Copy the training script into the project directory\n",
    "shutil.copy('train.py', project_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect or Create a Remote AML compute cluster\n",
    "\n",
    "Try to use the compute target you had created before (make sure you provide the same name here in the variable `cpu_cluster_name`).\n",
    "If not available, create a new cluster from the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found existing training cluster.\n",
      "Checking cluster status...\n",
      "Succeeded\n",
      "AmlCompute wait for completion finished\n",
      "Minimum number of nodes requested have been provisioned\n"
     ]
    }
   ],
   "source": [
    "# Define remote compute target to use\n",
    "# Further docs on Remote Compute Target: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-remote\n",
    "\n",
    "# Choose a name for your cluster.\n",
    "amlcompute_cluster_name = \"cpu-cluster\"\n",
    "\n",
    "found = False\n",
    "# Check if this compute target already exists in the workspace.\n",
    "cts = ws.compute_targets\n",
    "\n",
    "if amlcompute_cluster_name in cts and cts[amlcompute_cluster_name].type == 'AmlCompute':\n",
    "     found = True\n",
    "     print('Found existing training cluster.')\n",
    "     # Get existing cluster\n",
    "     # Method 1:\n",
    "     aml_remote_compute = cts[amlcompute_cluster_name]\n",
    "     # Method 2:\n",
    "     # aml_remote_compute = ComputeTarget(ws, amlcompute_cluster_name)\n",
    "    \n",
    "if not found:\n",
    "     print('Creating a new training cluster...')\n",
    "     provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D13_V2\", # for GPU, use \"STANDARD_NC12\"\n",
    "                                                                 #vm_priority = 'lowpriority', # optional\n",
    "                                                                 max_nodes = 20)\n",
    "     # Create the cluster.\n",
    "     aml_remote_compute = ComputeTarget.create(ws, amlcompute_cluster_name, provisioning_config)\n",
    "    \n",
    "print('Checking cluster status...')\n",
    "# Can poll for a minimum number of nodes and for a specific timeout.\n",
    "# If no min_node_count is provided, it will use the scale settings for the cluster.\n",
    "aml_remote_compute.wait_for_completion(show_output = True, min_node_count = 0, timeout_in_minutes = 20)\n",
    "    \n",
    "# For a more detailed view of current AmlCompute status, use get_status()."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fetch the AML Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "aml_dataset = ws.datasets['IBM-Employee-Attrition']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Environment \n",
    "#### Optionally list all the available environments and packages in your AML Workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name AzureML-Tutorial\n",
      "Name AzureML-Minimal\n",
      "Name AzureML-Chainer-5.1.0-GPU\n",
      "Name AzureML-PyTorch-1.2-CPU\n",
      "Name AzureML-TensorFlow-1.12-CPU\n",
      "Name AzureML-TensorFlow-1.13-CPU\n",
      "Name AzureML-PyTorch-1.1-CPU\n",
      "Name AzureML-TensorFlow-1.10-CPU\n",
      "Name AzureML-PyTorch-1.0-GPU\n",
      "Name AzureML-TensorFlow-1.12-GPU\n",
      "Name AzureML-TensorFlow-1.13-GPU\n",
      "Name AzureML-Chainer-5.1.0-CPU\n",
      "Name AzureML-PyTorch-1.0-CPU\n",
      "Name AzureML-Scikit-learn-0.20.3\n",
      "Name AzureML-PyTorch-1.2-GPU\n",
      "Name AzureML-PyTorch-1.1-GPU\n",
      "Name AzureML-TensorFlow-1.10-GPU\n",
      "Name AzureML-PyTorch-1.3-GPU\n",
      "Name AzureML-TensorFlow-2.0-CPU\n",
      "Name AzureML-PyTorch-1.3-CPU\n",
      "Name AzureML-TensorFlow-2.0-GPU\n",
      "Name AzureML-PySpark-MmlSpark-0.15\n"
     ]
    }
   ],
   "source": [
    "from azureml.core import Environment\n",
    "\n",
    "envs = Environment.list(workspace=ws)\n",
    "\n",
    "# List Environments and packages in my workspace\n",
    "for env in envs:\n",
    "    if env.startswith(\"AzureML\"):\n",
    "        print(\"Name\",env)\n",
    "        #print(\"packages\", envs[env].python.conda_dependencies.serialize_to_string())\n",
    "        \n",
    "# Use curated environment from AML named \"AzureML-Tutorial\"\n",
    "curated_environment = Environment.get(workspace=ws, name=\"AzureML-Tutorial\")  # Custom environment: Environment.get(workspace=ws,name=\"myenv\",version=\"1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Option A.1) Configure & Run using ScriptRunConfig & Environment \n",
    "### (Easiest path using curated environments, but less flexible than Estimator..?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Experiment</th><th>Id</th><th>Type</th><th>Status</th><th>Details Page</th><th>Docs Page</th></tr><tr><td>classif-attrition-amlcompute</td><td>classif-attrition-amlcompute_1578780432_feb64c59</td><td>azureml.scriptrun</td><td>Starting</td><td><a href=\"https://ml.azure.com/experiments/classif-attrition-amlcompute/runs/classif-attrition-amlcompute_1578780432_feb64c59?wsid=/subscriptions/102a16c3-37d3-48a8-9237-4c9b1e8e80e0/resourcegroups/automlpmdemo/workspaces/cesardl-automl-northcentralus-ws\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.script_run.ScriptRun?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "Run(Experiment: classif-attrition-amlcompute,\n",
       "Id: classif-attrition-amlcompute_1578780432_feb64c59,\n",
       "Type: azureml.scriptrun,\n",
       "Status: Starting)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add training script to run config\n",
    "from azureml.core import ScriptRunConfig, RunConfiguration, Experiment\n",
    "\n",
    "script_runconfig = ScriptRunConfig(source_directory=project_folder, \n",
    "                            script=\"train.py\",\n",
    "                            arguments=[aml_dataset.as_named_input('attrition')]\n",
    "                           )\n",
    "\n",
    "# Attach compute target to run config\n",
    "script_runconfig.run_config.target = cluster\n",
    "# runconfig.run_config.target = \"local\"\n",
    "\n",
    "# Attach environment to run config\n",
    "script_runconfig.run_config.environment = curated_environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A.2 (Optional) Run the experiment with a single ScriptRunConfig and single run (Optional Jump)\n",
    "Jump this step if you want to use HyperDrive (Go to Option C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Submit the Experiment Run to the AML Compute \n",
    "run = experiment.submit(script_runconfig)\n",
    "run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Option B.1.a) Configure an Estimator with specific pkgs versions (Using pip and conda) - Risky!\n",
    "Risks: Overriding remote compute Docker image packages with pip and conda might cause issues with inconsistent package versions.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.train.estimator import Estimator\n",
    "from azureml.train.sklearn import SKLearn\n",
    "\n",
    "script_params = {\n",
    "    '--solver': 'liblinear',\n",
    "    '--penalty': 'l2'\n",
    "}\n",
    "\n",
    "pip_packages = [\n",
    "                'azureml-defaults==1.0.76', 'azureml-core==1.0.76', 'azureml-telemetry==1.0.76',\n",
    "                'azureml-dataprep==1.1.31', 'joblib==0.14.0', 'sklearn-pandas==1.7.0', 'pandas==0.23.4' \n",
    "               ]\n",
    "\n",
    "# Using plain Estimator class\n",
    "estimator = Estimator(source_directory=project_folder, \n",
    "                      script_params=script_params,\n",
    "                      compute_target=aml_remote_compute,\n",
    "                      entry_script='train.py',\n",
    "                      pip_packages=pip_packages,\n",
    "                      conda_packages=['scikit-learn==0.20.3'],\n",
    "                      inputs=[ws.datasets['IBM-Employee-Attrition'].as_named_input('attrition')])\n",
    "\n",
    "\n",
    "# Using SKLearn estimator class\n",
    "# estimator = SKLearn(source_directory=project_folder, \n",
    "#                     script_params=script_params,\n",
    "#                     compute_target=aml_remote_compute,\n",
    "#                     entry_script='train.py',\n",
    "#                     pip_packages=pip_packages,\n",
    "#                     conda_packages=['scikit-learn==0.20.3'],\n",
    "#                     inputs=[aml_dataset.as_named_input('attrition')])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Option B.1.b) Configure an Estimator with Environment (Better! Easier! Consistent!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Copy the curated environment to a new custom environment and register it into your Workspace\n",
    "##### You can run this step just once since the custom Environment will be registered in your Workspace after this step. \n",
    "\n",
    "Need to copy the curated environment into a custom environment which will be updated by the Estimator.\n",
    "\n",
    "If you try to directly use a curated environment with an Estimator you'll get the error:\n",
    "Error: \"Environment name can not start with the prefix AzureML. To alter a curated environment first create a copy of it.\"\n",
    "\n",
    "#### This a potential PAPER-CUT: \n",
    "- Copying a curated environment to a custom environment and registering is not straight forward! You need to save Conda dependecies as a file, load it from there, etc.\n",
    "- An Estimator should be able to use a curated environment, too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Environment Name custom-workshop-environment\n",
      "packages channels:\n",
      "- conda-forge\n",
      "dependencies:\n",
      "- python=3.6.2\n",
      "- pip:\n",
      "  - azureml-core==1.0.81.1\n",
      "  - azureml-defaults==1.0.81\n",
      "  - azureml-telemetry==1.0.81.1\n",
      "  - azureml-train-restclients-hyperdrive==1.0.81\n",
      "  - azureml-train-core==1.0.81\n",
      "  - azureml-widgets==1.0.81\n",
      "  - azureml-pipeline-core==1.0.81\n",
      "  - azureml-pipeline-steps==1.0.81\n",
      "  - azureml-opendatasets==1.0.81\n",
      "  - azureml-automl-core==1.0.81.1\n",
      "  - azureml-automl-runtime==1.0.81\n",
      "  - azureml-train-automl-client==1.0.81.1\n",
      "  - azureml-train-automl-runtime==1.0.81\n",
      "  - azureml-train-automl==1.0.81\n",
      "  - azureml-train==1.0.81\n",
      "  - azureml-sdk==1.0.81\n",
      "  - azureml-interpret==1.0.81\n",
      "  - azureml-tensorboard==1.0.81\n",
      "  - azureml-mlflow==1.0.81\n",
      "  - mlflow\n",
      "  - sklearn-pandas\n",
      "- pandas\n",
      "- numpy\n",
      "- tqdm\n",
      "- scikit-learn\n",
      "- matplotlib\n",
      "name: azureml_9c3b026f45c1aef228bf1fb00f90ec20\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# (CDLTLL) POTENTIAL PAPER-CUT: Copying a curated environment to a custom environment and registering is not straight forward!\n",
    "\n",
    "# Save curated environment definition to folder (Two files, one for conda_dependencies.yml and another file for azureml_environment.json)\n",
    "curated_environment.save_to_directory(path=\"./curated_environment_definition\", overwrite=True)\n",
    "\n",
    "# Create custom Environment from Conda specification file\n",
    "custom_environment = Environment.from_conda_specification(name=\"custom-workshop-environment\", file_path=\"./curated_environment_definition/conda_dependencies.yml\")\n",
    "\n",
    "# Save curated environment definition to folder (Two files, one for conda_dependencies.yml and another file for azureml_environment.json)\n",
    "custom_environment.save_to_directory(path=\"./custom_environment_definition\", overwrite=True)\n",
    "\n",
    "custom_environment.register(ws)\n",
    "\n",
    "envs = Environment.list(workspace=ws)\n",
    "\n",
    "# List Environments and packages in my workspace\n",
    "for env in envs:\n",
    "    if env.startswith(\"custom\"):\n",
    "        print(\"Environment Name\",env)\n",
    "        print(\"packages\", envs[env].python.conda_dependencies.serialize_to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use Estimator with Custom Environment "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Environment(Name: custom-workshop-environment,\n",
      "Version: 1)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - This compute target type doesn't support non-Docker runs; overriding run configuration enable Docker.\n"
     ]
    }
   ],
   "source": [
    "from azureml.train.estimator import Estimator\n",
    "from azureml.train.sklearn import SKLearn\n",
    "\n",
    "# Need to use custom environment with an Estimator as the Estimator will updated it when creating a Docker Image even if you don't have new pip or Conda packages.\n",
    "# If you try to directly use a curated environment with an Estimator you'll get the error:\n",
    "# Error: \"Environment name can not start with the prefix AzureML. To alter a curated environment first create a copy of it.\"\n",
    "\n",
    "# Load Custom Environment from Workspace\n",
    "custom_environment = Environment.get(workspace=ws,name=\"custom-workshop-environment\")  # ,version=\"1\"\n",
    "print(custom_environment)\n",
    "\n",
    "script_params = {\n",
    "    '--solver': 'liblinear',\n",
    "    '--penalty': 'l2'\n",
    "}\n",
    "\n",
    "# Using plain Estimator class with custom Environment\n",
    "estimator = Estimator(source_directory=project_folder, \n",
    "                      script_params=script_params,\n",
    "                      compute_target=aml_remote_compute,\n",
    "                      # use_docker=True, #AML Cluster only supports Docker runs\n",
    "                      entry_script='train.py',\n",
    "                      environment_definition= custom_environment,\n",
    "                      inputs=[ws.datasets['IBM-Employee-Attrition'].as_named_input('attrition')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Option B.2) Run the experiment with a single Estimator (Optional Jump)\n",
    "Jump this step if you want to use HyperDrive (Go to Option B2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Experiment</th><th>Id</th><th>Type</th><th>Status</th><th>Details Page</th><th>Docs Page</th></tr><tr><td>classif-attrition-amlcompute</td><td>classif-attrition-amlcompute_1578780933_0428d479</td><td>azureml.scriptrun</td><td>Starting</td><td><a href=\"https://ml.azure.com/experiments/classif-attrition-amlcompute/runs/classif-attrition-amlcompute_1578780933_0428d479?wsid=/subscriptions/102a16c3-37d3-48a8-9237-4c9b1e8e80e0/resourcegroups/automlpmdemo/workspaces/cesardl-automl-northcentralus-ws\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.script_run.ScriptRun?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "Run(Experiment: classif-attrition-amlcompute,\n",
       "Id: classif-attrition-amlcompute_1578780933_0428d479,\n",
       "Type: azureml.scriptrun,\n",
       "Status: Starting)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run = experiment.submit(estimator)\n",
    "run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monitor Run (Optional Jump)\n",
    "Jump this step if you want to use HyperDrive (Go to Option B2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "307b17b5321d4e1da110e8779fb4b002",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_UserRunWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'NOTSET',…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Running\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/classif-attrition-amlcompute/runs/classif-attrition-amlcompute_1578780933_0428d479?wsid=/subscriptions/102a16c3-37d3-48a8-9237-4c9b1e8e80e0/resourcegroups/automlpmdemo/workspaces/cesardl-automl-northcentralus-ws\", \"run_id\": \"classif-attrition-amlcompute_1578780933_0428d479\", \"run_properties\": {\"run_id\": \"classif-attrition-amlcompute_1578780933_0428d479\", \"created_utc\": \"2020-01-11T22:15:35.501324Z\", \"properties\": {\"_azureml.ComputeTargetType\": \"amlcompute\", \"ContentSnapshotId\": \"318fc238-6e9d-4aac-8d44-89500509bbb9\", \"azureml.git.repository_uri\": \"https://github.com/CESARDELATORRE/azureml-workshop-2019.git\", \"mlflow.source.git.repoURL\": \"https://github.com/CESARDELATORRE/azureml-workshop-2019.git\", \"azureml.git.branch\": \"master\", \"mlflow.source.git.branch\": \"master\", \"azureml.git.commit\": \"a402da72246e3addd4c81c8a237a21d754c583b9\", \"mlflow.source.git.commit\": \"a402da72246e3addd4c81c8a237a21d754c583b9\", \"azureml.git.dirty\": \"True\", \"ProcessInfoFile\": \"azureml-logs/process_info.json\", \"ProcessStatusFile\": \"azureml-logs/process_status.json\"}, \"tags\": {\"_aml_system_ComputeTargetStatus\": \"{\\\"AllocationState\\\":\\\"steady\\\",\\\"PreparingNodeCount\\\":2,\\\"RunningNodeCount\\\":1,\\\"CurrentNodeCount\\\":3}\"}, \"script_name\": null, \"arguments\": null, \"end_time_utc\": null, \"status\": \"Running\", \"log_files\": {\"azureml-logs/55_azureml-execution-tvmps_5bc40263223408b9ea41f79b161fdb30f0cd3ae739c0c3bb9228ef2705518174_d.txt\": \"https://cesardlautomln5648400225.blob.core.windows.net/azureml/ExperimentRun/dcid.classif-attrition-amlcompute_1578780933_0428d479/azureml-logs/55_azureml-execution-tvmps_5bc40263223408b9ea41f79b161fdb30f0cd3ae739c0c3bb9228ef2705518174_d.txt?sv=2019-02-02&sr=b&sig=HO%2BmybL0w7oxbCaym8CJ9KAW50wiGQVrJwEZNQv0c9M%3D&st=2020-01-11T22%3A08%3A30Z&se=2020-01-12T06%3A18%3A30Z&sp=r\"}, \"log_groups\": [[\"azureml-logs/55_azureml-execution-tvmps_5bc40263223408b9ea41f79b161fdb30f0cd3ae739c0c3bb9228ef2705518174_d.txt\"]], \"run_duration\": \"0:02:55\"}, \"child_runs\": [], \"children_metrics\": {}, \"run_metrics\": [], \"run_logs\": \"2020-01-11T22:17:40Z Starting output-watcher...\\nLogin Succeeded\\nUsing default tag: latest\\nlatest: Pulling from azureml/azureml_5bb2715cb1ae959eaf94451122757b70\\na1298f4ce990: Pulling fs layer\\n04a3282d9c4b: Pulling fs layer\\n9b0d3db6dc03: Pulling fs layer\\n8269c605f3f1: Pulling fs layer\\n6504d449e70c: Pulling fs layer\\n4e38f320d0d4: Pulling fs layer\\nb0a763e8ee03: Pulling fs layer\\n11917a028ca4: Pulling fs layer\\na6c378d11cbf: Pulling fs layer\\n8269c605f3f1: Waiting\\n6cc007ad9140: Pulling fs layer\\n6c1698a608f3: Pulling fs layer\\n470beb826e90: Pulling fs layer\\nb3801d1554a4: Pulling fs layer\\n4749015222a2: Pulling fs layer\\n4b3f33d906f2: Pulling fs layer\\ncf9062f3cd62: Pulling fs layer\\n8412f94f6bc4: Pulling fs layer\\n6504d449e70c: Waiting\\n4e38f320d0d4: Waiting\\nb0a763e8ee03: Waiting\\n11917a028ca4: Waiting\\na6c378d11cbf: Waiting\\n6cc007ad9140: Waiting\\n4b3f33d906f2: Waiting\\nb3801d1554a4: Waiting\\ncf9062f3cd62: Waiting\\n6c1698a608f3: Waiting\\n8412f94f6bc4: Waiting\\n4749015222a2: Waiting\\n470beb826e90: Waiting\\n04a3282d9c4b: Verifying Checksum\\n04a3282d9c4b: Download complete\\n9b0d3db6dc03: Verifying Checksum\\n9b0d3db6dc03: Download complete\\n8269c605f3f1: Verifying Checksum\\n8269c605f3f1: Download complete\\n4e38f320d0d4: Verifying Checksum\\n4e38f320d0d4: Download complete\\na1298f4ce990: Verifying Checksum\\na1298f4ce990: Download complete\\n6504d449e70c: Download complete\\n11917a028ca4: Verifying Checksum\\n11917a028ca4: Download complete\\nb0a763e8ee03: Verifying Checksum\\nb0a763e8ee03: Download complete\\n6cc007ad9140: Verifying Checksum\\n6cc007ad9140: Download complete\\n6c1698a608f3: Download complete\\na1298f4ce990: Pull complete\\n04a3282d9c4b: Pull complete\\n9b0d3db6dc03: Pull complete\\n8269c605f3f1: Pull complete\\n470beb826e90: Verifying Checksum\\n470beb826e90: Download complete\\n4749015222a2: Download complete\\nb3801d1554a4: Verifying Checksum\\nb3801d1554a4: Download complete\\na6c378d11cbf: Verifying Checksum\\na6c378d11cbf: Download complete\\n4b3f33d906f2: Verifying Checksum\\n4b3f33d906f2: Download complete\\n8412f94f6bc4: Verifying Checksum\\n8412f94f6bc4: Download complete\\n6504d449e70c: Pull complete\\n4e38f320d0d4: Pull complete\\nb0a763e8ee03: Pull complete\\n11917a028ca4: Pull complete\\na6c378d11cbf: Pull complete\\n6cc007ad9140: Pull complete\\n6c1698a608f3: Pull complete\\n470beb826e90: Pull complete\\nb3801d1554a4: Pull complete\\n4749015222a2: Pull complete\\n4b3f33d906f2: Pull complete\\ncf9062f3cd62: Verifying Checksum\\ncf9062f3cd62: Download complete\\n\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"NOTSET\", \"sdk_version\": \"1.0.76\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "RunDetails(run).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get log results upon completion\n",
    "Model training and monitoring happen in the background. Wait until the model has finished training before you run more code. Use wait_for_completion to show when the model training is finished:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run.wait_for_completion(show_output=True)  # specify True for a verbose log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Option C) Configure and Run with Intelligent hyperparameter tuning (HyperDrive using Estimator)\n",
    "\n",
    "IMPORTANT: You need to have created either an Estimator or an ScriptRunConfig in the previous steps (Without submitting the experiment). \n",
    "\n",
    "The adjustable parameters that govern the training process are referred to as the **hyperparameters** of the model. The goal of hyperparameter tuning is to search across various hyperparameter configurations and find the configuration that results in the best performance.\n",
    "\n",
    "To demonstrate how Azure Machine Learning can help you automate the process of hyperarameter tuning, we will launch multiple runs with different values for numbers in the sequence. First let's define the parameter space using random sampling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a hyperparameter sweep\n",
    "First, we will define the hyperparameter space to sweep over. \n",
    "In this example we will use random sampling to try different configuration sets of hyperparameters to maximize our primary metric, Accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from azureml.train.hyperdrive import *\n",
    "from azureml.train.hyperdrive import RandomParameterSampling, BayesianParameterSampling \n",
    "from azureml.train.hyperdrive import BanditPolicy, HyperDriveConfig, PrimaryMetricGoal\n",
    "from azureml.train.hyperdrive import choice, loguniform\n",
    "    \n",
    "# solver{'newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'}, default='lbfgs'\n",
    "# penalty{'l1', 'l2', 'elasticnet', 'none'}, default='l2' --- Note that some penalty parameters are not supported by some algorithms..\n",
    "param_sampling = RandomParameterSampling( {\n",
    "    \"--solver\": choice('newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'),\n",
    "    \"--penalty\": choice('l2')\n",
    "    }\n",
    ")\n",
    "\n",
    "# Details on Scikit-Learn LogisticRegression hyper-parameters:\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will define an early termination policy. The BanditPolicy basically states to check the job every 2 iterations. If the primary metric (defined later) falls outside of the top 10% range, Azure ML terminate the job. This saves us from continuing to explore hyperparameters that don't show promise of helping reach our target metric.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "early_termination_policy = BanditPolicy(evaluation_interval=2, slack_factor=0.1)\n",
    "# Note that early termination policy is currently NOT supported with Bayesian sampling\n",
    "# Check here for recommendations on the multiple policies:\n",
    "# https://docs.microsoft.com/en-us/azure/machine-learning/how-to-tune-hyperparameters#picking-an-early-termination-policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to configure a run configuration object, and specify the primary metric 'Accuracy' that's recorded in your training runs. \n",
    "If you go back to visit the training script, you will notice that this value is being logged. \n",
    "We also want to tell the service that we are looking to maximizing this value. \n",
    "We also set the number of samples to 20, and maximal concurrent job to 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note that in this case when using HyperDrive, the original Estimator's parameters are not used but the HyperDrive parameters...\n",
    "hyperdrive_config = HyperDriveConfig(run_config=script_runconfig, \n",
    "                                     # estimator=estimator,\n",
    "                                     hyperparameter_sampling=param_sampling, \n",
    "                                     policy=early_termination_policy,\n",
    "                                     primary_metric_name='Accuracy',\n",
    "                                     primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,\n",
    "                                     max_total_runs=5,\n",
    "                                     max_concurrent_runs=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, lauch the hyperparameter tuning job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The same input parameter(s) are specified in estimator/run_config script params and HyperDrive parameter space. HyperDrive parameter space definition will override these duplicate entries. ['--solver', '--penalty'] is the list of overridden parameter(s).\n"
     ]
    }
   ],
   "source": [
    "# start the HyperDrive run\n",
    "hyperdrive_run = experiment.submit(hyperdrive_config)\n",
    "\n",
    "# Check here how to submit the hyperdrive run as a step of an AML Pipeline:\n",
    "# https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-parameter-tuning-with-hyperdrive.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor HyperDrive run\n",
    "\n",
    "Monitor the progress of the runs with the Jupyter widget."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a512094ad59a4211995bb5d35529b4c7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_HyperDriveWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'NOTSE…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Running\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/classif-attrition-amlcompute/runs/classif-attrition-amlcompute_1578780735491421?wsid=/subscriptions/102a16c3-37d3-48a8-9237-4c9b1e8e80e0/resourcegroups/automlpmdemo/workspaces/cesardl-automl-northcentralus-ws\", \"run_id\": \"classif-attrition-amlcompute_1578780735491421\", \"run_properties\": {\"run_id\": \"classif-attrition-amlcompute_1578780735491421\", \"created_utc\": \"2020-01-11T22:12:15.697227Z\", \"properties\": {\"primary_metric_config\": \"{\\\"name\\\": \\\"Accuracy\\\", \\\"goal\\\": \\\"maximize\\\"}\", \"resume_from\": \"null\", \"runTemplate\": \"HyperDrive\", \"azureml.runsource\": \"hyperdrive\", \"platform\": \"AML\", \"ContentSnapshotId\": \"318fc238-6e9d-4aac-8d44-89500509bbb9\"}, \"tags\": {\"max_concurrent_jobs\": \"4\", \"max_total_jobs\": \"5\", \"max_duration_minutes\": \"10080\", \"policy_config\": \"{\\\"name\\\": \\\"BANDIT\\\", \\\"properties\\\": {\\\"evaluation_interval\\\": 2, \\\"delay_evaluation\\\": 0, \\\"slack_factor\\\": 0.1}}\", \"generator_config\": \"{\\\"name\\\": \\\"RANDOM\\\", \\\"parameter_space\\\": {\\\"--solver\\\": [\\\"choice\\\", [[\\\"newton-cg\\\", \\\"lbfgs\\\", \\\"liblinear\\\", \\\"sag\\\", \\\"saga\\\"]]], \\\"--penalty\\\": [\\\"choice\\\", [[\\\"l2\\\"]]]}}\", \"primary_metric_config\": \"{\\\"name\\\": \\\"Accuracy\\\", \\\"goal\\\": \\\"maximize\\\"}\", \"platform_config\": \"{\\\"ServiceAddress\\\": \\\"https://northcentralus.experiments.azureml.net\\\", \\\"ServiceArmScope\\\": \\\"subscriptions/102a16c3-37d3-48a8-9237-4c9b1e8e80e0/resourceGroups/automlpmdemo/providers/Microsoft.MachineLearningServices/workspaces/cesardl-automl-northcentralus-ws/experiments/classif-attrition-amlcompute\\\", \\\"SubscriptionId\\\": \\\"102a16c3-37d3-48a8-9237-4c9b1e8e80e0\\\", \\\"ResourceGroupName\\\": \\\"automlpmdemo\\\", \\\"WorkspaceName\\\": \\\"cesardl-automl-northcentralus-ws\\\", \\\"ExperimentName\\\": \\\"classif-attrition-amlcompute\\\", \\\"Definition\\\": {\\\"Overrides\\\": {\\\"script\\\": \\\"train.py\\\", \\\"arguments\\\": [], \\\"target\\\": \\\"cesardl-cpu-clus\\\", \\\"framework\\\": \\\"Python\\\", \\\"communicator\\\": \\\"None\\\", \\\"maxRunDurationSeconds\\\": null, \\\"nodeCount\\\": 1, \\\"environment\\\": {\\\"name\\\": \\\"custom-workshop-environment\\\", \\\"version\\\": \\\"1\\\", \\\"environmentVariables\\\": {\\\"EXAMPLE_ENV_VAR\\\": \\\"EXAMPLE_VALUE\\\"}, \\\"python\\\": {\\\"userManagedDependencies\\\": false, \\\"interpreterPath\\\": \\\"python\\\", \\\"condaDependenciesFile\\\": null, \\\"baseCondaEnvironment\\\": null, \\\"condaDependencies\\\": {\\\"channels\\\": [\\\"conda-forge\\\"], \\\"dependencies\\\": [\\\"python=3.6.2\\\", {\\\"pip\\\": [\\\"azureml-core==1.0.81.1\\\", \\\"azureml-defaults==1.0.81\\\", \\\"azureml-telemetry==1.0.81.1\\\", \\\"azureml-train-restclients-hyperdrive==1.0.81\\\", \\\"azureml-train-core==1.0.81\\\", \\\"azureml-widgets==1.0.81\\\", \\\"azureml-pipeline-core==1.0.81\\\", \\\"azureml-pipeline-steps==1.0.81\\\", \\\"azureml-opendatasets==1.0.81\\\", \\\"azureml-automl-core==1.0.81.1\\\", \\\"azureml-automl-runtime==1.0.81\\\", \\\"azureml-train-automl-client==1.0.81.1\\\", \\\"azureml-train-automl-runtime==1.0.81\\\", \\\"azureml-train-automl==1.0.81\\\", \\\"azureml-train==1.0.81\\\", \\\"azureml-sdk==1.0.81\\\", \\\"azureml-interpret==1.0.81\\\", \\\"azureml-tensorboard==1.0.81\\\", \\\"azureml-mlflow==1.0.81\\\", \\\"mlflow\\\", \\\"sklearn-pandas\\\"]}, \\\"pandas\\\", \\\"numpy\\\", \\\"tqdm\\\", \\\"scikit-learn\\\", \\\"matplotlib\\\"], \\\"name\\\": \\\"azureml_9c3b026f45c1aef228bf1fb00f90ec20\\\"}}, \\\"docker\\\": {\\\"enabled\\\": true, \\\"baseImage\\\": \\\"mcr.microsoft.com/azureml/base:intelmpi2018.3-ubuntu16.04\\\", \\\"baseDockerfile\\\": null, \\\"sharedVolumes\\\": true, \\\"shmSize\\\": null, \\\"arguments\\\": [], \\\"baseImageRegistry\\\": {\\\"address\\\": null, \\\"username\\\": null, \\\"password\\\": null}}, \\\"spark\\\": {\\\"repositories\\\": [], \\\"packages\\\": [], \\\"precachePackages\\\": true}, \\\"databricks\\\": {\\\"mavenLibraries\\\": [], \\\"pypiLibraries\\\": [], \\\"rcranLibraries\\\": [], \\\"jarLibraries\\\": [], \\\"eggLibraries\\\": []}, \\\"inferencingStackVersion\\\": null}, \\\"history\\\": {\\\"outputCollection\\\": true, \\\"snapshotProject\\\": true, \\\"directoriesToWatch\\\": [\\\"logs\\\"]}, \\\"spark\\\": {\\\"configuration\\\": {\\\"spark.app.name\\\": \\\"Azure ML Experiment\\\", \\\"spark.yarn.maxAppAttempts\\\": 1}}, \\\"hdi\\\": {\\\"yarnDeployMode\\\": \\\"cluster\\\"}, \\\"tensorflow\\\": {\\\"workerCount\\\": 1, \\\"parameterServerCount\\\": 1}, \\\"mpi\\\": {\\\"processCountPerNode\\\": 1}, \\\"dataReferences\\\": {}, \\\"data\\\": {\\\"attrition\\\": {\\\"dataLocation\\\": {\\\"dataset\\\": {\\\"id\\\": \\\"f5793886-0886-4d96-9e9a-a50c897f841a\\\"}, \\\"datapath\\\": null}, \\\"createOutputDirectories\\\": false, \\\"mechanism\\\": \\\"direct\\\", \\\"environmentVariableName\\\": \\\"attrition\\\", \\\"pathOnCompute\\\": null, \\\"overwrite\\\": false}}, \\\"sourceDirectoryDataStore\\\": null, \\\"amlcompute\\\": {\\\"vmSize\\\": null, \\\"vmPriority\\\": null, \\\"retainCluster\\\": false, \\\"name\\\": null, \\\"clusterMaxNodeCount\\\": 1}}, \\\"TargetDetails\\\": null, \\\"SnapshotId\\\": \\\"318fc238-6e9d-4aac-8d44-89500509bbb9\\\", \\\"TelemetryValues\\\": {\\\"amlClientType\\\": \\\"azureml-sdk-train\\\", \\\"amlClientModule\\\": \\\"azureml.train.hyperdrive._search\\\", \\\"amlClientFunction\\\": \\\"search\\\", \\\"tenantId\\\": \\\"72f988bf-86f1-41af-91ab-2d7cd011db47\\\", \\\"amlClientRequestId\\\": \\\"934553b8-eaa1-4e2f-98a7-24cc43146437\\\", \\\"amlClientSessionId\\\": \\\"e2c8c68a-904e-4903-b435-3a834e7bc76c\\\", \\\"subscriptionId\\\": \\\"102a16c3-37d3-48a8-9237-4c9b1e8e80e0\\\", \\\"estimator\\\": \\\"Estimator\\\", \\\"samplingMethod\\\": \\\"RANDOM\\\", \\\"terminationPolicy\\\": \\\"Bandit\\\", \\\"primaryMetricGoal\\\": \\\"maximize\\\", \\\"maxTotalRuns\\\": 5, \\\"maxConcurrentRuns\\\": 4, \\\"maxDurationMinutes\\\": 10080, \\\"computeTarget\\\": \\\"AmlCompute\\\", \\\"vmSize\\\": null}}}\", \"resume_child_runs\": \"null\", \"all_jobs_generated\": \"true\", \"cancellation_requested\": \"false\", \"progress_metadata_evaluation_timestamp\": \"\\\"2020-01-11T22:12:16.272844\\\"\", \"progress_metadata_digest\": \"\\\"539023b9eef559c3a5cbd05e9dc920608bd94a9e6c99ea2d754af327c9ea825e\\\"\", \"progress_metadata_active_timestamp\": \"\\\"2020-01-11T22:12:16.272844\\\"\", \"classif-attrition-amlcompute_1578780735491421_0\": \"{\\\"--penalty\\\": \\\"l2\\\", \\\"--solver\\\": \\\"newton-cg\\\"}\", \"classif-attrition-amlcompute_1578780735491421_1\": \"{\\\"--penalty\\\": \\\"l2\\\", \\\"--solver\\\": \\\"liblinear\\\"}\", \"classif-attrition-amlcompute_1578780735491421_2\": \"{\\\"--penalty\\\": \\\"l2\\\", \\\"--solver\\\": \\\"newton-cg\\\"}\", \"classif-attrition-amlcompute_1578780735491421_3\": \"{\\\"--penalty\\\": \\\"l2\\\", \\\"--solver\\\": \\\"newton-cg\\\"}\", \"environment_preparation_status\": \"PREPARED\", \"prepare_run_id\": \"classif-attrition-amlcompute_1578780735491421_preparation\", \"classif-attrition-amlcompute_1578780735491421_4\": \"{\\\"--penalty\\\": \\\"l2\\\", \\\"--solver\\\": \\\"sag\\\"}\", \"final_best_metric_update_retry_count\": \"1\"}, \"end_time_utc\": null, \"status\": \"Running\", \"log_files\": {\"azureml-logs/hyperdrive.txt\": \"https://cesardlautomln5648400225.blob.core.windows.net/azureml/ExperimentRun/dcid.classif-attrition-amlcompute_1578780735491421/azureml-logs/hyperdrive.txt?sv=2019-02-02&sr=b&sig=Qjn31bbdnYfxHQx%2BpEcwZZkCyMafXkw5w3c8V6nu79M%3D&st=2020-01-11T22%3A08%3A18Z&se=2020-01-12T06%3A18%3A18Z&sp=r\"}, \"log_groups\": [[\"azureml-logs/hyperdrive.txt\"]], \"run_duration\": \"0:06:02\", \"hyper_parameters\": {\"--solver\": [\"choice\", [[\"newton-cg\", \"lbfgs\", \"liblinear\", \"sag\", \"saga\"]]], \"--penalty\": [\"choice\", [[\"l2\"]]]}}, \"child_runs\": [{\"run_id\": \"classif-attrition-amlcompute_1578780735491421_3\", \"run_number\": 180, \"metric\": 0.87414966, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-01-11T22:13:00.722799Z\", \"end_time\": \"2020-01-11T22:13:54.011294Z\", \"created_time\": \"2020-01-11T22:12:50.047787Z\", \"created_time_dt\": \"2020-01-11T22:12:50.047787Z\", \"duration\": \"0:01:03\", \"hyperdrive_id\": \"1578780735491421\", \"arguments\": null, \"param_--penalty\": \"l2\", \"param_--solver\": \"newton-cg\", \"best_metric\": 0.87414966}, {\"run_id\": \"classif-attrition-amlcompute_1578780735491421_2\", \"run_number\": 181, \"metric\": 0.87414966, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-01-11T22:14:59.623933Z\", \"end_time\": \"2020-01-11T22:15:48.883769Z\", \"created_time\": \"2020-01-11T22:12:50.099688Z\", \"created_time_dt\": \"2020-01-11T22:12:50.099688Z\", \"duration\": \"0:02:58\", \"hyperdrive_id\": \"1578780735491421\", \"arguments\": null, \"param_--penalty\": \"l2\", \"param_--solver\": \"newton-cg\", \"best_metric\": 0.87414966}, {\"run_id\": \"classif-attrition-amlcompute_1578780735491421_0\", \"run_number\": 182, \"metric\": 0.87414966, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-01-11T22:15:49.754944Z\", \"end_time\": \"2020-01-11T22:16:44.135292Z\", \"created_time\": \"2020-01-11T22:12:50.259421Z\", \"created_time_dt\": \"2020-01-11T22:12:50.259421Z\", \"duration\": \"0:03:53\", \"hyperdrive_id\": \"1578780735491421\", \"arguments\": null, \"param_--penalty\": \"l2\", \"param_--solver\": \"newton-cg\", \"best_metric\": 0.87414966}, {\"run_id\": \"classif-attrition-amlcompute_1578780735491421_1\", \"run_number\": 183, \"metric\": 0.8707483, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-01-11T22:13:57.20369Z\", \"end_time\": \"2020-01-11T22:14:56.572311Z\", \"created_time\": \"2020-01-11T22:12:50.286029Z\", \"created_time_dt\": \"2020-01-11T22:12:50.286029Z\", \"duration\": \"0:02:06\", \"hyperdrive_id\": \"1578780735491421\", \"arguments\": null, \"param_--penalty\": \"l2\", \"param_--solver\": \"liblinear\", \"best_metric\": 0.87414966}, {\"run_id\": \"classif-attrition-amlcompute_1578780735491421_4\", \"run_number\": 184, \"metric\": 0.8707483, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-01-11T22:16:48.859663Z\", \"end_time\": \"2020-01-11T22:17:36.215674Z\", \"created_time\": \"2020-01-11T22:14:22.375857Z\", \"created_time_dt\": \"2020-01-11T22:14:22.375857Z\", \"duration\": \"0:03:13\", \"hyperdrive_id\": \"1578780735491421\", \"arguments\": null, \"param_--penalty\": \"l2\", \"param_--solver\": \"sag\", \"best_metric\": 0.87414966}], \"children_metrics\": {\"categories\": [0], \"series\": {\"Accuracy\": [{\"categories\": [180, 181, 182, 183, 184], \"mode\": \"markers\", \"name\": \"Accuracy\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.8741496598639455, 0.8741496598639455, 0.8741496598639455, 0.8707482993197279, 0.8707482993197279]}, {\"categories\": [180, 181, 182, 183, 184], \"mode\": \"lines\", \"name\": \"Accuracy_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.8741496598639455, 0.8741496598639455, 0.8741496598639455, 0.8741496598639455, 0.8741496598639455]}]}, \"metricName\": null, \"primaryMetricName\": \"Accuracy\", \"showLegend\": false}, \"run_metrics\": [{\"name\": \"best_child_by_primary_metric\", \"run_id\": \"classif-attrition-amlcompute_1578780735491421\", \"categories\": [0], \"series\": [{\"data\": [{\"metric_name\": \"Accuracy\", \"timestamp\": \"2020-01-11 22:14:17.020971+00:00\", \"run_id\": \"classif-attrition-amlcompute_1578780735491421_3\", \"metric_value\": 0.8741496598639455, \"final\": false}]}]}], \"run_logs\": \"[2020-01-11T22:12:15.937534][API][INFO]Experiment created\\r\\n[2020-01-11T22:12:16.448644][GENERATOR][INFO]Trying to sample '4' jobs from the hyperparameter space\\r\\n[2020-01-11T22:12:16.664141][GENERATOR][INFO]Successfully sampled '4' jobs, they will soon be submitted to the execution target.\\r\\n[2020-01-11T22:12:18.5688275Z][SCHEDULER][INFO]The execution environment is being prepared. Please be patient as it can take a few minutes.\\r\\n[2020-01-11T22:12:49.2314659Z][SCHEDULER][INFO]Scheduling job, id='classif-attrition-amlcompute_1578780735491421_0'\\r\\n[2020-01-11T22:12:49.2353499Z][SCHEDULER][INFO]Scheduling job, id='classif-attrition-amlcompute_1578780735491421_3'\\r\\n[2020-01-11T22:12:49.2335496Z][SCHEDULER][INFO]Scheduling job, id='classif-attrition-amlcompute_1578780735491421_1'\\r\\n[2020-01-11T22:12:49.2309812Z][SCHEDULER][INFO]The execution environment was successfully prepared.\\r\\n[2020-01-11T22:12:49.2343849Z][SCHEDULER][INFO]Scheduling job, id='classif-attrition-amlcompute_1578780735491421_2'\\r\\n[2020-01-11T22:12:50.1405294Z][SCHEDULER][INFO]Successfully scheduled a job. Id='classif-attrition-amlcompute_1578780735491421_3'\\r\\n[2020-01-11T22:12:50.2778249Z][SCHEDULER][INFO]Successfully scheduled a job. Id='classif-attrition-amlcompute_1578780735491421_2'\\r\\n[2020-01-11T22:12:50.3749484Z][SCHEDULER][INFO]Successfully scheduled a job. Id='classif-attrition-amlcompute_1578780735491421_1'\\r\\n[2020-01-11T22:12:50.4393757Z][SCHEDULER][INFO]Successfully scheduled a job. Id='classif-attrition-amlcompute_1578780735491421_0'\\r\\n[2020-01-11T22:13:15.664320][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://northcentralus.experiments.azureml.net', subscription_id='102a16c3-37d3-48a8-9237-4c9b1e8e80e0', resource_group='automlpmdemo', workspace_name='cesardl-automl-northcentralus-ws', project_name='classif-attrition-amlcompute'), run_id='classif-attrition-amlcompute_1578780735491421_3', data_container_id=None, baggage=None)] do not contain any metrics with the primary metric name at this moment, policy cannot be applied.\\r\\n[2020-01-11T22:13:45.676751][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://northcentralus.experiments.azureml.net', subscription_id='102a16c3-37d3-48a8-9237-4c9b1e8e80e0', resource_group='automlpmdemo', workspace_name='cesardl-automl-northcentralus-ws', project_name='classif-attrition-amlcompute'), run_id='classif-attrition-amlcompute_1578780735491421_3', data_container_id=None, baggage=None)] do not contain any metrics with the primary metric name at this moment, policy cannot be applied.\\r\\n[2020-01-11T22:14:15.996282][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://northcentralus.experiments.azureml.net', subscription_id='102a16c3-37d3-48a8-9237-4c9b1e8e80e0', resource_group='automlpmdemo', workspace_name='cesardl-automl-northcentralus-ws', project_name='classif-attrition-amlcompute'), run_id='classif-attrition-amlcompute_1578780735491421_1', data_container_id=None, baggage=None)] do not contain any metrics with the primary metric name at this moment, policy cannot be applied.\\r\\n[2020-01-11T22:14:16.715817][GENERATOR][INFO]Trying to sample '1' jobs from the hyperparameter space\\r\\n[2020-01-11T22:14:16.846759][GENERATOR][INFO]Successfully sampled '1' jobs, they will soon be submitted to the execution target.\\r\\n[2020-01-11T22:14:21.2851234Z][SCHEDULER][INFO]Scheduling job, id='classif-attrition-amlcompute_1578780735491421_4'\\r\\n[2020-01-11T22:14:22.5367949Z][SCHEDULER][INFO]Successfully scheduled a job. Id='classif-attrition-amlcompute_1578780735491421_4'\\r\\n[2020-01-11T22:14:45.489439][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://northcentralus.experiments.azureml.net', subscription_id='102a16c3-37d3-48a8-9237-4c9b1e8e80e0', resource_group='automlpmdemo', workspace_name='cesardl-automl-northcentralus-ws', project_name='classif-attrition-amlcompute'), run_id='classif-attrition-amlcompute_1578780735491421_1', data_container_id=None, baggage=None)] do not contain any metrics with the primary metric name at this moment, policy cannot be applied.\\r\\n[2020-01-11T22:14:47.113017][GENERATOR][INFO]Max number of jobs '5' reached for experiment.\\r\\n[2020-01-11T22:14:47.884712][GENERATOR][INFO]All jobs generated.\\r\\n[2020-01-11T22:15:15.886897][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://northcentralus.experiments.azureml.net', subscription_id='102a16c3-37d3-48a8-9237-4c9b1e8e80e0', resource_group='automlpmdemo', workspace_name='cesardl-automl-northcentralus-ws', project_name='classif-attrition-amlcompute'), run_id='classif-attrition-amlcompute_1578780735491421_2', data_container_id=None, baggage=None)] do not contain any metrics with the primary metric name at this moment, policy cannot be applied.\\r\\n[2020-01-11T22:15:45.541262][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://northcentralus.experiments.azureml.net', subscription_id='102a16c3-37d3-48a8-9237-4c9b1e8e80e0', resource_group='automlpmdemo', workspace_name='cesardl-automl-northcentralus-ws', project_name='classif-attrition-amlcompute'), run_id='classif-attrition-amlcompute_1578780735491421_2', data_container_id=None, baggage=None)] do not contain any metrics with the primary metric name at this moment, policy cannot be applied.\\r\\n[2020-01-11T22:16:16.120230][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://northcentralus.experiments.azureml.net', subscription_id='102a16c3-37d3-48a8-9237-4c9b1e8e80e0', resource_group='automlpmdemo', workspace_name='cesardl-automl-northcentralus-ws', project_name='classif-attrition-amlcompute'), run_id='classif-attrition-amlcompute_1578780735491421_0', data_container_id=None, baggage=None)] do not contain any metrics with the primary metric name at this moment, policy cannot be applied.\\r\\n[2020-01-11T22:17:15.968546][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://northcentralus.experiments.azureml.net', subscription_id='102a16c3-37d3-48a8-9237-4c9b1e8e80e0', resource_group='automlpmdemo', workspace_name='cesardl-automl-northcentralus-ws', project_name='classif-attrition-amlcompute'), run_id='classif-attrition-amlcompute_1578780735491421_4', data_container_id=None, baggage=None)] do not contain any metrics with the primary metric name at this moment, policy cannot be applied.\\n\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"NOTSET\", \"sdk_version\": \"1.0.76\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "RunDetails(hyperdrive_run).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get details for debugging:\n",
    "# RunDetails(run).get_widget_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get log results upon completion\n",
    "Model training and monitoring happen in the background. Wait until the model has finished training before you run more code. Use wait_for_completion to show when the model training is finished:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hyperdrive_run.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find and get the best model found by HyperDrive¶ \n",
    "When all jobs finish, we can find out the one that has the highest accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['--penalty', 'l2', '--solver', 'newton-cg']\n"
     ]
    }
   ],
   "source": [
    "best_run = hyperdrive_run.get_best_run_by_primary_metric()\n",
    "print(best_run.get_details()['runDefinition']['arguments'])\n",
    "# print(best_run.get_details())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy 'best_run' to 'run' to re-use the same code also used without HyperDrive\n",
    "run = best_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Display run metrics results\n",
    "You now have a model trained on a remote cluster. Retrieve the accuracy of the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Accuracy': 0.8741496598639455}\n"
     ]
    }
   ],
   "source": [
    "print(run.get_metrics())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## See files associated with the run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['azureml-logs/55_azureml-execution-tvmps_bcb36db862e3c9e906c8553abcb0c1d35a8229810ba8acf1830ab57fe25cc9d1_d.txt', 'azureml-logs/65_job_prep-tvmps_bcb36db862e3c9e906c8553abcb0c1d35a8229810ba8acf1830ab57fe25cc9d1_d.txt', 'azureml-logs/70_driver_log.txt', 'azureml-logs/75_job_post-tvmps_bcb36db862e3c9e906c8553abcb0c1d35a8229810ba8acf1830ab57fe25cc9d1_d.txt', 'azureml-logs/process_info.json', 'azureml-logs/process_status.json', 'logs/azureml/153_azureml.log', 'logs/azureml/azureml.log', 'outputs/classif-empl-attrition.pkl', 'outputs/x_test.pkl', 'outputs/y_test.pkl']\n"
     ]
    }
   ],
   "source": [
    "print(run.get_file_names())\n",
    "\n",
    "run.download_file('azureml-logs/70_driver_log.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register the model\n",
    "Once you've trained the model, you can save and register it to your workspace. Model registration lets you store and version your models in your workspace to simplify model management and deployment.\n",
    "\n",
    "Running the following code will register the model to your workspace, and will make it available to reference by name in remote compute contexts or deployment scripts. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Model(workspace=Workspace.create(name='cesardl-automl-northcentralus-ws', subscription_id='102a16c3-37d3-48a8-9237-4c9b1e8e80e0', resource_group='automlpmdemo'), name=classif-empl-attrition-aml-comp, id=classif-empl-attrition-aml-comp:24, version=24, tags={'ml-task': 'binary-classification', 'business-area': 'HR'}, properties={'joblib-version': '0.14.0', 'pandas-version': '0.23.4'})"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from azureml.core.model import Model\n",
    "\n",
    "model_reg = run.register_model(model_name='classif-empl-attrition-aml-comp',    # Name of the registered model in your workspace.\n",
    "                               description='Binary classification model for employees attrition',\n",
    "                               model_path='outputs/classif-empl-attrition.pkl', # Local file to upload and register as a model.\n",
    "                               model_framework=Model.Framework.SCIKITLEARN,     # Framework used to create the model.\n",
    "                               model_framework_version='0.20.3',                # Version of scikit-learn used to create the model.\n",
    "                               tags={'ml-task': \"binary-classification\", 'business-area': \"HR\"},\n",
    "                               properties={'joblib-version': \"0.14.0\", 'pandas-version': \"0.23.4\"},\n",
    "                               sample_input_dataset=aml_dataset\n",
    "                              )\n",
    "model_reg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Scikit-Learn model pickle file from the run (Option A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# retrieve model for visualization and deployment\n",
    "\n",
    "# Download the model .pkl file to local (Using the 'run' object)\n",
    "run.download_file('outputs/classif-empl-attrition.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Scikit-Learn model pickle file from the model registry (Option B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "azureml-models/classif-empl-attrition-aml-comp/24/classif-empl-attrition.pkl\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'classif-empl-attrition.pkl'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(Model.get_model_path('classif-empl-attrition-aml-comp', _workspace=ws))\n",
    "\n",
    "model_from_registry = Model(ws,'classif-empl-attrition-aml-comp')\n",
    "model_from_registry.download(target_dir='.', exist_ok=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Try model predictions in this notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load model into memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('preprocessor', DataFrameMapper(default=False, df_out=False,\n",
       "        features=[(['Age'], Pipeline(memory=None,\n",
       "     steps=[('imputer', SimpleImputer(copy=True, fill_value=None, missing_values=nan,\n",
       "       strategy='median', verbose=0)), ('scaler', StandardScaler(copy=True, with_mean=True, wit...enalty='l2', random_state=None, solver='lbfgs',\n",
       "          tol=0.0001, verbose=0, warm_start=False))])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the model into memory\n",
    "model = joblib.load('classif-empl-attrition.pkl')\n",
    "\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load model and test datasets from .pkl files\n",
    "\n",
    "# Download the test datasets to local\n",
    "run.download_file('outputs/x_test.pkl')\n",
    "run.download_file('outputs/y_test.pkl')\n",
    "\n",
    "# Load the test datasets into memory\n",
    "x_test = joblib.load('x_test.pkl')\n",
    "y_test = joblib.load('y_test.pkl')\n",
    "\n",
    "# joblib\n",
    "# https://joblib.readthedocs.io/en/latest/installing.html\n",
    "# https://joblib.readthedocs.io/en/latest/generated/joblib.load.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make Predictions and calculate Accuracy metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.8741496598639455"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# Make Multiple Predictions\n",
    "y_predictions = model.predict(x_test)  # .predict(X[0:1])\n",
    "\n",
    "y_predictions\n",
    "\n",
    "print('Accuracy:')\n",
    "accuracy_score(y_test, y_predictions)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[243   4]\n",
      " [ 33  14]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQwAAADwCAYAAAD1q1pFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAV/0lEQVR4nO3debBcZZnH8e8vISyRNQZCCMEgBJShJEJAxBHjiAiIgpYoiIjKGEFcUQdQHHEBmbIGFVE0iAIugTjKwAAaY2oQsViDbBnIAoLEBEJA2Zfk3mf+OG9Lc7m3++1Od04vv0/Vqdtn6XOe7uQ+993OeRURmJnlGFV2AGbWPZwwzCybE4aZZXPCMLNsThhmls0Jw8yyOWGUTNJGkv5H0qOSfrEW5zlS0m9bGVtZJL1e0qKy47AXk8dh5JH0XuAE4BXA48AtwGkRcc1anvco4OPAPhGxZq0D7XCSApgaEUvLjsUa5xJGBkknAN8CTgcmANsB3wMOacHpXwYs7odkkUPSemXHYDVEhJcaC7AZ8ARwWI1jNqBIKMvT8i1gg7RvBrAM+AywElgBfDDt+zLwHLA6XeMY4FTgp1XnngIEsF5a/wBwD0Up58/AkVXbr6l63z7AjcCj6ec+VfuuAr4K/DGd57fA+BE+WyX+f6uK/1DgIGAx8Ajw+arj9wKuBf6ejj0bWD/tuzp9lifT531P1flPBB4AflLZlt6zQ7rG7ml9G2AVMKPs/xutWPafMTb2eNUGWQvwm7LjdTav77XAhsAlNY75ArA3MI3iF+JS4BTgi2n/1hSJZxLwZuC/JP13RHwpFdF3jIj3AUg6daSLSHoJcBawZ0QskjQRGDfMceOAK4BPALOBw4ArJO0YEQ+nw94LHAjcD/wa+Cxw0giX3jp9B5MoEtO5wDxgD4rS1gJJF0XEPcAA8GngJmDbdO6PAt+KiH3T590tUpVE0ox0/nEUpa1RwGsqF46IuyWdCPxM0h7Aj4HzI+Kqkb6nbrLqkQGun7tt1rFjJt49vs3h1OUqSX0vBVZF7SrDkcBXImJlRDxEUXI4qmr/6rR/dURcSfHXdecm4xkEdpW0UUSsiIiFwxzzVmBJRPwkItZExGzgLuBtVcf8OCIWR8TTwByKZDeS1RTtNauBi4DxwLcj4vF0/YXAqwAiYkFEXJeuey/wA+ANGZ/pSxHxbIrnBSLiXGAJcD0wkSJB94hgIAazlk7ghFHfw8D4OnXrbYD7qtbvS9v+cY4hCecpYONGA4mIJymK8ccCKyRdIekVGfFUYppUtf5AA/E8HBED6XXlF/rBqv1PV94vaSdJl0t6QNJjFO0+9f4yPhQRz9Q55lxgV+A7EfFsnWO7RgCDRNbSCZww6rsWeIai3j6S5RTF6Yrt0rZmPAmMrVrfunpnRMyNiDdT/KW9i+IXqV48lZj+2mRMjTiHIq6pEbEp8HlAdd5T87dB0sYU7ULnAaemKldPCILVMZC1dAInjDoi4lHg34HvSjpU0lhJYySdIukRSUsp/nqfImlLSePT8T9t8pK3APtK2k7SZsDJlR2SJkh6e2rLeJaiajPc/6QrgZ0kvVfSepLeA+wCXN5kTI3YBHgMeCKVfo4bsv9B4OUZ59lC0kpJdwDfBhZExL9StM18v5UBl80ljB4TEWdSjME4BXiIoqHwZGAmxS/ieIqei9uA24Gbga81ea15wMXpXAt44S/5KIreluUUPQdvoGhQHHqOh4GD07EPU/RwHBwRq5qJqUGfpWhQfZyi9HPxkP2nAhdI+rukd9c4z5PAARQJ6ACKahgU/w67SzqylUGXJYABImvpBB641QRJrwVOjYi3pPWTASLi66UG1mMkTQEuj4hdSw6lbabttn7M+/WWWcduNWn5goiY3uaQanK3anMmUZQyKpZR1RVoliuAgS76o+2E0ZzhGvG651/dOkpndJjmccJozjJgctX6tjTfK2J9LDqofSKHE0ZzbgSmStqeoqvycIqGPrOGRMDq7skX7iVpRhqE9TFgLnAnMGeEEZfWJEmzKcbA7CxpmaRjyo6pPcRA5tIJXMJoUhrifWXZcfSqiDii7BjWhQAGu6iE4YRhVrJOKT3kcMIwK1ExcMsJw8wyDYYThpllcAnDzLIFYnWMLjuMbO5WXQuSZpYdQ6/r9e+4UsLolm5VJ4y109P/mTtEj3/HYiBGZS2dwFUSsxIVT9zqjGSQo6MSxvhxo2PK5DFlh5Ftu0nrMX23Dbto2A0svm1s/YM6yIaMZVON66rv+Bme5Ll4NrsO0SnVjRwdlTCmTB7DDXMn1z/QmvaWbWo969da4fqYn31shDqmupGjoxKGWT8adAnDzHIE4rnonl/D7onUrAe50dPMGjLgoeFmliMQAy5hmFmuQfeSmFmOYmi4E4aZZei2m8+cMMxKFIEHbplZLnnglpnlKWY+cwnDzDJ1U6Nn90Rq1oMCMRh5Sz2SJkv6X0l3Sloo6ZNp+zhJ8yQtST+3SNsl6SxJSyXdJmn3etdwwjAr2QCjspYMa4DPRMQrgb2B4yXtApwEzI+IqcD8tA5wIDA1LTOBc+pdwFUSsxK1sls1IlYAK9LrxyXdCUwCDgFmpMMuAK4CTkzbL4yIAK6TtLmkiek8w3LCMCtRMfNZ6wv6kqYArwauByZUkkBErJC0VTpsEnB/1duWpW1OGGadqoEnbo2XdFPV+qyImDX0IEkbA78EPhURj0kjnn+4HTWfbuaEYVaiCDVSwlgVEdNrHSBpDEWy+FlE/CptfrBS1ZA0EViZti8Dqh9xty2wvNb53ehpVrJWPTVcRVHiPODOiDizatdlwNHp9dHApVXb3596S/YGHq3VfgEuYZiVqniATstGer4OOAq4XdItadvngTOAOZKOAf4CHJb2XQkcBCwFngI+WO8CThhmpWrdQ4Aj4hqGb5cAeNMwxwdwfCPXcMIwK1GA71Y1szyVkZ7dwgnDrGR+CLCZZSmeh+EShpllcpXEzLIUbRiukphZJk/GbGZZArFm0N2qZpbJz/Q0syzuJTGzhrjR08yyeKSnmTXEbRhmlqV4RJ8ThpnlCHermlmmFj9Ap+2cMMxK5iqJmWXptjaMtnYASzpA0qI0FdtJ9d9h1n9aNVXiutC2Eoak0cB3gTdTPM78RkmXRcT/teuaZt3G4zCetxewNCLuAZB0EcXUbE4YZhUBazzSExh+GrbXtPF6Zl2n29ow2pkwsqZhkzSTYuZotpvkNljrP92UMNpZFsqahi0iZkXE9IiYvuVLu2cAi1krVNowuqXRs50J40ZgqqTtJa0PHE4xNZuZVYlQ1tIJ2lYHiIg1kj4GzAVGAz+KiIXtup5Zt/JIzyQirqSYv9HMhhHRXW0YbmU0K5UYGHS3qpll6pT2iRxOGGYl8jgMM8sXRTtGt3DCMCuZe0nMLEvgNgwzy9Y5ozhzOGGYlWxw0AnDzDJEuEpiZg1wlcTMsnVTt2r3jEk161GtvFtV0o8krZR0R9W2UyX9VdItaTmoat/J6Zm7iyS9pd75nTDMShTkJYsG2jnOBw4YZvs3I2JaWq4EkLQLxWMn/im953vpWbwjcsIwK1lkLlnnirgaeCTz8EOAiyLi2Yj4M7CU4lm8I3LCMCtTQAwqawHGS7qpapnZwJU+Jum2VGXZIm0b7rm7k2qdZMRGT0mb1npjRDyWG6mZjayB6saqiJjexCXOAb5KUVD5KvCfwIfIfO5utVq9JAvTm6tPWlkPYLv8eM1sJO3uJYmIByuvJZ0LXJ5Ws567W23EhBERk0faZ2atsS7uJZE0MSJWpNV3AJUelMuAn0s6E9gGmArcUOtcWeMwJB0OvDwiTpe0LTAhIhY0Fb2ZPS+AFiYMSbOBGRTtHcuALwEzJE1LV7sX+AhARCyUNIdicrE1wPERMVDr/HUThqSzgTHAvsDpwFPA94E9m/tIZlatlVWSiDhimM3n1Tj+NOC03PPnlDD2iYjdJf0pXeCRNG2AmbVCF430zEkYqyWNIn0sSS8FBtsalVnf+EeXaVfIGYfxXeCXwJaSvgxcA/xHW6My6xfRYxMZRcSFkhYA+6VNh0XEHbXeY2YN6LEqCRQzl62m+GgeHWrWUp1ReshR95df0heA2RT9tNtS9Nue3O7AzPpGK28mabOcEsb7gD0i4ikASacBC4CvtzMws77RIckgR07CuG/IcesB97QnHLM+k24+6xa1bj77JkXuewpYKGluWt+foqfEzFqhR0oYlZ6QhcAVVduva184Zn2oQ7pMc9S6+WzE4aRm1jrqkRIGAJJ2oBhrvguwYWV7ROzUxrjM+kMH9YDkyBlTcT7wY4rO4gOBOcBFbYzJrI+oqJLkLB0gJ2GMjYi5ABFxd0ScAryxvWGZ9ZEeG4fxrCQBd0s6FvgrsFV7wzLrI110K2dOwvg0sDHwCYq2jM0ongdoZmurxQ/Qabecm8+uTy8fB45qbzhm/acnekkkXUKNmlNEvLMtEZn1m15IGMDZ6yyKZMniLThov3ev68v2ldGbPlB2CD1PT/TuDd21Bm7NX5eBmPWrnqiSmNk60kuNnmbWRkHPdasCIGmDiHi2ncGY9aNuqpLkPHFrL0m3A0vS+m6SvtP2yMz6RReN9Mxpzj0LOBh4GCAibsVDw81ap4sSRk6VZFRE3FeMDv+HmtOpmVkeRXdVSXISxv2S9gJC0mjg48Di9oZl1kd6rJfkOIpqyXbAg8Dv0jYza4VeKmFExErg8HUQi1lfUi91q0o6l2FyYETMbEtEZv2kB9swflf1ekPgHcD97QnHrA/1UsKIiIur1yX9BJjXtojM+k0vJYxhbA+8rNWBmPWrnqqSSPobz+fAUcAjwEntDMrMOlPNhJGe5bkbxXM8AQYjoovyoVkX6KLfqJpDw1NyuCQiBtLSRR/NrAtE0a2as3SCnHtJbpC0e9sjMetXXXQvyYgJQ1KluvLPFEljkaSbJf1J0s3rJjyz3iaev5+k3pJ1PulHklZKuqNq2zhJ8yQtST+3SNsl6SxJSyXdllMwqFXCuCH9PBTYGTgIOAx4V/ppZq3Q2hLG+cABQ7adBMyPiKnAfJ7vtDgQmJqWmcA59U5eq9FTUMx2lh2qmTWmxSM9I+JqSVOGbD4EmJFeXwBcBZyYtl+Y2iavk7S5pIkRsWKk89dKGFtKOqFGYGfWjd7M6mt/+8SEShKIiBWSKjMXTuKFo7aXpW1NJYzRFDOedc+9t2ZdqIEekPGSbqpanxURs9bm0sNsq5m+aiWMFRHxlbUIxsxy5JcwVkXE9Cau8GClqiFpIrAybV8GTK46bltgea0T1Wr0dMnCrN1yGzzXrtpyGXB0en00cGnV9ven3pK9gUdrtV9A7RLGm9YqRDPL0spGT0mzKRo4x0taBnwJOAOYI+kY4C8838t5JUXv51LgKeCD9c5fa+azR9YqcjPL09pekiNG2PWiAkDqHTm+kfN7IiOzkvXU3apm1mZOGGaWoxenGTCzdnLCMLNcLmGYWT4nDDPL5oRhZlnc6GlmDXHCMLNcnfK8zhxOGGYlc5XEzPJ00AN+czhhmJXNCcPMclSeGt4tcuYlacpwjzs3s2H0wrwkLXA+L37cuZkNoYispRO0rUoywuPOzaxauFvVzBrRGYWHLKUnDEkzKWZdYsMxm5Ycjdm650bPBkTErIiYHhHT1x89tuxwzNa9Lmr0LL2EYdbXuuzms3Z2q84GrgV2lrQsPeLczIZyCaPm487NLOm2gVuukpiVTIPdkzGcMMzK1EHVjRxOGGYl88AtM8vnEoaZ5XKjp5nlCaBDbizL4YRhVjK3YZhZFo/DMLN8Ea6SmFk+lzDMLJ8ThpnlcgnDzPIE4HtJzCyXu1XNLJ97ScwsVyvbMCTdCzwODABrImK6pHHAxcAU4F7g3RHxt2bOX/ozPc36Wu7TthpLKm+MiGkRMT2tnwTMj4ipwPy03hQnDLMSFSM92z6R0SHABen1BcChzZ7ICcOsbIOZS54AfitpQZrCA2BCRKwASD+3ajZUt2GYlayB0sN4STdVrc+KiFlDjnldRCyXtBUwT9JdLQkyccIwK1NEI+MwVlW1S4xwuliefq6UdAmwF/CgpIkRsULSRGBls+G6SmJWMkXeUvc80kskbVJ5DewP3AFcBhydDjsauLTZWF3CMCtb68ZhTAAukQTF7/bPI+I3km4E5qS5gf4CHNbsBZwwzMrUwtnbI+IeYLdhtj8MvKkV13DCMCubR3qaWbbuyRdOGGZlW8tBWeuUE4ZZmQIYcMIwswxirYd9r1NOGGZlc8Iws2xOGGaWJWjkxrLSOWGYlcxtGGaWzwnDzLJEwGD31EmcMMzK1j35wgnDrGxuwzCzfE4YZpbFM58177FnHlg1d+Hp95UdRwPGA6vKDqLHdeN3/LL8Q8MljGZFxJZlx9AISTfVe8airZ2++I6dMMwsSwAD3dNN4oRhVqqAcMLoF0PnhLDW6/3vuIuqJJ5mYC0MM4nMC0gakHSLpDsk/ULS2GavJWmGpMvT67dLGnF+TEmbS/poE9c4VdJnc7cPOeZ8Se9q4FpTJN1R77h633HXq/SS5CwdwAmjvZ5Ok+LuCjwHHFu9U4WG/w0i4rKIOKPGIZsDDScMK0lE3tIBnDDWnT8AO6a/rHdK+h5wMzBZ0v6SrpV0cyqJbAwg6QBJd0m6Bnhn5USSPiDp7PR6gqRLJN2aln2AM4AdUunmG+m4z0m6UdJtkr5cda4vSFok6XfAzvU+hKQPp/PcKumXQ0pN+0n6g6TFkg5Ox4+W9I2qa39kbb/InuOEYdUkrQccCNyeNu0MXBgRrwaeBE4B9ouI3YGbgBMkbQicC7wNeD2w9QinPwv4fUTsBuwOLAROAu5OpZvPSdofmEoxbd40YA9J+0raAzgceDVFQtoz4+P8KiL2TNe7Ezimat8U4A3AW4Hvp89wDPBoROyZzv9hSdtnXKc/RMDAQN7SAdzo2V4bSbolvf4DcB6wDXBfRFyXtu8N7AL8Mc1YtT5wLfAK4M8RsQRA0k+BmbzYvwDvB4iIAeBRSVsMOWb/tPwprW9MkUA2AS6JiKfSNS7L+Ey7SvoaRbVnY2Bu1b45ETEILJF0T/oM+wOvqmrf2Cxde3HGtfpDh5QecjhhtNfTETGtekNKCk9WbwLmRcQRQ46bRutmrBDw9Yj4wZBrfKqJa5wPHBoRt0r6ADCjat/Qc0W69scjojqxIGlKg9ftXV2UMFwlKd91wOsk7QggaayknYC7gO0l7ZCOO2KE988HjkvvHS1pU+BxitJDxVzgQ1VtI5MkbQVcDbxD0kZpEt+3ZcS7CbBC0hjgyCH7DpM0KsX8cmBRuvZx6Xgk7ZQmCjYAMntIOqSXxCWMkkXEQ+kv9WxJG6TNp0TEYkkzgSskrQKuAXYd5hSfBGaliXYHgOMi4lpJf0zdlr9O7RivBK5NJZwngPdFxM2SLgZuAe6jqDbV80Xg+nT87bwwMS0Cfk8xKfCxEfGMpB9StG3crOLiDwGH5n07fSAgumjglqKLikNmvWaz9baM126alz/n/u2HC8q+r8YlDLOyddEfbScMszJVulW7hBOGWcnCDwE2szydM4ozhxOGWZn8iD4za0gXdas6YZiVKIBwCcPMsoSfuGVmDYgu6lb1SE+zEkn6DcVUCjlWRcQB7YynHicMM8vmu1XNLJsThpllc8Iws2xOGGaWzQnDzLL9P/bfjoY7ChNoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 288x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "cm = confusion_matrix(y_test, y_predictions)\n",
    "\n",
    "print(cm)\n",
    "\n",
    "# Show confusion matrix in a separate window\n",
    "plt.matshow(cm)\n",
    "plt.title('Confusion matrix')\n",
    "plt.colorbar()\n",
    "plt.ylabel('True label')\n",
    "plt.xlabel('Predicted label')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "One Prediction: \n",
      "0\n",
      "20 Predictions: \n",
      "[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>BusinessTravel</th>\n",
       "      <th>DailyRate</th>\n",
       "      <th>Department</th>\n",
       "      <th>DistanceFromHome</th>\n",
       "      <th>Education</th>\n",
       "      <th>EducationField</th>\n",
       "      <th>EnvironmentSatisfaction</th>\n",
       "      <th>Gender</th>\n",
       "      <th>HourlyRate</th>\n",
       "      <th>...</th>\n",
       "      <th>PerformanceRating</th>\n",
       "      <th>RelationshipSatisfaction</th>\n",
       "      <th>StockOptionLevel</th>\n",
       "      <th>TotalWorkingYears</th>\n",
       "      <th>TrainingTimesLastYear</th>\n",
       "      <th>WorkLifeBalance</th>\n",
       "      <th>YearsAtCompany</th>\n",
       "      <th>YearsInCurrentRole</th>\n",
       "      <th>YearsSinceLastPromotion</th>\n",
       "      <th>YearsWithCurrManager</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>899</th>\n",
       "      <td>49</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>1098</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Medical</td>\n",
       "      <td>1</td>\n",
       "      <td>Male</td>\n",
       "      <td>85</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>23</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>610</th>\n",
       "      <td>27</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>269</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>Technical Degree</td>\n",
       "      <td>3</td>\n",
       "      <td>Male</td>\n",
       "      <td>42</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>667</th>\n",
       "      <td>41</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>1085</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>2</td>\n",
       "      <td>Female</td>\n",
       "      <td>57</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>857</th>\n",
       "      <td>44</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>1097</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>3</td>\n",
       "      <td>Male</td>\n",
       "      <td>96</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1063</th>\n",
       "      <td>29</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>1246</td>\n",
       "      <td>Sales</td>\n",
       "      <td>19</td>\n",
       "      <td>3</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>3</td>\n",
       "      <td>Male</td>\n",
       "      <td>77</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1022</th>\n",
       "      <td>37</td>\n",
       "      <td>Non-Travel</td>\n",
       "      <td>1413</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>Technical Degree</td>\n",
       "      <td>3</td>\n",
       "      <td>Male</td>\n",
       "      <td>84</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>665</th>\n",
       "      <td>47</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>1454</td>\n",
       "      <td>Sales</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>4</td>\n",
       "      <td>Female</td>\n",
       "      <td>65</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>840</th>\n",
       "      <td>35</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>982</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>Medical</td>\n",
       "      <td>4</td>\n",
       "      <td>Male</td>\n",
       "      <td>58</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>796</th>\n",
       "      <td>25</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>1219</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Technical Degree</td>\n",
       "      <td>4</td>\n",
       "      <td>Male</td>\n",
       "      <td>32</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1221</th>\n",
       "      <td>45</td>\n",
       "      <td>Non-Travel</td>\n",
       "      <td>1238</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>3</td>\n",
       "      <td>Male</td>\n",
       "      <td>74</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>23</td>\n",
       "      <td>15</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>19</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>1181</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Medical</td>\n",
       "      <td>2</td>\n",
       "      <td>Female</td>\n",
       "      <td>79</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139</th>\n",
       "      <td>32</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>634</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>Other</td>\n",
       "      <td>2</td>\n",
       "      <td>Female</td>\n",
       "      <td>35</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1420</th>\n",
       "      <td>41</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>642</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>4</td>\n",
       "      <td>Male</td>\n",
       "      <td>76</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>322</th>\n",
       "      <td>32</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>1018</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>Medical</td>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "      <td>74</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>777</th>\n",
       "      <td>21</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>1334</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>3</td>\n",
       "      <td>Female</td>\n",
       "      <td>36</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>37</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>408</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>19</td>\n",
       "      <td>2</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>2</td>\n",
       "      <td>Male</td>\n",
       "      <td>73</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1049</th>\n",
       "      <td>30</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>1358</td>\n",
       "      <td>Sales</td>\n",
       "      <td>16</td>\n",
       "      <td>1</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>4</td>\n",
       "      <td>Male</td>\n",
       "      <td>96</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1403</th>\n",
       "      <td>39</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>119</td>\n",
       "      <td>Sales</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>Marketing</td>\n",
       "      <td>2</td>\n",
       "      <td>Male</td>\n",
       "      <td>77</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>20</td>\n",
       "      <td>8</td>\n",
       "      <td>11</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>51</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>432</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>4</td>\n",
       "      <td>Male</td>\n",
       "      <td>96</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1331</th>\n",
       "      <td>48</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>1224</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>4</td>\n",
       "      <td>Male</td>\n",
       "      <td>91</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>29</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>22</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Age BusinessTravel  DailyRate              Department  DistanceFromHome  \\\n",
       "899    49  Travel_Rarely       1098  Research & Development                 4   \n",
       "610    27  Travel_Rarely        269  Research & Development                 5   \n",
       "667    41  Travel_Rarely       1085  Research & Development                 2   \n",
       "857    44  Travel_Rarely       1097  Research & Development                10   \n",
       "1063   29  Travel_Rarely       1246                   Sales                19   \n",
       "1022   37     Non-Travel       1413  Research & Development                 5   \n",
       "665    47  Travel_Rarely       1454                   Sales                 2   \n",
       "840    35  Travel_Rarely        982  Research & Development                 1   \n",
       "796    25  Travel_Rarely       1219  Research & Development                 4   \n",
       "1221   45     Non-Travel       1238  Research & Development                 1   \n",
       "149    19  Travel_Rarely       1181  Research & Development                 3   \n",
       "1139   32  Travel_Rarely        634  Research & Development                 5   \n",
       "1420   41  Travel_Rarely        642  Research & Development                 1   \n",
       "322    32  Travel_Rarely       1018  Research & Development                 2   \n",
       "777    21  Travel_Rarely       1334  Research & Development                10   \n",
       "47     37  Travel_Rarely        408  Research & Development                19   \n",
       "1049   30  Travel_Rarely       1358                   Sales                16   \n",
       "1403   39  Travel_Rarely        119                   Sales                15   \n",
       "87     51  Travel_Rarely        432  Research & Development                 9   \n",
       "1331   48  Travel_Rarely       1224  Research & Development                10   \n",
       "\n",
       "      Education    EducationField  EnvironmentSatisfaction  Gender  \\\n",
       "899           2           Medical                        1    Male   \n",
       "610           1  Technical Degree                        3    Male   \n",
       "667           4     Life Sciences                        2  Female   \n",
       "857           4     Life Sciences                        3    Male   \n",
       "1063          3     Life Sciences                        3    Male   \n",
       "1022          2  Technical Degree                        3    Male   \n",
       "665           4     Life Sciences                        4  Female   \n",
       "840           4           Medical                        4    Male   \n",
       "796           1  Technical Degree                        4    Male   \n",
       "1221          1     Life Sciences                        3    Male   \n",
       "149           1           Medical                        2  Female   \n",
       "1139          4             Other                        2  Female   \n",
       "1420          3     Life Sciences                        4    Male   \n",
       "322           4           Medical                        1  Female   \n",
       "777           3     Life Sciences                        3  Female   \n",
       "47            2     Life Sciences                        2    Male   \n",
       "1049          1     Life Sciences                        4    Male   \n",
       "1403          4         Marketing                        2    Male   \n",
       "87            4     Life Sciences                        4    Male   \n",
       "1331          3     Life Sciences                        4    Male   \n",
       "\n",
       "      HourlyRate          ...           PerformanceRating  \\\n",
       "899           85          ...                           3   \n",
       "610           42          ...                           3   \n",
       "667           57          ...                           3   \n",
       "857           96          ...                           3   \n",
       "1063          77          ...                           3   \n",
       "1022          84          ...                           3   \n",
       "665           65          ...                           3   \n",
       "840           58          ...                           3   \n",
       "796           32          ...                           3   \n",
       "1221          74          ...                           4   \n",
       "149           79          ...                           3   \n",
       "1139          35          ...                           3   \n",
       "1420          76          ...                           4   \n",
       "322           74          ...                           3   \n",
       "777           36          ...                           3   \n",
       "47            73          ...                           4   \n",
       "1049          96          ...                           3   \n",
       "1403          77          ...                           3   \n",
       "87            96          ...                           4   \n",
       "1331          91          ...                           3   \n",
       "\n",
       "      RelationshipSatisfaction StockOptionLevel  TotalWorkingYears  \\\n",
       "899                          3                1                 23   \n",
       "610                          2                1                  9   \n",
       "667                          3                1                 10   \n",
       "857                          3                0                  6   \n",
       "1063                         3                2                 10   \n",
       "1022                         1                0                  7   \n",
       "665                          1                0                  3   \n",
       "840                          2                1                 10   \n",
       "796                          2                1                  7   \n",
       "1221                         4                1                 25   \n",
       "149                          4                0                  1   \n",
       "1139                         4                2                  6   \n",
       "1420                         1                1                 12   \n",
       "322                          3                0                 10   \n",
       "777                          1                0                  1   \n",
       "47                           1                0                  8   \n",
       "1049                         3                2                  4   \n",
       "1403                         1                0                 21   \n",
       "87                           2                2                 10   \n",
       "1331                         4                0                 29   \n",
       "\n",
       "     TrainingTimesLastYear  WorkLifeBalance  YearsAtCompany  \\\n",
       "899                      2                4               1   \n",
       "610                      3                3               9   \n",
       "667                      1                2               7   \n",
       "857                      4                3               6   \n",
       "1063                     3                3              10   \n",
       "1022                     2                1               6   \n",
       "665                      3                2               3   \n",
       "840                      2                3               8   \n",
       "796                      3                4               7   \n",
       "1221                     3                2              23   \n",
       "149                      3                3               1   \n",
       "1139                     3                3               3   \n",
       "1420                     3                3               5   \n",
       "322                      0                2               7   \n",
       "777                      6                2               1   \n",
       "47                       1                3               1   \n",
       "1049                     2                2               2   \n",
       "1403                     3                3              20   \n",
       "87                       4                3               4   \n",
       "1331                     3                3              22   \n",
       "\n",
       "      YearsInCurrentRole  YearsSinceLastPromotion  YearsWithCurrManager  \n",
       "899                    0                        0                     0  \n",
       "610                    8                        0                     8  \n",
       "667                    7                        1                     0  \n",
       "857                    4                        0                     2  \n",
       "1063                   7                        0                     4  \n",
       "1022                   5                        1                     3  \n",
       "665                    2                        1                     2  \n",
       "840                    0                        1                     7  \n",
       "796                    7                        5                     6  \n",
       "1221                  15                       14                     4  \n",
       "149                    0                        0                     0  \n",
       "1139                   2                        0                     2  \n",
       "1420                   3                        1                     0  \n",
       "322                    7                        0                     7  \n",
       "777                    0                        1                     0  \n",
       "47                     0                        0                     0  \n",
       "1049                   1                        2                     2  \n",
       "1403                   8                       11                    10  \n",
       "87                     2                        0                     3  \n",
       "1331                  10                       12                     9  \n",
       "\n",
       "[20 rows x 30 columns]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# One Prediction\n",
    "instance_num = 6\n",
    "# Get the prediction for the first member of the test set and explain why model made that prediction\n",
    "prediction_value = model.predict(x_test)[instance_num]\n",
    "\n",
    "print(\"One Prediction: \")\n",
    "print(prediction_value)\n",
    "\n",
    "print(\"20 Predictions: \")\n",
    "print(y_predictions[:20])\n",
    "\n",
    "x_test.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "899     0\n",
       "610     0\n",
       "667     1\n",
       "857     1\n",
       "1063    0\n",
       "Name: Attrition, dtype: int64"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test.head(5)"
   ]
  }
 ],
 "metadata": {
  "file_extension": ".py",
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "mimetype": "text/x-python",
  "name": "python",
  "npconvert_exporter": "python",
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  },
  "pygments_lexer": "ipython3",
  "version": 3
 },
 "nbformat": 4,
 "nbformat_minor": 4
}