examples/sentence_similarity/automl_with_pipelines_deployment_aks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AzureML Pipeline, AutoML, AKS Deployment for Sentence Similarity"
   ]
  },
    {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/nlp/examples/sentence_similarity/automl_with_pipelines_deployment_aks.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook builds off of the [AutoML Local Deployment ACI](automl_local_deployment_aci.ipynb) notebook and demonstrates how to use [Azure Machine Learning](https://azure.microsoft.com/en-us/services/machine-learning-service/\n",
    ") pipelines and Automated Machine Learning ([AutoML](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-automated-ml\n",
    ")) to streamline the creation of a machine learning workflow for predicting sentence similarity. The pipeline contains two steps:   \n",
    "1. PythonScriptStep: embeds sentences using a popular sentence embedding model, Google Universal Sentence Encoder\n",
    "2. AutoMLStep: demonstrates how to use Automated Machine Learning (AutoML) to automate model selection for predicting sentence similarity (regression)\n",
    "\n",
    "After creating the pipeline, the notebook demonstrates the deployment of our sentence similarity model using Azure Kubernetes Service ([AKS](https://docs.microsoft.com/en-us/azure/aks/intro-kubernetes\n",
    ")).\n",
    "\n",
    "This notebook showcases how to use the following AzureML features:  \n",
    "- AzureML Pipelines (PythonScriptStep and AutoMLStep)\n",
    "- Automated Machine Learning\n",
    "- AmlCompute\n",
    "- Datastore\n",
    "- Logging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "1. [Introduction](#1.-Introduction)  \n",
    "    * 1.1 [What are AzureML Pipelines?](#1.1-What-are-AzureML-Pipelines?)  \n",
    "    * 1.2 [What is Azure AutoML?](#1.2-What-is-Azure-AutoML?)  \n",
    "    * 1.3 [Modeling Problem](#1.3-Modeling-Problem)  \n",
    "2. [Data Preparation](#2.-Data-Preparation)  \n",
    "3. [AzureML Setup](#3.-AzureML-Setup)  \n",
    "    * 3.1 [Link to or create a `Workspace`](#3.1-Link-to-or-create-a-Workspace)  \n",
    "    * 3.2 [Set up an `Experiment` and Logging](#3.2-Set-up-an-Experiment-and-Logging)  \n",
    "    * 3.3 [Link `AmlCompute` compute target](#3.3-Link-AmlCompute-compute-target)  \n",
    "    * 3.4 [Upload data to `Datastore`](#3.4-Upload-data-to-Datastore)  \n",
    "4. [Create AzureML Pipeline](#4.-Create-AzureML-Pipeline)  \n",
    "    * 4.1 [Set up run configuration file](#4.1-Set-up-run-configuration-file)  \n",
    "    * 4.2 [PythonScriptStep](#4.2-PythonScriptStep)  \n",
    "        * 4.2.1 [Define python script to run](#4.2.1-Define-python-script-to-run)\n",
    "        * 4.2.2 [Create PipelineData object](#4.2.2-Create-PipelineData-object)\n",
    "        * 4.2.3 [Create PythonScriptStep](#4.2.3-Create-PythonScriptStep)\n",
    "    * 4.3 [AutoMLStep](#4.3-AutoMLStep)\n",
    "        * 4.3.1 [Define get_data script to load data](#4.3.1-Define-get_data-script-to-load-data)\n",
    "        * 4.3.2 [Create AutoMLConfig object](#4.3.2-Create-AutoMLConfig-object)\n",
    "        * 4.3.3 [Create AutoMLStep](#4.3.3-Create-AutoMLStep)    \n",
    "5. [Run Pipeline](#5.-Run-Pipeline)  \n",
    "6. [Deploy Sentence Similarity Model](#6.-Deploy-Sentence-Similarity-Model)\n",
    "    * 6.1 [Register/Retrieve AutoML and Google Universal Sentence Encoder Models for Deployment](#6.1-Register/Retrieve-AutoML-and-Google-Universal-Sentence-Encoder-Models-for-Deployment)  \n",
    "    * 6.2 [Create Scoring Script](#6.2-Create-Scoring-Script)\n",
    "    * 6.3 [Create a YAML File for the Environment](#6.3-Create-a-YAML-File-for-the-Environment)   \n",
    "    * 6.4 [Image Creation](#6.4-Image-Creation) \n",
    "    * 6.5 [Provision the AKS Cluster](#6.5-Provision-the-AKS-Cluster)   \n",
    "    * 6.6 [Deploy the image as a Web Service to Azure Kubernetes Service](#6.6-Deploy-the-image-as-a-Web-Service-to-Azure-Kubernetes-Service)  \n",
    "    * 6.7 [Test Deployed Model](#6.7-Test-Deployed-Webservice)  \n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 What are AzureML Pipelines?\n",
    "\n",
    "[AzureML Pipelines](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-ml-pipelines) define reusable machine learning workflows that can be used as a template for your machine learning scenarios. Pipelines allow you to optimize your workflow and spend time on machine learning rather than infrastructure. A Pipeline is defined by a series of steps; the following steps are available: AdlaStep, AutoMLStep, AzureBatchStep, DataTransferStep, DatabricksStep, EstimatorStep, HyperDriveStep, ModuleStep, MpiStep, and PythonScriptStep (see [here](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/?view=azure-ml-py) for details of each step). When the pipeline is run, cached results are used for all steps that have not changed, optimizing the run time. Data sources and intermediate data can be used across multiple steps in a pipeline, saving time and resources. Below we see an example of an AzureML pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://nlpbp.blob.core.windows.net/images/pipelines.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 What is Azure AutoML?\n",
    "\n",
    "Automated machine learning ([AutoML](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-automated-ml)) is a capability of Microsoft's [Azure Machine Learning service](https://azure.microsoft.com/en-us/services/machine-learning-service/\n",
    "). The goal of AutoML is to improve the productivity of data scientists and democratize AI by allowing for the rapid development and deployment of machine learning models. To acheive this goal, AutoML automates the process of selecting a ML model and tuning the model. All the user is required to provide is a dataset (suitable for a classification, regression, or time-series forecasting problem) and a metric to optimize in choosing the model and hyperparameters. The user is also given the ability to set time and cost constraints for the model selection and tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](automl.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The AutoML model selection and tuning process can be easily tracked through the Azure portal or directly in python notebooks through the use of widgets. AutoML quickly selects a high quality machine learning model tailored for your prediction problem. In this notebook, we walk through the steps of preparing data, setting up an AutoML experiment, and evaluating the results of our best model. More information about running AutoML experiments in Python can be found [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Modeling Problem\n",
    "\n",
    "The regression problem we will demonstrate is predicting sentence similarity scores on the STS Benchmark dataset. The [STS Benchmark dataset](http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark#STS_benchmark_dataset_and_companion_dataset) contains a selection of English datasets that were used in Semantic Textual Similarity (STS) tasks 2012-2017. The dataset contains 8,628 sentence pairs with a human-labeled integer representing the sentences' similarity (ranging from 0, for no meaning overlap, to 5, meaning equivalence).\n",
    "\n",
    "For each sentence in the sentence pair, we will use Google's pretrained Universal Sentence Encoder (details provided below) to generate a $512$-dimensional embedding. Both embeddings in the sentence pair will be concatenated and the resulting $1024$-dimensional vector will be used as features in our regression problem. Our target variable is the sentence similarity score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\cocochra\\AppData\\Local\\Continuum\\anaconda3\\envs\\nlp_gpu\\lib\\site-packages\\sklearn\\externals\\joblib\\__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.\n",
      "  warnings.warn(msg, category=DeprecationWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Turning diagnostics collection on. \n",
      "System version: 3.6.8 |Anaconda, Inc.| (default, Feb 21 2019, 18:30:04) [MSC v.1916 64 bit (AMD64)]\n",
      "Azure ML SDK Version: 1.0.48\n",
      "Pandas version: 0.24.2\n"
     ]
    }
   ],
   "source": [
    "# Set the environment path to find NLP\n",
    "import sys\n",
    "\n",
    "sys.path.append(\"../../\")\n",
    "import time\n",
    "import logging\n",
    "import csv\n",
    "import os\n",
    "import pandas as pd\n",
    "import shutil\n",
    "import numpy as np\n",
    "import sys\n",
    "from scipy.stats import pearsonr\n",
    "from scipy.spatial import distance\n",
    "from sklearn.externals import joblib\n",
    "import json\n",
    "\n",
    "# Import utils\n",
    "from utils_nlp.azureml import azureml_utils\n",
    "from utils_nlp.dataset import stsbenchmark\n",
    "from utils_nlp.dataset.preprocess import (\n",
    "    to_lowercase,\n",
    "    to_spacy_tokens,\n",
    "    rm_spacy_stopwords,\n",
    ")\n",
    "from utils_nlp.common.timer import Timer\n",
    "\n",
    "# Google Universal Sentence Encoder loader\n",
    "import tensorflow_hub as hub\n",
    "\n",
    "# AzureML packages\n",
    "import azureml as aml\n",
    "import logging\n",
    "from azureml.telemetry import set_diagnostics_collection\n",
    "\n",
    "set_diagnostics_collection(send_diagnostics=True)\n",
    "from azureml.core import Datastore, Experiment, Workspace\n",
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.runconfig import RunConfiguration\n",
    "from azureml.core.conda_dependencies import CondaDependencies\n",
    "from azureml.core.webservice import AksWebservice, Webservice\n",
    "from azureml.core.compute import AksCompute, ComputeTarget\n",
    "from azureml.core.image import ContainerImage\n",
    "from azureml.core.model import Model\n",
    "from azureml.train.automl import AutoMLStep, AutoMLStepRun, AutoMLConfig\n",
    "from azureml.pipeline.core import Pipeline, PipelineData, TrainingOutput\n",
    "from azureml.pipeline.steps import PythonScriptStep\n",
    "from azureml.data.data_reference import DataReference\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Azure ML SDK Version:\", aml.core.VERSION)\n",
    "print(\"Pandas version: {}\".format(pd.__version__))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "BASE_DATA_PATH = \"../../data\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"task\": 'regression',  # type of task: classification, regression or forecasting\n",
    "    \"iteration_timeout_minutes\": 15,  # How long each iteration can take before moving on\n",
    "    \"iterations\": 50,  # Number of algorithm options to try\n",
    "    \"primary_metric\": \"spearman_correlation\",  # Metric to optimize\n",
    "    \"preprocess\": True,  # Whether dataset preprocessing should be applied\n",
    "    \"verbosity\": logging.INFO,\n",
    "    \"blacklist_models\": ['XGBoostRegressor'] #this model is blacklisted due to installation issues\n",
    "}\n",
    "\n",
    "config_path = (\n",
    "    \"./.azureml\"\n",
    ")  # Path to the directory containing config.json with azureml credentials\n",
    "\n",
    "# Azure resources\n",
    "subscription_id = \"YOUR_SUBSCRIPTION_ID\"\n",
    "resource_group = \"YOUR_RESOURCE_GROUP_NAME\"  \n",
    "workspace_name = \"YOUR_WORKSPACE_NAME\"  \n",
    "workspace_region = \"YOUR_WORKSPACE_REGION\" #Possible values eastus, eastus2 and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**STS Benchmark Dataset**\n",
    "\n",
    "As described above, the STS Benchmark dataset contains 8.6K sentence pairs along with a human-annotated score for how similar the two sentences are. We will load the training, development (validation), and test sets provided by STS Benchmark and preprocess the data (lowercase the text, drop irrelevant columns, and rename the remaining columns) using the utils contained in this repo. Each dataset will ultimately have three columns: _sentence1_ and _sentence2_ which contain the text of the sentences in the sentence pair, and _score_ which contains the human-annotated similarity score of the sentence pair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████| 401/401 [00:02<00:00, 198KB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data downloaded to ../../data\\raw\\stsbenchmark\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████| 401/401 [00:02<00:00, 174KB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data downloaded to ../../data\\raw\\stsbenchmark\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████| 401/401 [00:02<00:00, 148KB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data downloaded to ../../data\\raw\\stsbenchmark\n"
     ]
    }
   ],
   "source": [
    "# Load in the raw datasets as pandas dataframes\n",
    "train_raw = stsbenchmark.load_pandas_df(BASE_DATA_PATH, file_split=\"train\")\n",
    "dev_raw = stsbenchmark.load_pandas_df(BASE_DATA_PATH, file_split=\"dev\")\n",
    "test_raw = stsbenchmark.load_pandas_df(BASE_DATA_PATH, file_split=\"test\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean each dataset by lowercasing text, removing irrelevant columns,\n",
    "# and renaming the remaining columns\n",
    "train_clean = stsbenchmark.clean_sts(train_raw)\n",
    "dev_clean = stsbenchmark.clean_sts(dev_raw)\n",
    "test_clean = stsbenchmark.clean_sts(test_raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert all text to lowercase\n",
    "train = to_lowercase(train_clean)\n",
    "dev = to_lowercase(dev_clean)\n",
    "test = to_lowercase(test_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set has 5749 sentences\n",
      "Development set has 1500 sentences\n",
      "Testing set has 1379 sentences\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set has {} sentences\".format(len(train)))\n",
    "print(\"Development set has {} sentences\".format(len(dev)))\n",
    "print(\"Testing set has {} sentences\".format(len(test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>sentence1</th>\n",
       "      <th>sentence2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.00</td>\n",
       "      <td>a plane is taking off.</td>\n",
       "      <td>an air plane is taking off.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.80</td>\n",
       "      <td>a man is playing a large flute.</td>\n",
       "      <td>a man is playing a flute.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.80</td>\n",
       "      <td>a man is spreading shreded cheese on a pizza.</td>\n",
       "      <td>a man is spreading shredded cheese on an uncoo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.60</td>\n",
       "      <td>three men are playing chess.</td>\n",
       "      <td>two men are playing chess.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.25</td>\n",
       "      <td>a man is playing the cello.</td>\n",
       "      <td>a man seated is playing the cello.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   score                                      sentence1  \\\n",
       "0   5.00                         a plane is taking off.   \n",
       "1   3.80                a man is playing a large flute.   \n",
       "2   3.80  a man is spreading shreded cheese on a pizza.   \n",
       "3   2.60                   three men are playing chess.   \n",
       "4   4.25                    a man is playing the cello.   \n",
       "\n",
       "                                           sentence2  \n",
       "0                        an air plane is taking off.  \n",
       "1                          a man is playing a flute.  \n",
       "2  a man is spreading shredded cheese on an uncoo...  \n",
       "3                         two men are playing chess.  \n",
       "4                 a man seated is playing the cello.  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the cleaned data\n",
    "if not os.path.isdir(\"data\"):\n",
    "    os.mkdir(\"data\")\n",
    "\n",
    "train.to_csv(\"data/train.csv\", index=False)\n",
    "test.to_csv(\"data/test.csv\", index=False)\n",
    "dev.to_csv(\"data/dev.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. AzureML Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we set up the necessary components for running this as an AzureML experiment\n",
    "1. Create or link to an existing `Workspace`\n",
    "2. Set up an `Experiment` with `logging`\n",
    "3. Create or attach existing `AmlCompute`\n",
    "4. Upload our data to a `Datastore`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Link to or create a Workspace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell looks to set up the connection to your [Azure Machine Learning service Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace). You can choose to connect to an existing workspace or create a new one. \n",
    "\n",
    "**To access an existing workspace:**\n",
    "1. If you have a `config.json` file, you do not need to provide the workspace information; you will only need to update the `config_path` variable that is defined above which contains the file.\n",
    "2. Otherwise, you will need to supply the following:\n",
    "    * The name of your workspace\n",
    "    * Your subscription id\n",
    "    * The resource group name\n",
    "\n",
    "**To create a new workspace:**\n",
    "\n",
    "Set the following information:\n",
    "* A name for your workspace\n",
    "* Your subscription id\n",
    "* The resource group name\n",
    "* [Azure region](https://azure.microsoft.com/en-us/global-infrastructure/regions/) to create the workspace in, such as `eastus2`. \n",
    "\n",
    "This will automatically create a new resource group for you in the region provided if a resource group with the name given does not already exist. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performing interactive authentication. Please follow the instructions on the terminal.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - Note, we have launched a browser for you to login. For old experience with device code, use \"az login --use-device-code\"\n",
      "WARNING - You have logged in. Now let us find all the subscriptions to which you have access...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Interactive authentication successfully completed.\n"
     ]
    }
   ],
   "source": [
    "ws = azureml_utils.get_or_create_workspace(\n",
    "    config_path=config_path,\n",
    "    subscription_id=subscription_id,\n",
    "    resource_group=resource_group,\n",
    "    workspace_name=workspace_name,\n",
    "    workspace_region=workspace_region,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    \"Workspace name: \" + ws.name,\n",
    "    \"Azure region: \" + ws.location,\n",
    "    \"Subscription id: \" + ws.subscription_id,\n",
    "    \"Resource group: \" + ws.resource_group,\n",
    "    sep=\"\\n\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Set up an Experiment and Logging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a folder for the project\n",
    "project_folder = \"./automl-sentence-similarity\"\n",
    "os.makedirs(project_folder, exist_ok=True)\n",
    "\n",
    "# Set up an experiment\n",
    "experiment_name = \"NLP-SS-googleUSE\"\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "\n",
    "# Add logging to our experiment\n",
    "run = experiment.start_logging()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Link AmlCompute Compute Target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use AzureML Pipelines we need to link a compute target as they can not be run locally. The different options include AmlCompute, Azure Databricks, Remote VMs, etc. All [compute options](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets#supported-compute-targets) can be found in this table with details about whether the given options work with automated ML, pipelines, and GPU. For the following example, we will use an AmlCompute target because it supports Azure Pipelines and GPU. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found existing compute target.\n",
      "{'currentNodeCount': 0, 'targetNodeCount': 0, 'nodeStateCounts': {'preparingNodeCount': 0, 'runningNodeCount': 0, 'idleNodeCount': 0, 'unusableNodeCount': 0, 'leavingNodeCount': 0, 'preemptedNodeCount': 0}, 'allocationState': 'Steady', 'allocationStateTransitionTime': '2019-07-16T01:06:19.136000+00:00', 'errors': None, 'creationTime': '2019-07-09T16:20:30.625908+00:00', 'modifiedTime': '2019-07-09T16:20:46.601973+00:00', 'provisioningState': 'Succeeded', 'provisioningStateTransitionTime': None, 'scaleSettings': {'minNodeCount': 0, 'maxNodeCount': 4, 'nodeIdleTimeBeforeScaleDown': 'PT120S'}, 'vmPriority': 'Dedicated', 'vmSize': 'STANDARD_NC6'}\n"
     ]
    }
   ],
   "source": [
    "# choose your cluster\n",
    "cluster_name = \"gpu-test\"\n",
    "\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
    "    print(\"Found existing compute target.\")\n",
    "except ComputeTargetException:\n",
    "    print(\"Creating a new compute target...\")\n",
    "    compute_config = AmlCompute.provisioning_configuration(\n",
    "        vm_size=\"STANDARD_NC6\", max_nodes=4\n",
    "    )\n",
    "\n",
    "    # create the cluster\n",
    "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
    "\n",
    "    compute_target.wait_for_completion(show_output=True)\n",
    "\n",
    "# use get_status() to get a detailed status for the current AmlCompute.\n",
    "print(compute_target.get_status().serialize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 Upload data to Datastore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This step uploads our local data to a `Datastore` so that the data is accessible from the remote compute target and creates a `DataReference` to point to the location of the data on the Datastore. A DataStore is backed either by a Azure File Storage (default option) or Azure Blob Storage ([how to decide between these options](https://docs.microsoft.com/en-us/azure/storage/common/storage-decide-blobs-files-disks)) and data is made accessible by mounting or copying data to the compute target. `ws.datastores` lists all options for datastores and `ds.account_name` gets the name of the datastore that can be used to find it in the Azure portal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading an estimated of 3 files\n",
      "Uploading ./data\\dev.csv\n",
      "Uploading ./data\\test.csv\n",
      "Uploading ./data\\train.csv\n",
      "Uploaded ./data\\dev.csv, 1 files out of an estimated total of 3\n",
      "Uploaded ./data\\test.csv, 2 files out of an estimated total of 3\n",
      "Uploaded ./data\\train.csv, 3 files out of an estimated total of 3\n",
      "Uploaded 3 files\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "$AZUREML_DATAREFERENCE_6a3eb209d2a04cc6b66a68fa25213980"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select a specific datastore or you can call ws.get_default_datastore()\n",
    "datastore_name = \"workspacefilestore\"\n",
    "ds = ws.datastores[datastore_name]\n",
    "\n",
    "# Upload files in data folder to the datastore\n",
    "ds.upload(\n",
    "    src_dir=\"./data\",\n",
    "    target_path=\"stsbenchmark_data\",\n",
    "    overwrite=True,\n",
    "    show_progress=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also set up a `DataReference` object that points to the data we just uploaded into the stsbenchmark_data folder. DataReference objects point to data that is accessible from a datastore and will be used an an input into our pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data = DataReference(\n",
    "    datastore=ds,\n",
    "    data_reference_name=\"stsbenchmark\",\n",
    "    path_on_datastore=\"stsbenchmark_data/\",\n",
    "    overwrite=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Create AzureML Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we set up our pipeline which is made of two steps:  \n",
    "1. `PythonScriptStep`: takes each sentence pair from the data in the `Datastore` and concatenates the Google USE embeddings for each sentence into one vector. This step saves the embedding feature matrix back to our `Datastore` and uses a `PipelineData` object to represent this intermediate data.  \n",
    "2. `AutoMLStep`: takes the intermediate data produced by the previous step and passes it to an `AutoMLConfig` which performs the automatic model selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 Set up run configuration file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we set up a `RunConfiguration` object which configures the execution environment for an experiment (sets up the conda dependencies, etc.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "format": "row"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "run config is ready\n"
     ]
    }
   ],
   "source": [
    "# create a new RunConfig object\n",
    "conda_run_config = RunConfiguration(framework=\"python\")\n",
    "\n",
    "# Set compute target to AmlCompute\n",
    "conda_run_config.target = compute_target\n",
    "\n",
    "conda_run_config.environment.docker.enabled = True\n",
    "conda_run_config.environment.docker.base_image = aml.core.runconfig.DEFAULT_CPU_IMAGE\n",
    "\n",
    "# Specify our own conda dependencies for the execution environment\n",
    "conda_run_config.environment.python.user_managed_dependencies = False\n",
    "conda_run_config.environment.python.conda_dependencies = CondaDependencies.create(\n",
    "    pip_packages=[\n",
    "        \"azureml-sdk[automl]==1.0.48\",\n",
    "        \"azureml-dataprep==1.1.8\",\n",
    "        \"azureml-train-automl==1.0.48\",\n",
    "    ],\n",
    "    conda_packages=[\n",
    "        \"numpy\",\n",
    "        \"py-xgboost<=0.80\",\n",
    "        \"pandas\",\n",
    "        \"tensorflow\",\n",
    "        \"tensorflow-hub\",\n",
    "        \"scikit-learn\",\n",
    "    ],\n",
    "    pin_sdk_version=False,\n",
    ")\n",
    "\n",
    "print(\"run config is ready\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 PythonScriptStep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`PythonScriptStep` is a step which runs a user-defined Python script ([documentation](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.python_script_step.pythonscriptstep?view=azure-ml-py) here). In this `PythonScriptStep`, we will convert our sentences into a numerical representation in order to use them in our machine learning model. We will embed both sentences using the Google Universal Sentence Encoder (provided by tensorflow-hub) and concatenate their representations into a $1024$-dimensional vector to use as features for AutoML.\n",
    "\n",
    "**Google Universal Sentence Encoder:**\n",
    "We'll use a popular sentence encoder called Google Universal Sentence Encoder (see [original paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46808.pdf)). Google provides two pretrained models based on different design goals: a Transformer model (targets high accuracy even if this reduces model complexity) and a Deep Averaging Network model (DAN; targets efficient inference). Both models are trained on a variety of web sources (Wikipedia, news, question-answers pages, and discussion forums) and produced 512-dimensional embeddings. This notebook utilizes the Transformer-based encoding model which can be downloaded [here](https://tfhub.dev/google/universal-sentence-encoder-large/3) because of its better performance relative to the DAN model on the STS Benchmark dataset (see Table 2 in Google Research's [paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46808.pdf)). The Transformer model produces sentence embeddings using the \"encoding sub-graph of the transformer architecture\" (original architecture introduced [here](https://arxiv.org/abs/1706.03762)). \"This sub-graph uses attention to compute context aware representations of words in a sentence that take into account both the ordering and identity of all the other workds. The context aware word representations are converted to a fixed length sentence encoding vector by computing the element-wise sum of the representations at each word position.\" The input to the model is lowercase PTB-tokenized strings and the model is designed to be useful for multiple different tasks by using multi-task learning. More details about the model can be found in the [paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46808.pdf) by Google Research."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2.1 Define python script to run\n",
    "\n",
    "Define the script (called embed.py) that the `PythonScriptStep` will execute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./automl-sentence-similarity/embed.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $project_folder/embed.py\n",
    "import argparse\n",
    "import os\n",
    "import azureml.core\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)  # reduce logging output\n",
    "\n",
    "\n",
    "def google_encoder(dataset):\n",
    "    \"\"\" Function that embeds sentences using the Google Universal\n",
    "    Sentence Encoder pretrained model\n",
    "    \n",
    "    Parameters:\n",
    "    ----------\n",
    "    dataset: pandas dataframe with sentences and scores\n",
    "    \n",
    "    Returns:\n",
    "    -------\n",
    "    emb1: 512-dimensional representation of sentence1\n",
    "    emb2: 512-dimensional representation of sentence2\n",
    "    \"\"\"\n",
    "    sts_input1 = tf.placeholder(tf.string, shape=(None))\n",
    "    sts_input2 = tf.placeholder(tf.string, shape=(None))\n",
    "\n",
    "    # Apply embedding model and normalize the input\n",
    "    sts_encode1 = tf.nn.l2_normalize(embedding_model(sts_input1), axis=1)\n",
    "    sts_encode2 = tf.nn.l2_normalize(embedding_model(sts_input2), axis=1)\n",
    "\n",
    "    with tf.Session() as session:\n",
    "        session.run(tf.global_variables_initializer())\n",
    "        session.run(tf.tables_initializer())\n",
    "        emb1, emb2 = session.run(\n",
    "            [sts_encode1, sts_encode2],\n",
    "            feed_dict={\n",
    "                sts_input1: dataset[\"sentence1\"],\n",
    "                sts_input2: dataset[\"sentence2\"],\n",
    "            },\n",
    "        )\n",
    "    return emb1, emb2\n",
    "\n",
    "\n",
    "def feature_engineering(dataset):\n",
    "    \"\"\"Extracts embedding features from the dataset and returns\n",
    "    features and target in a dataframe\n",
    "    \n",
    "    Parameters:\n",
    "    ----------\n",
    "    dataset: pandas dataframe with sentences and scores\n",
    "    \n",
    "    Returns:\n",
    "    -------\n",
    "    df: pandas dataframe with embedding features\n",
    "    scores: list of target variables\n",
    "    \"\"\"\n",
    "    google_USE_emb1, google_USE_emb2 = google_encoder(dataset)\n",
    "    n_google = google_USE_emb1.shape[1]  # length of the embeddings\n",
    "    df = np.concatenate((google_USE_emb1, google_USE_emb2), axis=1)\n",
    "    names = [\"USEEmb1_\" + str(i) for i in range(n_google)] + [\n",
    "        \"USEEmb2_\" + str(i) for i in range(n_google)\n",
    "    ]\n",
    "    df = pd.DataFrame(df, columns=names)\n",
    "    return df, dataset[\"score\"]\n",
    "\n",
    "\n",
    "def write_output(df, path, name):\n",
    "    \"\"\"Write dataframes to correct path\"\"\"\n",
    "    os.makedirs(path, exist_ok=True)\n",
    "    print(\"%s created\" % path)\n",
    "    df.to_csv(path + \"/\" + name, index=False)\n",
    "\n",
    "\n",
    "# Parse arguments\n",
    "parser = argparse.ArgumentParser()\n",
    "parser.add_argument(\"--sentence_data\", type=str)\n",
    "parser.add_argument(\"--embedded_data\", type=str)\n",
    "args = parser.parse_args()\n",
    "\n",
    "# Import the Universal Sentence Encoder's TF Hub module\n",
    "module_url = \"https://tfhub.dev/google/universal-sentence-encoder-large/3\"\n",
    "embedding_model = hub.Module(module_url)\n",
    "\n",
    "# Read data\n",
    "train = pd.read_csv(args.sentence_data + \"/train.csv\")\n",
    "dev = pd.read_csv(args.sentence_data + \"/dev.csv\")\n",
    "\n",
    "# Get Google USE features\n",
    "training_data, training_scores = feature_engineering(train)\n",
    "validation_data, validation_scores = feature_engineering(dev)\n",
    "\n",
    "# Write out training data to Datastore\n",
    "write_output(training_data, args.embedded_data, \"X_train.csv\")\n",
    "write_output(\n",
    "    pd.DataFrame(training_scores, columns=[\"score\"]), args.embedded_data, \"y_train.csv\"\n",
    ")\n",
    "\n",
    "# Write out validation data to Datastore\n",
    "write_output(validation_data, args.embedded_data, \"X_dev.csv\")\n",
    "write_output(\n",
    "    pd.DataFrame(validation_scores, columns=[\"score\"]), args.embedded_data, \"y_dev.csv\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2.2 Create PipelineData object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`PipelineData` objects represent a piece of intermediate data in a pipeline. Generally they are produced by one step (as an output) and then consumed by the next step (as an input), introducing an implicit order between steps in a pipeline. We create a PipelineData object that can represent the data produced by our first pipeline step that will be consumed by our second pipeline step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedded_data = PipelineData(\"embedded_data\", datastore=ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2.3 Create PythonScriptStep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This step defines the `PythonScriptStep`. We give the step a name, tell the step which python script to run (embed.py) and what directory that script is located in (source_directory). \n",
    "\n",
    "We also link the compute target and run configuration that we made previously. Our input is the `DataReference` object (input_data) where our raw sentence data was uploaded and our ouput is the `PipelineData` object (embedded_data) where the embedded data produced by this step will be stored. These are also passed in as arguments so that we have access to the correct data paths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "embed_step = PythonScriptStep(\n",
    "    name=\"Embed\",\n",
    "    script_name=\"embed.py\",\n",
    "    arguments=[\"--embedded_data\", embedded_data, \"--sentence_data\", input_data],\n",
    "    inputs=[input_data],\n",
    "    outputs=[embedded_data],\n",
    "    compute_target=compute_target,\n",
    "    runconfig=conda_run_config,\n",
    "    source_directory=project_folder,\n",
    "    allow_reuse=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3 AutoMLStep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`AutoMLStep` creates an AutoML step in a pipeline (see [documentation](https://docs.microsoft.com/en-us/python/api/azureml-train-automl/azureml.train.automl.automlstep?view=azure-ml-py) and [basic example](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-automated-machine-learning-step.ipynb)). When using AutoML on remote compute, rather than passing our data directly into the `AutoMLConfig` object as we did in the local example, we must define a get_data.py script with a get_data() function to pass as the data_script argument. This workflow can be used for both local and remote executions (see [details](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-auto-train-remote)). \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3.1 Define get_data script to load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the get_data.py file and get_data() function that the `AutoMLStep` will execute to collect data. When AutoML is used with a remote compute, the data can not be passed directly as parameters. Rather, a get_data function must be defined to access the data (see [this resource](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-auto-train-remote) for further details). Note that we can directly access the path of the intermediate data (called embedded_data) through `os.environ['AZUREML_DATAREFERENCE_embedded_data']`. This is necessary because the AutoMLStep does not accept additional parameters like the PythonScriptStep does with `arguments`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./automl-sentence-similarity/get_data.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $project_folder/get_data.py\n",
    "\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "# get location of the embedded_data for future use\n",
    "EMBEDDED_DATA_REF = os.environ[\"AZUREML_DATAREFERENCE_embedded_data\"]\n",
    "\n",
    "def get_data():\n",
    "    \"\"\"Function needed to load data for use on remote AutoML experiments\"\"\"\n",
    "    X_train = pd.read_csv(EMBEDDED_DATA_REF + \"/X_train.csv\")\n",
    "    y_train = pd.read_csv(EMBEDDED_DATA_REF + \"/y_train.csv\")\n",
    "    X_dev = pd.read_csv(EMBEDDED_DATA_REF + \"/X_dev.csv\")\n",
    "    y_dev = pd.read_csv(EMBEDDED_DATA_REF + \"/y_dev.csv\")\n",
    "    return {\"X\": X_train.values, \"y\": y_train.values.flatten(), \"X_valid\": X_dev.values, \"y_valid\": y_dev.values.flatten()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3.2 Create AutoMLConfig object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we specify the parameters for the `AutoMLConfig` class:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**task**  \n",
    "AutoML supports the following base learners for the regression task: Elastic Net, Light GBM, Gradient Boosting, Decision Tree, K-nearest Neighbors, LARS Lasso, Stochastic Gradient Descent, Random Forest, Extremely Randomized Trees, XGBoost, DNN Regressor, Linear Regression. In addition, AutoML also supports two kinds of ensemble methods: voting (weighted average of the output of multiple base learners) and stacking (training a second \"metalearner\" which uses the base algorithms' predictions to predict the target variable). Specific base learners can be included or excluded in the parameters for the AutoMLConfig class (whitelist_models and blacklist_models) and the voting/stacking ensemble options can be specified as well (enable_voting_ensemble and enable_stack_ensemble)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**preprocess**  \n",
    "AutoML also has advanced preprocessing methods, eliminating the need for users to perform this manually. Data is automatically scaled and normalized but an additional parameter in the AutoMLConfig class enables the use of more advanced techniques including imputation, generating additional features, transformations, word embeddings, etc. (full list found [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-create-portal-experiments#preprocess)). Note that algorithm-specific preprocessing will be applied even if preprocess=False. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**primary_metric**  \n",
    "The regression metrics available are the following: Spearman Correlation (spearman_correlation), Normalized RMSE (normalized_root_mean_squared_error), Normalized MAE (normalized_mean_absolute_error), and R2 score (r2_score) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Constraints:**  \n",
    "There is a cost_mode parameter to set cost prediction modes (see options [here](https://docs.microsoft.com/en-us/python/api/azureml-train-automl/azureml.train.automl.automlconfig?view=azure-ml-py)). To set constraints on time there are multiple parameters including experiment_exit_score (target score to exit the experiment after achieving), experiment_timeout_minutes (maximum amount of time for all combined iterations), and iterations (total number of different algorithm and parameter combinations to try)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_config = AutoMLConfig(\n",
    "    debug_log=\"automl_errors.log\",\n",
    "    path=project_folder,\n",
    "    compute_target=compute_target,\n",
    "    run_configuration=conda_run_config,\n",
    "    data_script=project_folder\n",
    "    + \"/get_data.py\",  # local path to script with get_data() function\n",
    "    **automl_settings #where the AutoML main settings are defined\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3.3 Create AutoMLStep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we create `PipelineData` objects for the model data (our outputs) and then create the `AutoMLStep`. The `AutoMLStep` requires a `AutoMLConfig` object and we pass our intermediate data (embedded_data) in as the inputs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create PipelineData objects for tracking AutoML metrics\n",
    "\n",
    "metrics_data = PipelineData(\n",
    "    name=\"metrics_data\",\n",
    "    datastore=ds,\n",
    "    pipeline_output_name=\"metrics_output\",\n",
    "    training_output=TrainingOutput(type=\"Metrics\"),\n",
    ")\n",
    "model_data = PipelineData(\n",
    "    name=\"model_data\",\n",
    "    datastore=ds,\n",
    "    pipeline_output_name=\"best_model_output\",\n",
    "    training_output=TrainingOutput(type=\"Model\"),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_step = AutoMLStep(\n",
    "    name=\"AutoML\",\n",
    "    automl_config=automl_config,  # the AutoMLConfig object created previously\n",
    "    inputs=[\n",
    "        embedded_data\n",
    "    ],  # inputs is the PipelineData that was the output of the previous pipeline step\n",
    "    outputs=[\n",
    "        metrics_data,\n",
    "        model_data,\n",
    "    ],  # PipelineData objects to reference metric and model information\n",
    "    allow_reuse=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Run Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we set up our pipeline which requires specifying our `Workspace` and the ordering of the steps that we created (steps parameter). We submit the pipeline and inspect the run details using a RunDetails widget. For remote runs, the execution of iterations is asynchronous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - 'auto_prepare_environment' is deprecated and unused. It will be removed in a future release.\n"
     ]
    }
   ],
   "source": [
    "pipeline = Pipeline(\n",
    "    description=\"pipeline_embed_automl\",  # give a name for the pipeline\n",
    "    workspace=ws,\n",
    "    steps=[embed_step, automl_step],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created step Embed [d14b211c][a0500165-a3a6-4963-9d8a-7b3dc5981478], (This step will run and generate new outputs)\n",
      "Created step AutoML [b676f3ac][f37c2b71-e7be-486e-85cf-65c95d59d04f], (This step will run and generate new outputs)\n",
      "Using data reference stsbenchmark for StepId [35ea3ed1][e3340790-c54f-4147-8dd0-bcb80a9b7b46], (Consumers of this data are eligible to reuse prior runs.)\n",
      "Submitted pipeline run: 61516c51-af97-458b-a743-93fd3cbd7abf\n"
     ]
    }
   ],
   "source": [
    "pipeline_run = experiment.submit(pipeline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect the run details using the provided widget\n",
    "RunDetails(pipeline_run).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://nlpbp.blob.core.windows.net/images/pipelineWidget.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, block until the run has completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_run.wait_for_completion(\n",
    "    show_output=True\n",
    ")  # show console output while run is in progress"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cancel the Run**\n",
    "\n",
    "Interrupting/Restarting the jupyter kernel will not properly cancel the run, which can lead to wasted compute resources. To avoid this, we recommend explicitly canceling a run with the following code:\n",
    "\n",
    "`pipeline_run.cancel()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Deploy Sentence Similarity Model\n",
    "\n",
    "Deploying an Azure Machine Learning model as a web service creates a REST API. You can send data to this API and receive the prediction returned by the model.\n",
    "In general, you create a webservice by deploying a model as an image to a Compute Target.\n",
    "\n",
    "Some of the Compute Targets are: \n",
    "1. Azure Container Instance\n",
    "2. Azure Kubernetes Service\n",
    "3. Local web service\n",
    "\n",
    "The general workflow for deploying a model is as follows:\n",
    "1. Register a model\n",
    "2. Prepare to deploy\n",
    "3. Deploy the model to the compute target\n",
    "4. Test the deployed model (webservice)\n",
    "\n",
    "In this notebook we walk you through the process of creating a webservice running on Azure Kubernetes Service ([AKS](https://docs.microsoft.com/en-us/azure/aks/intro-kubernetes\n",
    ")) by deploying the model as an image. AKS is good for high-scale production deployments. It provides fast response time and autoscaling of the deployed service. Cluster autoscaling is not supported through the Azure Machine Learning SDK. \n",
    "\n",
    "You can find more information on deploying and serving models [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-deploy-and-where)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1 Register/Retrieve AutoML and Google Universal Sentence Encoder Models for Deployment\n",
    "\n",
    "Registering a model means registering one or more files that make up a model. The Machine Learning models are registered in your current Aure Machine Learning Workspace. The model can either come from Azure Machine Learning or another location, such as your local machine.\n",
    "\n",
    "See other ways to register a model [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-deploy-and-where)\n",
    "\n",
    "Below we show how to register a new model and also how to retrieve and register an existing model.\n",
    "\n",
    "### Register a new automl model\n",
    "Register the best AutoML model based on the pipeline results or load the saved model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - Received unrecognized parameter: time_column_name None\n",
      "WARNING - Received unrecognized parameter: grain_column_names None\n",
      "WARNING - Received unrecognized parameter: drop_column_names None\n",
      "WARNING - Received unrecognized parameter: group None\n",
      "WARNING - Received unrecognized parameter: target_lags None\n",
      "WARNING - Received unrecognized parameter: target_rolling_window_size None\n",
      "WARNING - Received unrecognized parameter: max_horizon None\n",
      "WARNING - Received unrecognized parameter: country_or_region None\n",
      "WARNING - Received unrecognized parameter: seasonality None\n",
      "WARNING - Received unrecognized parameter: use_stl None\n",
      "WARNING - Received unrecognized parameter: season_trend None\n",
      "WARNING - Received unrecognized parameter: season None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Registering model 711e9373160c4a8best\n",
      "711e9373160c4a8best\n"
     ]
    }
   ],
   "source": [
    "automl_step_run = AutoMLStepRun(step_run=pipeline_run.find_step_run(\"AutoML\")[0])\n",
    "\n",
    "# to register the fitted_mode\n",
    "description = \"Pipeline AutoML Model\"\n",
    "tags = {\"area\": \"nlp\", \"type\": \"sentencesimilarity pipelines\"}\n",
    "model = automl_step_run.register_model(description=description, tags=tags)\n",
    "automl_model_name = automl_step_run.model_id\n",
    "print(\n",
    "    automl_step_run.model_id\n",
    ")  # Use this id to deploy the model as a web service in Azure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve existing model from Azure\n",
    "If you already have a best model then you can skip registering the model by just retrieving the latest version of model by providing its name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found model with name 711e9373160c4a8best\n"
     ]
    }
   ],
   "source": [
    "automl_model_name = \"711e9373160c4a8best\"  # best fit model registered in the workspace\n",
    "model = Model(ws, name=automl_model_name)\n",
    "print(\"Found model with name\", automl_model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Register Google Universal Sentence Encoder Model\n",
    "Register the Google Universal Sentence Encoder model if not already registered in your workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Registering model googleUSEmodel\n",
      "Registered googleUSEembeddings model\n"
     ]
    }
   ],
   "source": [
    "# set location for where to download google tensorflow model\n",
    "os.environ[\"TFHUB_CACHE_DIR\"] = \"./googleUSE\"\n",
    "# download model\n",
    "hub.Module(\"https://tfhub.dev/google/universal-sentence-encoder-large/3\")\n",
    "# register model\n",
    "embedding_model = Model.register(\n",
    "    model_path=\"googleUSE\",\n",
    "    model_name=\"googleUSEmodel\",\n",
    "    tags={\"Model\": \"GoogleUSE\"},\n",
    "    description=\"Google Universal Sentence Embedding pretrained model\",\n",
    "    workspace=ws,\n",
    ")\n",
    "print(\"Registered googleUSEembeddings model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve existing Google USE model from Azure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found model with name googleUSEembeddings\n"
     ]
    }
   ],
   "source": [
    "embedding_model = Model(ws, name=\"googleUSEmodel\")\n",
    "print(\"Found model with name googleUSEembeddings\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2 Create Scoring Script\n",
    "\n",
    "In this section we show an example of an entry script, which is called from the deployed webservice. `score.py` is our entry script. The script must contain:\n",
    "1. init() - This function loads the model in a global object.\n",
    "2. run() - This function is used for model prediction. The inputs and outputs to `run()` typically use JSON for serialization and deserilization. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing score.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile score.py\n",
    "import pickle\n",
    "import json\n",
    "import numpy as np\n",
    "import azureml.train.automl\n",
    "from sklearn.externals import joblib\n",
    "from azureml.core.model import Model\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "import os\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)  # reduce logging output\n",
    "\n",
    "\n",
    "def google_encoder(dataset):\n",
    "    \"\"\" Function that embeds sentences using the Google Universal\n",
    "    Sentence Encoder pretrained model\n",
    "    \n",
    "    Parameters:\n",
    "    ----------\n",
    "    dataset: pandas dataframe with sentences and scores\n",
    "    \n",
    "    Returns:\n",
    "    -------\n",
    "    emb1: 512-dimensional representation of sentence1\n",
    "    emb2: 512-dimensional representation of sentence2\n",
    "    \"\"\"\n",
    "    global embedding_model, sess\n",
    "    sts_input1 = tf.placeholder(tf.string, shape=(None))\n",
    "    sts_input2 = tf.placeholder(tf.string, shape=(None))\n",
    "\n",
    "    # Apply embedding model and normalize the input\n",
    "    sts_encode1 = tf.nn.l2_normalize(embedding_model(sts_input1), axis=1)\n",
    "    sts_encode2 = tf.nn.l2_normalize(embedding_model(sts_input2), axis=1)\n",
    "\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    sess.run(tf.tables_initializer())\n",
    "    emb1, emb2 = sess.run(\n",
    "        [sts_encode1, sts_encode2],\n",
    "        feed_dict={sts_input1: dataset[\"sentence1\"], sts_input2: dataset[\"sentence2\"]},\n",
    "    )\n",
    "    return emb1, emb2\n",
    "\n",
    "\n",
    "def feature_engineering(dataset):\n",
    "    \"\"\"Extracts embedding features from the dataset and returns\n",
    "    features and target in a dataframe\n",
    "    \n",
    "    Parameters:\n",
    "    ----------\n",
    "    dataset: pandas dataframe with sentences and scores\n",
    "    \n",
    "    Returns:\n",
    "    -------\n",
    "    df: pandas dataframe with embedding features\n",
    "    scores: list of target variables\n",
    "    \"\"\"\n",
    "    google_USE_emb1, google_USE_emb2 = google_encoder(dataset)\n",
    "    return np.concatenate((google_USE_emb1, google_USE_emb2), axis=1)\n",
    "\n",
    "\n",
    "def init():\n",
    "    global model, googleUSE_dir_path\n",
    "    model_path = Model.get_model_path(\n",
    "        model_name=\"<<modelid>>\"\n",
    "    )  # this name is model.id of model that we want to deploy\n",
    "    # deserialize the model file back into a sklearn model\n",
    "    model = joblib.load(model_path)\n",
    "\n",
    "    # load the path for google USE embedding model\n",
    "    googleUSE_dir_path = Model.get_model_path(model_name=\"googleUSEmodel\")\n",
    "    os.environ[\"TFHUB_CACHE_DIR\"] = googleUSE_dir_path\n",
    "\n",
    "\n",
    "def run(rawdata):\n",
    "    global embedding_model, sess, googleUSE_dir_path, model\n",
    "    try:\n",
    "        # load data and convert to dataframe\n",
    "        data = json.loads(rawdata)[\"data\"]\n",
    "        data_df = pd.DataFrame(data, columns=[\"sentence1\", \"sentence2\"])\n",
    "\n",
    "        # begin a tensorflow session and load tensorhub module\n",
    "        sess = tf.Session()\n",
    "        embedding_model = hub.Module(\n",
    "            googleUSE_dir_path + \"/96e8f1d3d4d90ce86b2db128249eb8143a91db73\"\n",
    "        )\n",
    "\n",
    "        # Embed sentences using Google USE model\n",
    "        embedded_data = feature_engineering(data_df)\n",
    "        # Predict using AutoML saved model\n",
    "        result = model.predict(embedded_data)\n",
    "\n",
    "    except Exception as e:\n",
    "        result = str(e)\n",
    "        sess.close()\n",
    "        return json.dumps({\"error\": result})\n",
    "\n",
    "    sess.close()\n",
    "    return json.dumps({\"result\": result.tolist()})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Substitute the actual model id in the script file.\n",
    "script_file_name = \"score.py\"\n",
    "\n",
    "with open(script_file_name, \"r\") as cefr:\n",
    "    content = cefr.read()\n",
    "\n",
    "with open(script_file_name, \"w\") as cefw:\n",
    "    cefw.write(content.replace(\"<<modelid>>\", automl_model_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.3 Create a YAML File for the Environment\n",
    "\n",
    "To ensure the fit results are consistent with the training results, the SDK dependency versions need to be the same as the environment that trains the model. The following cells create a file, pipeline_env.yml, which specifies the dependencies from the run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'pipeline_env.yml'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myenv = CondaDependencies.create(\n",
    "    conda_packages=[\n",
    "        \"numpy\",\n",
    "        \"scikit-learn\",\n",
    "        \"py-xgboost<=0.80\",\n",
    "        \"pandas\",\n",
    "        \"tensorflow\",\n",
    "        \"tensorflow-hub\",\n",
    "    ],\n",
    "    pip_packages=[\"azureml-sdk[automl]==1.0.48.*\"],\n",
    "    python_version=\"3.6.8\",\n",
    ")\n",
    "\n",
    "conda_env_file_name = \"pipeline_env.yml\"\n",
    "myenv.save_to_file(\".\", conda_env_file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.4 Image Creation\n",
    "\n",
    "In this step we create a container image which is wrapper containing the entry script, yaml file with package dependencies and the model. The created image is then deployed as a webservice in the next step. This step can take up to 10 minutes and even longer if the model is large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating image\n",
      "Running..................................................................................\n",
      "Succeeded\n",
      "Image creation operation finished for image pipeline-automl-image:1, operation \"Succeeded\"\n"
     ]
    }
   ],
   "source": [
    "# trying to add dependencies\n",
    "image_config = ContainerImage.image_configuration(\n",
    "    execution_script=script_file_name,\n",
    "    runtime=\"python\",\n",
    "    conda_file=conda_env_file_name,\n",
    "    description=\"Image with aml pipeline model\",\n",
    "    tags={\"area\": \"nlp\", \"type\": \"sentencesimilarity pipeline\"},\n",
    ")\n",
    "\n",
    "image = ContainerImage.create(\n",
    "    name=\"pipeline-automl-image\",\n",
    "    # this is the model object\n",
    "    models=[model, embedding_model],  # add both embedding and autoML models\n",
    "    image_config=image_config,\n",
    "    workspace=ws,\n",
    ")\n",
    "\n",
    "image.wait_for_creation(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the above step fails, then use below command to see logs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# image.get_logs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.5 Provision the AKS Cluster\n",
    "\n",
    "**Time estimate:** Approximately 20 minutes.\n",
    "\n",
    "Creating or attaching an AKS cluster is a one time process for your workspace. You can reuse this cluster for multiple deployments. If you delete the cluster or the resource group that contains it, you must create a new cluster the next time you need to deploy. You can have multiple AKS clusters attached to your workspace.\n",
    "\n",
    "**Note:** Check the Azure Portal to make sure that the AKS Cluster has been provisioned properly before moving forward with this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create aks cluser\n",
    "\n",
    "# Use the default configuration (can also provide parameters to customize)\n",
    "prov_config = AksCompute.provisioning_configuration()\n",
    "\n",
    "# Create the cluster\n",
    "aks_target = ComputeTarget.create(\n",
    "    workspace=ws, name=\"nlp-aks-cluster\", provisioning_configuration=prov_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 6.6 Deploy the Image as a Web Service on Azure Kubernetes Service\n",
    "\n",
    "In the case of deployment on AKS, in addition to the Docker image, we need to define computational resources. This is typically a cluster of CPUs or a cluster of GPUs. If we already have a Kubernetes-managed cluster in our workspace, we can use it, otherwise, we can create a new one.\n",
    "\n",
    "In this notebook we will use the cluster in the above cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the web service configuration\n",
    "aks_config = AksWebservice.deploy_configuration()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to deploy our web service. We will deploy from the Docker image. It contains our AutoML model as well as the Google Universal Sentence Encoder model and the conda environment needed for the scoring script to work properly. The parameters to pass to the Webservice.deploy_from_image() command are similar to those used for deployment on Azure Container Instance ([ACI](https://azure.microsoft.com/en-us/services/container-instances/\n",
    ")). The only major difference is the compute target (aks_target), i.e. the CPU cluster we just spun up.\n",
    "\n",
    "**Note:** This deployment takes a few minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating service\n",
      "Running.........................\n",
      "SucceededAKS service creation operation finished, operation \"Succeeded\"\n",
      "Healthy\n"
     ]
    }
   ],
   "source": [
    "# deploy image as web service\n",
    "aks_service_name = \"aks-pipelines-service\"\n",
    "\n",
    "aks_service = Webservice.deploy_from_image(\n",
    "    workspace=ws,\n",
    "    name=aks_service_name,\n",
    "    image=image,\n",
    "    deployment_config=aks_config,\n",
    "    deployment_target=aks_target,\n",
    ")\n",
    "aks_service.wait_for_deployment(show_output=True)\n",
    "print(aks_service.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the above step fails then use below command to see logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# aks_service.get_logs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.7 Test Deployed Webservice\n",
    "\n",
    "Testing the deployed model means running the created webservice. <br>\n",
    "The deployed model can be tested by passing a list of sentence pairs. The output will be a score between 0 and 5, with 0 indicating no meaning overlap between the sentences and 5 meaning equivalence.\n",
    "\n",
    "The run method expects input in json format. The Run() method retrieves API keys behind the scenes to make sure that the call is authenticated. The service has a timeout (default of ~30 seconds) which does not allow passing the large test dataset. To overcome this, you can batch data and send it to the service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = [\n",
    "    [\"This is sentence1\", \"This is sentence1\"],\n",
    "    [\"A hungry cat.\", \"A sleeping cat\"],\n",
    "    [\"Its summer time \", \"Winter is coming\"],\n",
    "]\n",
    "data = {\"data\": sentences}\n",
    "data = json.dumps(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time elapsed: 12.8143\n",
      "Number of samples predicted: 3\n",
      "[3.7827566108065453, 2.7329700382428097, 2.4850673912463717]\n"
     ]
    }
   ],
   "source": [
    "# Set up a Timer to see how long the model takes to predict\n",
    "t = Timer()\n",
    "\n",
    "t.start()\n",
    "score = aks_service.run(input_data=data)\n",
    "t.stop()\n",
    "\n",
    "print(\"Time elapsed: {}\".format(t))\n",
    "\n",
    "result = json.loads(score)\n",
    "try:\n",
    "    output = result[\"result\"]\n",
    "    print(\"Number of samples predicted: {}\".format(len(output)))\n",
    "    print(output)\n",
    "except:\n",
    "    print(result[\"error\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we'll calculate the Pearson Correlation on the test set.\n",
    "\n",
    "**What is Pearson Correlation?**\n",
    "\n",
    "Our evaluation metric is Pearson correlation ($\\rho$) which is a measure of the linear correlation between two variables. The formula for calculating Pearson correlation is as follows:  \n",
    "\n",
    "$$\\rho_{X,Y} = \\frac{E[(X-\\mu_X)(Y-\\mu_Y)]}{\\sigma_X \\sigma_Y}$$\n",
    "\n",
    "This metric takes a value in [-1,1] where -1 represents a perfect negative correlation, 1 represents a perfect positive correlation, and 0 represents no correlation. We utilize the Pearson correlation metric as this is the main metric that [SentEval](http://nlpprogress.com/english/semantic_textual_similarity.html), a widely-used evaluation toolkit for evaluation sentence representations, uses for the STS Benchmark dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load test set sentences\n",
    "data = pd.read_csv(\"data/test.csv\")\n",
    "train_y = data[\"score\"].values.flatten()\n",
    "train_x = data.drop(\"score\", axis=1).values.tolist()\n",
    "data = {\"data\": train_x[:500]}\n",
    "data = json.dumps(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time elapsed: 17.3619\n",
      "Number of sample predicted : 500\n"
     ]
    }
   ],
   "source": [
    "# Set up a Timer to see how long the model takes to predict\n",
    "t = Timer()\n",
    "\n",
    "t.start()\n",
    "score = aks_service.run(input_data=data)\n",
    "t.stop()\n",
    "\n",
    "print(\"Time elapsed: {}\".format(t))\n",
    "\n",
    "result = json.loads(score)\n",
    "\n",
    "try:\n",
    "    output = result[\"result\"]\n",
    "    print(\"Number of sample predicted : {}\".format(len(output)))\n",
    "except:\n",
    "    print(result[\"error\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8706075673971211\n"
     ]
    }
   ],
   "source": [
    "# get Pearson Correlation\n",
    "print(pearsonr(output, train_y[:500])[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This notebook demonstrated how to use AzureML Pipelines and AutoML to streamline the creation of a machine learning workflow for predicting sentence similarity. After creating the pipeline, the notebook demonstrated the deployment of our sentence similarity model using AKS. The model results reported in this notebook (using Google USE embeddings) are much stronger than the results from using AutoML with its built-in embedding capabilities (as in [AutoML Local Deployment ACI](automl_local_deployment_aci.ipynb)). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}