assignments/P6-deploying-model/L2-building-a-model-using-sagemaker/IMDB Sentiment Analysis - XGBoost (Batch Transform).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Analysis\n",
    "\n",
    "## Using XGBoost in SageMaker\n",
    "\n",
    "_Deep Learning Nanodegree Program | Deployment_\n",
    "\n",
    "---\n",
    "\n",
    "As our first example of using Amazon's SageMaker service we will construct a random tree model to predict the sentiment of a movie review. You may have seen a version of this example in a pervious lesson although it would have been done using the sklearn package. Instead, we will be using the XGBoost package as it is provided to us by Amazon.\n",
    "\n",
    "## Instructions\n",
    "\n",
    "Some template code has already been provided for you, and you will need to implement additional functionality to successfully complete this notebook. You will not need to modify the included code beyond what is requested. Sections that begin with '**TODO**' in the header indicate that you need to complete or implement some portion within them. Instructions will be provided for each section and the specifics of the implementation are marked in the code block with a `# TODO: ...` comment. Please be sure to read the instructions carefully!\n",
    "\n",
    "In addition to implementing code, there may be questions for you to answer which relate to the task and your implementation. Each section where you will answer a question is preceded by a '**Question:**' header. Carefully read each question and provide your answer below the '**Answer:**' header by editing the Markdown cell.\n",
    "\n",
    "> **Note**: Code and Markdown cells can be executed using the **Shift+Enter** keyboard shortcut. In addition, a cell can be edited by typically clicking it (double-click for Markdown cells) or by pressing **Enter** while it is highlighted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Downloading the data\n",
    "\n",
    "The dataset we are going to use is very popular among researchers in Natural Language Processing, usually referred to as the [IMDb dataset](http://ai.stanford.edu/~amaas/data/sentiment/). It consists of movie reviews from the website [imdb.com](http://www.imdb.com/), each labeled as either '**pos**itive', if the reviewer enjoyed the film, or '**neg**ative' otherwise.\n",
    "\n",
    "> Maas, Andrew L., et al. [Learning Word Vectors for Sentiment Analysis](http://ai.stanford.edu/~amaas/data/sentiment/). In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, 2011.\n",
    "\n",
    "We begin by using some Jupyter Notebook magic to download and extract the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘../data’: File exists\n",
      "--2019-04-09 20:03:17--  http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
      "Resolving ai.stanford.edu (ai.stanford.edu)... 171.64.68.10\n",
      "Connecting to ai.stanford.edu (ai.stanford.edu)|171.64.68.10|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 84125825 (80M) [application/x-gzip]\n",
      "Saving to: ‘../data/aclImdb_v1.tar.gz’\n",
      "\n",
      "../data/aclImdb_v1. 100%[===================>]  80.23M  24.3MB/s    in 3.9s    \n",
      "\n",
      "2019-04-09 20:03:21 (20.6 MB/s) - ‘../data/aclImdb_v1.tar.gz’ saved [84125825/84125825]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%mkdir ../data\n",
    "!wget -O ../data/aclImdb_v1.tar.gz http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
    "!tar -zxf ../data/aclImdb_v1.tar.gz -C ../data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 82160\r\n",
      "drwxr-xr-x 4 ec2-user ec2-user     4096 Jun 26  2011 aclImdb\r\n",
      "-rw-rw-r-- 1 ec2-user ec2-user 84125825 Jun 26  2011 aclImdb_v1.tar.gz\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l ../data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 1724\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user 903029 Jun 11  2011 imdbEr.txt\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user 845980 Apr 12  2011 imdb.vocab\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user   4037 Jun 26  2011 README\r\n",
      "drwxr-xr-x 4 ec2-user ec2-user   4096 Apr  9 20:03 test\r\n",
      "drwxr-xr-x 5 ec2-user ec2-user   4096 Apr  9 20:03 train\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l ../data/aclImdb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 21600\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user 20205283 Apr 12  2011 labeledBow.feat\r\n",
      "drwxr-xr-x 2 ec2-user ec2-user   339968 Apr  9 20:03 neg\r\n",
      "drwxr-xr-x 2 ec2-user ec2-user   344064 Apr  9 20:03 pos\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user   612500 Apr 12  2011 urls_neg.txt\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user   612500 Apr 12  2011 urls_pos.txt\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l ../data/aclImdb/test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 ec2-user ec2-user 1645 Apr 12  2011 9991_1.txt\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user 3864 Apr 12  2011 9992_4.txt\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user 1203 Apr 12  2011 9993_2.txt\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user 1624 Apr 12  2011 999_3.txt\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user 1123 Apr 12  2011 9994_3.txt\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user 1025 Apr 12  2011 9995_2.txt\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user 1899 Apr 12  2011 9996_2.txt\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user  721 Apr 12  2011 9997_2.txt\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user  645 Apr 12  2011 9998_1.txt\r\n",
      "-rw-r--r-- 1 ec2-user ec2-user  877 Apr 12  2011 9999_1.txt\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l ../data/aclImdb/test/neg | tail -10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "When all we have anymore is pretty much reality TV shows with people making fools of themselves for whatever reason be it too fat or can't sing or cook worth a damn than I know Hollywood has run out of original ideas. I can not recall a time when anything original or intelligent came out on TV in the last 15 years. What is our obsession with watching bums make fools of themselves? I would have thought these types of programs would have run full circle but every year they come up with something new that is more strange then the one before. OK so people in this one need to lose weight...most Americans need to lose weight. I just think we all to some degree enjoy watching people humiliated. Maybe it makes us feel better when we see someone else looking like a jerk. I don't know but I just wish something intelligent would come out that did not insult your intelligence."
     ]
    }
   ],
   "source": [
    "!cat ../data/aclImdb/test/neg/9999_1.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Preparing the data\n",
    "\n",
    "The data we have downloaded is split into various files, each of which contains a single review. It will be much easier going forward if we combine these individual files into two large files, one for training and one for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "\n",
    "def read_imdb_data(data_dir='../data/aclImdb'):\n",
    "    data = {}\n",
    "    labels = {}\n",
    "    \n",
    "    for data_type in ['train', 'test']:\n",
    "        data[data_type] = {}\n",
    "        labels[data_type] = {}\n",
    "        \n",
    "        for sentiment in ['pos', 'neg']:\n",
    "            data[data_type][sentiment] = []\n",
    "            labels[data_type][sentiment] = []\n",
    "            \n",
    "            path = os.path.join(data_dir, data_type, sentiment, '*.txt')\n",
    "            files = glob.glob(path)\n",
    "            \n",
    "            for f in files:\n",
    "                with open(f) as review:\n",
    "                    data[data_type][sentiment].append(review.read())\n",
    "                    # Here we represent a positive review by '1' \n",
    "                    # and a negative review by '0'\n",
    "                    labels[data_type][sentiment].append( \\\n",
    "                        1 if sentiment == 'pos' else 0)\n",
    "                    \n",
    "            assert len(data[data_type][sentiment]) == len(labels[data_type][sentiment]), \\\n",
    "                    \"{}/{} data size does not match labels size\".format(data_type, \n",
    "                                                                        sentiment)\n",
    "                \n",
    "    return data, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IMDB reviews: train = 12500 pos / 12500 neg, test = 12500 pos / 12500 neg\n"
     ]
    }
   ],
   "source": [
    "data, labels = read_imdb_data()\n",
    "print(\"IMDB reviews: train = {} pos / {} neg, test = {} pos / {} neg\".format(\n",
    "            len(data['train']['pos']), len(data['train']['neg']),\n",
    "            len(data['test']['pos']), len(data['test']['neg'])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "def prepare_imdb_data(data, labels):\n",
    "    \"\"\"Prepare training and test sets from IMDb movie reviews.\"\"\"\n",
    "    \n",
    "    #Combine positive and negative reviews and labels\n",
    "    data_train = data['train']['pos'] + data['train']['neg']\n",
    "    data_test = data['test']['pos'] + data['test']['neg']\n",
    "    labels_train = labels['train']['pos'] + labels['train']['neg']\n",
    "    labels_test = labels['test']['pos'] + labels['test']['neg']\n",
    "    \n",
    "    #Shuffle reviews and corresponding labels within training and test sets\n",
    "    data_train, labels_train = shuffle(data_train, labels_train)\n",
    "    data_test, labels_test = shuffle(data_test, labels_test)\n",
    "    \n",
    "    # Return a unified training data, test data, training labels, test labets\n",
    "    return data_train, data_test, labels_train, labels_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IMDb reviews (combined): train = 25000, test = 25000\n"
     ]
    }
   ],
   "source": [
    "train_X, test_X, train_y, test_y = prepare_imdb_data(data, labels)\n",
    "print(\"IMDb reviews (combined): train = {}, test = {}\".format(len(train_X), \n",
    "                                                              len(test_X)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'this movie is ok if you like mindless action ,corny acting, and a very small plot !The special effects are decent considering this movie is from the director of\"Event Horizon\". The costumes are like something from a mad max movie .None of the soldiers talk ,so others tell them what to do.It eventually end up as a big shoot\\'em up movie with explosions all the place. I personally liked Russell better in \"tango and cash\"\"escape from la/new york\" and \"executive decision\". It must see this movie leave your mind at the door for a no brainer action science fiction movie!!'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_X[100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Processing the data\n",
    "\n",
    "Now that we have our training and testing datasets merged and ready to use, we need to start processing the raw data into something that will be useable by our machine learning algorithm. To begin with, we remove any html formatting that may appear in the reviews and perform some standard natural language processing in order to homogenize the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /home/ec2-user/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download(\"stopwords\")\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.porter import *\n",
    "stemmer = PorterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "def review_to_words(review):\n",
    "    text = BeautifulSoup(review, \"html.parser\").get_text() # Remove HTML tags\n",
    "    text = re.sub(r\"[^a-zA-Z0-9]\", \" \", text.lower()) # Convert to lower case\n",
    "    words = text.split() # Split string into words\n",
    "    # Remove stopwords\n",
    "    words = [w for w in words if w not in stopwords.words(\"english\")]\n",
    "    words = [PorterStemmer().stem(w) for w in words] # stem\n",
    "    \n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "# where to store cache files\n",
    "cache_dir = os.path.join(\"../cache\", \"sentiment_analysis\")\n",
    "# ensure cache directory exists\n",
    "os.makedirs(cache_dir, exist_ok=True)\n",
    "\n",
    "def preprocess_data(data_train, data_test, labels_train, labels_test,\n",
    "                    cache_dir=cache_dir, cache_file=\"preprocessed_data.pkl\"):\n",
    "    \"\"\"Convert each review to words; read from cache if available.\"\"\"\n",
    "\n",
    "    # If cache_file is not None, try to read from it first\n",
    "    cache_data = None\n",
    "    if cache_file is not None:\n",
    "        try:\n",
    "            with open(os.path.join(cache_dir, cache_file), \"rb\") as f:\n",
    "                cache_data = pickle.load(f)\n",
    "            print(\"Read preprocessed data from cache file:\", cache_file)\n",
    "        except:\n",
    "            pass  # unable to read from cache, but that's okay\n",
    "    \n",
    "    # If cache is missing, then do the heavy lifting\n",
    "    if cache_data is None:\n",
    "        # Preprocess training and test data to obtain words for each review\n",
    "        #words_train = list(map(review_to_words, data_train))\n",
    "        #words_test = list(map(review_to_words, data_test))\n",
    "        words_train = [review_to_words(review) for review in data_train]\n",
    "        words_test = [review_to_words(review) for review in data_test]\n",
    "        \n",
    "        # Write to cache file for future runs\n",
    "        if cache_file is not None:\n",
    "            cache_data = dict(words_train=words_train, words_test=words_test,\n",
    "                              labels_train=labels_train, labels_test=labels_test)\n",
    "            with open(os.path.join(cache_dir, cache_file), \"wb\") as f:\n",
    "                pickle.dump(cache_data, f)\n",
    "            print(\"Wrote preprocessed data to cache file:\", cache_file)\n",
    "    else:\n",
    "        # Unpack data loaded from cache file\n",
    "        words_train, words_test, labels_train, labels_test = \\\n",
    "            (cache_data['words_train'], cache_data['words_test'], \\\n",
    "             cache_data['labels_train'], cache_data['labels_test'])\n",
    "    \n",
    "    return words_train, words_test, labels_train, labels_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read preprocessed data from cache file: preprocessed_data.pkl\n"
     ]
    }
   ],
   "source": [
    "# Preprocess data\n",
    "train_X, test_X, train_y, test_y = preprocess_data(train_X, test_X, train_y, test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract Bag-of-Words features\n",
    "\n",
    "For the model we will be implementing, rather than using the reviews directly, we are going to transform each review into a Bag-of-Words feature representation. Keep in mind that 'in the wild' we will only have access to the training set so our transformer can only use the training set to construct a representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.externals import joblib\n",
    "# joblib is an enhanced version of pickle that is more efficient for storing NumPy arrays\n",
    "\n",
    "def extract_BoW_features(words_train, words_test, vocabulary_size=5000,\n",
    "                         cache_dir=cache_dir, cache_file=\"bow_features.pkl\"):\n",
    "    \"\"\"Extract Bag-of-Words for a given set of documents, already preprocessed into words.\"\"\"\n",
    "    \n",
    "    # If cache_file is not None, try to read from it first\n",
    "    cache_data = None\n",
    "    if cache_file is not None:\n",
    "        try:\n",
    "            with open(os.path.join(cache_dir, cache_file), \"rb\") as f:\n",
    "                cache_data = joblib.load(f)\n",
    "            print(\"Read features from cache file:\", cache_file)\n",
    "        except:\n",
    "            pass  # unable to read from cache, but that's okay\n",
    "    \n",
    "    # If cache is missing, then do the heavy lifting\n",
    "    if cache_data is None:\n",
    "        # Fit a vectorizer to training documents and use it to transform them\n",
    "        # NOTE: Training documents have already been preprocessed and tokenized into words;\n",
    "        #       pass in dummy functions to skip those steps, e.g. preprocessor=lambda x: x\n",
    "        vectorizer = CountVectorizer(max_features=vocabulary_size,\n",
    "                preprocessor=lambda x: x, tokenizer=lambda x: x)  # already preprocessed\n",
    "        features_train = vectorizer.fit_transform(words_train).toarray()\n",
    "\n",
    "        # Apply the same vectorizer to transform the test documents (ignore unknown words)\n",
    "        features_test = vectorizer.transform(words_test).toarray()\n",
    "        \n",
    "        # NOTE: Remember to convert the features using .toarray() for a compact representation\n",
    "        \n",
    "        # Write to cache file for future runs (store vocabulary as well)\n",
    "        if cache_file is not None:\n",
    "            vocabulary = vectorizer.vocabulary_\n",
    "            cache_data = dict(features_train=features_train, features_test=features_test,\n",
    "                             vocabulary=vocabulary)\n",
    "            with open(os.path.join(cache_dir, cache_file), \"wb\") as f:\n",
    "                joblib.dump(cache_data, f)\n",
    "            print(\"Wrote features to cache file:\", cache_file)\n",
    "    else:\n",
    "        # Unpack data loaded from cache file\n",
    "        features_train, features_test, vocabulary = (cache_data['features_train'],\n",
    "                cache_data['features_test'], cache_data['vocabulary'])\n",
    "    \n",
    "    # Return both the extracted features as well as the vocabulary\n",
    "    return features_train, features_test, vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read features from cache file: bow_features.pkl\n"
     ]
    }
   ],
   "source": [
    "# Extract Bag of Words features for both training and test datasets\n",
    "train_X, test_X, vocabulary = extract_BoW_features(train_X, test_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 5000)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 5000)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5000"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocabulary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1989"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocabulary['good']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "579"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocabulary['book']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 2041616\r\n",
      "-rw-rw-r-- 1 ec2-user ec2-user 2000232487 Apr  9 19:57 bow_features.pkl\r\n",
      "-rw-rw-r-- 1 ec2-user ec2-user   90371161 Apr  9 19:57 preprocessed_data.pkl\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l ../cache/sentiment_analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Classification using XGBoost\n",
    "\n",
    "Now that we have created the feature representation of our training (and testing) data, it is time to start setting up and using the XGBoost classifier provided by SageMaker.\n",
    "\n",
    "### (TODO) Writing the dataset\n",
    "\n",
    "The XGBoost classifier that we will be using requires the dataset to be written to a file and stored using Amazon S3. To do this, we will start by splitting the training dataset into two parts, the data we will train the model with and a validation set. Then, we will write those datasets to a file and upload the files to S3. In addition, we will write the test set input to a file and upload the file to S3. This is so that we can use SageMakers Batch Transform functionality to test our model once we've fit it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# TODO: Split the train_X and train_y arrays into the DataFrames \n",
    "#       val_X, train_X and val_y, train_y. Make sure that\n",
    "#       val_X and val_y contain 10 000 entires while train_X and \n",
    "#       train_y contain the remaining 15 000 entries.\n",
    "\n",
    "val_X = pd.DataFrame(train_X[:10000])\n",
    "train_X = pd.DataFrame(train_X[10000:])\n",
    "\n",
    "val_y = pd.DataFrame(train_y[:10000])\n",
    "train_y = pd.DataFrame(train_y[10000:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>4990</th>\n",
       "      <th>4991</th>\n",
       "      <th>4992</th>\n",
       "      <th>4993</th>\n",
       "      <th>4994</th>\n",
       "      <th>4995</th>\n",
       "      <th>4996</th>\n",
       "      <th>4997</th>\n",
       "      <th>4998</th>\n",
       "      <th>4999</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 5000 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0     1     2     3     4     5     6     7     8     9     ...   4990  \\\n",
       "0     0     0     0     0     0     0     0     0     0     0  ...      0   \n",
       "1     0     0     0     0     0     0     0     0     0     0  ...      0   \n",
       "2     0     0     0     0     0     0     0     0     0     0  ...      0   \n",
       "3     0     0     0     0     0     0     0     0     0     0  ...      0   \n",
       "4     0     0     0     0     0     0     0     0     0     0  ...      0   \n",
       "5     0     0     0     0     0     0     0     0     0     0  ...      0   \n",
       "6     0     0     0     0     0     0     0     0     0     0  ...      0   \n",
       "7     0     0     0     0     0     0     0     0     0     0  ...      0   \n",
       "8     0     0     0     0     0     0     0     0     0     0  ...      0   \n",
       "9     0     0     0     1     1     0     1     1     1     0  ...      0   \n",
       "\n",
       "   4991  4992  4993  4994  4995  4996  4997  4998  4999  \n",
       "0     0     0     0     0     0     0     0     0     0  \n",
       "1     0     0     0     0     0     0     0     0     0  \n",
       "2     0     0     0     0     0     0     0     0     0  \n",
       "3     0     0     0     0     0     0     0     0     0  \n",
       "4     0     0     0     0     0     0     0     0     0  \n",
       "5     0     0     0     0     0     0     0     0     0  \n",
       "6     0     0     0     0     0     0     0     0     0  \n",
       "7     0     0     0     0     0     0     0     0     0  \n",
       "8     0     2     0     0     0     0     0     0     0  \n",
       "9     0     0     0     0     0     0     1     0     0  \n",
       "\n",
       "[10 rows x 5000 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_X[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0\n",
       "0  1\n",
       "1  1\n",
       "2  1\n",
       "3  1\n",
       "4  0\n",
       "5  0\n",
       "6  0\n",
       "7  1\n",
       "8  1\n",
       "9  0"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_y[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15000"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(val_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25000"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(test_X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The documentation for the XGBoost algorithm in SageMaker requires that the saved datasets should contain no headers or index and that for the training and validation data, the label should occur first for each sample.\n",
    "\n",
    "For more information about this and other algorithms, the SageMaker developer documentation can be found on __[Amazon's website.](https://docs.aws.amazon.com/sagemaker/latest/dg/)__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we make sure that the local directory in which we'd like to store \n",
    "# the training and validation csv files exists.\n",
    "data_dir = '../data/xgboost'\n",
    "if not os.path.exists(data_dir):\n",
    "    os.makedirs(data_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, save the test data to test.csv in the data_dir directory. \n",
    "# Note that we do not save the associated ground truth\n",
    "# labels, instead we will use them later to compare with our model output.\n",
    "\n",
    "pd.DataFrame(test_X).to_csv(os.path.join(data_dir, 'test.csv'), \n",
    "                            header=False, index=False)\n",
    "\n",
    "# TODO: Save the training and validation data to train.csv and \n",
    "#       validation.csv in the data_dir directory.\n",
    "#       Make sure that the files you create are in the correct format.\n",
    "pd.concat([val_y, val_X], axis=1).to_csv(os.path.join(data_dir, 'validation.csv'), \n",
    "                                         header=False, index=False)\n",
    "pd.concat([train_y, train_X], axis=1).to_csv(os.path.join(data_dir, 'train.csv'), \n",
    "                                             header=False, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 488340\r\n",
      "-rw-rw-r-- 1 ec2-user ec2-user 250002134 Apr  9 20:10 test.csv\r\n",
      "-rw-rw-r-- 1 ec2-user ec2-user 150031465 Apr  9 20:13 train.csv\r\n",
      "-rw-rw-r-- 1 ec2-user ec2-user 100020988 Apr  9 20:11 validation.csv\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l ../data/xgboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To save a bit of memory we can set \n",
    "# text_X, train_X, val_X, train_y and val_y to None.\n",
    "\n",
    "test_X = train_X = val_X = train_y = val_y = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (TODO) Uploading Training / Validation files to S3\n",
    "\n",
    "Amazon's S3 service allows us to store files that can be access by both the built-in training models such as the XGBoost model we will be using as well as custom models such as the one we will see a little later.\n",
    "\n",
    "For this, and most other tasks we will be doing using SageMaker, there are two methods we could use. The first is to use the low level functionality of SageMaker which requires knowing each of the objects involved in the SageMaker environment. The second is to use the high level functionality in which certain choices have been made on the user's behalf. The low level approach benefits from allowing the user a great deal of flexibility while the high level approach makes development much quicker. For our purposes we will opt to use the high level approach although using the low-level approach is certainly an option.\n",
    "\n",
    "Recall the method `upload_data()` which is a member of object representing our current SageMaker session. What this method does is upload the data to the default bucket (which is created if it does not exist) into the path described by the key_prefix variable. To see this for yourself, once you have uploaded the data files, go to the S3 console and look to see where the files have been uploaded.\n",
    "\n",
    "For additional resources, see the __[SageMaker API documentation](http://sagemaker.readthedocs.io/en/latest/)__ and in addition the __[SageMaker Developer Guide.](https://docs.aws.amazon.com/sagemaker/latest/dg/)__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Created S3 bucket: sagemaker-us-east-1-112107346316\n"
     ]
    }
   ],
   "source": [
    "import sagemaker\n",
    "\n",
    "session = sagemaker.Session() # Store the current SageMaker session\n",
    "\n",
    "# S3 prefix (which folder will we use)\n",
    "prefix = 'sentiment-xgboost'\n",
    "\n",
    "# TODO: Upload the test.csv, train.csv and validation.csv files which \n",
    "#       are contained in data_dir to S3 using sess.upload_data().\n",
    "test_location = session.upload_data(os.path.join(data_dir, 'test.csv'), \n",
    "                                    key_prefix=prefix)\n",
    "val_location = session.upload_data(os.path.join(data_dir, 'validation.csv'), \n",
    "                                   key_prefix=prefix)\n",
    "train_location = session.upload_data(os.path.join(data_dir, 'train.csv'), \n",
    "                                     key_prefix=prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (TODO) Creating the XGBoost model\n",
    "\n",
    "Now that the data has been uploaded it is time to create the XGBoost model. To begin with, we need to do some setup. At this point it is worth discussing what a model is in SageMaker. It is easiest to think of a model of comprising three different objects in the SageMaker ecosystem, which interact with one another.\n",
    "\n",
    "- Model Artifacts\n",
    "- Training Code (Container)\n",
    "- Inference Code (Container)\n",
    "\n",
    "The Model Artifacts are what you might think of as the actual model itself. For example, if you were building a neural network, the model artifacts would be the weights of the various layers. In our case, for an XGBoost model, the artifacts are the actual trees that are created during training.\n",
    "\n",
    "The other two objects, the training code and the inference code are then used the manipulate the training artifacts. More precisely, the training code uses the training data that is provided and creates the model artifacts, while the inference code uses the model artifacts to make predictions on new data.\n",
    "\n",
    "The way that SageMaker runs the training and inference code is by making use of Docker containers. For now, think of a container as being a way of packaging code up so that dependencies aren't an issue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import get_execution_role\n",
    "\n",
    "# Our current execution role is require when creating the model as the training\n",
    "# and inference code will need to access the model artifacts.\n",
    "role = get_execution_role()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need to retrieve the location of the container  \n",
    "# which is provided by Amazon for using XGBoost.\n",
    "# As a matter of convenience, the training and \n",
    "# inference code both use the same container.\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "container = get_image_uri(session.boto_region_name, 'xgboost')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Create a SageMaker estimator using the container \n",
    "#       location determined in the previous cell.\n",
    "#       It is recommended that you use a single training \n",
    "#       instance of type ml.m4.xlarge. It is also\n",
    "#       recommended that you use \n",
    "#       's3://{}/{}/output'.format(session.default_bucket(), prefix) \n",
    "#       as the output path.\n",
    "\n",
    "xgb = sagemaker.estimator.Estimator(\n",
    "    container, # The location of the container we wish to use\n",
    "    role,                                    # What is our current IAM Role\n",
    "    train_instance_count=1,                  # How many compute instances\n",
    "    train_instance_type='ml.m4.xlarge',      # What kind of compute instances\n",
    "    output_path='s3://{}/{}/output'.format(session.default_bucket(), prefix),\n",
    "    sagemaker_session=session)\n",
    "\n",
    "\n",
    "# TODO: Set the XGBoost hyperparameters in the xgb object. \n",
    "#       Don't forget that in this case we have a binary\n",
    "#       label so we should be using the 'binary:logistic' objective.\n",
    "\n",
    "xgb.set_hyperparameters(max_depth=5,\n",
    "                        eta=0.2,\n",
    "                        gamma=4,\n",
    "                        min_child_weight=6,\n",
    "                        subsample=0.8,\n",
    "                        silent=0,\n",
    "                        objective='binary:logistic',\n",
    "                        early_stopping_rounds=10,\n",
    "                        num_round=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fit the XGBoost model\n",
    "\n",
    "Now that our model has been set up we simply need to attach the training and validation datasets and then ask SageMaker to set up the computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_input_train = sagemaker.s3_input(s3_data=train_location, content_type='csv')\n",
    "s3_input_validation = sagemaker.s3_input(s3_data=val_location, content_type='csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating training-job with name: xgboost-2019-04-09-20-20-11-160\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-04-09 20:20:11 Starting - Starting the training job...\n",
      "2019-04-09 20:20:13 Starting - Launching requested ML instances.........\n",
      "2019-04-09 20:21:49 Starting - Preparing the instances for training......\n",
      "2019-04-09 20:23:04 Downloading - Downloading input data\n",
      "2019-04-09 20:23:04 Training - Downloading the training image...\n",
      "2019-04-09 20:23:27 Training - Training image download completed. Training in progress.\n",
      "\u001b[31mArguments: train\u001b[0m\n",
      "\u001b[31m[2019-04-09:20:23:26:INFO] Running standalone xgboost training.\u001b[0m\n",
      "\u001b[31m[2019-04-09:20:23:26:INFO] File size need to be processed in the node: 238.47mb. Available memory size in the node: 8394.54mb\u001b[0m\n",
      "\u001b[31m[2019-04-09:20:23:26:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[31m[20:23:26] S3DistributionType set as FullyReplicated\u001b[0m\n",
      "\u001b[31m[20:23:28] 15000x5000 matrix with 75000000 entries loaded from /opt/ml/input/data/train?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[31m[2019-04-09:20:23:28:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[31m[20:23:28] S3DistributionType set as FullyReplicated\u001b[0m\n",
      "\u001b[31m[20:23:29] 10000x5000 matrix with 50000000 entries loaded from /opt/ml/input/data/validation?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[31m[20:23:33] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 36 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[0]#011train-error:0.299533#011validation-error:0.2949\u001b[0m\n",
      "\u001b[31mMultiple eval metrics have been passed: 'validation-error' will be used for early stopping.\n",
      "\u001b[0m\n",
      "\u001b[31mWill train until validation-error hasn't improved in 10 rounds.\u001b[0m\n",
      "\u001b[31m[20:23:34] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 46 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[1]#011train-error:0.274133#011validation-error:0.2712\u001b[0m\n",
      "\u001b[31m[20:23:36] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 44 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[2]#011train-error:0.264133#011validation-error:0.2661\u001b[0m\n",
      "\u001b[31m[20:23:37] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[3]#011train-error:0.262067#011validation-error:0.2645\u001b[0m\n",
      "\u001b[31m[20:23:38] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[4]#011train-error:0.259467#011validation-error:0.2627\u001b[0m\n",
      "\u001b[31m[20:23:40] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 44 extra nodes, 0 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[5]#011train-error:0.245133#011validation-error:0.2532\u001b[0m\n",
      "\u001b[31m[20:23:41] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[6]#011train-error:0.2418#011validation-error:0.2481\u001b[0m\n",
      "\u001b[31m[20:23:42] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[7]#011train-error:0.233733#011validation-error:0.242\u001b[0m\n",
      "\u001b[31m[20:23:44] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[8]#011train-error:0.229333#011validation-error:0.2398\u001b[0m\n",
      "\u001b[31m[20:23:45] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[9]#011train-error:0.2226#011validation-error:0.2337\u001b[0m\n",
      "\u001b[31m[20:23:46] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 40 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[10]#011train-error:0.218267#011validation-error:0.2301\u001b[0m\n",
      "\u001b[31m[20:23:47] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[11]#011train-error:0.214067#011validation-error:0.2252\u001b[0m\n",
      "\u001b[31m[20:23:49] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[12]#011train-error:0.208#011validation-error:0.2217\u001b[0m\n",
      "\u001b[31m[20:23:50] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[13]#011train-error:0.201533#011validation-error:0.2167\u001b[0m\n",
      "\u001b[31m[20:23:51] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[14]#011train-error:0.197867#011validation-error:0.2119\u001b[0m\n",
      "\u001b[31m[20:23:53] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 42 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[15]#011train-error:0.192067#011validation-error:0.2102\u001b[0m\n",
      "\u001b[31m[20:23:54] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[16]#011train-error:0.191267#011validation-error:0.2069\u001b[0m\n",
      "\u001b[31m[20:23:55] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[17]#011train-error:0.188667#011validation-error:0.2045\u001b[0m\n",
      "\u001b[31m[20:23:57] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[18]#011train-error:0.187267#011validation-error:0.2044\u001b[0m\n",
      "\u001b[31m[20:23:58] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[19]#011train-error:0.1846#011validation-error:0.2051\u001b[0m\n",
      "\u001b[31m[20:23:59] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[20]#011train-error:0.1826#011validation-error:0.2038\u001b[0m\n",
      "\u001b[31m[20:24:01] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[21]#011train-error:0.177667#011validation-error:0.2014\u001b[0m\n",
      "\u001b[31m[20:24:02] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[22]#011train-error:0.175467#011validation-error:0.1989\u001b[0m\n",
      "\u001b[31m[20:24:03] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[23]#011train-error:0.173333#011validation-error:0.1975\u001b[0m\n",
      "\u001b[31m[20:24:05] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[24]#011train-error:0.1726#011validation-error:0.1963\u001b[0m\n",
      "\u001b[31m[20:24:06] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 38 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[25]#011train-error:0.169733#011validation-error:0.1939\u001b[0m\n",
      "\u001b[31m[20:24:07] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[26]#011train-error:0.166933#011validation-error:0.192\u001b[0m\n",
      "\u001b[31m[20:24:08] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[27]#011train-error:0.1652#011validation-error:0.1895\u001b[0m\n",
      "\u001b[31m[20:24:10] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[28]#011train-error:0.163933#011validation-error:0.1902\u001b[0m\n",
      "\u001b[31m[20:24:11] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[29]#011train-error:0.160267#011validation-error:0.1891\u001b[0m\n",
      "\u001b[31m[20:24:12] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[30]#011train-error:0.157933#011validation-error:0.1882\u001b[0m\n",
      "\u001b[31m[20:24:14] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[31]#011train-error:0.156667#011validation-error:0.1886\u001b[0m\n",
      "\u001b[31m[20:24:15] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[32]#011train-error:0.156933#011validation-error:0.1874\u001b[0m\n",
      "\u001b[31m[20:24:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[33]#011train-error:0.154#011validation-error:0.185\u001b[0m\n",
      "\u001b[31m[20:24:18] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[34]#011train-error:0.153#011validation-error:0.1845\u001b[0m\n",
      "\u001b[31m[20:24:19] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[35]#011train-error:0.151867#011validation-error:0.1818\u001b[0m\n",
      "\u001b[31m[20:24:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[36]#011train-error:0.1506#011validation-error:0.1811\u001b[0m\n",
      "\u001b[31m[20:24:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[37]#011train-error:0.149067#011validation-error:0.1807\u001b[0m\n",
      "\u001b[31m[20:24:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[38]#011train-error:0.148533#011validation-error:0.1809\u001b[0m\n",
      "\u001b[31m[20:24:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[39]#011train-error:0.147933#011validation-error:0.1783\u001b[0m\n",
      "\u001b[31m[20:24:25] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[40]#011train-error:0.146333#011validation-error:0.1787\u001b[0m\n",
      "\u001b[31m[20:24:27] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[41]#011train-error:0.143667#011validation-error:0.177\u001b[0m\n",
      "\u001b[31m[20:24:28] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[42]#011train-error:0.1416#011validation-error:0.175\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[20:24:29] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[43]#011train-error:0.1414#011validation-error:0.1739\u001b[0m\n",
      "\u001b[31m[20:24:31] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[44]#011train-error:0.1414#011validation-error:0.1749\u001b[0m\n",
      "\u001b[31m[20:24:32] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[45]#011train-error:0.1408#011validation-error:0.1744\u001b[0m\n",
      "\u001b[31m[20:24:33] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[46]#011train-error:0.1402#011validation-error:0.1734\u001b[0m\n",
      "\u001b[31m[20:24:35] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[47]#011train-error:0.1402#011validation-error:0.1735\u001b[0m\n",
      "\u001b[31m[20:24:36] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[48]#011train-error:0.139#011validation-error:0.1725\u001b[0m\n",
      "\u001b[31m[20:24:37] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[49]#011train-error:0.138467#011validation-error:0.1716\u001b[0m\n",
      "\u001b[31m[20:24:39] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[50]#011train-error:0.1362#011validation-error:0.1705\u001b[0m\n",
      "\u001b[31m[20:24:40] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[51]#011train-error:0.1346#011validation-error:0.1704\u001b[0m\n",
      "\u001b[31m[20:24:41] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[52]#011train-error:0.133533#011validation-error:0.169\u001b[0m\n",
      "\u001b[31m[20:24:42] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[53]#011train-error:0.1322#011validation-error:0.1684\u001b[0m\n",
      "\u001b[31m[20:24:44] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[54]#011train-error:0.130867#011validation-error:0.1678\u001b[0m\n",
      "\u001b[31m[20:24:45] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[55]#011train-error:0.130733#011validation-error:0.1685\u001b[0m\n",
      "\u001b[31m[20:24:46] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[56]#011train-error:0.131267#011validation-error:0.1665\u001b[0m\n",
      "\u001b[31m[20:24:48] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[57]#011train-error:0.1298#011validation-error:0.1673\u001b[0m\n",
      "\u001b[31m[20:24:49] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[58]#011train-error:0.128067#011validation-error:0.167\u001b[0m\n",
      "\u001b[31m[20:24:50] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[59]#011train-error:0.1268#011validation-error:0.1668\u001b[0m\n",
      "\u001b[31m[20:24:52] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[60]#011train-error:0.1254#011validation-error:0.1659\u001b[0m\n",
      "\u001b[31m[20:24:53] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[61]#011train-error:0.1252#011validation-error:0.1652\u001b[0m\n",
      "\u001b[31m[20:24:54] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[62]#011train-error:0.124067#011validation-error:0.1641\u001b[0m\n",
      "\u001b[31m[20:24:56] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[63]#011train-error:0.122867#011validation-error:0.1642\u001b[0m\n",
      "\u001b[31m[20:24:57] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[64]#011train-error:0.122467#011validation-error:0.1639\u001b[0m\n",
      "\u001b[31m[20:24:58] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[65]#011train-error:0.122#011validation-error:0.1637\u001b[0m\n",
      "\u001b[31m[20:25:00] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[66]#011train-error:0.1216#011validation-error:0.163\u001b[0m\n",
      "\u001b[31m[20:25:01] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[67]#011train-error:0.121067#011validation-error:0.1625\u001b[0m\n",
      "\u001b[31m[20:25:02] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[68]#011train-error:0.1208#011validation-error:0.1622\u001b[0m\n",
      "\u001b[31m[20:25:03] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[69]#011train-error:0.12#011validation-error:0.1613\u001b[0m\n",
      "\u001b[31m[20:25:05] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[70]#011train-error:0.119067#011validation-error:0.1603\u001b[0m\n",
      "\u001b[31m[20:25:06] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 0 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[71]#011train-error:0.118467#011validation-error:0.1603\u001b[0m\n",
      "\u001b[31m[20:25:07] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[72]#011train-error:0.1182#011validation-error:0.1614\u001b[0m\n",
      "\u001b[31m[20:25:09] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[73]#011train-error:0.117133#011validation-error:0.1602\u001b[0m\n",
      "\u001b[31m[20:25:10] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[74]#011train-error:0.1158#011validation-error:0.1594\u001b[0m\n",
      "\u001b[31m[20:25:11] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[75]#011train-error:0.1152#011validation-error:0.1599\u001b[0m\n",
      "\u001b[31m[20:25:13] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[76]#011train-error:0.1148#011validation-error:0.159\u001b[0m\n",
      "\u001b[31m[20:25:14] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[77]#011train-error:0.114333#011validation-error:0.1588\u001b[0m\n",
      "\u001b[31m[20:25:15] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[78]#011train-error:0.114267#011validation-error:0.1588\u001b[0m\n",
      "\u001b[31m[20:25:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[79]#011train-error:0.113733#011validation-error:0.1587\u001b[0m\n",
      "\u001b[31m[20:25:18] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[80]#011train-error:0.1126#011validation-error:0.1587\u001b[0m\n",
      "\u001b[31m[20:25:19] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[81]#011train-error:0.111667#011validation-error:0.1576\u001b[0m\n",
      "\u001b[31m[20:25:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[82]#011train-error:0.110933#011validation-error:0.1579\u001b[0m\n",
      "\u001b[31m[20:25:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[83]#011train-error:0.110933#011validation-error:0.1576\u001b[0m\n",
      "\u001b[31m[20:25:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[84]#011train-error:0.110467#011validation-error:0.1574\u001b[0m\n",
      "\u001b[31m[20:25:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[85]#011train-error:0.110667#011validation-error:0.1567\u001b[0m\n",
      "\u001b[31m[20:25:26] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[86]#011train-error:0.111067#011validation-error:0.1564\u001b[0m\n",
      "\u001b[31m[20:25:27] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[87]#011train-error:0.110067#011validation-error:0.1552\u001b[0m\n",
      "\u001b[31m[20:25:28] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[88]#011train-error:0.110067#011validation-error:0.155\u001b[0m\n",
      "\u001b[31m[20:25:30] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[89]#011train-error:0.108533#011validation-error:0.1536\u001b[0m\n",
      "\u001b[31m[20:25:31] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[90]#011train-error:0.108#011validation-error:0.1535\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[20:25:32] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[91]#011train-error:0.107067#011validation-error:0.1543\u001b[0m\n",
      "\u001b[31m[20:25:33] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[92]#011train-error:0.106733#011validation-error:0.1538\u001b[0m\n",
      "\u001b[31m[20:25:35] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[93]#011train-error:0.1072#011validation-error:0.1527\u001b[0m\n",
      "\u001b[31m[20:25:36] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[94]#011train-error:0.106133#011validation-error:0.1535\u001b[0m\n",
      "\u001b[31m[20:25:37] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[95]#011train-error:0.105933#011validation-error:0.1525\u001b[0m\n",
      "\u001b[31m[20:25:39] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[96]#011train-error:0.106067#011validation-error:0.1516\u001b[0m\n",
      "\u001b[31m[20:25:40] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[97]#011train-error:0.105267#011validation-error:0.1515\u001b[0m\n",
      "\u001b[31m[20:25:41] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[98]#011train-error:0.104867#011validation-error:0.1523\u001b[0m\n",
      "\u001b[31m[20:25:43] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[99]#011train-error:0.104733#011validation-error:0.1519\u001b[0m\n",
      "\u001b[31m[20:25:44] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[100]#011train-error:0.1044#011validation-error:0.1522\u001b[0m\n",
      "\u001b[31m[20:25:45] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[101]#011train-error:0.1032#011validation-error:0.1517\u001b[0m\n",
      "\u001b[31m[20:25:47] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[102]#011train-error:0.1034#011validation-error:0.1514\u001b[0m\n",
      "\u001b[31m[20:25:48] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[103]#011train-error:0.1032#011validation-error:0.1515\u001b[0m\n",
      "\u001b[31m[20:25:49] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[104]#011train-error:0.101933#011validation-error:0.1509\u001b[0m\n",
      "\u001b[31m[20:25:50] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[105]#011train-error:0.101533#011validation-error:0.1508\u001b[0m\n",
      "\u001b[31m[20:25:52] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[106]#011train-error:0.1018#011validation-error:0.1504\u001b[0m\n",
      "\u001b[31m[20:25:53] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[107]#011train-error:0.101733#011validation-error:0.1495\u001b[0m\n",
      "\u001b[31m[20:25:54] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[108]#011train-error:0.100933#011validation-error:0.1502\u001b[0m\n",
      "\u001b[31m[20:25:56] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[109]#011train-error:0.101333#011validation-error:0.1502\u001b[0m\n",
      "\u001b[31m[20:25:57] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[110]#011train-error:0.100533#011validation-error:0.1504\u001b[0m\n",
      "\u001b[31m[20:25:58] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 22 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[111]#011train-error:0.100067#011validation-error:0.1506\u001b[0m\n",
      "\u001b[31m[20:26:00] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[112]#011train-error:0.099533#011validation-error:0.151\u001b[0m\n",
      "\u001b[31m[20:26:01] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[113]#011train-error:0.098#011validation-error:0.1504\u001b[0m\n",
      "\u001b[31m[20:26:02] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[114]#011train-error:0.098067#011validation-error:0.1502\u001b[0m\n",
      "\u001b[31m[20:26:04] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[115]#011train-error:0.097933#011validation-error:0.1505\u001b[0m\n",
      "\u001b[31m[20:26:05] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[116]#011train-error:0.096867#011validation-error:0.1494\u001b[0m\n",
      "\u001b[31m[20:26:06] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[117]#011train-error:0.096333#011validation-error:0.149\u001b[0m\n",
      "\u001b[31m[20:26:07] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[118]#011train-error:0.096067#011validation-error:0.1494\u001b[0m\n",
      "\u001b[31m[20:26:09] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[119]#011train-error:0.0964#011validation-error:0.1491\u001b[0m\n",
      "\u001b[31m[20:26:10] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[120]#011train-error:0.0964#011validation-error:0.1482\u001b[0m\n",
      "\u001b[31m[20:26:11] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[121]#011train-error:0.0958#011validation-error:0.1473\u001b[0m\n",
      "\u001b[31m[20:26:13] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[122]#011train-error:0.0956#011validation-error:0.1473\u001b[0m\n",
      "\u001b[31m[20:26:14] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[123]#011train-error:0.095333#011validation-error:0.1464\u001b[0m\n",
      "\u001b[31m[20:26:15] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 0 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[124]#011train-error:0.095467#011validation-error:0.1469\u001b[0m\n",
      "\u001b[31m[20:26:17] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[125]#011train-error:0.0948#011validation-error:0.147\u001b[0m\n",
      "\u001b[31m[20:26:18] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[126]#011train-error:0.094267#011validation-error:0.1472\u001b[0m\n",
      "\u001b[31m[20:26:19] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[127]#011train-error:0.0936#011validation-error:0.147\u001b[0m\n",
      "\u001b[31m[20:26:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[128]#011train-error:0.093733#011validation-error:0.147\u001b[0m\n",
      "\u001b[31m[20:26:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[129]#011train-error:0.093133#011validation-error:0.1466\u001b[0m\n",
      "\u001b[31m[20:26:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[130]#011train-error:0.0932#011validation-error:0.1462\u001b[0m\n",
      "\u001b[31m[20:26:25] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[131]#011train-error:0.0926#011validation-error:0.1457\u001b[0m\n",
      "\u001b[31m[20:26:26] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[132]#011train-error:0.091867#011validation-error:0.1456\u001b[0m\n",
      "\u001b[31m[20:26:27] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[133]#011train-error:0.092067#011validation-error:0.1457\u001b[0m\n",
      "\u001b[31m[20:26:28] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[134]#011train-error:0.0916#011validation-error:0.1456\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[20:26:30] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[135]#011train-error:0.0916#011validation-error:0.1455\u001b[0m\n",
      "\u001b[31m[20:26:31] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[136]#011train-error:0.091#011validation-error:0.1451\u001b[0m\n",
      "\u001b[31m[20:26:32] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[137]#011train-error:0.090667#011validation-error:0.145\u001b[0m\n",
      "\u001b[31m[20:26:34] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[138]#011train-error:0.0906#011validation-error:0.1451\u001b[0m\n",
      "\u001b[31m[20:26:35] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[139]#011train-error:0.090267#011validation-error:0.1448\u001b[0m\n",
      "\u001b[31m[20:26:36] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[140]#011train-error:0.089933#011validation-error:0.1445\u001b[0m\n",
      "\u001b[31m[20:26:38] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[141]#011train-error:0.09#011validation-error:0.1438\u001b[0m\n",
      "\u001b[31m[20:26:39] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[142]#011train-error:0.089533#011validation-error:0.1437\u001b[0m\n",
      "\u001b[31m[20:26:40] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[143]#011train-error:0.0894#011validation-error:0.1441\u001b[0m\n",
      "\u001b[31m[20:26:42] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[144]#011train-error:0.089467#011validation-error:0.1446\u001b[0m\n",
      "\u001b[31m[20:26:43] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[145]#011train-error:0.089667#011validation-error:0.144\u001b[0m\n",
      "\u001b[31m[20:26:44] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[146]#011train-error:0.089067#011validation-error:0.1439\u001b[0m\n",
      "\u001b[31m[20:26:45] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[147]#011train-error:0.0886#011validation-error:0.1441\u001b[0m\n",
      "\u001b[31m[20:26:47] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[148]#011train-error:0.088533#011validation-error:0.1429\u001b[0m\n",
      "\u001b[31m[20:26:48] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[149]#011train-error:0.0882#011validation-error:0.1427\u001b[0m\n",
      "\u001b[31m[20:26:49] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[150]#011train-error:0.088133#011validation-error:0.1429\u001b[0m\n",
      "\u001b[31m[20:26:51] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[151]#011train-error:0.087133#011validation-error:0.1421\u001b[0m\n",
      "\u001b[31m[20:26:52] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[152]#011train-error:0.086667#011validation-error:0.1419\u001b[0m\n",
      "\u001b[31m[20:26:53] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[153]#011train-error:0.0862#011validation-error:0.1414\u001b[0m\n",
      "\u001b[31m[20:26:55] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[154]#011train-error:0.085867#011validation-error:0.1422\u001b[0m\n",
      "\u001b[31m[20:26:56] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[155]#011train-error:0.0852#011validation-error:0.1421\u001b[0m\n",
      "\u001b[31m[20:26:57] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 0 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[156]#011train-error:0.084733#011validation-error:0.1419\u001b[0m\n",
      "\u001b[31m[20:26:59] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[157]#011train-error:0.084467#011validation-error:0.1418\u001b[0m\n",
      "\u001b[31m[20:27:00] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[158]#011train-error:0.0842#011validation-error:0.1413\u001b[0m\n",
      "\u001b[31m[20:27:01] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[159]#011train-error:0.084133#011validation-error:0.1415\u001b[0m\n",
      "\u001b[31m[20:27:03] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[160]#011train-error:0.0842#011validation-error:0.1417\u001b[0m\n",
      "\u001b[31m[20:27:04] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[161]#011train-error:0.083933#011validation-error:0.1418\u001b[0m\n",
      "\u001b[31m[20:27:05] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[162]#011train-error:0.0838#011validation-error:0.1418\u001b[0m\n",
      "\u001b[31m[20:27:06] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[163]#011train-error:0.083333#011validation-error:0.1412\u001b[0m\n",
      "\u001b[31m[20:27:08] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[164]#011train-error:0.083533#011validation-error:0.142\u001b[0m\n",
      "\u001b[31m[20:27:09] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[165]#011train-error:0.083267#011validation-error:0.1413\u001b[0m\n",
      "\u001b[31m[20:27:10] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[166]#011train-error:0.083867#011validation-error:0.1415\u001b[0m\n",
      "\u001b[31m[20:27:12] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[167]#011train-error:0.083333#011validation-error:0.1414\u001b[0m\n",
      "\u001b[31m[20:27:13] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[168]#011train-error:0.082867#011validation-error:0.1412\u001b[0m\n",
      "\u001b[31m[20:27:14] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[169]#011train-error:0.082667#011validation-error:0.1409\u001b[0m\n",
      "\u001b[31m[20:27:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[170]#011train-error:0.082733#011validation-error:0.1407\u001b[0m\n",
      "\u001b[31m[20:27:17] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[171]#011train-error:0.082733#011validation-error:0.1405\u001b[0m\n",
      "\u001b[31m[20:27:18] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[172]#011train-error:0.082867#011validation-error:0.1409\u001b[0m\n",
      "\u001b[31m[20:27:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[173]#011train-error:0.082667#011validation-error:0.1405\u001b[0m\n",
      "\u001b[31m[20:27:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[174]#011train-error:0.082667#011validation-error:0.1404\u001b[0m\n",
      "\u001b[31m[20:27:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[175]#011train-error:0.082267#011validation-error:0.1402\u001b[0m\n",
      "\u001b[31m[20:27:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[176]#011train-error:0.082#011validation-error:0.1399\u001b[0m\n",
      "\u001b[31m[20:27:25] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[177]#011train-error:0.082133#011validation-error:0.14\u001b[0m\n",
      "\u001b[31m[20:27:26] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[178]#011train-error:0.080867#011validation-error:0.1393\u001b[0m\n",
      "\u001b[31m[20:27:27] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[179]#011train-error:0.080867#011validation-error:0.1401\u001b[0m\n",
      "\u001b[31m[20:27:29] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[180]#011train-error:0.0808#011validation-error:0.1399\u001b[0m\n",
      "\u001b[31m[20:27:30] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[181]#011train-error:0.080467#011validation-error:0.1401\u001b[0m\n",
      "\u001b[31m[20:27:31] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[182]#011train-error:0.0796#011validation-error:0.1395\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[20:27:33] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[183]#011train-error:0.078867#011validation-error:0.1394\u001b[0m\n",
      "\u001b[31m[20:27:34] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[184]#011train-error:0.079067#011validation-error:0.139\u001b[0m\n",
      "\u001b[31m[20:27:35] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[185]#011train-error:0.078333#011validation-error:0.1388\u001b[0m\n",
      "\u001b[31m[20:27:37] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[186]#011train-error:0.078#011validation-error:0.1394\u001b[0m\n",
      "\u001b[31m[20:27:38] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[187]#011train-error:0.0776#011validation-error:0.1387\u001b[0m\n",
      "\u001b[31m[20:27:39] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[188]#011train-error:0.078067#011validation-error:0.1379\u001b[0m\n",
      "\u001b[31m[20:27:40] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[189]#011train-error:0.0776#011validation-error:0.1382\u001b[0m\n",
      "\u001b[31m[20:27:42] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[190]#011train-error:0.0772#011validation-error:0.1385\u001b[0m\n",
      "\u001b[31m[20:27:43] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[191]#011train-error:0.077333#011validation-error:0.1383\u001b[0m\n",
      "\u001b[31m[20:27:44] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[192]#011train-error:0.077467#011validation-error:0.1386\u001b[0m\n",
      "\u001b[31m[20:27:46] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[193]#011train-error:0.0768#011validation-error:0.138\u001b[0m\n",
      "\u001b[31m[20:27:47] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[194]#011train-error:0.0764#011validation-error:0.1383\u001b[0m\n",
      "\u001b[31m[20:27:48] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[195]#011train-error:0.075867#011validation-error:0.1384\u001b[0m\n",
      "\u001b[31m[20:27:50] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[196]#011train-error:0.075667#011validation-error:0.1382\u001b[0m\n",
      "\u001b[31m[20:27:51] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[197]#011train-error:0.075333#011validation-error:0.1385\u001b[0m\n",
      "\u001b[31m[20:27:52] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[198]#011train-error:0.074933#011validation-error:0.1381\u001b[0m\n",
      "\u001b[31mStopping. Best iteration:\u001b[0m\n",
      "\u001b[31m[188]#011train-error:0.078067#011validation-error:0.1379\n",
      "\u001b[0m\n",
      "\n",
      "2019-04-09 20:28:38 Uploading - Uploading generated training model\n",
      "2019-04-09 20:28:38 Completed - Training job completed\n",
      "Billable seconds: 347\n"
     ]
    }
   ],
   "source": [
    "xgb.fit({'train': s3_input_train, 'validation': s3_input_validation})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (TODO) Testing the model\n",
    "\n",
    "Now that we've fit our XGBoost model, it's time to see how well it performs. To do this we will use SageMakers Batch Transform functionality. Batch Transform is a convenient way to perform inference on a large dataset in a way that is not realtime. That is, we don't necessarily need to use our model's results immediately and instead we can peform inference on a large number of samples. An example of this in industry might be peforming an end of month report. This method of inference can also be useful to us as it means to can perform inference on our entire test set. \n",
    "\n",
    "To perform a Batch Transformation we need to first create a transformer objects from our trained estimator object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model with name: xgboost-2019-04-09-20-20-11-160\n"
     ]
    }
   ],
   "source": [
    "# TODO: Create a transformer object from the trained model. Using an \n",
    "#       instance count of 1 and an instance type of ml.m4.xlarge\n",
    "#       should be more than enough.\n",
    "xgb_transformer = xgb.transformer(instance_count = 1, \n",
    "                                  instance_type = 'ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we actually perform the transform job. When doing so we need to make sure to specify the type of data we are sending so that it is serialized correctly in the background. In our case we are providing our model with csv data so we specify `text/csv`. Also, if the test data that we have provided is too large to process all at once then we need to specify how the data file should be split up. Since each line is a single entry in our data set we tell SageMaker that it can split the input on each line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating transform job with name: xgboost-2019-04-09-20-30-05-405\n"
     ]
    }
   ],
   "source": [
    "# TODO: Start the transform job. Make sure to specify the \n",
    "#       content type and the split type of the test data.\n",
    "xgb_transformer.transform(test_location, content_type='text/csv', \n",
    "                          split_type='Line')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently the transform job is running but it is doing so in the background. Since we wish to wait until the transform job is done and we would like a bit of feedback we can run the `wait()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "....................................................!\n"
     ]
    }
   ],
   "source": [
    "xgb_transformer.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the transform job has executed and the result, the estimated sentiment of each review, has been saved on S3. Since we would rather work on this file locally we can perform a bit of notebook magic to copy the file to the `data_dir`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed 256.0 KiB/371.0 KiB (1.9 MiB/s) with 1 file(s) remaining\r",
      "Completed 371.0 KiB/371.0 KiB (2.7 MiB/s) with 1 file(s) remaining\r",
      "download: s3://sagemaker-us-east-1-112107346316/xgboost-2019-04-09-20-30-05-405/test.csv.out to ../data/xgboost/test.csv.out\r\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp --recursive $xgb_transformer.output_path $data_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last step is now to read in the output from our model, convert the output to something a little more usable, in this case we want the sentiment to be either `1` (positive) or `0` (negative), and then compare to the ground truth labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = pd.read_csv(os.path.join(data_dir, 'test.csv.out'), header=None)\n",
    "predictions = [round(num) for num in predictions.squeeze().values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.86632"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(test_y, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional: Clean up\n",
    "\n",
    "The default notebook instance on SageMaker doesn't have a lot of excess disk space available. As you continue to complete and execute notebooks you will eventually fill up this disk space, leading to errors which can be difficult to diagnose. Once you are completely finished using a notebook it is a good idea to remove the files that you created along the way. Of course, you can do this from the terminal or from the notebook hub if you would like. The cell below contains some commands to clean up the created files from within the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we will remove all of the files contained in the data_dir directory\n",
    "!rm $data_dir/*\n",
    "\n",
    "# And then we delete the directory itself\n",
    "!rmdir $data_dir\n",
    "\n",
    "# Similarly we will remove the files in the cache_dir directory and the directory itself\n",
    "!rm $cache_dir/*\n",
    "!rmdir $cache_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}