t81_558_class_14_other.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 14: Other Neural Network Techniques**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module Video Material\n",
    "\n",
    "Main video lecture:\n",
    "\n",
    "* [Part 14.1: What is AutoML]()\n",
    "* [Part 14.2: Using Denoising AutoEncoders in Keras]()\n",
    "* [Part 14.3: Anomaly Detection in Keras]()\n",
    "* [Part 14.4: Training an Intrusion Detection System with KDD99]()\n",
    "* [Part 14.5: The Deep Learning Technologies I am Excited About]()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 14.2: Using Denoising AutoEncoders in Keras\n",
    "\n",
    "# Function Approximation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Activation\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "rng = np.random.RandomState(1)\n",
    "x = np.sort((360 * rng.rand(100, 1)), axis=0)\n",
    "y = np.array([np.sin(x*(np.pi/180.0)).ravel()]).T\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu'))\n",
    "model.add(Dense(50, activation='relu'))\n",
    "model.add(Dense(25, activation='relu'))\n",
    "model.add(Dense(1))\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,y,verbose=0,batch_size=len(x),epochs=25000)\n",
    "\n",
    "pred = model.predict(x)\n",
    "\n",
    "print(\"Actual\")\n",
    "print(y[0:5])\n",
    "\n",
    "print(\"Pred\")\n",
    "print(pred[0:5])\n",
    "\n",
    "chart_regression(pred.flatten(),y,sort=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-Output Regression\n",
    "\n",
    "Unlike most models, neural networks can provide multiple regression outputs.  This allows a neural network to generate multiple outputs for the same input.  For example, the MPG data set might be trained to predict both MPG and horsepower.  One area that multiple regression outputs can be useful for is auto encoders.  The following diagram shows a multi-regression neural network.  As you can see, there are multiple output neurons.  Usually multiple output neurons are used for classification.  However, in this case it is a regression neural network.\n",
    "\n",
    "![Multi-Output Regression](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_13_multi_output.png \"Multi-Output Regression\")\n",
    "\n",
    "The following program uses a multi-output regression to predict both [sin](https://en.wikipedia.org/wiki/Trigonometric_functions#Sine.2C_cosine_and_tangent) and [cos](https://en.wikipedia.org/wiki/Trigonometric_functions#Sine.2C_cosine_and_tangent) from the same input data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "rng = np.random.RandomState(1)\n",
    "x = np.sort((360 * rng.rand(100, 1)), axis=0)\n",
    "y = np.array([np.pi * np.sin(x*(np.pi/180.0)).ravel(), np.pi * np.cos(x*(np.pi/180.0)).ravel()]).T\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu'))\n",
    "model.add(Dense(50, activation='relu'))\n",
    "model.add(Dense(25, activation='relu'))\n",
    "model.add(Dense(2)) # Two output neurons\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,y,verbose=0,batch_size=len(x),epochs=25000)\n",
    "\n",
    "\n",
    "# Fit regression DNN model.\n",
    "pred = model.predict(x)\n",
    "\n",
    "score = np.sqrt(metrics.mean_squared_error(pred, y))\n",
    "print(\"Score (RMSE): {}\".format(score))\n",
    "\n",
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "print(\"Predicted:\")\n",
    "print(np.array(pred[20:25]))\n",
    "\n",
    "print(\"Expected:\")\n",
    "print(np.array(y[20:25]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple Auto Encoder\n",
    "\n",
    "An auto encoder is a neural network that has the same number of input neurons as it does outputs.  The hidden layers of the neural network will have fewer neurons than the input/output neurons.  Because there are fewer neurons, the auto-encoder must learn to encode the input to the fewer hidden neurons.  The predictors (x) and output (y) are exactly the same in an auto encoder.  Because of this, auto encoders are said to be unsupervised.  \n",
    "\n",
    "![Simple Auto Encoder](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_13_auto_encode.png \"Simple Auto Encoder\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following program demonstrates a very simple auto encoder that learns to encode a sequence of numbers.  Fewer hidden neurons will make it much more difficult for the auto encoder to learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML \n",
    "import tensorflow as tf\n",
    "\n",
    "x = np.array([range(10)]).astype(np.float32)\n",
    "print(x)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(3, input_dim=x.shape[1], activation='relu'))\n",
    "model.add(Dense(x.shape[1])) # Multiple output neurons\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,x,verbose=0,epochs=1000)\n",
    "\n",
    "pred = model.predict(x)\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,x))\n",
    "print(\"Fold score (RMSE): {}\".format(score))\n",
    "np.set_printoptions(suppress=True)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Auto Encode (single image)\n",
    "\n",
    "We are now ready to build a simple image auto encoder.  The program below learns an effective encoding for the image.  You can see the distortions that occur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from PIL import Image, ImageFile\n",
    "from matplotlib.pyplot import imshow\n",
    "from keras.optimizers import SGD\n",
    "import requests\n",
    "from io import BytesIO\n",
    "\n",
    "url = \"https://upload.wikimedia.org/wikipedia/commons/9/92/Brookings.jpg\"\n",
    "    \n",
    "response = requests.get(url)\n",
    "img = Image.open(BytesIO(response.content))\n",
    "img.load()\n",
    "img = img.resize((128,128), Image.ANTIALIAS)\n",
    "img_array = np.asarray(img)\n",
    "img_array = img_array.flatten()\n",
    "img_array = np.array([ img_array ])\n",
    "img_array = img_array.astype(np.float32)\n",
    "print(img_array.shape[1])\n",
    "print(img_array)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(10, input_dim=img_array.shape[1], activation='relu'))\n",
    "model.add(Dense(img_array.shape[1])) # Multiple output neurons\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(img_array,img_array,verbose=0,epochs=20)\n",
    "\n",
    "print(\"Neural network output\")\n",
    "pred = model.predict(img_array)\n",
    "print(pred)\n",
    "print(img_array)\n",
    "cols,rows = img.size\n",
    "img_array2 = pred[0].reshape(rows,cols,3)\n",
    "img_array2 = img_array2.astype(np.uint8)\n",
    "img2 = Image.fromarray(img_array2, 'RGB')\n",
    "img2   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Standardize Images\n",
    "\n",
    "When processing several images together it is sometimes important to standardize them.  The following code reads a sequence of images and causes them to all be of the same size and perfectly square.  If the input images are not square, cropping will occur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from PIL import Image, ImageFile\n",
    "from matplotlib.pyplot import imshow\n",
    "import requests\n",
    "import numpy as np\n",
    "from io import BytesIO\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "#url = \"http://www.heatonresearch.com/images/about-jeff.jpg\"\n",
    "\n",
    "images = [\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/9/92/Brookings.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/f/ff/WashU_Graham_Chapel.JPG\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/9/9e/SeigleHall.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/a/aa/WUSTLKnight.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/3/32/WashUABhall.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/c/c0/Brown_Hall.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/f/f4/South40.jpg\"    \n",
    "]\n",
    "\n",
    "\n",
    "def make_square(img):\n",
    "    cols,rows = img.size\n",
    "    \n",
    "    if rows>cols:\n",
    "        pad = (rows-cols)/2\n",
    "        img = img.crop((pad,0,cols,cols))\n",
    "    else:\n",
    "        pad = (cols-rows)/2\n",
    "        img = img.crop((0,pad,rows,rows))\n",
    "    \n",
    "    return img\n",
    "    \n",
    "x = [] \n",
    "    \n",
    "for url in images:\n",
    "    ImageFile.LOAD_TRUNCATED_IMAGES = False\n",
    "    response = requests.get(url)\n",
    "    img = Image.open(BytesIO(response.content))\n",
    "    img.load()\n",
    "    img = make_square(img)\n",
    "    img = img.resize((128,128), Image.ANTIALIAS)\n",
    "    print(url)\n",
    "    display(img)\n",
    "    img_array = np.asarray(img)\n",
    "    img_array = img_array.flatten()\n",
    "    img_array = img_array.astype(np.float32)\n",
    "    img_array = (img_array-128)/128\n",
    "    x.append(img_array)\n",
    "    \n",
    "\n",
    "x = np.array(x)\n",
    "\n",
    "print(x.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Image Auto Encoder (multi-image)\n",
    "\n",
    "Auto encoders can learn the same encoding for multiple images.  The following code learns a single encoding for multiple images. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from PIL import Image, ImageFile\n",
    "from matplotlib.pyplot import imshow\n",
    "import requests\n",
    "from io import BytesIO\n",
    "from sklearn import metrics\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from IPython.display import display, HTML \n",
    "\n",
    "# Fit regression DNN model.\n",
    "print(\"Creating/Training neural network\")\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu'))\n",
    "model.add(Dense(x.shape[1])) # Multiple output neurons\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,x,verbose=0,epochs=1000)\n",
    "\n",
    "print(\"Score neural network\")\n",
    "pred = model.predict(x)\n",
    "\n",
    "cols,rows = img.size\n",
    "for i in range(len(pred)):\n",
    "    print(pred[i])\n",
    "    img_array2 = pred[i].reshape(rows,cols,3)\n",
    "    img_array2 = (img_array2*128)+128\n",
    "    img_array2 = img_array2.astype(np.uint8)\n",
    "    img2 = Image.fromarray(img_array2, 'RGB')\n",
    "    display(img2)   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding Noise to an Image\n",
    "\n",
    "Auto encoders can handle noise.  First it is important to see how to add noise to an image.  There are many ways to add such noise.  The following code adds random black squares to the image to produce noise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image, ImageFile\n",
    "from matplotlib.pyplot import imshow\n",
    "import requests\n",
    "from io import BytesIO\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "\n",
    "def add_noise(a):\n",
    "    a2 = a.copy()\n",
    "    rows = a2.shape[0]\n",
    "    cols = a2.shape[1]\n",
    "    s = int(min(rows,cols)/20) # size of spot is 1/20 of smallest dimension\n",
    "    \n",
    "    for i in range(100):\n",
    "        x = np.random.randint(cols-s)\n",
    "        y = np.random.randint(rows-s)\n",
    "        a2[y:(y+s),x:(x+s)] = 0\n",
    "        \n",
    "    return a2\n",
    "\n",
    "url = \"https://upload.wikimedia.org/wikipedia/commons/9/92/Brookings.jpg\"\n",
    "#url = \"http://www.heatonresearch.com/images/about-jeff.jpg\"\n",
    "\n",
    "response = requests.get(url)\n",
    "img = Image.open(BytesIO(response.content))\n",
    "img.load()\n",
    "\n",
    "img_array = np.asarray(img)\n",
    "rows = img_array.shape[0]\n",
    "cols = img_array.shape[1]\n",
    "\n",
    "print(\"Rows: {}, Cols: {}\".format(rows,cols))\n",
    "\n",
    "# Create new image\n",
    "img2_array = img_array.astype(np.uint8)\n",
    "print(img2_array.shape)\n",
    "img2_array = add_noise(img2_array)\n",
    "img2 = Image.fromarray(img2_array, 'RGB')\n",
    "img2        \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Denoising Autoencoder\n",
    "\n",
    "A denoising auto encoder is designed to remove noise from input signals.  To do this the $y$ becomes each image/signal (just like a normal auto encoder), however, the $x$ becomes a version of $y$ with noise added.  Noise is artificially added to the images to produce $x$.  The following code creates 10 noisy version of each of the images.  The network is trained to convert noisy data ($x$) to the origional input ($y$)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from PIL import Image, ImageFile\n",
    "from matplotlib.pyplot import imshow\n",
    "import requests\n",
    "import numpy as np\n",
    "from io import BytesIO\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "#url = \"http://www.heatonresearch.com/images/about-jeff.jpg\"\n",
    "\n",
    "images = [\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/9/92/Brookings.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/f/ff/WashU_Graham_Chapel.JPG\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/9/9e/SeigleHall.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/a/aa/WUSTLKnight.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/3/32/WashUABhall.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/c/c0/Brown_Hall.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/f/f4/South40.jpg\"    \n",
    "]\n",
    "\n",
    "\n",
    "def make_square(img):\n",
    "    cols,rows = img.size\n",
    "    \n",
    "    if rows>cols:\n",
    "        pad = (rows-cols)/2\n",
    "        img = img.crop((pad,0,cols,cols))\n",
    "    else:\n",
    "        pad = (cols-rows)/2\n",
    "        img = img.crop((0,pad,rows,rows))\n",
    "    \n",
    "    return img\n",
    "    \n",
    "x = []    \n",
    "y = []\n",
    "loaded_images = []\n",
    "    \n",
    "for url in images:\n",
    "    ImageFile.LOAD_TRUNCATED_IMAGES = False\n",
    "    response = requests.get(url)\n",
    "    img = Image.open(BytesIO(response.content))\n",
    "    img.load()\n",
    "    img = make_square(img)\n",
    "    img = img.resize((128,128), Image.ANTIALIAS)\n",
    "    \n",
    "    loaded_images.append(img)\n",
    "    print(url)\n",
    "    display(img)\n",
    "    for i in range(10):\n",
    "        img_array = np.asarray(img)\n",
    "        img_array_noise = add_noise(img_array)\n",
    "        \n",
    "        img_array = img_array.flatten()\n",
    "        img_array = img_array.astype(np.float32)\n",
    "        img_array = (img_array-128)/128\n",
    "        \n",
    "        img_array_noise = img_array_noise.flatten()\n",
    "        img_array_noise = img_array_noise.astype(np.float32)\n",
    "        img_array_noise = (img_array_noise-128)/128\n",
    "        \n",
    "        x.append(img_array_noise)\n",
    "        y.append(img_array)\n",
    "    \n",
    "x = np.array(x)\n",
    "y = np.array(y)\n",
    "\n",
    "print(x.shape)\n",
    "print(y.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from PIL import Image, ImageFile\n",
    "from matplotlib.pyplot import imshow\n",
    "import requests\n",
    "from io import BytesIO\n",
    "from sklearn import metrics\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from IPython.display import display, HTML \n",
    "\n",
    "# Fit regression DNN model.\n",
    "print(\"Creating/Training neural network\")\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu'))\n",
    "model.add(Dense(50, activation='relu'))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(x.shape[1])) # Multiple output neurons\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,y,verbose=1,epochs=20)\n",
    "\n",
    "print(\"Neural network trained\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for z in range(10):\n",
    "    print(\"*** Trial {}\".format(z+1))\n",
    "    \n",
    "    # Choose random image\n",
    "    i = np.random.randint(len(loaded_images))\n",
    "    img = loaded_images[i]\n",
    "    img_array = np.asarray(img)\n",
    "    cols, rows = img.size\n",
    "\n",
    "    # Add noise\n",
    "    img_array_noise = add_noise(img_array)    \n",
    "\n",
    "    #Display noisy image\n",
    "    img2 = img_array_noise.astype(np.uint8)\n",
    "    img2 = Image.fromarray(img2, 'RGB')\n",
    "    print(\"With noise:\")\n",
    "    display(img2)\n",
    "\n",
    "    # Present noisy image to auto encoder\n",
    "    img_array_noise = img_array_noise.flatten()\n",
    "    img_array_noise = img_array_noise.astype(np.float32)\n",
    "    img_array_noise = (img_array_noise-128)/128\n",
    "    img_array_noise = np.array([img_array_noise])\n",
    "    pred = model.predict(img_array_noise)[0]\n",
    "\n",
    "    # Display neural result\n",
    "    img_array2 = pred.reshape(rows,cols,3)\n",
    "    img_array2 = (img_array2*128)+128\n",
    "    img_array2 = img_array2.astype(np.uint8)\n",
    "    img2 = Image.fromarray(img_array2, 'RGB')\n",
    "    print(\"After auto encode noise removal\")\n",
    "    display(img2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 14.4: Training an Intrusion Detection System with KDD99\n",
    "\n",
    "The [KDD-99 dataset](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html) is very famous in the security field and almost a \"hello world\" of intrusion detection systems in machine learning.\n",
    "\n",
    "# Read in Raw KDD-99 Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from http://kdd.ics.uci.edu/databases/kddcup99/kddcup.data_10_percent.gz\n",
      "2146304/2144903 [==============================] - 11s 5us/step\n",
      "/Users/jheaton/.keras/datasets/kddcup.data_10_percent.gz\n",
      "Read 494021 rows.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>service</th>\n",
       "      <th>flag</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>...</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <th>outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>181</td>\n",
       "      <td>5450</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.11</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>239</td>\n",
       "      <td>486</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>19</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>235</td>\n",
       "      <td>1337</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>29</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>219</td>\n",
       "      <td>1337</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>39</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>217</td>\n",
       "      <td>2032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>49</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration protocol_type service flag  src_bytes  dst_bytes  land  \\\n",
       "0         0           tcp    http   SF        181       5450     0   \n",
       "1         0           tcp    http   SF        239        486     0   \n",
       "2         0           tcp    http   SF        235       1337     0   \n",
       "3         0           tcp    http   SF        219       1337     0   \n",
       "4         0           tcp    http   SF        217       2032     0   \n",
       "\n",
       "   wrong_fragment  urgent  hot   ...     dst_host_srv_count  \\\n",
       "0               0       0    0   ...                      9   \n",
       "1               0       0    0   ...                     19   \n",
       "2               0       0    0   ...                     29   \n",
       "3               0       0    0   ...                     39   \n",
       "4               0       0    0   ...                     49   \n",
       "\n",
       "   dst_host_same_srv_rate  dst_host_diff_srv_rate  \\\n",
       "0                     1.0                     0.0   \n",
       "1                     1.0                     0.0   \n",
       "2                     1.0                     0.0   \n",
       "3                     1.0                     0.0   \n",
       "4                     1.0                     0.0   \n",
       "\n",
       "   dst_host_same_src_port_rate  dst_host_srv_diff_host_rate  \\\n",
       "0                         0.11                          0.0   \n",
       "1                         0.05                          0.0   \n",
       "2                         0.03                          0.0   \n",
       "3                         0.03                          0.0   \n",
       "4                         0.02                          0.0   \n",
       "\n",
       "   dst_host_serror_rate  dst_host_srv_serror_rate  dst_host_rerror_rate  \\\n",
       "0                   0.0                       0.0                   0.0   \n",
       "1                   0.0                       0.0                   0.0   \n",
       "2                   0.0                       0.0                   0.0   \n",
       "3                   0.0                       0.0                   0.0   \n",
       "4                   0.0                       0.0                   0.0   \n",
       "\n",
       "   dst_host_srv_rerror_rate  outcome  \n",
       "0                       0.0  normal.  \n",
       "1                       0.0  normal.  \n",
       "2                       0.0  normal.  \n",
       "3                       0.0  normal.  \n",
       "4                       0.0  normal.  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.utils.data_utils import get_file\n",
    "\n",
    "try:\n",
    "    path = get_file('kddcup.data_10_percent.gz', origin='http://kdd.ics.uci.edu/databases/kddcup99/kddcup.data_10_percent.gz')\n",
    "except:\n",
    "    print('Error downloading')\n",
    "    raise\n",
    "    \n",
    "print(path) \n",
    "\n",
    "# This file is a CSV, just no CSV extension or headers\n",
    "# Download from: http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html\n",
    "df = pd.read_csv(path, header=None)\n",
    "\n",
    "print(\"Read {} rows.\".format(len(df)))\n",
    "# df = df.sample(frac=0.1, replace=False) # Uncomment this line to sample only 10% of the dataset\n",
    "df.dropna(inplace=True,axis=1) # For now, just drop NA's (rows with missing values)\n",
    "\n",
    "# The CSV file has no column heads, so add them\n",
    "df.columns = [\n",
    "    'duration',\n",
    "    'protocol_type',\n",
    "    'service',\n",
    "    'flag',\n",
    "    'src_bytes',\n",
    "    'dst_bytes',\n",
    "    'land',\n",
    "    'wrong_fragment',\n",
    "    'urgent',\n",
    "    'hot',\n",
    "    'num_failed_logins',\n",
    "    'logged_in',\n",
    "    'num_compromised',\n",
    "    'root_shell',\n",
    "    'su_attempted',\n",
    "    'num_root',\n",
    "    'num_file_creations',\n",
    "    'num_shells',\n",
    "    'num_access_files',\n",
    "    'num_outbound_cmds',\n",
    "    'is_host_login',\n",
    "    'is_guest_login',\n",
    "    'count',\n",
    "    'srv_count',\n",
    "    'serror_rate',\n",
    "    'srv_serror_rate',\n",
    "    'rerror_rate',\n",
    "    'srv_rerror_rate',\n",
    "    'same_srv_rate',\n",
    "    'diff_srv_rate',\n",
    "    'srv_diff_host_rate',\n",
    "    'dst_host_count',\n",
    "    'dst_host_srv_count',\n",
    "    'dst_host_same_srv_rate',\n",
    "    'dst_host_diff_srv_rate',\n",
    "    'dst_host_same_src_port_rate',\n",
    "    'dst_host_srv_diff_host_rate',\n",
    "    'dst_host_serror_rate',\n",
    "    'dst_host_srv_serror_rate',\n",
    "    'dst_host_rerror_rate',\n",
    "    'dst_host_srv_rerror_rate',\n",
    "    'outcome'\n",
    "]\n",
    "\n",
    "# display 5 rows\n",
    "df[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyzing a Dataset\n",
    "\n",
    "The following script can be used to give a high-level overview of how a dataset appears."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "ENCODING = 'utf-8'\n",
    "\n",
    "def expand_categories(values):\n",
    "    result = []\n",
    "    s = values.value_counts()\n",
    "    t = float(len(values))\n",
    "    for v in s.index:\n",
    "        result.append(\"{}:{}%\".format(v,round(100*(s[v]/t),2)))\n",
    "    return \"[{}]\".format(\",\".join(result))\n",
    "        \n",
    "def analyze(filename):\n",
    "    print()\n",
    "    print(\"Analyzing: {}\".format(filename))\n",
    "    df = pd.read_csv(filename,encoding=ENCODING)\n",
    "    cols = df.columns.values\n",
    "    total = float(len(df))\n",
    "\n",
    "    print(\"{} rows\".format(int(total)))\n",
    "    for col in cols:\n",
    "        uniques = df[col].unique()\n",
    "        unique_count = len(uniques)\n",
    "        if unique_count>100:\n",
    "            print(\"** {}:{} ({}%)\".format(col,unique_count,int(((unique_count)/total)*100)))\n",
    "        else:\n",
    "            print(\"** {}:{}\".format(col,expand_categories(df[col])))\n",
    "            expand_categories(df[col])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Analyze KDD-99\n",
    "\n",
    "import tensorflow.contrib.learn as skflow\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Encode the feature vector\n",
    "Encode every row in the database.  This is not instant!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>num_failed_logins</th>\n",
       "      <th>num_compromised</th>\n",
       "      <th>root_shell</th>\n",
       "      <th>su_attempted</th>\n",
       "      <th>...</th>\n",
       "      <th>flag-S3</th>\n",
       "      <th>flag-SF</th>\n",
       "      <th>flag-SH</th>\n",
       "      <th>land-0</th>\n",
       "      <th>land-1</th>\n",
       "      <th>logged_in-0</th>\n",
       "      <th>logged_in-1</th>\n",
       "      <th>is_host_login-0</th>\n",
       "      <th>is_guest_login-0</th>\n",
       "      <th>is_guest_login-1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002879</td>\n",
       "      <td>0.138664</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002820</td>\n",
       "      <td>-0.011578</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002824</td>\n",
       "      <td>0.014179</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002840</td>\n",
       "      <td>0.014179</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002842</td>\n",
       "      <td>0.035214</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 121 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration  src_bytes  dst_bytes  wrong_fragment    urgent       hot  \\\n",
       "0 -0.067792  -0.002879   0.138664        -0.04772 -0.002571 -0.044136   \n",
       "1 -0.067792  -0.002820  -0.011578        -0.04772 -0.002571 -0.044136   \n",
       "2 -0.067792  -0.002824   0.014179        -0.04772 -0.002571 -0.044136   \n",
       "3 -0.067792  -0.002840   0.014179        -0.04772 -0.002571 -0.044136   \n",
       "4 -0.067792  -0.002842   0.035214        -0.04772 -0.002571 -0.044136   \n",
       "\n",
       "   num_failed_logins  num_compromised  root_shell  su_attempted  \\\n",
       "0          -0.009782        -0.005679   -0.010552     -0.004676   \n",
       "1          -0.009782        -0.005679   -0.010552     -0.004676   \n",
       "2          -0.009782        -0.005679   -0.010552     -0.004676   \n",
       "3          -0.009782        -0.005679   -0.010552     -0.004676   \n",
       "4          -0.009782        -0.005679   -0.010552     -0.004676   \n",
       "\n",
       "         ...         flag-S3  flag-SF  flag-SH  land-0  land-1  logged_in-0  \\\n",
       "0        ...               0        1        0       1       0            0   \n",
       "1        ...               0        1        0       1       0            0   \n",
       "2        ...               0        1        0       1       0            0   \n",
       "3        ...               0        1        0       1       0            0   \n",
       "4        ...               0        1        0       1       0            0   \n",
       "\n",
       "   logged_in-1  is_host_login-0  is_guest_login-0  is_guest_login-1  \n",
       "0            1                1                 1                 0  \n",
       "1            1                1                 1                 0  \n",
       "2            1                1                 1                 0  \n",
       "3            1                1                 1                 0  \n",
       "4            1                1                 1                 0  \n",
       "\n",
       "[5 rows x 121 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now encode the feature vector\n",
    "\n",
    "encode_numeric_zscore(df, 'duration')\n",
    "encode_text_dummy(df, 'protocol_type')\n",
    "encode_text_dummy(df, 'service')\n",
    "encode_text_dummy(df, 'flag')\n",
    "encode_numeric_zscore(df, 'src_bytes')\n",
    "encode_numeric_zscore(df, 'dst_bytes')\n",
    "encode_text_dummy(df, 'land')\n",
    "encode_numeric_zscore(df, 'wrong_fragment')\n",
    "encode_numeric_zscore(df, 'urgent')\n",
    "encode_numeric_zscore(df, 'hot')\n",
    "encode_numeric_zscore(df, 'num_failed_logins')\n",
    "encode_text_dummy(df, 'logged_in')\n",
    "encode_numeric_zscore(df, 'num_compromised')\n",
    "encode_numeric_zscore(df, 'root_shell')\n",
    "encode_numeric_zscore(df, 'su_attempted')\n",
    "encode_numeric_zscore(df, 'num_root')\n",
    "encode_numeric_zscore(df, 'num_file_creations')\n",
    "encode_numeric_zscore(df, 'num_shells')\n",
    "encode_numeric_zscore(df, 'num_access_files')\n",
    "encode_numeric_zscore(df, 'num_outbound_cmds')\n",
    "encode_text_dummy(df, 'is_host_login')\n",
    "encode_text_dummy(df, 'is_guest_login')\n",
    "encode_numeric_zscore(df, 'count')\n",
    "encode_numeric_zscore(df, 'srv_count')\n",
    "encode_numeric_zscore(df, 'serror_rate')\n",
    "encode_numeric_zscore(df, 'srv_serror_rate')\n",
    "encode_numeric_zscore(df, 'rerror_rate')\n",
    "encode_numeric_zscore(df, 'srv_rerror_rate')\n",
    "encode_numeric_zscore(df, 'same_srv_rate')\n",
    "encode_numeric_zscore(df, 'diff_srv_rate')\n",
    "encode_numeric_zscore(df, 'srv_diff_host_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_count')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_count')\n",
    "encode_numeric_zscore(df, 'dst_host_same_srv_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_diff_srv_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_same_src_port_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_diff_host_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_serror_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_serror_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_rerror_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_rerror_rate')\n",
    "outcomes = encode_text_index(df, 'outcome')\n",
    "num_classes = len(outcomes)\n",
    "\n",
    "# display 5 rows\n",
    "\n",
    "df.dropna(inplace=True,axis=1)\n",
    "df[0:5]\n",
    "# This is the numeric feature vector, as it goes to the neural net\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train the Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 370515 samples, validate on 123506 samples\n",
      "Epoch 1/1000\n",
      " - 14s - loss: 0.1870 - val_loss: 0.0568\n",
      "Epoch 2/1000\n",
      " - 14s - loss: 0.0493 - val_loss: 0.0506\n",
      "Epoch 3/1000\n",
      " - 14s - loss: 0.0427 - val_loss: 0.0529\n",
      "Epoch 4/1000\n",
      " - 20s - loss: 0.0410 - val_loss: 0.0450\n",
      "Epoch 5/1000\n",
      " - 13s - loss: 0.0402 - val_loss: 0.0447\n",
      "Epoch 6/1000\n",
      " - 13s - loss: 0.0396 - val_loss: 0.0455\n",
      "Epoch 7/1000\n",
      " - 13s - loss: 0.0412 - val_loss: 0.0435\n",
      "Epoch 8/1000\n",
      " - 13s - loss: 0.0386 - val_loss: 0.0419\n",
      "Epoch 9/1000\n",
      " - 13s - loss: 0.0382 - val_loss: 0.0402\n",
      "Epoch 10/1000\n",
      " - 13s - loss: 0.0371 - val_loss: 0.0434\n",
      "Epoch 11/1000\n",
      " - 13s - loss: 0.0377 - val_loss: 0.0406\n",
      "Epoch 12/1000\n",
      " - 13s - loss: 0.0367 - val_loss: 0.0390\n",
      "Epoch 13/1000\n",
      " - 14s - loss: 0.0360 - val_loss: 0.0393\n",
      "Epoch 14/1000\n",
      " - 13s - loss: 0.0355 - val_loss: 0.0376\n",
      "Epoch 15/1000\n",
      " - 13s - loss: 0.0349 - val_loss: 0.0392\n",
      "Epoch 16/1000\n",
      " - 13s - loss: 0.0343 - val_loss: 0.0392\n",
      "Epoch 17/1000\n",
      " - 13s - loss: 0.0328 - val_loss: 0.0370\n",
      "Epoch 18/1000\n",
      " - 13s - loss: 0.0309 - val_loss: 0.0329\n",
      "Epoch 19/1000\n",
      " - 13s - loss: 0.0316 - val_loss: 0.0329\n",
      "Epoch 20/1000\n",
      " - 13s - loss: 0.0305 - val_loss: 0.0367\n",
      "Epoch 21/1000\n",
      " - 13s - loss: 0.0306 - val_loss: 0.0324\n",
      "Epoch 22/1000\n",
      " - 17s - loss: 0.0330 - val_loss: 0.0358\n",
      "Epoch 23/1000\n",
      " - 18s - loss: 0.0295 - val_loss: 0.0306\n",
      "Epoch 24/1000\n",
      " - 16s - loss: 0.0303 - val_loss: 0.0307\n",
      "Epoch 25/1000\n",
      " - 14s - loss: 0.0288 - val_loss: 0.0334\n",
      "Epoch 26/1000\n",
      " - 14s - loss: 0.0290 - val_loss: 0.0343\n",
      "Epoch 27/1000\n",
      " - 21s - loss: 0.0293 - val_loss: 0.0399\n",
      "Epoch 28/1000\n",
      " - 14s - loss: 0.0332 - val_loss: 0.0329\n",
      "Epoch 00028: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1a2b4c5748>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "import os\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Activation\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "# Break into X (predictors) & y (prediction)\n",
    "x, y = to_xy(df,'outcome')\n",
    "\n",
    "# Create a test/train split.  25% test\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(\n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Create neural net\n",
    "model = Sequential()\n",
    "model.add(Dense(10, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))\n",
    "model.add(Dense(50, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))\n",
    "model.add(Dense(10, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))\n",
    "model.add(Dense(1, kernel_initializer='normal'))\n",
    "model.add(Dense(y.shape[1],activation='softmax'))\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto')\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation score: 0.9975547746668179\n"
     ]
    }
   ],
   "source": [
    "# Measure accuracy\n",
    "pred = model.predict(x_test)\n",
    "pred = np.argmax(pred,axis=1)\n",
    "y_eval = np.argmax(y_test,axis=1)\n",
    "score = metrics.accuracy_score(y_eval, pred)\n",
    "print(\"Validation score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}