t81_558_class_03_3_save_load.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 3: Introduction to TensorFlow**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 3 Material\n",
    "\n",
    "* Part 3.1: Deep Learning and Neural Network Introduction [[Video]](https://www.youtube.com/watch?v=zYnI4iWRmpc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_01_neural_net.ipynb)\n",
    "* Part 3.2: Introduction to Tensorflow & Keras [[Video]](https://www.youtube.com/watch?v=PsE73jk55cE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_02_keras.ipynb)\n",
    "* **Part 3.3: Saving and Loading a Keras Neural Network** [[Video]](https://www.youtube.com/watch?v=-9QfbGM1qGw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_03_save_load.ipynb)\n",
    "* Part 3.4: Early Stopping in Keras to Prevent Overfitting [[Video]](https://www.youtube.com/watch?v=m1LNunuI2fk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_04_early_stop.ipynb)\n",
    "* Part 3.5: Extracting Weights and Manual Calculation [[Video]](https://www.youtube.com/watch?v=7PWgx16kH8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_05_weights.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3.3: Saving and Loading a Keras Neural Network\n",
    "\n",
    "Complex neural networks will take a long time to fit/train.  It is helpful to be able to save these neural networks so that they can be reloaded later.  A reloaded neural network will not require retraining.  Keras provides three formats for neural network saving.\n",
    "\n",
    "* **YAML** - Stores the neural network structure (no weights) in the [YAML file format](https://en.wikipedia.org/wiki/YAML).\n",
    "* **JSON** - Stores the neural network structure (no weights) in the [JSON file format](https://en.wikipedia.org/wiki/JSON).\n",
    "* **HDF5** - Stores the complete neural network (with weights) in the [HDF5 file format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format). Do not confuse HDF5 with [HDFS](https://en.wikipedia.org/wiki/Apache_Hadoop).  They are different.  We do not use HDFS in this class.\n",
    "\n",
    "Usually you will want to save in HDF5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "import pandas as pd\n",
    "import io\n",
    "import os\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "\n",
    "save_path = \".\"\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "cars = df['name']\n",
    "\n",
    "# Handle missing value\n",
    "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
    "\n",
    "# Pandas to Numpy\n",
    "x = df[['cylinders', 'displacement', 'horsepower', 'weight',\n",
    "       'acceleration', 'year', 'origin']].values\n",
    "y = df['mpg'].values # regression\n",
    "\n",
    "# Build the neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)\n",
    "\n",
    "# Predict\n",
    "pred = model.predict(x)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(f\"Before save score (RMSE): {score}\")\n",
    "\n",
    "# save neural network structure to JSON (no weights)\n",
    "model_json = model.to_json()\n",
    "with open(os.path.join(save_path,\"network.json\"), \"w\") as json_file:\n",
    "    json_file.write(model_json)\n",
    "\n",
    "# save neural network structure to YAML (no weights)\n",
    "model_yaml = model.to_yaml()\n",
    "with open(os.path.join(save_path,\"network.yaml\"), \"w\") as yaml_file:\n",
    "    yaml_file.write(model_yaml)\n",
    "\n",
    "# save entire network to HDF5 (save everything, suggested)\n",
    "model.save(os.path.join(save_path,\"network.h5\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below sets up a neural network and reads the data (for predictions), but it does not clear the model directory or fit the neural network.  The weights from the previous fit are used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we reload the network and perform another prediction.  The RMSE should match the previous one exactly if the neural network was really saved and reloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import load_model\n",
    "model2 = load_model(os.path.join(save_path,\"network.h5\"))\n",
    "pred = model2.predict(x)\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(f\"After load score (RMSE): {score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3.4: Early Stopping in Keras to Prevent Overfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Overfitting** occurs when a neural network is trained to the point that it begins to memorize rather than generalize.  \n",
    "\n",
    "![Training vs Validation Error for Overfitting](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_3_training_val.png \"Training vs Validation Error for Overfitting\")\n",
    "\n",
    "It is important to segment the original dataset into several datasets:\n",
    "\n",
    "* **Training Set**\n",
    "* **Validation Set**\n",
    "* **Holdout Set**\n",
    "\n",
    "There are several different ways that these sets can be constructed.  The following programs demonstrate some of these.\n",
    "\n",
    "The first method is a training and validation set.  The training data are used to train the neural network until the validation set no longer improves.  This attempts to stop at a near optimal training point.  This method will only give accurate \"out of sample\" predictions for the validation set, this is usually 20% or so of the data.  The predictions for the training data will be overly optimistic, as these were the data that the neural network was trained on.  \n",
    "\n",
    "![Training with a Validation Set](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_1_train_val.png \"Training with a Validation Set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Early Stopping with Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 112 samples, validate on 38 samples\n",
      "Epoch 1/1000\n",
      "112/112 - 0s - loss: 1.2631 - val_loss: 1.1849\n",
      "Epoch 2/1000\n",
      "112/112 - 0s - loss: 1.1055 - val_loss: 1.0706\n",
      "Epoch 3/1000\n",
      "112/112 - 0s - loss: 1.0157 - val_loss: 1.0093\n",
      "Epoch 4/1000\n",
      "112/112 - 0s - loss: 0.9774 - val_loss: 0.9663\n",
      "Epoch 5/1000\n",
      "112/112 - 0s - loss: 0.9449 - val_loss: 0.9235\n",
      "Epoch 6/1000\n",
      "112/112 - 0s - loss: 0.9073 - val_loss: 0.8794\n",
      "Epoch 7/1000\n",
      "112/112 - 0s - loss: 0.8659 - val_loss: 0.8314\n",
      "Epoch 8/1000\n",
      "112/112 - 0s - loss: 0.8203 - val_loss: 0.7881\n",
      "Epoch 9/1000\n",
      "112/112 - 0s - loss: 0.7832 - val_loss: 0.7506\n",
      "Epoch 10/1000\n",
      "112/112 - 0s - loss: 0.7527 - val_loss: 0.7151\n",
      "Epoch 11/1000\n",
      "112/112 - 0s - loss: 0.7208 - val_loss: 0.6813\n",
      "Epoch 12/1000\n",
      "112/112 - 0s - loss: 0.6904 - val_loss: 0.6484\n",
      "Epoch 13/1000\n",
      "112/112 - 0s - loss: 0.6607 - val_loss: 0.6162\n",
      "Epoch 14/1000\n",
      "112/112 - 0s - loss: 0.6322 - val_loss: 0.5841\n",
      "Epoch 15/1000\n",
      "112/112 - 0s - loss: 0.6048 - val_loss: 0.5549\n",
      "Epoch 16/1000\n",
      "112/112 - 0s - loss: 0.5787 - val_loss: 0.5287\n",
      "Epoch 17/1000\n",
      "112/112 - 0s - loss: 0.5544 - val_loss: 0.5034\n",
      "Epoch 18/1000\n",
      "112/112 - 0s - loss: 0.5315 - val_loss: 0.4796\n",
      "Epoch 19/1000\n",
      "112/112 - 0s - loss: 0.5096 - val_loss: 0.4580\n",
      "Epoch 20/1000\n",
      "112/112 - 0s - loss: 0.4902 - val_loss: 0.4378\n",
      "Epoch 21/1000\n",
      "112/112 - 0s - loss: 0.4706 - val_loss: 0.4188\n",
      "Epoch 22/1000\n",
      "112/112 - 0s - loss: 0.4536 - val_loss: 0.4019\n",
      "Epoch 23/1000\n",
      "112/112 - 0s - loss: 0.4396 - val_loss: 0.3865\n",
      "Epoch 24/1000\n",
      "112/112 - 0s - loss: 0.4215 - val_loss: 0.3709\n",
      "Epoch 25/1000\n",
      "112/112 - 0s - loss: 0.4077 - val_loss: 0.3573\n",
      "Epoch 26/1000\n",
      "112/112 - 0s - loss: 0.3938 - val_loss: 0.3440\n",
      "Epoch 27/1000\n",
      "112/112 - 0s - loss: 0.3815 - val_loss: 0.3318\n",
      "Epoch 28/1000\n",
      "112/112 - 0s - loss: 0.3707 - val_loss: 0.3208\n",
      "Epoch 29/1000\n",
      "112/112 - 0s - loss: 0.3579 - val_loss: 0.3102\n",
      "Epoch 30/1000\n",
      "112/112 - 0s - loss: 0.3465 - val_loss: 0.3000\n",
      "Epoch 31/1000\n",
      "112/112 - 0s - loss: 0.3372 - val_loss: 0.2905\n",
      "Epoch 32/1000\n",
      "112/112 - 0s - loss: 0.3268 - val_loss: 0.2841\n",
      "Epoch 33/1000\n",
      "112/112 - 0s - loss: 0.3190 - val_loss: 0.2756\n",
      "Epoch 34/1000\n",
      "112/112 - 0s - loss: 0.3139 - val_loss: 0.2665\n",
      "Epoch 35/1000\n",
      "112/112 - 0s - loss: 0.3011 - val_loss: 0.2597\n",
      "Epoch 36/1000\n",
      "112/112 - 0s - loss: 0.2998 - val_loss: 0.2516\n",
      "Epoch 37/1000\n",
      "112/112 - 0s - loss: 0.2883 - val_loss: 0.2436\n",
      "Epoch 38/1000\n",
      "112/112 - 0s - loss: 0.2789 - val_loss: 0.2397\n",
      "Epoch 39/1000\n",
      "112/112 - 0s - loss: 0.2717 - val_loss: 0.2315\n",
      "Epoch 40/1000\n",
      "112/112 - 0s - loss: 0.2655 - val_loss: 0.2248\n",
      "Epoch 41/1000\n",
      "112/112 - 0s - loss: 0.2585 - val_loss: 0.2189\n",
      "Epoch 42/1000\n",
      "112/112 - 0s - loss: 0.2498 - val_loss: 0.2168\n",
      "Epoch 43/1000\n",
      "112/112 - 0s - loss: 0.2465 - val_loss: 0.2124\n",
      "Epoch 44/1000\n",
      "112/112 - 0s - loss: 0.2418 - val_loss: 0.2028\n",
      "Epoch 45/1000\n",
      "112/112 - 0s - loss: 0.2334 - val_loss: 0.1979\n",
      "Epoch 46/1000\n",
      "112/112 - 0s - loss: 0.2278 - val_loss: 0.1952\n",
      "Epoch 47/1000\n",
      "112/112 - 0s - loss: 0.2225 - val_loss: 0.1889\n",
      "Epoch 48/1000\n",
      "112/112 - 0s - loss: 0.2178 - val_loss: 0.1840\n",
      "Epoch 49/1000\n",
      "112/112 - 0s - loss: 0.2128 - val_loss: 0.1829\n",
      "Epoch 50/1000\n",
      "112/112 - 0s - loss: 0.2073 - val_loss: 0.1759\n",
      "Epoch 51/1000\n",
      "112/112 - 0s - loss: 0.2010 - val_loss: 0.1701\n",
      "Epoch 52/1000\n",
      "112/112 - 0s - loss: 0.2002 - val_loss: 0.1659\n",
      "Epoch 53/1000\n",
      "112/112 - 0s - loss: 0.1927 - val_loss: 0.1655\n",
      "Epoch 54/1000\n",
      "112/112 - 0s - loss: 0.1937 - val_loss: 0.1653\n",
      "Epoch 55/1000\n",
      "112/112 - 0s - loss: 0.1870 - val_loss: 0.1557\n",
      "Epoch 56/1000\n",
      "112/112 - 0s - loss: 0.1794 - val_loss: 0.1523\n",
      "Epoch 57/1000\n",
      "112/112 - 0s - loss: 0.1760 - val_loss: 0.1492\n",
      "Epoch 58/1000\n",
      "112/112 - 0s - loss: 0.1716 - val_loss: 0.1447\n",
      "Epoch 59/1000\n",
      "112/112 - 0s - loss: 0.1692 - val_loss: 0.1408\n",
      "Epoch 60/1000\n",
      "112/112 - 0s - loss: 0.1666 - val_loss: 0.1405\n",
      "Epoch 61/1000\n",
      "112/112 - 0s - loss: 0.1662 - val_loss: 0.1408\n",
      "Epoch 62/1000\n",
      "112/112 - 0s - loss: 0.1584 - val_loss: 0.1322\n",
      "Epoch 63/1000\n",
      "112/112 - 0s - loss: 0.1567 - val_loss: 0.1296\n",
      "Epoch 64/1000\n",
      "112/112 - 0s - loss: 0.1541 - val_loss: 0.1314\n",
      "Epoch 65/1000\n",
      "112/112 - 0s - loss: 0.1497 - val_loss: 0.1279\n",
      "Epoch 66/1000\n",
      "112/112 - 0s - loss: 0.1469 - val_loss: 0.1234\n",
      "Epoch 67/1000\n",
      "112/112 - 0s - loss: 0.1453 - val_loss: 0.1198\n",
      "Epoch 68/1000\n",
      "112/112 - 0s - loss: 0.1430 - val_loss: 0.1194\n",
      "Epoch 69/1000\n",
      "112/112 - 0s - loss: 0.1404 - val_loss: 0.1211\n",
      "Epoch 70/1000\n",
      "112/112 - 0s - loss: 0.1395 - val_loss: 0.1156\n",
      "Epoch 71/1000\n",
      "112/112 - 0s - loss: 0.1355 - val_loss: 0.1163\n",
      "Epoch 72/1000\n",
      "112/112 - 0s - loss: 0.1328 - val_loss: 0.1123\n",
      "Epoch 73/1000\n",
      "112/112 - 0s - loss: 0.1307 - val_loss: 0.1094\n",
      "Epoch 74/1000\n",
      "112/112 - 0s - loss: 0.1297 - val_loss: 0.1063\n",
      "Epoch 75/1000\n",
      "112/112 - 0s - loss: 0.1291 - val_loss: 0.1045\n",
      "Epoch 76/1000\n",
      "112/112 - 0s - loss: 0.1262 - val_loss: 0.1055\n",
      "Epoch 77/1000\n",
      "112/112 - 0s - loss: 0.1291 - val_loss: 0.1192\n",
      "Epoch 78/1000\n",
      "112/112 - 0s - loss: 0.1264 - val_loss: 0.1033\n",
      "Epoch 79/1000\n",
      "112/112 - 0s - loss: 0.1227 - val_loss: 0.0987\n",
      "Epoch 80/1000\n",
      "112/112 - 0s - loss: 0.1198 - val_loss: 0.0987\n",
      "Epoch 81/1000\n",
      "112/112 - 0s - loss: 0.1194 - val_loss: 0.1028\n",
      "Epoch 82/1000\n",
      "112/112 - 0s - loss: 0.1151 - val_loss: 0.0956\n",
      "Epoch 83/1000\n",
      "112/112 - 0s - loss: 0.1159 - val_loss: 0.0934\n",
      "Epoch 84/1000\n",
      "112/112 - 0s - loss: 0.1139 - val_loss: 0.0978\n",
      "Epoch 85/1000\n",
      "112/112 - 0s - loss: 0.1133 - val_loss: 0.0967\n",
      "Epoch 86/1000\n",
      "112/112 - 0s - loss: 0.1140 - val_loss: 0.0994\n",
      "Epoch 87/1000\n",
      "112/112 - 0s - loss: 0.1090 - val_loss: 0.0894\n",
      "Epoch 88/1000\n",
      "112/112 - 0s - loss: 0.1090 - val_loss: 0.0875\n",
      "Epoch 89/1000\n",
      "112/112 - 0s - loss: 0.1119 - val_loss: 0.0882\n",
      "Epoch 90/1000\n",
      "112/112 - 0s - loss: 0.1082 - val_loss: 0.0950\n",
      "Epoch 91/1000\n",
      "112/112 - 0s - loss: 0.1065 - val_loss: 0.0925\n",
      "Epoch 92/1000\n",
      "112/112 - 0s - loss: 0.1047 - val_loss: 0.0842\n",
      "Epoch 93/1000\n",
      "112/112 - 0s - loss: 0.1091 - val_loss: 0.0829\n",
      "Epoch 94/1000\n",
      "112/112 - 0s - loss: 0.1082 - val_loss: 0.0909\n",
      "Epoch 95/1000\n",
      "112/112 - 0s - loss: 0.1018 - val_loss: 0.0858\n",
      "Epoch 96/1000\n",
      "112/112 - 0s - loss: 0.0999 - val_loss: 0.0804\n",
      "Epoch 97/1000\n",
      "112/112 - 0s - loss: 0.1024 - val_loss: 0.0802\n",
      "Epoch 98/1000\n",
      "112/112 - 0s - loss: 0.0995 - val_loss: 0.0883\n",
      "Epoch 99/1000\n",
      "112/112 - 0s - loss: 0.0988 - val_loss: 0.0917\n",
      "Epoch 100/1000\n",
      "112/112 - 0s - loss: 0.1002 - val_loss: 0.0826\n",
      "Epoch 101/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "112/112 - 0s - loss: 0.0962 - val_loss: 0.0797\n",
      "Epoch 00101: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1a32f0c9e8>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/iris.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x = df[['sepal_l', 'sepal_w', 'petal_l', 'petal_w']].values\n",
    "dummies = pd.get_dummies(df['species']) # Classification\n",
    "species = dummies.columns\n",
    "y = dummies.values\n",
    "\n",
    "# Split into validation and training sets\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Build neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto',\n",
    "        restore_best_weights=True)\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see from above, the entire number of requested epocs were not used.  The neural network training stopped once the validation set no longer improved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "pred = model.predict(x_test)\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y_test,axis=1)\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Early Stopping with Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 298 samples, validate on 100 samples\n",
      "Epoch 1/1000\n",
      "298/298 - 0s - loss: 137009.1588 - val_loss: 87400.7916\n",
      "Epoch 2/1000\n",
      "298/298 - 0s - loss: 64687.9471 - val_loss: 34750.5637\n",
      "Epoch 3/1000\n",
      "298/298 - 0s - loss: 14863.5487 - val_loss: 437.6791\n",
      "Epoch 4/1000\n",
      "298/298 - 0s - loss: 1920.5421 - val_loss: 4129.8140\n",
      "Epoch 5/1000\n",
      "298/298 - 0s - loss: 2874.4793 - val_loss: 669.1977\n",
      "Epoch 6/1000\n",
      "298/298 - 0s - loss: 388.7574 - val_loss: 602.8470\n",
      "Epoch 7/1000\n",
      "298/298 - 0s - loss: 574.0564 - val_loss: 379.9243\n",
      "Epoch 8/1000\n",
      "298/298 - 0s - loss: 277.5623 - val_loss: 286.2166\n",
      "Epoch 9/1000\n",
      "298/298 - 0s - loss: 300.7139 - val_loss: 269.4909\n",
      "Epoch 10/1000\n",
      "298/298 - 0s - loss: 258.5796 - val_loss: 257.4868\n",
      "Epoch 11/1000\n",
      "298/298 - 0s - loss: 258.8578 - val_loss: 254.6119\n",
      "Epoch 12/1000\n",
      "298/298 - 0s - loss: 252.2497 - val_loss: 245.8697\n",
      "Epoch 13/1000\n",
      "298/298 - 0s - loss: 248.8389 - val_loss: 243.8566\n",
      "Epoch 14/1000\n",
      "298/298 - 0s - loss: 244.4187 - val_loss: 238.7814\n",
      "Epoch 15/1000\n",
      "298/298 - 0s - loss: 238.9818 - val_loss: 234.5445\n",
      "Epoch 16/1000\n",
      "298/298 - 0s - loss: 236.7845 - val_loss: 231.0035\n",
      "Epoch 17/1000\n",
      "298/298 - 0s - loss: 232.2934 - val_loss: 228.9457\n",
      "Epoch 18/1000\n",
      "298/298 - 0s - loss: 229.4774 - val_loss: 223.4574\n",
      "Epoch 19/1000\n",
      "298/298 - 0s - loss: 227.1223 - val_loss: 218.9363\n",
      "Epoch 20/1000\n",
      "298/298 - 0s - loss: 221.9513 - val_loss: 216.4789\n",
      "Epoch 21/1000\n",
      "298/298 - 0s - loss: 218.9378 - val_loss: 211.9149\n",
      "Epoch 22/1000\n",
      "298/298 - 0s - loss: 215.4493 - val_loss: 209.9197\n",
      "Epoch 23/1000\n",
      "298/298 - 0s - loss: 214.1163 - val_loss: 203.0125\n",
      "Epoch 24/1000\n",
      "298/298 - 0s - loss: 207.9805 - val_loss: 199.8764\n",
      "Epoch 25/1000\n",
      "298/298 - 0s - loss: 204.4530 - val_loss: 196.3883\n",
      "Epoch 26/1000\n",
      "298/298 - 0s - loss: 200.6158 - val_loss: 191.7415\n",
      "Epoch 27/1000\n",
      "298/298 - 0s - loss: 198.5157 - val_loss: 189.0866\n",
      "Epoch 28/1000\n",
      "298/298 - 0s - loss: 194.7323 - val_loss: 184.6138\n",
      "Epoch 29/1000\n",
      "298/298 - 0s - loss: 191.0196 - val_loss: 181.2299\n",
      "Epoch 30/1000\n",
      "298/298 - 0s - loss: 188.4134 - val_loss: 178.2012\n",
      "Epoch 31/1000\n",
      "298/298 - 0s - loss: 184.4825 - val_loss: 176.0514\n",
      "Epoch 32/1000\n",
      "298/298 - 0s - loss: 181.8757 - val_loss: 169.4620\n",
      "Epoch 33/1000\n",
      "298/298 - 0s - loss: 177.8738 - val_loss: 171.5512\n",
      "Epoch 34/1000\n",
      "298/298 - 0s - loss: 174.6477 - val_loss: 162.5667\n",
      "Epoch 35/1000\n",
      "298/298 - 0s - loss: 177.4287 - val_loss: 159.0651\n",
      "Epoch 36/1000\n",
      "298/298 - 0s - loss: 169.2184 - val_loss: 160.3227\n",
      "Epoch 37/1000\n",
      "298/298 - 0s - loss: 167.7838 - val_loss: 152.2763\n",
      "Epoch 38/1000\n",
      "298/298 - 0s - loss: 165.2119 - val_loss: 154.2723\n",
      "Epoch 39/1000\n",
      "298/298 - 0s - loss: 159.2664 - val_loss: 145.9608\n",
      "Epoch 40/1000\n",
      "298/298 - 0s - loss: 156.9077 - val_loss: 145.2549\n",
      "Epoch 41/1000\n",
      "298/298 - 0s - loss: 154.3356 - val_loss: 140.0966\n",
      "Epoch 42/1000\n",
      "298/298 - 0s - loss: 152.0847 - val_loss: 139.5768\n",
      "Epoch 43/1000\n",
      "298/298 - 0s - loss: 151.5686 - val_loss: 135.4901\n",
      "Epoch 44/1000\n",
      "298/298 - 0s - loss: 148.0668 - val_loss: 131.2476\n",
      "Epoch 45/1000\n",
      "298/298 - 0s - loss: 143.9285 - val_loss: 132.7091\n",
      "Epoch 46/1000\n",
      "298/298 - 0s - loss: 140.7696 - val_loss: 126.0253\n",
      "Epoch 47/1000\n",
      "298/298 - 0s - loss: 138.6575 - val_loss: 126.5558\n",
      "Epoch 48/1000\n",
      "298/298 - 0s - loss: 135.8795 - val_loss: 121.8080\n",
      "Epoch 49/1000\n",
      "298/298 - 0s - loss: 133.8290 - val_loss: 121.2612\n",
      "Epoch 50/1000\n",
      "298/298 - 0s - loss: 131.2323 - val_loss: 116.3520\n",
      "Epoch 51/1000\n",
      "298/298 - 0s - loss: 129.6460 - val_loss: 115.3310\n",
      "Epoch 52/1000\n",
      "298/298 - 0s - loss: 126.7419 - val_loss: 114.1026\n",
      "Epoch 53/1000\n",
      "298/298 - 0s - loss: 125.9019 - val_loss: 111.1985\n",
      "Epoch 54/1000\n",
      "298/298 - 0s - loss: 125.0483 - val_loss: 108.5914\n",
      "Epoch 55/1000\n",
      "298/298 - 0s - loss: 121.6166 - val_loss: 110.9844\n",
      "Epoch 56/1000\n",
      "298/298 - 0s - loss: 118.7622 - val_loss: 103.5990\n",
      "Epoch 57/1000\n",
      "298/298 - 0s - loss: 117.9291 - val_loss: 104.9465\n",
      "Epoch 58/1000\n",
      "298/298 - 0s - loss: 115.4261 - val_loss: 100.3285\n",
      "Epoch 59/1000\n",
      "298/298 - 0s - loss: 114.4067 - val_loss: 100.5623\n",
      "Epoch 60/1000\n",
      "298/298 - 0s - loss: 112.1221 - val_loss: 97.5862\n",
      "Epoch 61/1000\n",
      "298/298 - 0s - loss: 111.3041 - val_loss: 96.5822\n",
      "Epoch 62/1000\n",
      "298/298 - 0s - loss: 108.3902 - val_loss: 93.8154\n",
      "Epoch 63/1000\n",
      "298/298 - 0s - loss: 107.3160 - val_loss: 93.5471\n",
      "Epoch 64/1000\n",
      "298/298 - 0s - loss: 105.4188 - val_loss: 90.5398\n",
      "Epoch 65/1000\n",
      "298/298 - 0s - loss: 106.7369 - val_loss: 91.4318\n",
      "Epoch 66/1000\n",
      "298/298 - 0s - loss: 103.6693 - val_loss: 87.3595\n",
      "Epoch 67/1000\n",
      "298/298 - 0s - loss: 102.3900 - val_loss: 87.9832\n",
      "Epoch 68/1000\n",
      "298/298 - 0s - loss: 100.5562 - val_loss: 85.1072\n",
      "Epoch 69/1000\n",
      "298/298 - 0s - loss: 98.5360 - val_loss: 85.6733\n",
      "Epoch 70/1000\n",
      "298/298 - 0s - loss: 97.5029 - val_loss: 83.7361\n",
      "Epoch 71/1000\n",
      "298/298 - 0s - loss: 97.5912 - val_loss: 82.2137\n",
      "Epoch 72/1000\n",
      "298/298 - 0s - loss: 95.3032 - val_loss: 80.9050\n",
      "Epoch 73/1000\n",
      "298/298 - 0s - loss: 95.0331 - val_loss: 80.0183\n",
      "Epoch 74/1000\n",
      "298/298 - 0s - loss: 93.8799 - val_loss: 78.0280\n",
      "Epoch 75/1000\n",
      "298/298 - 0s - loss: 92.1775 - val_loss: 76.6894\n",
      "Epoch 76/1000\n",
      "298/298 - 0s - loss: 93.9282 - val_loss: 81.0375\n",
      "Epoch 77/1000\n",
      "298/298 - 0s - loss: 92.5300 - val_loss: 74.6066\n",
      "Epoch 78/1000\n",
      "298/298 - 0s - loss: 89.0045 - val_loss: 76.7518\n",
      "Epoch 79/1000\n",
      "298/298 - 0s - loss: 87.9711 - val_loss: 73.6240\n",
      "Epoch 80/1000\n",
      "298/298 - 0s - loss: 86.4187 - val_loss: 73.5404\n",
      "Epoch 81/1000\n",
      "298/298 - 0s - loss: 85.1361 - val_loss: 72.9517\n",
      "Epoch 82/1000\n",
      "298/298 - 0s - loss: 84.2285 - val_loss: 70.8196\n",
      "Epoch 83/1000\n",
      "298/298 - 0s - loss: 83.5711 - val_loss: 73.0940\n",
      "Epoch 84/1000\n",
      "298/298 - 0s - loss: 83.3114 - val_loss: 68.8215\n",
      "Epoch 85/1000\n",
      "298/298 - 0s - loss: 85.2272 - val_loss: 77.8712\n",
      "Epoch 86/1000\n",
      "298/298 - 0s - loss: 84.8784 - val_loss: 66.9690\n",
      "Epoch 87/1000\n",
      "298/298 - 0s - loss: 84.3240 - val_loss: 74.7963\n",
      "Epoch 88/1000\n",
      "298/298 - 0s - loss: 82.6671 - val_loss: 66.4218\n",
      "Epoch 89/1000\n",
      "298/298 - 0s - loss: 78.9253 - val_loss: 66.0465\n",
      "Epoch 90/1000\n",
      "298/298 - 0s - loss: 78.2948 - val_loss: 67.1132\n",
      "Epoch 91/1000\n",
      "298/298 - 0s - loss: 77.8454 - val_loss: 64.5615\n",
      "Epoch 92/1000\n",
      "298/298 - 0s - loss: 76.5378 - val_loss: 63.7461\n",
      "Epoch 93/1000\n",
      "298/298 - 0s - loss: 76.3956 - val_loss: 65.1476\n",
      "Epoch 94/1000\n",
      "298/298 - 0s - loss: 75.6534 - val_loss: 64.3530\n",
      "Epoch 95/1000\n",
      "298/298 - 0s - loss: 74.4515 - val_loss: 62.2992\n",
      "Epoch 96/1000\n",
      "298/298 - 0s - loss: 75.0175 - val_loss: 65.6056\n",
      "Epoch 97/1000\n",
      "298/298 - 0s - loss: 75.4062 - val_loss: 60.3103\n",
      "Epoch 98/1000\n",
      "298/298 - 0s - loss: 73.4676 - val_loss: 63.0665\n",
      "Epoch 99/1000\n",
      "298/298 - 0s - loss: 73.8604 - val_loss: 59.1017\n",
      "Epoch 100/1000\n",
      "298/298 - 0s - loss: 75.2177 - val_loss: 69.7093\n",
      "Epoch 101/1000\n",
      "298/298 - 0s - loss: 76.5957 - val_loss: 57.7874\n",
      "Epoch 102/1000\n",
      "298/298 - 0s - loss: 70.4228 - val_loss: 63.4300\n",
      "Epoch 103/1000\n",
      "298/298 - 0s - loss: 69.7966 - val_loss: 56.9887\n",
      "Epoch 104/1000\n",
      "298/298 - 0s - loss: 70.7405 - val_loss: 63.1973\n",
      "Epoch 105/1000\n",
      "298/298 - 0s - loss: 75.2547 - val_loss: 55.7992\n",
      "Epoch 106/1000\n",
      "298/298 - 0s - loss: 67.7083 - val_loss: 59.9058\n",
      "Epoch 107/1000\n",
      "298/298 - 0s - loss: 66.1408 - val_loss: 54.8671\n",
      "Epoch 108/1000\n",
      "298/298 - 0s - loss: 70.5219 - val_loss: 68.1857\n",
      "Epoch 109/1000\n",
      "298/298 - 0s - loss: 68.6804 - val_loss: 53.9880\n",
      "Epoch 110/1000\n",
      "298/298 - 0s - loss: 66.7275 - val_loss: 56.5978\n",
      "Epoch 111/1000\n",
      "298/298 - 0s - loss: 65.0595 - val_loss: 54.1060\n",
      "Epoch 112/1000\n",
      "298/298 - 0s - loss: 63.8615 - val_loss: 56.1610\n",
      "Epoch 113/1000\n",
      "298/298 - 0s - loss: 63.5641 - val_loss: 52.9413\n",
      "Epoch 114/1000\n",
      "298/298 - 0s - loss: 63.4639 - val_loss: 54.6012\n",
      "Epoch 115/1000\n",
      "298/298 - 0s - loss: 62.6168 - val_loss: 53.1920\n",
      "Epoch 116/1000\n",
      "298/298 - 0s - loss: 61.7351 - val_loss: 50.9684\n",
      "Epoch 117/1000\n",
      "298/298 - 0s - loss: 63.5669 - val_loss: 56.6385\n",
      "Epoch 118/1000\n",
      "298/298 - 0s - loss: 61.4415 - val_loss: 50.0127\n",
      "Epoch 119/1000\n",
      "298/298 - 0s - loss: 60.7425 - val_loss: 57.8498\n",
      "Epoch 120/1000\n",
      "298/298 - 0s - loss: 61.2164 - val_loss: 49.3895\n",
      "Epoch 121/1000\n",
      "298/298 - 0s - loss: 60.2546 - val_loss: 49.6976\n",
      "Epoch 122/1000\n",
      "298/298 - 0s - loss: 59.2859 - val_loss: 48.5382\n",
      "Epoch 123/1000\n",
      "298/298 - 0s - loss: 58.9762 - val_loss: 50.3346\n",
      "Epoch 124/1000\n",
      "298/298 - 0s - loss: 58.9458 - val_loss: 47.6183\n",
      "Epoch 125/1000\n",
      "298/298 - 0s - loss: 57.7480 - val_loss: 50.9688\n",
      "Epoch 126/1000\n",
      "298/298 - 0s - loss: 57.2021 - val_loss: 48.4334\n",
      "Epoch 127/1000\n",
      "298/298 - 0s - loss: 56.0933 - val_loss: 48.6423\n",
      "Epoch 128/1000\n",
      "298/298 - 0s - loss: 56.0916 - val_loss: 46.6327\n",
      "Epoch 129/1000\n",
      "298/298 - 0s - loss: 55.1348 - val_loss: 47.3380\n",
      "Epoch 130/1000\n",
      "298/298 - 0s - loss: 54.5392 - val_loss: 45.5798\n",
      "Epoch 131/1000\n",
      "298/298 - 0s - loss: 55.0560 - val_loss: 46.4440\n",
      "Epoch 132/1000\n",
      "298/298 - 0s - loss: 55.1137 - val_loss: 50.3307\n",
      "Epoch 133/1000\n",
      "298/298 - 0s - loss: 54.9968 - val_loss: 45.0395\n",
      "Epoch 134/1000\n",
      "298/298 - 0s - loss: 54.1360 - val_loss: 48.7353\n",
      "Epoch 135/1000\n",
      "298/298 - 0s - loss: 52.8437 - val_loss: 43.5325\n",
      "Epoch 136/1000\n",
      "298/298 - 0s - loss: 51.9040 - val_loss: 50.6398\n",
      "Epoch 137/1000\n",
      "298/298 - 0s - loss: 52.5419 - val_loss: 43.6542\n",
      "Epoch 138/1000\n",
      "298/298 - 0s - loss: 51.0480 - val_loss: 44.7285\n",
      "Epoch 139/1000\n",
      "298/298 - 0s - loss: 51.3169 - val_loss: 44.8340\n",
      "Epoch 140/1000\n",
      "298/298 - 0s - loss: 50.1141 - val_loss: 41.7541\n",
      "Epoch 141/1000\n",
      "298/298 - 0s - loss: 50.2745 - val_loss: 42.6962\n",
      "Epoch 142/1000\n",
      "298/298 - 0s - loss: 49.2871 - val_loss: 43.4447\n",
      "Epoch 143/1000\n",
      "298/298 - 0s - loss: 49.1571 - val_loss: 41.6067\n",
      "Epoch 144/1000\n",
      "298/298 - 0s - loss: 48.6497 - val_loss: 41.7728\n",
      "Epoch 145/1000\n",
      "298/298 - 0s - loss: 47.9751 - val_loss: 41.4525\n",
      "Epoch 146/1000\n",
      "298/298 - 0s - loss: 47.7075 - val_loss: 39.7075\n",
      "Epoch 147/1000\n",
      "298/298 - 0s - loss: 47.1009 - val_loss: 42.2388\n",
      "Epoch 148/1000\n",
      "298/298 - 0s - loss: 46.8240 - val_loss: 39.7196\n",
      "Epoch 149/1000\n",
      "298/298 - 0s - loss: 46.4609 - val_loss: 42.2989\n",
      "Epoch 150/1000\n",
      "298/298 - 0s - loss: 48.0583 - val_loss: 38.4602\n",
      "Epoch 151/1000\n",
      "298/298 - 0s - loss: 46.5334 - val_loss: 43.8147\n",
      "Epoch 152/1000\n",
      "298/298 - 0s - loss: 46.7716 - val_loss: 40.3131\n",
      "Epoch 153/1000\n",
      "298/298 - 0s - loss: 47.0893 - val_loss: 37.4092\n",
      "Epoch 154/1000\n",
      "298/298 - 0s - loss: 44.4238 - val_loss: 38.6971\n",
      "Epoch 155/1000\n",
      "298/298 - 0s - loss: 44.1358 - val_loss: 36.7729\n",
      "Epoch 156/1000\n",
      "298/298 - 0s - loss: 43.6989 - val_loss: 39.5008\n",
      "Epoch 157/1000\n",
      "298/298 - 0s - loss: 43.0400 - val_loss: 36.1819\n",
      "Epoch 158/1000\n",
      "298/298 - 0s - loss: 44.1855 - val_loss: 36.5305\n",
      "Epoch 159/1000\n",
      "298/298 - 0s - loss: 43.9221 - val_loss: 38.8542\n",
      "Epoch 160/1000\n",
      "298/298 - 0s - loss: 42.5634 - val_loss: 35.4242\n",
      "Epoch 161/1000\n",
      "298/298 - 0s - loss: 41.8758 - val_loss: 37.1866\n",
      "Epoch 162/1000\n",
      "298/298 - 0s - loss: 41.5447 - val_loss: 35.2277\n",
      "Epoch 163/1000\n",
      "298/298 - 0s - loss: 41.2064 - val_loss: 34.6725\n",
      "Epoch 164/1000\n",
      "298/298 - 0s - loss: 40.9822 - val_loss: 35.0514\n",
      "Epoch 165/1000\n",
      "298/298 - 0s - loss: 40.1962 - val_loss: 34.8705\n",
      "Epoch 166/1000\n",
      "298/298 - 0s - loss: 39.8260 - val_loss: 33.5002\n",
      "Epoch 167/1000\n",
      "298/298 - 0s - loss: 40.1474 - val_loss: 35.9322\n",
      "Epoch 168/1000\n",
      "298/298 - 0s - loss: 39.7041 - val_loss: 33.0713\n",
      "Epoch 169/1000\n",
      "298/298 - 0s - loss: 39.3530 - val_loss: 32.5961\n",
      "Epoch 170/1000\n",
      "298/298 - 0s - loss: 39.3592 - val_loss: 34.3898\n",
      "Epoch 171/1000\n",
      "298/298 - 0s - loss: 39.9318 - val_loss: 37.0751\n",
      "Epoch 172/1000\n",
      "298/298 - 0s - loss: 39.9988 - val_loss: 31.7385\n",
      "Epoch 173/1000\n",
      "298/298 - 0s - loss: 38.5538 - val_loss: 31.9029\n",
      "Epoch 174/1000\n",
      "298/298 - 0s - loss: 39.0118 - val_loss: 33.9289\n",
      "Epoch 175/1000\n",
      "298/298 - 0s - loss: 38.1769 - val_loss: 34.0615\n",
      "Epoch 176/1000\n",
      "298/298 - 0s - loss: 37.7205 - val_loss: 31.3360\n",
      "Epoch 177/1000\n",
      "298/298 - 0s - loss: 38.3103 - val_loss: 30.4050\n",
      "Epoch 178/1000\n",
      "298/298 - 0s - loss: 37.4394 - val_loss: 34.8860\n",
      "Epoch 179/1000\n",
      "298/298 - 0s - loss: 36.0916 - val_loss: 30.1852\n",
      "Epoch 180/1000\n",
      "298/298 - 0s - loss: 35.0402 - val_loss: 29.9700\n",
      "Epoch 181/1000\n",
      "298/298 - 0s - loss: 35.0743 - val_loss: 30.8582\n",
      "Epoch 182/1000\n",
      "298/298 - 0s - loss: 34.9778 - val_loss: 29.4909\n",
      "Epoch 183/1000\n",
      "298/298 - 0s - loss: 35.5301 - val_loss: 32.6701\n",
      "Epoch 184/1000\n",
      "298/298 - 0s - loss: 35.1328 - val_loss: 31.4080\n",
      "Epoch 185/1000\n",
      "298/298 - 0s - loss: 34.8071 - val_loss: 28.7977\n",
      "Epoch 186/1000\n",
      "298/298 - 0s - loss: 33.5718 - val_loss: 29.4352\n",
      "Epoch 187/1000\n",
      "298/298 - 0s - loss: 32.8953 - val_loss: 27.9879\n",
      "Epoch 188/1000\n",
      "298/298 - 0s - loss: 33.0373 - val_loss: 32.1209\n",
      "Epoch 189/1000\n",
      "298/298 - 0s - loss: 32.7321 - val_loss: 27.8982\n",
      "Epoch 190/1000\n",
      "298/298 - 0s - loss: 32.6381 - val_loss: 27.3850\n",
      "Epoch 191/1000\n",
      "298/298 - 0s - loss: 32.2747 - val_loss: 27.2659\n",
      "Epoch 192/1000\n",
      "298/298 - 0s - loss: 32.4851 - val_loss: 27.0773\n",
      "Epoch 193/1000\n",
      "298/298 - 0s - loss: 31.9399 - val_loss: 26.8742\n",
      "Epoch 194/1000\n",
      "298/298 - 0s - loss: 31.4582 - val_loss: 30.4675\n",
      "Epoch 195/1000\n",
      "298/298 - 0s - loss: 31.7919 - val_loss: 26.1536\n",
      "Epoch 196/1000\n",
      "298/298 - 0s - loss: 30.5031 - val_loss: 27.7287\n",
      "Epoch 197/1000\n",
      "298/298 - 0s - loss: 30.5504 - val_loss: 26.4488\n",
      "Epoch 198/1000\n",
      "298/298 - 0s - loss: 29.9834 - val_loss: 25.7465\n",
      "Epoch 199/1000\n",
      "298/298 - 0s - loss: 31.6555 - val_loss: 26.2356\n",
      "Epoch 200/1000\n",
      "298/298 - 0s - loss: 29.6036 - val_loss: 24.8315\n",
      "Epoch 201/1000\n",
      "298/298 - 0s - loss: 29.1212 - val_loss: 24.8808\n",
      "Epoch 202/1000\n",
      "298/298 - 0s - loss: 29.4223 - val_loss: 24.4004\n",
      "Epoch 203/1000\n",
      "298/298 - 0s - loss: 30.4341 - val_loss: 26.0114\n",
      "Epoch 204/1000\n",
      "298/298 - 0s - loss: 29.7155 - val_loss: 30.6579\n",
      "Epoch 205/1000\n",
      "298/298 - 0s - loss: 30.2008 - val_loss: 25.4360\n",
      "Epoch 206/1000\n",
      "298/298 - 0s - loss: 28.6162 - val_loss: 25.2825\n",
      "Epoch 207/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "298/298 - 0s - loss: 28.0181 - val_loss: 25.5654\n",
      "Epoch 00207: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1a34b40b00>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "import pandas as pd\n",
    "import io\n",
    "import os\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "cars = df['name']\n",
    "\n",
    "# Handle missing value\n",
    "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
    "\n",
    "# Pandas to Numpy\n",
    "x = df[['cylinders', 'displacement', 'horsepower', 'weight',\n",
    "       'acceleration', 'year', 'origin']].values\n",
    "y = df['mpg'].values # regression\n",
    "\n",
    "# Split into validation and training sets\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Build the neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto',\n",
    "        restore_best_weights=True)\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 4.939672608054\n"
     ]
    }
   ],
   "source": [
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "pred = model.predict(x_test)\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(f\"Final score (RMSE): {score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3.5: Extracting Keras Weights and Manual Neural Network Calculation\n",
    "\n",
    "In this section we will build a neural network and analyze it down the individual weights.  We will train a simple neural network that learns the XOR function.  It is not hard to simply hand-code the neurons to provide an [XOR function](https://en.wikipedia.org/wiki/Exclusive_or); however, for simplicity, we will allow Keras to train this network for us.  We will just use 100K epochs on the ADAM optimizer.  This is massive overkill, but it gets the result, and our focus here is not on tuning.  The neural network is small.  Two inputs, two hidden neurons, and a single output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "import numpy as np\n",
    "\n",
    "# Create a dataset for the XOR function\n",
    "x = np.array([\n",
    "    [0,0],\n",
    "    [1,0],\n",
    "    [0,1],\n",
    "    [1,1]\n",
    "])\n",
    "\n",
    "y = np.array([\n",
    "    0,\n",
    "    1,\n",
    "    1,\n",
    "    0\n",
    "])\n",
    "\n",
    "# Build the network\n",
    "# sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)\n",
    "\n",
    "done = False\n",
    "cycle = 1\n",
    "\n",
    "while not done:\n",
    "    print(\"Cycle #{}\".format(cycle))\n",
    "    cycle+=1\n",
    "    model = Sequential()\n",
    "    model.add(Dense(2, input_dim=2, activation='relu')) \n",
    "    model.add(Dense(1)) \n",
    "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "    model.fit(x,y,verbose=0,epochs=10000)\n",
    "\n",
    "    # Predict\n",
    "    pred = model.predict(x)\n",
    "    \n",
    "    # Check if successful.  It takes several runs with this small of a network\n",
    "    done = pred[0]<0.01 and pred[3]<0.01 and pred[1] > 0.9 and pred[2] > 0.9 \n",
    "    print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output above should have two numbers near 0.0 for the first and forth spots (input [[0,0]] and [[1,1]]).  The middle two numbers should be near 1.0 (input [[1,0]] and [[0,1]]).  These numbers are in scientific notation.  Due to random starting weights, it is sometimes necessary to run the above through several cycles to get a good result.\n",
    "\n",
    "Now that the neural network is trained, lets dump the weights.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dump weights\n",
    "for layerNum, layer in enumerate(model.layers):\n",
    "    weights = layer.get_weights()[0]\n",
    "    biases = layer.get_weights()[1]\n",
    "    \n",
    "    for toNeuronNum, bias in enumerate(biases):\n",
    "        print(f'{layerNum}B -> L{layerNum+1}N{toNeuronNum}: {bias}')\n",
    "    \n",
    "    for fromNeuronNum, wgt in enumerate(weights):\n",
    "        for toNeuronNum, wgt2 in enumerate(wgt):\n",
    "            print(f'L{layerNum}N{fromNeuronNum} -> L{layerNum+1}N{toNeuronNum} = {wgt2}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you rerun this, you probably get different weights.  There are many ways to solve the XOR function.\n",
    "\n",
    "In the next section, we copy/paste the weights from above and recreate the calculations done by the neural network.  Because weights can change with each training, the weights used for the below code came from this:\n",
    "\n",
    "```\n",
    "0B -> L1N0: -1.2913415431976318\n",
    "0B -> L1N1: -3.021530048386012e-08\n",
    "L0N0 -> L1N0 = 1.2913416624069214\n",
    "L0N0 -> L1N1 = 1.1912699937820435\n",
    "L0N1 -> L1N0 = 1.2913411855697632\n",
    "L0N1 -> L1N1 = 1.1912697553634644\n",
    "1B -> L2N0: 7.626241297587034e-36\n",
    "L1N0 -> L2N0 = -1.548777461051941\n",
    "L1N1 -> L2N0 = 0.8394404649734497\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input0 = 0\n",
    "input1 = 1\n",
    "\n",
    "hidden0Sum = (input0*1.3)+(input1*1.3)+(-1.3)\n",
    "hidden1Sum = (input0*1.2)+(input1*1.2)+(0)\n",
    "\n",
    "print(hidden0Sum) # 0\n",
    "print(hidden1Sum) # 1.2\n",
    "\n",
    "hidden0 = max(0,hidden0Sum)\n",
    "hidden1 = max(0,hidden1Sum)\n",
    "\n",
    "print(hidden0) # 0\n",
    "print(hidden1) # 1.2\n",
    "\n",
    "outputSum = (hidden0*-1.6)+(hidden1*0.8)+(0)\n",
    "print(outputSum) # 0.96\n",
    "\n",
    "output = max(0,outputSum)\n",
    "\n",
    "print(output) # 0.96"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 3 Assignment\n",
    "\n",
    "You can find the first assignment here: [assignment 3](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class3.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow-2.0)",
   "language": "python",
   "name": "tensorflow-2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}