conv_net.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Definition and training of the ConvNet computational graph\n",
    "\n",
    "## Matching the models' \"expressivities\"\n",
    "\n",
    "The idea of ConvNet is to use a more common structure while keeping approximately the same expressivity as SpotNet. In this way, the improvement in performance of SpotNet over ConvNet can really be attributed to its beneficial structure. To achieve this goal, the total number of learnable parameters should be approximately the same in both models. The following\n",
    "\n",
    "| Model | # learnable $5\\times 5$ conv. kernels | # scalar parameters |\n",
    "| --    | --                                      | --                  |\n",
    "| SpotNet | 210 | 8 |\n",
    "| ConvNet | 216 | 6 |\n",
    "|<td colspan=3> <center> Tab. 1: Comparison of the number of learnable parameters in SpotNet and ConvNet. Separated into the number of learnable $5\\times5$ convolutional kernels and the number of learnable scalar parameters. For details on how these numbers were obtained, see the following sections. </center> |\n",
    "\n",
    "### Analysis of SpotNet in terms of number of parameters\n",
    "\n",
    "SpotNet has $30$ convolutions in its input half-layer, and $60$ convolutions in each hidden layer (of which there are $3$). This amounts to a total of $30+60\\times 3=\\mathbf{210}$ two-dimensional convolutions with $5\\times 5$ learnable kernels. Besides the learnable kernels, SpotNet has $1$ scalar parameter and $1$ one-dimensional bias parameter per each of its $3$ layers, and $1$ more of each for its input half-layer, amounting to a total of $4\\times 2 = \\mathbf{8}$ scalar paremeters.\n",
    "\n",
    "<a id=\"design\"></a>\n",
    "### Design of ConvNet\n",
    "\n",
    "A full convolutional layer with $N$ input channels, $M$ output channels, and $5 \\times 5$ kernels implies $N \\times M$ two-dimensional convolutions with such kernels. Similarly, a convolutional layer that turns each of the $N$ input channels into $K$ contains $N\\times K$ two-dimensional convolutions. Furthermore, a feature-wise convolutional layer involves only $1$ two-dimensional convolution per feature. Thereby, the structure we propose for ConvNet, i.e., \n",
    "\n",
    "1. $2$ full convolutional layers to turn the single input into $6$ features and, succesively, the $6$ features into $15$, \n",
    "2. $1$ layer that turns each feature into two, and \n",
    "3. $3$ layers with feature-wise convolutions,   \n",
    "\n",
    "results in a grand total of $1\\times  6 + 6 \\times 15 + 15 \\times 2 + 30 + 30 +30 = \\mathbf{216}$ two-dimensional convolutions with $5\\times 5$ learnable kernels. Furthermore, this structure we propose for ConvNet has $1$ one-dimensional bias term for each layer, and a total of $6$ layers, amounting ot $1\\times 6 = \\mathbf{6}$ scalar parameters. \n",
    "\n",
    "\n",
    "# Index\n",
    "1. Definition of computational graph of ConvNet\n",
    "    1. [Importing libraries](#libs) to build and initialize the network\n",
    "    2. Definition of the [full ConvNet computational graph](#convnet)\n",
    "    6. Definition of the [training strategy](#training_strategy) for ConvNet\n",
    "2. Training of ConvNet (not recommended for non-GPU systems)\n",
    "    1. [Loading of the training data-base](#data) and setting of parameters \n",
    "    2. Construction of the [computational graph](#session)\n",
    "    3. Running the [training](#training) and storing its results\n",
    "\n",
    "# Definition of computational graph of ConvNet\n",
    "\n",
    "<a id=\"libs\"></a>\n",
    "## Importing libraries to build and initialize the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"convnet\"></a>\n",
    "## Definition of the full ConvNet computational graph\n",
    "\n",
    "To compare with the [Design of ConvNet](#design) section above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def conv_net( image_height = 512, image_width = 512 ):\n",
    "    \n",
    "    # Seed for reproducibility\n",
    "    tf.set_random_seed( 8888 )\n",
    "    \n",
    "    # Input placeholder\n",
    "    source = tf.placeholder( tf.float32, [ 1, image_height, image_width ] )\n",
    "    \n",
    "    # First convolutional layer, from source to 6 channels\n",
    "    h1 = tf.Variable( tf.truncated_normal( [5, 5, 1, 6], stddev = 0.01 ) )\n",
    "    x1 = tf.nn.conv2d( tf.expand_dims( source, axis = 3 ), h1, strides = [1, 1, 1, 1], padding = 'SAME' )\n",
    "    b1 = tf.Variable( 1.0 )\n",
    "    x1 = tf.nn.relu(  x1 + b1 )\n",
    "    \n",
    "    # Second convolutional layer, from 6 to 15 channels\n",
    "    h2 = tf.Variable( tf.truncated_normal( [5, 5, 6, 15], stddev = 0.01 ) )\n",
    "    x2 = tf.nn.conv2d( x1, h2, strides = [1, 1, 1, 1], padding = 'SAME' )\n",
    "    b2 = tf.Variable( 1.0 )\n",
    "    x2 = tf.nn.relu( x2 + b2 )\n",
    "    \n",
    "    # Third convolutional layer, each of the 15 channels splits in 2\n",
    "    h3 = tf.Variable( tf.truncated_normal( [5, 5, 15, 2], stddev = 0.01 ) )\n",
    "    x3 = tf.nn.depthwise_conv2d( x2, h3, strides = [1, 1, 1, 1], padding = 'SAME' )\n",
    "    b3 = tf.Variable( 1.0 )\n",
    "    x3 = tf.nn.relu( x3 + b3  )\n",
    "    \n",
    "    x = [x3]; b = []; h = [];\n",
    "    # Three layers in which each channel is treated independently, i.e. from 30 channels to 30 channels\n",
    "    for layer_number in range(3):\n",
    "        h_next = tf.Variable( tf.truncated_normal( [5, 5, 30, 1], stddev = 0.01 ) )\n",
    "        x_next = tf.nn.depthwise_conv2d( x[-1], h_next, strides = [1, 1, 1, 1], padding = 'SAME' )\n",
    "        b_next = tf.Variable( 1.0 )\n",
    "        x_next = tf.nn.relu( x_next + b_next )\n",
    "        x.append( x_next )\n",
    "        b.append( b_next )\n",
    "        h.append( h_next )    \n",
    "    \n",
    "    # Output is the result at the end of the last layer\n",
    "    output = x[-1]\n",
    "    \n",
    "    # Placeholder for the target\n",
    "    target = tf.placeholder( tf.float32, [ 1, image_height, image_width, 30 ] )\n",
    "    \n",
    "    # MSE loss function\n",
    "    loss = tf.reduce_mean( ( output - target ) ** 2 )\n",
    "    \n",
    "    return (source, loss, target, output)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"training_strategy\"></a>\n",
    "## Definition of the training strategy for SpotNet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create and return the optimizer\n",
    "def network_training( loss, learning_rate ):\n",
    "    return tf.train.AdamOptimizer( learning_rate ).minimize( loss )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"training_sec\"></a>\n",
    "# Training of SpotNet\n",
    "\n",
    "<a id=\"data\"></a>\n",
    "## Loading of the training data-base and setting of parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset: Num images: 10, Image height: 512, Image width: 512, Num cells: 1250, Num kernels: 30\n"
     ]
    }
   ],
   "source": [
    "# Learning process parameters\n",
    "batch_size = 1\n",
    "learning_rate = 1e-3\n",
    "nrof_train_steps = 400000\n",
    "nrof_cells = 1250\n",
    "\n",
    "# Relevant directories\n",
    "results_dir = 'results/'\n",
    "data_dir = 'sim_data/'\n",
    "\n",
    "# Load dataset\n",
    "data = np.load( data_dir + 'result_' + str( nrof_cells ) + '_cells_10_images.npy' )[()]\n",
    "\n",
    "# Extract images and xs\n",
    "images = data['fluorospot']\n",
    "psdrs = data['psdrs']\n",
    "\n",
    "# Extract shape parameters\n",
    "nrof_images, image_height, image_width = images.shape\n",
    "_, _, _, nrof_kernels = psdrs.shape\n",
    "\n",
    "# Split dataset for training and validation\n",
    "nrof_training_samples = int( 0.7 * nrof_images )\n",
    "train_images, train_psdrs = (images[ : nrof_training_samples, ... ], psdrs[ : nrof_training_samples, ... ])\n",
    "val_images, val_psdrs = (images[nrof_training_samples : , ... ], psdrs[nrof_training_samples : , ... ])\n",
    "\n",
    "# Inform user\n",
    "print( 'Dataset: Num images: %d, Image height: %d, Image width: %d, Num cells: %d, Num kernels: %d'%( nrof_images, \n",
    "                                                                                                      image_height, \n",
    "                                                                                                      image_width,\n",
    "                                                                                                      nrof_cells,\n",
    "                                                                                                      nrof_kernels ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"session\"></a>\n",
    "## Construction of the computational graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build computational graph\n",
    "with tf.name_scope( 'ConvNet' ) as scope:\n",
    "    source, loss, target, output = conv_net( image_height, image_width )\n",
    "\n",
    "# Build training computational graph\n",
    "with tf.name_scope( 'Training' ) as scope:\n",
    "    train_step = network_training( loss, learning_rate )\n",
    "    \n",
    "# Create TensorFlow session to run the computational graph\n",
    "nrof_gpu = 1\n",
    "config = tf.ConfigProto( device_count = {'GPU': nrof_gpu} )\n",
    "sess = tf.Session( config = config )\n",
    "sess.run( tf.global_variables_initializer( ) )\n",
    "print( \"An error will be issued here if no GPU is present in the system. This is intentional.\\n\" + \n",
    "       \"Training is not recommended for non-GPU systems. Set 'nrof_gpus = 0' above if you still want to proceed.\" )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"training\"></a>\n",
    "## Running training and storing its results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training the network\n",
      "Train step 0, Batch loss: 2321.3057, Test loss: 1590.5524, Elapsed: 3s.\n",
      "Train step 4000, Batch loss: 1452.4852, Test loss: 1426.8415, Elapsed: 1964s. Best in test yet! Storing.\n",
      "WARNING:tensorflow:From /home/polgpu/tensorflow-new/lib/python3.5/site-packages/tensorflow/python/saved_model/simple_save.py:85: calling SavedModelBuilder.add_meta_graph_and_variables (from tensorflow.python.saved_model.builder_impl) with legacy_init_op is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Pass your op to the equivalent parameter main_op instead.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_4000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 8000, Batch loss: 1881.9786, Test loss: 1394.3916, Elapsed: 3909s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_8000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 12000, Batch loss: 1413.0874, Test loss: 1388.3988, Elapsed: 5853s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_12000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 16000, Batch loss: 1872.7573, Test loss: 1385.8140, Elapsed: 7810s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_16000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 20000, Batch loss: 1272.5352, Test loss: 1382.8471, Elapsed: 9693s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_20000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 24000, Batch loss: 1489.3918, Test loss: 1381.1123, Elapsed: 11654s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_24000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 28000, Batch loss: 1863.9000, Test loss: 1380.4369, Elapsed: 13624s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_28000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 32000, Batch loss: 1489.6536, Test loss: 1380.3583, Elapsed: 15501s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_32000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 36000, Batch loss: 1866.9066, Test loss: 1387.4642, Elapsed: 17470s.\n",
      "Train step 40000, Batch loss: 1270.8966, Test loss: 1378.5295, Elapsed: 19444s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_40000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 44000, Batch loss: 1493.2151, Test loss: 1386.9553, Elapsed: 21321s.\n",
      "Train step 48000, Batch loss: 1989.5453, Test loss: 1379.9991, Elapsed: 23198s.\n",
      "Train step 52000, Batch loss: 1655.5103, Test loss: 1380.7638, Elapsed: 25076s.\n",
      "Train step 56000, Batch loss: 1317.4935, Test loss: 1293.2635, Elapsed: 26954s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_56000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 60000, Batch loss: 1527.6094, Test loss: 1278.1179, Elapsed: 28832s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_60000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 64000, Batch loss: 1171.3572, Test loss: 1277.6740, Elapsed: 30796s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_64000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 68000, Batch loss: 1598.8911, Test loss: 1133.9457, Elapsed: 32744s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_68000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 72000, Batch loss: 1376.3573, Test loss: 1040.7609, Elapsed: 34719s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_72000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 76000, Batch loss: 1103.0494, Test loss: 1031.8590, Elapsed: 36692s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_76000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 80000, Batch loss: 1358.0425, Test loss: 1027.0955, Elapsed: 38629s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_80000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 84000, Batch loss: 1464.6305, Test loss: 1060.4536, Elapsed: 40508s.\n",
      "Train step 88000, Batch loss: 1131.7362, Test loss: 1026.5550, Elapsed: 42388s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_88000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 92000, Batch loss: 1427.6304, Test loss: 1028.4776, Elapsed: 44351s.\n",
      "Train step 96000, Batch loss: 1314.9944, Test loss: 1028.8957, Elapsed: 46323s.\n",
      "Train step 100000, Batch loss: 1067.6622, Test loss: 1008.2988, Elapsed: 48296s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_100000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 104000, Batch loss: 1118.1652, Test loss: 1009.2066, Elapsed: 50138s.\n",
      "Train step 108000, Batch loss: 1009.1857, Test loss: 1004.2726, Elapsed: 52007s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_108000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 112000, Batch loss: 1325.9093, Test loss: 1003.9886, Elapsed: 53873s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_112000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 116000, Batch loss: 1018.2363, Test loss: 1006.2570, Elapsed: 55770s.\n",
      "Train step 120000, Batch loss: 1408.2792, Test loss: 998.2184, Elapsed: 57671s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_120000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 124000, Batch loss: 1296.8856, Test loss: 1000.8432, Elapsed: 59640s.\n",
      "Train step 128000, Batch loss: 1169.4192, Test loss: 1018.6312, Elapsed: 61613s.\n",
      "Train step 132000, Batch loss: 1407.1370, Test loss: 997.6457, Elapsed: 63583s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_132000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 136000, Batch loss: 1057.1263, Test loss: 995.3231, Elapsed: 65499s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_136000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 140000, Batch loss: 1105.0287, Test loss: 992.9384, Elapsed: 67459s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_140000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 144000, Batch loss: 1390.3242, Test loss: 1002.9410, Elapsed: 69421s.\n",
      "Train step 148000, Batch loss: 1041.9705, Test loss: 992.8294, Elapsed: 71390s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_148000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 152000, Batch loss: 1095.2675, Test loss: 985.7089, Elapsed: 73346s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_152000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 156000, Batch loss: 1004.6012, Test loss: 994.3376, Elapsed: 75319s.\n",
      "Train step 160000, Batch loss: 1284.1036, Test loss: 990.7282, Elapsed: 77294s.\n",
      "Train step 164000, Batch loss: 985.5270, Test loss: 983.2510, Elapsed: 79267s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_164000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 168000, Batch loss: 1064.0646, Test loss: 1011.4656, Elapsed: 81183s.\n",
      "Train step 172000, Batch loss: 990.3528, Test loss: 984.8436, Elapsed: 83106s.\n",
      "Train step 176000, Batch loss: 920.4621, Test loss: 996.3977, Elapsed: 85026s.\n",
      "Train step 180000, Batch loss: 1038.8396, Test loss: 990.3020, Elapsed: 86952s.\n",
      "Train step 184000, Batch loss: 1377.0148, Test loss: 1004.9018, Elapsed: 88863s.\n",
      "Train step 188000, Batch loss: 916.6488, Test loss: 995.6585, Elapsed: 90789s.\n",
      "Train step 192000, Batch loss: 1087.4929, Test loss: 999.9535, Elapsed: 92714s.\n",
      "Train step 196000, Batch loss: 1384.0037, Test loss: 983.0239, Elapsed: 94638s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_196000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 200000, Batch loss: 1407.0045, Test loss: 985.2688, Elapsed: 96554s.\n",
      "Train step 204000, Batch loss: 1084.9614, Test loss: 982.3210, Elapsed: 98471s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_204000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 208000, Batch loss: 1136.0950, Test loss: 985.4192, Elapsed: 100434s.\n",
      "Train step 212000, Batch loss: 1084.1869, Test loss: 990.2749, Elapsed: 102411s.\n",
      "Train step 216000, Batch loss: 1034.7827, Test loss: 988.3309, Elapsed: 104387s.\n",
      "Train step 220000, Batch loss: 1041.6223, Test loss: 986.6673, Elapsed: 106365s.\n",
      "Train step 224000, Batch loss: 1268.0579, Test loss: 990.6945, Elapsed: 108342s.\n",
      "Train step 228000, Batch loss: 1139.8883, Test loss: 1013.7863, Elapsed: 110320s.\n",
      "Train step 232000, Batch loss: 1028.5054, Test loss: 987.0336, Elapsed: 112297s.\n",
      "Train step 236000, Batch loss: 1368.1620, Test loss: 987.1572, Elapsed: 114275s.\n",
      "Train step 240000, Batch loss: 1139.2946, Test loss: 1007.5775, Elapsed: 116252s.\n",
      "Train step 244000, Batch loss: 1402.3790, Test loss: 993.8728, Elapsed: 118231s.\n",
      "Train step 248000, Batch loss: 980.7451, Test loss: 983.8595, Elapsed: 120205s.\n",
      "Train step 252000, Batch loss: 1035.1549, Test loss: 985.4783, Elapsed: 122182s.\n",
      "Train step 256000, Batch loss: 1410.1461, Test loss: 978.2589, Elapsed: 124161s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_256000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 260000, Batch loss: 1371.7910, Test loss: 981.6657, Elapsed: 126017s.\n",
      "Train step 264000, Batch loss: 984.6527, Test loss: 985.0247, Elapsed: 127872s.\n",
      "Train step 268000, Batch loss: 1392.6896, Test loss: 991.5236, Elapsed: 129745s.\n",
      "Train step 272000, Batch loss: 1269.2466, Test loss: 1007.3300, Elapsed: 131612s.\n",
      "Train step 276000, Batch loss: 988.5262, Test loss: 992.6923, Elapsed: 133479s.\n",
      "Train step 280000, Batch loss: 1083.4633, Test loss: 977.7537, Elapsed: 135357s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_280000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 284000, Batch loss: 1087.4559, Test loss: 981.3038, Elapsed: 137319s.\n",
      "Train step 288000, Batch loss: 1138.2474, Test loss: 975.7980, Elapsed: 139293s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_288000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 292000, Batch loss: 1035.1000, Test loss: 990.1171, Elapsed: 141149s.\n",
      "Train step 296000, Batch loss: 900.7518, Test loss: 983.4026, Elapsed: 143023s.\n",
      "Train step 300000, Batch loss: 1051.7599, Test loss: 1000.1262, Elapsed: 144900s.\n",
      "Train step 304000, Batch loss: 932.1952, Test loss: 1028.9119, Elapsed: 146775s.\n",
      "Train step 308000, Batch loss: 891.1969, Test loss: 980.9376, Elapsed: 148653s.\n",
      "Train step 312000, Batch loss: 1363.0844, Test loss: 985.6753, Elapsed: 150531s.\n",
      "Train step 316000, Batch loss: 1275.8470, Test loss: 989.2797, Elapsed: 152409s.\n",
      "Train step 320000, Batch loss: 1388.2933, Test loss: 981.6227, Elapsed: 154287s.\n",
      "Train step 324000, Batch loss: 1273.9095, Test loss: 989.6844, Elapsed: 156166s.\n",
      "Train step 328000, Batch loss: 1092.0314, Test loss: 979.0125, Elapsed: 158045s.\n",
      "Train step 332000, Batch loss: 1363.8644, Test loss: 994.3871, Elapsed: 159922s.\n",
      "Train step 336000, Batch loss: 1023.0265, Test loss: 976.5054, Elapsed: 161801s.\n",
      "Train step 340000, Batch loss: 908.9329, Test loss: 1006.0644, Elapsed: 163680s.\n",
      "Train step 344000, Batch loss: 1141.1687, Test loss: 978.1420, Elapsed: 165559s.\n",
      "Train step 348000, Batch loss: 1157.1334, Test loss: 980.4649, Elapsed: 167437s.\n",
      "Train step 352000, Batch loss: 971.8505, Test loss: 975.1649, Elapsed: 169317s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_352000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 356000, Batch loss: 1030.1417, Test loss: 984.6429, Elapsed: 171291s.\n",
      "Train step 360000, Batch loss: 895.2697, Test loss: 978.1816, Elapsed: 173265s.\n",
      "Train step 364000, Batch loss: 1376.6743, Test loss: 979.2205, Elapsed: 175240s.\n",
      "Train step 368000, Batch loss: 1261.9673, Test loss: 989.5524, Elapsed: 177216s.\n",
      "Train step 372000, Batch loss: 1080.3176, Test loss: 980.1757, Elapsed: 179190s.\n",
      "Train step 376000, Batch loss: 1281.9993, Test loss: 980.3916, Elapsed: 181163s.\n",
      "Train step 380000, Batch loss: 1371.2188, Test loss: 976.1876, Elapsed: 183137s.\n",
      "Train step 384000, Batch loss: 921.4305, Test loss: 1006.7783, Elapsed: 185112s.\n",
      "Train step 388000, Batch loss: 1265.0707, Test loss: 984.6764, Elapsed: 187088s.\n",
      "Train step 392000, Batch loss: 1138.5031, Test loss: 974.7727, Elapsed: 189063s. Best in test yet! Storing.\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: spotnet_results/trained_convnet_392000_train-steps_5_kersize/saved_model.pb\n",
      "Train step 396000, Batch loss: 976.4579, Test loss: 975.4981, Elapsed: 191039s.\n",
      "Train step 400000, Batch loss: 1270.8306, Test loss: 978.7233, Elapsed: 193010s.\n"
     ]
    }
   ],
   "source": [
    "# Set TensorFlow logging level to non-debug (only WARN and ERROR log messages)\n",
    "tf.logging.set_verbosity( tf.logging.WARN )\n",
    "\n",
    "# Time library to time training\n",
    "import time\n",
    "\n",
    "# Inform user\n",
    "print( 'Training the network' )\n",
    "\n",
    "# Space to store loss values\n",
    "train_loss_record = np.empty( (0, ) )\n",
    "val_loss_record  = np.empty( (0, ) )\n",
    "iterations_record = np.empty( (0, ) )\n",
    "\n",
    "# Indices that will be picked at random at each iteration\n",
    "indices = np.arange( train_images.shape[0] )\n",
    "\n",
    "# Seed for reproducibility\n",
    "np.random.seed( 8888 )\n",
    "\n",
    "# Train the network\n",
    "start = time.time( )\n",
    "for training_iteration in range( nrof_train_steps + 1 ):\n",
    "    # Shuffle indices\n",
    "    np.random.shuffle( indices )\n",
    "    # Extract current batch\n",
    "    batch_indices = np.take( indices, np.arange( batch_size ), mode = 'wrap' )\n",
    "    batch_input = train_images[ batch_indices, ... ]\n",
    "    batch_target = train_psdrs[ batch_indices, ... ]\n",
    "    # Run a training step\n",
    "    _, train_loss = sess.run( [ train_step, loss ], feed_dict = { source: batch_input, \n",
    "                                                                  target: batch_target } )\n",
    "    \n",
    "    # Every 100th of total iterations, record progress in terms of validation loss\n",
    "    if training_iteration % (nrof_train_steps / 100) == 0:\n",
    "        # Sum validation losses\n",
    "        val_loss = 0\n",
    "        for val_image_index in range( val_images.shape[0] ):\n",
    "            val_loss += sess.run( loss, feed_dict = { source: np.expand_dims( val_images[ val_image_index, ... ], axis = 0 ), \n",
    "                                                       target: np.expand_dims( val_psdrs[  val_image_index, ... ], axis = 0 ) } )\n",
    "        # Divide by number of validation images\n",
    "        val_loss = val_loss / val_images.shape[0]\n",
    "        # Store losses\n",
    "        val_loss_record   = np.append( val_loss_record, val_loss )\n",
    "        train_loss_record = np.append( train_loss_record, train_loss )\n",
    "        iterations_record = np.append( iterations_record, training_iteration )\n",
    "        # Inform the user and store the model if it is the best one yet\n",
    "        if (val_loss == val_loss_record.min() and training_iteration > 10):\n",
    "            print( 'Train step %d, Batch loss: %0.4f, Test loss: %0.4f, Elapsed: %ds. Best in test yet! Storing.'%( \n",
    "                                                                                    training_iteration, \n",
    "                                                                                    train_loss, \n",
    "                                                                                    val_loss, \n",
    "                                                                                    time.time() - start ) ) \n",
    "            tf.saved_model.simple_save( sess,\n",
    "                results_dir + 'trained_convnet_' + str( training_iteration ) + '_train-steps_5_kersize/',\n",
    "                inputs = {'image': source},\n",
    "                outputs = {'psdr': output} )\n",
    "        else:\n",
    "            print( 'Train step %d, Batch loss: %0.4f, Test loss: %0.4f, Elapsed: %ds.'%( \n",
    "                                                                                    training_iteration, \n",
    "                                                                                    train_loss, \n",
    "                                                                                    val_loss, \n",
    "                                                                                    time.time() - start ) )\n",
    "# Store loss values\n",
    "# Store loss values\n",
    "np.savez( results_dir + 'trained_convnet_history_train-steps_5_kersize', train_loss_record = train_loss_record, \n",
    "                                                                         val_loss_record   = val_loss_record, \n",
    "                                                                         iterations_record = iterations_record )\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}