6. Simple MNIST with Tensorboard.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensorboard Example using Simple MNIST\n",
    "\n",
    "This tutorial will show you how to get started with Tensorboard.\n",
    "Tensorboard is a visualization tool that you can use along with Tensorflow. You can do a lot of different things with Tensorboard, such as plot metrics about the execution of your (computational) graphs, visualize images that pass through your graphs, etc.\n",
    "\n",
    "In brief, the steps to use Tensorboard is as follow:\n",
    "1. Indicate which nodes in your program from which you would like to collect the summary of data you want to visualize\n",
    "2. Within a running session, generate and merge all the summary nodes, by using tf.summary.merge_all\n",
    "3. Write the merged summary to disk, using tf.summary.FileWriter\n",
    "4. Access the Tensorboard to view the summary\n",
    "\n",
    "We will go through the four steps above in the following tutorial, using a simple learning on MNIST dataset. \n",
    "\n",
    "This tutorial is based on \n",
    "*https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/4_Utils/tensorboard_basic.ipynb \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you haven't already, download the MNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "# Import MINST data\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Declare a directory to which your graph summaries will be saved. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "logs_path = '/tmp/tensorflow_logs/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following part, we will declare the fixed parameters and input parameters for our learning. \n",
    "This part is the same as the previous examples on declaring parameters for learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "learning_rate = 0.01\n",
    "training_epochs = 25\n",
    "batch_size = 100\n",
    "display_step = 1\n",
    "\n",
    "# tf Graph Input\n",
    "# mnist data image of shape 28*28=784\n",
    "x = tf.placeholder(tf.float32, [None, 784], name='InputData')\n",
    "# 0-9 digits recognition => 10 classes\n",
    "y = tf.placeholder(tf.float32, [None, 10], name='LabelData')\n",
    "\n",
    "# Set model weights\n",
    "W = tf.Variable(tf.zeros([784, 10]), name='Weights')\n",
    "b = tf.Variable(tf.zeros([10]), name='Bias')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After declaring our fixed parameters, we would continue with declaring our learning method: the model we would learn, its cost function, its optimizer and accuracy measure.\n",
    "\n",
    "Below, you see that we declare our nodes/ops into scopes. A scope can contain more than one node/op. Having a scope would particularly help when you want to visualize your network. \n",
    "This is not a scope of this tutorial, but you can learn more about visualizing your network here: https://www.tensorflow.org/get_started/graph_viz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Construct model and encapsulating all ops into scopes, making\n",
    "# Tensorboard's Graph visualization more convenient\n",
    "with tf.name_scope('Model'):\n",
    "    # Model\n",
    "    pred = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax\n",
    "with tf.name_scope('Loss'):\n",
    "    # Minimize error using cross entropy\n",
    "    cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))\n",
    "with tf.name_scope('SGD'):\n",
    "    # Gradient Descent\n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)\n",
    "with tf.name_scope('Accuracy'):\n",
    "    # Accuracy\n",
    "    acc = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))\n",
    "    acc = tf.reduce_mean(tf.cast(acc, tf.float32))\n",
    "\n",
    "# Initializing the variables\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following section, we declare which nodes we would like to summarize, and in which way. \n",
    "Depending on what you want to visualize, you can choose from different types of summary ops: scalar, histogram, audio, image, etc. \n",
    "In this example, we will choose a scalar summary, as we would like to monitor our cost function in each epoch, as well as the accuracy of our learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a summary to monitor cost tensor\n",
    "tf.summary.scalar(\"loss\", cost)\n",
    "# Create a summary to monitor accuracy tensor\n",
    "tf.summary.scalar(\"accuracy\", acc)\n",
    "# Merge all summaries into a single op\n",
    "merged_summary_op = tf.summary.merge_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have declared the summary ops, and the merged summary Tensor, we can then continue with running our Session.\n",
    "\n",
    "To use Tensorboard, in our running Session, we would need to create the tf.summary.Filewriter Tensor, and let it know which directory (logs_path) it should write the summary logs to.\n",
    "\n",
    "Afterwards, we need to run our merged_summary_op along with the other ops in the learning process (optimizer, cost). Finally, the output summary is added into the log file, by using summary_writer.add_summary()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 cost= 1.182138977\n",
      "Epoch: 0002 cost= 0.664631298\n",
      "Epoch: 0003 cost= 0.552558806\n",
      "Epoch: 0004 cost= 0.498570960\n",
      "Epoch: 0005 cost= 0.465417103\n",
      "Epoch: 0006 cost= 0.442526688\n",
      "Epoch: 0007 cost= 0.425479051\n",
      "Epoch: 0008 cost= 0.412129293\n",
      "Epoch: 0009 cost= 0.401385706\n",
      "Epoch: 0010 cost= 0.392419814\n",
      "Epoch: 0011 cost= 0.384733562\n",
      "Epoch: 0012 cost= 0.378202107\n",
      "Epoch: 0013 cost= 0.372406618\n",
      "Epoch: 0014 cost= 0.367222395\n",
      "Epoch: 0015 cost= 0.362659236\n",
      "Epoch: 0016 cost= 0.358608620\n",
      "Epoch: 0017 cost= 0.354867672\n",
      "Epoch: 0018 cost= 0.351428938\n",
      "Epoch: 0019 cost= 0.348337940\n",
      "Epoch: 0020 cost= 0.345400420\n",
      "Epoch: 0021 cost= 0.342753728\n",
      "Epoch: 0022 cost= 0.340205403\n",
      "Epoch: 0023 cost= 0.337913331\n",
      "Epoch: 0024 cost= 0.335709043\n",
      "Epoch: 0025 cost= 0.333699787\n",
      "Optimization Finished!\n",
      "Accuracy: 0.9143\n"
     ]
    }
   ],
   "source": [
    "# Launch the graph\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "\n",
    "    # op to write logs to Tensorboard\n",
    "    summary_writer = tf.summary.FileWriter(logs_path, graph=tf.get_default_graph())\n",
    "\n",
    "    # Training cycle\n",
    "    for epoch in range(training_epochs):\n",
    "        avg_cost = 0.\n",
    "        total_batch = int(mnist.train.num_examples/batch_size)\n",
    "        # Loop over all batches\n",
    "        for i in range(total_batch):\n",
    "            batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n",
    "            # Run optimization op (backprop), cost op (to get loss value)\n",
    "            # and summary nodes\n",
    "            _, c, summary = sess.run([optimizer, cost, merged_summary_op],\n",
    "                                     feed_dict={x: batch_xs, y: batch_ys})\n",
    "            # Write logs at every iteration\n",
    "            summary_writer.add_summary(summary, epoch * total_batch + i)\n",
    "            # Compute average loss\n",
    "            avg_cost += c / total_batch\n",
    "        # Display logs per epoch step\n",
    "        if (epoch+1) % display_step == 0:\n",
    "            print \"Epoch:\", '%04d' % (epoch+1), \"cost=\", \"{:.9f}\".format(avg_cost)\n",
    "\n",
    "    print \"Optimization Finished!\"\n",
    "\n",
    "    # Test model\n",
    "    # Calculate accuracy\n",
    "    print \"Accuracy:\", acc.eval({x: mnist.test.images, y: mnist.test.labels})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Up until this point, the summaries of the graphs we are interested in have been saved in a log file in the '/tmp/tensorflow_logs/' directory (or whatever path you gave before). \n",
    "\n",
    "To visualize it on Tensorboard, we need to first activate it through the command line, and then open it in our browser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Run the command line:\n",
      "--> tensorboard --logdir=/tmp/tensorflow_logs \n",
      "Then open http://0.0.0.0:6006/ into your web browser\n"
     ]
    }
   ],
   "source": [
    "print \"Run the command line:\\n\" \\\n",
    "      \"--> tensorboard --logdir=/tmp/tensorflow_logs \" \\\n",
    "      \"\\nThen open http://0.0.0.0:6006/ into your web browser\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tensorflow",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}