2-seq2seq-advanced.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced dynamic seq2seq with TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encoder is bidirectional now. Decoder is implemented using `tf.nn.raw_rnn`. \n",
    "It feeds previously generated tokens during training as inputs, instead of target sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**UPDATE (16.02.2017)**: I learned some things after I wrote this tutorial. In particular:\n",
    " - [DONE] Replacing projection (one-hot encoding followed by linear layer) with embedding (indexing weights of linear layer directly) is more efficient.\n",
    " - When decoding, feeding previously generated tokens as inputs adds robustness to model's errors. However feeding ground truth speeds up training. Apperantly best practice is to mix both randomly when training.\n",
    "\n",
    "I will update tutorial to reflect this at some point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import helpers\n",
    "\n",
    "tf.reset_default_graph()\n",
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.3.0'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PAD = 0\n",
    "EOS = 1\n",
    "\n",
    "vocab_size = 10\n",
    "input_embedding_size = 20\n",
    "\n",
    "encoder_hidden_units = 20\n",
    "decoder_hidden_units = encoder_hidden_units * 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "encoder_inputs = tf.placeholder(shape=(None, None), dtype=tf.int32, name='encoder_inputs')\n",
    "encoder_inputs_length = tf.placeholder(shape=(None,), dtype=tf.int32, name='encoder_inputs_length')\n",
    "\n",
    "decoder_targets = tf.placeholder(shape=(None, None), dtype=tf.int32, name='decoder_targets')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously we elected to manually feed `decoder_inputs` to better understand what is going on. Here we implement decoder with `tf.nn.raw_rnn` and will construct `decoder_inputs` step by step in the loop."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "Setup embeddings (see tutorial 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "embeddings = tf.Variable(tf.random_uniform([vocab_size, input_embedding_size], -1.0, 1.0), dtype=tf.float32)\n",
    "\n",
    "encoder_inputs_embedded = tf.nn.embedding_lookup(embeddings, encoder_inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoder\n",
    "\n",
    "We are replacing unidirectional `tf.nn.dynamic_rnn` with `tf.nn.bidirectional_dynamic_rnn` as the encoder.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.contrib.rnn import LSTMCell, LSTMStateTuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder_cell = LSTMCell(encoder_hidden_units)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "((encoder_fw_outputs,\n",
    "  encoder_bw_outputs),\n",
    " (encoder_fw_final_state,\n",
    "  encoder_bw_final_state)) = (\n",
    "    tf.nn.bidirectional_dynamic_rnn(cell_fw=encoder_cell,\n",
    "                                    cell_bw=encoder_cell,\n",
    "                                    inputs=encoder_inputs_embedded,\n",
    "                                    sequence_length=encoder_inputs_length,\n",
    "                                    dtype=tf.float32, time_major=True)\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'bidirectional_rnn/fw/fw/TensorArrayStack/TensorArrayGatherV3:0' shape=(?, ?, 20) dtype=float32>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoder_fw_outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'ReverseSequence:0' shape=(?, ?, 20) dtype=float32>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoder_bw_outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LSTMStateTuple(c=<tf.Tensor 'bidirectional_rnn/fw/fw/while/Exit_2:0' shape=(?, 20) dtype=float32>, h=<tf.Tensor 'bidirectional_rnn/fw/fw/while/Exit_3:0' shape=(?, 20) dtype=float32>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoder_fw_final_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LSTMStateTuple(c=<tf.Tensor 'bidirectional_rnn/bw/bw/while/Exit_2:0' shape=(?, 20) dtype=float32>, h=<tf.Tensor 'bidirectional_rnn/bw/bw/while/Exit_3:0' shape=(?, 20) dtype=float32>)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoder_bw_final_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have to concatenate forward and backward outputs and state. In this case we will not discard outputs, they would be used for attention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "encoder_outputs = tf.concat((encoder_fw_outputs, encoder_bw_outputs), 2)\n",
    "\n",
    "encoder_final_state_c = tf.concat(\n",
    "    (encoder_fw_final_state.c, encoder_bw_final_state.c), 1)\n",
    "\n",
    "encoder_final_state_h = tf.concat(\n",
    "    (encoder_fw_final_state.h, encoder_bw_final_state.h), 1)\n",
    "\n",
    "encoder_final_state = LSTMStateTuple(\n",
    "    c=encoder_final_state_c,\n",
    "    h=encoder_final_state_h\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "decoder_cell = LSTMCell(decoder_hidden_units)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time and batch dimensions are dynamic, i.e. they can change in runtime, from batch to batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder_max_time, batch_size = tf.unstack(tf.shape(encoder_inputs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we need to decide how far to run decoder. There are several options for stopping criteria:\n",
    "- Stop after specified number of unrolling steps\n",
    "- Stop after model produced <EOS> token\n",
    "\n",
    "The choice will likely be time-dependant. In legacy `translate` tutorial we can see that decoder unrolls for `len(encoder_input)+10` to allow for possibly longer translated sequence. Here we are doing a toy copy task, so how about we unroll decoder for `len(encoder_input)+2`, to allow model some room to make mistakes over 2 additional steps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "decoder_lengths = encoder_inputs_length + 3\n",
    "# +2 additional steps, +1 leading <EOS> token for decoder inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Output projection\n",
    "\n",
    "Decoder will contain manually specified by us transition step:\n",
    "```\n",
    "output(t) -> output projection(t) -> prediction(t) (argmax) -> input embedding(t+1) -> input(t+1)\n",
    "```\n",
    "\n",
    "In tutorial 1, we used `tf.contrib.layers.linear` layer to initialize weights and biases and apply operation for us. This is convenient, however now we need to specify parameters `W` and `b`  of the output layer in global scope, and apply them at every step of the decoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "W = tf.Variable(tf.random_uniform([decoder_hidden_units, vocab_size], -1, 1), dtype=tf.float32)\n",
    "b = tf.Variable(tf.zeros([vocab_size]), dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoder via `tf.nn.raw_rnn`\n",
    "\n",
    "`tf.nn.dynamic_rnn` allows for easy RNN construction, but is limited. \n",
    "\n",
    "For example, a nice way to increase robustness of the model is to feed as decoder inputs tokens that it previously generated, instead of shifted true sequence.\n",
    "\n",
    "![seq2seq-feed-previous](pictures/2-seq2seq-feed-previous.png)\n",
    "*Image borrowed from http://www.wildml.com/2016/04/deep-learning-for-chatbots-part-1-introduction/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First prepare tokens. Decoder would operate on column vectors of shape `(batch_size,)` representing single time steps of the batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "assert EOS == 1 and PAD == 0\n",
    "\n",
    "eos_time_slice = tf.ones([batch_size], dtype=tf.int32, name='EOS')\n",
    "pad_time_slice = tf.zeros([batch_size], dtype=tf.int32, name='PAD')\n",
    "\n",
    "eos_step_embedded = tf.nn.embedding_lookup(embeddings, eos_time_slice)\n",
    "pad_step_embedded = tf.nn.embedding_lookup(embeddings, pad_time_slice)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for the tricky part.\n",
    "\n",
    "Remember that standard `tf.nn.dynamic_rnn` requires all inputs `(t, ..., t+n)` be passed in advance as a single tensor. \"Dynamic\" part of its name refers to the fact that `n` can change from batch to batch.\n",
    "\n",
    "Now, what if we want to implement more complex mechanic like when we want decoder to receive previously generated tokens as input at every timestamp (instead of lagged target sequence)? Or when we want to implement soft attention, where at every timestep we add additional fixed-len representation, derived from query produced by previous step's hidden state? `tf.nn.raw_rnn` is a way to solve this problem.\n",
    "\n",
    "Main part of specifying RNN with `tf.nn.raw_rnn` is *loop transition function*. It defines inputs of step `t` given outputs and state of step `t-1`.\n",
    "\n",
    "Loop transition function is a mapping `(time, previous_cell_output, previous_cell_state, previous_loop_state) -> (elements_finished, input, cell_state, output, loop_state)`. It is called *before* RNNCell to prepare its inputs and state. Everything is a Tensor except for initial call at time=0 when everything is `None` (except `time`).\n",
    "\n",
    "Note that decoder inputs are returned from the transition function but passed into it. You are supposed to index inputs manually using `time` Tensor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loop transition function is called two times:\n",
    " 1. Initial call at time=0 to provide initial cell_state and input to RNN.\n",
    " 2. Transition call for all following timesteps where you define transition between two adjacent steps.\n",
    "\n",
    "Lets define both cases separately."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loop initial state is function of only `encoder_final_state` and embeddings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def loop_fn_initial():\n",
    "    initial_elements_finished = (0 >= decoder_lengths)  # all False at the initial step\n",
    "    initial_input = eos_step_embedded\n",
    "    initial_cell_state = encoder_final_state\n",
    "    initial_cell_output = None\n",
    "    initial_loop_state = None  # we don't need to pass any additional information\n",
    "    return (initial_elements_finished,\n",
    "            initial_input,\n",
    "            initial_cell_state,\n",
    "            initial_cell_output,\n",
    "            initial_loop_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define transition function such that previously generated token (as judged in greedy manner by `argmax` over output projection) is passed as next input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def loop_fn_transition(time, previous_output, previous_state, previous_loop_state):\n",
    "\n",
    "    def get_next_input():\n",
    "        output_logits = tf.add(tf.matmul(previous_output, W), b)\n",
    "        prediction = tf.argmax(output_logits, axis=1)\n",
    "        next_input = tf.nn.embedding_lookup(embeddings, prediction)\n",
    "        return next_input\n",
    "    \n",
    "    elements_finished = (time >= decoder_lengths) # this operation produces boolean tensor of [batch_size]\n",
    "                                                  # defining if corresponding sequence has ended\n",
    "\n",
    "    finished = tf.reduce_all(elements_finished) # -> boolean scalar\n",
    "    input = tf.cond(finished, lambda: pad_step_embedded, get_next_input)\n",
    "    state = previous_state\n",
    "    output = previous_output\n",
    "    loop_state = None\n",
    "\n",
    "    return (elements_finished, \n",
    "            input,\n",
    "            state,\n",
    "            output,\n",
    "            loop_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combine initializer and transition functions and create raw_rnn.\n",
    "\n",
    "Note that while all operations above are defined with TF's control flow and reduction ops, here we rely on checking if state is `None` to determine if it is an initializer call or transition call. This is not very clean API and might be changed in the future (indeed, `tf.nn.raw_rnn`'s doc contains warning that API is experimental)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def loop_fn(time, previous_output, previous_state, previous_loop_state):\n",
    "    if previous_state is None:    # time == 0\n",
    "        assert previous_output is None and previous_state is None\n",
    "        return loop_fn_initial()\n",
    "    else:\n",
    "        return loop_fn_transition(time, previous_output, previous_state, previous_loop_state)\n",
    "\n",
    "decoder_outputs_ta, decoder_final_state, _ = tf.nn.raw_rnn(decoder_cell, loop_fn)\n",
    "decoder_outputs = decoder_outputs_ta.stack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'TensorArrayStack/TensorArrayGatherV3:0' shape=(?, ?, 40) dtype=float32>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To do output projection, we have to temporarilly flatten `decoder_outputs` from `[max_steps, batch_size, hidden_dim]` to `[max_steps*batch_size, hidden_dim]`, as `tf.matmul` needs rank-2 tensors at most."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "decoder_max_steps, decoder_batch_size, decoder_dim = tf.unstack(tf.shape(decoder_outputs))\n",
    "decoder_outputs_flat = tf.reshape(decoder_outputs, (-1, decoder_dim))\n",
    "decoder_logits_flat = tf.add(tf.matmul(decoder_outputs_flat, W), b)\n",
    "decoder_logits = tf.reshape(decoder_logits_flat, (decoder_max_steps, decoder_batch_size, vocab_size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "decoder_prediction = tf.argmax(decoder_logits, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RNN outputs tensor of shape `[max_time, batch_size, hidden_units]` which projection layer maps onto `[max_time, batch_size, vocab_size]`. `vocab_size` part of the shape is static, while `max_time` and `batch_size` is dynamic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stepwise_cross_entropy = tf.nn.softmax_cross_entropy_with_logits(\n",
    "    labels=tf.one_hot(decoder_targets, depth=vocab_size, dtype=tf.float32),\n",
    "    logits=decoder_logits,\n",
    ")\n",
    "\n",
    "loss = tf.reduce_mean(stepwise_cross_entropy)\n",
    "train_op = tf.train.AdamOptimizer().minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training on the toy task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the copy task — given a random sequence of integers from a `vocabulary`, learn to memorize and reproduce input sequence. Because sequences are random, they do not contain any structure, unlike natural language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "head of the batch:\n",
      "[7, 3, 7, 3, 8, 4]\n",
      "[8, 9, 7]\n",
      "[5, 2, 6, 7, 3, 9, 9, 8]\n",
      "[6, 7, 4, 2, 9, 6, 3, 3]\n",
      "[5, 2, 2, 3, 7, 3]\n",
      "[5, 6, 7, 9, 9]\n",
      "[6, 3, 3, 4]\n",
      "[4, 6, 5, 6, 4]\n",
      "[7, 7, 7, 3]\n",
      "[4, 6, 3, 9, 2, 5]\n"
     ]
    }
   ],
   "source": [
    "batch_size = 100\n",
    "\n",
    "batches = helpers.random_sequences(length_from=3, length_to=8,\n",
    "                                   vocab_lower=2, vocab_upper=10,\n",
    "                                   batch_size=batch_size)\n",
    "\n",
    "print('head of the batch:')\n",
    "for seq in next(batches)[:10]:\n",
    "    print(seq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def next_feed():\n",
    "    batch = next(batches)\n",
    "    encoder_inputs_, encoder_input_lengths_ = helpers.batch(batch)\n",
    "    decoder_targets_, _ = helpers.batch(\n",
    "        [(sequence) + [EOS] + [PAD] * 2 for sequence in batch]\n",
    "    )\n",
    "    return {\n",
    "        encoder_inputs: encoder_inputs_,\n",
    "        encoder_inputs_length: encoder_input_lengths_,\n",
    "        decoder_targets: decoder_targets_,\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss_track = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch 0\n",
      "  minibatch loss: 2.3642187118530273\n",
      "  sample 1:\n",
      "    input     > [2 7 2 6 0 0 0 0]\n",
      "    predicted > [1 1 1 1 6 1 6 0 0 0 0]\n",
      "  sample 2:\n",
      "    input     > [8 8 7 9 0 0 0 0]\n",
      "    predicted > [1 1 1 6 1 6 6 0 0 0 0]\n",
      "  sample 3:\n",
      "    input     > [4 4 6 2 6 7 8 0]\n",
      "    predicted > [0 0 6 6 6 6 6 3 7 4 0]\n",
      "\n",
      "batch 1000\n",
      "  minibatch loss: 0.5487109422683716\n",
      "  sample 1:\n",
      "    input     > [3 6 9 4 5 6 5 0]\n",
      "    predicted > [3 6 6 9 6 5 5 1 0 0 0]\n",
      "  sample 2:\n",
      "    input     > [2 3 3 7 2 0 0 0]\n",
      "    predicted > [2 3 3 3 2 1 0 0 0 0 0]\n",
      "  sample 3:\n",
      "    input     > [6 2 5 2 4 0 0 0]\n",
      "    predicted > [6 2 2 2 4 1 0 0 0 0 0]\n",
      "\n",
      "batch 2000\n",
      "  minibatch loss: 0.277699738740921\n",
      "  sample 1:\n",
      "    input     > [7 6 7 9 9 0 0 0]\n",
      "    predicted > [7 6 7 9 9 1 0 0 0 0 0]\n",
      "  sample 2:\n",
      "    input     > [7 5 3 4 3 3 3 3]\n",
      "    predicted > [7 5 4 3 3 3 3 3 1 0 0]\n",
      "  sample 3:\n",
      "    input     > [7 9 8 4 6 8 6 3]\n",
      "    predicted > [7 9 8 4 8 6 6 3 1 0 0]\n",
      "\n",
      "batch 3000\n",
      "  minibatch loss: 0.13742178678512573\n",
      "  sample 1:\n",
      "    input     > [8 5 2 4 7 5 4 5]\n",
      "    predicted > [8 5 2 7 4 4 5 5 1 0 0]\n",
      "  sample 2:\n",
      "    input     > [8 7 2 8 6 0 0 0]\n",
      "    predicted > [8 7 2 8 6 1 0 0 0 0 0]\n",
      "  sample 3:\n",
      "    input     > [9 8 9 7 0 0 0 0]\n",
      "    predicted > [9 8 9 7 1 0 0 0 0 0 0]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "max_batches = 3001\n",
    "batches_in_epoch = 1000\n",
    "\n",
    "try:\n",
    "    for batch in range(max_batches):\n",
    "        fd = next_feed()\n",
    "        _, l = sess.run([train_op, loss], fd)\n",
    "        loss_track.append(l)\n",
    "\n",
    "        if batch == 0 or batch % batches_in_epoch == 0:\n",
    "            print('batch {}'.format(batch))\n",
    "            print('  minibatch loss: {}'.format(sess.run(loss, fd)))\n",
    "            predict_ = sess.run(decoder_prediction, fd)\n",
    "            for i, (inp, pred) in enumerate(zip(fd[encoder_inputs].T, predict_.T)):\n",
    "                print('  sample {}:'.format(i + 1))\n",
    "                print('    input     > {}'.format(inp))\n",
    "                print('    predicted > {}'.format(pred))\n",
    "                if i >= 2:\n",
    "                    break\n",
    "            print()\n",
    "\n",
    "except KeyboardInterrupt:\n",
    "    print('training interrupted')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.1332 after 300100 examples (batch_size=100)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecFeW9x/HPb3ujLbuANJeyiqg0V0DA3igmGmOMmogl\nkRjjvZpEEyKxcRPlGks0GguRWGI08dowgiAWQESQ3puAdFjKUraX5/5xjssu29mzO6d836/Xvpgz\n85wzv3Hky7NznnnGnHOIiEh4ifK6ABERCTyFu4hIGFK4i4iEIYW7iEgYUriLiIQhhbuISBiqM9zN\nrIuZfWpmq8xspZndUU2b88zsoJkt8f/c1zTliohIfcTUo00J8Gvn3CIzawEsNLOPnHOrjmk32zl3\nWeBLFBGRhqqz5+6c2+mcW+RfPgysBjo1dWEiInL86tNzL2dmGUB/YF41m4eY2TJgO3CXc25lbZ+V\nlpbmMjIyGrJ7EZGIt3Dhwr3OufS62tU73M0sBXgLuNM5d+iYzYuArs65I2Y2EngXyKzmM8YAYwC6\ndu3KggUL6rt7EREBzOyb+rSr12gZM4vFF+yvOefePna7c+6Qc+6If3kKEGtmadW0e8E5l+Wcy0pP\nr/MfHhEROU71GS1jwIvAaufc4zW06eBvh5kN9H/uvkAWKiIi9VefyzJDgeuB5Wa2xL/uHqArgHPu\nOeAq4OdmVgLkA9c4TTcpIuKZOsPdOfc5YHW0eRp4OlBFiYhI4+gOVRGRMKRwFxEJQwp3EZEwFHLh\nvnbXYR6dtpYDuUVelyIiErRCLtw37T3C059uYHtOvteliIgErZAL99ZJcQDk5BV7XImISPAKuXBP\nifeN3swtKvG4EhGR4BVy4Z4UFw1AflGpx5WIiASvEAx3X889T+EuIlKjkAv3RH/PPU+XZUREahRy\n4Z4SH0NcTBTZhwu9LkVEJGiFXLhHRxltkmI1WkZEpBYhF+4AyXExGi0jIlKLkAz3xLhojZYREalF\nSIa7eu4iIrULyXBXz11EpHYhGe7J8dHkKtxFRGoUkuGeGBujnruISC1CMtx9PXddcxcRqUlIhvsX\nX+8jJ6+YTXtzvS5FRCQohWS4b92fB8CybTkeVyIiEpxCMtyf/fEAAFKT4zyuREQkOIVkuLdvmQDA\n9gN6GpOISHVCMtyjzAAY+/ZyjysREQlOIRnuPduleF2CiEhQC8lwj40+WnZZmfOwEhGR4BSS4Q6Q\nlhIPwBGNdxcRqSJkw/03w08G4FC+5nUXETlWyIZ7y4RYAA4q3EVEqgjZcAfftfbnZ270uA4RkeAT\nsuGe2b4FAJOX7vC4EhGR4BOy4d4jXcMhRURqErLhXtFDU1Z7XYKISFAJ6XDvlpYMwAuzdN1dRKSi\nOsPdzLqY2admtsrMVprZHdW0MTN7ysw2mNkyMxvQNOVWdvOwbuXLuw4WNMcuRURCQn167iXAr51z\nvYHBwC/MrPcxbUYAmf6fMcCzAa2yBq0SY8uXx7y6oDl2KSISEuoMd+fcTufcIv/yYWA10OmYZpcD\nrzifL4HWZnZCwKs9xmWnH93Fqh2Hmnp3IiIho0HX3M0sA+gPzDtmUydga4XX26j6D0DARUUZ55+c\nDkCJ5pgRESlX73A3sxTgLeBO59xxdZPNbIyZLTCzBdnZ2cfzEVU8fGWf8uWb/j4/IJ8pIhLq6hXu\nZhaLL9hfc869XU2T7UCXCq87+9dV4px7wTmX5ZzLSk9PP556q+jQKqF8+dO1gfkHQ0Qk1NVntIwB\nLwKrnXOP19BsMjDaP2pmMHDQObczgHXW6sJe7ZprVyIiISGmHm2GAtcDy81siX/dPUBXAOfcc8AU\nYCSwAcgDbgp8qTW769KT+XjNnubcpYhIUKsz3J1znwNWRxsH/CJQRTXUKSe09GrXIiJBKaTvUK3I\n/1hVPlzRbFeDRESCVtiEu/OPhFy67aC3hYiIBIGwCfeEWN+hFJeUeVyJiIj3wibcv30y05FCPVNV\nRCRswv3KAZ0B+Grzfo8rERHxXtiE+12XnATA19m5HlciIuK9sAn3mOiwORQRkUYLy0S84pk5Xpcg\nIuKpsAz3JVtzvC5BRMRTYRXuT13b3+sSRESCQliF+8jTOnhdgohIUAircK/4pWpBcamHlYiIeCus\nwh2gt38SsSc+WudxJSIi3gm7cO/ftTUAhZqGQEQiWNiF++0X9AQgs32Kx5WIiHgn7MI9Jd43Rf22\nA/keVyIi4p2wC/fkOF+4P/vZ1x5XIiLinbAL96ioow+Nyskr8rASERHvhF24V5STV+x1CSIingjL\ncH/++jMAze0uIpErLMP92y9VcxXuIhKhwjrcF2sCMRGJUGEZ7ie0TgBgwtQ1HlciIuKNsAz39JR4\nr0sQEfFUWIa7mdXdSEQkjIVluFdUpDlmRCQChW24n9DKd939gG5kEpEIFLbhfu9lvQHYn6twF5HI\nE7bhnpocB8CybRoOKSKRJ2zDvWVCLAC/fWu5x5WIiDS/sA339BYaDikikSvG6wKaSnqLeJLiornw\nlPZelyIi0uzCtucO0DU1iR05emiHiESesO25A6zZdRiAguJSEmKjPa5GRKT51NlzN7NJZrbHzFbU\nsP08MztoZkv8P/cFvszGOZSved1FJLLU57LMS8DwOtrMds718/+Mb3xZgXVQ4S4iEabOcHfOzQL2\nN0MtTealLzZ7XYKISLMK1BeqQ8xsmZlNNbNTA/SZjfbIVX0AeG3eFo8rERFpXoEI90VAV+dcH+Av\nwLs1NTSzMWa2wMwWZGdnB2DXtbtqQOcm34eISDBqdLg75w455474l6cAsWaWVkPbF5xzWc65rPT0\n9Mbuuk5RUZr6V0QiU6PD3cw6mH8CdTMb6P/MfY393EC5fvCJACz8JqS/NhARaZD6DIV8HZgLnGxm\n28zsJ2Z2q5nd6m9yFbDCzJYCTwHXOOdc05XcMJee2gGA7z871+NKRESaT503MTnnrq1j+9PA0wGr\nKMBOap/idQkiIs0urKcfAEiOD+ubcEVEqhX24Z5YYdqBILpaJCLSpMI+3CuOmPnDB6s9rEREpPmE\nfbgDTLoxC4AXP9/kcSUiIs0jIsL9gl5H53TXPDMiEgkiItwr6vvgdK9LEBFpchEX7iIikSBiwr1/\n19ZelyAi0mwiJtxfuD6rfLm4tMzDSkREml7EhHt6i3jaJMUCkDluqsfViIg0rYgJd4DZv73A6xJE\nRJpFRIV7SoWpCO5+c6mHlYiINK2ICveK3ly4jW/25XpdhohIk4i4cP/9qFPKl7cdyPewEhGRphNx\n4X5J7w7lyz/62zwPKxERaToRF+5d2yZVev3VZj2hSUTCT8SFO8Cwnkcf8fqD5+ZypLDEw2pERAIv\nIsP9pZvO5H8uP7X89bJtOR5WIyISeBEZ7jHRUQyt0Hu/bqKuvYtIeInIcAfonp5CanJc+euHp+pB\nHiISPiI23AHm/u7oHavPz9zI4QLN9S4i4SGiwz0+JrrS69MfmE5pmZ6zKiKhL6LDHWDlg5dWet3j\nninszy3yqBoRkcCI+HBPjo/h3z87q9K61+dv8agaEZHAiPhwBxjYLZWbh3Yrf/2naWs9rEZEpPEU\n7n6XnNq+0ushD3/MnsMFHlUjItI4Cne/wd3bsuGPI8pf7zhYwMA/fsyOHE0uJiKhR+FeQUx0FLN/\nc36ldUMmfOJRNSIix0/hfowuqUlV1v19ziYPKhEROX4K92pMu/OcSq8ffH8Vt722kILiUo8qEhFp\nGIV7NU7u0II1/zO80ropy3fR694Pmfv1Po+qEhGpP4V7DRJio7mgV7sq66+d+KUH1YiINIzCvRbP\n/fgMLqwm4EtKyzyoRkSk/hTutYiLieLFG8+ssv6TNXs8qEZEpP7qDHczm2Rme8xsRQ3bzcyeMrMN\nZrbMzAYEvszgMubVhWSM/YBi9eBFJEjVp+f+EjC8lu0jgEz/zxjg2caXFVw2TxjF5gmjePu2IZXW\nZ46bqhE0IhKU6gx359wsoLanSF8OvOJ8vgRam9kJgSowmAzo2obuacmV1vW690PeW7Ldo4pERKoX\niGvunYCtFV5v868LSy+Mzqqy7o43lnDvuyso01zwIhIkmvULVTMbY2YLzGxBdnZ2c+46YHq2S6ly\nkxPAq19+w6n3T/OgIhGRqgIR7tuBLhVed/avq8I594JzLss5l5Wenh6AXXvj5A4tmH/PhVXW5xeX\nkjH2A7buz/OgKhGRowIR7pOB0f5RM4OBg865nQH43KDWrmUCM351Lh//+twq285+5FMPKhIROao+\nQyFfB+YCJ5vZNjP7iZndama3+ptMATYCG4CJwG1NVm2Q6dkuhR7p1V+myRj7AZv25npQlYgImHPe\nfAmYlZXlFixY4Mm+m8LW/XnV9tjHjujFLWd3JzrKPKhKRMKNmS10zlUd2XEM3aEaIF1Sk9g8YRQ9\n0isPlZwwdQ3vLdnOoYJiTVsgIs1G4R5gH//6PAZ3T6207lf/XkqfB6Yz4snZHlUlIpFG4d4EXrl5\nULXr1+85wkt68IeINAOFexOIi4ni07vOq3bbA++vat5iRCQiKdybSLe0ZBbdezF3XpRZZVvG2A/I\nLSxhyvKd5BaWeFCdiIQ7hXsTSk2O4/sDOle77dT7p3Hba4u4553lzVyViEQChXsT65KaxLo/jGDp\n/ZdUu/29JTv0AG4RCTiFezOIi4miVWIsc8ZeUO32B99fxadr9QAQEQkchXsz6tQ6kYevPL3abTf9\n/Ssyxn7AL/+1BK9uLBOR8KFwb2bXnNmFnwzrRmJsdLXb31m8nfH/WcWeQwWs3HGwmasTkXCh6Qc8\nUlBcSq97P6yz3eYJo5qhGhEJFZp+IMglxEbzys0DGTuiF3+44rQa213xzJxmrEpEwkWM1wVEsnNO\nSueck3zz2v/10w3sOFhQpc2SrTn84rVF9O/amu/07UjLhFgS46q/pCMi8i1dlgkieUUl9L6v9qc5\nnXFiG976+ZBa24hI+NJlmRCUFBfD3286k/iYmk/Lwm8OsO9IYTNWJSKhSD33IFRa5li98xCX/eXz\nGttc0KsdI07rwOSlO9h9qIDpv6z6RCgRCT/17bnrmnsQio4yTuvUiqX3X0LfB6dX2+aTNXv4ZE3V\nG58O5hUTHxtFQg1DLUUkMuiyTBBrlRjL4nsvZt0fRtTZdvLSHQD0HT+dayd+2dSliUiQU7gHuTbJ\nccTFRLF6/HDO9Y+sqc5/v76YmeuyAVi8Jae5yhORIKVwDxGJcdG8fPNAEmJrPmU3TJpfvvz0J+ub\noywRCVL6QjXEOOeYs2Efs9dnc+dFJ3HxEzPZdiC/1vfMu+dC2rdMaKYKRaQp6QvVMGVmDMtMY1hm\nGgB//dEA/jlvC2dmpPLrN5dW+56py3eSFBfDpn25/HZ4r+YsV0Q8onAPcX06t6ZP59YAPPPZBjZm\n51ZpU/HRfj87pzutk+KarT4R8YauuYeRB797ap1t+o3/iMufmcM7i7cxf9P+ZqhKRLygcA8jZ2em\n868xg+tst3RrDr/811Kufn4uAPtziygpLWvq8kSkGekL1TCUV1TCjpwCTmybxLYD+Zz/6Gf1et/d\nl55M/66tWbXjEDcOySAmWv/2iwSb+n6hqnCPALsPFTDooY8b9J5OrRNrfCygiHhHE4dJufYtE3jz\n1rP46bBu9X7P9px8Fm85QG5hCS9/sZn8otImrFBEAk099wiUMfaD43rfpodHYmYczCsmLiZK88qL\neEA9d6nR4nsv5h8/GdTg91313FwO5hfTd/x0Lnp8ZhNUJiKBonCPQG2S4xiWmcb3+ncC4JazuzHm\nnO5c0a9jre9b+M2B8lkqt+fk49VvfSJSN12WiWBFJWXsPJjPiW2Ty9cdyC3iosdnsi+3qF6fMXF0\nFi9/sZkbhmSwdX8eV5/ZhZR43Rsn0lQ0WkaOW0FxKTNW7+b2fy5u8HtvObsb40b1xjnH+j1HyGyX\ngpk1QZUikUnhLo3mnKPb76Y06jMeuaoPV2d1CVBFIqKJw6TRzIxZd5/Psu05dGqdyKx1e3lixroG\nfcZ/lu1kQNfW7DxYQLsWCZzcoUUTVSsiFdUr3M1sOPAkEA38zTk34Zjt5wHvAZv8q952zo0PYJ3i\nka5tk+jaNgmAvp1bsyH7CO/7n/pUH7PWZXPR49nlrzf8cQQlZa7SYwBLSstYvv0g/bu2CVzhIhGu\nztEyZhYNPAOMAHoD15pZ72qaznbO9fP/KNjDUFSUlU9O1rNdCp/edR4v3zyQzm0S6/0ZPcdNpde9\nH/L5+r1c/vTnvLdkO499tI7v/fULVu442FSli0Sc+vTcBwIbnHMbAczsDeByYFWt75KwlJocx5PX\n9GNozzTSUuLplpbM7N+cT5mDHvfU//r8j1+cB8AdbyyhbbJvCuJ9R4ooK3NERekLWJHGqs84907A\n1gqvt/nXHWuImS0zs6lmVu3cs2Y2xswWmNmC7Ozs6ppICLi8XyfSUuLLX5sZ0VHGO7cNOa7P+3bY\n5ehJ8+l+zxQen76WguJSTXkg0gh1jpYxs6uA4c65n/pfXw8Mcs7dXqFNS6DMOXfEzEYCTzrnMmv7\nXI2WCU9FJWXkF5WSGBfNSb+f2ujPW//HEcRWmJ3SOUeZg2j17iVCBXL6ge1AxbFsnf3ryjnnDjnn\njviXpwCxZpbWgHolTMTFRNEqKZa4mCj+dFWf45rmoKKrn59LxtgPeH3+Fg7kFpH1hxnll3/+9dUW\n9hwuqNR+f24R63YfbtQ+RcJBfXruMcA64EJ8of4VcJ1zbmWFNh2A3c45Z2YDgf8DTnS1fLh67pHj\ncEEx/1m2k7Mz03j5i838ZngvMsc1vlf/rY9+eQ6Z7X1DLAf+cQZ7DheyecKoSm2OFJaQGButHr+E\nvID13J1zJcDtwDRgNfBv59xKM7vVzG71N7sKWGFmS4GngGtqC3aJLC0SYrl2YFc6t0li3KjexEZH\n8bfRWQzMSOW+y6obeNUwFz8xi8/X72X+pv3sOVwIwLYDeQDkF5VSVuY47f5p3PveikbvSyRU6A5V\n8ZRzjomzN/LQlDUB/+xXbh7I6Enz+e8LevLUJxsA+OuPBjDy9BOYvnIXU1fs4okf9gv4fkWakqb8\nlZBgZow5pwebJ4xi5t3ncfPQbvV6Dmx9fLhyF0B5sAPc9toituzLY8yrC3ln8XY+XLGT0jJfB2fp\n1hxy8uo3YZpIsFPPXYLSP+dtYfhpHUiKi6bPA9MpasIHeD/wnd7ExkQx7p0VZLZL4aNfndtk+xJp\nLPXcJaRdN6grqclxJMRGc36vdMB3SaUpPPD+Ksa947sev37PETbtzaWsrHKn50BuEc/P/BrnHEUl\nZTw2fS15RSVNUo9IIGjiMAl6Y87pwcx12Qzqlsqsu8+nsKSU7MOFjJ40n0d/0Jc7/7UkoPs7/9HP\nAPjb6Cw6tk6kfct4hj85m+zDhQw4sQ3rdh/mL59soLjUMXZEr4DuWyRQFO4S9M44sQ1r/mcEAG1T\nfOsy27dgw0MjAejTuRUXPBb4x/799JWqlw2XbMkpf3bswfzigO9TJFB0zV3CwrYDeWQfLqS41HH1\n83Pp2CqBF0ZncdlfPg/4vuJjoigsKaNDywTGjTqFR6atIaNtMhed0p77J69k1fhLSYpTv0mahh7W\nIRFry748uqQmYmaUlJbxyLS1vDBrIwBX9u/E24u31/EJxyfKoMzBn3/Yj0HdU8kvKiU+NpqOrRL0\nNCoJGIW7iJ9zjnmb9pPZLoW2KfF8tGo3t/gvuSTHRTPxhiyumzivyfb/0PdO57pBXSut23O4gOS4\nGJL1vFlpIIW7SA2KSsp4Z/E2vtu3U/n18/7jp3Mgr2mvoQ/rmcaNQzLISEvmosdnktE2iYmjs/i/\nRdu4sn/nSk+pOphfTKvE2CatR0KTwl2kAfKKSigsLuOb/XkczC/mhknzGTfyFP44ZXWz1fDZXeeR\nkZbMjFW7+ekrC3jz1rM4MyOVFdsPcmrHlszZsI9l23O47byezVaTBB+Fu0gjHcwrpu/46fzv908n\nv6iUB95v3ufTjDitA4u2HGD3oULO6t6WuRv3AbDp4ZHl1/BLyxxlzlWaFvlb01buYv3uw9x+Qa2z\nb0uI0U1MIo3UKimWzRNG8cMzu3Lj0G7l67+dWPKKfh1p1yK+hnc33tQVu9h9yDcR2rfBDrBoywGc\ncxSXltHjnilkjpvK3K9927fn5PPZ2j2Uljl+9upCHp2+jgO5Rfzu7WUUFJeyYPN+Nu3NbbKaJXio\n5y5ST898uoEWCTH8aNCJzN+0n7N6tGXJ1hxumDSfHunJPHZ1v/IboLzw1LX9+e/XF1dZ/+PBXfnH\nl1sY2rMtczb4/hF46+dn0TIhtnyq5GN9uGIXt/5jISsevJQU/5e+uYUlOCh/Ld7QZRkRjxSVlFV5\nCtXPzu3O8zM3lr++6JT2zFi9u7lLq+Lrh0ZSWFJaZVz+pU/MYu3uw/zzlkHc/eYyJo7OYuRTswHK\n58qfs2EvB/KKuKxPx2avO5LpsoyIR+JiosoDsGOrBEadfgK/vvhkhvRoW97m3stOIT4mipuGZnhU\npU+Pe6bQ+75pjH9/FQXFpTw5Yz2HC4rJL/Y9v/a6ifPYnpPPXz87OrPmS3M2AfCjv83j9n8u1hw7\nQUo9d5EmkldUQmx0VPmXnftzi7j33RU89L3TaZVUdZjj5KU7eHTaWrbsz2vuUhvsVxefxOMfrSt/\nXfHJVwdyi9iXW0TPdilelBb2dFlGJAQVlpSydX8+Fz0+k75dWvP7Uafwg+fmAvDoD/rSNjmOm176\nyuMqq5p593mc+6fPOOekdGatywZ8v7VM/q9hjHllASVljsm3Dytvv2lvLocLiunTuXWj930wr5iW\niTERcxewwl0kxDnnMDPmbNjL+0t3MOH7fQAoKC5lza7DXPHMnFrff9clJ/Ho9HW1tmlql57anmkr\nj3638O+fncXAbqlkjP0AgNdvGUzrpFg6t0kkKS6GO95YzAW92hEdZdzxxhJ+Oqwbf/t8E3N/dwEn\ntEqs8vl7DhUw8KGPueuSkyJmyKfCXSTMvTDra9JbxPO9/p3L15WVOca9u5zrB2fQu2NLVu04xHee\n/pzv9DmBd5fs8LDaul0/+ERe/fKbGrcf+9Dz0+6fxpHCo9f7J47O4uLe7XHOcf/klVx1Rufy3wxK\nSsuIjrJ69e6/2ryfdbsP86NBJx7nkTQthbuIVLIjJ5/Z67PplpbC1c/PZcKVpzP27eXl2zPbpbB+\nzxEPK6zdTUMzOCcznZho4+Uvvql2tNE/bxlEt7Rkznr4E1okxPDRL8+lZWIMve+bxt2Xnswvzq98\nd29OXhF7jxz9fmDVjkNVRgWt232YlgmxdGiV0MRHWD8KdxGp07eXRzZPGEVeUQl/nrG+fAZNgPd+\nMZQnP17PkB5t+cMHzTcVQ1NaNf5Sdh8q5LHpa/nPsp0AvH3bEHqf0JJe935Y3m7p/ZfQKjG20n+j\n4/H+0h38Z9kO/vzD/uVzGTWGwl1E6nQgt4ji0jLatTzaK80+XMjv313OEz/sV2n8e2FJKc99tpEn\nZviu42+eMIqt+/M4+5FPAeiamhQSI32O13f7duSqMzozetJ87rusNzcP8921nFtYghmMeHI2f7qq\nLwO7pZa/p+JvAo9f3ZcrB3Su9rMbor7hrlvNRCJYm+S4KuvSW8Tz/PVVsyM+Jpo7Lsqka9tEjhT4\nrnV3SU3i/duHces/FvL+7cMoLCll/ub9vLlgGzP9o2be+vkQnpv5NR+t8v6mrcaYvHQHk5f6vrcY\n/59VnNg2iS6pSVzyxCzat4xn96FCfvLyVzz43VN5bubX3Dy0G+kVpqeIaubRPOq5i0iz2JGTz9b9\neczZsJefnduDU++f5nVJnrhpaAb3f+fU436/eu4iElQ6tk6kY+tEBnX33al77knpREcZz/34DErL\nHNmHC+naNontOfkMnfAJAKd2bMldl5xcPrb/7Mw0Zq/fC8Ds35zPXz/bwOvzt3pzQMfp73M2M6xn\nGhee0r5J96Oeu4gEnfW7D/P6/K3cM7IXMdFRlJU5CkpKSYyNZtb6vezIyefagV1xzvHx6j38+eN1\nrNh+qPzyyKQbs7j5pdrzpW1yHJNuPJPL67hfoCmkxMew4sFLj+u9+kJVRCLGwbxiVu86xODuR+fv\n2XOogOIyR6vEWDbvzeW0Tq3KR74s+P1FpKX4roc/OWM9T8xYx4W92vHijWey+1ABy7cdJCkumm/2\n5/Hlxn1c0b8TKfExPD9zI09f15+Jszby2EfHf4NY386teK/CHbsNoXAXETlGTl4RRwpL6NwmqXzd\nkcISHvlwDWNH9KoyO2ZNDhUU0+eB6QAsuvdi7vzXEmaty+a/LujJry4+iakrdnHba4sAeP/2YXzn\n6c8rvX/ePRfSvuXxjZvXNXcRkWO0ToqjdVLlEUIp8TGMv/y0Bn1Oy4RYnrymH6nJcaQmx/HKzQMr\nbY+POTrh7mmdWnLuSencOCSD83u1O/7iG0g9dxGRACstczw6fS23nN2d1GqGmzaGeu4iIh6JjjJ+\nO7yXpzXoYR0iImFI4S4iEoYU7iIiYahe4W5mw81srZltMLOx1Ww3M3vKv32ZmQ0IfKkiIlJfdYa7\nmUUDzwAjgN7AtWbW+5hmI4BM/88Y4NkA1ykiIg1Qn577QGCDc26jc64IeAO4/Jg2lwOvOJ8vgdZm\ndkKAaxURkXqqT7h3AirOzLPNv66hbTCzMWa2wMwWZGdnN7RWERGpp2b9QtU594JzLss5l5Went6c\nuxYRiSj1uYlpO9ClwuvO/nUNbVPJwoUL95pZzU/DrV0asPc43xtsdCzBKVyOJVyOA3Qs36rXk7vr\nE+5fAZlm1g1fYF8DXHdMm8nA7Wb2BjAIOOic21nbhzrnjrvrbmYL6nP7bSjQsQSncDmWcDkO0LE0\nVJ3h7pwrMbPbgWlANDDJObfSzG71b38OmAKMBDYAecBNTVeyiIjUpV5zyzjnpuAL8Irrnquw7IBf\nBLY0ERE5XqF6h+oLXhcQQDqW4BQuxxIuxwE6lgbxbMpfERFpOqHacxcRkVqEXLjXNc9NsDGzzWa2\n3MyWmNns01SsAAADaUlEQVQC/7pUM/vIzNb7/2xTof3v/Me21syO7wm6AWJmk8xsj5mtqLCuwbWb\n2Rn+/wYb/HMQWZAcywNmtt1/bpaY2chgPxYz62Jmn5rZKjNbaWZ3+NeH3Hmp5VhC8bwkmNl8M1vq\nP5YH/eu9Oy/OuZD5wTda52ugOxAHLAV6e11XHTVvBtKOWfcIMNa/PBb4X/9yb/8xxQPd/Mca7WHt\n5wADgBWNqR2YDwwGDJgKjAiSY3kAuKuatkF7LMAJwAD/cgtgnb/ekDsvtRxLKJ4XA1L8y7HAPH89\nnp2XUOu512eem1BwOfCyf/ll4IoK699wzhU65zbhG1o6sJr3Nwvn3Cxg/zGrG1S7+eYYaumc+9L5\n/s99pcJ7mk0Nx1KToD0W59xO59wi//JhYDW+qT5C7rzUciw1CeZjcc65I/6Xsf4fh4fnJdTCvV5z\n2AQZB8wws4VmNsa/rr07epPXLqC9fzkUjq+htXfyLx+7Plj8l/mmqZ5U4VfmkDgWM8sA+uPrJYb0\neTnmWCAEz4uZRZvZEmAP8JFzztPzEmrhHoqGOef64ZsW+Rdmdk7Fjf5/nUNyyFIo1+73LL5LfP2A\nncBj3pZTf2aWArwF3OmcO1RxW6idl2qOJSTPi3Ou1P93vTO+Xvhpx2xv1vMSauHe4DlsvOac2+7/\ncw/wDr7LLLv9v37h/3OPv3koHF9Da9/uXz52veecc7v9fyHLgIkcvQQW1MdiZrH4wvA159zb/tUh\neV6qO5ZQPS/fcs7lAJ8Cw/HwvIRauJfPc2NmcfjmuZnscU01MrNkM2vx7TJwCbACX803+JvdALzn\nX54MXGNm8eabyycT35crwaRBtft/JT1kZoP93/qPrvAeT1nlZw58D9+5gSA+Fv9+XwRWO+cer7Ap\n5M5LTccSoucl3cxa+5cTgYuBNXh5XprzG+VA/OCbw2Ydvm+Xx3ldTx21dsf3jfhSYOW39QJtgY+B\n9cAMILXCe8b5j20tHowqOab+1/H9WlyM79rfT46ndiAL31/Qr4Gn8d88FwTH8iqwHFjm/8t2QrAf\nCzAM36/2y4Al/p+RoXheajmWUDwvfYDF/ppXAPf513t2XnSHqohIGAq1yzIiIlIPCncRkTCkcBcR\nCUMKdxGRMKRwFxEJQwp3EZEwpHAXEQlDCncRkTD0/wNUeX4fMLUzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11c53aeb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.plot(loss_track)\n",
    "print('loss {:.4f} after {} examples (batch_size={})'.format(loss_track[-1], len(loss_track)*batch_size, batch_size))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}