IG_Intro_to_NN_4_(HW)_work.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "id": "l4VAzKlG8aHj",
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "source": [
    "## Neural Part Of Speech Tagging\n",
    "\n",
    "We're now going to solve the same problem of POS tagging with neural networks.\n",
    "<img src=https://i.stack.imgur.com/6pdIT.png width=320>\n",
    "\n",
    "From deep learning perspective, this is a task of predicting a sequence of outputs aligned to a sequence of inputs. There are several problems that match this formulation:\n",
    "* Part Of Speech Tagging -  an auxuliary task for many NLP problems\n",
    "* Named Entity Recognition - for chat bots and web crawlers\n",
    "* Protein structure prediction - for bioinformatics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "#import sklearn\n",
    "#from tqdm import tqdm_notebook\n",
    "import nltk\n",
    "import sys\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wnu9D3YZoH2e",
    "outputId": "1eeda8c6-e6ce-49a6-f15d-3a68a756a8e2"
   },
   "outputs": [],
   "source": [
    "#%tensorflow_version 2.x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yxFUBtb88aHq",
    "outputId": "27838e37-f152-4fa5-fd75-39acf836e7c8",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package brown to\n",
      "[nltk_data]     C:\\Users\\grebe\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package brown is already up-to-date!\n",
      "[nltk_data] Downloading package universal_tagset to\n",
      "[nltk_data]     C:\\Users\\grebe\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package universal_tagset is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "nltk.download('brown')\n",
    "nltk.download('universal_tagset') #загрузка тегов по каждой части речи\n",
    "data = nltk.corpus.brown.tagged_sents(tagset='universal') #присвоение словам корпуса brown тегов частей речи\n",
    "all_tags = ['#EOS#','#UNK#','ADV', 'NOUN', 'ADP', 'PRON', 'DET', '.', 'PRT', 'VERB', 'X', 'NUM', 'CONJ', 'ADJ']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.array([[(word.lower(),tag) for word,tag in sentence] for sentence in data ], dtype = object)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "Bq_OXuD38aHs"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "train_data, test_data = train_test_split(data,test_size=0.25,random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 157
    },
    "id": "sAlXrDmU8aHs",
    "outputId": "423e76c6-966b-4364-92cf-b96796510cc9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><td>DET</td><td>NOUN</td><td>NOUN</td><td>ADJ</td><td>NOUN</td><td>VERB</td><td>NOUN</td><td>DET</td><td>NOUN</td><td>ADP</td><td>NOUN</td><td>ADJ</td><td>NOUN</td><td>NOUN</td><td>VERB</td><td>.</td><td>DET</td><td>NOUN</td><td>.</td><td>ADP</td><td>DET</td><td>NOUN</td><td>VERB</td><td>NOUN</td><td>.</td></tr><tr><td>the</td><td>fulton</td><td>county</td><td>grand</td><td>jury</td><td>said</td><td>friday</td><td>an</td><td>investigation</td><td>of</td><td>atlanta's</td><td>recent</td><td>primary</td><td>election</td><td>produced</td><td>``</td><td>no</td><td>evidence</td><td>''</td><td>that</td><td>any</td><td>irregularities</td><td>took</td><td>place</td><td>.</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import HTML, display\n",
    "def draw(sentence):\n",
    "    words,tags = zip(*sentence)\n",
    "    #display(HTML('<table><tr>{tags}</tr>{words}<tr></table>'.format(\n",
    "    display(HTML('<table><tr>{tags}</tr><tr>{words}</tr></table>'.format(\n",
    "                words = '<td>{}</td>'.format('</td><td>'.join(words)),\n",
    "                tags = '<td>{}</td>'.format('</td><td>'.join(tags)))))\n",
    "\n",
    "\n",
    "draw(data[0])\n",
    "#draw(data[10])\n",
    "#draw(data[7])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LAViiL2C8aHt"
   },
   "source": [
    "### Building vocabularies\n",
    "\n",
    "Just like before, we have to build a mapping from tokens to integer ids. This time around, our model operates on a word level, processing one word per RNN step. This means we'll have to deal with far larger vocabulary.\n",
    "\n",
    "Luckily for us, we only receive those words as input i.e. we don't have to predict them. This means we can have a large vocabulary for free by using word embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZXK_k-mo8aHt",
    "outputId": "beea459d-536f-43f0-b5fe-b45f98cca8b1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coverage = 0.92876\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter # helps to count objects and create dictionaries {'object':'quantity'}\n",
    "word_counts = Counter()\n",
    "for sentence in data:\n",
    "    words,tags = zip(*sentence)\n",
    "    word_counts.update(words)\n",
    "\n",
    "all_words = ['#EOS#','#UNK#'] + list(list(zip(*word_counts.most_common(10000)))[0]) #List the n most common elements and their counts\n",
    "\n",
    "#let's measure what fraction of data words are in the dictionary\n",
    "print(\"Coverage = %.5f\" % (float(sum(word_counts[w] for w in all_words)) / sum(word_counts.values())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "T0hee8L88aHt"
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "word_to_id = defaultdict(lambda:1, { word: i for i, word in enumerate(all_words) })\n",
    "tag_to_id = { tag: i for i, tag in enumerate(all_tags)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RCmGbwpP8aHu"
   },
   "source": [
    "convert words and tags into fixed-size matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "X7kx6jWn8aHu"
   },
   "outputs": [],
   "source": [
    "def to_matrix(lines, token_to_id, max_len=None, pad=0, dtype='int32', time_major=False):\n",
    "    \"\"\"Converts a list of names into rnn-digestable matrix with paddings added after the end\"\"\"\n",
    "\n",
    "    max_len = max_len or max(map(len,lines))\n",
    "    matrix = np.empty([len(lines), max_len],dtype)\n",
    "    matrix.fill(pad)\n",
    "\n",
    "    for i in range(len(lines)):\n",
    "        line_ix = list(map(token_to_id.__getitem__,lines[i]))[:max_len]\n",
    "        matrix[i,:len(line_ix)] = line_ix\n",
    "\n",
    "    return matrix.T if time_major else matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BCaE-i5u8aHu",
    "outputId": "8c09c604-59ea-4449-9773-de8d1988bdf5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word ids:\n",
      "[[   2 3057    5    2 2238 1334 4238 2454    3    6   19   26 1070   69\n",
      "     8 2088    6    3    1    3  266   65  342    2    1    3    2  315\n",
      "     1    9   87  216 3322   69 1558    4    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0]\n",
      " [  45   12    8  511 8419    6   60 3246   39    2    1    1    3    2\n",
      "   845    1    3    1    3   10 9910    2    1 3470    9   43    1    1\n",
      "     3    6    2 1046  385   73 4562    3    9    2    1    1 3250    3\n",
      "    12   10    2  861 5240   12    8 8936  121    1    4]\n",
      " [  33   64   26   12  445    7 7346    9    8 3337    3    1 2811    3\n",
      "     2  463  572    2    1    1 1649   12    1    4    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0]]\n",
      "Tag ids:\n",
      "[[ 6  3  4  6  3  3  9  9  7 12  4  5  9  4  6  3 12  7  9  7  9  8  4  6\n",
      "   3  7  6 13  3  4  6  3  9  4  3  7  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0]\n",
      " [ 5  9  6  9  3 12  6  3  7  6 13  3  7  6 13  3  7 13  7  5  9  6  3  3\n",
      "   4  6 13  3  7 12  6  3  6 13  3  7  4  6  3  9  3  7  9  4  6 13  3  9\n",
      "   6  3  2 13  7]\n",
      " [ 4  6  5  9 13  4  3  4  6 13  7 13  3  7  6  3  4  6 13  3  3  9  9  7\n",
      "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0]]\n"
     ]
    }
   ],
   "source": [
    "batch_words, batch_tags = zip(*[zip(*sentence) for sentence in data[-3:]])\n",
    "\n",
    "print(\"Word ids:\")\n",
    "print(to_matrix(batch_words, word_to_id))\n",
    "print(\"Tag ids:\")\n",
    "print(to_matrix(batch_tags, tag_to_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "id": "_oK_i5Xa8aHv",
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "source": [
    "### Build model\n",
    "\n",
    "Unlike our previous lab, this time we'll focus on a high-level keras interface to recurrent neural networks. It is as simple as you can get with RNN, allbeit somewhat constraining for complex tasks like seq2seq.\n",
    "\n",
    "By default, all keras RNNs apply to a whole sequence of inputs and produce a sequence of hidden states `(return_sequences=True` or just the last hidden state `(return_sequences=False)`. All the recurrence is happening under the hood.\n",
    "\n",
    "At the top of our model we need to apply a Dense layer to each time-step independently. As of now, by default keras.layers.Dense would apply once to all time-steps concatenated. We use __keras.layers.TimeDistributed__ to modify Dense layer so that it would apply across both batch and time axes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "OQeb8d5j8aHv"
   },
   "outputs": [],
   "source": [
    "import keras\n",
    "import keras.layers as L\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(L.InputLayer([None],dtype='int32'))\n",
    "model.add(L.Embedding(len(all_words),50))\n",
    "model.add(L.SimpleRNN(64,return_sequences=True))\n",
    "\n",
    "#add top layer that predicts tag probabilities\n",
    "stepwise_dense = L.Dense(len(all_tags),activation='softmax')\n",
    "stepwise_dense = L.TimeDistributed(stepwise_dense)\n",
    "model.add(stepwise_dense)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lQbJqM2n8aHv"
   },
   "source": [
    "__Training:__ in this case we don't want to prepare the whole training dataset in advance. The main cause is that the length of every batch depends on the maximum sentence length within the batch. This leaves us two options: use custom training code as in previous seminar or use generators.\n",
    "\n",
    "Keras models have a __`model.fit_generator`__ method that accepts a python generator yielding one batch at a time. But first we need to implement such generator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "kpeMsDi18aHw"
   },
   "outputs": [],
   "source": [
    "import tensorflow\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "BATCH_SIZE=32\n",
    "def generate_batches(sentences,batch_size=BATCH_SIZE,max_len=None,pad=0):\n",
    "    assert isinstance(sentences,np.ndarray),\"Make sure sentences is q numpy array\"\n",
    "\n",
    "    while True:\n",
    "        indices = np.random.permutation(np.arange(len(sentences)))\n",
    "        for start in range(0,len(indices)-1,batch_size):\n",
    "            batch_indices = indices[start:start+batch_size]\n",
    "            batch_words,batch_tags = [],[]\n",
    "            for sent in sentences[batch_indices]:\n",
    "                words,tags = zip(*sent)\n",
    "                batch_words.append(words)\n",
    "                batch_tags.append(tags)\n",
    "\n",
    "            batch_words = to_matrix(batch_words,word_to_id,max_len,pad)\n",
    "            batch_tags = to_matrix(batch_tags,tag_to_id,max_len,pad)\n",
    "\n",
    "            batch_tags_1hot = to_categorical(batch_tags,len(all_tags)).reshape(batch_tags.shape+(-1,))\n",
    "            yield batch_words,batch_tags_1hot\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zYoR9vgn8aHw"
   },
   "source": [
    "__Callbacks:__ Another thing we need is to measure model performance. The tricky part is not to count accuracy after sentence ends (on padding) and making sure we count all the validation data exactly once.\n",
    "\n",
    "While it isn't impossible to persuade Keras to do all of that, we may as well write our own callback that does that.\n",
    "Keras callbacks allow you to write a custom code to be ran once every epoch or every minibatch. We'll define one via LambdaCallback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "CC8woNtV8aHx"
   },
   "outputs": [],
   "source": [
    "def compute_test_accuracy(model):\n",
    "    test_words,test_tags = zip(*[zip(*sentence) for sentence in test_data])\n",
    "    test_words,test_tags = to_matrix(test_words,word_to_id),to_matrix(test_tags,tag_to_id)\n",
    "\n",
    "    #predict tag probabilities of shape [batch,time,n_tags]\n",
    "    predicted_tag_probabilities = model.predict(test_words,verbose=1)\n",
    "    predicted_tags = predicted_tag_probabilities.argmax(axis=-1)\n",
    "\n",
    "    #compute accurary excluding padding\n",
    "    numerator = np.sum(np.logical_and((predicted_tags == test_tags),(test_words != 0)))\n",
    "    denominator = np.sum(test_words != 0)\n",
    "    return float(numerator)/denominator\n",
    "\n",
    "\n",
    "class EvaluateAccuracy(keras.callbacks.Callback):\n",
    "    def on_epoch_end(self,epoch,logs=None):\n",
    "        sys.stdout.flush()\n",
    "        print(\"\\nMeasuring validation accuracy...\")\n",
    "        acc = compute_test_accuracy(self.model)\n",
    "        print(\"\\nValidation accuracy: %.5f\\n\"%acc)\n",
    "        sys.stdout.flush()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 473
    },
    "id": "5eJGEWu58aHx",
    "outputId": "b9077914-92f5-44ed-8a3c-42a2637af0a3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grebe\\AppData\\Local\\Temp\\ipykernel_14724\\1670692242.py:3: UserWarning: `Model.fit_generator` is deprecated and will be removed in a future version. Please use `Model.fit`, which supports generators.\n",
      "  model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1341/1343 [============================>.] - ETA: 0s - loss: 0.2518\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 4s 10ms/step\n",
      "\n",
      "Validation accuracy: 0.93995\n",
      "\n",
      "1343/1343 [==============================] - 27s 19ms/step - loss: 0.2515\n",
      "Epoch 2/5\n",
      "1341/1343 [============================>.] - ETA: 0s - loss: 0.0582\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 4s 10ms/step\n",
      "\n",
      "Validation accuracy: 0.94483\n",
      "\n",
      "1343/1343 [==============================] - 25s 19ms/step - loss: 0.0582\n",
      "Epoch 3/5\n",
      "1344/1343 [==============================] - ETA: 0s - loss: 0.0515\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 4s 10ms/step\n",
      "\n",
      "Validation accuracy: 0.94617\n",
      "\n",
      "1343/1343 [==============================] - 25s 19ms/step - loss: 0.0515\n",
      "Epoch 4/5\n",
      "1344/1343 [==============================] - ETA: 0s - loss: 0.0471\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 5s 11ms/step\n",
      "\n",
      "Validation accuracy: 0.94630\n",
      "\n",
      "1343/1343 [==============================] - 25s 19ms/step - loss: 0.0471\n",
      "Epoch 5/5\n",
      "1342/1343 [============================>.] - ETA: 0s - loss: 0.0430\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 5s 10ms/step\n",
      "\n",
      "Validation accuracy: 0.94550\n",
      "\n",
      "1343/1343 [==============================] - 25s 19ms/step - loss: 0.0430\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x1b9c6d820d0>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile('adam','categorical_crossentropy')\n",
    "\n",
    "model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,\n",
    "                    callbacks=[EvaluateAccuracy()], epochs=5,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TTN7C34V8aHy"
   },
   "source": [
    "Measure final accuracy on the whole test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "tHgxnYB68aHy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "448/448 [==============================] - 4s 10ms/step\n",
      "Final accuracy: 0.94550\n"
     ]
    }
   ],
   "source": [
    "acc = compute_test_accuracy(model)\n",
    "print(\"Final accuracy: %.5f\"%acc)\n",
    "\n",
    "assert acc>0.94, \"Keras has gone on a rampage again, please contact course staff.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5L5Prr4I8aHy"
   },
   "source": [
    "### Going bidirectional\n",
    "\n",
    "Since we're analyzing a full sequence, it's legal for us to look into future data.\n",
    "\n",
    "A simple way to achieve that is to go both directions at once, making a __bidirectional RNN__.\n",
    "\n",
    "In Keras you can achieve that both manually (using two LSTMs and Concatenate) and by using __`keras.layers.Bidirectional`__.\n",
    "\n",
    "This one works just as `TimeDistributed` we saw before: you wrap it around a recurrent layer (SimpleRNN now and LSTM/GRU later) and it actually creates two layers under the hood.\n",
    "\n",
    "Your first task is to use such a layer our POS-tagger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "xWfCrbh-8aHy"
   },
   "outputs": [],
   "source": [
    "#Define a model that utilizes bidirectional SimpleRNN\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(L.InputLayer([None],dtype='int32'))\n",
    "model.add(L.Embedding(len(all_words),50))\n",
    "model.add(L.Bidirectional(L.SimpleRNN(64,return_sequences=True)))\n",
    "model.add(L.TimeDistributed(L.Dense(len(all_tags),activation='softmax')))\n",
    "\n",
    "\n",
    "#<Your code here!>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "Ort64W348aHz"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grebe\\AppData\\Local\\Temp\\ipykernel_14724\\1670692242.py:3: UserWarning: `Model.fit_generator` is deprecated and will be removed in a future version. Please use `Model.fit`, which supports generators.\n",
      "  model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1344/1343 [==============================] - ETA: 0s - loss: 0.1919\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 7s 15ms/step\n",
      "\n",
      "Validation accuracy: 0.95513\n",
      "\n",
      "1343/1343 [==============================] - 37s 26ms/step - loss: 0.1919\n",
      "Epoch 2/5\n",
      "1344/1343 [==============================] - ETA: 0s - loss: 0.0428\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 6s 14ms/step\n",
      "\n",
      "Validation accuracy: 0.96061\n",
      "\n",
      "1343/1343 [==============================] - 34s 25ms/step - loss: 0.0428\n",
      "Epoch 3/5\n",
      "1343/1343 [============================>.] - ETA: 0s - loss: 0.0355\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 7s 15ms/step\n",
      "\n",
      "Validation accuracy: 0.96131\n",
      "\n",
      "1343/1343 [==============================] - 34s 26ms/step - loss: 0.0355\n",
      "Epoch 4/5\n",
      "1344/1343 [==============================] - ETA: 0s - loss: 0.0300\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 7s 15ms/step\n",
      "\n",
      "Validation accuracy: 0.96224\n",
      "\n",
      "1343/1343 [==============================] - 34s 25ms/step - loss: 0.0300\n",
      "Epoch 5/5\n",
      "1342/1343 [============================>.] - ETA: 0s - loss: 0.0255\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 7s 15ms/step\n",
      "\n",
      "Validation accuracy: 0.96157\n",
      "\n",
      "1343/1343 [==============================] - 34s 25ms/step - loss: 0.0255\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x1b9cc1aa110>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile('adam','categorical_crossentropy')\n",
    "\n",
    "model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,\n",
    "                    callbacks=[EvaluateAccuracy()], epochs=5,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "iWHSkF648aHz"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "448/448 [==============================] - 6s 14ms/step\n",
      "\n",
      "Final accuracy: 0.96157\n",
      "Well done!\n"
     ]
    }
   ],
   "source": [
    "acc = compute_test_accuracy(model)\n",
    "print(\"\\nFinal accuracy: %.5f\"%acc)\n",
    "\n",
    "assert acc>0.96, \"Bidirectional RNNs are better than this!\"\n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2DyAflw066m9"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y2kLQyJz7FRw"
   },
   "source": [
    "Create **at least one experiment** from the list bellow, you can choose the most interesting and promising options to improve the performance of Bidirectional LSTM:\n",
    "\n",
    "* __Go beyond SimpleRNN__: there's `keras.layers.LSTM` and `keras.layers.GRU`\n",
    "  * If you want to use a custom recurrent Cell, read [this](https://keras.io/layers/recurrent/#rnn)\n",
    "  * You can also use 1D Convolutions (`keras.layers.Conv1D`). They are often as good as recurrent layers but with less overfitting.\n",
    "* __Stack more layers__: if there is a common motif to this course it's about stacking layers\n",
    "  * You can just add recurrent and 1dconv layers on top of one another and keras will understand it\n",
    "  * Just remember that bigger networks may need more epochs to train\n",
    "* __Regularization__: you can apply dropouts as usual but also in an RNN-specific way\n",
    "  * `keras.layers.Dropout` works inbetween RNN layers\n",
    "  * Recurrent layers also have `recurrent_dropout` parameter\n",
    "* __Gradient clipping__: If your training isn't as stable as you'd like, set `clipnorm` in your optimizer.\n",
    "  * Which is to say, it's a good idea to watch over your loss curve at each minibatch. Try tensorboard callback or something similar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Stack more Layers**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "TF1T00yr8RvM"
   },
   "outputs": [],
   "source": [
    "#<Your code here!>\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(L.InputLayer([None],dtype='int32'))\n",
    "model.add(L.Embedding(len(all_words),50))\n",
    "\n",
    "model.add(L.Bidirectional(L.SimpleRNN(64,return_sequences=True), merge_mode='concat'))\n",
    "model.add(L.Bidirectional(L.SimpleRNN(64,return_sequences=True), merge_mode='concat'))\n",
    "\n",
    "model.add(L.Conv1D(64, kernel_size=3, padding='same', activation='relu'))\n",
    "\n",
    "model.add(L.TimeDistributed(L.Dense(len(all_tags),activation='softmax')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "V2T0SMV37eSP"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grebe\\AppData\\Local\\Temp\\ipykernel_14724\\3444266715.py:3: UserWarning: `Model.fit_generator` is deprecated and will be removed in a future version. Please use `Model.fit`, which supports generators.\n",
      "  model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1343/1343 [============================>.] - ETA: 0s - loss: 0.1277\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 15s 32ms/step\n",
      "\n",
      "Validation accuracy: 0.95858\n",
      "\n",
      "1343/1343 [==============================] - 70s 50ms/step - loss: 0.1277\n",
      "Epoch 2/4\n",
      "1344/1343 [==============================] - ETA: 0s - loss: 0.0397\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 14s 31ms/step\n",
      "\n",
      "Validation accuracy: 0.96043\n",
      "\n",
      "1343/1343 [==============================] - 66s 49ms/step - loss: 0.0397\n",
      "Epoch 3/4\n",
      "1344/1343 [==============================] - ETA: 0s - loss: 0.0316\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 14s 31ms/step\n",
      "\n",
      "Validation accuracy: 0.96213\n",
      "\n",
      "1343/1343 [==============================] - 66s 49ms/step - loss: 0.0316\n",
      "Epoch 4/4\n",
      "1344/1343 [==============================] - ETA: 0s - loss: 0.0259\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 13s 30ms/step\n",
      "\n",
      "Validation accuracy: 0.96152\n",
      "\n",
      "1343/1343 [==============================] - 64s 47ms/step - loss: 0.0259\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x1b9d05efb90>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile('adam','categorical_crossentropy')\n",
    "\n",
    "model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,\n",
    "                    callbacks=[EvaluateAccuracy()], epochs=4,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "448/448 [==============================] - 13s 30ms/step\n",
      "\n",
      "Final accuracy: 0.96152\n",
      "Well done!\n"
     ]
    }
   ],
   "source": [
    "acc = compute_test_accuracy(model)\n",
    "print(\"\\nFinal accuracy: %.5f\"%acc)\n",
    "\n",
    "assert acc>0.96, \"Bidirectional RNNs are better than this!\"\n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Stack more Layers + Regularization**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#<Your code here!>\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(L.InputLayer([None],dtype='int32'))\n",
    "model.add(L.Embedding(len(all_words),50))\n",
    "\n",
    "model.add(L.Bidirectional(L.SimpleRNN(64,return_sequences=True, dropout=0.2, recurrent_dropout=0.2), merge_mode='concat'))\n",
    "model.add(L.Bidirectional(L.SimpleRNN(64,return_sequences=True,dropout=0.2, recurrent_dropout=0.2), merge_mode='concat'))\n",
    "\n",
    "model.add(L.Conv1D(64, kernel_size=3, padding='same', activation='relu'))\n",
    "\n",
    "model.add(L.TimeDistributed(L.Dense(len(all_tags),activation='softmax')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grebe\\AppData\\Local\\Temp\\ipykernel_14724\\1611267849.py:3: UserWarning: `Model.fit_generator` is deprecated and will be removed in a future version. Please use `Model.fit`, which supports generators.\n",
      "  model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1344/1343 [==============================] - ETA: 0s - loss: 0.2055\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 14s 31ms/step\n",
      "\n",
      "Validation accuracy: 0.95424\n",
      "\n",
      "1343/1343 [==============================] - 86s 61ms/step - loss: 0.2055\n",
      "Epoch 2/7\n",
      "1343/1343 [============================>.] - ETA: 0s - loss: 0.0500\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 14s 32ms/step\n",
      "\n",
      "Validation accuracy: 0.96152\n",
      "\n",
      "1343/1343 [==============================] - 84s 63ms/step - loss: 0.0500\n",
      "Epoch 3/7\n",
      "1343/1343 [============================>.] - ETA: 0s - loss: 0.0421\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 14s 32ms/step\n",
      "\n",
      "Validation accuracy: 0.96339\n",
      "\n",
      "1343/1343 [==============================] - 82s 61ms/step - loss: 0.0421\n",
      "Epoch 4/7\n",
      "1343/1343 [============================>.] - ETA: 0s - loss: 0.0377\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 13s 30ms/step\n",
      "\n",
      "Validation accuracy: 0.96539\n",
      "\n",
      "1343/1343 [==============================] - 81s 60ms/step - loss: 0.0377\n",
      "Epoch 5/7\n",
      "1343/1343 [============================>.] - ETA: 0s - loss: 0.0347\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 16s 35ms/step\n",
      "\n",
      "Validation accuracy: 0.96519\n",
      "\n",
      "1343/1343 [==============================] - 91s 68ms/step - loss: 0.0347\n",
      "Epoch 6/7\n",
      "1344/1343 [==============================] - ETA: 0s - loss: 0.0328\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 20s 44ms/step\n",
      "\n",
      "Validation accuracy: 0.96609\n",
      "\n",
      "1343/1343 [==============================] - 118s 88ms/step - loss: 0.0328\n",
      "Epoch 7/7\n",
      "1344/1343 [==============================] - ETA: 0s - loss: 0.0306\n",
      "Measuring validation accuracy...\n",
      "448/448 [==============================] - 19s 43ms/step\n",
      "\n",
      "Validation accuracy: 0.96647\n",
      "\n",
      "1343/1343 [==============================] - 118s 88ms/step - loss: 0.0306\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x1b9ded07850>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile('adam','categorical_crossentropy')\n",
    "\n",
    "model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,\n",
    "                    callbacks=[EvaluateAccuracy()], epochs=7,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "448/448 [==============================] - 16s 36ms/step\n",
      "\n",
      "Final accuracy: 0.96647\n",
      "Well done!\n"
     ]
    }
   ],
   "source": [
    "acc = compute_test_accuracy(model)\n",
    "print(\"\\nFinal accuracy: %.5f\"%acc)\n",
    "\n",
    "assert acc>0.96, \"Bidirectional RNNs are better than this!\"\n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}