t81_558_class_09_4_transfer_nlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 9: Regularization: L1, L2 and Dropout**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 9 Material\n",
    "\n",
    "* Part 9.1: Introduction to Keras Transfer Learning [[Video]](https://www.youtube.com/watch?v=xyymDGReKdY&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=26) [[Notebook]](t81_558_class_09_1_keras_transfer.ipynb)\n",
    "* Part 9.2: Popular Pretrained Neural Networks for Keras [[Video]](https://www.youtube.com/watch?v=CEFcwpBneFo&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=27) [[Notebook]](t81_558_class_09_2_popular_transfer.ipynb)\n",
    "* Part 9.3: Transfer Learning for Computer Vision and Keras [[Video]](https://www.youtube.com/watch?v=JPqwyuK7bPg&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=28) [[Notebook]](t81_558_class_09_3_transfer_cv.ipynb)\n",
    "* **Part 9.4: Transfer Learning for Languages and Keras** [[Video]](https://www.youtube.com/watch?v=JPqwyuK7bPg&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=28) [[Notebook]](t81_558_class_09_4_transfer_nlp.ipynb)\n",
    "* Part 9.5: Transfer Learning for Keras Feature Engineering [[Video]](https://www.youtube.com/watch?v=JPqwyuK7bPg&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=28) [[Notebook]](t81_558_class_09_5_transfer_feature_eng.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 9.4: Transfer Learning for Languages and Keras\n",
    "\n",
    "Transfer learning is commonly used with Natural Language Processing (NLP).  This course has an entire module that covers NLP.  However, for now we will look how a NLP network can be loaded into Keras for transfer learning.  The following three sources were helpful for the creation of this section.\n",
    "\n",
    "* Cer, D., Yang, Y., Kong, S. Y., Hua, N., Limtiaco, N., John, R. S., ... & Sung, Y. H. (2018). [Universal sentence encoder](https://arxiv.org/abs/1803.11175). arXiv preprint arXiv:1803.11175.\n",
    "* [Deep Transfer Learning for Natural Language Processing — Text Classification with Universal Embeddings](https://towardsdatascience.com/deep-transfer-learning-for-natural-language-processing-text-classification-with-universal-1a2c69e5baa9)\n",
    "* [Keras Tutorial - How to Use Google's Universal Sentence Encoder for Spam Classification](http://hunterheidenreich.com/blog/google-universal-sentence-encoder-in-keras/)\n",
    "\n",
    "These examples make use of TensorFlow Hub, which allows pretrained models to easily be loaded into TensorFlow.  To install TensorHub use the following command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install tensorflow_hub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also necessary to install TensorFlow Datasets.  This can be done with the following command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install tensorflow_datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/tf2_text_classification.ipynb#scrollTo=2ew7HTbPpCJH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the Internet Movie DataBase (IMDB) reviews data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "import tensorflow_datasets as tfds\n",
    "\n",
    "train_data, test_data = tfds.load(name=\"imdb_reviews\", split=[\"train\", \"test\"], \n",
    "                                  batch_size=-1, as_supervised=True)\n",
    "\n",
    "train_examples, train_labels = tfds.as_numpy(train_data)\n",
    "test_examples, test_labels = tfds.as_numpy(test_data)\n",
    "\n",
    "# /Users/jheaton/tensorflow_datasets/imdb_reviews/plain_text/0.1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load a pretrained embedding model called [gnews-swivel-20dim](https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1).  This was trained by Google on gnews data and can convert RAW text into vectors.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = \"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1\"\n",
    "hub_layer = hub.KerasLayer(model, output_shape=[20], input_shape=[], \n",
    "                           dtype=tf.string, trainable=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the following 3 movie reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([b\"I absolutely LOVED this movie when I was a kid. I cried every time I watched it. It wasn't weird to me. I totally identified with the characters. I would love to see it again (and hope I wont be disappointed!). Pufnstuf rocks!!!! I was really drawn in to the fantasy world. And to me the movie was loooong. I wonder if I ever saw the series and have confused them? The acting I thought was strong. I loved Jack Wilde. He was so dreamy to an 10 year old (when I first saw the movie, not in 1970. I can still remember the characters vividly. The flute was totally believable and I can still 'feel' the evil woods. Witchy poo was scary - I wouldn't want to cross her path.\",\n",
       "       b'A very close and sharp discription of the bubbling and dynamic emotional world of specialy one 18year old guy, that makes his first experiences in his gay love to an other boy, during an vacation with a part of his family.<br /><br />I liked this film because of his extremly clear and surrogated storytelling , with all this \"Sound-close-ups\" and quiet moments wich had been full of intensive moods.<br /><br />',\n",
       "       b\"As a lifelong fan of Dickens, I have invariably been disappointed by adaptations of his novels.<br /><br />Although his works presented an extremely accurate re-telling of human life at every level in Victorian Britain, throughout them all was a pervasive thread of humour that could be both playful or sarcastic as the narrative dictated. In a way, he was a literary caricaturist and cartoonist. He could be serious and hilarious in the same sentence. He pricked pride, lampooned arrogance, celebrated modesty, and empathised with loneliness and poverty. It may be a clich\\xc3\\xa9, but he was a people's writer.<br /><br />And it is the comedy that is so often missing from his interpretations. At the time of writing, Oliver Twist is being dramatised in serial form on BBC television. All of the misery and cruelty is their, but non of the humour, irony, and savage lampoonery. The result is just a dark, dismal experience: the story penned by a journalist rather than a novelist. It's not really Dickens at all.<br /><br />'Oliver!', on the other hand, is much closer to the mark. The mockery of officialdom is perfectly interpreted, from the blustering beadle to the drunken magistrate. The classic stand-off between the beadle and Mr Brownlow, in which the law is described as 'a ass, a idiot' couldn't have been better done. Harry Secombe is an ideal choice.<br /><br />But the blinding cruelty is also there, the callous indifference of the state, the cold, hunger, poverty and loneliness are all presented just as surely as The Master would have wished.<br /><br />And then there is crime. Ron Moody is a treasure as the sleazy Jewish fence, whilst Oliver Reid has Bill Sykes to perfection.<br /><br />Perhaps not surprisingly, Lionel Bart - himself a Jew from London's east-end - takes a liberty with Fagin by re-interpreting him as a much more benign fellow than was Dicken's original. In the novel, he was utterly ruthless, sending some of his own boys to the gallows in order to protect himself (though he was also caught and hanged). Whereas in the movie, he is presented as something of a wayward father-figure, a sort of charitable thief rather than a corrupter of children, the latter being a long-standing anti-semitic sentiment. Otherwise, very few liberties are taken with Dickens's original. All of the most memorable elements are included. Just enough menace and violence is retained to ensure narrative fidelity whilst at the same time allowing for children' sensibilities. Nancy is still beaten to death, Bullseye narrowly escapes drowning, and Bill Sykes gets a faithfully graphic come-uppance.<br /><br />Every song is excellent, though they do incline towards schmaltz. Mark Lester mimes his wonderfully. Both his and my favourite scene is the one in which the world comes alive to 'who will buy'. It's schmaltzy, but it's Dickens through and through.<br /><br />I could go on. I could commend the wonderful set-pieces, the contrast of the rich and poor. There is top-quality acting from more British regulars than you could shake a stick at.<br /><br />I ought to give it 10 points, but I'm feeling more like Scrooge today. Soak it up with your Christmas dinner. No original has been better realised.\"],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_examples[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The embedding layer can convert each to 20-number vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=309, shape=(3, 20), dtype=float32, numpy=\n",
       "array([[ 2.0660517 , -4.491833  ,  3.6127415 , -0.6699504 , -7.310619  ,\n",
       "        -4.402696  , -4.3286967 ,  2.5536625 ,  5.506356  ,  1.5327201 ,\n",
       "        -4.539875  ,  0.2960692 ,  1.5209124 ,  1.1145658 , -6.6717415 ,\n",
       "         3.1345682 ,  5.0351944 , -2.2035742 , -3.4724264 , -0.7029331 ],\n",
       "       [ 2.2850306 , -1.3994653 ,  1.9646566 , -1.8079202 , -0.3574569 ,\n",
       "        -0.12123204, -0.82485604,  0.20197354,  2.4308808 , -1.8707466 ,\n",
       "        -1.8008713 ,  0.9650812 , -1.3530476 , -0.10591842, -2.7889197 ,\n",
       "         0.69957316,  2.0551753 , -1.6120912 , -2.8612442 , -0.7083347 ],\n",
       "       [ 3.9819887 , -4.4838037 ,  5.177359  , -2.3643482 , -3.2938678 ,\n",
       "        -3.5364532 , -2.4786978 ,  2.5525482 ,  6.688532  , -2.3076782 ,\n",
       "        -1.9807833 ,  1.1315885 , -3.0339816 , -0.7604128 , -5.743445  ,\n",
       "         3.4242578 ,  4.790099  , -4.03061   , -5.992149  , -1.7297493 ]],\n",
       "      dtype=float32)>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hub_layer(train_examples[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We add addition layers to attempt to classify the movie reviews as either positive or negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "keras_layer (KerasLayer)     (None, 20)                400020    \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 16)                336       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 17        \n",
      "=================================================================\n",
      "Total params: 400,373\n",
      "Trainable params: 400,373\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.Sequential()\n",
    "model.add(hub_layer)\n",
    "model.add(tf.keras.layers.Dense(16, activation='relu'))\n",
    "model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Compile the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split and train the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_val = train_examples[:10000]\n",
    "partial_x_train = train_examples[10000:]\n",
    "\n",
    "y_val = train_labels[:10000]\n",
    "partial_y_train = train_labels[10000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 15000 samples, validate on 10000 samples\n",
      "Epoch 1/40\n",
      "15000/15000 [==============================] - 2s 147us/sample - loss: 0.8996 - accuracy: 0.5348 - val_loss: 0.7075 - val_accuracy: 0.5826\n",
      "Epoch 2/40\n",
      "15000/15000 [==============================] - 2s 119us/sample - loss: 0.6725 - accuracy: 0.5999 - val_loss: 0.6488 - val_accuracy: 0.6217\n",
      "Epoch 3/40\n",
      "15000/15000 [==============================] - 2s 115us/sample - loss: 0.6324 - accuracy: 0.6427 - val_loss: 0.6163 - val_accuracy: 0.6603\n",
      "Epoch 4/40\n",
      "15000/15000 [==============================] - 2s 114us/sample - loss: 0.5967 - accuracy: 0.6821 - val_loss: 0.5823 - val_accuracy: 0.6916\n",
      "Epoch 5/40\n",
      "15000/15000 [==============================] - 2s 116us/sample - loss: 0.5579 - accuracy: 0.7186 - val_loss: 0.5474 - val_accuracy: 0.7224\n",
      "Epoch 6/40\n",
      "15000/15000 [==============================] - 2s 114us/sample - loss: 0.5190 - accuracy: 0.7516 - val_loss: 0.5132 - val_accuracy: 0.7492\n",
      "Epoch 7/40\n",
      "15000/15000 [==============================] - 2s 115us/sample - loss: 0.4796 - accuracy: 0.7821 - val_loss: 0.4793 - val_accuracy: 0.7753\n",
      "Epoch 8/40\n",
      "15000/15000 [==============================] - 2s 115us/sample - loss: 0.4418 - accuracy: 0.8053 - val_loss: 0.4487 - val_accuracy: 0.7949\n",
      "Epoch 9/40\n",
      "15000/15000 [==============================] - 2s 113us/sample - loss: 0.4063 - accuracy: 0.8273 - val_loss: 0.4229 - val_accuracy: 0.8092\n",
      "Epoch 10/40\n",
      "15000/15000 [==============================] - 2s 115us/sample - loss: 0.3743 - accuracy: 0.8451 - val_loss: 0.4008 - val_accuracy: 0.8205\n",
      "Epoch 11/40\n",
      "15000/15000 [==============================] - 2s 114us/sample - loss: 0.3456 - accuracy: 0.8601 - val_loss: 0.3800 - val_accuracy: 0.8327\n",
      "Epoch 12/40\n",
      "15000/15000 [==============================] - 2s 114us/sample - loss: 0.3197 - accuracy: 0.8755 - val_loss: 0.3633 - val_accuracy: 0.8409\n",
      "Epoch 13/40\n",
      "15000/15000 [==============================] - 2s 115us/sample - loss: 0.2965 - accuracy: 0.8875 - val_loss: 0.3495 - val_accuracy: 0.8485\n",
      "Epoch 14/40\n",
      "15000/15000 [==============================] - 2s 113us/sample - loss: 0.2761 - accuracy: 0.8965 - val_loss: 0.3392 - val_accuracy: 0.8539\n",
      "Epoch 15/40\n",
      "15000/15000 [==============================] - 2s 114us/sample - loss: 0.2579 - accuracy: 0.9032 - val_loss: 0.3278 - val_accuracy: 0.8604\n",
      "Epoch 16/40\n",
      "15000/15000 [==============================] - 2s 114us/sample - loss: 0.2410 - accuracy: 0.9104 - val_loss: 0.3205 - val_accuracy: 0.8630\n",
      "Epoch 17/40\n",
      "15000/15000 [==============================] - 2s 114us/sample - loss: 0.2260 - accuracy: 0.9167 - val_loss: 0.3141 - val_accuracy: 0.8652\n",
      "Epoch 18/40\n",
      "15000/15000 [==============================] - 2s 115us/sample - loss: 0.2122 - accuracy: 0.9242 - val_loss: 0.3083 - val_accuracy: 0.8685\n",
      "Epoch 19/40\n",
      "15000/15000 [==============================] - 2s 115us/sample - loss: 0.1999 - accuracy: 0.9293 - val_loss: 0.3051 - val_accuracy: 0.8726\n",
      "Epoch 20/40\n",
      "15000/15000 [==============================] - 2s 116us/sample - loss: 0.1878 - accuracy: 0.9353 - val_loss: 0.3012 - val_accuracy: 0.8726\n",
      "Epoch 21/40\n",
      "15000/15000 [==============================] - 2s 114us/sample - loss: 0.1759 - accuracy: 0.9399 - val_loss: 0.3008 - val_accuracy: 0.8749\n",
      "Epoch 22/40\n",
      "15000/15000 [==============================] - 2s 115us/sample - loss: 0.1668 - accuracy: 0.9432 - val_loss: 0.2976 - val_accuracy: 0.8762\n",
      "Epoch 23/40\n",
      "15000/15000 [==============================] - 2s 114us/sample - loss: 0.1543 - accuracy: 0.9485 - val_loss: 0.2997 - val_accuracy: 0.8753\n",
      "Epoch 24/40\n",
      "15000/15000 [==============================] - 2s 116us/sample - loss: 0.1452 - accuracy: 0.9529 - val_loss: 0.2975 - val_accuracy: 0.8776\n",
      "Epoch 25/40\n",
      "15000/15000 [==============================] - 2s 117us/sample - loss: 0.1363 - accuracy: 0.9566 - val_loss: 0.2984 - val_accuracy: 0.8781\n",
      "Epoch 26/40\n",
      "15000/15000 [==============================] - 2s 118us/sample - loss: 0.1278 - accuracy: 0.9591 - val_loss: 0.3004 - val_accuracy: 0.8780\n",
      "Epoch 27/40\n",
      "15000/15000 [==============================] - 2s 116us/sample - loss: 0.1198 - accuracy: 0.9632 - val_loss: 0.3025 - val_accuracy: 0.8773\n",
      "Epoch 28/40\n",
      "15000/15000 [==============================] - 2s 117us/sample - loss: 0.1126 - accuracy: 0.9658 - val_loss: 0.3116 - val_accuracy: 0.8751\n",
      "Epoch 29/40\n",
      "15000/15000 [==============================] - 2s 115us/sample - loss: 0.1065 - accuracy: 0.9688 - val_loss: 0.3142 - val_accuracy: 0.8741\n",
      "Epoch 30/40\n",
      "15000/15000 [==============================] - 2s 115us/sample - loss: 0.0991 - accuracy: 0.9723 - val_loss: 0.3125 - val_accuracy: 0.8766\n",
      "Epoch 31/40\n",
      "15000/15000 [==============================] - 2s 115us/sample - loss: 0.0927 - accuracy: 0.9753 - val_loss: 0.3163 - val_accuracy: 0.8758\n",
      "Epoch 32/40\n",
      "15000/15000 [==============================] - 2s 117us/sample - loss: 0.0870 - accuracy: 0.9783 - val_loss: 0.3208 - val_accuracy: 0.8752\n",
      "Epoch 33/40\n",
      "15000/15000 [==============================] - 2s 116us/sample - loss: 0.0813 - accuracy: 0.9807 - val_loss: 0.3257 - val_accuracy: 0.8743\n",
      "Epoch 34/40\n",
      "15000/15000 [==============================] - 2s 116us/sample - loss: 0.0759 - accuracy: 0.9828 - val_loss: 0.3327 - val_accuracy: 0.8747\n",
      "Epoch 35/40\n",
      "15000/15000 [==============================] - 2s 115us/sample - loss: 0.0710 - accuracy: 0.9845 - val_loss: 0.3385 - val_accuracy: 0.8739\n",
      "Epoch 36/40\n",
      "15000/15000 [==============================] - 2s 116us/sample - loss: 0.0665 - accuracy: 0.9864 - val_loss: 0.3430 - val_accuracy: 0.8735\n",
      "Epoch 37/40\n",
      "15000/15000 [==============================] - 2s 116us/sample - loss: 0.0616 - accuracy: 0.9879 - val_loss: 0.3510 - val_accuracy: 0.8719\n",
      "Epoch 38/40\n",
      "15000/15000 [==============================] - 2s 114us/sample - loss: 0.0573 - accuracy: 0.9893 - val_loss: 0.3625 - val_accuracy: 0.8711\n",
      "Epoch 39/40\n",
      "15000/15000 [==============================] - 2s 117us/sample - loss: 0.0542 - accuracy: 0.9891 - val_loss: 0.3702 - val_accuracy: 0.8706\n",
      "Epoch 40/40\n",
      "15000/15000 [==============================] - 2s 114us/sample - loss: 0.0498 - accuracy: 0.9915 - val_loss: 0.3727 - val_accuracy: 0.8695\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=40,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val),\n",
    "                    verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Evaluate the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 3s 116us/sample - loss: 0.4037 - accuracy: 0.8580\n",
      "[0.4036788517403603, 0.85804]\n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(test_data, test_labels)\n",
    "\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmYFOXV9/HvAVlE9kVEh9WVVcAR9UECuL2oEaISBMdETRTliTGJMQmPGGM0PFFjkGCIr5qEGEEJ0dd9IUZJ0CQq4AIiIi6AI4iAgiCLDpz3j7u66RlmaWampnpmfp/rqqurq6urz9TM1Ol7LXN3REREABokHYCIiOQOJQUREUlTUhARkTQlBRERSVNSEBGRNCUFERFJU1KQamVmDc1sq5l1qc59k2Rmh5lZtffdNrNTzGxlxvPlZjYkm30r8Vm/N7NrKvv+co77CzP7U3UfV5KzX9IBSLLMbGvG02bATmBX9Pwyd5+1L8dz911A8+retz5w9yOr4zhmdglwgbsPyzj2JdVxbKn7lBTqOXdPX5Sjb6KXuPvfy9rfzPZz96KaiE1Eap6qj6RcUfXAX8zsfjPbAlxgZieY2YtmtsnM1prZNDNrFO2/n5m5mXWLns+MXn/KzLaY2X/MrPu+7hu9frqZvW1mm83sdjP7l5ldVEbc2cR4mZm9Y2afmtm0jPc2NLPbzGyjmb0HjCjn/Ewys9kltk03synR+iVmtiz6ed6NvsWXdaxCMxsWrTczs3uj2JYCx5TY91ozey867lIzGxlt7wv8FhgSVc1tyDi312e8//LoZ99oZg+bWadszk1FzOzsKJ5NZvacmR2Z8do1ZrbGzD4zs7cyftbjzeyVaPs6M/tVtp8nMXB3LVpwd4CVwCkltv0C+AI4i/AlYn/gWOA4QkmzB/A2cEW0/36AA92i5zOBDUA+0Aj4CzCzEvseCGwBRkWvXQV8CVxUxs+STYyPAK2AbsAnqZ8duAJYCuQB7YD54V+l1M/pAWwFDsg49sdAfvT8rGgfA04CtgP9otdOAVZmHKsQGBat3wr8A2gDdAXeLLHvGKBT9Ds5P4qhY/TaJcA/SsQ5E7g+Wj8tirE/0BT4HfBcNuemlJ//F8CfovWeURwnRb+ja4Dl0XpvYBVwULRvd6BHtL4AGBettwCOS/p/oT4vKilINl5w98fcfbe7b3f3Be7+krsXuft7wF3A0HLe/4C7L3T3L4FZhIvRvu77VeA1d38keu02QgIpVZYx/tLdN7v7SsIFOPVZY4Db3L3Q3TcCN5XzOe8BbxCSFcCpwKfuvjB6/TF3f8+D54BngVIbk0sYA/zC3T9191WEb/+ZnzvH3ddGv5P7CAk9P4vjAhQAv3f319x9BzARGGpmeRn7lHVuyjMWeNTdn4t+RzcREstxQBEhAfWOqiDfj84dhOR+uJm1c/ct7v5Slj+HxEBJQbLxQeYTMzvKzJ4ws4/M7DPgBqB9Oe//KGN9G+U3Lpe178GZcbi7E75ZlyrLGLP6LMI33PLcB4yL1s+Pnqfi+KqZvWRmn5jZJsK39PLOVUqn8mIws4vM7PWommYTcFSWx4Xw86WP5+6fAZ8Ch2Tssy+/s7KOu5vwOzrE3ZcDPyT8Hj6OqiMPina9GOgFLDezl83sjCx/DomBkoJko2R3zDsJ344Pc/eWwHWE6pE4rSVU5wBgZkbxi1hJVYlxLdA543lFXWbnAKeY2SGEEsN9UYz7Aw8AvyRU7bQG/pZlHB+VFYOZ9QDuACYA7aLjvpVx3Iq6z64hVEmljteCUE31YRZx7ctxGxB+Zx8CuPtMdx9MqDpqSDgvuPtydx9LqCL8NfCgmTWtYixSSUoKUhktgM3A52bWE7isBj7zcWCgmZ1lZvsB3wM6xBTjHOD7ZnaImbUDflLezu7+EfAC8CdgubuviF5qAjQG1gO7zOyrwMn7EMM1ZtbawjiOKzJea0648K8n5MdLCSWFlHVAXqphvRT3A982s35m1oRwcX7e3cssee1DzCPNbFj02T8itAO9ZGY9zWx49Hnbo2U34Qf4hpm1j0oWm6OfbXcVY5FKUlKQyvghcCHhH/5OQoNwrNx9HXAeMAXYCBwKvEoYV1HdMd5BqPtfQmgEfSCL99xHaDhOVx25+ybgB8BDhMba0YTklo2fEUosK4GngD9nHHcxcDvwcrTPkUBmPfwzwApgnZllVgOl3v80oRrnoej9XQjtDFXi7ksJ5/wOQsIaAYyM2heaALcQ2oE+IpRMJkVvPQNYZqF3263Aee7+RVXjkcqxUDUrUruYWUNCdcVod38+6XhE6gqVFKTWMLMRUXVKE+CnhF4rLycclkidoqQgtcmJwHuEqon/A5zt7mVVH4lIJaj6SERE0lRSEBGRtFo3IV779u29W7duSYchIlKrLFq0aIO7l9eNG6iFSaFbt24sXLgw6TBERGoVM6toZD6g6iMREcmgpCAiImlKCiIiklbr2hREpGZ9+eWXFBYWsmPHjqRDkSw0bdqUvLw8GjUqa+qr8ikpiEi5CgsLadGiBd26dSNMTiu5yt3ZuHEjhYWFdO/eveI3lKJeVB/NmgXdukGDBuFx1j7dil6kftuxYwft2rVTQqgFzIx27dpVqVRX50sKs2bB+PGwbVt4vmpVeA5QUOV5IUXqByWE2qOqv6s6X1KYNGlPQkjZti1sFxGR4up8Uli9et+2i0hu2bhxI/3796d///4cdNBBHHLIIennX3yR3W0XLr74YpYvX17uPtOnT2dWNdUtn3jiibz22mvVcqyaVuerj7p0CVVGpW0Xkeo3a1Yoia9eHf7PJk+uWlVtu3bt0hfY66+/nubNm3P11VcX28fdcXcaNCj9e+6MGTMq/JzvfOc7lQ+yDqnzJYXJk6FZs+LbmjUL20WkeqXa8FatAvc9bXhxdO5455136NWrFwUFBfTu3Zu1a9cyfvx48vPz6d27NzfccEN639Q396KiIlq3bs3EiRM5+uijOeGEE/j4448BuPbaa5k6dWp6/4kTJzJo0CCOPPJI/v3vfwPw+eefc+6559KrVy9Gjx5Nfn5+hSWCmTNn0rdvX/r06cM111wDQFFREd/4xjfS26dNmwbAbbfdRq9evejXrx8XXHBBtZ+zbNT5kkLqG0p1fnMRkdKV14YXx//cW2+9xZ///Gfy8/MBuOmmm2jbti1FRUUMHz6c0aNH06tXr2Lv2bx5M0OHDuWmm27iqquu4o9//CMTJ07c69juzssvv8yjjz7KDTfcwNNPP83tt9/OQQcdxIMPPsjrr7/OwIEDy42vsLCQa6+9loULF9KqVStOOeUUHn/8cTp06MCGDRtYsmQJAJs2bQLglltuYdWqVTRu3Di9rabV+ZIChD/GlSth9+7wqIQgEo+absM79NBD0wkB4P7772fgwIEMHDiQZcuW8eabb+71nv3335/TTz8dgGOOOYaVK1eWeuxzzjlnr31eeOEFxo4dC8DRRx9N7969y43vpZde4qSTTqJ9+/Y0atSI888/n/nz53PYYYexfPlyrrzySubOnUurVq0A6N27NxdccAGzZs2q9OCzqqoXSUFEakZZbXVxteEdcMAB6fUVK1bwm9/8hueee47FixczYsSIUvvrN27cOL3esGFDioqKSj12kyZNKtynstq1a8fixYsZMmQI06dP57LLLgNg7ty5XH755SxYsIBBgwaxa9euav3cbCgpiEi1SbIN77PPPqNFixa0bNmStWvXMnfu3Gr/jMGDBzNnzhwAlixZUmpJJNNxxx3HvHnz2LhxI0VFRcyePZuhQ4eyfv163J2vf/3r3HDDDbzyyivs2rWLwsJCTjrpJG655RY2bNjAtpJ1cTWgzrcpiEjNSbINb+DAgfTq1YujjjqKrl27Mnjw4Gr/jO9+97t885vfpFevXuklVfVTmry8PG688UaGDRuGu3PWWWdx5pln8sorr/Dtb38bd8fMuPnmmykqKuL8889ny5Yt7N69m6uvvpoWLVpU+89QkVp3j+b8/HzXTXZEas6yZcvo2bNn0mHkhKKiIoqKimjatCkrVqzgtNNOY8WKFey3X259vy7td2Zmi9w9v4y3pOXWTyIiksO2bt3KySefTFFREe7OnXfemXMJoarq1k8jIhKj1q1bs2jRoqTDiJUamkVEJE1JQURE0pQUREQkTUlBRETSlBREJKcNHz58r4FoU6dOZcKECeW+r3nz5gCsWbOG0aNHl7rPsGHDqKiL+9SpU4sNIjvjjDOqZV6i66+/nltvvbXKx6luSgoiktPGjRvH7Nmzi22bPXs248aNy+r9Bx98MA888EClP79kUnjyySdp3bp1pY+X65QURCSnjR49mieeeCJ9Q52VK1eyZs0ahgwZkh43MHDgQPr27csjjzyy1/tXrlxJnz59ANi+fTtjx46lZ8+enH322Wzfvj2934QJE9LTbv/sZz8DYNq0aaxZs4bhw4czfPhwALp168aGDRsAmDJlCn369KFPnz7pabdXrlxJz549ufTSS+nduzennXZasc8pzWuvvcbxxx9Pv379OPvss/n000/Tn5+aSjs1Ed8///nP9E2GBgwYwJYtWyp9bkujcQoikrXvfx+q+4Zi/ftDdD0tVdu2bRk0aBBPPfUUo0aNYvbs2YwZMwYzo2nTpjz00EO0bNmSDRs2cPzxxzNy5Mgy71N8xx130KxZM5YtW8bixYuLTX09efJk2rZty65duzj55JNZvHgxV155JVOmTGHevHm0b9++2LEWLVrEjBkzeOmll3B3jjvuOIYOHUqbNm1YsWIF999/P3fffTdjxozhwQcfLPf+CN/85je5/fbbGTp0KNdddx0///nPmTp1KjfddBPvv/8+TZo0SVdZ3XrrrUyfPp3BgwezdetWmjZtug9nu2IqKYhIzsusQsqsOnJ3rrnmGvr168cpp5zChx9+yLp168o8zvz589MX5379+tGvX7/0a3PmzGHgwIEMGDCApUuXVjjZ3QsvvMDZZ5/NAQccQPPmzTnnnHN4/vnnAejevTv9+/cHyp+eG8L9HTZt2sTQoUMBuPDCC5k/f346xoKCAmbOnJkeOT148GCuuuoqpk2bxqZNm6p9RHWsJQUzGwH8BmgI/N7dbyrxehfgHqB1tM9Ed38yzphEpPLK+0Yfp1GjRvGDH/yAV155hW3btnHMMccAMGvWLNavX8+iRYto1KgR3bp1K3W67Iq8//773HrrrSxYsIA2bdpw0UUXVeo4KalptyFMvV1R9VFZnnjiCebPn89jjz3G5MmTWbJkCRMnTuTMM8/kySefZPDgwcydO5ejjjqq0rGWFFtJwcwaAtOB04FewDgz61Vit2uBOe4+ABgL/C6ueESk9mrevDnDhw/nW9/6VrEG5s2bN3PggQfSqFEj5s2bx6rSbsie4Stf+Qr33XcfAG+88QaLFy8GwrTbBxxwAK1atWLdunU89dRT6fe0aNGi1Hr7IUOG8PDDD7Nt2zY+//xzHnroIYYMGbLPP1urVq1o06ZNupRx7733MnToUHbv3s0HH3zA8OHDufnmm9m8eTNbt27l3XffpW/fvvzkJz/h2GOP5a233trnzyxPnCWFQcA77v4egJnNBkYBmWUyB1pG662ANTHGIyK12Lhx4zj77LOL9UQqKCjgrLPOom/fvuTn51f4jXnChAlcfPHF9OzZk549e6ZLHEcffTQDBgzgqKOOonPnzsWm3R4/fjwjRozg4IMPZt68eentAwcO5KKLLmLQoEEAXHLJJQwYMKDcqqKy3HPPPVx++eVs27aNHj16MGPGDHbt2sUFF1zA5s2bcXeuvPJKWrduzU9/+lPmzZtHgwYN6N27d/ouctUltqmzzWw0MMLdL4mefwM4zt2vyNinE/A3oA1wAHCKu+8125SZjQfGA3Tp0uWYir4NiEj10dTZtU9Vps5OuqF5HPAnd88DzgDuNbO9YnL3u9w9393zO3ToUONBiojUF3EmhQ+BzhnP86Jtmb4NzAFw9/8ATYH2iIhIIuJMCguAw82su5k1JjQkP1pin9XAyQBm1pOQFNbHGJOIVEJtu0NjfVbV31VsScHdi4ArgLnAMkIvo6VmdoOZjYx2+yFwqZm9DtwPXOT66xPJKU2bNmXjxo1KDLWAu7Nx48YqDWjTPZpFpFxffvklhYWFVeq3LzWnadOm5OXl0ahRo2LbdY9mEakWjRo1onv37kmHITUk6d5HIiKSQ5QUREQkTUlBRETSlBRERCRNSUFERNKUFEREJE1JQURE0pQUREQkTUlBRETSlBRERCRNSUFERNKUFEREJE1JQURE0upVUti1K+kIRERyW71JCjNmQL9+sH170pGIiOSuepMUuneHN9+EKVOSjkREJHfVm6QwbBiccw788pewZk3S0YiI5KZ6kxQAfvUr+PJLuOaapCMREclN9Sop9OgBP/gB3HMP6DbPIiJ7q1dJAUIpoWNH+P73wT3paEREcku9SwotW8LkyfCvf8GcOWHbrFnQrRs0aBAeZ81KMkIRkeTsl3QASbjoIvjtb+HHP4Zt2+CKK8IjwKpVMH58WC8oSCxEEZFE1LuSAkDDhnDbbbB6dWhjSCWElG3bYNKkZGITEUlSvUwKsKeL6ubNpb++enWNhiMikhPqbVKA0EW1LF261FwcIiK5ol4nhR494Ktf3Xt7s2ahMVpEpL6p10kBQk+jVq2gSZPwvGtXuOsuNTKLSP1U75NCy5bw61/Dzp0wezasXKmEICL1V71PChC6qPbvH7qmLl6cdDQiIslRUiB0UZ0zB5o2heHDYdGi4q9rcJuI1BdKCpHDD4f580N10sknw4svhu2zZoXBbKtWhWkxUoPblBhEpC5SUsjQvTv885/QoQOceio8/3wYxKbBbSJSXygplNClS0gMeXkwYkQoGZRGg9tEpC5SUijFwQeHxHDooWXvo8FtIlIXKSmU4cADYd680LBckga3iUhdpaRQjnbt4NVXi5cYNLhNROoyJYUKtG4Nr7wCJ54Yuq5OmaKEICJ1V6xJwcxGmNlyM3vHzCaWsc8YM3vTzJaa2X1xxlNZLVvCk0/CccfBeefBww8nHZGISDxiSwpm1hCYDpwO9ALGmVmvEvscDvwPMNjdewPfjyueqmrRAp56CvLzYcwYeOyxPa9pcJuI1BVx3nltEPCOu78HYGazgVHAmxn7XApMd/dPAdz94xjjqbKWLeHpp+G00+Dcc+Ghh2DTpjCYTXduE5G6IM7qo0OADzKeF0bbMh0BHGFm/zKzF81sRGkHMrPxZrbQzBauX78+pnCz06oVzJ0L/fqFm/RcdZUGt4lI3ZF0Q/N+wOHAMGAccLeZtS65k7vf5e757p7foUOHGg5xb61bw9/+Br17w8dllG00uE1EaqM4k8KHQOeM53nRtkyFwKPu/qW7vw+8TUgSOa9tW3jmGWjUqPTXNbhNRGqjOJPCAuBwM+tuZo2BscCjJfZ5mFBKwMzaE6qT3osxpmrVrh1MmwZmxbdrcJuI1FaxJQV3LwKuAOYCy4A57r7UzG4ws5HRbnOBjWb2JjAP+JG7b4wrpjhcfjlMn76nxNCxowa3iUjtZe6edAz7JD8/3xcuXJh0GHtZuxaGDoV16+C55+CYY5KOSERkDzNb5O75Fe2XdENzndGpEzz7LLRpE7qsLlmSdEQiIvtOSaEade4cSglNm4b7MSxfHrZrcJuI1BZKCtWsR49QYnAPd3C77TbduU1Eag8lhRgcdVTorrp9O/zoRxrcJiK1h5JCTPr1CyOfd+0q/XUNbhORXKSkEKP8/NBFtTQa3CYiuUhJIWa//jU0aVJ8mwa3iUiuUlKIWUEB/OEP4faeAI0bh8ZnDW4TkVykpFADCgrCoLYnngg9kP7wB9i8OemoRET2pqRQg844Ax54INze8/TTYcuWpCMSESlOSaGGjRwJs2fDyy/DmWeGUoMGtolIrojzzmtShnPPDRf/cePgX/+C3bvDdt21TUSSppJCQs47L9yTIZUQUjSwTUSSpKSQoE8+KX27BraJSFKUFBJU1gA2DWwTkaRklRTM7FAzaxKtDzOzK0u7l7Lsm8mTw0C2TA0bws9/nkw8IiLZlhQeBHaZ2WHAXYR7L98XW1T1REFBuEtb167hlp5t24a5kh56CL74IunoRKQ+yjYp7I5ur3k2cLu7/wjoFF9Y9UdBAaxcGRqcN26E3/4WHnkExo6FL79MOjoRqW+yTQpfmtk44ELg8Whbo3hCqt++8x2YNi2UFsaNU2IQkZqVbVK4GDgBmOzu75tZd+De+MKq3777XZg6FR58EE48MVQvaXCbiNSErAavufubwJUAZtYGaOHuN8cZWH33ve+FUc/3ZbTcaHCbiMQt295H/zCzlmbWFngFuNvMpsQbmvzrX3tv0+A2EYlTttVHrdz9M+Ac4M/ufhxwSnxhCZQ9iE2D20QkLtkmhf3MrBMwhj0NzRKzsgaxHXJIzcYhIvVHtknhBmAu8K67LzCzHsCK+MISKH1wG4TuqyotiEgcskoK7v5Xd+/n7hOi5++5+7nxhiYlB7d17QrXXgtbt8LgwfDmm0lHKCJ1TbYNzXlm9pCZfRwtD5pZXtzBSfHBbStXwo03wvz5UFQEQ4bAf/6TdIQiUpdkW300A3gUODhaHou2SQKOPjr0TGrTBk4+GZ56KumIRKSuyDYpdHD3Ge5eFC1/AjrEGJdUoEePkBgOPDDc5tNMg9tEpOqyTQobzewCM2sYLRcAG+MMTCr297/Dxx/veb5qFVx6qRKDiFRetknhW4TuqB8Ba4HRwEUxxSRZmjQJtm8vvm37dvjxj5OJR0Rqv2x7H61y95Hu3sHdD3T3rwHqfZSwsrqlrlkDTzxRs7GISN1QlTuvXVVtUUillDW4rVEj+OpXQ0li166ajUlEareqJAWrtiikUkob3NasGdx5J1xyCfzv/8Jpp8G6dcnEJyK1T1WSgldbFFIppQ1uu+suuPhiuPtumDED/v1vGDgQnnsu6WhFpDYw97Kv7Wa2hdIv/gbs7+5ZTb1dnfLz833hwoU1/bG11uuvw9e/DitWwGWXwS23QMuWSUclIjXNzBa5e35F+5VbUnD3Fu7espSlRRIJQfbdG2/Ajh1h/c47oXt3ePrpZGMSkdxVleqjCpnZCDNbbmbvmNnEcvY718zczCrMYpK9WbPCTXk++GDPtk8/hdNPD1VMn36aXGwikptiSwpm1hCYDpwO9ALGmVmvUvZrAXwPeCmuWOqrSZPCTXkyuYfqo3vvhV694JFHkolNRHJTnCWFQcA70YyqXwCzgVGl7HcjcDOwI8ZY6qWyxjFs2QILFkDHjvC1r8G4ccVHRotI/RVnUjgEyKi4oDDalmZmA4HO7l7uUCszG29mC81s4fr166s/0jqqrHEMXbrAgAEhMdx4Izz4IBx5ZOixtHt3zcYoIrkl1jaF8phZA2AK8MOK9nX3u9w9393zO3TQPHzZKmscw+TJYb1Ro3B/htdfDzOvjh8PX/lKaJwWkfopzqTwIdA543letC2lBdAH+IeZrQSOBx5VY3P1KWscQ0FB8f169oR588K4hrfeCqWIiRP3bo8Qkbqv3HEKVTqw2X7A28DJhGSwADjf3ZeWsf8/gKvdvdxBCBqnUL1mzQoN0qtXh2qliRPh5ZdDgujWDaZPD1Nzi0jtVi3jFKrC3YuAKwj3dl4GzHH3pWZ2g5mNjOtzJXupLqurVoVeSatWwQ9/GG7c849/QNOmcOaZMHp0KEGISN0XW0khLiopVJ9u3UIiKKlr13Drz5074Ve/gl/+MkzJPXZsaIPotVfHYhHJddmWFJQU6rEGDUIJoSSz4r2Q1q+HX/8afvvb0M7w9a/DT38KffrUXKwiddmGDeF+66tXw8aNe5ZPPin+fMqUMPC0MrJNCpqqoh7r0qX0kkLJrqwdOsBNN8HVV4c/yttvhzlzQrXSdddB3741E69IXbB7NyxfHm6n++9/h8e33y6+T8uW0K7dnuWww8LjEUfEH59KCvVYqk0hs5dRs2al91DKtHEj3HYbTJsWBsKNGgVXXAEnnRRKHyJS3Jo14YvU3/8eSgSffBK2t2sHgwfDf/1XeDz8cGjbNnQXr26qPpKslOx9NHly+Qkh06efwtSp8LvfheLvEUfAhAlw4YXQpk28cYvkug0bwsDQ+++H+fNDVe2RR8KJJ+5JBEccEapra4KSglRZtgljxw544IGQHP7zH9h/fzj/fPjv/w73chCpLzZvhocfhtmz4Zlnwp0PjzwyTCVz3nlw1FHJxaakIFVS2aql116DO+6AmTPDewcNCg1jI0fCwQfHH7dITdq6FV58EZ5/Hl54IbQP7NwZevaNHRuWfv1qrjRQHiUFqZKKuqtWZPNm+POfQ+khNcbh2GNDchg1KvRcyoV/FJF9sW5duPCnksCrr4bSQIMG0L8/DBsGY8aEL0O59vetpCBVkm131Yq4w9Kl8OijYZrul18O27t3Dwli5Mgw39J+6gcnOWD37tAo/O67pS+pe5A0bQrHHQdDhoQ2ghNOyP07GiopSJVUtaRQlrVr4fHHQ5J45plQ1G7fHs45J9S5Dh0KDRtW/vgi2XCHDz8Mkz+mliVLYNmyMFAzpWHD8Dd/6KFhOfzwkACOOQYaN04u/spQUpAqqWybwr74/HOYOxf++ld47LHw/MADw/iHMWPCNzAlCKmqXbvCxX7BAli4EBYvDklg06Y9+3TqFKo0e/cOPYJSSaBLl3i6hyZBSUGqrKLeR1XpzlrStm3w5JPwl7/AE0+Eb2udOsG558Lw4XD88Wqoloq5w/vvhwSQWhYtCl84AFq0CHX/ffrsWXr3DuMF6jolBYlVnCWJrVtDFdOcOSFR7NwZtuflheSQWgYODN1fpf7ZvDmMAi5t2bo17NOkSUgAxx4bGn6PPTaUAurrAEslBYlVXG0OJe3cGbq5vvhiWF56KXwThNA43a9fSA4DBoSlXz844IDq+3xJRqrB97339l7efbf47WPNwt/jEUeEOv/evUMS6NOn9tX7x0lJQWJVXb2TKmPdupAcUsurr+6ZNqBBg3BxSCWJ/v3D3EwdO+ZeF8G6Yteu0Ctnw4awbNwYznWLFtC8eXhMrTdvHtqJNm+GDz5wO54PAAANRklEQVSAwsLwmLm+enX4YvHFF3s+o0ED6NwZevQIy5FH7kkCPXqE3kBSPiUFiVVNlRSy4R4uJq++Wnz5IOMO4e3aheTQt2/4Btm3b/hGmevdCHPJ+vXw7LOh19hbb4WL/4YNISHvy2WkcePiF3wISaRTp3Dhz7z4p5a61OCbFM2SKrGaPLn0NoXU/Z9rklm4aHTpEgbGpWzYsKenyZIlYfnjH/c0OkLo7XTQQXuWjh2LP8/LCxep+th2sWNHGKD1zDNhefXVsL1161AKO/ro0J245NKuXUgSW7aE+v2Sj59/HvZLJYDOnUNC0EU/N6ikIJVWk72Tqsvu3aGEk0oUq1bBRx+FZd268Jhq2M504IF7Ek+XLqFElJcXqi0aNgxLgwZ71hs2DA2dHTuGJVfqtouKQtXNpk17L5nbly8Pk7jt2BEu1iecAKedBqeeGvroq6tw7aPqI0lUTYxziIN7uDh+9FEYaFdYGJJaalm1Kjxmljay0bZt8RLIQQeFBvGyLtCpPvRt2oT3tmmzZ0k9h1ANs3NnWDLXd+6Ezz7b+4JfUdxmoSRwyCHhtqynnhoGFDZvvu/nUnKLkoIkKpfaHKqbe2hYLSwMF+Jdu0pfduwIvWRSJZFUokk9bt8OrVqFi3Dm0qZN2J76nMzlk0/CY2ZpJlUqadw4PKbWM49dcr1Vq/A5JT+7efP622WzrlObgiRq9ep9216bmIVv623bVv4Y7mGp7AU4NRVD48aqypHqpe8EEouSt/QsbfusWaFE0aBBeJw1qyYiyw1mVftGvv/+YVFCkOqmpCCxmDw5tCFkyuydlGpzWLUqfGNetSo8r0+JQSQXKSlILAoKQqNy167hW3HXrsUbmSdNKt4IDeH5pEk1H6uI7KGGZklEkiOiReqjbBuaVVKQRGTT5iAiNU9JQRJRUZsD1O+GaJGkKClIIipqc1BDtEgy1KYgOakuD34TSYLaFKRWq2jwm6qWROKhpCA5qbyGaFUticRHSUFyUnkN0RrjIBIfJQXJSeU1RNfleZVEkqakIDmroCA0Ku/eHR5TPZM0r5JIfJQUpNbRvEoi8VFSkFpH8yqJxEdJQWqlsqqWILs2B1UviZROSUHqnIraHFS9JFI2JQWpcypqc1D1kkjZYk0KZjbCzJab2TtmNrGU168yszfNbLGZPWtmXeOMR+qHitocVL0kUrbY5j4ys4bA28CpQCGwABjn7m9m7DMceMndt5nZBGCYu59X3nE195FUVUXzKqWqlzJLE82aFU8sIrVNLsx9NAh4x93fc/cvgNnAqMwd3H2eu6f+9V4E8mKMRwRQ9ZJIeeJMCocAH2Q8L4y2leXbwFOlvWBm481soZktXL9+fTWGKPWRqpdEyrZf0gEAmNkFQD4wtLTX3f0u4C4I1Uc1GJrUUQUFZVcFdelSevVSyd5LqdJEqvdS6rgitVmcJYUPgc4Zz/OibcWY2SnAJGCku++MMR6RrKh6SeqzOJPCAuBwM+tuZo2BscCjmTuY2QDgTkJC+DjGWESyVtXqJVUtSW0WW/WRuxeZ2RXAXKAh8Ed3X2pmNwAL3f1R4FdAc+CvZgaw2t1HxhWTSLYqW72kqiWp7WIdp+DuT7r7Ee5+qLtPjrZdFyUE3P0Ud+/o7v2jRQlBcl5V7/WgkoTkMo1oFtlHVbnXg6bYkFwX2+C1uGjwmuSyigbGVfS6SFxyYfCaSL1TUc8ljYGQXKekIFKNKuq5pBlcJdcpKYhUs/Lu9VAdYyBUkpA4KSmI1KDqGAOhkoTESUlBpIaVV5KoqHpJJQmJm5KCSA6pakO1ShJSVUoKIjmkqg3VFZUkVIqQiigpiOSYqjRUl1eSUClCsqGkIFKLVKUkofYIyYaSgkgtU9mSRHW0Ryhp1H1KCiJ1SHkliepoj1D1U92npCBSx5RVkqhqzyZVP9UPSgoi9URVezap+ql+0CypIgLsfYMgCCWJVOKo6gywFR1f4qVZUkVkn1RUkoi7+kmliNygpCAiaeX1bIqz+klVT7lD1UciUi2qUv0EqnqKm6qPRKRGVaX6ST2fcoeSgohUm8pWP6nnU+5QUhCRGlPZMRQ1MfBOSSNQUhCRxOVCz6fykka9ShjuXquWY445xkWk/pk5071rV3ez8Dhz5p7XunZ1D5fz4kvXruF1s9JfN6v4/TNnujdrVnx7s2bFP7+82HIFsNCzuMYmfpHf10VJQURKqujCXZWkUdF7a0vSyDYpqPpIRGq9qlY/lddmEXfVVGqfnKmeyiZz5NKikoKIVEZ539bL+7YfZ9VURZ+dTezZQtVHIiLZK+vCG2fVVDbvzyZpZCPbpKDqIxERyu4uG2fVFFTPwL3qpKQgIlKBqswJFXfSqG5KCiIiVZRk0qhuSgoiIjGLM2lUt/3iOayIiGSroKDs2V5T2ydNClVGXbqEhBDX7LBKCiIiOa68pFHdVH0kIiJpSgoiIpKmpCAiImlKCiIikqakICIiaRamxKg9zGw9UMotvgFoD2yowXD2VS7Hp9gqR7FVjmKrnKrE1tXdO1S0U61LCuUxs4Xunp90HGXJ5fgUW+UotspRbJVTE7Gp+khERNKUFEREJK2uJYW7kg6gArkcn2KrHMVWOYqtcmKPrU61KYiISNXUtZKCiIhUgZKCiIik1ZmkYGYjzGy5mb1jZhOTjieTma00syVm9pqZLUw4lj+a2cdm9kbGtrZm9oyZrYge2+RQbNeb2YfRuXvNzM5IKLbOZjbPzN40s6Vm9r1oe+LnrpzYEj93ZtbUzF42s9ej2H4ebe9uZi9F/69/MbPGORTbn8zs/Yzz1r+mY8uIsaGZvWpmj0fP4z9v2dzIOdcXoCHwLtADaAy8DvRKOq6M+FYC7ZOOI4rlK8BA4I2MbbcAE6P1icDNORTb9cDVOXDeOgEDo/UWwNtAr1w4d+XElvi5AwxoHq03Al4CjgfmAGOj7f8XmJBDsf0JGJ3031wU11XAfcDj0fPYz1tdKSkMAt5x9/fc/QtgNjAq4ZhykrvPBz4psXkUcE+0fg/wtRoNKlJGbDnB3de6+yvR+hZgGXAIOXDuyoktcR5sjZ42ihYHTgIeiLYndd7Kii0nmFkecCbw++i5UQPnra4khUOADzKeF5Ij/xQRB/5mZovMbHzSwZSio7uvjdY/AjomGUwprjCzxVH1UiJVW5nMrBswgPDNMqfOXYnYIAfOXVQF8hrwMfAMoVS/yd2Lol0S+38tGZu7p87b5Oi83WZmTZKIDZgK/BjYHT1vRw2ct7qSFHLdie4+EDgd+I6ZfSXpgMrioVyaM9+WgDuAQ4H+wFrg10kGY2bNgQeB77v7Z5mvJX3uSoktJ86du+9y9/5AHqFUf1QScZSmZGxm1gf4H0KMxwJtgZ/UdFxm9lXgY3dfVNOfXVeSwodA54znedG2nODuH0aPHwMPEf4xcsk6M+sEED1+nHA8ae6+LvrH3Q3cTYLnzswaES66s9z9/0Wbc+LclRZbLp27KJ5NwDzgBKC1maVuB5z4/2tGbCOi6jh3953ADJI5b4OBkWa2klAdfhLwG2rgvNWVpLAAODxqmW8MjAUeTTgmAMzsADNrkVoHTgPeKP9dNe5R4MJo/ULgkQRjKSZ1wY2cTULnLqrP/QOwzN2nZLyU+LkrK7ZcOHdm1sHMWkfr+wOnEto85gGjo92SOm+lxfZWRpI3Qp19jZ83d/8fd89z926E69lz7l5ATZy3pFvXq2sBziD0ungXmJR0PBlx9SD0hnodWJp0bMD9hKqELwl1kt8m1FU+C6wA/g60zaHY7gWWAIsJF+BOCcV2IqFqaDHwWrSckQvnrpzYEj93QD/g1SiGN4Drou09gJeBd4C/Ak1yKLbnovP2BjCTqIdSUgswjD29j2I/b5rmQkRE0upK9ZGIiFQDJQUREUlTUhARkTQlBRERSVNSEBGRNCUFkYiZ7cqYGfM1q8bZds2sW+bsryK5ar+KdxGpN7Z7mPJApN5SSUGkAhbuh3GLhXtivGxmh0Xbu5nZc9HEac+aWZdoe0czeyiap/91M/uv6FANzezuaO7+v0WjaDGzK6N7ISw2s9kJ/ZgigJKCSKb9S1QfnZfx2mZ37wv8ljB7JcDtwD3u3g+YBUyLtk8D/unuRxPuD7E02n44MN3dewObgHOj7ROBAdFxLo/rhxPJhkY0i0TMbKu7Ny9l+0rgJHd/L5p47iN3b2dmGwhTR3wZbV/r7u3NbD2Q52FCtdQxuhGmZj48ev4ToJG7/8LMnga2Ag8DD/ueOf5FapxKCiLZ8TLW98XOjPVd7GnTOxOYTihVLMiYBVOkxikpiGTnvIzH/0Tr/ybMYAlQADwfrT8LTID0TVxalXVQM2sAdHb3eYR5+1sBe5VWRGqKvpGI7LF/dBeulKfdPdUttY2ZLSZ82x8XbfsuMMPMfgSsBy6Otn8PuMvMvk0oEUwgzP5amobAzChxGDDNw9z+IolQm4JIBaI2hXx335B0LCJxU/WRiIikqaQgIiJpKimIiEiakoKIiKQpKYiISJqSgoiIpCkpiIhI2v8HftgIlszvdXQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc = history_dict['accuracy']\n",
    "val_acc = history_dict['val_accuracy']\n",
    "loss = history_dict['loss']\n",
    "val_loss = history_dict['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "# \"bo\" is for \"blue dot\"\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "# b is for \"solid blue line\"\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPW9//HXB2QHAQGtiiaoqKzBkKIt4r7gft2uILa1VqlUrVfr9VK1av3V+rDWpV5te7HXViWKXC2KS9W61aXWEhUQ1CKyKIgQVgkBZfn8/vieGYYwyQxJJmeSeT8fj/OYmXPOzHzmTHI+813O92vujoiICECruAMQEZH8oaQgIiJJSgoiIpKkpCAiIklKCiIikqSkICIiSUoKsh0za21mVWa2d2PuGycz28/MGr3/tZkdY2YLUh7/y8xGZLNvPd7rD2Z2TX2fL5KNneIOQBrOzKpSHnYEvgI2R49/6O7lO/J67r4Z6NzY+xYCdz+gMV7HzC4EznP3I1Je+8LGeG2RuigptADunjwpR79EL3T3F2vb38x2cvdNTRGbSCb6e8wvqj4qAGb2CzN71MweMbO1wHlm9i0z+4eZrTazJWZ2t5m1ifbfyczczIqjxxOj7X8xs7Vm9paZ9dnRfaPtJ5jZHDNbY2b/bWZvmtn5tcSdTYw/NLO5ZrbKzO5OeW5rM7vTzFaY2TxgZB3H51ozm1Rj3b1mdkd0/0Iz+zD6PJ9Ev+Jre61FZnZEdL+jmT0UxTYbGFpj3+vMbF70urPN7NRo/SDgHmBEVDW3POXY3pjy/Iujz77CzJ4ws92zOTY7cpwT8ZjZi2a20sy+MLOrU97nZ9Ex+dLMKsxsj3RVdWb2RuJ7jo7na9H7rASuM7O+ZvZK9B7Lo+PWNeX5RdFnrIy2/8bM2kcx90vZb3czqzazHrV9XsnA3bW0oAVYABxTY90vgK+BUwg/BDoA3wQOJpQW9wHmAJdG++8EOFAcPZ4ILAfKgDbAo8DEeuy7K7AWOC3adiWwETi/ls+STYxPAl2BYmBl4rMDlwKzgd5AD+C18Oee9n32AaqATimvvQwoix6fEu1jwFHAemBwtO0YYEHKay0Cjoju/xp4FegOFAEf1Nj334Hdo+/k3CiG3aJtFwKv1ohzInBjdP+4KMYhQHvgt8DL2RybHTzOXYGlwOVAO2BnYFi07afADKBv9BmGALsA+9U81sAbie85+mybgHFAa8Lf4/7A0UDb6O/kTeDXKZ9nVnQ8O0X7D4+2TQBuTnmfnwBT4v4/bM5L7AFoaeQvtPak8HKG510F/F90P92J/vcp+54KzKrHvhcAr6dsM2AJtSSFLGM8JGX7n4GrovuvEarREttOrHmiqvHa/wDOje6fAPyrjn2fBi6J7teVFD5N/S6AH6Xum+Z1ZwEnRfczJYUHgF+mbNuZ0I7UO9Ox2cHj/B1gWi37fZKIt8b6bJLCvAwxnJV4X2AE8AXQOs1+w4H5gEWPpwNnNPb/VSEtqj4qHJ+lPjCzA83smag64EvgJqBnHc//IuV+NXU3Lte27x6pcXj4L15U24tkGWNW7wUsrCNegIeB0dH9c6PHiThONrO3o6qN1YRf6XUdq4Td64rBzM43sxlRFchq4MAsXxfC50u+nrt/CawC9kzZJ6vvLMNx3otw8k+nrm2Z1Px7/IaZTTazxVEMf6oRwwIPnRq24e5vEkodh5rZQGBv4Jl6xiSoTaGQ1OyO+T+EX6b7ufvOwPWEX+65tITwSxYAMzO2PYnV1JAYlxBOJgmZusxOBo4xsz0J1VsPRzF2AB4DbiFU7XQDXsgyji9qi8HM9gF+R6hC6RG97kcpr5up++znhCqpxOt1IVRTLc4irprqOs6fAfvW8rzatq2LYuqYsu4bNfap+fluJfSaGxTFcH6NGIrMrHUtcTwInEco1Ux2969q2U+yoKRQuLoAa4B1UUPdD5vgPZ8GSs3sFDPbiVBP3StHMU4G/sPM9owaHf+rrp3d/QtCFcefCFVHH0eb2hHquSuBzWZ2MqHuO9sYrjGzbhau47g0ZVtnwomxkpAfLyKUFBKWAr1TG3xreAT4gZkNNrN2hKT1urvXWvKqQ13HeSqwt5ldambtzGxnMxsWbfsD8Asz29eCIWa2CyEZfkHo0NDazMaSksDqiGEdsMbM9iJUYSW8BawAfmmh8b6DmQ1P2f4QobrpXEKCkAZQUihcPwG+R2j4/R9Cg3BOuftS4BzgDsI/+b7Ae4RfiI0d4++Al4D3gWmEX/uZPExoI0hWHbn7auAKYAqhsfYsQnLLxg2EEssC4C+knLDcfSbw38A/o30OAN5Oee5fgY+BpWaWWg2UeP5zhGqeKdHz9wbGZBlXTbUeZ3dfAxwLnElIVHOAw6PNtwFPEI7zl4RG3/ZRteBFwDWETgf71fhs6dwADCMkp6nA4ykxbAJOBvoRSg2fEr6HxPYFhO/5K3f/+w5+dqkh0Tgj0uSi6oDPgbPc/fW445Hmy8weJDRe3xh3LM2dLl6TJmVmIwk9fdYTujRuJPxaFqmXqH3mNGBQ3LG0BKo+kqZ2KDCPUJd+PHC6GgalvszsFsK1Er9090/jjqclUPWRiIgkqaQgIiJJza5NoWfPnl5cXBx3GCIizco777yz3N3r6gIONMOkUFxcTEVFRdxhiIg0K2aW6ap+QNVHIiKSImdJwczuN7NlZjarlu0WDZ0718xmmllprmIREZHs5LKk8CfqGMOeMBJl32gZS7gCVUREYpSzNgV3f82iiVdqcRrwYHRJ/D+i8WF2d/clO/peGzduZNGiRWzYsKGe0UpTaN++Pb1796ZNm9qG8xGRuMXZ0Lwn2w6fuyhat8NJYdGiRXTp0oXi4mLCwJuSb9ydFStWsGjRIvr06ZP5CSISi2bR0GxmY6Op/ioqKyu3275hwwZ69OihhJDHzIwePXqoNCdSD+XlUFwMrVqF2/Ly3L1XnElhMduONd+bWsaCd/cJ7l7m7mW9eqXvZquEkP/0HUmhynRSr2t7eTmMHQsLF4J7uB07NneJIc6kMBX4btQL6RBgTX3aE0REci2XJ/VM26+9Fqqrt32/6uqwPidyNc8nYRKQJYRRMBcBPwAuBi6OthtwL2E6v/eJJknPtAwdOtRr+uCDD7Zb15SWL1/uJSUlXlJS4rvttpvvscceycdfffVVVq9x/vnn+0cffVTnPvfcc49PnDixMUKOTdzflUhtJk50LypyNwu3iX+1iRPdO3Z0D6fssHTsmP32oqJttyWWoqLstpul3262Y58PqPBszt3Z7JRPS2Mkhdq+/MZwww03+G233bbd+i1btvjmzZsb742aKSUFiUtd//d1ndhzfVLPtD3T62cr26TQLBqaG1NT1s/NnTuX/v37M2bMGAYMGMCSJUsYO3YsZWVlDBgwgJtuuim576GHHsr06dPZtGkT3bp1Y/z48ZSUlPCtb32LZcuWAXDddddx1113JfcfP348w4YN44ADDuDvfw8TTq1bt44zzzyT/v37c9ZZZ1FWVsb06dO3i+2GG27gm9/8JgMHDuTiiy9OlO6YM2cORx11FCUlJZSWlrJgwQIAfvnLXzJo0CBKSkq4NmflVpH6a0gVTl1VNJ/WMiB3Yn2m7XvXMjt4Yn2m7TffDB07brutY8ewPieyyRz5tDS0pNBYWbc2qSWFjz/+2M3Mp02blty+YsUKd3ffuHGjH3rooT579mx3dx8+fLi/9957vnHjRgf82WefdXf3K664wm+55RZ3d7/22mv9zjvvTO5/9dVXu7v7k08+6ccff7y7u99yyy3+ox/9yN3dp0+f7q1atfL33ntvuzgTcWzZssVHjRqVfL/S0lKfOnWqu7uvX7/e161b51OnTvVDDz3Uq6urt3lufaikIPVV31/67g37Nd/QkkJDq58yffZsoZJCepmyemPbd999KSsrSz5+5JFHKC0tpbS0lA8//JAPPvhgu+d06NCBE044AYChQ4cmf63XdMYZZ2y3zxtvvMGoUaMAKCkpYcCAAWmf+9JLLzFs2DBKSkr429/+xuzZs1m1ahXLly/nlFNOAcLFZh07duTFF1/kggsuoEOHDgDssssuO34gRDLI1S99aNiv+Uy/1DNtHzMGJkyAoiIwC7cTJoT12WxP7LNgAWzZEm7H1Hc27iwUXFLIVFRrbJ06dUre//jjj/nNb37Dyy+/zMyZMxk5cmTafvtt27ZN3m/dujWbNm1K+9rt2rXLuE861dXVXHrppUyZMoWZM2dywQUX6PoBiVWcJ32o+8TeFCf1pjzpZ1JwSaHJ6+dSfPnll3Tp0oWdd96ZJUuW8Pzzzzf6ewwfPpzJkycD8P7776ctiaxfv55WrVrRs2dP1q5dy+OPPw5A9+7d6dWrF0899RQQLgqsrq7m2GOP5f7772f9+vUArFy5stHjlsJQW2kgzpM+ZHfiby4n9YYquKSQTVbPldLSUvr378+BBx7Id7/7XYYPH97o73HZZZexePFi+vfvz89//nP69+9P165dt9mnR48efO9736N///6ccMIJHHzwwclt5eXl3H777QwePJhDDz2UyspKTj75ZEaOHElZWRlDhgzhzjvvbPS4pWWobxVQ3Cf9xD4t5cTeINk0POTTko/XKeSTjRs3+vr1693dfc6cOV5cXOwbN26MOaqt9F01b7lq7G1oY22m2CT7hubYT/I7uigp1G3VqlVeWlrqgwcP9kGDBvnzzz8fd0jb0HeV3+Lq4aOTfu4pKUhe0neVv3J50s/m+Trp51a2SaHg2hRECllddf750NirOv34KSmItDC1nfgzdfvMh8ZeyQPZFCfyaVH1UfOm7yq3cjmGj+r9mzdUfSTSMtW3CihTSUDdOgVUfdQojjzyyO0uRLvrrrsYN25cnc/r3LkzAJ9//jlnnXVW2n2OOOIIKioq6nydu+66i+qUM8GJJ57I6tWrswldmpmGVAFlqv7RSV9ASaFRjB49mkmTJm2zbtKkSYwePTqr5++xxx489thj9X7/mknh2WefpVu3bvV+PYlXQxqDGzKGD+ikL0oKjeKss87imWee4euvvwZgwYIFfP7554wYMYKqqiqOPvpoSktLGTRoEE8++eR2z1+wYAEDBw4EwhAUo0aNol+/fpx++unJoSUAxo0blxx2+4YbbgDg7rvv5vPPP+fII4/kyCOPBKC4uJjly5cDcMcddzBw4EAGDhyYHHZ7wYIF9OvXj4suuogBAwZw3HHHbfM+CU899RQHH3wwBx10EMcccwxLly4FoKqqiu9///sMGjSIwYMHJ4fJeO655ygtLaWkpISjjz66UY5toWloY3BDxvARAVpeQ/Pll7sffnjjLpdfnrkR56STTvInnnjC3cPw1T/5yU/cPVxhvGbNGnd3r6ys9H333de3bNni7u6dOnVyd/f58+f7gAED3N399ttv9+9///vu7j5jxgxv3bp1cujtxJDVmzZt8sMPP9xnzJjh7u5FRUVeWVmZjCXxuKKiwgcOHOhVVVW+du1a79+/v7/77rs+f/58b926dXJI7bPPPtsfeuih7T7TypUrk7Hed999fuWVV7q7+9VXX+2XpxyUlStX+rJly7x3794+b968bWKtSQ3NdTfGNrQxONPrS+FCDc1NK7UKKbXqyN255pprGDx4MMcccwyLFy9O/uJO57XXXuO8884DYPDgwQwePDi5bfLkyZSWlnLQQQcxe/bstIPdpXrjjTc4/fTT6dSpE507d+aMM87g9ddfB6BPnz4MGTIEqH147kWLFnH88cczaNAgbrvtNmbPng3Aiy++yCWXXJLcr3v37vzjH//gsMMOo0+fPoCG165NLksCCaoCkobYKe4AGltUQ9LkTjvtNK644greffddqqurGTp0KBAGmKusrOSdd96hTZs2FBcX12uY6vnz5/PrX/+aadOm0b17d84///wGDXedGHYbwtDb6aqPLrvsMq688kpOPfVUXn31VW688cZ6v18hKS/f2tsnUZefODHX1SYwZkzYf+HC7V8ztTE48TrpXl+koVRSaCSdO3fmyCOP5IILLtimgXnNmjXsuuuutGnThldeeYWF6f7jUxx22GE8/PDDAMyaNYuZM2cCYdjtTp060bVrV5YuXcpf/vKX5HO6dOnC2rVrt3utESNG8MQTT1BdXc26deuYMmUKI0aMyPozrVmzhj333BOABx54ILn+2GOP5d57700+XrVqFYcccgivvfYa8+fPBwp3eG2VBKS5U1JoRKNHj2bGjBnbJIUxY8ZQUVHBoEGDePDBBznwwAPrfI1x48ZRVVVFv379uP7665MljpKSEg466CAOPPBAzj333G2G3R47diwjR45MNjQnlJaWcv755zNs2DAOPvhgLrzwQg466KCsP8+NN97I2WefzdChQ+nZs2dy/XXXXceqVasYOHAgJSUlvPLKK/Tq1YsJEyZwxhlnUFJSwjnnnJP1+zRH9Z0XoDG6hYrkkoX2h+ajrKzMa/bb//DDD+nXr19MEcmOaAnfVaI0kHry79gxnLy/851QQqjJLPyyr+u5OvFLLpnZO+5elmk/lRRE0qjvtQIqCUhz1+IamkUaquav+US7AISTd13tAg89lL4kULNNQElA8lWLKSk0t2qwQpRP31GurhpWSUCauxaRFNq3b8+KFSvy6qQj23J3VqxYQfv27eMOJec9hNQ7SJqzFtHQvHHjRhYtWtSgfvuSe+3bt6d37960adMm1jiKi9NfC1BUFE7imbZD3dciiOSjbBuaW0RSENkRrVqph5AUHvU+koJWV5uBegiJ1E5JQVqcTG0GumpYpHZKCtIsNaT3kEoCIrVTm4I0O5nq/DO1GYgUIrUpSIvV0PGFRKR2SgqSt2qrImqMkUZFJD0NcyF5qa6hJjTngEju5LSkYGYjzexfZjbXzMan2V5kZi+Z2Uwze9XMeucyHmk+6qoiUu8hkdzJWVIws9bAvcAJQH9gtJn1r7Hbr4EH3X0wcBNwS67ikfxTVw+iuqqI1HtIJHdyWX00DJjr7vMAzGwScBqQOrFwf+DK6P4rwBM5jEfySKaRSLOpIlISEGl8uaw+2hP4LOXxomhdqhnAGdH904EuZtaj5guZ2VgzqzCzisrKypwEK00rUw8iNRaLxCPu3kdXAYeb2XvA4cBiYHPNndx9gruXuXtZr169mjpGyYFMPYhURSQSj1wmhcXAXimPe0frktz9c3c/w90PAq6N1q3OYUzShBoy/hCosVgkDrlMCtOAvmbWx8zaAqOAqak7mFlPM0vE8FPg/hzGI02oMcYfEpGml7Ok4O6bgEuB54EPgcnuPtvMbjKzU6PdjgD+ZWZzgN0AnRJaCI0/JNI8aewjyQmNPySSXzT2kcRK4w+JNE9KClJvdTUkq81ApHnS2EdSL5kuPtP4Q7mzenXojfXll/DVV+mXr78O1XetWoUqu5q3rVtDhw4hUadbunWDnXeO+5NKHNSmIPWSzeT2hcwd1qyBFStg5cqtt6n3N2yAzp2hS5ew1Ly/bh3Mnx+WBQu23q5uok7bXbuG77OoKHzfiftFRfCNb4TtnTuHRFPbMaishLlz4ZNPtt4uXBiSUqdO4fmpt4mlQ4ewtG+/9TZxv0OHrcepSxdo06Zpjkdzl22bgkoKUi+ZLj5rCb7+Opy4a7NpE3z22dYTdurJe/58qKqq/bldu0K7dmGfmr20aurQIZyU+/SBb3873BYXQ/fu4TXSLW3bhhKBe2jYr3m7eTOsXx/eO3GbuqxYEb7LBQvC8uqrsHZt+vi6dAmlisTSpQssXx6SQOoxMAslxuLiEMcXX4TEl1iqqmDjxrqPRTrt2m2bUNu0Cd9dYtm4cdvHbdtuTT7pklLi+LVtG14rcb9t25CYevaE3XYLy667hsetW+943PlKSUHqJdPYRM3R4sXw97+H5c034b33wok/W506hRN2nz5w5JHhWPTqBbvsEpYePcJt9+6wU8p/3ubN4aS4dm1YqqrCbYcO4bV23TWcUOPkHkooCxeGZdmyUH1Vc1mzJtzusQeMGAH77Qf77htui4vDCbcuGzeGY7FhQ0hW69dvvZ+4ra7e9jilLonE0q7dtif0xP02bcL2qqptk1EiCVZVpU8kdVWotGoVEkMiQdRMNKm3HTrUnsjbtQtJrVu38KOhfftG/QqzpqQg9XLzzemnxGwuDcnV1TBrFrz99tZEkCjldOgAw4bBVVeFk3ptWrUKJ79EIujRo34n79att/7KzldmIZl17w5DhuTufdq0CSfFfLN5c0gO69eHKrGlS0NiXLp02/vLl4cfF6kJp6oqPH9HtWu3NUEkbi+/HE46qfE/XyolBamX5tKQ7A6LFsGMGdsuH3+89dffHnvA8OFw5ZWhembIENVTy7YSDfMdOoTS3gEHZP9c95BQ1q0LP0Zq6xywYUNIIKtXhxLX6tXb3l+zJrxOrqmhWWpVXp7/J30I/0iJao0FC7a9P2cOrFq1dd8+faCkBAYPDrdDh4bPFnf1jEiuqaFZGiRTl9OmtmpV+HU/d264TSyffBLqg1O1bRtO9EVFcPbZW5PA4MH5XUUjkg9UUpC04u5yumQJTJoEf/4zfPjhtif+RC+Wvn23NmDW7C5ZWzdJkUKlkoI0SBxdTquqYMoUmDgRXnwxdJ0sLYUzzwwJILHss098PTNEWjolBUmrqbqcbtoUEsDEiSEhVFeHX/7XXBOqqQ48sHHfT0TqpqQgaeWyy+natfDCC/D00/DMM6GLX/fu8J3vwHnnhZ5AavgViYeSgqTV2F1OFyyAp54KieDVV0PXum7d4IQTQmPwiSdmvrBJRHJPDc0FLpfdTj//HO67Dx57LFwoBrD//nDKKWH59rd1PYBIU1FDs2SUi26n7mGIiHvugccfD1dyHnYY3H47nHxySAoikr9UUihgjdntdN06ePjhkAxmzgxVQxdcAOPGhW6jIhIvlRQko8bodjp/fkgE998fLsUfPDhUGZ177vaT7IhI/lNSKGAN6XY6bRrcdluoImrVKlxLcOml6jkk0tzpus8CtqNTZm7ZEnoQHX54GEX0hRfgP/8zVDVNmgSHHqqEINLcqaRQwLLtdrphQ7i47Pbb4aOPYK+94I474MILw/jvItJyqKTQwpWXhwblVq3CbXn5ttvHjAm/9LdsCbepCcEd/vSn8LyLLgpDS5SXh0HorrhCCUGkJVJJoQVrSJfTJUvCvk8/HdoJysvhqKNUPSTS0qmk0IJde+328/9WV4f1tXGHRx6BAQPCmER33gmvvQZHH62EIFIIlBRasB3tclpZCf/+76E76QEHwPTp8B//oWGoRQqJ/t1bsNq6lqZbP2VKKB1MnQq33AKvv75jUw6KSMugpNCCZdPldMWKMDLpGWdA797wzjswfjzspNYmkYKkpNCCjRkDEyaEYSvMwu2ECWG9Ozz4YJiv4NFH4cYb4e23YeDAuKMWkTjp92ALN2bM9j2NPv4YLr4YXn45jFT6P/+jZCAigUoKBeSrr+D//T8YNChUE/3+96HtQAlBRBJUUigQr78OP/whfPhh6GF0112w++5xRyUi+UYlhWYu0xXL1dXhIrTDDgv3n3kmtCEoIYhIOhmTgpldZmbdmyIY2TGJK5YXLgwNx4krlhOJYf78cDXyH/4AV10Fs2eHaS9FRGqTTUlhN2CamU02s5Fm2V/XGu3/LzOba2bj02zf28xeMbP3zGymmemUtQPqumL5hRdg6NAwntEzz4Rhrjt1iiVMEWlGspp5LUoExwHfB8qAycD/uvsndTynNTAHOBZYBEwDRrv7Byn7TADec/ffmVl/4Fl3L64rFs28tlWrVqGEkI5ZaECeMgX23bdp4xKR/JPtzGtZtSl4yBxfRMsmoDvwmJn9qo6nDQPmuvs8d/8amAScVvOlgZ2j+12Bz7OJR4K6JsM55xx46y0lBBHZMdm0KVxuZu8AvwLeBAa5+zhgKHBmHU/dE/gs5fGiaF2qG4HzzGwR8CxwWS0xjDWzCjOrqKyszBRywUh3xTKEsYsefljVRSKy47IpKewCnOHux7v7/7n7RgB33wKc3MD3Hw38yd17AycCD5nZdjG5+wR3L3P3sl69ejXwLVuOxBXLiUPSqhVcc01oaNaIpiJSH9lcp/AXYGXigZntDPRz97fd/cM6nrcY2Cvlce9oXaofACMB3P0tM2sP9ASWZRGXAD16wKpVoVH5z3/Obn5lEZHaZFNS+B1QlfK4KlqXyTSgr5n1MbO2wChgao19PgWOBjCzfkB7QPVDWXrrLTjzzNCg/NJLSggi0nDZJAXzlC5KUbVRxhKGu28CLgWeBz4EJrv7bDO7ycxOjXb7CXCRmc0AHgHO92y6QwmzZ8NJJ8Eee8Bzz0HXrnFHJCItQTZJYZ6Z/djM2kTL5cC8bF7c3Z919/3dfV93vzlad727T43uf+Duw929xN2HuPsL9f8oLVO6K5YXLIDjjgtzJv/1r7DbbjEHKSItRjZtChcDdwPXEbqQvgSMzWVQEqSbY/mii2DnncPgdq+/HhKFiEhjyaYaaBmhPUCaWLorltevhw0b4I03NLqpiDS+jEkh6hH0A2AAoSEYAHe/IIdxCbXPpewe5kEQEWls2bQpPAR8Azge+Buha+naXAYlQW29iYqKmjYOESkc2SSF/dz9Z8A6d38AOAk4OLdhCWQ3x7KISGPKJilsjG5Xm9lAwhhFu+YuJEkYMwYuvXTr49Q5lkVEciGb3kcTovkUriNcfNYZ+FlOoxIAVq6Ehx6C/v3D9Jnt22d+johIQ9SZFKJxiL5091XAa8A+TRKVAHDJJVBZCU8/rYQgIk2jzuqj6Orlq5soFkkxaVJYbrgBSkvjjkZECkU2bQovmtlVZraXme2SWHIeWQFbvBjGjYNDDoHx281XJyKSO9m0KZwT3V6Sss5RVVJOuMMFF8DXX8ODD8JO2XxDIiKNJJsrmvs0RSAS/Pa3YX7l3/4W+vaNOxoRKTTZXNH83XTr3f3Bxg+n8JSXh+EsPv0Udt8dli+H44+Hiy+OOzIRKUTZVE58M+V+e8L8B+8CSgoNVHPAu8+jGapPPlkzp4lIPGxHpy8ws27AJHcfmZuQ6lZWVuYVFRVxvHWjKy4OI5/WVFRZU1p2AAAP8ElEQVQUhscWEWksZvaOu5dl2i+b3kc1rQPUztAIahvwrrb1IiK5lk2bwlOE3kYQkkh/YHIugyoUe++dvqSgaTVFJC7ZtCn8OuX+JmChuy/KUTwF5eab4fzzYdOmres04J2IxCmbpPApsMTdNwCYWQczK3b3BTmNrADstx9s3gydO8O6daGEcPPNGvBOROKTTVL4PyB1SpfN0bpvpt9dsrF+fSgl9O4N778PXbvGHZGISHZJYSd3/zrxwN2/NrO2OYypIFx/PXz0UbhQTQlBRPJFNr2PKs3s1MQDMzsNWJ67kFq+N9+E228PF6gde2zc0YiIbJVNSeFioNzM7okeLwLSXuUsmVVXh2qjoiL41a/ijkZEZFvZjH30CXCImXWOHlflPKoW7JprYO5cePll6NIl7mhERLaVsfrIzH5pZt3cvcrdq8ysu5n9oimCa2n+9jf4zW/gssvgyCPjjkZEZHvZtCmc4O6rEw+iWdhOzF1ILVNVVRgSe9994ZZb4o5GRCS9bJJCazNrl3hgZh2AdnXsLzWUl4cRUOfNC8nhiSfijkhEJL1sGprLgZfM7I+AAecDD+QyqJakvBx+8AP46qvweOnSMDIq6CI1Eck/GUsK7n4r8AugH3AA8DxQlOO4WoxrrtmaEBKqq8McCiIi+SbbUVKXEgbFOxs4CvgwZxG1MBoJVUSak1qrj8xsf2B0tCwHHiXMv6B+M1lauRJatYItW7bfppFQRSQf1dWm8BHwOnCyu88FMLMrmiSqFuL668Ed2reHDRu2rtdIqCKSr+qqPjoDWAK8Ymb3mdnRhIZmycLMmfC738Ell8Af/hCuYDYLtxMmqJFZRPJTxuk4zawTcBqhGukowtzMU9z9hdyHt73mMB2ne7g4bdYsmDMHdtkl7ohEpNA12nSc7r7O3R9291OA3sB7wH9lGcRIM/uXmc01s/Fptt9pZtOjZY6ZrU73Os3NY4+Fq5dvvlkJQUSal4wlhXq/sFlrYA5wLGEQvWnAaHf/oJb9LwMOcvcL6nrdfC8pVFfDgQdCjx5QUQGtW8cdkYhII5YUGmAYMNfd50XzMUwiVEPVZjTwSA7jaRK33gqffQZ3362EICLNTy6Twp7AZymPF0XrtmNmRUAf4OVato81swozq6isrGz0QBvL/PkhKYweDSNGxB2NiMiOy2VS2BGjgMfcfXO6je4+wd3L3L2sV69eTRxa9q66KpQONE+CiDRXuUwKi4G9Uh73jtalM4pmXnX00kvw5z+H4St69447GhGR+sllUpgG9DWzPtGczqOAqTV3MrMDge7AWzmMJac2bYLLL4d99oErr4w7GhGR+stmlNR6cfdNZnYpYQC91sD97j7bzG4CKtw9kSBGAZM8V92gmsDvfw+zZ8OUKeHqZRGR5ipnXVJzJd+6pK5YEa5S3rw5DGVRVBSuT9AVyyKST7LtkpqzkkKhGD0a1q3b+njhQs2XICLNV770PmqWZs2Cv/51+/WaL0FEmislhXpyhyvqGDNW8yWISHOkpFBPTz0FL74I3bun3675EkSkOVJSqIevvgpdT/v1gzvvDPMjpNJ8CSLSXKmhuR7uvhs++QSeew6OPx522im0IXz6aSghqPeRiDRX6pK6g5Yuhb594fDDQxWSiEhzkA+jpLZI114brke4/fa4IxERaXxKCjvg3Xfh/vvhxz+G/fePOxoRkcanpJAl9zC+Uc+e8LOfxR2NiEhuqKE5S5MnwxtvwIQJ0LVr3NGIiOSGSgpZ2LQJfvpTGDIELqhzslARkeZNJYUsPPpomFVt6lRNsSkiLZtKChm4hyk2+/eHk06KOxoRkdxSSSGDv/wF3n8fHngAWimFikgLp9NcBrfeCnvtFYbIFhFp6ZQU6vDWW/Daa+Hq5b59Q0mhuBjKy+OOTEQkN1R9VIdbb4XOneHxx2H9+rBOk+iISEumkkItPvgAnnwy9DZKJIQETaIjIi2VkkItbrsNOnSANWvSb9ckOiLSEikppPHZZzBxIlx0ERQVpd9Hk+iISEukpJDGHXeE6xOuvDLMjaBJdESkUKihuYYVK+C+++Dcc0MpIVFS0CQ6IlIIlBRquPdeWLcOrr5667oxY5QERKQwqPooxbp1YarNk0+GgQPjjkZEpOkpKaS4//5QfTR+fNyRiIjEQ0khsnFjmGJz+PCwiIgUIrUpRB59NFytfM89cUciIhIflRQI3U9/9SsYMABOPDHuaERE4qOSAvD88xoeW0QEVFIAwsB3vXvDqFFxRyIiEq+CTwr//Ce8+ipccQW0bRt3NCIi8Sr4pHDbbdC1axjnSESk0BV0Upg7N8yV8KMfQZcucUcjIhK/gk4Kt98ObdrAj38cdyQiIvkhp0nBzEaa2b/MbK6Zpb1O2Mz+3cw+MLPZZvZwLuNJtXQp/PGP8L3vwUsvhWk2Nd2miBS6nHVJNbPWwL3AscAiYJqZTXX3D1L26Qv8FBju7qvMbNdcxVPTPffA11+HuZfHjg2zqYGm2xSRwpbLksIwYK67z3P3r4FJwGk19rkIuNfdVwG4+7IcxpNUVRVGQ/23fwu3iYSQoOk2RaRQ5TIp7Al8lvJ4UbQu1f7A/mb2ppn9w8xGpnshMxtrZhVmVlFZWdngwP73f2HVqjA8dm3Tamq6TREpRHE3NO8E9AWOAEYD95lZt5o7ufsEdy9z97JevXo16A03bgwzq40YAYccUvu0mppuU0QKUS6TwmJgr5THvaN1qRYBU919o7vPB+YQkkTOTJ4cSgGJSXQ03aaIyFa5TArTgL5m1sfM2gKjgKk19nmCUErAzHoSqpPm5SqgxMB3/ftvHfhuzBiYMCFMu2kWbidMUCOziBSmnPU+cvdNZnYp8DzQGrjf3Web2U1AhbtPjbYdZ2YfAJuB/3T3FbmK6YUXYObM0BU1deA7TbcpIhKYu8cdww4pKyvzioqKej336KPho49g/nyNcyQihcXM3nH3skz7xd3Q3GQqKuDllzXwnYhIXQomKbz6KnTrtvXCNBER2V7BJIWrroJ582DnneOOREQkfxVMUgDo3j3uCERE8ltBJQUREambkoKIiCQpKYiISJKSgoiIJCkpiIhIkpKCiIgkKSmIiEiSkoKIiCQpKYiISJKSgoiIJCkpiIhIkpKCiIgkKSmIiEiSkoKIiCQpKYiISFJBJIXyciguhlatwm15edwRiYjkp53iDiDXysvDFJzV1eHxwoVbp+QcMya+uERE8lGLLylce+3WhJBQXR3Wi4jItlp8Uvj00x1bLyJSyFp8Uth77x1bLyJSyFp8Urj5ZujYcdt1HTuG9SIisq0WnxTGjIEJE6CoCMzC7YQJamQWEUmnxfc+gpAAlARERDJr8SUFERHJnpKCiIgkKSmIiEiSkoKIiCQpKYiISJK5e9wx7BAzqwQW1rK5J7C8CcPZUfkcn2KrH8VWP4qtfhoSW5G798q0U7NLCnUxswp3L4s7jtrkc3yKrX4UW/0otvppithUfSQiIklKCiIiktTSksKEuAPIIJ/jU2z1o9jqR7HVT85ja1FtCiIi0jAtraQgIiINoKQgIiJJLSYpmNlIM/uXmc01s/Fxx5PKzBaY2ftmNt3MKmKO5X4zW2Zms1LW7WJmfzWzj6Pb7nkU241mtjg6dtPN7MSYYtvLzF4xsw/MbLaZXR6tj/3Y1RFb7MfOzNqb2T/NbEYU28+j9X3M7O3o//VRM2ubR7H9yczmpxy3IU0dW0qMrc3sPTN7Onqc++Pm7s1+AVoDnwD7AG2BGUD/uONKiW8B0DPuOKJYDgNKgVkp634FjI/ujwduzaPYbgSuyoPjtjtQGt3vAswB+ufDsasjttiPHWBA5+h+G+Bt4BBgMjAqWv97YFwexfYn4Ky4/+aiuK4EHgaejh7n/Li1lJLCMGCuu89z96+BScBpMceUl9z9NWBljdWnAQ9E9x8A/q1Jg4rUEltecPcl7v5udH8t8CGwJ3lw7OqILXYeVEUP20SLA0cBj0Xr4zputcWWF8ysN3AS8IfosdEEx62lJIU9gc9SHi8iT/4pIg68YGbvmNnYuINJYzd3XxLd/wLYLc5g0rjUzGZG1UuxVG2lMrNi4CDCL8u8OnY1YoM8OHZRFch0YBnwV0KpfrW7b4p2ie3/tWZs7p44bjdHx+1OM2sXR2zAXcDVwJbocQ+a4Li1lKSQ7w5191LgBOASMzss7oBq46Fcmje/loDfAfsCQ4AlwO1xBmNmnYHHgf9w9y9Tt8V97NLElhfHzt03u/sQoDehVH9gHHGkUzM2MxsI/JQQ4zeBXYD/auq4zOxkYJm7v9PU791SksJiYK+Ux72jdXnB3RdHt8uAKYR/jHyy1Mx2B4hul8UcT5K7L43+cbcA9xHjsTOzNoSTbrm7/zlanRfHLl1s+XTsonhWA68A3wK6mVliOuDY/19TYhsZVce5u38F/JF4jttw4FQzW0CoDj8K+A1NcNxaSlKYBvSNWubbAqOAqTHHBICZdTKzLon7wHHArLqf1eSmAt+L7n8PeDLGWLaROOFGTiemYxfV5/4v8KG735GyKfZjV1ts+XDszKyXmXWL7ncAjiW0ebwCnBXtFtdxSxfbRylJ3gh19k1+3Nz9p+7e292LCeezl919DE1x3OJuXW+sBTiR0OviE+DauONJiWsfQm+oGcDsuGMDHiFUJWwk1En+gFBX+RLwMfAisEsexfYQ8D4wk3AC3j2m2A4lVA3NBKZHy4n5cOzqiC32YwcMBt6LYpgFXB+t3wf4JzAX+D+gXR7F9nJ03GYBE4l6KMW1AEewtfdRzo+bhrkQEZGkllJ9JCIijUBJQUREkpQUREQkSUlBRESSlBRERCRJSUEkYmabU0bGnG6NONqumRWnjv4qkq92yryLSMFY72HIA5GCpZKCSAYW5sP4lYU5Mf5pZvtF64vN7OVo4LSXzGzvaP1uZjYlGqd/hpl9O3qp1mZ2XzR2/wvRVbSY2Y+juRBmmtmkmD6mCKCkIJKqQ43qo3NStq1x90HAPYTRKwH+G3jA3QcD5cDd0fq7gb+5ewlhfojZ0fq+wL3uPgBYDZwZrR8PHBS9zsW5+nAi2dAVzSIRM6ty985p1i8AjnL3edHAc1+4ew8zW04YOmJjtH6Ju/c0s0qgt4cB1RKvUUwYmrlv9Pi/gDbu/gszew6oAp4AnvCtY/yLNDmVFESy47Xc3xFfpdzfzNY2vZOAewmlimkpo2CKNDklBZHsnJNy+1Z0/++EESwBxgCvR/dfAsZBchKXrrW9qJm1AvZy91cI4/Z3BbYrrYg0Ff0iEdmqQzQLV8Jz7p7oltrdzGYSfu2PjtZdBvzRzP4TqAS+H62/HJhgZj8glAjGEUZ/Tac1MDFKHAbc7WFsf5FYqE1BJIOoTaHM3ZfHHYtIrqn6SEREklRSEBGRJJUUREQkSUlBRESSlBRERCRJSUFERJKUFEREJOn/A7yZRcwTCBM8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.clf()   # clear figure\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow-2.0)",
   "language": "python",
   "name": "tensorflow-2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}