t81_558_class_07_2_Keras_gan.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 7: Generative Adversarial Networks**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 7 Material\n",
    "\n",
    "* Part 7.1: Introduction to GANS for Image and Data Generation [[Video]](https://www.youtube.com/watch?v=u8xn393mDPM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=21) [[Notebook]](t81_558_class_07_1_gan_intro.ipynb)\n",
    "* **Part 7.2: Implementing a GAN in Keras** [[Video]](https://www.youtube.com/watch?v=cf6FDLFNWEk&index=22&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_07_2_Keras_gan.ipynb)\n",
    "* Part 7.3: Face Generation with StyleGAN and Python [[Video]](https://www.youtube.com/watch?v=LSSH_NdXwhU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=23) [[Notebook]](t81_558_class_07_3_style_gan.ipynb)\n",
    "* Part 7.4: GANS for Semi-Supervised Learning in Keras [[Video]](https://www.youtube.com/watch?v=LSSH_NdXwhU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=23) [[Notebook]](t81_558_class_07_4_gan_semi_supervised.ipynb)\n",
    "* Part 7.5: An Overview of GAN Research [[Video]](https://www.youtube.com/watch?v=LSSH_NdXwhU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=23)  [[Notebook]](t81_558_class_07_5_gan_research.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 7.2: Implementing DCGANs in Keras\n",
    "\n",
    "Paper that described the type of DCGAN that we will create in this module.\n",
    "\n",
    "* Radford, A., Metz, L., & Chintala, S. (2015). [Unsupervised representation learning with deep convolutional generative adversarial networks](https://arxiv.org/abs/1511.06434). *arXiv preprint arXiv:1511.06434*.\n",
    "\n",
    "This paper implements a DCGAN as follows:\n",
    "\n",
    "* No pre-processing was applied to training images besides scaling to the range of the tanh activation function [-1, 1]. \n",
    "* All models were trained with mini-batch stochastic gradient descent (SGD) with a mini-batch size of 128. \n",
    "* All weights were initialized from a zero-centered Normal distribution with standard deviation 0.02. \n",
    "* In the LeakyReLU, the slope of the leak was set to 0.2 in all models.\n",
    "* we used the Adam optimizer(Kingma & Ba, 2014) with tuned hyperparameters. We found the suggested learning rate of 0.001, to be too high, using 0.0002 instead. \n",
    "* Additionally, we found leaving the momentum term $\\beta{1}$ at the suggested value of 0.9 resulted in training oscillation and instability while reducing it to 0.5 helped stabilize training.\n",
    "\n",
    "The paper also provides the following architecture guidelines for stable Deep Convolutional GANs:\n",
    "\n",
    "* Replace any pooling layers with strided convolutions (discriminator) and fractional-strided convolutions (generator).\n",
    "* Use batchnorm in both the generator and the discriminator.\n",
    "* Remove fully connected hidden layers for deeper architectures.\n",
    "* Use ReLU activation in generator for all layers except for the output, which uses Tanh.\n",
    "* Use LeakyReLU activation in the discriminator for all layers.\n",
    "\n",
    "While creating the material for this module I used a number of Internet resources, some of the most helpful were:\n",
    "\n",
    "* [Keep Calm and train a GAN. Pitfalls and Tips on training Generative Adversarial Networks](https://medium.com/@utk.is.here/keep-calm-and-train-a-gan-pitfalls-and-tips-on-training-generative-adversarial-networks-edd529764aa9)\n",
    "* [Collection of Keras implementations of Generative Adversarial Networks GANs](https://github.com/eriklindernoren/Keras-GAN)\n",
    "* [dcgan-facegenerator](https://github.com/platonovsimeon/dcgan-facegenerator), [Semi-Paywalled Article by GitHub Author](https://medium.com/datadriveninvestor/generating-human-faces-with-keras-3ccd54c17f16)\n",
    "\n",
    "The program created next will generate faces similar to these.  While these faces are not perfect, they demonstrate how we can construct and train a GAN on or own.  Later we will see how to import very advanced weights from nVidia to produce high resolution, realistic looking faces.\n",
    "\n",
    "![GAN](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/gan-3.png \"GAN\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following packages will be used to implement a basic GAN system in Python/Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Input, Reshape, Dropout, Dense, Flatten, BatchNormalization, Activation, ZeroPadding2D\n",
    "from keras.layers.advanced_activations import LeakyReLU\n",
    "from keras.layers.convolutional import UpSampling2D, Conv2D\n",
    "from keras.models import Sequential, Model, load_model\n",
    "from keras.optimizers import Adam\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "from tqdm import tqdm\n",
    "import os "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I suggest running this code with a GPU, it will be very slow on a CPU alone.  The following code mounts your Google drive for use with Google CoLab.  If you are not using CoLab, the following code will not work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this for Google CoLab\n",
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the constants that define how the GANs will be created for this example.  The higher the resolution, the more memory that will be needed.  Higher resolution will also result in longer run times.  For Google CoLab (with GPU) 128x128 resolution is as high as can be used (due to memoru).  Note that the resolution is specified as a multiple of 32.  So **GENERATE_RES** of 1 is 32, 2 is 64, etc.\n",
    "\n",
    "To run this you will need training data.  The training data can be any collection of images.  I suggest using training data from the following two locations.  Simply unzip and combine to a common directory.  This directory should be uploaded to Google Drive (if you are using CoLab). The constant **DATA_PATH** defines where these images are stored.\n",
    "\n",
    "The source data (faces) used in this module can be found here:\n",
    "\n",
    "* [Kaggle Faces Data New](https://www.kaggle.com/gasgallo/faces-data-new)\n",
    "* [Kaggle Lag Dataset: Dataset of faces, from more than 1k different subjects](https://www.kaggle.com/gasgallo/lag-dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generation resolution - Must be square \n",
    "# Training data is also scaled to this.\n",
    "# Note GENERATE_RES higher than 4 will blow Google CoLab's memory.\n",
    "GENERATE_RES = 2 # (1=32, 2=64, 3=96, etc.)\n",
    "GENERATE_SQUARE = 32 * GENERATE_RES # rows/cols (should be square)\n",
    "IMAGE_CHANNELS = 3\n",
    "\n",
    "# Preview image \n",
    "PREVIEW_ROWS = 4\n",
    "PREVIEW_COLS = 7\n",
    "PREVIEW_MARGIN = 16\n",
    "SAVE_FREQ = 100\n",
    "\n",
    "# Size vector to generate images from\n",
    "SEED_SIZE = 100\n",
    "\n",
    "# Configuration\n",
    "DATA_PATH = '/content/drive/My Drive/projects/faces'\n",
    "EPOCHS = 10000\n",
    "BATCH_SIZE = 32\n",
    "\n",
    "print(f\"Will generate {GENERATE_SQUARE}px square images.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will load and preprocess the images.  This can take awhile.  Google CoLab took around an hour to process.  Because of this we store the processed file as a binary.  This way we can simply reload the processed training data and quickly use it.  It is most efficient to only perform this operation once.  The dimensions of the image are encoded into the filename of the binary file because we need to regenerate it if these change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image set has 11,682 images.  Can take over an hour for initial preprocessing.\n",
    "# Because of this time needed, save a Numpy preprocessed file.\n",
    "# Note, that file is large enough to cause problems for sume verisons of Pickle,\n",
    "# so Numpy binary files are used.\n",
    "training_binary_path = os.path.join(DATA_PATH,f'training_data_{GENERATE_SQUARE}_{GENERATE_SQUARE}.npy')\n",
    "\n",
    "print(f\"Looking for file: {training_binary_path}\")\n",
    "\n",
    "if not os.path.isfile(training_binary_path):\n",
    "  print(\"Loading training images...\")\n",
    "\n",
    "  training_data = []\n",
    "  faces_path = os.path.join(DATA_PATH,'face_images')\n",
    "  for filename in tqdm(os.listdir(faces_path)):\n",
    "      path = os.path.join(faces_path,filename)\n",
    "      image = Image.open(path).resize((GENERATE_SQUARE,GENERATE_SQUARE),Image.ANTIALIAS)\n",
    "      training_data.append(np.asarray(image))\n",
    "  training_data = np.reshape(training_data,(-1,GENERATE_SQUARE,GENERATE_SQUARE,IMAGE_CHANNELS))\n",
    "  training_data = training_data / 127.5 - 1.\n",
    "\n",
    "  print(\"Saving training image binary...\")\n",
    "  np.save(training_binary_path,training_data)\n",
    "else:\n",
    "  print(\"Loading previous training pickle...\")\n",
    "  training_data = np.load(training_binary_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below creates the generator and discriminator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_generator(seed_size, channels):\n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Dense(4*4*256,activation=\"relu\",input_dim=seed_size))\n",
    "    model.add(Reshape((4,4,256)))\n",
    "\n",
    "    model.add(UpSampling2D())\n",
    "    model.add(Conv2D(256,kernel_size=3,padding=\"same\"))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(Activation(\"relu\"))\n",
    "\n",
    "    model.add(UpSampling2D())\n",
    "    model.add(Conv2D(256,kernel_size=3,padding=\"same\"))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(Activation(\"relu\"))\n",
    "   \n",
    "    # Output resolution, additional upsampling\n",
    "    for i in range(GENERATE_RES):\n",
    "      model.add(UpSampling2D())\n",
    "      model.add(Conv2D(128,kernel_size=3,padding=\"same\"))\n",
    "      model.add(BatchNormalization(momentum=0.8))\n",
    "      model.add(Activation(\"relu\"))\n",
    "\n",
    "    # Final CNN layer\n",
    "    model.add(Conv2D(channels,kernel_size=3,padding=\"same\"))\n",
    "    model.add(Activation(\"tanh\"))\n",
    "\n",
    "    input = Input(shape=(seed_size,))\n",
    "    generated_image = model(input)\n",
    "\n",
    "    return Model(input,generated_image)\n",
    "\n",
    "\n",
    "def build_discriminator(image_shape):\n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Conv2D(32, kernel_size=3, strides=2, input_shape=image_shape, padding=\"same\"))\n",
    "    model.add(LeakyReLU(alpha=0.2))\n",
    "\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Conv2D(64, kernel_size=3, strides=2, padding=\"same\"))\n",
    "    model.add(ZeroPadding2D(padding=((0,1),(0,1))))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(LeakyReLU(alpha=0.2))\n",
    "\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Conv2D(128, kernel_size=3, strides=2, padding=\"same\"))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(LeakyReLU(alpha=0.2))\n",
    "\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Conv2D(256, kernel_size=3, strides=1, padding=\"same\"))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(LeakyReLU(alpha=0.2))\n",
    "\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Conv2D(512, kernel_size=3, strides=1, padding=\"same\"))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(LeakyReLU(alpha=0.2))\n",
    "\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "    input_image = Input(shape=image_shape)\n",
    "\n",
    "    validity = model(input_image)\n",
    "\n",
    "    return Model(input_image, validity)\n",
    "  \n",
    "def save_images(cnt,noise):\n",
    "  image_array = np.full(( \n",
    "      PREVIEW_MARGIN + (PREVIEW_ROWS * (GENERATE_SQUARE+PREVIEW_MARGIN)), \n",
    "      PREVIEW_MARGIN + (PREVIEW_COLS * (GENERATE_SQUARE+PREVIEW_MARGIN)), 3), \n",
    "      255, dtype=np.uint8)\n",
    "  \n",
    "  generated_images = generator.predict(noise)\n",
    "\n",
    "  generated_images = 0.5 * generated_images + 0.5\n",
    "\n",
    "  image_count = 0\n",
    "  for row in range(PREVIEW_ROWS):\n",
    "      for col in range(PREVIEW_COLS):\n",
    "        r = row * (GENERATE_SQUARE+16) + PREVIEW_MARGIN\n",
    "        c = col * (GENERATE_SQUARE+16) + PREVIEW_MARGIN\n",
    "        image_array[r:r+GENERATE_SQUARE,c:c+GENERATE_SQUARE] = generated_images[image_count] * 255\n",
    "        image_count += 1\n",
    "\n",
    "          \n",
    "  output_path = os.path.join(DATA_PATH,'output')\n",
    "  if not os.path.exists(output_path):\n",
    "    os.makedirs(output_path)\n",
    "  \n",
    "  filename = os.path.join(output_path,f\"train-{cnt}.png\")\n",
    "  im = Image.fromarray(image_array)\n",
    "  im.save(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we actually build the discriminator and the generator.  Both will be trained with the Adam optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_shape = (GENERATE_SQUARE,GENERATE_SQUARE,IMAGE_CHANNELS)\n",
    "optimizer = Adam(1.5e-4,0.5) # learning rate and momentum adjusted from paper\n",
    "\n",
    "discriminator = build_discriminator(image_shape)\n",
    "discriminator.compile(loss=\"binary_crossentropy\",optimizer=optimizer,metrics=[\"accuracy\"])\n",
    "generator = build_generator(SEED_SIZE,IMAGE_CHANNELS)\n",
    "\n",
    "random_input = Input(shape=(SEED_SIZE,))\n",
    "\n",
    "generated_image = generator(random_input)\n",
    "\n",
    "discriminator.trainable = False\n",
    "\n",
    "validity = discriminator(generated_image)\n",
    "\n",
    "combined = Model(random_input,validity)\n",
    "combined.compile(loss=\"binary_crossentropy\",optimizer=optimizer,metrics=[\"accuracy\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_real = np.ones((BATCH_SIZE,1))\n",
    "y_fake = np.zeros((BATCH_SIZE,1))\n",
    "\n",
    "fixed_seed = np.random.normal(0, 1, (PREVIEW_ROWS * PREVIEW_COLS, SEED_SIZE))\n",
    "\n",
    "cnt = 1\n",
    "for epoch in range(EPOCHS):\n",
    "    idx = np.random.randint(0,training_data.shape[0],BATCH_SIZE)\n",
    "    x_real = training_data[idx]\n",
    "\n",
    "    # Generate some images\n",
    "    seed = np.random.normal(0,1,(BATCH_SIZE,SEED_SIZE))\n",
    "    x_fake = generator.predict(seed)\n",
    "\n",
    "    # Train discriminator on real and fake\n",
    "    discriminator_metric_real = discriminator.train_on_batch(x_real,y_real)\n",
    "    discriminator_metric_generated = discriminator.train_on_batch(x_fake,y_fake)\n",
    "    discriminator_metric = 0.5 * np.add(discriminator_metric_real,discriminator_metric_generated)\n",
    "    \n",
    "    # Train generator on Calculate losses\n",
    "    generator_metric = combined.train_on_batch(seed,y_real)\n",
    "    \n",
    "    # Time for an update?\n",
    "    if epoch % SAVE_FREQ == 0:\n",
    "        save_images(cnt, fixed_seed)\n",
    "        cnt += 1\n",
    "        print(f\"Epoch {epoch}, Discriminator accuarcy: {discriminator_metric[1]}, Generator accuracy: {generator_metric[1]}\")\n",
    "        \n",
    "generator.save(os.path.join(DATA_PATH,\"face_generator.h5\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow-2.0)",
   "language": "python",
   "name": "tensorflow-2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}