CV2020 - 14 - Generative Adversarial Network (PyTorch).ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.5"},"colab":{"name":"CV2020 - 14 - Generative Adversarial Network (PyTorch).ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"f_ChrQTZyAZU"},"source":["![title](https://i.ibb.co/f2W87Fg/logo2020.png)\n","\n","---\n"]},{"cell_type":"markdown","metadata":{"id":"MQjL2XtX7uKJ"},"source":["<table  class=\"tfo-notebook-buttons\" align=\"left\"><tr><td>\n","    \n","<a href=\"https://colab.research.google.com/github/adf-telkomuniv/CV2020_Exercises/blob/main/CV2020 - 14 - Generative Adversarial Network (PyTorch).ipynb\" source=\"blank\" ><img src=\"https://colab.research.google.com/assets/colab-badge.svg\"></a>\n","</td><td>\n","<a href=\"https://github.com/adf-telkomuniv/CV2020_Exercises/blob/main/CV2020 - 14 - Generative Adversarial Network (PyTorch).ipynb\" source=\"blank\" ><img src=\"https://i.ibb.co/6NxqGSF/pinpng-com-github-logo-png-small.png\"></a>\n","    \n","</td></tr></table>"]},{"cell_type":"markdown","metadata":{"id":"8WGg98rBctSg"},"source":["\n","# Task 14 - Generative Adversarial Network (PyTorch)\n","\n","\n","So far in this course, all the applications of neural networks that we have explored have been **DISCRIMINATIVE MODELS**: that take an input and are trained to produce a labeled output.\n","\n"," This has ranged from straightforward classification of image categories to sentence generation (which was still phrased as a classification problem, our labels were in vocabulary space and we’d learned a recurrence to capture multi-word labels). \n","\n","In this notebook, we will expand our repetoire, and build **GENERATIVE MODELS** using neural networks. \n","\n","Specifically, we will learn how to build models which generate novel images that resemble a set of training images.\n"]},{"cell_type":"markdown","metadata":{"id":"SF71bN55cZzi"},"source":["Write down your Name and Student ID"]},{"cell_type":"code","metadata":{"id":"S5yg44U8cZzk"},"source":["## --- start your code here ----\n","\n","NIM  = ??\n","Nama = ??\n","\n","## --- end your code here ----"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"VXMkb4poctSu"},"source":["---\n","---\n","# [Part 0] What is GAN?\n","\n","In 2014, [Goodfellow et al.](https://arxiv.org/abs/1406.2661) presented a method for training generative models called Generative Adversarial Networks (GANs for short). \n","\n","GANs are not the only way to train a generative model! For other approaches to generative modeling check out the [deep generative model chapter](http://www.deeplearningbook.org/contents/generative_models.html) of the Deep Learning [book](http://www.deeplearningbook.org). \n","\n","As you've tried before, another popular way of training neural networks as generative models is Variational Autoencoders (co-discovered [here](https://arxiv.org/abs/1312.6114) and [here](https://arxiv.org/abs/1401.4082)). Variational autoencoders combine neural networks with Variational inference to train deep generative models. These models tend to be far more stable and easier to train but currently don't produce samples that are as pretty as GANs.\n","\n","<br>\n","\n","Since 2014, GANs have exploded into a huge research area, with massive [workshops](https://sites.google.com/site/nips2016adversarial/), and [hundreds of new papers](https://github.com/hindupuravinash/the-gan-zoo). Compared to other approaches for generative models, they often produce the highest quality samples but are some of the most difficult and finicky models to train \n","\n","\n","Improving the stabiilty and robustness of GAN training is an open research question, with new papers coming out every day! 2017 is said to be the year of GAN with so many research and papers published in investigating and improving GANs application. \n","\n","Several popular GAN architectures and implementation are: [WGAN](https://arxiv.org/abs/1701.07875), [WGAN-GP](https://arxiv.org/abs/1704.00028), [CGAN](https://arxiv.org/abs/1411.1784), [InfoGAN](https://arxiv.org/abs/1606.03657), [Pix2Pix](https://arxiv.org/abs/1611.07004), [CycleGAN](https://arxiv.org/abs/1703.10593), and many others."]},{"cell_type":"markdown","metadata":{"id":"UYvvSQVSctSv"},"source":["---\n","## 1 - GAN Networks\n","\n","In a GAN, we build two different neural networks. \n","\n","Our first network is a traditional classification network, called the **DISCRIMINATOR**. We will train the discriminator to take images, and classify them as being real (belonging to the training set) or fake (not present in the training set). \n","\n","Our other network, called the **GENERATOR**, will take random noise as input and transform it using a neural network to produce images. The goal of the generator is to fool the discriminator into thinking the images it produced are real.\n","\n","<br>\n","\n","We can think of this back and forth process of the generator ($G$) trying to fool the discriminator ($D$), and the discriminator trying to correctly classify real vs. fake as a minimax game:\n","\n","$$\\underset{G}{\\text{minimize}}\\; \\underset{D}{\\text{maximize}}\\; \\mathbb{E}_{x \\sim p_\\text{data}}\\left[\\log D(x)\\right] + \\mathbb{E}_{z \\sim p(z)}\\left[\\log \\left(1-D(G(z))\\right)\\right]$$\n","where \n","* $z \\sim p(z)$ are the random noise samples,\n","* $G(z)$ are the generated images using the neural network generator $G$, and\n","* $D$ is the output of the discriminator, specifying the probability of an input being real.\n","\n","<br>\n","\n","In [Goodfellow et al.](https://arxiv.org/abs/1406.2661), they analyze this minimax game and show how it relates to minimizing the Jensen-Shannon divergence between the training data distribution and the generated samples from $G$."]},{"cell_type":"markdown","metadata":{"id":"AVgCDu6sctSw"},"source":["---\n","## 2 - GAN Objectives\n","\n","To optimize this minimax game, we will aternate between taking gradient *descent* steps on the objective for $G$, and gradient *ascent* steps on the objective for $D$:\n","1. update the **generator** ($G$) to minimize the probability of the **discriminator making the CORRECT choice**. \n","\n","2. update the **discriminator** ($D$) to maximize the probability of the **discriminator making the CORRECT choice**.\n","\n","While these updates are useful for analysis, they do not perform well in practice. Instead, we will use a different objective when we update the generator: maximize the probability of the **discriminator making the INCORRECT choice**. This small change helps to allevaiate problems with the generator gradient vanishing when the discriminator is confident. \n","\n","<br>\n","\n","This is the standard update used in most GAN papers, and was used in the original paper from [Goodfellow et al.](https://arxiv.org/abs/1406.2661). "]},{"cell_type":"markdown","metadata":{"id":"2xvUaF9WctSy"},"source":["---\n","## 3 - In this Assignment\n","\n","In this assignment, we will alternate the following updates:\n","\n","1. Update the generator ($G$) to maximize the probability of the discriminator making the incorrect choice on generated data:\n","\n","$$\\underset{G}{\\text{maximize}}\\;  \\mathbb{E}_{z \\sim p(z)}\\left[\\log D(G(z))\\right]$$\n","\n","2. Update the discriminator ($D$), to maximize the probability of the discriminator making the correct choice on real and generated data:\n","\n","$$\\underset{D}{\\text{maximize}}\\; \\mathbb{E}_{x \\sim p_\\text{data}}\\left[\\log D(x)\\right] + \\mathbb{E}_{z \\sim p(z)}\\left[\\log \\left(1-D(G(z))\\right)\\right]$$"]},{"cell_type":"markdown","metadata":{"id":"jCK04OxLctTA"},"source":["---\n","## 4 - MNIST GAN \n","\n","GANs are notoriously finicky with hyperparameters, and also require many training epochs. In order to make this assignment approachable without a GPU, we will be working on the MNIST dataset, which is 60,000 training and 10,000 test images. \n","\n","Each picture contains a centered image of white digit on black background (0 through 9). This was one of the first datasets used to train convolutional neural networks and it is fairly easy -- a standard CNN model can easily exceed 99% accuracy. \n","\n","<br>\n","\n","\n","Here's an example of what your outputs from the 3 different models you're going to train should look like... note that GANs are sometimes finicky, so your outputs might not look exactly like this... this is just meant to be a *rough* guideline of the kind of quality you can expect:\n","\n","<br>\n","\n","<center>\n","<img src='https://image.ibb.co/d3SZsq/gan-outputs-pytorch.png'>"]},{"cell_type":"markdown","metadata":{"id":"ozHDPCYxeUWw"},"source":["---\n","---\n","# [Part 1] Import Libraries and Load Data"]},{"cell_type":"markdown","metadata":{"id":"12SMnaunBw96"},"source":["---\n","## 1 - Import Libraries\n","\n","Import requiered libraries"]},{"cell_type":"code","metadata":{"id":"gyI_HVN-ctSk"},"source":["import torch\n","from torch import nn, optim\n","from torch.nn import init\n","\n","import torchvision\n","import torchvision.transforms as T\n","\n","import os, shutil\n","import imageio, glob\n","import numpy as np\n","from IPython.display import Image\n","\n","import matplotlib.pyplot as plt\n","import matplotlib.gridspec as gridspec\n","\n","%matplotlib inline\n","plt.rcParams['figure.figsize'] = (12.0, 10.0) # set default size of plots\n","plt.rcParams['image.interpolation'] = 'nearest'\n","plt.rcParams['image.cmap'] = 'gray'\n","\n","np.set_printoptions(precision=5)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ChFnN6aNktMf"},"source":["---\n","## 2 - CPU/GPU Setting\n","This is the optional setting if you're using CPU Processing"]},{"cell_type":"code","metadata":{"id":"HkAsz5K2ktMh"},"source":["dtype = torch.cuda.FloatTensor \n","\n","## Uncomment out the following line if you're on a machine with a CPU set up for PyTorch!\n","# dtype = torch.FloatTensor "],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"M2yt0aybctTB"},"source":["---\n","## 3 - MNIST Dataset\n","\n","To simplify our code here, we will use the PyTorch MNIST wrapper, which downloads and loads the MNIST dataset. See the [documentation](https://github.com/pytorch/vision/blob/master/torchvision/datasets/mnist.py) for more information about the interface. \n"]},{"cell_type":"markdown","metadata":{"id":"gIaMNQRmBdbz"},"source":["---\n","### a. MNIST Parameters\n","\n","The default parameters will take $5,000$ of the training examples and place them into a validation dataset.\n","\n","The data will be saved into a folder called `MNIST_data`. "]},{"cell_type":"code","metadata":{"id":"DMk4OSViBqLw"},"source":["NUM_TRAIN = 50000\n","NUM_VAL   = 5000\n","NOISE_DIM = 96\n","\n","MNIST_DIR = './MNIST_data'\n","if os.path.exists(MNIST_DIR):\n","    shutil.rmtree(MNIST_DIR)\n","os.mkdir(MNIST_DIR)\n","\n","batch_size = 128"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"0gONoyJdAXUP"},"source":["---\n","### b. Image Sampler\n","\n","Class below is a sampler routine used to sequentially samples MNIST images  from some offset."]},{"cell_type":"code","metadata":{"id":"1WDxga1fAH78"},"source":["from torch.utils.data import sampler\n","\n","class ChunkSampler(sampler.Sampler):\n","    def __init__(self, num_samples, start=0):\n","        self.num_samples = num_samples\n","        self.start = start\n","\n","    def __iter__(self):\n","        return iter(range(self.start, self.start + self.num_samples))\n","\n","    def __len__(self):\n","        return self.num_samples\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"1ATT9LmzCFit"},"source":["---\n","### c. Load MNIST images\n","\n","Load data using DataLoader"]},{"cell_type":"code","metadata":{"id":"GVnqvs6kctTD"},"source":["import torchvision.datasets as dset\n","from torch.utils.data import DataLoader\n","\n","mnist_train = dset.MNIST(MNIST_DIR, train=True, download=True,\n","                           transform=T.ToTensor())\n","loader_train = DataLoader(mnist_train, batch_size=batch_size,\n","                          sampler=ChunkSampler(NUM_TRAIN, 0))\n","\n","mnist_val = dset.MNIST(MNIST_DIR, train=True, download=True,\n","                           transform=T.ToTensor())\n","loader_val = DataLoader(mnist_val, batch_size=batch_size,\n","                        sampler=ChunkSampler(NUM_VAL, NUM_TRAIN))\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Pdvd_1CFctS1"},"source":["---\n","## 4 - Helper Functions\n","\n","We provide you with some helper functions needed for this exercise\n","\n","You don't need to do anything in these cells."]},{"cell_type":"markdown","metadata":{"id":"okLO72JrwBCR"},"source":["---\n","### a. Show Images\n","function to display a set of image in a two-dimensional gdrid"]},{"cell_type":"code","metadata":{"id":"Cibe7q9j-1Xz"},"source":["def show_images(images):\n","    # images reshape to (batch_size, D)\n","    images  = np.reshape(images, [images.shape[0], -1])  \n","    sqrtn   = int(np.ceil(np.sqrt(images.shape[0])))\n","    sqrtimg = int(np.ceil(np.sqrt(images.shape[1])))\n","\n","    fig = plt.figure(figsize=(sqrtn, sqrtn))\n","    gs  = gridspec.GridSpec(sqrtn, sqrtn)\n","    gs.update(wspace=0.05, hspace=0.05)\n","\n","    for i, img in enumerate(images):\n","        ax = plt.subplot(gs[i])\n","        plt.axis('off')\n","        ax.set_xticklabels([])\n","        ax.set_yticklabels([])\n","        ax.set_aspect('equal')\n","        plt.imshow(img.reshape([sqrtimg,sqrtimg]))\n","        \n","    return "],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"BXiWCZTNctTI"},"source":["Run the code below to show some MNIST data sample"]},{"cell_type":"code","metadata":{"scrolled":false,"id":"8n4LrFmsctTK"},"source":["imgs = loader_train.__iter__().next()[0]\n","imgs = imgs.view(batch_size, 784).numpy().squeeze()\n","\n","show_images(imgs[:121])\n","plt.show()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"MyCam7Jg-12k"},"source":["---\n","### b. Preprocess Image\n","\n","`preprocess_img()` function is used to preprocess MNIST dataset images by normalizing it to range of $-1..1$\n","\n","and another `deprocess_img()` function to return the image back to range of $0..1$"]},{"cell_type":"code","metadata":{"id":"C0GotRAlctS7"},"source":["def preprocess_img(x):\n","    return 2 * x - 1.0\n","\n","def deprocess_img(x):\n","    return (x + 1.0) / 2.0"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"MCnbsppvwD-Q"},"source":["---\n","### c. Relative Error Function\n","\n","Function to calculate difference between your matrix and our expected results"]},{"cell_type":"code","metadata":{"id":"KQN7GSth_wMc"},"source":["def rel_error(x,y):\n","    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"RMMuO-WiA2RQ"},"source":["---\n","### d. Model Summary\n","\n","Function to display model graph and count its number of parameters \n","\n","Kind of a simple version of keras `.summary()` in PyTorch\n","\n","Actually, you can just use `print()` function to show the model's architecture. But this way, we can also print the number of trained parameters."]},{"cell_type":"code","metadata":{"id":"JVzQVK3qA2RY"},"source":["def count_params(model):\n","    param_count = np.sum([np.prod(p.size()) for p in model.parameters()])\n","\n","    base = str(model).split('\\n')\n","    base = base[1:-1]\n","    # base[-1] =\"(-) : Output()\"\n","    summary = []\n","    for layer in base:\n","        new_l = []\n","        layer = layer.replace('in_features=','')\n","        layer = layer.replace('out_features=','')\n","        layer = layer.replace('kernel_size','f')\n","        layer = layer.replace('=(','=[')\n","        layer = layer.replace('), ','], ')\n","        layer = layer.replace('momentum','m')\n","        layer = layer.replace('stride','s')\n","        layer = layer.replace('padding','p')\n","        layer = layer.replace('))','])')\n","        layer = layer.replace(', dilation=1, ceil_mode=False','') \n","        layer = layer.replace('negative_slope','alpha') \n","        layer = layer.replace(', affine=True, track_running_stats=True','') \n","        layer = layer.split(': ')\n","        new_l.append(layer[0].replace(' ',''))\n","        new_l.extend(layer[1].replace('(','\\n(').split('\\n'))\n","        summary.append(new_l)\n","        \n","    params = []\n","    for param in model.parameters():\n","        params.append(np.prod(param.size()))\n","    \n","    i = 0\n","    for s in summary:\n","        if s[1] in ['Linear', 'BatchNorm1d', 'BatchNorm2d', 'Conv2d', 'ConvTranspose2d' ]:\n","            s.append(params[i]+params[i+1])\n","            i+=2\n","\n","    line = \"{: ^4} {:>15}  {:>40} {:>10}\".format(\"ID\", \"Layer (type)\", \"Input-Output Shape\", \"Param #\")\n","    print(line)\n","    print(\"=========================================================================\")    \n","    for s in summary:\n","        if(len(s)<4):\n","            s.append('-')\n","        line = \"{: >4} {:>15}  {:>40} {:>10}\".format(*s)\n","        print(line)\n","\n","    # print(model)\n","    print('\\nTotal Parameters: {:,.0f}'.format(param_count))\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"5ApNwbbGctTa"},"source":["---\n","### e. Flatten and Unflatten\n","\n","Recall way back in our Task 08 Exercise?\n","\n","There, we provide you with Flatten operation.\n","\n","<br>\n","\n","\n","This time we also provide an **Unflatten** operation, which reshape an input vector into a 3-dimensional tensor.\n","\n","You'll it use when implementing the convolutional generator. \n"]},{"cell_type":"code","metadata":{"id":"eNb5KKKnctTb"},"source":["class Flatten(nn.Module):\n","    def forward(self, x):\n","        N, C, H, W = x.size()                 # read in N, C, H, W\n","        return x.view(N, -1)                  # \"flatten\" the C * H * W values into a single vector per image\n","    "],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vDhu-yP2G03p"},"source":["\n","An Unflatten module receives an input of shape $(N, C^*H^*W)$ and reshapes it to produce an output of shape $(N, C, H, W)$."]},{"cell_type":"code","metadata":{"id":"3-I4BnGZGvT_"},"source":["class Unflatten(nn.Module):\n","    def __init__(self, N=-1, C=128, H=7, W=7):\n","        super(Unflatten, self).__init__()\n","        self.N = N\n","        self.C = C\n","        self.H = H\n","        self.W = W\n","        \n","    def forward(self, x):\n","        return x.view(self.N, self.C, self.H, self.W)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"cj_3xhRYGpSm"},"source":["---\n","### f. Xavier Weight Initializer\n","\n","\n","We also provide a weight initializer (and call it for you) that uses Xavier initialization instead of PyTorch's uniform default."]},{"cell_type":"code","metadata":{"id":"s3JbrAxiGuT1"},"source":["def initialize_weights(m):\n","    if isinstance(m, nn.Linear) or isinstance(m, nn.ConvTranspose2d):\n","        init.xavier_uniform_(m.weight.data)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"6Uit1y7BUYzK"},"source":["---\n","### g. Generate GIF\n","\n","This function generate a GIF animation from the saved images and display it to your notebook"]},{"cell_type":"code","metadata":{"id":"jOFuzDwAUxVL"},"source":["def show_gif(base_dir, anim_file): \n","\n","    with imageio.get_writer(anim_file, mode='I') as writer:\n","        filenames = glob.glob(base_dir+'/image*.png')\n","        filenames = sorted(filenames)\n","        for filename in filenames:\n","            image = imageio.imread(filename)\n","            for i in range(3):\n","              writer.append_data(image)\n","        image = imageio.imread(filename)\n","        writer.append_data(image)\n","\n","    print('GIF saved as', anim_file)\n","\n","    with open(anim_file,'rb') as f:\n","      display(Image(data=f.read(), format='png'))"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"C0m1RBxo6NBW"},"source":["---\n","## 5 - Load File Checker\n","\n","we also provided a reference matrix to check whether the matrix produced by your implementation matched our expected output"]},{"cell_type":"code","metadata":{"id":"qWOvhJih6NBa"},"source":["!wget -q 'https://github.com/CNN-ADF/Task2019/raw/master/resources/gan-checks.npz'\n","\n","answers = dict(np.load('gan-checks.npz'))"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"AcO47wDG8RsZ"},"source":["---\n","---\n","# [Part 2] Vanilla GAN\n","\n","For starter, let's build our Vanilla GAN using Normal Neural Network (Linear Layer, not Convolutional Layer)\n","\n","After we build the training function for Vanilla GAN, for the other architecture, it's as simple as replacing the Loss and the Networks (Generator and Discriminator network)."]},{"cell_type":"markdown","metadata":{"id":"hmzfc7u2ctTO"},"source":["---\n","## 1 - Random Noise\n","\n","Similar to Variational Autoencoder from previous exercise, GAN Generator network generate an image from a input random vector, called `seed` or `noise`\n","\n","It is preferable that the seed is uniformly distributed in range of $(-1..1)$\n","\n","However, PyTorch only provides random uniform generator &nbsp;`torch.rand()`&nbsp; function which generate a PyTorch Tensor in range of $(0..1)$\n","\n","So we need to define a function to scale and shift it to the desired range"]},{"cell_type":"markdown","metadata":{"id":"aorPmaAuI8px"},"source":["---\n","#### <font color='red'>**EXERCISE:** </font>\n","\n","complete the function generate uniform noise from $-1$ to $1$ with shape `[batch_size, dim]`.\n","\n","Hint: use `torch.rand()`."]},{"cell_type":"code","metadata":{"id":"ERU1IUcnctTP"},"source":["def sample_noise(batch_size, dim):\n","\n","    # use torch.rand() to generate random matrix of shape (batch_size, dim) \n","    # then scale (multiply) by 2 and subtract by 1.\n","    noise = ??\n","        \n","    return noise\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"HXuwExjhctTT"},"source":["Make sure noise is the correct shape and type:"]},{"cell_type":"code","metadata":{"id":"5XsOUntRctTU"},"source":["torch.manual_seed(231)\n","batch_size = 3\n","dim        = 4\n","\n","z = sample_noise(batch_size, dim)\n","np_z = z.cpu().numpy()\n","print(np_z)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"lNFsH0WLctTZ"},"source":["**Expected Output**:\n","<pre>\n","[[-0.87505066 -0.9412632   0.11506891  0.799615  ]\n"," [-0.52444327  0.3785026   0.45648217 -0.93902993]\n"," [ 0.36680496 -0.26814795 -0.44410968 -0.9003619 ]]"]},{"cell_type":"markdown","metadata":{"id":"J3DrTVYMctTj"},"source":["---\n","## 2 - Discriminator Model\n","Our first step is to build a discriminator. Fill in the architecture as part of the `nn.Sequential` constructor in the function below. All fully connected layers should include bias terms. \n","\n","The architecture is:\n","<pre>\n","    * <b>Flatten</b> layer \n","    * <b>Fully connected</b> layer with input size <font color='blue'><b>784</b></font> and output size <font color='blue'><b>256</b></font>,\n","    * <b>LeakyReLU</b> activation with alpha <font color='blue'><b>0.01</b></font> \n","    * <b>Fully connected</b> layer with input size <font color='blue'><b>256</b></font> and output size <font color='blue'><b>256</b></font>,\n","    * <b>LeakyReLU</b> activation with alpha <font color='blue'><b>0.01</b></font> \n","    * <b>Fully connected</b> layer with input size <font color='blue'><b>256</b></font> and output size <font color='blue'><b>1</b></font>,\n","</pre>\n","\n","\n","Recall Leaky ReLU:\n","  * Leaky ReLU nonlinearity computes&nbsp; $f(x) = \\max(\\alpha x, x)$ &nbsp;for some fixed constant $\\alpha$; \n","  * for this architecture, we set&nbsp; $\\alpha=0.01$.\n"," \n"," <br>\n"," \n","The output of the discriminator should have shape `[batch_size, 1]`, and contain real numbers corresponding to the scores that each of the `batch_size` inputs is a real image.\n"]},{"cell_type":"markdown","metadata":{"id":"ZmC8f-rrIFsA"},"source":["---\n","#### <font color='red'>**EXERCISE:** </font>\n","Build and return a PyTorch model implementing the architecture above."]},{"cell_type":"code","metadata":{"id":"3-Q5pNC-ctTl"},"source":["def discriminator():\n","    model = nn.Sequential(\n","        \n","        # add Flatten() custom layer        \n","        ??,\n","        \n","        # add nn.Linear() layer with defined size above, \n","        ??,\n","        \n","        # add nn.LeakyRelu() with defined alpha above, \n","        ??,\n","        \n","        # add nn.Linear() layer with defined size above, \n","        ??,\n","        \n","        # add nn.LeakyRelu() with defined alpha above, \n","        ??,\n","        \n","        # add nn.Linear() layer with defined size above, \n","        ??\n","    )\n","    return model"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"8dHNuvHoctTn"},"source":["Test to make sure the number of parameters in the discriminator is correct:"]},{"cell_type":"code","metadata":{"id":"JwEkkvjqctTp"},"source":["model = discriminator()\n","\n","count_params(model) "],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Uo9JOxCtctTt"},"source":["**Expected Result**:\n","<pre>\n"," ID     Layer (type)               Input-Output Shape    Param #\n","================================================================\n"," (0)         Flatten                               ()          -\n"," (1)          Linear            (784, 256, bias=True)     200960\n"," (2)       LeakyReLU                     (alpha=0.01)          -\n"," (3)          Linear            (256, 256, bias=True)      65792\n"," (4)       LeakyReLU                     (alpha=0.01)          -\n"," (5)          Linear              (256, 1, bias=True)        257\n","\n","Total Parameters: 267,009"]},{"cell_type":"markdown","metadata":{"id":"BMcEOeroctTt"},"source":["---\n","## 3 - Generator Model\n","\n","Now to build the generator network\n","\n","\n","The architecture is:\n","<pre>\n","    * <b>Flatten</b> layer \n","    * <b>Fully connected</b> layer with input size <font color='blue'><b>noise_dim</b></font> and output size <font color='blue'><b>1024</b></font>,\n","    * <b>ReLU</b> activation\n","    * <b>Fully connected</b> layer with input size <font color='blue'><b>1024</b></font> and output size <font color='blue'><b>1024</b></font>,\n","    * <b>ReLU</b> activation\n","    * <b>Fully connected</b> layer with input size <font color='blue'><b>1024</b></font> and output size <font color='blue'><b>784</b></font>,\n","    * <b>TanH</b> activation\n","</pre>"]},{"cell_type":"markdown","metadata":{"id":"Ej2Tc92akD6R"},"source":["---\n","#### <font color='red'>**EXERCISE:** </font>\n","Build and return a PyTorch model implementing the architecture above."]},{"cell_type":"code","metadata":{"id":"Z9w7kgrgctTu"},"source":["def generator(noise_dim=NOISE_DIM):\n","    \"\"\"\n","    Build and return a PyTorch model implementing the architecture above.\n","    \"\"\"\n","    model = nn.Sequential(\n","        \n","        # add nn.Linear() layer with defined size above, \n","        ??,\n","        \n","        # add nn.Relu() activation\n","        ??,\n","        \n","        # add nn.Linear() layer with defined size above, \n","        ??,\n","        \n","        # add nn.Relu() activation\n","        ??,\n","        \n","        # add nn.Linear() layer with defined size above, \n","        ??,\n","        \n","        # add nn.Tanh() activation\n","        ??\n","    )\n","    return model"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"KLjCUaUUctTy"},"source":["Test to make sure the number of parameters in the generator is correct:"]},{"cell_type":"code","metadata":{"id":"93m7E-S1ctTz"},"source":["model = generator(4)\n","\n","count_params(model) "],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"lwS_vPmuctT3"},"source":["**Expected Result**:\n","<pre>\n"," ID     Layer (type)                 Input-Output Shape    Param #\n","==================================================================\n"," (0)          Linear               (4, 1024, bias=True)       5120\n"," (1)            ReLU                                 ()          -\n"," (2)          Linear            (1024, 1024, bias=True)    1049600\n"," (3)            ReLU                                 ()          -\n"," (4)          Linear             (1024, 784, bias=True)     803600\n"," (5)            Tanh                                 ()          -\n","\n","Total Parameters: 1,858,320"]},{"cell_type":"markdown","metadata":{"id":"jB7RQxe8ctT4"},"source":["---\n","---\n","# [Part 3] GAN Loss\n","\n","Compute the generator and discriminator loss. \n","\n","The generator loss is:\n","\n","$$\n","\\ell_G  =  -\\mathbb{E}_{z \\sim p(z)}\\left[\\log D(G(z))\\right]\n","$$\n","\n","and the discriminator loss is:\n","\n","$$ \n","\\ell_D = -\\mathbb{E}_{x \\sim p_\\text{data}}\\left[\\log D(x)\\right] - \\mathbb{E}_{z \\sim p(z)}\\left[\\log \\left(1-D(G(z))\\right)\\right]\n","$$\n","\n","<br>\n","\n","**Note** that these are negated from the equations presented earlier as we will be ***MINIMIZING*** these losses.\n","\n","<br>\n","\n","Instead of computing the expectation of $\\log D(G(z))$, $\\log D(x)$ and $\\log \\left(1-D(G(z))\\right)$, \n","\n","we will be averaging over elements of the minibatch, so make sure to combine the loss by averaging instead of summing."]},{"cell_type":"markdown","metadata":{"id":"FKe7ZAxNctT5"},"source":["---\n","## 1 - Binary Cross Entropy\n","\n","To compute the log probability of the true label given the logits output from the discriminator, we need to calculate the Binary Cross Entropy Loss.\n","\n","Given a score $s\\in\\mathbb{R}$ and a label $y\\in\\{0, 1\\}$, the binary cross entropy loss is\n","\n","$$ bce(s, y) = -y * \\log(s) - (1 - y) * \\log(1 - s) $$\n","\n","<br>\n","\n","A naive implementation of this formula can be numerically unstable, so we have provided a numerically stable implementation for you below.\n","\n","**Note: consult this [link](https://github.com/pytorch/pytorch/issues/751)*\n","\n"]},{"cell_type":"code","metadata":{"id":"PvKOijguctT7"},"source":["def bce_loss(input, target):\n","\n","    neg_abs = - input.abs()\n","    \n","    loss = input.clamp(min=0) - input * target + (1 + neg_abs.exp()).log()\n","    \n","    return loss.mean()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"GUi2ljTHctT9"},"source":["---\n","## 2 - Data Type conversion\n","\n","You will also need to compute labels corresponding to real or fake and use the logit arguments to determine their size. This computation will be performed in your selected device (CPU or GPU).\n","\n","If you are using GPU to train the network, you need to cast these labels to the correct data type using the global `dtype` variable, for example:\n","\n","```python\n","      true_labels = torch.ones(size).type(dtype)\n","```"]},{"cell_type":"markdown","metadata":{"id":"AENYMoQpctT_"},"source":["---\n","## 3 - Discriminator Loss\n","Discriminator Loss calculate the total loss of Discriminator Model in recognizing real images and fake images.\n","\n","The loss formula is as follow:\n","$$ \\ell_D = -\\mathbb{E}_{x \\sim p_\\text{data}}\\left[\\log D(x)\\right] - \\mathbb{E}_{z \\sim p(z)}\\left[\\log \\left(1-D(G(z))\\right)\\right]$$\n","\n","or in simple term:\n","\n","```python\n","      loss = loss_real + loss_fake\n","```\n","\n","where \n","* `loss_real` is the binary cross entropy loss result between `logits_real` and `ones` matrix,\n","\n","* while `loss_fake` is the binary cross entropy loss result between `logits_fake` and `zeros` matrix."]},{"cell_type":"markdown","metadata":{"id":"czXFhry9oeBW"},"source":["---\n","#### <font color='red'>**EXERCISE:** </font>\n","\n","Implement the Discriminator Loss as  described above."]},{"cell_type":"code","metadata":{"id":"fuUxbbCHctUA"},"source":["def discriminator_loss(logits_real, logits_fake):\n","    \n","    # get size of batch data\n","    N = logits_real.size(0)\n","    \n","    # create ONES tensor as real-data label\n","    # use torch.ones() with size N, remember to cast the type \n","    # (read instruction/example above)\n","    labels_real = ??.type(dtype)\n","    \n","    # calculate the loss_real\n","    # call bce_loss() function with input logits_real and labels_real\n","    loss_real = ??\n","    \n","    # create ZEROS tensor as fake-data label\n","    # use torch.zeros() with size N, remember to cast the type \n","    # (read instruction/example above)\n","    labels_fake = ??.type(dtype)\n","    \n","    # calculate the loss_fake\n","    # call bce_loss() function with input logits_fake and labels_fake\n","    loss_fake = ??\n","    \n","    # calculate the total discriminator loss\n","    loss = ??\n","        \n","    return loss"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"OlNXdOhDctUC"},"source":["Test your discriminator loss. \n","\n","You should see errors < `1e-7`."]},{"cell_type":"code","metadata":{"id":"Y0N3ysHictUC"},"source":["d_loss = discriminator_loss(torch.Tensor(answers['logits_real']).type(dtype),\n","                            torch.Tensor(answers['logits_fake']).type(dtype)).cpu().numpy()\n","\n","difference = rel_error(answers['d_loss_true'], d_loss)\n","\n","print(\"Maximum error in d_loss: %g\" % difference)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"B8LX-CM7ctUG"},"source":["**Expected Result**:\n","<pre>\n","Maximum error in d_loss: 2.83811e-08"]},{"cell_type":"markdown","metadata":{"id":"cQF2v0awctUH"},"source":["---\n","## 4 - Generator Loss\n","\n","Generator Model is trained to *fool* the Discriminator. It means that we want to maximize the log probability when predicting fake images.\n","\n","Thus, the Generator Loss is calculated using a simple Binary Cross Entropy as follow:\n","\n","$$\\ell_G  =  -\\mathbb{E}_{z \\sim p(z)}\\left[\\log D(G(z))\\right]$$\n","\n","the `loss` calculates the binary cross entropy loss result between `logits_fake` and `ones` matrix"]},{"cell_type":"markdown","metadata":{"id":"L2auHQxvrieG"},"source":["---\n","#### <font color='red'>**EXERCISE:** </font>\n","\n","Implement the Generator Loss as  described above."]},{"cell_type":"code","metadata":{"id":"Q74gPc5_ctUI"},"source":["def generator_loss(logits_fake):\n","    \n","    # get the number of batch data\n","    N = logits_fake.size(0)\n","    \n","    # create ONES tensor as fake-data label\n","    # use torch.ones() with size N, remember to cast the type \n","    labels_fake = ??\n","    \n","    # calculate the loss_fake\n","    # call bce_loss() function with input logits_fake and labels_fake\n","    loss = ??\n","    \n","    return loss"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"RWuObAAWctUJ"},"source":["Test your generator and discriminator loss. You should see errors < 1e-7."]},{"cell_type":"code","metadata":{"id":"XWL-jEPkctUK"},"source":["g_loss = generator_loss(torch.Tensor(answers['logits_fake']).type(dtype)).cpu().numpy()\n","\n","difference = rel_error(answers['g_loss_true'], g_loss)\n","\n","print(\"Maximum error in g_loss: %g\" % difference)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"cjx6oE6JctUN"},"source":["**Expected Result**:\n","<pre>\n","Maximum error in g_loss: 3.4188e-08\n"]},{"cell_type":"markdown","metadata":{"id":"GwuN37KxslBO"},"source":["---\n","---\n","# [Part 4] Training Function"]},{"cell_type":"markdown","metadata":{"id":"w0qEBCi5ctUN"},"source":["---\n","## 1 - Adam Optimizer\n","\n","To train our models, we will optimize it using Adam optimizer with a `1e-3` learning rate, beta1=`0.5`, beta2=`0.999`.\n","\n","Note that training a model in PyTorch requires an [Optimizer](https://pytorch.org/docs/stable/optim.html) object to perform the step and weights update. \n","\n","For that, we need two optimizers for each generator and discriminator models. And later, we need another four when we implement LS-GAN and DC-GAN. We set all of them use the same Adam optimizer with the same learning rate and betas.\n","\n","Therefore, rather than defining each of optimizers one-by-one, let's build a function to get the optimizer for any given model. We'll use this to construct optimizers for the generators and discriminators for the rest of the notebook.\n"]},{"cell_type":"markdown","metadata":{"id":"VOac3usffMjW"},"source":["---\n","#### <font color='red'>**EXERCISE:** </font>\n","\n","Construct and return an Adam optimizer for the model with `learning rate = 1e-3` and `betas = (0.5, 0.999)`."]},{"cell_type":"code","metadata":{"id":"LQwnHpxfctUO"},"source":["def get_optimizer(model):\n","\n","    # call optim.Adam() function with input model.parameters() \n","    # and defined learning rate and betas\n","    optimizer = ??\n","    \n","    return optimizer"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"q731E88actUS"},"source":["---\n","## 2 - Training Loop Function\n","\n","We provide you the main training loop... you won't need to change this function,\n","\n","but we **DO** encourage you to read through and understand it. "]},{"cell_type":"code","metadata":{"id":"PMxGHFZxctUT"},"source":["def train_gan(Discriminator, Generator, base_dir,\n","              show_every=250, batch_size=128, noise_size=96, num_epochs=10):\n","    \"\"\"\n","    Train a GAN!\n","    \n","    Inputs:\n","    - Discriminator  : Tuple of Discriminator package, containing:\n","      - D_model      : PyTorch models for the discriminator \n","      - D_solver     : torch.optim Optimizers for training \n","      - D_loss       : Functions to use for computing the discriminator loss.\n","\n","    - Generator      : Tuple of Generator package, containing:\n","      - G_model      : PyTorch models for the generator \n","      - G_solver     : torch.optim Optimizers for training \n","      - G_loss       : Functions to use for computing the generator loss.\n","\n","    - base_dir           : Directory to save the generated image example\n","    - show_every         : Show and save samples after every show_every iterations.\n","    - batch_size         : Batch size to use for training.\n","    - noise_size         : Dimension of the noise to use as input to the generator.\n","    - num_epochs         : Number of epochs over the training dataset to use for training.\n","    \"\"\"\n","\n","    # unpack Discriminator\n","    D_model, D_solver, D_loss = Discriminator\n","\n","    # unpack Generator\n","    G_model, G_solver, G_loss = Generator\n","\n","    # parameter setting\n","    iter_count = 0\n","    img_count  = 0\n","    seed_saved = sample_noise(batch_size, noise_size).type(dtype)\n","\n","    # make directory for saving image\n","    if os.path.exists(base_dir):\n","        shutil.rmtree(base_dir)\n","    os.mkdir(base_dir)\n","\n","    # start training\n","    for epoch in range(num_epochs):\n","        for x, _ in loader_train:\n","            if len(x) != batch_size:\n","                continue\n","            \n","            # reset the gradients\n","            D_solver.zero_grad()\n","            \n","            # get real images\n","            real_data   = x.type(dtype)\n","            logits_real = D_model(2* (real_data - 0.5)).type(dtype)\n","\n","            # 1. GENERATOR NETWORK FORWARD\n","            # generate new fake images\n","            g_fake_seed = sample_noise(batch_size, noise_size).type(dtype)\n","            fake_images = G_model(g_fake_seed).detach()\n","            logits_fake = D_model(fake_images.view(batch_size, 1, 28, 28))\n","\n","\n","            # 2. DISCRIMINATOR NETWORK FORWARD\n","            # forward fake and real logits to discriminator\n","            d_total_error = D_loss(logits_real, logits_fake)\n","            \n","            \n","            # 3. DISCRIMINATOR NETWORK BACKWARD\n","            # backward the error\n","            d_total_error.backward() \n","\n","            # update discriminator weights\n","            D_solver.step()\n","\n","\n","            # 4. GENERATOR NETWORK BACKWARD            \n","            # reset the gradients\n","            G_solver.zero_grad()\n","\n","            # generate new fake images\n","            g_fake_seed = sample_noise(batch_size, noise_size).type(dtype)\n","            fake_images = G_model(g_fake_seed)\n","\n","            # get logits fake throuch discriminator network\n","            gen_logits_fake = D_model(fake_images.view(batch_size, 1, 28, 28))\n","\n","            # calculate generator loss\n","            g_error = G_loss(gen_logits_fake)\n","\n","            # backward the error\n","            g_error.backward()\n","\n","            # update generator weights\n","            G_solver.step()\n","\n","            if (iter_count % show_every == 0):\n","\n","                print('Iter: {}, D loss: {:.4}, G loss:{:.4}'.format(iter_count,d_total_error.item(),g_error.item()))\n","\n","                # show randomly generated images from this iteration\n","                imgs_numpy = fake_images.data.cpu().numpy()\n","                show_images(imgs_numpy[0:16])\n","                plt.show()\n","\n","                # save images generated from the same initial seed\n","                # to visualize the generator progress along training\n","                filename = base_dir+'/image_'+'{:03}'.format(img_count)+'.png'\n","                fake_images = G_model(seed_saved)                \n","                imgs_numpy = fake_images.data.cpu().numpy()\n","                show_images(imgs_numpy[0:16])\n","                plt.savefig(filename)\n","                plt.close()\n","                print()\n","                img_count += 1\n","\n","            iter_count += 1\n","\n","    print('training done')\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vgc-5Bm0k33v"},"source":["---\n","## 3 - Train Vanilla GAN\n","\n","Well, that's it\n","\n","Now let's try to train a Vanilla GAN!"]},{"cell_type":"code","metadata":{"scrolled":true,"id":"Vnoy-EXlctUW"},"source":["# Make the discriminator\n","D_V = discriminator().type(dtype)\n","\n","# Make the generator\n","G_V = generator().type(dtype)\n","\n","# Use the function you wrote earlier to get optimizers for the Discriminator and the Generator\n","D_solver = get_optimizer(D_V)\n","G_solver = get_optimizer(G_V)\n","\n","# pack the model\n","D = ( D_V, D_solver, discriminator_loss )\n","G = ( G_V, G_solver, generator_loss )\n","\n","# Run it!\n","base_dir = 'v_gan'\n","train_gan(D, G, base_dir)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"SBod32HoctUY"},"source":["Well that wasn't so hard, was it? In the iterations in the low 100s you should see black backgrounds, fuzzy shapes as you approach iteration 1000, and decent shapes, about half of which will be sharp and clearly recognizable as we pass 3000."]},{"cell_type":"markdown","metadata":{"id":"ikYqrmMVUj16"},"source":["---\n","## 4 - Generate GIF\n","\n","Now this just for fun, we combine the saved image generated from the same initial seed each epoch while training into a GIF animation"]},{"cell_type":"code","metadata":{"id":"MwLP7vdEXNqg"},"source":["base_dir = 'v_gan'\n","\n","show_gif(base_dir, 'vanilla_gan.gif')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"3AhB2wsictUZ"},"source":["---\n","---\n","# [Part 5] Least Squares GAN\n","We'll now look at [Least Squares GAN](https://arxiv.org/abs/1611.04076), a newer, more stable alernative to the original GAN loss function. \n","\n","For this part, all we have to do is change the loss function and retrain the model. \n","\n","<br>\n","\n","We'll implement equation (9) in the paper, with the generator loss:\n","$$\n","\\ell_G  =  \\frac{1}{2}\\mathbb{E}_{z \\sim p(z)}\\left[\\left(D(G(z))-1\\right)^2\\right]$$\n","\n","and the discriminator loss:\n","\n","$$ \\ell_D = \\frac{1}{2}\\mathbb{E}_{x \\sim p_\\text{data}}\\left[\\left(D(x)-1\\right)^2\\right] + \\frac{1}{2}\\mathbb{E}_{z \\sim p(z)}\\left[ \\left(D(G(z))\\right)^2\\right]\n","$$"]},{"cell_type":"markdown","metadata":{"id":"LFNPOjP_ctUZ"},"source":["---\n","## 1 - LS-Discriminator Loss\n","\n","In <font color='red'>**Vanilla GAN**</font>, the Discriminator Loss is the total <font color='red'>binary cross entropy loss</font> from discriminator when recognizing real images and fake images. And we train the network to **MINIMIZE** that total loss\n","\n","<br>\n","\n","In <font color='blue'>**LS-GAN**</font>, we change the BCE loss calculation into a simple <font color='blue'>score averaging</font>. \n","\n","The idea is that since the Discriminator is a binary classification, it means that:\n","\n","* we want the `real_score` when predicting **real images** to be **as close to 1 as possible**,\n","\n","* and the `fake_score` when prediction **fake images** to be **as close to 0 as possible**.\n","\n","<br>\n","\n","Now if we ***flip*** the real scores by subtracting it with 1, this equalize our objectives so that now we want both `fake_score` and `real_score-1` to be as close to 0 as possible. And if we square the scores, now the values are always positive, and our objectives all become <font color='blue'>**minimization to 0**</font>.\n","\n","<br>\n","\n","As so we can just simply average the squared scores, and train the network to **MINIMIZE** the average\n","\n","Thus we have:\n","\n","$$\n","\\ell_D = \\frac{1}{2}\\mathbb{E}_{x \\sim p_\\text{data}}\\left[\\left(D(x)-1\\right)^2\\right] + \\frac{1}{2}\\mathbb{E}_{z \\sim p(z)}\\left[ \\left(D(G(z))\\right)^2\\right]\n","$$\n","\n","or its just simply,\n","\n","```python\n","    loss_real = mean( square( scores_real-1 ) )\n","    loss_fake = mean( square( scores_fake ) )\n","\n","    loss = 0.5 * loss_real + 0.5 * loss_fake\n","```\n","\n","However, instead of computing the expectation, we will be averaging over elements of the minibatch, so make sure to combine the loss by averaging instead of summing. \n","\n","When plugging in for $D(x)$ and $D(G(z))$ use the direct output from the discriminator (`scores_real` and `scores_fake`)."]},{"cell_type":"markdown","metadata":{"id":"E3zuwNhNn7Pu"},"source":["---\n","#### <font color='red'>**EXERCISE:** </font>\n","\n","Implement the Least Squares GAN Discriminator loss as defined above"]},{"cell_type":"code","metadata":{"id":"Sqpv9waRctUb"},"source":["def ls_discriminator_loss(scores_real, scores_fake):\n","\n","    # calculate loss_real:\n","    # - first calculate the square of (scores_real-1)\n","    # - use torch.mean with input to average the squared score\n","    loss_real = ??\n","    \n","    # calculate loss_fake:\n","    # - use torch.mean() with input square of (scores_fake)\n","    loss_fake = ??\n","    \n","    # average the loss_real and loss_fake\n","    loss = ??\n","\n","    return loss"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"1psf4C9mctUd"},"source":["Before running a GAN with our new loss function, let's check it:\n","\n","You should see error <`1e-7`"]},{"cell_type":"code","metadata":{"id":"DeGEoK9sctUe"},"source":["score_real = torch.Tensor(answers['logits_real']).type(dtype)\n","score_fake = torch.Tensor(answers['logits_fake']).type(dtype)\n","\n","d_loss = ls_discriminator_loss(score_real, score_fake).cpu().numpy()\n","\n","difference = rel_error(answers['d_loss_lsgan_true'], d_loss)\n","\n","print(\"Maximum error in d_loss: %g\" % difference)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"0Ef2rYhNctUh"},"source":["**Expected Result**:\n","<pre>\n","Maximum error in g_loss: 1.53171e-08\n"]},{"cell_type":"markdown","metadata":{"id":"aXyDMaz3ctUh"},"source":["---\n","## 2 - LS-Generator Loss\n","\n","Similar to Vanilla GAN, Generator Loss is designed to minimize the correct classification of discriminator in predicting fake images. \n","\n","Before we use BCE loss, now in LS-GAN, since the Discriminator is using simple mean square error, then we have to treat the Generator the same way.\n","\n","Thus our objective for Generator Model is to maximize the `fake_score` to be as close to 1 as possible, or flip it to minimizing `fake_score-1` to be as close to 0 as possible\n","\n","And so we have:\n","\n","$$\n","\\ell_G  =  \\frac{1}{2}\\mathbb{E}_{z \\sim p(z)}\\left[\\left(D(G(z))-1\\right)^2\\right]\n","$$\n","\n","or in simply, \n","```python\n","    loss = 1/2 * mean( square( scores_fake-1 ) )\n","```\n"]},{"cell_type":"markdown","metadata":{"id":"YdQ8OufvwkJk"},"source":["---\n","#### <font color='red'>**EXERCISE:** </font>\n","\n","Implement the Least Squares GAN Generator loss"]},{"cell_type":"code","metadata":{"id":"Pl4n6HxActUi"},"source":["def ls_generator_loss(scores_fake):\n","    \n","    # calculate generator loss,\n","    # first use torch.mean() with input square of (scores_fake-1)\n","    # then divide by 2\n","    loss = ??\n","\n","    return loss"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"f2b9QHumctUk"},"source":["Before running a GAN with our new loss function, let's check it:\n","\n","You should see error <`1e-7`"]},{"cell_type":"code","metadata":{"id":"HzrX73H-ctUl"},"source":["score_real = torch.Tensor(answers['logits_real']).type(dtype)\n","score_fake = torch.Tensor(answers['logits_fake']).type(dtype)\n","\n","d_loss = ls_discriminator_loss(score_real, score_fake).cpu().numpy()\n","g_loss = ls_generator_loss(score_fake).cpu().numpy()\n","\n","g_difference = rel_error(answers['d_loss_lsgan_true'], d_loss)\n","d_difference = rel_error(answers['g_loss_lsgan_true'], g_loss)\n","\n","print(\"Maximum error in d_loss: %g\" % g_difference)\n","print(\"Maximum error in g_loss: %g\" % d_difference)\n","\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"miT18EBFctUn"},"source":["**Expected Result**:\n","<pre>\n","Maximum error in d_loss: 1.53171e-08\n","Maximum error in g_loss: 2.7837e-09\n"]},{"cell_type":"markdown","metadata":{"id":"jIC6iJeNctUn"},"source":["---\n","## 3 - Train LS-GAN\n","Run the following cell to train your model!"]},{"cell_type":"code","metadata":{"scrolled":true,"id":"Nhn2FlmTctUo"},"source":["# Make new discriminator model\n","D_LS = discriminator().type(dtype)\n","\n","# Make new generator model\n","G_LS = generator().type(dtype)\n","\n","# Use the get_optimizer() function to get optimizers for the Discriminator and the Generator\n","D_LS_solver = get_optimizer(D_LS)\n","G_LS_solver = get_optimizer(G_LS)\n","\n","# pack the model into tuple\n","# pack D_LS, D_LS_solver, and ls_discriminator_loss into a single tuple\n","D = ( D_LS, D_LS_solver, ls_discriminator_loss )\n","\n","# pack G_LS, G_LS_solver, and ls_generator_loss into a single tuple\n","G = ( G_LS, G_LS_solver, ls_generator_loss )\n","\n","base_dir = 'ls_gan'\n","\n","# train the networks by calling train_gan() function\n","train_gan(D, G, base_dir)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"dwDClYWK-Hqy"},"source":["You should see that using LS-GAN, the generated image looks more sharp and clearly recognizable compared to Vanilla GAN."]},{"cell_type":"markdown","metadata":{"id":"76uYAfsS3ei1"},"source":["---\n","## 4 - Generate GIF\n","\n","Now this just for fun, we combine the saved image generated from the same initial seed each epoch while training into a GIF animation"]},{"cell_type":"code","metadata":{"id":"Uv6E7bc13ei5"},"source":["base_dir = 'ls_gan'\n","\n","show_gif(base_dir, 'ls_gan.gif')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"uW9-8WaxctUq"},"source":["---\n","---\n","# [Part 6] Deeply Convolutional GANs\n","In the first part of the notebook, we implemented an almost direct copy of the original GAN network from Ian Goodfellow. \n","\n","However, this network architecture allows no real spatial reasoning. It is unable to reason about things like \"sharp edges\" in general because it lacks any convolutional layers. \n","\n","Thus, in this section, we will implement some of the ideas from [DCGAN](https://arxiv.org/abs/1511.06434), where we use convolutional networks "]},{"cell_type":"markdown","metadata":{"id":"1LkxLE4CctUr"},"source":["---\n","## 1 - DC-Discriminator Model\n","\n","We will use a 4 layer ConvNet as our Discriminator Network \n","\n","The architecture is as follow:<pre>\n","    01. <b>Unflatten</b> layer       to reshape input into <font color='blue'><b>(1,28,28)</b></font> \n","    02. <b>Conv2D</b> layer          with <font color='blue'><b>32</b></font> filters of <font color='blue'><b>5x5</b></font> using stride <font color='blue'><b>1</b></font>,\n","    03. <b>LeakyReLU</b> activation  with alpha <font color='blue'><b>0.01</b></font> \n","    04. <b>Max Pool</b> layer        with kernel size of <font color='blue'><b>2x2</b></font> \n","    05. <b>Conv2D</b> layer          with <font color='blue'><b>64</b></font> filters of <font color='blue'><b>5x5</b></font> using stride <font color='blue'><b>1</b></font>,\n","    06. <b>LeakyReLU</b> activation  with alpha <font color='blue'><b>0.01</b></font> \n","    07. <b>Max Pool</b> layer        with kernel size of <font color='blue'><b>2x2</b></font> \n","    08. <b>Flatten</b> layer         to reshape activation into vector\n","    09. <b>Fully Connected</b>       with input size <font color='blue'><b>4\\*4\\*64</b></font> and output size also <font color='blue'><b>4\\*4\\*64</b></font>,\n","    10. <b>LeakyReLU</b> activation  with alpha <font color='blue'><b>0.01</b></font> \n","    11. <b>Fully Connected</b>       with input size <font color='blue'><b>4\\*4\\*64</b></font> and output size <font color='blue'><b>1</b></font>,\n","</pre>\n","\n","---\n","RTFM: Read the [Documentation](https://pytorch.org/docs/stable/nn.html)"]},{"cell_type":"markdown","metadata":{"id":"6-k26867Im5G"},"source":["---\n","#### <font color='red'>**EXERCISE:** </font>\n","Build and return a PyTorch model implementing the architecture above.\n","\n","Note: \n","\n","[.Conv2d()](https://pytorch.org/docs/stable/nn.html#conv2d) layer from .nn module receive required input as follow:\n","<pre>\n","* in_channel  : input channel dimension\n","* out_channel : output channel dimension (number of filter)\n","* kernel_size : either integer or tuple\n","* stride      : optional argument, default=1</pre>\n","\n","while [.MaxPool2d()](https://pytorch.org/docs/stable/nn.html#maxpool2d) layer only receive required input as follow:\n","<pre>\n","* kernel_size : either integer or tuple\n","* stride      : optional argument, default equal to kernel_size</pre>"]},{"cell_type":"code","metadata":{"scrolled":false,"id":"rafpCGIcctUs"},"source":["def dc_discriminator():\n","  \n","    return nn.Sequential(\n","        # 01. call Unflatten() function with input N=batch_size, C=1, W=28, and H=28\n","        ??,\n","        \n","        # 02. add Conv2d() from .nn module with 32 Filters, kernel size 5, Stride 1 \n","        # (from input channel = 1)\n","        ??,\n","        \n","        # 03. add Leaky ReLU from .nn module with alpha=0.01\n","        ??,\n","        \n","        # 04. add MaxPool2d() from .nn module with kernel size 2, Stride 2\n","        ??,\n","        \n","        # 05. add Conv2d() from .nn module with 64 Filters, kernel size 5, Stride 1 \n","        # (from input channel = 32)\n","        ??,\n","        \n","        # 06. add Leaky ReLU from .nn module with alpha=0.01\n","        ??,\n","        \n","        # 07. add MaxPool2d() from .nn module with kernel size 2x2, Stride 2\n","        ??,\n","        \n","        # 08. add Flatten() function\n","        ??,\n","        \n","        # 09. add Linear layer from .nn module\n","        # with both input and output size 4*4*64\n","        ??,\n","        \n","        # 10. add Leaky ReLU from .nn module with alpha=0.01\n","        ??,\n","        \n","        # 11. add Linear layer from .nn module\n","        # with input 4*4*64 and output 1\n","        ??\n","        \n","    )\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"FMqnJDpuO5lb"},"source":["Test to make sure the number of parameters in the discriminator is correct:"]},{"cell_type":"code","metadata":{"id":"3bqcnFp-O5li"},"source":["model = dc_discriminator()\n","\n","count_params(model) "],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"NzIQOBWMO5ls"},"source":["**Expected Result**:\n","<pre>\n"," ID     Layer (type)                      Input-Output Shape    Param #\n","=======================================================================\n"," (0)       Unflatten                                      ()          -\n"," (1)          Conv2d             (1, 32, f=[5, 5], s=[1, 1])        832\n"," (2)       LeakyReLU                            (alpha=0.01)          -\n"," (3)       MaxPool2d                         (f=2, s=2, p=0)          -\n"," (4)          Conv2d            (32, 64, f=[5, 5], s=[1, 1])      51264\n"," (5)       LeakyReLU                            (alpha=0.01)          -\n"," (6)       MaxPool2d                         (f=2, s=2, p=0)          -\n"," (7)         Flatten                                      ()          -\n"," (8)          Linear                 (1024, 1024, bias=True)    1049600\n"," (9)       LeakyReLU                            (alpha=0.01)          -\n","(10)          Linear                    (1024, 1, bias=True)       1025\n","\n","Total Parameters: 1,102,721"]},{"cell_type":"markdown","metadata":{"id":"8hKaYDR9PM3x"},"source":["Now try to feed an input image and check the output shape"]},{"cell_type":"code","metadata":{"id":"yzkHy56zPA_1"},"source":["batch_size = 128\n","\n","data = next(enumerate(loader_train))[-1][0].type(dtype)\n","print('Data shape   :', data.shape)\n","\n","b   = dc_discriminator().type(dtype)\n","out = b(data)\n","print('Output shape :', out.size())"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-tN-BpBfctUu"},"source":["**Expected Result**:\n","<pre>\n","Data shape   : torch.Size([128, 1, 28, 28])\n","Output shape : torch.Size([128, 1])\n"]},{"cell_type":"markdown","metadata":{"id":"yyEMzdm5ctUz"},"source":["---\n","## 2 - DC-Generator Model\n","For the generator, we will copy the architecture exactly from the [InfoGAN paper](https://arxiv.org/pdf/1606.03657.pdf). \n","\n","The architecture is as follow:<pre>\n","    01. <b>Fully connected</b>  with input size <font color='blue'><b>noise_dim</b></font> and output size <font color='blue'><b>1024</b></font>,\n","    02. <b>ReLU</b> activation\n","    03. <b>BatchNorm1D</b>      with input size <font color='blue'><b>1024</b></font>\n","    04. <b>Fully connected</b>  with input size <font color='blue'><b>1024</b></font> and output size <font color='blue'><b>128\\*7\\*7</b></font>,\n","    05. <b>ReLU</b> activation\n","    06. <b>BatchNorm1D</b>      with input size <font color='blue'><b>128\\*7\\*7</b></font>\n","    07. <b>Unflatten</b> layer  to reshape activation into <font color='blue'><b>(128,7,7)</b></font> \n","    08. <b>Conv2DTranspose</b>  with <font color='blue'><b>64</b></font> filters of <font color='blue'><b>4x4</b></font> using stride <font color='blue'><b>2</b></font>, and <font color='blue'><b>padding 1</b></font>\n","    09. <b>ReLU</b> activation\n","    10. <b>BatchNorm2D</b>      with input size <font color='blue'><b>64</b></font>\n","    11. <b>Conv2DTranspose</b>  with <font color='blue'><b>1</b></font> filters of <font color='blue'><b>4x4</b></font> using stride <font color='blue'><b>2</b></font>, and <font color='blue'><b>padding 1</b></font>\n","    12. <b>TanH</b> activation\n","    13. <b>Flatten</b> layer    to reshape activation into <font color='blue'><b>784</b></font> vector    \n","</pre>\n","\n","---\n","RTFM: Read the [Documentation](https://pytorch.org/docs/stable/nn.html)"]},{"cell_type":"markdown","metadata":{"id":"P00D4tTmhu66"},"source":["<font size=3><b> Something Weird? </b></font>\n","\n","Note that the architecture reshape back it's already made output image of $(1,28,28)$ into a vector if $(784,)$\n","\n","*weird, huh?*\n","\n","<br>\n","\n","Well, we need to reshape it back since we made our training function to receive and save images from a vector-shaped output from the previously Vanilla GAN\n","\n","Since we don't want to change the training function just for this DC-GAN, it's easier to design the generator model to just return a vector"]},{"cell_type":"markdown","metadata":{"id":"lVVu478gcPPo"},"source":["---\n","#### <font color='red'>**EXERCISE:** </font>\n","Build and return a PyTorch model implementing the architecture above."]},{"cell_type":"code","metadata":{"scrolled":true,"id":"HjEFnznFctUz"},"source":["def dc_generator(noise_dim=NOISE_DIM):\n","\n","    return nn.Sequential(\n","        \n","        # 01. add Linear() with input noise_dim and output 1024\n","        ??,\n","        \n","        # 02. add ReLU() from .nn module\n","        ??,\n","        \n","        # 03. add BatchNorm1d() with input 1024\n","        ??,\n","        \n","        # 04. add Linear() with input 1024 and output 7*7*128\n","        ??,\n","        \n","        # 05. add ReLU() from .nn module\n","        ??,\n","        \n","        # 06. add BatchNorm1d() with input 7*7*128\n","        ??,\n","        \n","        # 07. add Unflatten() with input N=batch_size, C=128, W=7, and H=7\n","        ??,\n","        \n","        # 08. add ConvTranspose2d() with input channel = 128, 64 Filters, \n","        # kernel size 4, stride 2, and padding 1 \n","        ??,\n","        \n","        # 09. add ReLU() from .nn module\n","        ??,\n","        \n","        # 10. add BatchNorm2d() with input 64\n","        ??,\n","        \n","        # 11. add ConvTranspose2d() with input channel = 64, 1 Filter, \n","        # kernel size 4, stride 2, and padding 1 \n","        ??,\n","        \n","        # 12. add Tanh() from .nn module\n","        ??,\n","        \n","        # 13. add back Flatten() \n","        ??\n","        \n","    )\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ulAX1YIUkJfZ"},"source":["Test to make sure the number of parameters in the discriminator is correct:"]},{"cell_type":"code","metadata":{"id":"fiGZ8aDkkJff"},"source":["model = dc_generator(4)\n","\n","count_params(model) "],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"B794kUoFkJfr"},"source":["**Expected Result**:\n","\n","<pre>\n"," ID     Layer (type)                        Input-Output Shape    Param #\n","=========================================================================\n"," (0)          Linear                      (4, 1024, bias=True)       5120\n"," (1)            ReLU                                        ()          -\n"," (2)     BatchNorm1d                  (1024, eps=1e-05, m=0.1)       2048\n"," (3)          Linear                   (1024, 6272, bias=True)    6428800\n"," (4)            ReLU                                        ()          -\n"," (5)     BatchNorm1d                  (6272, eps=1e-05, m=0.1)      12544\n"," (6)       Unflatten                                        ()          -\n"," (7) ConvTranspose2d   (128, 64, f=[4, 4], s=[2, 2], p=[1, 1])     131136\n"," (8)            ReLU                                        ()          -\n"," (9)     BatchNorm2d                    (64, eps=1e-05, m=0.1)        128\n","(10) ConvTranspose2d     (64, 1, f=[4, 4], s=[2, 2], p=[1, 1])       1025\n","(11)            Tanh                                        ()          -\n","(12)         Flatten                                        ()          -\n","\n","Total Parameters: 6,580,801"]},{"cell_type":"markdown","metadata":{"id":"TKAUZL-KkJfs"},"source":["Now try to feed an input image and check the output shape"]},{"cell_type":"code","metadata":{"id":"yxoTuKu7kAXM"},"source":["batch_size = 128\n","\n","test_g_gan = dc_generator().type(dtype)\n","test_g_gan.apply(initialize_weights)\n","\n","fake_seed   = torch.randn(batch_size, NOISE_DIM).type(dtype)\n","fake_images = test_g_gan.forward(fake_seed)\n","\n","print('Output shape :',fake_images.size())"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Lgg2UAR7ctU2"},"source":["**Expected Result**:\n","<pre>\n","Output shape : torch.Size([128, 784])\n"]},{"cell_type":"markdown","metadata":{"id":"ls03xv6d8Oz4"},"source":["---\n","## 3 - Train DC-GAN\n","Run the following cell to train your model!"]},{"cell_type":"code","metadata":{"scrolled":true,"id":"VRBmShhf8O0L"},"source":["# Make new discriminator model\n","D_DC = dc_discriminator().type(dtype)\n","\n","# Make new generator model\n","G_DC = dc_generator().type(dtype)\n","\n","# apply weight initialization\n","G_DC.apply(initialize_weights)\n","D_DC.apply(initialize_weights)\n","\n","\n","# Use the get_optimizer() function to get optimizers for the Discriminator and the Generator\n","D_DC_solver = get_optimizer(D_DC)\n","G_DC_solver = get_optimizer(G_DC)\n","\n","# pack the model into tuple\n","# pack D_dc, D_dc_solver, and discriminator_loss into a single tuple\n","D = ( D_DC, D_DC_solver, discriminator_loss )\n","\n","# pack G_dc, G_dc_solver, and generator_loss into a single tuple\n","G = ( G_DC, G_DC_solver, generator_loss )\n","\n","base_dir = 'dc_gan'\n","\n","# train the networks by calling train_gan() function\n","train_gan(D, G, base_dir)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"mTkpwOD58O0S"},"source":["You should see that now using DC-GAN, from the start, the generated image looks much more clean and natural compared to using Linear GAN."]},{"cell_type":"markdown","metadata":{"id":"oBAjP-ue8O0V"},"source":["---\n","## 4 - Generate GIF\n","\n","Now this just for fun, we combine the saved image generated from the same initial seed each epoch while training into a GIF animation"]},{"cell_type":"code","metadata":{"id":"7CUFPbUb8O0W"},"source":["base_dir = 'dc_gan'\n","\n","show_gif(base_dir, 'dc_gan.gif')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"d5gmDwFrctVg"},"source":["Pretty Neat, eh?"]},{"cell_type":"markdown","metadata":{"id":"HoGECWnwctU-"},"source":["---\n","---\n","# [Part 7] Generate new Images\n","\n","Let's see the result of our GAN with the same sample seed start"]},{"cell_type":"markdown","metadata":{"id":"CXxgRLTmctU_"},"source":["---\n","## 1 - Generate random noise\n"]},{"cell_type":"code","metadata":{"id":"RPrfTFL4ctVA"},"source":["fake_seed = sample_noise(128, 96).type(dtype)\n","\n","print(fake_seed.shape)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"GqdouV-LctVC"},"source":["Example of seed of an image"]},{"cell_type":"code","metadata":{"id":"y_aHbw-qctVC"},"source":["print(fake_seed[0])"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"imMJbbbHctVF"},"source":["---\n","## 2 - Vanilla GAN Generated Images"]},{"cell_type":"code","metadata":{"id":"GiWSU7jvctVG"},"source":["fake_images = G_V(fake_seed)\n","\n","print(fake_images.shape)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"a-b_n1v99iFq"},"source":["View the generated images"]},{"cell_type":"code","metadata":{"id":"GYbWFbXgctVI"},"source":["image_to_view = 20\n","imgs_numpy = fake_images.data.cpu().numpy()\n","show_images(imgs_numpy[0:image_to_view])\n","plt.show()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"sEuZ-XBU-Muy"},"source":["We can generate single image from a vector of fake seed"]},{"cell_type":"code","metadata":{"id":"K8pOUYJp-Mu4"},"source":["single_seed = sample_noise(1, 96)\n","print('Vanilla GAN generated image\\nfrom seed', single_seed[0,:4],'...\\n')\n","\n","single_seed = single_seed.type(dtype)\n","fake_images = G_V(single_seed)\n","imgs_numpy = fake_images.data.cpu().numpy()\n","\n","plt.figure(figsize=(5,5))\n","plt.imshow(imgs_numpy.reshape((28,28)))\n","plt.axis('off')\n","plt.show()\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"QMvpMkYQctVK"},"source":["---\n","## 3 - LS-GAN Generated Images\n"]},{"cell_type":"code","metadata":{"id":"2VZ_7D0_ctVK"},"source":["fake_images = G_LS(fake_seed)\n","\n","print(fake_images.shape)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-jMZPDEI9yT-"},"source":["View the generated images"]},{"cell_type":"code","metadata":{"id":"KbnmH9nVctVN"},"source":["image_to_view = 20\n","imgs_numpy = fake_images.data.cpu().numpy()\n","show_images(imgs_numpy[0:image_to_view])\n","plt.show()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"zSs2ObJD-THU"},"source":["We can generate single image from a vector of fake seed"]},{"cell_type":"code","metadata":{"id":"foBcItTj-THv"},"source":["single_seed = sample_noise(1, 96)\n","print('LS GAN generated image\\nfrom seed', single_seed[0,:4],'...\\n')\n","\n","single_seed = single_seed.type(dtype)\n","fake_images = G_LS(single_seed)\n","imgs_numpy = fake_images.data.cpu().numpy()\n","\n","plt.figure(figsize=(5,5))\n","plt.imshow(imgs_numpy.reshape((28,28)))\n","plt.axis('off')\n","plt.show()\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"R5VrHdb1ctVP"},"source":["---\n","## 4 - DC-GAN Generated Images\n"]},{"cell_type":"code","metadata":{"id":"Rscjlgg3ctVQ"},"source":["fake_images = G_DC(fake_seed)\n","\n","print(fake_images.shape)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"eUIF9Zx39ysh"},"source":["View the generated images"]},{"cell_type":"code","metadata":{"scrolled":true,"id":"3V7y5ygqctVR"},"source":["image_to_view = 25\n","imgs_numpy = fake_images.data.cpu().numpy()\n","show_images(imgs_numpy[0:image_to_view])\n","plt.show()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"lnqaVmxFIH45"},"source":["Looks great isn't it?\n","\n","Now let's compare it to the real MNIST dataset"]},{"cell_type":"code","metadata":{"id":"4Q9UguU1Q2kX"},"source":["imgs = loader_train.__iter__().next()[0]\n","imgs = imgs.view(batch_size, 784).numpy().squeeze()\n","print(imgs.shape)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"SHgfHD9vIN-z"},"source":["combined = np.vstack([imgs_numpy[:50,:], imgs[:50,:]])\n","idx = np.arange(combined.shape[0])\n","combined = combined[np.random.shuffle(idx)][0]\n","show_images(combined)\n","plt.show()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"yWG28k59IRR3"},"source":["The images above are the combination between generated images and the real MNIST images.\n","\n","Can you tell which are the original images and which are the fake ones?\n","\n","Can you tell the difference?"]},{"cell_type":"markdown","metadata":{"id":"xuujlTQYhhVW"},"source":["---\n","---\n","\n","# Congratulation, You've Completed Exercise 14\n","\n","<p>Copyright &copy;  <a href=https://www.linkedin.com/in/andityaarifianto/>2020 - ADF</a> </p>"]},{"cell_type":"markdown","metadata":{"id":"7_Hkr9kg9Do-"},"source":["![footer](https://i.ibb.co/yX0jfMS/footer2020.png)"]}]}