Autoregression_Emojis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generating Emojis with an Autoregression model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project I implemented an autoregressive likelihood model for the task of image modeling. It was trained on the apple emojis from the [Full Emoji Image Dataset](https://www.kaggle.com/datasets/subinium/emojiimage-dataset/data).\n",
    "\n",
    "Firstly I tried to use all the emojis from the dataset, but after some testing I realised that better results were achieved by only training on the \"face-like\" emojis, There was insufficient training data on the other types of emojis since they differ a lot and that confused the model.\n",
    "\n",
    "The inspiration for this implementation was the [PixelCNN](https://arxiv.org/pdf/1606.05328.pdf) model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, let's import the libraries and initialize the device. (This was done on a mac device, change to \"cuda\" if applicable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mps\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "device = torch.device(\"mps\" if torch.backends.mps.is_available() else \"cpu\")\n",
    "print(\"mps\" if torch.backends.mps.is_available() else \"cpu\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Loading the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When I made the decision to drastically narrow down the dataset on only the face-like emojis, I chose the first 95 of them and copied them into a new folder that I named \"Apple_Faces\".\n",
    "\n",
    "Here we also initialized the image size and the number of classes (16 classes --> 4 bit colours)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image size: torch.Size([3, 32, 32])\n",
      "Label: grinning face\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/db/b4g1bwvj4qd9pwhjmkfm9pxh0000gn/T/ipykernel_36961/1414903246.py:59: DeprecationWarning: __array__ implementation doesn't accept a copy keyword, so passing copy=False failed. __array__ must implement 'dtype' and 'copy' keyword arguments.\n",
      "  image_np = np.array(image)\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from PIL import Image\n",
    "from torch.utils.data import Dataset\n",
    "from torchvision import transforms\n",
    "\n",
    "image_size = 32\n",
    "num_classes = 16\n",
    "\n",
    "class EmojiDataset(Dataset):\n",
    "    def __init__(self, csv_file, img_dir, transform=None):\n",
    "\n",
    "        self.emoji_labels = pd.read_csv(csv_file)\n",
    "        self.img_dir = img_dir\n",
    "        self.transform = transform\n",
    "\n",
    "    def __len__(self):\n",
    "        return 95 # this was added because I only chose the first 95 emojis from the dataset\n",
    "        return len(self.emoji_labels)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        img_name = os.path.join(self.img_dir, str(self.emoji_labels.iloc[idx, 0]) + '.png')\n",
    "\n",
    "        # Open the image with RGBA mode\n",
    "        image = Image.open(img_name).convert('RGBA')\n",
    "\n",
    "        # Create a black background image of the same size\n",
    "        black_background = Image.new(\"RGBA\", image.size, (0, 0, 0, 255))\n",
    "\n",
    "        # Composite the original image on top of the black background\n",
    "        composite_image = Image.alpha_composite(black_background, image)\n",
    "\n",
    "        # Convert back to RGB, discarding the alpha channel\n",
    "        image = composite_image.convert('RGB')\n",
    "\n",
    "        label = self.emoji_labels.iloc[idx, 3]  # Assuming fourth column is the label\n",
    "\n",
    "        if self.transform:\n",
    "            image = self.transform(image)\n",
    "\n",
    "        return image, label\n",
    "\n",
    "# Path to the folder containing the Apple emoji images\n",
    "img_dir = './Dataset/image/Apple_Faces'\n",
    "\n",
    "# Path to the CSV file with emoji labels\n",
    "csv_file = './Dataset/full_emoji.csv'\n",
    "\n",
    "\n",
    "def to_int_tensor(image):\n",
    "    # Convert PIL image to NumPy array and ensure it has integer values\n",
    "    image_np = np.array(image)\n",
    "    \n",
    "    # Convert NumPy array to a PyTorch tensor and ensure it's an integer type\n",
    "    return torch.tensor(image_np, dtype=torch.int64).permute(2, 0, 1)\n",
    "\n",
    "def to_scaled_int_tensor(image, num_classes):\n",
    "    # Convert PIL image to NumPy array and ensure it has integer values\n",
    "    image_np = np.array(image)\n",
    "\n",
    "    # Scale the pixel values from [0, 255] to [0, num_classes]\n",
    "    image_np_scaled = (image_np / 255.0) * num_classes\n",
    "    image_np_scaled = np.round(image_np_scaled).astype(np.int64)  # Ensure integer values\n",
    "\n",
    "    # Convert NumPy array to a PyTorch tensor\n",
    "    return torch.tensor(image_np_scaled, dtype=torch.int64)\n",
    "\n",
    "# Define any image transformations (e.g., resizing, normalization)\n",
    "transform = transforms.Compose([\n",
    "    transforms.Resize((image_size, image_size)),  # Resize images if necessary\n",
    "    transforms.Lambda(to_int_tensor),        # Convert PIL images to tensors\n",
    "    transforms.Lambda(lambda img: to_scaled_int_tensor(img, num_classes)),  # Convert and scale to discrete values\n",
    "])\n",
    "\n",
    "# Create the dataset\n",
    "emoji_dataset = EmojiDataset(csv_file=csv_file, img_dir=img_dir, transform=transform)\n",
    "\n",
    "# Example of how to load an image and label\n",
    "image, label = emoji_dataset[0]\n",
    "print(f\"Image size: {image.size()}\")\n",
    "print(f\"Label: {label}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a function that visualises an image from the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGFCAYAAAASI+9IAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAMtUlEQVR4nO3c3XEcV5aF0csJvTM9YHrQZUKZABNoAk2QCTSB7QFMKHlQ8qDgwYUFmAf2bEVMTE/fTTGFgrDW88mjFP6+qAeeDy8vLy8DAMYY//XaLwDA/RAFAEIUAAhRACBEAYAQBQBCFAAIUQAgflkd/PDhw5HvAcDBVv6tsk8KAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQCzfPuL9+ljOb8UD+95t37fiPYrZHzHnMbNjjHGdz4ftfl5fzTvkkwIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEMxd/E5/KWxTn0/oDD+du92lfn92auxVjjK25XXFHdy5meYti3tZnr8XsGGM8XtZnL9fuJsaTExpvnk8KAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQHx4eXl5WRr88OHod/nb+1jcJ3oobhONMcaXh+5d9tO2PLvtp2751sy3u7dmuNt9qFs33hw/mtdy9WV59nad1e6vj+uzj+VdpWd3lf60lT/3PikAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAOHPxJ3zqLlGMXx/WH3h46HZvp3P5QPEf2PZud3Veopl9y+Zrv8AfmrMY87FcfVmefVwfHWOM8fWxu3Px+1O3/z1w5gKAiigAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAuH30v/zj0/rs18/d8aPzeVsf3h+q3WM/d/Nv9ubQ9tov8BeZr/0C/zLL8ev67PVbtfpymdX8l2/rt5Ley50kt48AqIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIA8bc/c/Gpu0Qxvn1Znz2fi5sYY4xxelif3U7d7tp28P6jbK/9Aj9ovvYL/KB54OpbN399rMYvl/X9n7+un8QYY4ynbvxuOHMBQEUUAAhRACBEAYAQBQBCFAAIUQAgRAGAEAUAQhQACFEAIN7k7aOPxT2jrw/d8aPPD8Vwc8tojO6e0datPtb22i/ATzVf+wW+m+38rZsvbiV9e5zV6i+P68ePnu/oTpLbRwBURAGAEAUAQhQACFEAIEQBgBAFAEIUAAhRACBEAYD45bVf4Ec8nIrZc7l8L5ZvW7l8HjJ6uG22DxzwEvx789Dxu7GV88Xv8sP5Uq2+3NZn//lbtfrV+aQAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAxF3cPvr0sZv/fF5/YGvvE237+uzsVvc3hO7EPPKBrV3+TszXfoH7M8v54nd/2/dq9ZfzbXn2cn2udj914z+dTwoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBA3MXto/Opmz/txXB7+6g6sNLMjjHmtj5bjP6IOef67G19dowxtn1bn62/P0faDtw9u+nia958L8cYYyt+gbZqc2keubz8D5Q/h82ppPbv2z9/6+Z/Np8UAAhRACBEAYAQBQBCFAAIUQAgRAGAEAUAQhQACFEAIA47c/GxmH3Ym+nyX6S3ZxSakwFHntAoRn/E5dt1efZ6eap276dPy7MPX07V7mOvYszDNt/KUyGPX6/Ls3M+V7vPD+vvcj7v1e67Up7/aDQ/hw97t/vxuj773H3rl/ikAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQIgCACEKAMRht4+24pzRvrcHPIrl7f2T5qhJvbsbb7Sv0tzi2fdu97yt30qat63avZ26+WNty5O36+xWF/eM9vXXGGOMcbve1l/j3C3vpkvzyN3t8m15sv37tr55jANOH/mkAMAfRAGAEAUAQhQACFEAIEQBgBAFAEIUAAhRACBEAYA47MxF80/vm8sS/3qifaAwD9x9P6qv+Wx3F2dI3rS5PNn+jPe/E8XuanYe9BbvR/u9bP52Ph1w58InBQBCFAAIUQAgRAGAEAUAQhQACFEAIEQBgBAFAEIUAAhRACDu4/ZRvX3WT6yv3tZni9Hvu8v5Qntf5XRef+B27XZvxTd/37vdY87ygfuwn7r5efu0Plt+TU7nvXuZxjxudW0eNlzZRncLbN/WDxr99tS+zX/mkwIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgDEYWcumhMQs/xn4Ae9xts2u/HTaVue3YvZMd7R17ywlfPnh/UnZrm9mp7V6ndjHnkWYzts9RKfFAAIUQAgRAGAEAUAQhQACFEAIEQBgBAFAEIUAAhRACBEAYA47vbRkeaBu7didh64+2hzffS1b7Hw/9vaB+bPf4e/xHybuw9cfQifFAAIUQAgRAGAEAUAQhQACFEAIEQBgBAFAEIUAAhRACBEAYA47vbRPGh2jLu5ITTnLJ/Y1ifXR483y/ntgHfg35uv/QI/pv71KR/YjvwlmgeuPnD3Cp8UAAhRACBEAYAQBQBCFAAIUQAgRAGAEAUAQhQACFEAIA47c3Gb67NzPFe7t/GxmC5eZHT/xPzbY7d7a2b3avXYm+Xl/vpcwGxepFtdjt+NefgDxeril7P5Pf6+e3223b1t3QOfz810t7vaPLu/b+3X5WfzSQGAEAUAQhQACFEAIEQBgBAFAEIUAAhRACBEAYAQBQBCFACI+7h9VMx+f+CQ0e+2YrQ8ULSfvyzPzvKLcm2OzowxxmV9fpZfxa2a73a/H9vy5KwvQu3rb1H+jG+n9flu8xjj9rV9Ytk88IH275vbRwDcDVEAIEQBgBAFAEIUAAhRACBEAYAQBQBCFAAIUQAgDjtzMZ/XZ6+3bve+r89u3erqRMPWLi/sp3M1vx35Mu0RgGK8PaHxZq9ibO148UC5uzuhUSpuOtwuj9XqfavGO/O48dut3F387TyCTwoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAHHb7qDnfcbl1u8/FcZBtfKx2z2L2tFWrx+Pl2/JsfftoP3Xz217MbuW7rM93m/m/zOLe0Pf52/psebinmZ+3x2p3+SPefV2K0e+71/8GtX/fXvn0kU8KAPxBFAAIUQAgRAGAEAUAQhQACFEAIEQBgBAFAEIUAIjDzlw0rrfj5otrDt/nt2Z2Vrv37bo8e7usz363ddPV/+h+P7ur6WPNariark5RHLq7vf9QOJ/KB7Zyfh4yOsYYo7n+0Z65eG0+KQAQogBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgBxF7ePnp67+cfr+uxpK5fPj8ujzYmfMcY47+uzl9ntnvOpfKCYn793q7s3uZvd78VWTa//Powxxr4Xs92L1N/85iTUvHV/J5q/Qe3ft9fmkwIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgDEXZy5aF1ux8yOMca5OovRnQBozmKcTtXqcb127zLnG/u39/xE6z8rzdmKMcY4lfON5mzF9/n1n/H278RjOf+W+KQAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAxIeXl5eXpcEPH45+l0N86k4CjV8f1mfPe7d7bOsv09xJ+hHX2/rsrZj9zl2lv1TxczVGd59o36rVlfaW0SjvdTX3jH59rFaPpzf6I77y594nBQBCFAAIUQAgRAGAEAUAQhQACFEAIEQBgBAFAEIUAAhRACD+9rePWv8ozsh8eeh2NzdntvKezZG3kmY539xKau/flON3Yyvn973YXX7zt+arWIy247O8ZdTc6xpjjK+P67O/v9FbRi23jwCoiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgDhzMWf8Km7RDG+nNdnz3u3uz2L0dxdOPKERmtWByNmub3Z3W6eh+1uVadFmtnRna643LrdXy/d/NM7OV3RcOYCgIooABCiAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQLh99Bf6WJwnam8ffT5183uxfxvH3VVqHbi6Nl/7Bf7HbMfXjwLdbt3ux2sxW+5+dsvoT3P7CICKKAAQogBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABDOXPxNfCovUZz29dn25Eaze+tW39edi8Y8bvx663Y385dy95NTFHfNmQsAKqIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIA4fYR/1F5VmlsxQP71u1u5+/FbR43P8t7Q84TvV9uHwFQEQUAQhQACFEAIEQBgBAFAEIUAAhRACBEAYAQBQDCmQuAd8KZCwAqogBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgDxy+rg4okkAN4wnxQACFEAIEQBgBAFAEIUAAhRACBEAYAQBQBCFACI/wbuGWGlDISv4QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def show_tensor_image(image_tensor):\n",
    "    image_tensor = image_tensor.permute(1, 2, 0)\n",
    "\n",
    "    # Convert the tensor to a numpy array (bring channels to last dimension)\n",
    "    image_np = image_tensor.numpy()\n",
    "    \n",
    "    # Clip the values to be between 0 and 1 for proper visualization\n",
    "    image_np = image_np / num_classes\n",
    "    \n",
    "    plt.imshow(image_np)\n",
    "    plt.axis('off')\n",
    "    plt.show()\n",
    "\n",
    "# print(image)\n",
    "show_tensor_image(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Defining the Auto-Regression Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To prevent the model from \"cheating\" (looking at pixels that were not generated yet), we use Masked Convolutions. \n",
    "\n",
    "These Masked Convolutions follow a simple rule - Any pixel to the right and the bottom can not influence the value of the result, therefore, they have to be masked and ignored. That is because after the training phase, our model will generate the image from top to bottom, left to right, row by row.\n",
    "\n",
    "There are two types of masked convolutions we will use:\n",
    "* A mask --> the mask zeros out the center pixel, this is used only in the first layer to prevent the cheating\n",
    "* B mask --> the mask doesn't zero out the center pixel, this is used in all subsequent layers since the information about the center pixel has already been lost in the first layer.\n",
    "\n",
    "This implementation has the initial convolutional layer, three hidden convolutional layers and one more output layer.\n",
    "\n",
    "All images have three color channels (RGB), so that is something I had to account for, especially in the last layer and it's dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 3, 32, 32])\n",
      "torch.Size([1, 3, 32, 32, 16])\n"
     ]
    }
   ],
   "source": [
    "# Define the Masked Convolution Layer\n",
    "class MaskedConv2d(nn.Conv2d):\n",
    "    def __init__(self, in_channels, out_channels, kernel_size, mask_type, **kwargs):\n",
    "        super(MaskedConv2d, self).__init__(in_channels, out_channels, kernel_size, **kwargs)\n",
    "        # Create the mask\n",
    "        self.register_buffer('mask', torch.ones(self.weight.size()))\n",
    "        \n",
    "        # Define the center pixel of the kernel\n",
    "        _, _, height, width = self.weight.size()\n",
    "        yc, xc = height // 2, width // 2\n",
    "\n",
    "        if mask_type == 'A':  # Type A mask (used in the first layer)\n",
    "            self.mask[:, :, yc, xc:] = 0  # Zero out the pixels to the right\n",
    "            self.mask[:, :, yc+1:, :] = 0  # Zero out the pixels below\n",
    "        elif mask_type == 'B':  # Type B mask (used in deeper layers)\n",
    "            self.mask[:, :, yc, xc+1:] = 0  # Zero out the pixels to the right\n",
    "            self.mask[:, :, yc+1:, :] = 0  # Zero out the pixels below\n",
    "\n",
    "    def forward(self, x):\n",
    "        # Apply the mask to the weights\n",
    "        self.weight.data *= self.mask\n",
    "        return super(MaskedConv2d, self).forward(x)\n",
    "    \n",
    "# Define the PixelCNN model\n",
    "class PixelCNN(nn.Module):\n",
    "    def __init__(self, input_dim=3, num_classes=256, hidden_dim=64):\n",
    "        super(PixelCNN, self).__init__()\n",
    "        \n",
    "        # The first layer uses the Type A mask (to prevent cheating on the pixel itself)\n",
    "        self.conv1 = MaskedConv2d(input_dim, hidden_dim, kernel_size=3, mask_type='A', padding=1)\n",
    "        \n",
    "        # Intermediate layers use Type B mask (since pixel can see itself)\n",
    "        self.conv2 = MaskedConv2d(hidden_dim, hidden_dim, kernel_size=7, mask_type='B', padding=3)\n",
    "        self.conv3 = MaskedConv2d(hidden_dim, hidden_dim, kernel_size=9, mask_type='B', padding=4)\n",
    "        self.conv4 = MaskedConv2d(hidden_dim, hidden_dim, kernel_size=7, mask_type='B', padding=3)\n",
    "        \n",
    "        # Output layer (predicts the class of each pixel for each RGB channel)\n",
    "        self.conv_out = nn.Conv2d(hidden_dim, num_classes * 3, kernel_size=1)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        # [ 0, num_classes ] => [ -1, 1 ]\n",
    "        x = x / ((num_classes-1) * 1.0)\n",
    "\n",
    "        # Pass through masked convolutions\n",
    "        x = F.relu(self.conv1(x))\n",
    "        x = F.relu(self.conv2(x))\n",
    "        x = F.relu(self.conv3(x))\n",
    "        x = F.relu(self.conv4(x))\n",
    "        \n",
    "        # Output layer (no activation here as it will be followed by softmax during loss calculation)\n",
    "        x = self.conv_out(x)\n",
    "\n",
    "        # Reshape the output into three parts, one for each color channel (R, G, B)\n",
    "        r = x[:, 0:num_classes, :, :]   # Red channel\n",
    "        g = x[:, num_classes:2*num_classes, :, :] # Green channel\n",
    "        b = x[:, 2*num_classes:3*num_classes, :, :] # Blue channel\n",
    "\n",
    "        # Stack them back along the second dimension (RGB channels)\n",
    "        x = torch.stack([r, g, b], dim=1)\n",
    "\n",
    "        return x.transpose(2, 3).transpose(3, 4)\n",
    "\n",
    "\n",
    "# Define the model\n",
    "input_dim = 3  # RGB channels\n",
    "# num_classes = 16  # Grayscale pixel values (0 to 255)\n",
    "model = PixelCNN(input_dim=input_dim, num_classes=num_classes).to(device)\n",
    "\n",
    "# Example input: Batch of grayscale images (batch_size=1, height=16, width=16)\n",
    "batch_size = 1\n",
    "dummy_input = torch.randint(0, num_classes, (batch_size, input_dim, image_size, image_size), dtype=torch.float32) / 255.0\n",
    "print(dummy_input.shape)\n",
    "\n",
    "# Forward pass\n",
    "output = model(dummy_input.to(device))\n",
    "\n",
    "# Output will have shape (batch_size, num_classes, height, width), \n",
    "# where num_classes = 256 (for pixel value classification in grayscale)\n",
    "print(output.shape)  # Example output: torch.Size([1, 3, 256, 16, 16])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sampling images\n",
    "\n",
    "Here I wrote a helper function that will randomly choose pixel values using the given probability distributions.\n",
    "\n",
    "* sample_one_pixel - chooses an element from a given distributions;\n",
    "* sample_from_pixelcc - performs sample_one_pixel for each pixel in each color channel of the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 3, 32, 32])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGFCAYAAAASI+9IAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAWvUlEQVR4nO3cwdEQN/fl4cuU94gIEBEgIkBEgBwB7QgsR/A1ESAioImAdgRuIqCJABEBcgTvLL7635rde89qpqZ+z/r4lAwvnOoF98nDw8ODAQBgZv/r//YDAAD/72AUAACOUQAAOEYBAOAYBQCAYxQAAI5RAAA4RgEA4H6LBtuxS8WrjXj2WZK69396OJtW/B1mZnnGu1/cp9T9T1nxd2yb1H0dScr33MLZtMezZmb1imfnuUvda8azI0nV1nbtP0jtDmd/3n9I3S/zn+Hs9xJ/h5nZ0/k1nN2vt1L3Jvxcldql7sO0fF7x7J0OqbvfKZxNd5W6zeLd026pec3z0QxfCgAAxygAAByjAABwjAIAwDEKAADHKAAAHKMAAHCMAgDAMQoAAMcoAAAcowAAcOHbR5dw08TMrFxXOHs9jd9iMTPbR41n7/g7zMxySuHs01N793bHs+czqdr6P1PK/ztaOJuWVG13P8LZeg6p++PRwtk3pUvdf7ZDys+thrNX/iF1L+H/M9dN7E7h7NyW1H2nIjzklrrTmFJ+E24rzWtob0lnOFuzVG33Hu/ue9HKA/hSAAA4RgEA4BgFAIBjFAAAjlEAADhGAQDgGAUAgGMUAACOUQAAOEYBAODiZy7eLKm4/irh7G1V6j6Ffwbe2iZ2x7P79knqbmcKZ9fTU+q+xF/DzUo4e1btLYeNcPa6te4Pdwln64tL6s5fupbvLZydtqTu/U7h7Ei31D2EMyR5K1L3sadwdu+31D2vTcrff+/hbH99SN3n3sPZj/WN1P1tfxvOzp6k7gi+FAAAjlEAADhGAQDgGAUAgGMUAACOUQAAOEYBAOAYBQCAYxQAAI5RAAA4RgEA4J48PDw8RIL17FJxv3M4m4Q7PGZmfWzhbEta99hGOLv1KXX3K/6WazSpu6Uu5Ws5wtlya91pxbPt1u7C2PEyHL2OJlVv9l7K1/1HOFuuLnXnVMPZ7TylbjtyODpmk6o/53j3a+Fn0MzsTFPK176Fs+k6pe6v/QpnX1uSuue+wtmShtR9Bv5M8KUAAHCMAgDAMQoAAMcoAAAcowAAcIwCAMAxCgAAxygAAByjAABwjAIAwIXPXDxZm1T8ul3h7Bin1H3cQj7H32Fmdh8rnC2rSt313sLZXJvUfe6XlN+OO5xt7Xepu51C+JCqzdI/QneXqo+jSvlVP4aztcTPc5iZJeGkg/rnZ7cczm6H1i1c57DyeZO673+eSfnZvoWzKb+SukeO/37mXaq2Na5wdqZb6j5HfTTDlwIAwDEKAADHKAAAHKMAAHCMAgDAMQoAAMcoAAAcowAAcIwCAMAxCgAAxygAANxv0eDLa0jF51zh7Jq31G1nPDq1ZiulhbOrdak7tRnvrpvUnVuT8mcr8fD4U+o+rvhNoJ8/X0vdn9abcDaZcCfJzNKWpfzVUji7r1PqHmc8f68udd+rhLNnPGpmZpdy/Oh1k7pX+SXlWynhbN+021Sf7Xs83LV3W1rh6De7tG6rjyb4UgAAOEYBAOAYBQCAYxQAAI5RAAA4RgEA4BgFAIBjFAAAjlEAADhGAQDgwmcuRi5ac07haCnCPxk3s22P/7Pxuh1Sdzneh7P3oXXv+Qxnt5Kl7iyeOjjfH+Hs9W6Tuo/9Wzhb5i51W38ejt5bk6qvObSnlDOc/Xx0qduOeHfpU6u2Hs528ZTLseLZMqVqm/OZlD/Tl3B2jCJ1f9pqOLu+xrNmZpdwx+fV8yx1PwQyfCkAAByjAABwjAIAwDEKAADHKAAAHKMAAHCMAgDAMQoAAMcoAAAcowAAcIwCAMA9eXh4iJzDsDGaVLzsCGf7PKXuMks4O+5b6j7WEc5edzxrZpbsCmf3XqXua+xSvo4Uf8uUqm3NI5y9ZpW66xXP/rua1P1tz1L+VbvD2R/bLnW/aJuQLlL3y5WF5iV1f04pnP02pWpbLUv5fO/h7KYcbTIzq/F8XrdUfZYRzrbzkLqPdD2a4UsBAOAYBQCAYxQAAI5RAAA4RgEA4BgFAIBjFAAAjlEAADhGAQDgGAUAgAufuVi5S8X7OcPZkovUfbUUzp5fs9Rtr7dwNKVTqi5Ctt9StS3h5IKZmeU9nlVuS5iZzXj3uXep+hbORfR9k7r3cUv5UzgBcbcudac8hHfEs2ZmY7Zw9j6lartTvPsSz5D8XV5I+Q+v4tnyWqq2nF/Gu+spdY99D2dr/yx15/74X/d8KQAAHKMAAHCMAgDAMQoAAMcoAAAcowAAcIwCAMAxCgAAxygAAByjAABwjAIAwIVvH+X9iVTca/yYSN2H1L2lHn/HPKXu8r2Gs+lblrqTcP1oS5vUba1J8Srcy7mPS+oedoazV9K6U1nxdxxJ6t7uKeWn8Ja/5pC6f6xLSDep23oJR6/tkKpbXeFsFv9sjjq0t2w9nD1zkrrtmEL0lqrb7/Hs+HVI3TM9/ha+FAAAjlEAADhGAQDgGAUAgGMUAACOUQAAOEYBAOAYBQCAYxQAAI5RAAA4RgEA4MK3j84ypOJpKx5u76Xuq38IZ9MuVVvvf4Wzoz+Xukcp4ezRu9R97VLc2or330Urv+sMZ7f7lrqHcOPpXMIRGTO79l9S/o/rDGc/ifeJ5h7vLj1L3b/PLZx9J91gMrvSCmeTeMvou3BTy8zsQ27h7BR+Zs3MUq3h7LbF32FmVno8e/b4O8zM6lqPZvhSAAA4RgEA4BgFAIBjFAAAjlEAADhGAQDgGAUAgGMUAACOUQAAOEYBAODCZy7W2aXiWzhz0eyWuv9dJZx9l6bUvcoRD2+31D2FUyE5S9W21y7lu3Beoh5V6h6nEN7j7zAzu/sKZ1/UTer+kqqUt3SFo2Vp1bb3cPSsWnWqJZyt2yF1b7OFs2V2qfvjpb3l9bzD2VGa1H0Lf5bv/F3qLulHONvEX5MUuIfDlwIAwDEKAADHKAAAHKMAAHCMAgDAMQoAAMcoAAAcowAAcIwCAMAxCgAAxygAAFz49lF5sqTi8jaerylJ3bXMeDafUvc1ejh7p03q3rYczv6bp9T9o2r5a1fyTepuecWzvUvdWbhNtZbW3fYq5eddwtnjbFJ3Fm52Ndukbmt7OHr8MbTqP1c8nF9J3bn+I+WvmePd2yV1t/sMZ8cef4eZ2S3cbNrE21Q98HcQXwoAAMcoAAAcowAAcIwCAMAxCgAAxygAAByjAABwjAIAwDEKAADHKAAA3G/RYHtbpOKy/wxn+/FF6t7LCmfbdUnds4xwtl9J6v533OHsh6+H1H19eiPl/7DX4eyvMaTuUZ+Fs1+PJHV/veP5H9sude910/KthLNr/SF1b/ZnONvzJXWXVsPZOYvUXXM8+6Zpf+7tvqX4j+uIV6cidY8ef0trSeo+9zOcLemSus3yowm+FAAAjlEAADhGAQDgGAUAgGMUAACOUQAAOEYBAOAYBQCAYxQAAI5RAAA4RgEA4MK3j86ya80pHj0D9zj+T6/6Gc7+ykXqrn+3cHa81rrLHc+Xd1nqtvulFP+U9nB2HE3qfv/qn3D21zal7n7Hbwi9OJ5L3a/FO1ntWQ1nt7fxW1NmZmNlIZukbhPuGS3lmJGZlW2Fs5/yLXWnW3vLec5w9rBd6u5X/C1zDak77yucvdYhdVdrj2b4UgAAOEYBAOAYBQCAYxQAAI5RAAA4RgEA4BgFAIBjFAAAjlEAADhGAQDgnjw8PDxEgr1dUrHwL+mtHVK1jesMZ7NtUvffdQ9n36UhdX8uPZx9u1Wpe7y5pPyLH/G3fHqRpe7zV/wteSapewlnS9a4pe5yafn69Yx3axc3zIQ/b3s/pOpbOEUx03up+2d+F87+c8XfYWaWcpbyOcWzfUjVZqOHo9uYUvWblIR3LKn7odVHM3wpAAAcowAAcIwCAMAxCgAAxygAAByjAABwjAIAwDEKAADHKAAAHKMAAHCMAgDA/RYNTvHOzy7c7zhsSt2llXD2c4u/w8zsab7D2S6++/OVw9lZq9Sdviwp/+HawtnzeZO6Z+C+yv+4+pS6481mS/j1VrvNzLYPPZxNf+1SdxX+vN1Nu0/Utpfh7HF+krqv4w5nx5xSt5VLitf7ezi7rz+l7hdrhbN5HFL3n72Es+13rdse7kcjfCkAAByjAABwjAIAwDEKAADHKAAAHKMAAHCMAgDAMQoAAMcoAAAcowAAcIwCAMCFbx+1OaTiV0L2P9eUutvdw9l8L617i2dfjV3q/rCf4ey5hIeY2WxZyh/P4v35U5K6s3L/ZmSpu4wVztajat13k/K77eHsen1I3U34Ndzsi9R97lc4O/OUul+UFc5+KEPqvsSflV7jb+lzl7qfpnj2WkLYzPK/LZw9P9xSdw1k+FIAADhGAQDgGAUAgGMUAACOUQAAOEYBAOAYBQCAYxQAAI5RAAA4RgEA4J48PDw8RIJn61LxuFY4+3VPUvc34Z/S53xK3fe2hbOHXVr31xTOri+71H2VKeV7G+HsfcWzZmajtHB2K0XqVq6tnNuSupedWl44X3DMKXVf2whnk3A+xcwsr3i+nZfUvYS/J5pNqXt/ob2lfrvD2XklqTuNFs72Fc+ama2ew9mzH1L3kcqjGb4UAACOUQAAOEYBAOAYBQCAYxQAAI5RAAA4RgEA4BgFAIBjFAAAjlEAADhGAQDgfosG89Wl4tn2cPZbPaXullI4u4m3W+4a7x5d6z7H13B2zl3qtpml+NZmPDyW1p32cLY2qdrWmuHsfcbfYWb2sZ9S/luv4Wwdl9Q9Azdq/kdfu9T9b7za2r5J3Ved4Wwfr6Tu/vQ/Un4Tfs2HeNttpRbOpnJK3W/qm3D27cpSt6XHI3wpAAAcowAAcIwCAMAxCgAAxygAAByjAABwjAIAwDEKAADHKAAAHKMAAHBPHh4eHiLBdDapeI4Rzq7Ste56hLNvxPMc9rGFo5++XFL1/L2Es9frXepupUr5Xfj9rNuSums5wtl5StW2Rglnu/hzdfUl5bcRz/ZjSt1mZzg51V/E2cPR432RquvrEc7mJVVbWvEzMWZm9eencPZ627XuXMLZUzj7YmZ29hQP703qTmM+muFLAQDgGAUAgGMUAACOUQAAOEYBAOAYBQCAYxQAAI5RAAA4RgEA4BgFAIBjFAAALnz7KO+X1nwmITq16vuOh+spdb+/Wzj7rk6p+9pyONtbPGtmtvZTytcUz25/CGEz27/E83MsqbsJN7VGm1L3aa+kvPUf8e71Qqq+y9twtt1d6j7sDmdHnlJ3mTWc/driWTOzb1uR8vOI57fxt9R925dwtotHnvLWw9lVNqn7WOPRDF8KAADHKAAAHKMAAHCMAgDAMQoAAMcoAAAcowAAcIwCAMAxCgAAxygAAFz4zEUbTSruM56/i1Rtf9Uezn6wKXWfL+Ld9dMpdRfhZEBKReoehxS3q8f/g33fpe65SjjbhX/Sb2a23ymcvW1J3XVMKd97CWe3MqTus17hbBF/f27hxsmz/Za6P9Uazq4Vf4eZ2bUOKZ9W/C1dfMtxjnC2Hk3qrmsPZ/ccf4eZ2Qj8zPKlAABwjAIAwDEKAADHKAAAHKMAAHCMAgDAMQoAAMcoAAAcowAAcIwCAMAxCgAA91s4uWvFfeRwdlvxrJnZr9XC2Wd7lbpffou/pZ6b1P1GuFPyvO9S9xJu5fz3Pyjh6CncsTIzm8L9m49ll7rfCb+fOWep+0i3lN9GPD+EWzlmZues4WwS3z2uPZx9PbPUnYTu43uRus+X2luU/8/17JC6e1nh7Lw2qbu+ErJfTqnbrDya4EsBAOAYBQCAYxQAAI5RAAA4RgEA4BgFAIBjFAAAjlEAADhGAQDgGAUAgAufuVh9k4pLucPZ9GpK3deXEs5+2C6te4+/5U1uUvfz3MPZegypewonF8zMthXPj3JI3cmucPZMQ+p+tZdw9t2cUnf6+5Dy9jaHo/vepOqcajibUpK6k3D+Y7Vd6h5XD2fbp0Pqbqf2llP4Ge9Ss9nn7XM4+7r/I3Ufz+PZtSep29rjEb4UAACOUQAAOEYBAOAYBQCAYxQAAI5RAAA4RgEA4BgFAIBjFAAAjlEAADhGAQDgwrePyvtTKl71Cmfvd1r3x3uEsz+2Teq+cjz7ciap+6xHOLvtwkPMbCoPN7OR93C2blr3MWs4e9db6n6Z4/kyitRdS5byNs54VLzB9WLFs9/uV1L3Wf4TD88udVvL4egSf+9TXtpThP71VHvL0/I2nD22InUP4R5YG/Hsf61HE3wpAAAcowAAcIwCAMAxCgAAxygAAByjAABwjAIAwDEKAADHKAAAHKMAAHCMAgDAhW8frS+XVJyE+x25Janbznj+aqdUfXyf8fCHQ+ruawtn12xS988ixe0s8f5lf0ndqca7r/2Wur+fPZydc0jdeT+l/J3jb7n6LXX/qiWcHRa/w2NmdtUZzubtkLqPvoWzs8Xf8d980t5yPYtnt3dSd01bPCz8uTcz23sJZ4/9kLprIMOXAgDAMQoAAMcoAAAcowAAcIwCAMAxCgAAxygAAByjAABwjAIAwDEKAAAXPnNxKf+s28x+bkL++yF1/xJObtRzSd3p6RnOjr92qfv85wpnV65S976KlL+Ff3r/Rx5S96+9hbPPzip1fytbOJvPKXU/u4eUfyn8f66ape6trXD2vv6WutcRP4vR/42/w8xs2hbOVvFn9rQk5bd0h7PXkqptH/HsdYu/P+cVztY3Seq2h/ZohC8FAIBjFAAAjlEAADhGAQDgGAUAgGMUAACOUQAAOEYBAOAYBQCAYxQAAI5RAAC4Jw8PDw+RYE1NKr5TCWf7kaXumVb8HeOQur+fKZx9eg6pu84tnC31u9Q91nMpP0cOZ2/h3WZmV7vC2e0vrXv/8CacbeOL1N3XLeW3Gs9mqdms5Cuc3dshttd4tBep+Z4pnL3uLHX3ekv5XFv8LZf2ljXib/lZL6nb9hyOvvw4per74Xw0w5cCAMAxCgAAxygAAByjAABwjAIAwDEKAADHKAAAHKMAAHCMAgDAMQoAAPdbNDi3XSo+zhXP5ix1jzXC2VLiWTOzH0L3PC+pu58lni271H1ZkvI15XC27JvU/Xn2cHY+L1L3Nv4Tzq4ype4l/qxM4XzBJpx/MDO76vtwtq9d6h4pnt/TIXX3coez56l1zzyl/FZzOHt93qTun8ppnrlL3R8+9nD2ejul7gi+FAAAjlEAADhGAQDgGAUAgGMUAACOUQAAOEYBAOAYBQCAYxQAAI5RAAA4RgEA4MK3j2wOqTilFs7+vXep29oRjo58S9XbtoWzeUjVdgg3as6rSd3zuKX8z97D2VIOqfv1sYWzaxSp+6xC931J3Xk1KT/3FM5uKZ41M9v2D+HsyrvWfdRw9tXXU+r+9GcJZ0eTqu08hpQfwv2w/DZL3fMo4ewxltSdfnwPZ/fxQ+qO4EsBAOAYBQCAYxQAAI5RAAA4RgEA4BgFAIBjFAAAjlEAADhGAQDgGAUAgHvy8PDwEAmWkqVi4V/SW9661F33Ox5ep9TdhNsVVxLeYWYlX+HsHX+GmZnln0vKp+c9nP1YhtT9warwkCx1r3WEs60sqft8f0r59GOGs2vTuqvt4ewunHMwM2u1h7N5Fam7CH/w09dd6ranhxTfrhXOJvFmzTW3cHat+DvMzO7jZzibPj+VutPD42/hSwEA4BgFAIBjFAAAjlEAADhGAQDgGAUAgGMUAACOUQAAOEYBAOAYBQCAYxQAAC58+wgA8P8/vhQAAI5RAAA4RgEA4BgFAIBjFAAAjlEAADhGAQDgGAUAgGMUAADufwOAx/Dn8a00GgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "def sample_one_pixel(output_distribution):\n",
    "    applied_softmax = F.softmax(output_distribution, dim=0)\n",
    "    dist = torch.distributions.categorical.Categorical(probs=applied_softmax)\n",
    "    sampled_pixel = dist.sample()\n",
    "    return sampled_pixel.item()\n",
    "\n",
    "def sample_from_pixelcnn(output):\n",
    "    sampled_image = []\n",
    "    # print(output.shape)\n",
    "    batch_size, num_channels, height, width, _ = output.shape\n",
    "\n",
    "    for b in range(batch_size):\n",
    "        image = torch.zeros((num_channels, height, width))\n",
    "        for c in range(num_channels):\n",
    "            for h in range(height):\n",
    "                for w in range(width):\n",
    "                    image[c][h][w] = sample_one_pixel(output[b][c][h][w])\n",
    "        sampled_image.append(image)\n",
    "\n",
    "    sampled_image = torch.stack(sampled_image)\n",
    "    return sampled_image\n",
    "\n",
    "# Sample from the model (using a random output)\n",
    "generated_image = sample_from_pixelcnn(output)\n",
    "print(generated_image.shape)\n",
    "show_tensor_image(generated_image[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training is extremely efficient due to the utilization of **Teacher Forcing**. Teacher forcing is a method where we do not feed the actual choices of our network as the next input, but we always use the ground truth as input, this parallelises the training process making it faster and also helps the model learn more efficiently in the early stages of training.\n",
    "\n",
    "I experimented with the hyperparameters of the network a little bit, but there is definitely room for improvement.\n",
    "\n",
    "As I have already mentiond, the training is extremely efficient. It only took around a minute to train on my laptop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/db/b4g1bwvj4qd9pwhjmkfm9pxh0000gn/T/ipykernel_36961/1234043459.py:65: DeprecationWarning: __array__ implementation doesn't accept a copy keyword, so passing copy=False failed. __array__ must implement 'dtype' and 'copy' keyword arguments.\n",
      "  image_np = np.array(image)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1/250], Step [5/12], Loss: 2.3662919998168945\n",
      "Epoch [1/250], Step [10/12], Loss: 2.1977052688598633\n",
      "Epoch [2/250], Step [5/12], Loss: 2.0824568271636963\n",
      "Epoch [2/250], Step [10/12], Loss: 1.8482893705368042\n",
      "Epoch [3/250], Step [5/12], Loss: 1.7759639024734497\n",
      "Epoch [3/250], Step [10/12], Loss: 1.6483122110366821\n",
      "Epoch [4/250], Step [5/12], Loss: 1.5862394571304321\n",
      "Epoch [4/250], Step [10/12], Loss: 1.429655909538269\n",
      "Epoch [5/250], Step [5/12], Loss: 1.6086554527282715\n",
      "Epoch [5/250], Step [10/12], Loss: 1.4813398122787476\n",
      "Epoch [6/250], Step [5/12], Loss: 1.3733493089675903\n",
      "Epoch [6/250], Step [10/12], Loss: 1.3632580041885376\n",
      "Epoch [7/250], Step [5/12], Loss: 1.3520432710647583\n",
      "Epoch [7/250], Step [10/12], Loss: 1.2656431198120117\n",
      "Epoch [8/250], Step [5/12], Loss: 1.1976510286331177\n",
      "Epoch [8/250], Step [10/12], Loss: 1.175576090812683\n",
      "Epoch [9/250], Step [5/12], Loss: 1.1968234777450562\n",
      "Epoch [9/250], Step [10/12], Loss: 1.2665716409683228\n",
      "Epoch [10/250], Step [5/12], Loss: 1.1709734201431274\n",
      "Epoch [10/250], Step [10/12], Loss: 1.2976951599121094\n",
      "Epoch [11/250], Step [5/12], Loss: 1.1811418533325195\n",
      "Epoch [11/250], Step [10/12], Loss: 1.134867787361145\n",
      "Epoch [12/250], Step [5/12], Loss: 1.1888494491577148\n",
      "Epoch [12/250], Step [10/12], Loss: 1.2230154275894165\n",
      "Epoch [13/250], Step [5/12], Loss: 1.0399807691574097\n",
      "Epoch [13/250], Step [10/12], Loss: 1.1826480627059937\n",
      "Epoch [14/250], Step [5/12], Loss: 1.0417815446853638\n",
      "Epoch [14/250], Step [10/12], Loss: 1.223756194114685\n",
      "Epoch [15/250], Step [5/12], Loss: 1.300699234008789\n",
      "Epoch [15/250], Step [10/12], Loss: 1.043725609779358\n",
      "Epoch [16/250], Step [5/12], Loss: 1.0309022665023804\n",
      "Epoch [16/250], Step [10/12], Loss: 1.2858588695526123\n",
      "Epoch [17/250], Step [5/12], Loss: 1.055746078491211\n",
      "Epoch [17/250], Step [10/12], Loss: 1.0121662616729736\n",
      "Epoch [18/250], Step [5/12], Loss: 1.1397954225540161\n",
      "Epoch [18/250], Step [10/12], Loss: 1.012919306755066\n",
      "Epoch [19/250], Step [5/12], Loss: 1.0208367109298706\n",
      "Epoch [19/250], Step [10/12], Loss: 1.2457585334777832\n",
      "Epoch [20/250], Step [5/12], Loss: 1.1563282012939453\n",
      "Epoch [20/250], Step [10/12], Loss: 1.0046933889389038\n",
      "Epoch [21/250], Step [5/12], Loss: 0.9488833546638489\n",
      "Epoch [21/250], Step [10/12], Loss: 0.9981830716133118\n",
      "Epoch [22/250], Step [5/12], Loss: 0.9473788142204285\n",
      "Epoch [22/250], Step [10/12], Loss: 0.9196782112121582\n",
      "Epoch [23/250], Step [5/12], Loss: 0.9044234156608582\n",
      "Epoch [23/250], Step [10/12], Loss: 0.9273665547370911\n",
      "Epoch [24/250], Step [5/12], Loss: 0.9288210868835449\n",
      "Epoch [24/250], Step [10/12], Loss: 0.8947162628173828\n",
      "Epoch [25/250], Step [5/12], Loss: 1.1003912687301636\n",
      "Epoch [25/250], Step [10/12], Loss: 0.8692889213562012\n",
      "Epoch [26/250], Step [5/12], Loss: 1.308242917060852\n",
      "Epoch [26/250], Step [10/12], Loss: 0.9154877662658691\n",
      "Epoch [27/250], Step [5/12], Loss: 0.9371142387390137\n",
      "Epoch [27/250], Step [10/12], Loss: 1.0083402395248413\n",
      "Epoch [28/250], Step [5/12], Loss: 0.789095938205719\n",
      "Epoch [28/250], Step [10/12], Loss: 0.931239664554596\n",
      "Epoch [29/250], Step [5/12], Loss: 0.8010236620903015\n",
      "Epoch [29/250], Step [10/12], Loss: 0.8836113810539246\n",
      "Epoch [30/250], Step [5/12], Loss: 0.7885653376579285\n",
      "Epoch [30/250], Step [10/12], Loss: 0.929297149181366\n",
      "Epoch [31/250], Step [5/12], Loss: 0.8230354189872742\n",
      "Epoch [31/250], Step [10/12], Loss: 0.9721824526786804\n",
      "Epoch [32/250], Step [5/12], Loss: 0.8984501957893372\n",
      "Epoch [32/250], Step [10/12], Loss: 0.9250404238700867\n",
      "Epoch [33/250], Step [5/12], Loss: 0.8929288387298584\n",
      "Epoch [33/250], Step [10/12], Loss: 0.9228451251983643\n",
      "Epoch [34/250], Step [5/12], Loss: 0.689702033996582\n",
      "Epoch [34/250], Step [10/12], Loss: 0.9521312713623047\n",
      "Epoch [35/250], Step [5/12], Loss: 0.9104768633842468\n",
      "Epoch [35/250], Step [10/12], Loss: 0.9796082377433777\n",
      "Epoch [36/250], Step [5/12], Loss: 0.7948476672172546\n",
      "Epoch [36/250], Step [10/12], Loss: 0.8205183148384094\n",
      "Epoch [37/250], Step [5/12], Loss: 0.9432908892631531\n",
      "Epoch [37/250], Step [10/12], Loss: 0.9368166923522949\n",
      "Epoch [38/250], Step [5/12], Loss: 0.7916944622993469\n",
      "Epoch [38/250], Step [10/12], Loss: 0.9157257080078125\n",
      "Epoch [39/250], Step [5/12], Loss: 0.7077506184577942\n",
      "Epoch [39/250], Step [10/12], Loss: 0.807100772857666\n",
      "Epoch [40/250], Step [5/12], Loss: 0.9305891990661621\n",
      "Epoch [40/250], Step [10/12], Loss: 0.8181187510490417\n",
      "Epoch [41/250], Step [5/12], Loss: 0.6094686388969421\n",
      "Epoch [41/250], Step [10/12], Loss: 0.9949837327003479\n",
      "Epoch [42/250], Step [5/12], Loss: 0.7574231624603271\n",
      "Epoch [42/250], Step [10/12], Loss: 0.846796452999115\n",
      "Epoch [43/250], Step [5/12], Loss: 0.7955639362335205\n",
      "Epoch [43/250], Step [10/12], Loss: 0.9355317950248718\n",
      "Epoch [44/250], Step [5/12], Loss: 0.8637145161628723\n",
      "Epoch [44/250], Step [10/12], Loss: 0.7425718903541565\n",
      "Epoch [45/250], Step [5/12], Loss: 0.9287071228027344\n",
      "Epoch [45/250], Step [10/12], Loss: 0.8247987627983093\n",
      "Epoch [46/250], Step [5/12], Loss: 0.7282002568244934\n",
      "Epoch [46/250], Step [10/12], Loss: 0.8470353484153748\n",
      "Epoch [47/250], Step [5/12], Loss: 0.7517433166503906\n",
      "Epoch [47/250], Step [10/12], Loss: 0.7961830496788025\n",
      "Epoch [48/250], Step [5/12], Loss: 0.6714428067207336\n",
      "Epoch [48/250], Step [10/12], Loss: 0.7932887673377991\n",
      "Epoch [49/250], Step [5/12], Loss: 0.7479178309440613\n",
      "Epoch [49/250], Step [10/12], Loss: 0.8889444470405579\n",
      "Epoch [50/250], Step [5/12], Loss: 0.6853248476982117\n",
      "Epoch [50/250], Step [10/12], Loss: 0.8558163046836853\n",
      "Epoch [51/250], Step [5/12], Loss: 0.9895113110542297\n",
      "Epoch [51/250], Step [10/12], Loss: 0.8501084446907043\n",
      "Epoch [52/250], Step [5/12], Loss: 0.9332640171051025\n",
      "Epoch [52/250], Step [10/12], Loss: 0.7358898520469666\n",
      "Epoch [53/250], Step [5/12], Loss: 0.6862943768501282\n",
      "Epoch [53/250], Step [10/12], Loss: 0.7070366740226746\n",
      "Epoch [54/250], Step [5/12], Loss: 0.684451162815094\n",
      "Epoch [54/250], Step [10/12], Loss: 0.6919271349906921\n",
      "Epoch [55/250], Step [5/12], Loss: 0.7745797634124756\n",
      "Epoch [55/250], Step [10/12], Loss: 1.0312048196792603\n",
      "Epoch [56/250], Step [5/12], Loss: 0.9251856803894043\n",
      "Epoch [56/250], Step [10/12], Loss: 0.7080877423286438\n",
      "Epoch [57/250], Step [5/12], Loss: 0.7813222408294678\n",
      "Epoch [57/250], Step [10/12], Loss: 0.6721928715705872\n",
      "Epoch [58/250], Step [5/12], Loss: 0.72477787733078\n",
      "Epoch [58/250], Step [10/12], Loss: 0.7807579636573792\n",
      "Epoch [59/250], Step [5/12], Loss: 0.7186132073402405\n",
      "Epoch [59/250], Step [10/12], Loss: 0.8396621346473694\n",
      "Epoch [60/250], Step [5/12], Loss: 0.7815963625907898\n",
      "Epoch [60/250], Step [10/12], Loss: 0.590423047542572\n",
      "Epoch [61/250], Step [5/12], Loss: 0.8206769824028015\n",
      "Epoch [61/250], Step [10/12], Loss: 0.6808817982673645\n",
      "Epoch [62/250], Step [5/12], Loss: 0.6732431054115295\n",
      "Epoch [62/250], Step [10/12], Loss: 0.8768976330757141\n",
      "Epoch [63/250], Step [5/12], Loss: 0.7517309188842773\n",
      "Epoch [63/250], Step [10/12], Loss: 0.7717606425285339\n",
      "Epoch [64/250], Step [5/12], Loss: 0.7805785536766052\n",
      "Epoch [64/250], Step [10/12], Loss: 0.6333797574043274\n",
      "Epoch [65/250], Step [5/12], Loss: 0.6896992325782776\n",
      "Epoch [65/250], Step [10/12], Loss: 0.6233115792274475\n",
      "Epoch [66/250], Step [5/12], Loss: 0.698830783367157\n",
      "Epoch [66/250], Step [10/12], Loss: 0.7621139883995056\n",
      "Epoch [67/250], Step [5/12], Loss: 0.8230531811714172\n",
      "Epoch [67/250], Step [10/12], Loss: 0.6353175044059753\n",
      "Epoch [68/250], Step [5/12], Loss: 0.6117867231369019\n",
      "Epoch [68/250], Step [10/12], Loss: 0.8070762753486633\n",
      "Epoch [69/250], Step [5/12], Loss: 0.6910956501960754\n",
      "Epoch [69/250], Step [10/12], Loss: 0.6576782464981079\n",
      "Epoch [70/250], Step [5/12], Loss: 0.7683966755867004\n",
      "Epoch [70/250], Step [10/12], Loss: 0.8580062985420227\n",
      "Epoch [71/250], Step [5/12], Loss: 0.7140696048736572\n",
      "Epoch [71/250], Step [10/12], Loss: 0.8058455586433411\n",
      "Epoch [72/250], Step [5/12], Loss: 0.8167967796325684\n",
      "Epoch [72/250], Step [10/12], Loss: 0.6022893190383911\n",
      "Epoch [73/250], Step [5/12], Loss: 0.734710693359375\n",
      "Epoch [73/250], Step [10/12], Loss: 0.7467791438102722\n",
      "Epoch [74/250], Step [5/12], Loss: 0.5561323761940002\n",
      "Epoch [74/250], Step [10/12], Loss: 0.6049438118934631\n",
      "Epoch [75/250], Step [5/12], Loss: 0.6397855877876282\n",
      "Epoch [75/250], Step [10/12], Loss: 0.6637974381446838\n",
      "Epoch [76/250], Step [5/12], Loss: 0.6573286652565002\n",
      "Epoch [76/250], Step [10/12], Loss: 0.5930176377296448\n",
      "Epoch [77/250], Step [5/12], Loss: 0.7149100303649902\n",
      "Epoch [77/250], Step [10/12], Loss: 0.6923632025718689\n",
      "Epoch [78/250], Step [5/12], Loss: 0.49788323044776917\n",
      "Epoch [78/250], Step [10/12], Loss: 0.629413366317749\n",
      "Epoch [79/250], Step [5/12], Loss: 0.6747123599052429\n",
      "Epoch [79/250], Step [10/12], Loss: 0.5930697917938232\n",
      "Epoch [80/250], Step [5/12], Loss: 0.596763551235199\n",
      "Epoch [80/250], Step [10/12], Loss: 0.5889268517494202\n",
      "Epoch [81/250], Step [5/12], Loss: 0.40135467052459717\n",
      "Epoch [81/250], Step [10/12], Loss: 0.7288634777069092\n",
      "Epoch [82/250], Step [5/12], Loss: 0.7171399593353271\n",
      "Epoch [82/250], Step [10/12], Loss: 0.8116133213043213\n",
      "Epoch [83/250], Step [5/12], Loss: 0.6515594124794006\n",
      "Epoch [83/250], Step [10/12], Loss: 0.763319194316864\n",
      "Epoch [84/250], Step [5/12], Loss: 0.6517018675804138\n",
      "Epoch [84/250], Step [10/12], Loss: 0.5705713033676147\n",
      "Epoch [85/250], Step [5/12], Loss: 0.7164950966835022\n",
      "Epoch [85/250], Step [10/12], Loss: 0.6837374567985535\n",
      "Epoch [86/250], Step [5/12], Loss: 0.5506877303123474\n",
      "Epoch [86/250], Step [10/12], Loss: 0.5204373002052307\n",
      "Epoch [87/250], Step [5/12], Loss: 0.6962397694587708\n",
      "Epoch [87/250], Step [10/12], Loss: 0.6395888924598694\n",
      "Epoch [88/250], Step [5/12], Loss: 0.5336721539497375\n",
      "Epoch [88/250], Step [10/12], Loss: 0.6602485179901123\n",
      "Epoch [89/250], Step [5/12], Loss: 0.7612788081169128\n",
      "Epoch [89/250], Step [10/12], Loss: 0.7034706473350525\n",
      "Epoch [90/250], Step [5/12], Loss: 0.6137232780456543\n",
      "Epoch [90/250], Step [10/12], Loss: 0.6016342043876648\n",
      "Epoch [91/250], Step [5/12], Loss: 0.7078998684883118\n",
      "Epoch [91/250], Step [10/12], Loss: 0.5581246018409729\n",
      "Epoch [92/250], Step [5/12], Loss: 0.6907145380973816\n",
      "Epoch [92/250], Step [10/12], Loss: 0.5737094283103943\n",
      "Epoch [93/250], Step [5/12], Loss: 0.6711478233337402\n",
      "Epoch [93/250], Step [10/12], Loss: 0.5999062061309814\n",
      "Epoch [94/250], Step [5/12], Loss: 0.768458366394043\n",
      "Epoch [94/250], Step [10/12], Loss: 0.5757535099983215\n",
      "Epoch [95/250], Step [5/12], Loss: 0.48073020577430725\n",
      "Epoch [95/250], Step [10/12], Loss: 0.6464473009109497\n",
      "Epoch [96/250], Step [5/12], Loss: 0.6394233703613281\n",
      "Epoch [96/250], Step [10/12], Loss: 0.6776993274688721\n",
      "Epoch [97/250], Step [5/12], Loss: 0.7528778910636902\n",
      "Epoch [97/250], Step [10/12], Loss: 0.4763181507587433\n",
      "Epoch [98/250], Step [5/12], Loss: 0.4863468408584595\n",
      "Epoch [98/250], Step [10/12], Loss: 0.6800742149353027\n",
      "Epoch [99/250], Step [5/12], Loss: 0.600294828414917\n",
      "Epoch [99/250], Step [10/12], Loss: 0.6526562571525574\n",
      "Epoch [100/250], Step [5/12], Loss: 0.7676653265953064\n",
      "Epoch [100/250], Step [10/12], Loss: 0.6269477009773254\n",
      "Epoch [101/250], Step [5/12], Loss: 0.7268717885017395\n",
      "Epoch [101/250], Step [10/12], Loss: 0.720143735408783\n",
      "Epoch [102/250], Step [5/12], Loss: 0.7146598696708679\n",
      "Epoch [102/250], Step [10/12], Loss: 0.48625853657722473\n",
      "Epoch [103/250], Step [5/12], Loss: 0.5319992899894714\n",
      "Epoch [103/250], Step [10/12], Loss: 0.658868134021759\n",
      "Epoch [104/250], Step [5/12], Loss: 0.5656092166900635\n",
      "Epoch [104/250], Step [10/12], Loss: 0.5621752142906189\n",
      "Epoch [105/250], Step [5/12], Loss: 0.8419973254203796\n",
      "Epoch [105/250], Step [10/12], Loss: 0.5264682769775391\n",
      "Epoch [106/250], Step [5/12], Loss: 0.6055259108543396\n",
      "Epoch [106/250], Step [10/12], Loss: 0.6472280025482178\n",
      "Epoch [107/250], Step [5/12], Loss: 0.5685010552406311\n",
      "Epoch [107/250], Step [10/12], Loss: 0.4947148263454437\n",
      "Epoch [108/250], Step [5/12], Loss: 0.6217973232269287\n",
      "Epoch [108/250], Step [10/12], Loss: 0.42401567101478577\n",
      "Epoch [109/250], Step [5/12], Loss: 0.523065447807312\n",
      "Epoch [109/250], Step [10/12], Loss: 0.6681222319602966\n",
      "Epoch [110/250], Step [5/12], Loss: 0.6541048884391785\n",
      "Epoch [110/250], Step [10/12], Loss: 0.4522257149219513\n",
      "Epoch [111/250], Step [5/12], Loss: 0.6520915627479553\n",
      "Epoch [111/250], Step [10/12], Loss: 0.5795755982398987\n",
      "Epoch [112/250], Step [5/12], Loss: 0.6348477005958557\n",
      "Epoch [112/250], Step [10/12], Loss: 0.5090774297714233\n",
      "Epoch [113/250], Step [5/12], Loss: 0.4684469401836395\n",
      "Epoch [113/250], Step [10/12], Loss: 0.5487300753593445\n",
      "Epoch [114/250], Step [5/12], Loss: 0.5744128823280334\n",
      "Epoch [114/250], Step [10/12], Loss: 0.5724759697914124\n",
      "Epoch [115/250], Step [5/12], Loss: 0.5911149382591248\n",
      "Epoch [115/250], Step [10/12], Loss: 0.6219180226325989\n",
      "Epoch [116/250], Step [5/12], Loss: 0.4721761643886566\n",
      "Epoch [116/250], Step [10/12], Loss: 0.7149360775947571\n",
      "Epoch [117/250], Step [5/12], Loss: 0.6957740783691406\n",
      "Epoch [117/250], Step [10/12], Loss: 0.5348675847053528\n",
      "Epoch [118/250], Step [5/12], Loss: 0.659222424030304\n",
      "Epoch [118/250], Step [10/12], Loss: 0.4960883557796478\n",
      "Epoch [119/250], Step [5/12], Loss: 0.5255290269851685\n",
      "Epoch [119/250], Step [10/12], Loss: 0.830420196056366\n",
      "Epoch [120/250], Step [5/12], Loss: 0.5645042061805725\n",
      "Epoch [120/250], Step [10/12], Loss: 0.5215747356414795\n",
      "Epoch [121/250], Step [5/12], Loss: 0.6195138096809387\n",
      "Epoch [121/250], Step [10/12], Loss: 0.6145480275154114\n",
      "Epoch [122/250], Step [5/12], Loss: 0.6540690064430237\n",
      "Epoch [122/250], Step [10/12], Loss: 0.766787588596344\n",
      "Epoch [123/250], Step [5/12], Loss: 0.5226128697395325\n",
      "Epoch [123/250], Step [10/12], Loss: 0.6030548810958862\n",
      "Epoch [124/250], Step [5/12], Loss: 0.7148452401161194\n",
      "Epoch [124/250], Step [10/12], Loss: 0.36636313796043396\n",
      "Epoch [125/250], Step [5/12], Loss: 0.6098440885543823\n",
      "Epoch [125/250], Step [10/12], Loss: 0.6579472422599792\n",
      "Epoch [126/250], Step [5/12], Loss: 0.5367874503135681\n",
      "Epoch [126/250], Step [10/12], Loss: 0.7062084078788757\n",
      "Epoch [127/250], Step [5/12], Loss: 0.4779360592365265\n",
      "Epoch [127/250], Step [10/12], Loss: 0.5555693507194519\n",
      "Epoch [128/250], Step [5/12], Loss: 0.5448893308639526\n",
      "Epoch [128/250], Step [10/12], Loss: 0.4706571102142334\n",
      "Epoch [129/250], Step [5/12], Loss: 0.4438749849796295\n",
      "Epoch [129/250], Step [10/12], Loss: 0.5861632227897644\n",
      "Epoch [130/250], Step [5/12], Loss: 0.48172926902770996\n",
      "Epoch [130/250], Step [10/12], Loss: 0.5449156165122986\n",
      "Epoch [131/250], Step [5/12], Loss: 0.8443935513496399\n",
      "Epoch [131/250], Step [10/12], Loss: 0.4660518169403076\n",
      "Epoch [132/250], Step [5/12], Loss: 0.41395798325538635\n",
      "Epoch [132/250], Step [10/12], Loss: 0.8073949217796326\n",
      "Epoch [133/250], Step [5/12], Loss: 0.6955594420433044\n",
      "Epoch [133/250], Step [10/12], Loss: 0.6981716752052307\n",
      "Epoch [134/250], Step [5/12], Loss: 0.5369669198989868\n",
      "Epoch [134/250], Step [10/12], Loss: 0.47646215558052063\n",
      "Epoch [135/250], Step [5/12], Loss: 0.6385157704353333\n",
      "Epoch [135/250], Step [10/12], Loss: 0.5048224329948425\n",
      "Epoch [136/250], Step [5/12], Loss: 0.5660095810890198\n",
      "Epoch [136/250], Step [10/12], Loss: 0.5386766791343689\n",
      "Epoch [137/250], Step [5/12], Loss: 0.6977777481079102\n",
      "Epoch [137/250], Step [10/12], Loss: 0.5311129689216614\n",
      "Epoch [138/250], Step [5/12], Loss: 0.6789419054985046\n",
      "Epoch [138/250], Step [10/12], Loss: 0.6062204837799072\n",
      "Epoch [139/250], Step [5/12], Loss: 0.6424932479858398\n",
      "Epoch [139/250], Step [10/12], Loss: 0.4849947392940521\n",
      "Epoch [140/250], Step [5/12], Loss: 0.49132809042930603\n",
      "Epoch [140/250], Step [10/12], Loss: 0.5788716673851013\n",
      "Epoch [141/250], Step [5/12], Loss: 0.6212441921234131\n",
      "Epoch [141/250], Step [10/12], Loss: 0.5554752945899963\n",
      "Epoch [142/250], Step [5/12], Loss: 0.5682074427604675\n",
      "Epoch [142/250], Step [10/12], Loss: 0.7214238047599792\n",
      "Epoch [143/250], Step [5/12], Loss: 0.559533417224884\n",
      "Epoch [143/250], Step [10/12], Loss: 0.6209107041358948\n",
      "Epoch [144/250], Step [5/12], Loss: 0.6507031321525574\n",
      "Epoch [144/250], Step [10/12], Loss: 0.45987406373023987\n",
      "Epoch [145/250], Step [5/12], Loss: 0.513490617275238\n",
      "Epoch [145/250], Step [10/12], Loss: 0.47854265570640564\n",
      "Epoch [146/250], Step [5/12], Loss: 0.3900632858276367\n",
      "Epoch [146/250], Step [10/12], Loss: 0.5036798119544983\n",
      "Epoch [147/250], Step [5/12], Loss: 0.5336616039276123\n",
      "Epoch [147/250], Step [10/12], Loss: 0.6263098120689392\n",
      "Epoch [148/250], Step [5/12], Loss: 0.6893370151519775\n",
      "Epoch [148/250], Step [10/12], Loss: 0.5566595792770386\n",
      "Epoch [149/250], Step [5/12], Loss: 0.5785121321678162\n",
      "Epoch [149/250], Step [10/12], Loss: 0.6246633529663086\n",
      "Epoch [150/250], Step [5/12], Loss: 0.46052059531211853\n",
      "Epoch [150/250], Step [10/12], Loss: 0.6124961972236633\n",
      "Epoch [151/250], Step [5/12], Loss: 0.5266459584236145\n",
      "Epoch [151/250], Step [10/12], Loss: 0.5756812691688538\n",
      "Epoch [152/250], Step [5/12], Loss: 0.4030889570713043\n",
      "Epoch [152/250], Step [10/12], Loss: 0.5560851097106934\n",
      "Epoch [153/250], Step [5/12], Loss: 0.6373293995857239\n",
      "Epoch [153/250], Step [10/12], Loss: 0.5470751523971558\n",
      "Epoch [154/250], Step [5/12], Loss: 0.5558340549468994\n",
      "Epoch [154/250], Step [10/12], Loss: 0.630460798740387\n",
      "Epoch [155/250], Step [5/12], Loss: 0.647042989730835\n",
      "Epoch [155/250], Step [10/12], Loss: 0.6054245829582214\n",
      "Epoch [156/250], Step [5/12], Loss: 0.636385977268219\n",
      "Epoch [156/250], Step [10/12], Loss: 0.5314143896102905\n",
      "Epoch [157/250], Step [5/12], Loss: 0.5784969925880432\n",
      "Epoch [157/250], Step [10/12], Loss: 0.5230282545089722\n",
      "Epoch [158/250], Step [5/12], Loss: 0.4949968755245209\n",
      "Epoch [158/250], Step [10/12], Loss: 0.6782801747322083\n",
      "Epoch [159/250], Step [5/12], Loss: 0.6202806830406189\n",
      "Epoch [159/250], Step [10/12], Loss: 0.5914879441261292\n",
      "Epoch [160/250], Step [5/12], Loss: 0.6870799660682678\n",
      "Epoch [160/250], Step [10/12], Loss: 0.6320522427558899\n",
      "Epoch [161/250], Step [5/12], Loss: 0.48908665776252747\n",
      "Epoch [161/250], Step [10/12], Loss: 0.7721757888793945\n",
      "Epoch [162/250], Step [5/12], Loss: 0.5689690709114075\n",
      "Epoch [162/250], Step [10/12], Loss: 0.6957208514213562\n",
      "Epoch [163/250], Step [5/12], Loss: 0.7140031456947327\n",
      "Epoch [163/250], Step [10/12], Loss: 0.6814512610435486\n",
      "Epoch [164/250], Step [5/12], Loss: 0.46915149688720703\n",
      "Epoch [164/250], Step [10/12], Loss: 0.6629310846328735\n",
      "Epoch [165/250], Step [5/12], Loss: 0.5727963447570801\n",
      "Epoch [165/250], Step [10/12], Loss: 0.5812936425209045\n",
      "Epoch [166/250], Step [5/12], Loss: 0.4165939390659332\n",
      "Epoch [166/250], Step [10/12], Loss: 0.4136977195739746\n",
      "Epoch [167/250], Step [5/12], Loss: 0.55246901512146\n",
      "Epoch [167/250], Step [10/12], Loss: 0.30603960156440735\n",
      "Epoch [168/250], Step [5/12], Loss: 0.4168109893798828\n",
      "Epoch [168/250], Step [10/12], Loss: 0.48200392723083496\n",
      "Epoch [169/250], Step [5/12], Loss: 0.6105254292488098\n",
      "Epoch [169/250], Step [10/12], Loss: 0.5242034792900085\n",
      "Epoch [170/250], Step [5/12], Loss: 0.5001739859580994\n",
      "Epoch [170/250], Step [10/12], Loss: 0.501693844795227\n",
      "Epoch [171/250], Step [5/12], Loss: 0.5478376746177673\n",
      "Epoch [171/250], Step [10/12], Loss: 0.41934260725975037\n",
      "Epoch [172/250], Step [5/12], Loss: 0.4464395344257355\n",
      "Epoch [172/250], Step [10/12], Loss: 0.4842757284641266\n",
      "Epoch [173/250], Step [5/12], Loss: 0.528644859790802\n",
      "Epoch [173/250], Step [10/12], Loss: 0.5157061219215393\n",
      "Epoch [174/250], Step [5/12], Loss: 0.6207395195960999\n",
      "Epoch [174/250], Step [10/12], Loss: 0.41896119713783264\n",
      "Epoch [175/250], Step [5/12], Loss: 0.45127812027931213\n",
      "Epoch [175/250], Step [10/12], Loss: 0.5038074851036072\n",
      "Epoch [176/250], Step [5/12], Loss: 0.5167074203491211\n",
      "Epoch [176/250], Step [10/12], Loss: 0.5774943232536316\n",
      "Epoch [177/250], Step [5/12], Loss: 0.47361281514167786\n",
      "Epoch [177/250], Step [10/12], Loss: 0.49255263805389404\n",
      "Epoch [178/250], Step [5/12], Loss: 0.43356263637542725\n",
      "Epoch [178/250], Step [10/12], Loss: 0.7756748199462891\n",
      "Epoch [179/250], Step [5/12], Loss: 0.5265530943870544\n",
      "Epoch [179/250], Step [10/12], Loss: 0.5404073596000671\n",
      "Epoch [180/250], Step [5/12], Loss: 0.4708535671234131\n",
      "Epoch [180/250], Step [10/12], Loss: 0.5667527318000793\n",
      "Epoch [181/250], Step [5/12], Loss: 0.5208846926689148\n",
      "Epoch [181/250], Step [10/12], Loss: 0.435342937707901\n",
      "Epoch [182/250], Step [5/12], Loss: 0.44634151458740234\n",
      "Epoch [182/250], Step [10/12], Loss: 0.5740182995796204\n",
      "Epoch [183/250], Step [5/12], Loss: 0.4374209940433502\n",
      "Epoch [183/250], Step [10/12], Loss: 0.6570439338684082\n",
      "Epoch [184/250], Step [5/12], Loss: 0.4620250165462494\n",
      "Epoch [184/250], Step [10/12], Loss: 0.441683292388916\n",
      "Epoch [185/250], Step [5/12], Loss: 0.5773098468780518\n",
      "Epoch [185/250], Step [10/12], Loss: 0.37220343947410583\n",
      "Epoch [186/250], Step [5/12], Loss: 0.5075629949569702\n",
      "Epoch [186/250], Step [10/12], Loss: 0.6122865676879883\n",
      "Epoch [187/250], Step [5/12], Loss: 0.5046077370643616\n",
      "Epoch [187/250], Step [10/12], Loss: 0.6265876293182373\n",
      "Epoch [188/250], Step [5/12], Loss: 0.4457426071166992\n",
      "Epoch [188/250], Step [10/12], Loss: 0.5896877646446228\n",
      "Epoch [189/250], Step [5/12], Loss: 0.6334258317947388\n",
      "Epoch [189/250], Step [10/12], Loss: 0.5004339814186096\n",
      "Epoch [190/250], Step [5/12], Loss: 0.5618569850921631\n",
      "Epoch [190/250], Step [10/12], Loss: 0.5064927935600281\n",
      "Epoch [191/250], Step [5/12], Loss: 0.41816583275794983\n",
      "Epoch [191/250], Step [10/12], Loss: 0.5314803123474121\n",
      "Epoch [192/250], Step [5/12], Loss: 0.3896982669830322\n",
      "Epoch [192/250], Step [10/12], Loss: 0.5303875207901001\n",
      "Epoch [193/250], Step [5/12], Loss: 0.6333739161491394\n",
      "Epoch [193/250], Step [10/12], Loss: 0.44971799850463867\n",
      "Epoch [194/250], Step [5/12], Loss: 0.5545072555541992\n",
      "Epoch [194/250], Step [10/12], Loss: 0.5693171620368958\n",
      "Epoch [195/250], Step [5/12], Loss: 0.43116775155067444\n",
      "Epoch [195/250], Step [10/12], Loss: 0.5982306599617004\n",
      "Epoch [196/250], Step [5/12], Loss: 0.5567825436592102\n",
      "Epoch [196/250], Step [10/12], Loss: 0.412104994058609\n",
      "Epoch [197/250], Step [5/12], Loss: 0.5536391139030457\n",
      "Epoch [197/250], Step [10/12], Loss: 0.4832352101802826\n",
      "Epoch [198/250], Step [5/12], Loss: 0.6024084687232971\n",
      "Epoch [198/250], Step [10/12], Loss: 0.46690282225608826\n",
      "Epoch [199/250], Step [5/12], Loss: 0.46420904994010925\n",
      "Epoch [199/250], Step [10/12], Loss: 0.38823723793029785\n",
      "Epoch [200/250], Step [5/12], Loss: 0.48187288641929626\n",
      "Epoch [200/250], Step [10/12], Loss: 0.5722905397415161\n",
      "Epoch [201/250], Step [5/12], Loss: 0.4697374403476715\n",
      "Epoch [201/250], Step [10/12], Loss: 0.45410022139549255\n",
      "Epoch [202/250], Step [5/12], Loss: 0.48376283049583435\n",
      "Epoch [202/250], Step [10/12], Loss: 0.4490412771701813\n",
      "Epoch [203/250], Step [5/12], Loss: 0.46905985474586487\n",
      "Epoch [203/250], Step [10/12], Loss: 0.6396721005439758\n",
      "Epoch [204/250], Step [5/12], Loss: 0.44873419404029846\n",
      "Epoch [204/250], Step [10/12], Loss: 0.5548318028450012\n",
      "Epoch [205/250], Step [5/12], Loss: 0.47319495677948\n",
      "Epoch [205/250], Step [10/12], Loss: 0.5177721381187439\n",
      "Epoch [206/250], Step [5/12], Loss: 0.6542184352874756\n",
      "Epoch [206/250], Step [10/12], Loss: 0.4774671494960785\n",
      "Epoch [207/250], Step [5/12], Loss: 0.507783830165863\n",
      "Epoch [207/250], Step [10/12], Loss: 0.5370334982872009\n",
      "Epoch [208/250], Step [5/12], Loss: 0.5756478905677795\n",
      "Epoch [208/250], Step [10/12], Loss: 0.4218917787075043\n",
      "Epoch [209/250], Step [5/12], Loss: 0.4819541871547699\n",
      "Epoch [209/250], Step [10/12], Loss: 0.4433853328227997\n",
      "Epoch [210/250], Step [5/12], Loss: 0.4604558050632477\n",
      "Epoch [210/250], Step [10/12], Loss: 0.42502686381340027\n",
      "Epoch [211/250], Step [5/12], Loss: 0.5868327617645264\n",
      "Epoch [211/250], Step [10/12], Loss: 0.5961112976074219\n",
      "Epoch [212/250], Step [5/12], Loss: 0.5208373665809631\n",
      "Epoch [212/250], Step [10/12], Loss: 0.4376973807811737\n",
      "Epoch [213/250], Step [5/12], Loss: 0.5669718384742737\n",
      "Epoch [213/250], Step [10/12], Loss: 0.43198931217193604\n",
      "Epoch [214/250], Step [5/12], Loss: 0.4566718637943268\n",
      "Epoch [214/250], Step [10/12], Loss: 0.5014210939407349\n",
      "Epoch [215/250], Step [5/12], Loss: 0.6302293539047241\n",
      "Epoch [215/250], Step [10/12], Loss: 0.37625908851623535\n",
      "Epoch [216/250], Step [5/12], Loss: 0.576160728931427\n",
      "Epoch [216/250], Step [10/12], Loss: 0.503012478351593\n",
      "Epoch [217/250], Step [5/12], Loss: 0.5102578997612\n",
      "Epoch [217/250], Step [10/12], Loss: 0.5113169550895691\n",
      "Epoch [218/250], Step [5/12], Loss: 0.3296416103839874\n",
      "Epoch [218/250], Step [10/12], Loss: 0.46643105149269104\n",
      "Epoch [219/250], Step [5/12], Loss: 0.4287164509296417\n",
      "Epoch [219/250], Step [10/12], Loss: 0.50275057554245\n",
      "Epoch [220/250], Step [5/12], Loss: 0.5335931181907654\n",
      "Epoch [220/250], Step [10/12], Loss: 0.49387356638908386\n",
      "Epoch [221/250], Step [5/12], Loss: 0.6015412211418152\n",
      "Epoch [221/250], Step [10/12], Loss: 0.5704613327980042\n",
      "Epoch [222/250], Step [5/12], Loss: 0.5521556735038757\n",
      "Epoch [222/250], Step [10/12], Loss: 0.41870245337486267\n",
      "Epoch [223/250], Step [5/12], Loss: 0.4756302833557129\n",
      "Epoch [223/250], Step [10/12], Loss: 0.4610620439052582\n",
      "Epoch [224/250], Step [5/12], Loss: 0.5153950452804565\n",
      "Epoch [224/250], Step [10/12], Loss: 0.45920583605766296\n",
      "Epoch [225/250], Step [5/12], Loss: 0.3345031440258026\n",
      "Epoch [225/250], Step [10/12], Loss: 0.5306186676025391\n",
      "Epoch [226/250], Step [5/12], Loss: 0.44113802909851074\n",
      "Epoch [226/250], Step [10/12], Loss: 0.36654409766197205\n",
      "Epoch [227/250], Step [5/12], Loss: 0.5393785834312439\n",
      "Epoch [227/250], Step [10/12], Loss: 0.488373726606369\n",
      "Epoch [228/250], Step [5/12], Loss: 0.4684091806411743\n",
      "Epoch [228/250], Step [10/12], Loss: 0.5146152377128601\n",
      "Epoch [229/250], Step [5/12], Loss: 0.5446910262107849\n",
      "Epoch [229/250], Step [10/12], Loss: 0.4382711946964264\n",
      "Epoch [230/250], Step [5/12], Loss: 0.49910274147987366\n",
      "Epoch [230/250], Step [10/12], Loss: 0.5086660385131836\n",
      "Epoch [231/250], Step [5/12], Loss: 0.47374463081359863\n",
      "Epoch [231/250], Step [10/12], Loss: 0.529356062412262\n",
      "Epoch [232/250], Step [5/12], Loss: 0.5051507353782654\n",
      "Epoch [232/250], Step [10/12], Loss: 0.47849011421203613\n",
      "Epoch [233/250], Step [5/12], Loss: 0.4701507091522217\n",
      "Epoch [233/250], Step [10/12], Loss: 0.516226589679718\n",
      "Epoch [234/250], Step [5/12], Loss: 0.5082336664199829\n",
      "Epoch [234/250], Step [10/12], Loss: 0.36191055178642273\n",
      "Epoch [235/250], Step [5/12], Loss: 0.4284961223602295\n",
      "Epoch [235/250], Step [10/12], Loss: 0.4748823344707489\n",
      "Epoch [236/250], Step [5/12], Loss: 0.4528276026248932\n",
      "Epoch [236/250], Step [10/12], Loss: 0.5855270028114319\n",
      "Epoch [237/250], Step [5/12], Loss: 0.48532557487487793\n",
      "Epoch [237/250], Step [10/12], Loss: 0.5158505439758301\n",
      "Epoch [238/250], Step [5/12], Loss: 0.46662846207618713\n",
      "Epoch [238/250], Step [10/12], Loss: 0.5520396828651428\n",
      "Epoch [239/250], Step [5/12], Loss: 0.5197553038597107\n",
      "Epoch [239/250], Step [10/12], Loss: 0.4596313536167145\n",
      "Epoch [240/250], Step [5/12], Loss: 0.5161696672439575\n",
      "Epoch [240/250], Step [10/12], Loss: 0.5567560791969299\n",
      "Epoch [241/250], Step [5/12], Loss: 0.6713992953300476\n",
      "Epoch [241/250], Step [10/12], Loss: 0.5731367468833923\n",
      "Epoch [242/250], Step [5/12], Loss: 0.37108638882637024\n",
      "Epoch [242/250], Step [10/12], Loss: 0.5100929141044617\n",
      "Epoch [243/250], Step [5/12], Loss: 0.3971511423587799\n",
      "Epoch [243/250], Step [10/12], Loss: 0.5131604075431824\n",
      "Epoch [244/250], Step [5/12], Loss: 0.42649826407432556\n",
      "Epoch [244/250], Step [10/12], Loss: 0.503757655620575\n",
      "Epoch [245/250], Step [5/12], Loss: 0.549298882484436\n",
      "Epoch [245/250], Step [10/12], Loss: 0.4970736801624298\n",
      "Epoch [246/250], Step [5/12], Loss: 0.5271530151367188\n",
      "Epoch [246/250], Step [10/12], Loss: 0.4739414155483246\n",
      "Epoch [247/250], Step [5/12], Loss: 0.5755369663238525\n",
      "Epoch [247/250], Step [10/12], Loss: 0.46954452991485596\n",
      "Epoch [248/250], Step [5/12], Loss: 0.38546547293663025\n",
      "Epoch [248/250], Step [10/12], Loss: 0.4378717839717865\n",
      "Epoch [249/250], Step [5/12], Loss: 0.4321870505809784\n",
      "Epoch [249/250], Step [10/12], Loss: 0.38624921441078186\n",
      "Epoch [250/250], Step [5/12], Loss: 0.45629969239234924\n",
      "Epoch [250/250], Step [10/12], Loss: 0.47849181294441223\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 250\n",
    "batch_size = 8\n",
    "learning_rate = 0.001\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
    "loss = nn.CrossEntropyLoss()\n",
    "\n",
    "# Create the DataLoader for the dataset\n",
    "from torch.utils.data import DataLoader\n",
    "data_loader = DataLoader(emoji_dataset, batch_size=batch_size, shuffle=True)\n",
    "\n",
    "# Training loop\n",
    "for epoch in range(num_epochs):\n",
    "    for i, (images, labels) in enumerate(data_loader):\n",
    "        # this break is just a workournd to avoid the error of the mismatched size of the output and the truth\n",
    "        if i >= 11: break \n",
    "\n",
    "        images = images.to(device)\n",
    "        \n",
    "        # Forward pass\n",
    "        outputs = model(images)\n",
    "\n",
    "        outputs_flat = outputs.reshape(batch_size * input_dim * image_size * image_size, num_classes)\n",
    "\n",
    "        # print(images.shape)\n",
    "        # print(outputs.shape)\n",
    "        truth_flat = images.view(batch_size * input_dim * image_size * image_size).long()\n",
    "\n",
    "        # Compute the loss\n",
    "        loss_value = loss(outputs_flat, truth_flat)\n",
    "        \n",
    "        # Zero the gradients\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        # Backward pass\n",
    "        loss_value.backward()\n",
    "        \n",
    "        # Update the parameters\n",
    "        optimizer.step()\n",
    "        \n",
    "        if (i+1) % 5 == 0:\n",
    "            print(f'Epoch [{epoch+1}/{num_epochs}], Step [{i+1}/{len(data_loader)}], Loss: {loss_value.item()}')\n",
    "    \n",
    "    if (epoch+1) % 1000 == 0:\n",
    "        # Sample from the model after each epoch\n",
    "        example_image, _  = emoji_dataset[10]\n",
    "        example_image = example_image.unsqueeze(0).to(device)\n",
    "        output = model(example_image)\n",
    "        generated_image = sample_from_pixelcnn(output)\n",
    "        show_tensor_image(generated_image[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test (with teacher forcing)\n",
    "\n",
    "Using teacher forcing will eliminate the possability of \"creativity\" for our model, since there are predermined inputs for the image generation. The result should resemble an existing image from our dataset and that is exactly what we see.\n",
    "\n",
    "Notice that sampling from the output takes a long time, this is a big dissadvantage of this architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/db/b4g1bwvj4qd9pwhjmkfm9pxh0000gn/T/ipykernel_36961/1414903246.py:59: DeprecationWarning: __array__ implementation doesn't accept a copy keyword, so passing copy=False failed. __array__ must implement 'dtype' and 'copy' keyword arguments.\n",
      "  image_np = np.array(image)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape:  torch.Size([1, 3, 32, 32, 16])\n",
      "tensor(0.3388, device='mps:0')\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGFCAYAAAASI+9IAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAOwElEQVR4nO3cwXEcWXYF0AdF7/k9YHrQaUJ5oDIBHogm0ASaAHlQ8qDGg2oPEh58WoBZcOJJoZgI/cthimD3Oetbl4kCwBu1wHt6e3t7KwCoqn/72Q8AwPthFABoRgGAZhQAaEYBgGYUAGhGAYBmFABov60Gn56eznwOAKqqPgTZr1n1yt8q+6QAQDMKADSjAEAzCgA0owBAMwoANKMAQDMKADSjAEAzCgA0owBAe3pbOYZRbh/92Zx4XiXy779n+THWn3xk1fELZpA9jqz7mOvv+kwepKq+nvkN5V1z+wiAiFEAoBkFAJpRAKAZBQCaUQCgGQUAmlEAoBkFAJpRAKD99rMfgH/uY3KH4jtc9vXs9fIx6t7HXM6OfYu6o1MUI+yODleE+UfYHNyueBzBc1TV/b5+5+L2iKrr1QmNX55PCgA0owBAMwoANKMAQDMKADSjAEAzCgA0owBAMwoANKMAQDMKALSnt7e3t6Xg09PZz/Kn9yE4IbRv2fGjL9fsWbZ9JOGsfCT5sLu25WRwPqiqqsbI8pnjvPy8h9Xr3ccjeI6q+nJbz96O7FDS19cozj+x8t+9TwoANKMAQDMKADSjAEAzCgA0owBAMwoANKMAQDMKADSjAEBz5uJf8DG7RFGfL+svuD5n3bVfs/y4BOEtqp7JY9SIuqNrEendijCefaFJuGoeQfWWdUcPPl+y6sdjOfrlHjxHVd1esrMYf2TxvwRnLgCIGAUAmlEAoBkFAJpRAKAZBQCaUQCgGQUAmlEAoBkFAJpRAKC5ffS//P5xPfvlmnXv16B8C8ujW0ZVMzj0s5785hFkt7A7f5rzmudJ2aqqI3jFCLv3KD3D9kcQfcmaw1tJn4JbSX+8RtW/LLePAIgYBQCaUQCgGQUAmlEAoBkFAJpRAKAZBQCaUQCgGQUA2p/+zMXHD1n+5Xn9Bft1RN0zOV0x9qh7ROmqmsEr4vIzjdOaZ/qCI4huWfUePM168pvxWM/e96x9BNk9eQOrqh63LH5f738OTmJUVb3+omcxnLkAIGIUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYBmFABoRgGA9kvePvoQ3DP6cs2OH10vQfhyjbqTc0Oz9qj7iNJV0UmbMbLy+309e7lE1SPIHvGdpBmlt7A9MYPsSMJxe2isR2dafRzZC47bcvR2y44ZfVqvrq/ZWaVTuX0EQMQoANCMAgDNKADQjAIAzSgA0IwCAM0oANCMAgDNKADQfvvZD/A9rnuQvYTle1Aenn8Y4ZGGxCW5oVFVx2M9e3+5Zd3Bs3+6bFF3ckdhhIcU4nMRYz06Z1Y+b8dy9na/Rd31aSxHn/fnrDuw/hTf+YJtX45eLzOqvh/rtyv+829R9U/nkwIAzSgA0IwCAM0oANCMAgDNKADQjAIAzSgA0IwCAM0oANCMAgDtXdw++vAhy3/ag3B4n2iObT38MqPu47qe3UdUnR0zqqpt29az44i6I48wv8/l6Air8xecV30L7kd9qfU7PFVVnx9jPbzNqPvM93CG+ehXf0vCVc+X9ez9kX1/XrP4D+eTAgDNKADQjAIAzSgA0IwCAM0oANCMAgDNKADQjAIAzSgA0IwCAO1d3D563rP8tq8fS5rh7aNHcGHlco2qayTXW4JoVcW3W+q4L0fn/TV7lOvvy9kR31WaQXaE3Wk+MGcUH49jOft5RNV12ZMHWX+Of7xgPTqz5pm/YNlIbp5V1b491rN7VF2vf8vyP5pPCgA0owBAMwoANKMAQDMKADSjAEAzCgA0owBAMwoANKMAQDvtzMWH9UsUtaXlI8km4apL8LfxMzyLsEXp1BGlb8nZhUvwzayq41h/luPLiLoryG9Zc6XvYd3Wo4/7I+uO75ysuwfV1/A8xxzBc5x5VqSq9hPfw+S/levIfn/u9XU5u55c55MCAM0oANCMAgDNKADQjAIAzSgA0IwCAM0oANCMAgDNKADQjAIA7bTbR3uQvYS3dZLjRyO83ZLcY0nuJFVVzbGeH/FdmPXuqqotyB9Rc+bxnLVfo4tGM+pOHZexHt62qHu+PJazY2a/P/PzDJ4jqq4RvOfp78+R3mEKfoe2sDv5P2jfX7Pm+3r26wnHj3xSAKAZBQCaUQCgGQUAmlEAoBkFAJpRAKAZBQCaUQCgGQUA2mlnLo6xnh0j/VvtLcyvS/70/gj/TH/MkYSj7vSiwxb8mf6jsj/TT77MaxL+1h5kw+6kuqq24Hv0CL+f276enY+ouq5B9zjx5zA/5ZLZj7keTr/OwBjZGZLk/8NXZy4AOJNRAKAZBQCaUQCgGQUAmlEAoBkFAJpRAKAZBQCaUQCgGQUA2mm3jy4jCIe3QaIDK0H0m7Gc3MZ6NjbPq66qGsFxqn2fUff9FoS39eeoquyw0tnmXI6Gb2Hdo3R2AOde679v1xlVh7LyPa0fp4Wr6jiteQ9e8Ed2lmyJTwoANKMAQDMKADSjAEAzCgA0owBAMwoANKMAQDMKADSjAEA77cxFZIb5kbxghOV/Eds8rfoa3DhJL4XM4IdlPfnNFubPLN+DMyT35/VsVdU1edOz6ppHUB12J2dF/vEvnNcdxbMzJCNK/3g+KQDQjAIAzSgA0IwCAM0oANCMAgDNKADQjAIAzSgA0IwCAM0oANDex+2jd2UuJx9h8/7Tr5r8D9tYjsbPvWfxxDgpe7qZxcc+lrPXrDozs3h8zyhrP7P8NLM+ZC8Y2a2kH80nBQCaUQCgGQUAmlEAoBkFAJpRAKAZBQCaUQCgGQUAmlEAoBkFANq7uH000hfMM8vXX7DNmVXf1qPHc1Y9ZpZPDjeNLexO8iPsDswwP054hu8unyc8w3e4h/lLEp5Z9yOL1/5e3sSZxY8w/6P5pABAMwoANKMAQDMKADSjAEAzCgA0owBAMwoANKMAQDMKALTTzlwkf6qd/ln3NpJ0WB7FR1R9C85iXGbWPY8oXo/gLsYWfp17kL2vP0ZVhWcUYjNMj/VsVh39jM9H1j324DnCb9DjCLJRc2XnU6pqC/Ijq47M+prl5znPsconBQCaUQCgGQUAmlEAoBkFAJpRAKAZBQCaUQCgGQUAmlEAoBkFANq7uH00Z3YbpOaHLJ8Y69HHPas+tut69r6F3bcon9yReflyRNWXy4jyidsjSc+sfM/i476efWwz6r4G8fvcou66r0fn+BRVb9uxnN3TN/x4RPExbkF4Rt3Jj1Z6y+gR5n80nxQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYB22pmL5HLF48i6t209O7LqyHbJ8kfwZ/3bkZWP8Zzlg7MLly9RdXRc4jGz8m1cg/ARdY+xRfnkW5ScrajKfm6fs+q6B79Al/j8w1iPhr/49+NzlN+TZz+i6uhn/BZ2V3j150fzSQGAZhQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYB22u2j5HzH45F1X/b17Miqq+Zcj0YXUKq2x3r2Nl6i7n1/jvJb8HXGN4Tqsl49PkfdtSXhPes+0XZi9wzzybfzcbxk5S/39ecI7m9VVV23KB69MUH0m2P9f7j0tttPPn3kkwIA/80oANCMAgDNKADQjAIAzSgA0IwCAM0oANCMAgDNKADQnt7e3t6Wgk9Ppz3Exw9Z/vN1PbtvWfm2RfHI7bGenWNk5cnZiqqac1vO7mOPumusd8+wOrqMkGSr6hG+YBxBNnwP9/lpOXvfRtSd/KwkX2NVVRLf9qz7kn6dxzwj+i3/WD9G8emedb+eeOdi5b97nxQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYBmFABo7+L2Ueo/fl/PXi9Zd3L7aFR4tGmsRx+PIFxVx3yN8hU9e3qM5b10n2wG2ZF+ncEP+XFk1VtQHX5/tu3jcvYaHbKq+JZV4jiyr/PzfT37X39kz3Imt48AiBgFAJpRAKAZBQCaUQCgGQUAmlEAoBkFAJpRAKAZBQDabz/7Ab7HyxGEb1n38/N6doysOzHD/OWanX+435I/6z/ztETa/V6eu6JzETWzZxl1rFcnz1FVc6xnt/A9vI4Z5RNp85zrPyv3I+tO878SnxQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYBmFABov+Tto6/B+ZuXsHs71rPXLbnDUzWCOzKXfUbdM4tXbcFNmyPsHkE2uE/zzcn3jBLzl6yuPbgfte8nvt8zzWc/K49jPfvlHlVH/wf9anxSAKAZBQCaUQCgGQUAmlEAoBkFAJpRAKAZBQCaUQCgGQUAmlEAoD29vb29LQWfns5+lnfhQ3Dq5csl69739ewY5974GTMIb1n3/RjL2XkkD1LZvZz1x/g+Sf9MwlVjW89eom9m9ihhdfTtmSfeMqrK7hn98Zp1/6pW/rv3SQGAZhQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGjOXPwLPoaXKD5d1rOXLeuOz2KMpDurnlk8LB/L0fy5Z5RP6mf4MONIwjPqfgTx/MzF+umKxyPrTs5WVFW9Zlc0/hKcuQAgYhQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYDm9tH/ow/BeaL09tFlZPna16PX9K5S4My7SmH1qeY7Kj+C/DGzA0L3x3r2dkTV9dUto3+Z20cARIwCAM0oANCMAgDNKADQjAIAzSgA0IwCAM0oANCMAgDNmYs/iY/hJYp9rGcv+3nd6ZmLU29XzCyePPsMu48kG3Y/7uvZe9j96hTFu+bMBQARowBAMwoANKMAQDMKADSjAEAzCgA0owBAMwoANKMAQDMKADS3j/g/hWeVaiQvGFn3FuS3mXWnz5LeHDqre4b3hpwn+uty+wiAiFEAoBkFAJpRAKAZBQCaUQCgGQUAmlEAoBkFAJpRAKA5cwHwF+HMBQARowBAMwoANKMAQDMKADSjAEAzCgA0owBAMwoANKMAQDMKALTfVoOLJ5IA+IX5pABAMwoANKMAQDMKADSjAEAzCgA0owBAMwoANKMAQPs72iXzg5a28ZcAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "referece_test_image, _ = emoji_dataset[1]\n",
    "referece_test_image = referece_test_image.unsqueeze(0).to(device)\n",
    "\n",
    "with torch.no_grad():\n",
    "    test_image = model(referece_test_image)\n",
    "    print('shape: ', test_image.shape)\n",
    "    # print(test_image.shape)\n",
    "    flat1 = test_image.reshape(-1, num_classes)\n",
    "    flat2 = referece_test_image.reshape(-1).long()\n",
    "    l = loss(flat1, flat2)\n",
    "\n",
    "    # print(F.softmax(flat1[29], dim=0))\n",
    "    # print(flat2)\n",
    "    print(l)\n",
    "    \n",
    "    generated_image = sample_from_pixelcnn(test_image)\n",
    "    show_tensor_image(generated_image[0])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a program sampling images from scratch using the trained model, performing a new forward pass for each new pixel and adding it to the image, recursively.\n",
    "\n",
    "Obviously, autoregression models are extremely inefficient when generating a new images from scratch.\n",
    "\n",
    "The outputs are not the best, but we can see that the model has learned some basic structures of the emojis, the shape of the face, eyes and mouth, etc.\n",
    "\n",
    "Feel free to play around with the temperature term which is currently 0.6 in the model.\n",
    "\n",
    "**Note** - The reason why many generated emojis have a straight bottom right part of the face is most likely due to the blind spot of our Masked Convolutional layers. This could have been solved by seperating the process into the vertical and horizontal stacks as stated in this [tutorial](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial12/Autoregressive_Image_Modeling.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_image(model, image_size=16, num_classes=256, temperature=1):\n",
    "    # Start with a blank image (batch_size=1, 3 channels for RGB, height, width)\n",
    "    generated_image = torch.zeros(1, 3, image_size, image_size).to(device)\n",
    "    \n",
    "    # Iterate over every pixel, row by row, column by column\n",
    "    for i in range(image_size):\n",
    "        # print('i: ', i)\n",
    "        for j in range(image_size):\n",
    "            # Forward pass through the model to predict the distribution for the current pixel\n",
    "            with torch.no_grad():  # No need to compute gradients\n",
    "                if i > 0 or j > 0 or True: output = model(generated_image)\n",
    "                else: \n",
    "                    output = torch.rand(1, 3, image_size, image_size, num_classes).to(device)\n",
    "                    # print(output.shape)\n",
    "            \n",
    "            output = output / temperature\n",
    "            \n",
    "            pixel_outputs_r = sample_one_pixel(output[0][0][i][j])\n",
    "            pixel_outputs_g = sample_one_pixel(output[0][1][i][j])\n",
    "            pixel_outputs_b = sample_one_pixel(output[0][2][i][j])\n",
    "            # print(pixel_outputs_r, pixel_outputs_g, pixel_outputs_b)\n",
    "\n",
    "            # print(pixel_outputs)\n",
    "            \n",
    "            # Assign the sampled pixel values to the corresponding pixel in the image\n",
    "            generated_image[0, 0, i, j] = pixel_outputs_r  # Normalize between 0 and 1\n",
    "            generated_image[0, 1, i, j] = pixel_outputs_g\n",
    "            generated_image[0, 2, i, j] = pixel_outputs_b\n",
    "\n",
    "    return generated_image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Higer temperature produces more diverse images, lower temperature produces more stable outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGFCAYAAAASI+9IAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAANtklEQVR4nO3c3XEcV5oE0IsNvfN6wDKhPVB7sPBgaYJMoAk0QeMB1oOmBz0eFD24sAD7wJmc2J+IrWSoBBA65zk7WQIpZNRDfw8vLy8vAwDGGP/22g8AwNthFAAIowBAGAUAwigAEEYBgDAKAIRRACB+ORp8eHg48zkAONmR7yp7UwAgjAIAYRQACKMAQBgFAMIoABBGAYAwCgCEUQAgjAIAYRQAiMO3j3hfPhTZ2YTHGHM7/oFtlt1lvupeXX4v8vf9ueouqsdzV/2XUf6zrbznH7k3BQDCKAAQRgGAMAoAhFEAIIwCAGEUAAijAEAYBQDCKAAQzly8UR/L7+hfL90HHq/Hs5dZVY/ZfODMuxWj7V5l/Hh+7V31vcjfbl33U3Fy49u3rvstec+nKM7kTQGAMAoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAhFEAIB5eXl5eDgUfHs5+lnfv3389nv187W4ZbdfZPcx2OZ6dRXaMMca1zDfm4eQ67Rm+m2Mv0reuvDmWtN+r6v2+Dme/PFXV4+l+/OLQs+NEf7ojv+69KQAQRgGAMAoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAcObif/hQXJd4vHTdnx+Pl89rWb596vJjK7Kz7G7zP6niXMSaXfXcjneP6tzGGGM9Hc/eb1X1UxH//NTdufj2rYrzf3DmAoCKUQAgjAIAYRQACKMAQBgFAMIoABBGAYAwCgCEUQAgjAIA8e5vHxWnjMYYY3z69Xj286ey/XI9HF3lLaO9vDdUndbpqqsPtNWrzJ/paa3D2cvedc8mfDn+HGN0V6/GuFfpcT+ev99uVfVvv3e3kv7uVtL/4vYRABWjAEAYBQDCKAAQRgGAMAoAhFEAIIwCAGEUAAijAEC8+zMX/1GcrRhjjC+PxemK62PVvW/Xw9l5r6rHfZ9VvkkXj/29uylfTfhtnbkYxZmLVvMzXG35fvwTbfcs7nnM+1PVfb8d7x5jjC/FWYz//IucxHDmAoCKUQAgjAIAYRQACKMAQBgFAMIoABBGAYAwCgCEUQAgjAIA8ctrP8CP+NicJ7oW4THG2C6Ho6vI/uMTRXfXfJnHu8cY1XGd6pbRKO/llN2zuDe0d9X1van276jqLrJbeaHoPo9n515Vj9Xk52PVPS+/V/nHon49Hb+TNMYYX9/xrSRvCgCEUQAgjAIAYRQACKMAQBgFAMIoABBGAYAwCgCEUQAgfsozF9fL8ezjVpZfjpfP8rzAPC3c51dzcmN15VW6PM9RPUpXPdaly88mvHfd+7a6DxQuswhvbfs6nNzrf+SPXfrydDi7quYx7r8fP4vx3F3QeHXeFAAIowBAGAUAwigAEEYBgDAKAIRRACCMAgBhFAAIowBAGAUA4k3cPvr4ocs/XooPzK0rb263HI+OMcozMrPrblX15X2iMx++aZ5N+GyX1cVPeYi3pb4dNss/oLhj9mncqur78erxt69V9avzpgBAGAUAwigAEEYBgDAKAIRRACCMAgBhFAAIowBAGAUAwigAEG/i9tG1uWU0xrhejmf38vbRWutw9nI7nh1jjHGdx7NF9LtV5us/4BSrzM8TnuGfVplvb/fw3/U/v1mlV3EsaW5d96fr8+Hs7V5Vj2/Hq0/hTQGAMAoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAxGlnLprDFY+Xsnwe/0D57fXqi/fr2nXP2bTPrvxUq0rvRX6rmsdYxc+lPaNwvLm3Tuyef8InTrFm+4Eq3bSv8hzOZVuHs9dLd7fib1+r+B/OmwIAYRQACKMAQBgFAMIoABBGAYAwCgCEUQAgjAIAYRQACKMAQJx2+2h+PJ69zLZ9P5xc7X2VeTxfNleXW+Zq0qN+mNU9TdW9VenWOrW9sRfZ7S1dP6r+bTXZzq187us65THGGP2drObRH4vsGGM8FdnuqtIx3hQACKMAQBgFAMIoABBGAYAwCgCEUQAgjAIAYRQACKMAQJx25qI6XTHbL2sfL28eY4wx9hO/1j+b6lmWl+c8qnT14J22eZ7wDD9qO7V9Ftn9pGc4V3u2or5YU5zz6K/KzMPZ7dL9fpu349nnE+5ceFMAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAhFEAIIwCAGEUAIjTbh/NKvvhvPLSVmTXSc/wI1b5NLMr/0tYZX6e8Az/sk5IfjfL/FlWeVOrPcF1K7LXrroyZ/f7bStuwX1z+wiAMxkFAMIoABBGAYAwCgCEUQAgjAIAYRQACKMAQBgFAOK0MxeNVearr7vPsvwnNV/7AX7QPLF7lfl5wjP8qPXaD/AnaM9WtC7n1p9mm8ezX7/98X++NwUAwigAEEYBgDAKAIRRACCMAgBhFAAIowBAGAUAwigAEEYBgHgTt4/Otcr8POEZfsA6uX+e3P8GzDK/TszPsrvNvxnrtR/gX/Yiu5Xds8w39hO7j/CmAEAYBQDCKAAQRgGAMAoAhFEAIIwCAGEUAAijAEAYBQDCKAAQp90+WmcV12YXX0VzWd1ZZXqeVn/uf+fPa3vtB/indWL3PLH7ZFuRnSc9w4+4FNmvJ/z53hQACKMAQBgFAMIoABBGAYAwCgCEUQAgjAIAYRQACKMAQJx35mI14eezHqM+AbA31Sfef5jlF++3U57iH9aZ5W/HfO0HeIPuq8tvRXZ21fU/w9l8oPqF1eVX+fut/Zn/0bwpABBGAYAwCgCEUQAgjAIAYRQACKMAQBgFAMIoABBGAYAwCgDEabeP9iK7yu7qpsnsurcT7xntRXaV3W1+Ftm97L4X2WvZPct8Y19d/l6l2/JZZMvu5m5P8RhjjLFvxz+wrl33tfzbX0V2q5o77Vklt48AeDOMAgBhFAAIowBAGAUAwigAEEYBgDAKAIRRACCMAgBx2pmL5qvd7dfAO2V5Ez/xJEZrnti9TuyeJ3avMn+b5Qf2Iru2qnpW8Vl1j9vq8oX9fjy7bV33vJT56m5Jk+3i7dmK5+cu/0fzpgBAGAUAwigAEEYBgDAKAIRRACCMAgBhFAAIowBAGAUAwigAEKfdPhrF/Y773lVfti5fmet4toh+756Ho1tZvU7Mb2X3pcyfZZb5x7Z/O55d5d/QbMJblR7rejx7r5rH2Fb5gULzv2at7G7OKt3vXfdr86YAQBgFAMIoABBGAYAwCgCEUQAgjAIAYRQACKMAQBgFAOK0MxfFlYv6zMVetG/jQ1feWF28vEZQWedV/2XMn7R7rFXFi2sr49pVn6v87zzz/4p9Hf8ddNtPe4xTeFMAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAhFEAIIwCAGEUAIjTbh812tsg1/vx7Lx03fO08BhjL7JbV726+JuxZpefq+juqmvztHB35qe5ZVRru9cJz/CjVhHdm2ttYzzdj2e/ddWvzpsCAGEUAAijAEAYBQDCKAAQRgGAMAoAhFEAIIwCAGEUAIg3ceai/Rp48xXzy9aVz/mhSK+qu3Ev7wtcyv7mvERzWqI1T/xA292cljhbdbpinfQQP9J95g+x7G7iX+5VdfU76GfjTQGAMAoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAhFEAIB5eXl5eDgUfHs5+lsM+FOeJPl277k+X49nuTtIYc6vinbZ8Ho+urrnKb2X3mzKL7Oqqm/jsqjvtLaMmf2L1GGPc7sfvnv321HU/l/fa3oojv+69KQAQRgGAMAoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAxE95+6jR3EkaY4zPj8ezj1vXPbbjDzNn2V1axR9wKx/mcjue3fdVdTf5tVXVY5tdflyOf+BaXihaXbzrLo4OXerbR0W0rG5uGY0xxufb8ey3b92z/KzcPgKgYhQACKMAQBgFAMIoABBGAYAwCgCEUQAgjAIAYRQAiF9e+wHO9tx9M358eSrCj1334zj+MGt29znm1j3LXMezl6563Lfj2X3vum978TO8dd3tz/B6P569/zar7ubkxlY1jzH2Iru66uZ0RfN3OcYYn5+q+PhW/r/Pd94UAAijAEAYBQDCKAAQRgGAMAoAhFEAIIwCAGEUAAijAEAYBQDi3d8+ajX3Uqo7SWOMcS2isz3cUt5KKo4fbWOW3cezt8eu+/G2Dme/FLemxhj1nZ9Lkd266s5q88c/sFb3M7zdj2fdMnqbvCkAEEYBgDAKAIRRACCMAgBhFAAIowBAGAUAwigAEEYBgHh4eXl5ORR8eDj7Wd69D8UlisvWdX+6dPnL5fjDbLPrXsVRh3tXPdbaD2e/lOWfVpe/fpqHs1v9Q1zHs3uRHWPs+/F7Ebe9qq5Ovzhb8ec78uvemwIAYRQACKMAQBgFAMIoABBGAYAwCgCEUQAgjAIAYRQACKMAQLh99E58/Njlm9tKny9d99rOu6s013b8OdruuboPjPIPaBQ3nppbRmOM8eV49Xh6qqrHs3tGb5rbRwBUjAIAYRQACKMAQBgFAMIoABBGAYAwCgCEUQAgjAIA4cwF/69fj1+tGGOM8Xg9nr0WJzHGGGMruvs7F6vLN8rqe3G64vNT1/31712e98OZCwAqRgGAMAoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAxOHbRwC8f94UAAijAEAYBQDCKAAQRgGAMAoAhFEAIIwCAGEUAIj/AjAabpd54eUgAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGFCAYAAAASI+9IAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAOSklEQVR4nO3cwXUkx5UF0IAO9wgPOjxgejDlgcoEmAAT2gSYAHlQ40HJg6QH2R4ELIAWoP5CM4t4FJNEN+9dv/qdLIB4pxb1Ht7f398bALTW/vZnPwAAn4dSAKAoBQCKUgCgKAUAilIAoCgFAIpSAKD8tBp8eHg48zkAONnKd5V9UgCgKAUAilIAoCgFAIpSAKAoBQCKUgCgKAUAilIAoCgFAIpSAKAsbx/Bqi9J+Ofs9qU/Lmd7z26fac7z8vvxlt0Osm/ZaX4APikAUJQCAEUpAFCUAgBFKQBQlAIARSkAUJQCAEUpAFCUAgDFzMUPYn384UP/n/Xs08iuXy/r2ZFuUYwgH55uM31BdDyMr+fnkZ3eg/x9z3Yubvf17DcTGp+STwoAFKUAQFEKABSlAEBRCgAUpQBAUQoAFKUAQFEKABSlAEBRCgCUh/f39/el4MPD2c/yw3sMJoSuW3b76ZrtE20jCI9LdLuNLctHktvjpGf4I+xBdman5309eyTP0dqxz+Xsyy06He8w2Vb6v1b+3PukAEBRCgAUpQBAUQoAFKUAQFEKABSlAEBRCgAUpQBAUQoAFDMX/4Uv2bJE+/q0/oLrJbvdtvAF/RqER3Y7cuLtGeb7Cc/wb0f6gnlC8kMfySuO7Pi8rWeDSYzWWtv3Pco/v67vXPzyLTr93TJzAUBEKQBQlAIARSkAUJQCAEUpAFCUAgBFKQBQlAIARSkAUJQCAMX20X/4+ct69uUpu71dguPjmh3vlyi+B0M/44hORxNCc6S3k+vfr9c5l7P9yG6PILtt68+Ru2fx/ZbF73M5+xTsJLXW2rfvdCvJ9hEAEaUAQFEKABSlAEBRCgAUpQBAUQoAFKUAQFEKABSlAED54Wcuvjxm+den9ex2DWYrPl6QhLPbofutL2ePcYtuX/t1OdtHdLrNYOZiPfnr7Znl97b+gi19muBh9vD0JYtH9mMuZ0d4u489e8FxW47ebzM6ncxivGULGqcycwFARCkAUJQCAEUpAFCUAgBFKQBQlAIARSkAUJQCAEUpAFCUAgDlu9w+egz2jF6u2fjR9dqXs3O7RLdb39ZvZ5fbOLL80fblbLI31FprY2xRPtFPu9za3NMXBNntvNtB9ENfj47w+LGvv2DfbtHtbY4oP5J35rhHt5+D7aN//DM6fSrbRwBElAIARSkAUJQCAEUpAFCUAgBFKQBQlAIARSkAUJQCAOWnP/sBfovrFmQv4fGx/oLee3R6Bl+7H9Hl/AUj2DqY4fEe/HeuJ/N8eruP8AWJexafcz0brKd85EcSDh6kZXMr80geJJytaK0lqxg9/OFft1+Ws7c9Ot3e1hc0TuGTAgBFKQBQlAIARSkAUJQCAEUpAFCUAgBFKQBQlAIARSkAUJQCAOVTbB99eczyT1vwgnCfKJgEygZqWvoo2e3swTOzHafd7ic+d3w5fMEMsseW3U52fnp2+lxjPbqNHp2eM8tHG1zh34kx1v8GXUY2ZvS/67NKp/BJAYCiFAAoSgGAohQAKEoBgKIUAChKAYCiFAAoSgGAohQAKEoBgPIpto+uI8tvWxAe4fFgL+UIL49oK6lnx3tyO9vtGUd0OjNm+IJ+wkP824zSPchuIzodPUk4wZW9g+HtZPvodMEb08Ptoz7Ws8lOUmuttV+yraTfm08KABSlAEBRCgAUpQBAUQoAFKUAQFEKABSlAEBRCgAUpQBAOW3mIvli9zX9Gnj0lfQk26Kv9YffjG/JswSP8atLlO7ttpw9juh0m/t6dnvKbqdzHt+rHoWjdPstv12nmFm8p3se0bt43u1L+xZdfg2yZwxi+KQAQFEKABSlAEBRCgAUpQBAUQoAFKUAQFEKABSlAEBRCgAUpQBAOW37qH8Jslt8fTk5P8vOS2tttuf1cM9u93jjaazfHl+j08lW0rxHp6Pflfvo0e1LlP48jvB3vN/W832PTrf2tH77N4yHheZ6MtxV6kF2bNHp6D1/y2aVlvikAEBRCgAUpQBAUQoAFKUAQFEKABSlAEBRCgAUpQBAUQoAlNNmLkaQ7f3tvOPhBMAMvsA+5kt0+wi+1j+iy63t4czF6HM52/vX6PblaT0/9+h09MZcTp442YPsdtIztNayB2mt3e7BC8LbT2MsZ+d1Rrd7lG7tCLIjvJ3o/THKX4K/h/8wcwHAmZQCAEUpAFCUAgBFKQBQlAIARSkAUJQCAEUpAFCUAgBFKQBQzts+6kk62wZpx1yO9mBvqLXW9v6ynB3hGst9D/L9iG4f0dJLa5dtW8+myz3zshzt7ZbdTt7DkZ2ewe9Va9m2zjai0y1Z+pnhxlNPfm+37PZxWc+O7HRsnHp9nne6n3d6hU8KABSlAEBRCgAUpQBAUQoAFKUAQFEKABSlAEBRCgAUpQBAOW3mIvmqdhBtrWVfMN/TL7vv6/l7z07f+76cPbLT7SmcoriE9xN729ez69EPx/oL0omTeUTx1o71+/vX7PQINiC27HRrl74cTd/DEaVPNpNsEg5vh/p5p5f4pABAUQoAFKUAQFEKABSlAEBRCgAUpQBAUQoAFKUAQFEKABSlAEA5b/tonhJtrbXWkxeEx/t22ul2nWM5u6WbM+unW2vZttIMl5j2/bqc7ftLdLuF70vilq7O9GM9+rpFp7fnuR4e0enodyX6f43/3zw5/zvzSQGAohQAKEoBgKIUAChKAYCiFAAoSgGAohQAKEoBgKIUAChKAYBy2vbRPOtway2ZqOkzCLfW+ljPBtHWWmszePD1ZB2PjOAfmPG4zlzPPl2j09u4LWf36HJrL/csvwcjQiMdEepZ/JOc/uuYSfQtOn0Pbp/BJwUAilIAoCgFAIpSAKAoBQCKUgCgKAUAilIAoCgFAIpSAKCcN3Mxk3B8PUge0eWtvQS3n6PbPcgeR3S69eR4yyY6wtNtBhsalyM83q7Lya3dstOXLJ6secQ/oGxH4a9hpvngBdEfrOxR9ux0O8L8780nBQCKUgCgKAUAilIAoCgFAIpSAKAoBQCKUgCgKAUAilIAoCgFAMpp20fJfscx36Lb23xczl7C7aPW7svJ3rbo8gzGdYL5oF9vZ45g6yV9lmxz6J4dj/ZsstPxyFOaT8wg28+7vYenexKe2e0R7hOdOgoVnL7v2em37M/h784nBQCKUgCgKAUAilIAoCgFAIpSAKAoBQCKUgCgKAUAilIAoJw2czHPOpzqM4rP+7F+ut+j2/vYlrOj9+h2T9/xJH7s0ekeTFfMeUS3ZzCkMKLLZ5tZvJ/xDB9mkN2ScGvRc6enY8E/kC5o3INpnnTm4s/mkwIARSkAUJQCAEUpAFCUAgBFKQBQlAIARSkAUJQCAEUpAFCUAgDltO2jtj4N0sL5mzbHejacEIomZ+a8R7eP4xKkt+j2CDeesm2lEd2eQXZPwq21I3jB9ZLd7rOHL8jikRlke3Y6ioe3P8np2Ay2jFpr7X5fz37LTv/pfFIAoCgFAIpSAKAoBQCKUgCgKAUAilIAoCgFAIpSAKAoBQDKaTMXyTe77+HtbQbhJBvq6bREsOcxwxGAI9n+aK3NbT2bTWK01ufzcvYyX6Lbt2Nfzt5ndLpto0f5Ed5PnHj63HmJmWSTcJ5P4vsenW63I8t/T3xSAKAoBQCKUgCgKAUAilIAoCgFAIpSAKAoBQCKUgCgKAUAilIAoJy2fZQIJoE+8tv6slKfj9nxYLxlHj07HWwl9S27PUYUb8cMwkm2tTaDraR9C2/fg/AtWeBqbb/MKD/Cn1HiCLI9Cbezt4/mOdk83u77+s//6z27/Zb9an1XfFIAoCgFAIpSAKAoBQCKUgCgKAUAilIAoCgFAIpSAKAoBQDKp5i5uB9Z/hLk+wi/j97Xo8dITx/rt8MJjXRxo+3r0efw9j5fl7NfX4MHaa31fuK+QLT90Vo7ceZiC7Kzn/QQrcXvyQy2KNLZij2YrWgtm6749gPPVqR8UgCgKAUAilIAoCgFAIpSAKAoBQCKUgCgKAUAilIAoCgFAIpSAKB8iu2jdHfkvq9n03maJD6y0232fTn70p6z27cs/zzGcva6R6dbC/472xb+8I8gO7PTLdzWOYJ/YFx79iyB+PKxHt3DgaKR3D7CLaNbFLdn9Bv5pABAUQoAFKUAQFEKABSlAEBRCgAUpQBAUQoAFKUAQFEKABSlAED5FNtHqduxnt2CbGutXbb1bM9OtzGP5ezz6yW6vW8jy/e5nJ2XI7rd9iA/H7Pb/bxBmxnmk12gZBPo4wV9Pbv+GL/mj+XoEe4TJf9v3m7RaVtGfxCfFAAoSgGAohQAKEoBgKIUAChKAYCiFAAoSgGAohQAKEoBgPLw/v7+vhR8eDj7WU7xJVxR+Hpdz15Gdru19Ye59J6dft2i+LdgG+Hn7EnaFmw6XLdv0e1goeFTGdfsF3FLwjM6HU1XvNyy28nMxZvZij/cyp97nxQAKEoBgKIUAChKAYCiFAAoSgGAohQAKEoBgKIUAChKAYCiFAAoP/z2UerLl/Xs83baY7SXe5b/duaOTLgf9fexnr1s2e2exT+NrWf5Ptaz9z27/Xpfz/4zm6bik7N9BEBEKQBQlAIARSkAUJQCAEUpAFCUAgBFKQBQlAIARSkAUMxc/Bcew/mHxNuZsxWfSPoejiDbe3Z7ZvFP4xdTFCwycwFARCkAUJQCAEUpAFCUAgBFKQBQlAIARSkAUJQCAEUpAFCUAgDF9hHAX4TtIwAiSgGAohQAKEoBgKIUAChKAYCiFAAoSgGAohQAKEoBgKIUAChKAYCiFAAoSgGAohQAKEoBgKIUAChKAYCiFAAoSgGAohQAKEoBgPLTavD9/f3M5wDgE/BJAYCiFAAoSgGAohQAKEoBgKIUAChKAYCiFAAoSgGA8i9Rv76kp21seAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGFCAYAAAASI+9IAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAANZElEQVR4nO3cwZUcR3oE4IQe70gPUB6oTYAJYwJMoAk0ASZwPRgTmh70elDwIGHB6MBlrJ4OUgWo5AyI7ztHBwsYouPVYf53Ly8vLwMAxhj/8doPAMDbYRQACKMAQBgFAMIoABBGAYAwCgCEUQAgfroafPfu3c7nAGCzK7+r7E0BgDAKAIRRACCMAgBhFAAIowBAGAUAwigAEEYBgDAKAIRRACAu3z7ix/W+zM/iA8fRtR+zeI4i+y3WKrJl96PpPr9W3V+7OD8YbwoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAhFEAIN69vLy8XAq+e7f7WfgT3n/o8k/FeYmnj1337biene0tiqMp76pra1t4jPN6/v7oqu/FDY3ne3cT44sTGm/ala97bwoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAhFEAIIwCAOH20V/o/fVzQ+PpVoTHGJ+eume53eb18HHrymeTL7vHUebfinNffrXV9+vRx/XsGGN8fr6efX50h5K+uqv0p7l9BEDFKAAQRgGAMAoAhFEAIIwCAGEUAAijAEAYBQDCKAAQzlz8CR+6SxTjl6frH3h66rrH7WOXn81/4Oi6x7ycPIvsGOWTrKq6tub17Hx03edc17uP69kxxpjVCY3nqnsUZzGer0fHGGP88tzdufjypev/EThzAUDFKAAQRgGAMAoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAcPvof/jPD9ezn5+67ttTUX505Wveqnx/z2iPWd4++l6t1eWfz/Ny9tZVj9ttFulVtj+K6K9d831V+Z9/vX4r6Z8/yJ0kt48AqBgFAMIoABBGAYAwCgCEUQAgjAIAYRQACKMAQBgFAOJvf+biw/su/+un6x+4Pc2u/PbUhKvqVaXHOIvzEreye695Obna6voD+8rPIl5drRhjrCL/eBQPMsY4iu55nFX33HgW41NxEmOMMb58p2cxnLkAoGIUAAijAEAYBQDCKAAQRgGAMAoAhFEAIIwCAGEUAAijAEB8l7eP3hf3jD5/7I4fPX0qwtUtozHGvF2Orq55792eObv4lof4BquMl/lHkT3K7lnk18eu+ygOK51ddWUeZX6d3Qcez5ejz8/dMaOfr1ePr91Zpa3cPgKgYhQACKMAQBgFAMIoABBGAYAwCgCEUQAgjAIAYRQAiJ9e+wG+xdOtyD6V5cf1D5zl+Yfm7sJRNp+zyzfxsrqytn+gqJ5d/rbjIf5lFdn5OKvu8zYvZ4+quXzusrv+4R+3y9Gnj133/bx+u+Ifv1XVr86bAgBhFAAIowBAGAUAwigAEEYBgDAKAIRRACCMAgBhFAAIowBAvInbRx/ed/lPH4twe5+oiB9rbetu77zM8pJMk17lszTpo72AU8bfSHVdPjceeXo73XPTU/xuFf/253FU3Z9u/7ycvT+q6vHl+lmlLbwpABBGAYAwCgCEUQAgjAIAYRQACKMAQBgFAMIoABBGAYAwCgDEm7h99PHW5W/H9WNJzf2TMbo7P13zGLO5lTSPrnvnPZuyuomvsry98bTXKpKzat7582ysMj+r8q69vcHVxMvmcbtd/w5qv9/+8dvrHj/ypgBAGAUAwigAEEYBgDAKAIRRACCMAgBhFAAIowBAGAUAYtuZi/fXfwt8fCrOVowxxmhOV5RnLo7iV+/7ExpFtgmPMY7Z5asTDWX30cVLa0Pyd7PMn0W2/XneivMfZXX152yyY4z+YQqzzD+KhznKf8vN98rTcf05xhjj+bfr2R0HMbwpABBGAYAwCgCEUQAgjAIAYRQACKMAQBgFAMIoABBGAYAwCgDEtttHt3n9ntFxK8tnE137yjceejnGo/zE6uLncT1b3m7pzDfUvMr+6/+FY+P/K9ef4g9rY3vT3TnLI1y3qruqrv5WjqO7UFR8dY6vG44feVMAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAhFEAIIwCALHtzMUxm9+/Ln6ve4ytv6V/zqK8eY4xxjG7fGWV+Xlej5bnBZp4f4bkura5+dGPMcZsPjC77u7hy/KN/4/v/Md5jLPKr6p7nzm7fPM98cWZCwB2MgoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAhFEAILbdPvpurdd+gG9zlvnm5tBsu69Xvylr+weum9Xf+ll1r9U8RxGu7ex+O2Z52625G/fbl/Zp/m/eFAAIowBAGAUAwigAEEYBgDAKAIRRACCMAgBhFAAIowBAvIkzF3Nn+eric+54iP2O1eXX9/oHfUPm1va1rXlua+6sMj83PMMfzjJ/bHiGP7z2P01vCgCEUQAgjAIAYRQACKMAQBgFAMIoABBGAYAwCgCEUQAgjAIA8SZuH60yP99Id229neq54Rn+sIrs3PQMu7v58+bq8mW8+kB7O6yxsXoLbwoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAhFEAIIwCALHt9tGau5o7c2P32pg/yu5Z5nceKCrjb8bamJ9ld5vfZr32A/zbLPPVd9BaZXmZr7r3VV/hTQGAMAoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAxLYzF6/9q9qxyvzc8Az/sorsY9Mz/OHWhNeeZxijP4cy146n+Ff3xvwqu38EZ5mfbX6VH9ikvYjxKPP/37wpABBGAYAwCgCEUQAgjAIAYRQACKMAQBgFAMIoABBGAYAwCgDEtttHzb2P9jbIrNJd+Srau+fo8ufG524d25r7+zRNfHbVW83XfoC/yCqyx6ZneGse62uZ3/McV3lTACCMAgBhFAAIowBAGAUAwigAEEYBgDAKAIRRACCMAgCx7czFua5nV/lr4DvNNYtw2V1kb2X5WaXHuDXhWZavLdFvyu/snpuyb8n52g/w37QnUXbe2mm+3+6Pqnp8feWvQ28KAIRRACCMAgBhFAAIowBAGAUAwigAEEYBgDAKAIRRACCMAgCx7fZRc87ocXbdR5GfR9ddHcCZZXdhlfm54Rli7YvPrnrrn7M8f9P9tcyuu4xvM7/T7t+ty8nmltEYYzzu17/gnh9d92vzpgBAGAUAwigAEEYBgDAKAIRRACCMAgBhFAAIowBAGAUAYtuZi+LKRX3m4mNxQ2Ou9135LB5mHV31LLJVc38Wo8nPsnuntbG7+fmMsfvv5Xr7Kv9WrjePMds/ZXsrpOrel3+czTfWGL88rme/dtWvzpsCAGEUAAijAEAYBQDCKAAQRgGAMAoAhFEAIIwCAGEUAAijAEBsu33UuK8u/1TkZ9ld3b8pu9/SEaG5sXsV2Z0/n7L6G/5Omk+sqrm9Z9SZl5M77yrVP6HyrtLz4/rRoc/37lG+fOny3xNvCgCEUQAgjAIAYRQACKMAQBgFAMIoABBGAYAwCgCEUQAg3sSZi1X+yviv5/XsL/P6r7qPMcZc74tw8SBjjHHO69HjevZbrCI7y+6zyD7K8qciW1Z/g7Uh2Zv1J9b17uvRunucXXl55aI6n3OeXfffmTcFAMIoABBGAYAwCgCEUQAgjAIAYRQACKMAQBgFAMIoABBGAYB4E7ePuutEY9zvRbbsfiqeZh7FnaQxqkMyR3En6fcPdPGzSs8yvS5n23s21aO03a25JbrdY13P3tbZlRfd9S2jR/dNcX++nv3afgn9jXlTACCMAgBhFAAIowBAGAUAwigAEEYBgDAKAIRRACCMAgBhFACIN3H7qNXcKfl877rnvJ79WGTHKO/fzLMrX0cVb/6c9RGhIn7rmitrbiwfb+ie0eri5/28nH3Mrnus4h/no6tu/y1/cc/om3hTACCMAgBhFAAIowBAGAUAwigAEEYBgDAKAIRRACCMAgDxXZ65aLS/6v75uUl35R9v7y9nZ9U8xhhnlZ7FnYt7+TSrepCqehwbu9fq8judTbh88Ob8x1k9yBjrfj37XHY35234dt4UAAijAEAYBQDCKAAQRgGAMAoAhFEAIIwCAGEUAAijAEAYBQDi3cvLy8ul4Lt3u5/lu/Ph+imjMcYYP3+8nv1467rn0T1McfporHlU3U35nKvrbpTVZ1k/N2XHGONRZM/HWXU394zOe3dw6PlxPeuW0V/vyte9NwUAwigAEEYBgDAKAIRRACCMAgBhFAAIowBAGAUAwigAED+99gN8z76Uv6b/y/169lNXXefHcT062wMQzVmM1VU31uzyx9rxFL9rq4/H9U80Zyt+/8D1/3GbsxVjOF3xd+BNAYAwCgCEUQAgjAIAYRQACKMAQBgFAMIoABBGAYAwCgCEUQAg3D76CzV3YX69d91zdEdnnqru91X3aG4lHbPsvp6fq6wudfVnlb5X1d3P/vPz9axbRj8ebwoAhFEAIIwCAGEUAAijAEAYBQDCKAAQRgGAMAoAhFEAIJy5eKPa8wKf711+FdlPR/cw83H9LMYsj0U0zvqERmmty9F7eYricb+evT9X1U5X8L/ypgBAGAUAwigAEEYBgDAKAIRRACCMAgBhFAAIowBAGAUAwigAEO9eXl5eXvshAHgbvCkAEEYBgDAKAIRRACCMAgBhFAAIowBAGAUAwigAEP8Fg1VpgWeUBwcAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGFCAYAAAASI+9IAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAOhElEQVR4nO3cwXEcV5YF0MeJ3vN7oDQhPejyQPBgaIJMkAkwQe0BTCh5kO1BwoNPCzAL9rxQzOpfjFIoUuesb91IFCHdyAXep7e3t7cCgKr6r49+AAAeh1EAoBkFAJpRAKAZBQCaUQCgGQUAmlEAoP1jNfjp06crnwOAi638rbI3BQCaUQCgGQUAmlEAoBkFAJpRAKAZBQCaUQCgGQUAmlEAoBkFANry7SN+LJ8vC1eNILtvYfe2/jDJc7zHnNdkq6qO8+t693q0qqrC+N9D+Dv+I3+J3hQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYD26e3t7W0p+OnT1c/CH/wU5m//zPJPwbmI2rLu5HTF2EZWPoJ8kq2q+AedZ5CdYfV69xE8RlXV/b6efTmyew6vP/D5hx/Byv/uvSkA0IwCAM0oANCMAgDNKADQjAIAzSgA0IwCAM0oANCMAgDNKADQ3D76C/0cnBt6+hKEq+rpNrKHGft6NojmH0jLx0XZqx1hfgbRsPu8B9HgOarqeb06vqv01V2l/ze3jwCIGAUAmlEAoBkFAJpRAKAZBQCaUQCgGQUAmlEAoBkFAJozF/9XcF3i5y2rfg5OV4zbLSvf9ixfSf+Ims+oOevO0qEzzG9JeEbVZ5Ddwu6ofd6z6uNlOfoSVv/6kt25eH3N+v8OnLkAIGIUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYBmFABoRgGA9sPfPvopzN9+Xr9P9PwUlu+39eyWlo/L8jNsTqS3jy41H6f6GOufuB1h+Z6Ez7A8yB+/Rc3HfUb5X35bv5X077/JnSS3jwCIGAUAmlEAoBkFAJpRAKAZBQCaUQCgGQUAmlEAoBkFANoPf+biv/+5fraiqur5SxC+JeGqc+zr2SOqrnneo/zYx3L2Vreou7b16EzPXMws/jjmw1SfQf4My/cgu+1H1F3HPYyfy9nkJEbV93sWw5kLACJGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaP/46Ad4j59/Ws/+egvLt305OscWVc/7XM/Ol6h71Bnl93lbf5Z9hs+ybs6sezuD7uRBqrIHT83r4iOrrjHW29Pu5MFnciSrqsZ2i/L7/rKc/SWrrl9e1m8lfc3OKn04bwoANKMAQDMKADSjAEAzCgA0owBAMwoANKMAQDMKADSjAEB7iDMXn8N88ifpY9+z8m1b7w7OVlRVdIjiaf0xvgme+5uxnExPUSTpsf4Y32xh/lGMK7tnFN8ueYjru+PvMDhZ83S7R9X3cz37r9+j6g/nTQGAZhQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYD2ELePxk9Zft+Da0nhXZgkPfYz6n5KjreMPepOzQvTI0qHgvIg+nBG+J1fZfvoB/ijGeaTw1pbkK2qL7evy9n7EVXX63r1JbwpANCMAgDNKADQjAIAzSgA0IwCAM0oANCMAgDNKADQjAIAzSgA0B7i9tHTFtwyqoqO2szwAs4MDqxkze/5wHVGlJ6XpUfU/L5PXGcGyRE1P8rto0vNyz+w3jy2KL9vczl727NjRv/6PYr/6bwpANCMAgDNKADQjAIAzSgA0IwCAM0oANCMAgDNKADQjAIA7bIzFz8Flytut7A8+JP0OcLuuR49wu5trpePEZY/0FmEM8ju6T/QmEE4yVYdUTp7lC167uw73KLm9BNnVj2zeOIM88ElitxYj6ZnfF5+Xz+LkR3QWONNAYBmFABoRgGAZhQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgHbZ7aNtrGf3IPut/JrnqKqq4D7ROfeoeoyn4Dmi6oov94wkvEfVe/26nI3vRwXZsLr28EufyX2qrLrG/VzOvqxH/2MuJ7cvWfMef+vXOYOfcwT/3f/nE8vJbXvNmoNTSV8vOH7kTQGAZhQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGgPceYiNpNoEK6qc9yWs6O+RN0z+NP4X4+ouqruUfo2vixn9y2qrjrXoyMJV2W/WEH0PUbwu7We/Oblvv6J53t2RmFu63cUnm9b1B3/rkTmZektvyuzLLmGUpX9ir86cwHAlYwCAM0oANCMAgDNKADQjAIAzSgA0IwCAM0oANCMAgDNKADQLrt9lBjpB+Zcjp73rP15vy1nv+xRdW3nWM7OcUTdt/olyj/t69kZfN9V2b2peY+q676tdz/dsu7UOKN0WD6Xo1twy6iq6inIhqePon/74Ef8ls/itV3aPoPm7N9nG+sHjX7Pzl4t8aYAQDMKADSjAEAzCgA0owBAMwoANKMAQDMKADSjAEAzCgC0hzhzMcP8CD5xbnvYfVvPjqi6kkf55QzCVbVvUTwywvysfT2737Py4GzJ3LLqLcwndxSS39mqqqen9fLbLeve0l/cBzE++gH+Ih/9z+NNAYBmFABoRgGAZhQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgHbZ7aN5Wbiq5m05OrYvUfVti+KRM8imd3vO8EscwSWZER5j2caX9Wx6cGi7B+GZdT+QsQXZC68CzTA/kg+MsDzpDuMjq47MC7uv4E0BgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYBmFABoRgGA9hi3j8Z13WPbo+59rGePqDlziz8x/vRn+F8zzI/ontEetr+sR2dYfaEzzG8XPMN7RLeMUld2V/jsMwm/I/8Y1Uu8KQDQjAIAzSgA0IwCAM0oANCMAgDNKADQjAIAzSgA0IwCAO2yMxfnXM+mf9Y9tqf17jGi7iN5juSHrOyERnq2YkbpqhncABhzZOXRdx52f6e2+dFP8D4zzI8Lu9P8o3znc36N8vfzmudY5U0BgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYBmFABoRgGAdtnto+SeURCtqqqx35az24XXW7bwrlIF94bSe1DpB2bwc64nc/M8o/wx9+XsLbpkVfkPeibRGVVv51jOzi2qzk5TPZIZ5kfSnZUn8eOMqus1O5X0p/OmAEAzCgA0owBAMwoANKMAQDMKADSjAEAzCgA0owBAMwoAtMvOXFTwp9rz3KLqUfcgu0fdyd/Gz+BsxbcPZPHIyOLzomxV9ijj3KPuOX5bzr7cZ9RdZ5afIwlH1dEH4msr2/oHxp6VJ/H0P5+g+psLb+3Muf4/uPuRdX80bwoANKMAQDMKADSjAEAzCgA0owBAMwoANKMAQDMKADSjAEAzCgC0y24fBaeP6j7OqPuW3LS5jag7uWcTnz4KbrEcwXNUVW1ZvPbkLkxwKyc1ty3K34JbSfd5ZM8Spd9xiycwg/+C5pl1JyeetvT2UZROzcvyyX8OVVXHuZ59CbKPwJsCAM0oANCMAgDNKADQjAIAzSgA0IwCAM0oANCMAgDNKADQLjtzkdg++gH+YAR/Gn+E3ee5LWdv2/pzVFWN8BTFTE4GJHcRqmomByDCEycz+DnHHlWnj1JzC8L3rHtE6c9Z91hvf4qaK7tEkd6WCM9cJPXzTA7zVD3f17Nfs+oP500BgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYBmFABoRgGA9hC3j+YZ5m/rHxjjyMrv23o2uCFTVbUH94zGWM9WVXoWJnKG5cn9qDFH1L0FB4q2sUXdta13V2Vf+T1qDo00P9ezQTQXlqfx4J7R85F1//s1y39PvCkA0IwCAM0oANCMAgDNKADQjAIAzSgA0IwCAM0oANCMAgDtIc5cxM7flqPzZUbVxwjCWXVtQfcZdo86sg+ce9CdGSPo3u5Z+XkG4Zl1jzAe1j+K6JrHPsP2sR4Nq+dcP1tRVXWc69mXI6r+oXlTAKAZBQCaUQCgGQUAmlEAoBkFAJpRAKAZBQCaUQCgGQUAmlEAoH2Xt49GcF9l1j3qviXxLaquGmM9GlbXnGH8vpwd4SGmsR/r4SD6zZl+YN28Lr+H1cf4vN69Zd3JDa74Owk+MM+s+R7mf72vZ79mZ5V+aN4UAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaA9x+2jM9BPrHxhpc/KBI+sOTh+948Gz+BF86eeZdSe3kkb44Ldb8iBRdS7o3/arHiK8ZRQKT2rVnOtHhO5H1v18z/Kv7hm9izcFAJpRAKAZBQCaUQCgGQUAmlEAoBkFAJpRAKAZBQCaUQCgXXbm4nOUzv4efc6sPbP+LC9b1ryf691jZD/jCG86jOCex0jvkAS3EcLmOoIP7GF3bF5XvSXhmXUnpyiOM+v+7VjP3sPur85W/CW8KQDQjAIAzSgA0IwCAM0oANCMAgDNKADQjAIAzSgA0IwCAM0oANAuu32UnCk5Z9Y9g/YtvMJUwc2hfWbVSTw4H/Qtv2X5W9B/bmH5eS5H07NKWxbPzDB+BtmR3vdazwZfd1VVvQT59PbRq/tE3z1vCgA0owBAMwoANKMAQDMKADSjAEAzCgA0owBAMwoANKMAQLvszEUiPXMRnSMYYXeQ39Pu+GHWzTB/jLGc3cKbG2fSvUXVFVTHt0LScxHHXL/pcH/Juu9zPRs8RlVlJ2j4+/GmAEAzCgA0owBAMwoANKMAQDMKADSjAEAzCgA0owBAMwoANKMAQHuI20fHvC4/wu6RhLes+0ojzG9BdkYHh6q24BJT1lzRPaP0ltH9nl0Feg76Xx0c4jvhTQGAZhQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGgPceZihicAjnM9u2/X3RcY83P6gSQcdmf5Lcjeo+aqW/Ls84y6z2P93/P5HlXXS/Yo9dXpCn5A3hQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaEYBgGYUAGhGAYBmFABon97e3t4++iEAeAzeFABoRgGAZhQAaEYBgGYUAGhGAYBmFABoRgGAZhQAaP8DPI7KJyTS+HgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i in range(4):\n",
    "    generated_image = generate_image(model, image_size=image_size, num_classes=num_classes, temperature=0.6)\n",
    "    generated_image = generated_image.cpu()\n",
    "    show_tensor_image(generated_image[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "Autoregression is a relatively simple approach to generative tasks as image creation, but might not be the most suitable. Generating a high resolution image with this model would require height*width  number of iterations (for a 1080p image that would be close to 1 million) in comparison with some more sophisticated models like diffusion models which need only around 100-1000 steps.\n",
    "\n",
    "The dataset used for training was very small, especially after I have narrowed it down only on face-like emojis, but we can still see that the resulting images resemble real emojis at least a little bit.\n",
    "\n",
    "With a larger dataset (maybe including other operating system emojis), with longer training and maybe a little playing around with the parameters and the model hidden layer structure, we could have achieved better results. But this is more than enough to demonstrate the functionality of autoregressive models like PixelCNN."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}