Quanvolutional_Neural_Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring the application of quantum circuits in convolutional neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial will guide you through implementing a hybrid quantum-classical convolutional neural network using Tequila along with other packages such as Tensorflow. We will then train the model on the MNIST dataset, which contains images of handwritten numbers classifed according to the digit they represent. Finally, we will compare the accuracy and loss of models with and without the quantum preprocessing.\n",
    "\n",
    "Inspriation for this tutorial comes from [Pennylane: Quanvolutional Neural Networks](https://pennylane.ai/qml/demos/tutorial_quanvolution.html). We will similarly follow the method proposed in the reference paper used for this tutorial, [Henderson at al (2020)](https://doi.org/10.1007/s42484-020-00012-y)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convolutional Neural Nets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An excellent high-level explanation of convolutional neural networks can be found [here](https://www.youtube.com/watch?v=FmpDIaiMIeA). Alternatively, an excellent written explanation can be found [here](http://neuralnetworksanddeeplearning.com/chap6.html) and for more information, the wikipedia article can be found [here](https://en.wikipedia.org/wiki/Convolutional_neural_network).\n",
    "\n",
    "In summary, a convolutional neural network includes preprocessing layers prior to optimisation layers so that features in the input (which are often images) are extracted and amplified. The result is a model with greater predictive power. This processing also improves classification of images as it extracts features even if they are translocated between images. This means that searching for a particular pixel distribution (for example the shape of a curve or line may be a useful feature when classifying digits) is not dependant on the distribution being in an identical location in each image where it is present. The convolutional process extracts this information even if it is slightly rotated or translocated.\n",
    "\n",
    "The implementation of the convolutional layer involves a grid for each feature being passed over the entire image. At each location, a score is calculated representing how well the feature and the section of the image match, and this becomes the value of the corresponding pixel in the output image. As a guide, a large score represents a close match, generally meaning that the feature is present at that location of the image, and a low score represents the absence of a match."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Our Approach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our general approach is similar to that used in a conventional convolutional neural network however the initial processing occurs by running the images through a quantum circuit instead of a convolutional filter. Each simulation of a circuit represents one 3x3 filter being applied to one 3x3 region of one image. The construction of the circuit is randomised (see below), however this construction only occurs once per filter such that each region of the image being transformed by the same filter gets run through the same circuit. A single, scalar output is generated from the circuit which is used as the pixel strength of the output image, and the remainder of the neural net uses only classical processing, specifically two further convolutional layers, max pooling and two fully connected layers. This architecture has been chosen to closely mimic the structure used in our reference paper (Henderson et al, 2020), however as they note, \"The QNN topology chosen in this work is not fixed by nature ... the QNN framework was designed to give users complete control over the number and order of quanvolutional layers in the architecture. The topology explored in this work was chosen because it was the simplest QNN architecture to use as a baseline for comparison against other purely classical networks. Future work would focus on exploring the impact of more complex architectural variations.\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"Quanv_Neural_Net/Our_approach.jpg\" width=\"700\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quantum Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Henderson et al summarise the use of quantum circuits as convolutional layers: \"Quanvolutional layers are made up of a group of N quantum filters which operate much like their classical convolutional layer counterparts, producing feature maps by locally transforming input data. The key difference is that quanvolutional filters extract features from input data by transforming spatially local subsections of data using quantum circuits.\" Our approach to the circuit design is based on the paper and is as follows:\n",
    "\n",
    "1) The input images are iterated over and each 3x3 region is embedded into the quantum circuit using the threshold function: \n",
    "$$|\\psi \\rangle = \\begin{cases} \n",
    "                      |0\\rangle & if & strength\\leq 0 \\\\\n",
    "                      |1\\rangle & if & strength > 0\n",
    "                   \\end{cases}$$ \n",
    "\n",
    "As the pixel strengths are normalised to values between -0.5 and 0.5, it is expected that brighter regions of the image will intialise their corresponding qubit in the state $|1\\rangle$ and darker regions will intitialise the state $|0\\rangle$. Each pixel is represented by one qubit, such that 9 qubits are used in total, and this quantum circuit is reused for each 3x3 square in the filter.\n",
    "\n",
    "2) We next apply a random circuit to the qubits. To implement this, a random choice from Rx, Ry and Rz gates is applied to a random qubit, and the total number of gates applied in each layer is equal to the number of qubits. With a set probability (which we set to 0.3), a CNOT gate will be applied instead of the rotation to two random qubits. We have chosen to set the parameters of rotation with random numbers between (0,2π) however futher optimisation of the model could be found from using a variational circuit and optimising these parameters.\n",
    "\n",
    "3) Further layers could be applied of the random gates. To simplify, we only apply one layer.\n",
    "\n",
    "4) A scalar is outputted from the circuit and used as the corresponding pixel in the output image. We generate this number using the following method. The state vector of the final state of the circuit is simulated and the state corresponding to the most likely output (largest modulus) is selected. We then calculate the number of qubits for this state which are measured as a $|1\\rangle$. \n",
    "\n",
    "5) A total of four filters are applied to each image, and for each filter steps 1-3 are repeated with a different randomised circuit. The output image therefore contains a third dimension with four channels representing the four different outputted values which each filters produced."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"Quanv_Neural_Net/Quantum_circuit.jpg\" width=\"700\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code and Running the Program"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code cell is used to import the necessary packages and to set parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import tequila as tq\n",
    "\n",
    "from operator import itemgetter\n",
    "from tensorflow import keras\n",
    "\n",
    "n_filters = 4               # Number of convolutional filters\n",
    "filter_size = 3             # Size of filter = nxn (here 3x3)\n",
    "pool_size = 2               # Used for the pooling layer\n",
    "n_qubits = filter_size ** 2 # Number of qubits\n",
    "n_layers = 1                # Number of quantum circuit layers\n",
    "n_train = 1000              # Size of the training dataset\n",
    "n_test = 200                # Size of the testing dataset\n",
    "n_epochs = 100              # Number of optimization epochs\n",
    "\n",
    "SAVE_PATH = \"quanvolution/\" # Data saving folder\n",
    "PREPROCESS = False          # If False, skip quantum processing and load data from SAVE_PATH\n",
    "tf.random.set_seed(1)       # Seed for TensorFlow random number generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by creating the Dataset class. Here, we load the images and labels of handwritten digits from the MNIST dataset. We then reduce the number of images from 60,000 and 10,000 (for the training and testing sets respectively) down to the variables n_train and n_test, normalise the pixel values to within the range (-0.5,0.5) and reshape the images by adding a third dimension. Each image's shape is therefore transformed from (28, 28) to (28, 28, 1) as this is necessary for the convolutional layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Dataset:\n",
    "\n",
    "    def __init__(self):\n",
    "        # Loading the full dataset of images from keras\n",
    "        # Shape of self.train_images is (60000, 28, 28), shape of self.train_labels is (60000,)\n",
    "        # For self.test_images and self.test_labels, shapes are (10000, 28, 28) and (10000,)\n",
    "        mnist_dataset = keras.datasets.mnist\n",
    "        (self.train_images, self.train_labels), (self.test_images, self.test_labels) = mnist_dataset.load_data()\n",
    "\n",
    "        # Reduce dataset size to n_train and n_test\n",
    "        # First dimension of shapes are reduced to n_train and n_test\n",
    "        self.train_images = self.train_images[:n_train]\n",
    "        self.train_labels = self.train_labels[:n_train]\n",
    "        self.test_images = self.test_images[:n_test]\n",
    "        self.test_labels = self.test_labels[:n_test]\n",
    "\n",
    "        # Normalize pixel values within -0.5 and +0.5\n",
    "        self.train_images = (self.train_images / 255) - 0.5\n",
    "        self.test_images = (self.test_images / 255) - 0.5\n",
    "\n",
    "        # Add extra dimension for convolution channels\n",
    "        self.train_images = self.train_images[..., tf.newaxis]\n",
    "        self.test_images = self.test_images[..., tf.newaxis]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next code cell contains the class used to generate the quantum circuit. In theory, the circuit could be either structured or random. We form a randomised circuit to match the reference paper (Henderson et al, 2020), however for simplicity, our implementation differs in some ways. We choose to use only use single qubit Rx($\\theta$), Ry($\\theta$) and Rz($\\theta$) gates and the two qubit CNOT gate compared to the choice of single qubit X($\\theta$), Y($\\theta$), Z($\\theta$), U($\\theta$), P, T, H and two qubit CNOT, SWAP, SQRTSWAP, or CU gates used in the paper. Furthermore, we chose to assign a two qubit gate to any random qubits with a certain probability (labelled ratio_imprim, set to 0.3) rather than setting a connection probabiltiy between each pair of qubits (this approach follows the Pennylane tutorial). The seed is used for reproducability and its value is set depending on which filter the circuit represents (see QuantumModel below).\n",
    "\n",
    "The parameters used for the rotation gates have the potential to be optimised using a cost function. For simplicity, and to mirror the paper, here we will use random parameters and we will not include these in the optimisation of the model. This means that the quantum processing only needs to happen once, prior to creating the neural net."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class QuantumCircuit:\n",
    "    \n",
    "    def __init__(self, seed=None):\n",
    "        # Set random seed for reproducability\n",
    "        if seed: np.random.seed(seed)\n",
    "        \n",
    "        # Encode classical information into quantum circuit\n",
    "        # Bit flip gate is applied if the pixel strength > 0\n",
    "        self.circ = tq.QCircuit()\n",
    "        for i in range(n_qubits):\n",
    "            self.circ += tq.gates.X(i, power='input_{}'.format(i))\n",
    "\n",
    "        # Add random layers to the circuit\n",
    "        self.circ += self.random_layers()\n",
    "    \n",
    "    def random_layers(self, ratio_imprim=0.3):\n",
    "        # Initialise circuit\n",
    "        circuit = tq.QCircuit()\n",
    "\n",
    "        # Iterate over the number of layers, adding rotational and CNOT gates\n",
    "        # The number of rotational gates added per layer is equal to the number of qubits in the circuit\n",
    "        for i in range(n_layers):\n",
    "            j = 0\n",
    "            while (j < n_qubits):\n",
    "                if np.random.random() > ratio_imprim:\n",
    "                    # Applies a random rotation gate to a random qubit with probability (1 - ratio_imprim)\n",
    "                    rnd_qubit = np.random.randint(n_qubits)\n",
    "                    circuit += np.random.choice(\n",
    "                        [tq.gates.Rx(angle='l_{},th_{}'.format(i,j), target=rnd_qubit),\n",
    "                         tq.gates.Ry(angle='l_{},th_{}'.format(i,j), target=rnd_qubit),\n",
    "                         tq.gates.Rz(angle='l_{},th_{}'.format(i,j), target=rnd_qubit)])\n",
    "                    j += 1\n",
    "                else:\n",
    "                    # Applies the CNOT gate to 2 random qubits with probability ratio_imprim\n",
    "                    if n_qubits > 1:\n",
    "                        rnd_qubits = np.random.choice(range(n_qubits), 2, replace=False)\n",
    "                        circuit += tq.gates.CNOT(target=rnd_qubits[0], control=rnd_qubits[1])\n",
    "        return circuit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example to show the circuit used in this program, an instance of a circuit is drawn below. This will differ between calls if you remove the seed variable due to the random nature of forming the circuit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "circuit = QuantumCircuit(seed=2)\n",
    "tq.draw(circuit.circ, backend='qiskit')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We next show the QuantumModel class, used to generate the neural network for the images which undergo pre-processing through the quantum convolutional layer. If PREPROCESSING is set to True, each image from the dataset undergoes processing through a number of quantum circuits, determined by n_filters. The embedding used, the structure of the circuit and the method of extracting the output are described in the background of this tutorial. \n",
    "\n",
    "We use tensorflow to construct the neural net. The implementation we use contains two conventional convolutional layers, each followed by max pooling, and then one fully connected with 1024 nodes before the softmax output layer. We use a Relu activation function for the convolutional and fully connected layers. See the background section of this tutorial for some context on this choice of neural net."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class QuantumModel:\n",
    "\n",
    "    def __init__(self, dataset, parameters):\n",
    "        # Initialize dataset and parameters\n",
    "        self.ds = dataset\n",
    "        self.params = parameters\n",
    "        \n",
    "        # The images are run through the quantum convolutional layer\n",
    "        self.convolutional_layer()\n",
    "\n",
    "        # The model is initialized\n",
    "        self.model = keras.models.Sequential([\n",
    "            keras.layers.Conv2D(n_filters, filter_size, activation='relu'),\n",
    "            keras.layers.MaxPooling2D(pool_size=pool_size),\n",
    "            keras.layers.Conv2D(n_filters, filter_size, activation='relu'),\n",
    "            keras.layers.MaxPooling2D(pool_size=pool_size),\n",
    "            keras.layers.Flatten(),\n",
    "            keras.layers.Dense(1024, activation=\"relu\"),\n",
    "            keras.layers.Dense(10, activation=\"softmax\")\n",
    "        ])\n",
    "\n",
    "        # Compile model using the Adam optimiser\n",
    "        self.model.compile(\n",
    "            optimizer=keras.optimizers.Adam(learning_rate=0.00001),\n",
    "            loss=\"sparse_categorical_crossentropy\",\n",
    "            metrics=[\"accuracy\"]\n",
    "        )\n",
    "    \n",
    "    def convolutional_layer(self):\n",
    "        if PREPROCESS == True:\n",
    "            # Initate arrays to store processed images\n",
    "            self.q_train_images = [np.zeros((28-2, 28-2, n_filters)) for _ in range(len(self.ds.train_images))]\n",
    "            self.q_test_images = [np.zeros((28-2, 28-2, n_filters)) for _ in range(len(self.ds.test_images))]\n",
    "            \n",
    "            # Loop over the number of filters, applying a different randomised quantum circuit for each\n",
    "            for i in range(n_filters):\n",
    "                print('Filter {}/{}\\n'.format(i+1, n_filters))\n",
    "                \n",
    "                # Construct circuit\n",
    "                # We set the seed to be i+1 so that the circuits are reproducable but the design differs between filters\n",
    "                # We use i+1 not i to avoid setting the seed as 0 which sometimes produces random behaviour\n",
    "                circuit = QuantumCircuit(seed=i+1)\n",
    "                \n",
    "                # Apply the quantum processing to the train_images, analogous to a convolutional layer\n",
    "                print(\"Quantum pre-processing of train images:\")\n",
    "                for j, img in enumerate(self.ds.train_images):\n",
    "                    print(\"{}/{}        \".format(j+1, n_train), end=\"\\r\")\n",
    "                    self.q_train_images[j][...,i] = (self.filter_(img, circuit, self.params[i]))\n",
    "                print('\\n')\n",
    "\n",
    "                # Similarly for the test_images\n",
    "                print(\"Quantum pre-processing of test images:\")\n",
    "                for j, img in enumerate(self.ds.test_images):\n",
    "                    print(\"{}/{}        \".format(j+1, n_test), end=\"\\r\")\n",
    "                    self.q_test_images[j][...,i] = (self.filter_(img, circuit, self.params[i]))\n",
    "                print('\\n')\n",
    "\n",
    "            # Transform images to numpy array\n",
    "            self.q_train_images = np.asarray(self.q_train_images)\n",
    "            self.q_test_images = np.asarray(self.q_test_images)\n",
    "            \n",
    "            # Save pre-processed images\n",
    "            np.save(SAVE_PATH + \"q_train_images.npy\", self.q_train_images)\n",
    "            np.save(SAVE_PATH + \"q_test_images.npy\", self.q_test_images)\n",
    "        \n",
    "        # Load pre-processed images\n",
    "        self.q_train_images = np.load(SAVE_PATH + \"q_train_images.npy\")\n",
    "        self.q_test_images = np.load(SAVE_PATH + \"q_test_images.npy\")\n",
    "\n",
    "    def filter_(self, image, circuit, variables):\n",
    "        # Initialize output image\n",
    "        output = np.zeros((28-2, 28-2))\n",
    "\n",
    "        # Loop over the image co-ordinates (i,j) using a 3x3 square filter\n",
    "        for i in range(28-2):\n",
    "            for j in range(28-2):\n",
    "\n",
    "                # Extract the value of each pixel in the 3x3 filter grid\n",
    "                image_pixels = [\n",
    "                    image[i,j,0],\n",
    "                    image[i,j+1,0],\n",
    "                    image[i,j+2,0],\n",
    "                    image[i+1,j,0],\n",
    "                    image[i+1,j+1,0],\n",
    "                    image[i+1,j+2,0],\n",
    "                    image[i+2,j,0],\n",
    "                    image[i+2,j+1,0],\n",
    "                    image[i+2,j+2,0]\n",
    "                ]\n",
    "\n",
    "                # Construct parameters used to embed the pixel strength into the circuit\n",
    "                input_variables = {}\n",
    "                for idx, strength in enumerate(image_pixels):\n",
    "                    # If strength > 0, the power of the bit flip gate is 1\n",
    "                    # Therefore this qubit starts in state |1>\n",
    "                    if strength > 0:\n",
    "                        input_variables['input_{}'.format(idx)] = 1\n",
    "                    # Otherwise the gate is not applied and the initial state is |0>\n",
    "                    else:\n",
    "                        input_variables['input_{}'.format(idx)] = 0\n",
    "\n",
    "                # Find the statevector of the circuit and determine the state which is most likely to be measured\n",
    "                wavefunction = tq.simulate(circuit.circ, variables={**variables, **input_variables})\n",
    "                amplitudes = [(k,(abs(wavefunction(k)))) for k in range(2**n_qubits) if wavefunction(k)]\n",
    "                max_idx = max(amplitudes,key=itemgetter(1))[0]\n",
    "                \n",
    "                # Count the number of qubits which output '1' in this state\n",
    "                result = len([k for k in str(bin(max_idx))[2::] if k == '1'])\n",
    "                output[i,j] = result\n",
    "        return output\n",
    "\n",
    "    def train(self):\n",
    "        # Train the model on the dataset\n",
    "        self.history = self.model.fit(\n",
    "            self.q_train_images,\n",
    "            self.ds.train_labels,\n",
    "            validation_data=(self.q_test_images, self.ds.test_labels),\n",
    "            batch_size=4,\n",
    "            epochs=n_epochs,\n",
    "            verbose=2\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also create a ClassicalModel class to run the images through a conventional convolutional neural network. The design of the neural net used here is identical to the QuantumModel class, however the images used are directly from the dataset and therefore have not been processed through the quantum layer. We include this as a control to compare the results from the quantum model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ClassicalModel:\n",
    "\n",
    "    def __init__(self, dataset):\n",
    "        # Initialize dataset and parameters\n",
    "        self.ds = dataset\n",
    "\n",
    "        # The model is initialized\n",
    "        self.model = keras.models.Sequential([\n",
    "            keras.layers.Conv2D(n_filters, filter_size, activation='relu'),\n",
    "            keras.layers.MaxPooling2D(pool_size=pool_size),\n",
    "            keras.layers.Conv2D(n_filters, filter_size, activation='relu'),\n",
    "            keras.layers.MaxPooling2D(pool_size=pool_size),\n",
    "            keras.layers.Flatten(),\n",
    "            keras.layers.Dense(1024, activation=\"relu\"),\n",
    "            keras.layers.Dense(10, activation=\"softmax\")\n",
    "        ])\n",
    "\n",
    "        # Compile model using the Adam optimiser\n",
    "        self.model.compile(\n",
    "            optimizer=keras.optimizers.Adam(learning_rate=0.00005),\n",
    "            loss=\"sparse_categorical_crossentropy\",\n",
    "            metrics=[\"accuracy\"]\n",
    "        )\n",
    "    \n",
    "    def train(self):\n",
    "        # Train the model on the dataset\n",
    "        self.history = self.model.fit(\n",
    "            self.ds.train_images,\n",
    "            self.ds.train_labels,\n",
    "            validation_data=(self.ds.test_images, self.ds.test_labels),\n",
    "            batch_size=4,\n",
    "            epochs=n_epochs,\n",
    "            verbose=2\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now able to run our program! The following code does this using the quantum_model and classical_model functions. Although the implementations are similar, quantum_model additionally defines the parameters used for the rotational gates in the circuit. We have limited the value of each parameter to the range (0,2π).\n",
    "\n",
    "Running the program takes some time. Our results are plotted below, so if you would rather not wait, either reduce the numbers in n_train and n_test or skip ahead!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def quantum_model():\n",
    "    # Generating parameters, each maps to a random number between 0 and 2*π\n",
    "    # parameters is a list of dictionaries, where each dictionary represents the parameter\n",
    "    # mapping for one filter\n",
    "    parameters = []\n",
    "    for i in range(n_filters):\n",
    "        filter_params = {}\n",
    "        for j in range(n_layers):\n",
    "            for k in range(n_qubits):\n",
    "                filter_params[tq.Variable(name='l_{},th_{}'.format(j,k))] = np.random.uniform(high=2*np.pi)\n",
    "        parameters.append(filter_params)\n",
    "        \n",
    "    # Initalise the dataset\n",
    "    ds = Dataset()\n",
    "    \n",
    "    # Initialise and train the model\n",
    "    model = QuantumModel(ds, parameters=parameters)\n",
    "    model.train()\n",
    "    \n",
    "    # Store the loss and accuracy of the model to return\n",
    "    loss = model.history.history['val_loss']\n",
    "    accuracy = model.history.history['val_accuracy']\n",
    "\n",
    "    return model\n",
    "\n",
    "def classical_model():\n",
    "    # Initialise the dataset\n",
    "    ds = Dataset()\n",
    "    \n",
    "    # Initialise and train the model\n",
    "    model = ClassicalModel(ds)\n",
    "    model.train()\n",
    "    \n",
    "    # Store the loss and accuracy of the model to return\n",
    "    loss = model.history.history['val_loss']\n",
    "    accuracy = model.history.history['val_accuracy']\n",
    "    \n",
    "    return model\n",
    "\n",
    "model_q = quantum_model()\n",
    "model_c = classical_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting the Results"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graphs showing the accuracy and loss of our models are included in this text box. These were generated using the function plot, available below. As shown, the results from the quantum processing lead to a model comparable to the classical control in both accuracy and loss. After running for 100 epochs, the quantum model results in a validation set accuracy of 0.9350, compared to the fully classical model which has a validation set accuracy of 0.9150.\n",
    "<img src=\"Quanv_Neural_Net/Plots.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot(model_q, model_c):\n",
    "\n",
    "    plt.style.use(\"seaborn\")\n",
    "    fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(6, 9))\n",
    "\n",
    "    # Plotting the graph for accuracy\n",
    "    ax1.plot(model_q.history.history['val_accuracy'], color=\"tab:red\", label=\"Quantum\")\n",
    "    ax1.plot(model_c.history.history['val_accuracy'], color=\"tab:green\", label=\"Classical\")\n",
    "    ax1.set_ylabel(\"Accuracy\")\n",
    "    ax1.set_ylim([0,1])\n",
    "    ax1.set_xlabel(\"Epoch\")\n",
    "    ax1.legend()\n",
    "\n",
    "    # Plotting the graph for loss\n",
    "    ax2.plot(model_q.history.history['val_loss'], color=\"tab:red\", label=\"Quantum\")\n",
    "    ax2.plot(model_c.history.history['val_loss'], color=\"tab:green\", label=\"Classical\")\n",
    "    ax2.set_ylabel(\"Loss\")\n",
    "    ax2.set_xlabel(\"Epoch\")\n",
    "    ax2.legend()\n",
    "    \n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "plot(model_q, model_c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now compare the behaviour of the two models. We do this by running the test images through each with the optimised weights and biases and seeing the results of the classification. This process is implemented using the Classification class, shown below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall, our quantum model misclassified images 34, 37, 42, 54, 67, 74, 120, 127, 143, 150, 152, 166, and 185. The classical model misclassified images 8, 16, 21, 23, 54, 60, 61, 67, 74, 93, 113, 125, 134, 160, 168, 178, and 196. This means that in total, the quantum model misclassified 13 images and the classical model misclassified 17 images. Of these, only images 54, 67, and 74 were misclassified by both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from termcolor import colored\n",
    "\n",
    "class Classification:\n",
    "    \n",
    "    def __init__(self, model, test_images):\n",
    "        # Initialising parameters\n",
    "        self.model = model\n",
    "        self.test_images = test_images\n",
    "        self.test_labels = model.ds.test_labels\n",
    "\n",
    "    def classify(self):\n",
    "        # Create predictions on the test set\n",
    "        self.predictions = np.argmax(self.model.model.predict(self.test_images), axis=-1)\n",
    "\n",
    "        # Keep track of the indices of images which were classified correctly and incorrectly\n",
    "        self.correct_indices = np.nonzero(self.predictions == self.test_labels)[0]\n",
    "        self.incorrect_indices = np.nonzero(self.predictions != self.test_labels)[0]\n",
    "    \n",
    "    def print_(self):\n",
    "        # Printing the total number of correctly and incorrectly classified images\n",
    "        print(len(self.correct_indices),\" classified correctly\")\n",
    "        print(len(self.incorrect_indices),\" classified incorrectly\")\n",
    "        print('\\n')\n",
    "\n",
    "        # Printing the classification of each image\n",
    "        for i in range(n_test):\n",
    "            print(\"Image {}/{}\".format(i+1, n_test))\n",
    "            if i in self.correct_indices:\n",
    "                # The image was correctly classified\n",
    "                print('model predicts: {} - true classification: {}'.format(\n",
    "                    self.predictions[i], self.test_labels[i]))\n",
    "            else:\n",
    "                # The image was not classified correctly\n",
    "                print(colored('model predicts: {} - true classification: {}'.format(\n",
    "                    self.predictions[i], self.test_labels[i]), 'red'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Quantum Model')\n",
    "q_class = Classification(model_q, model_q.q_test_images)\n",
    "q_class.classify()\n",
    "q_class.print_()\n",
    "\n",
    "print('\\n')\n",
    "\n",
    "print('Classical Model')\n",
    "c_class = Classification(model_c, model_c.ds.test_images)\n",
    "c_class.classify()\n",
    "c_class.print_()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we can see the effect that the quantum convolutional layer actually has on the images by plotting images after they have been run through the quantum filters, and to do this we use the function visualise, shown below. Included in this text box is a plot showing four images which have been run through our filters. The top row shows images from the original dataset, and each subsequent row shows the result from each of the four filters on that original image. It can be seen that the processing preserves the global shape of the digit while introducing local distortion.\n",
    "<img src=\"Quanv_Neural_Net/Filters.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def visualise(model):\n",
    "    # Setting n_samples to be the number of images to print\n",
    "    n_samples = 4\n",
    "    \n",
    "    fig, axes = plt.subplots(1 + n_filters, n_samples, figsize=(10, 10))\n",
    "    \n",
    "    # Iterate over each image\n",
    "    for i in range(n_samples):\n",
    "        \n",
    "        # Plot the original image from the dataset\n",
    "        axes[0, 0].set_ylabel(\"Input\")\n",
    "        if i != 0:\n",
    "            axes[0, i].yaxis.set_visible(False)\n",
    "        axes[0, i].imshow(model.ds.train_images[i, :, :, 0], cmap=\"gray\")\n",
    "\n",
    "        # Plot the images generated by each filter\n",
    "        for c in range(n_filters):\n",
    "            axes[c + 1, 0].set_ylabel(\"Output [ch. {}]\".format(c))\n",
    "            if i != 0:\n",
    "                axes[c, i].yaxis.set_visible(False)\n",
    "            axes[c + 1, i].imshow(model.q_train_images[i, :, :, c], cmap=\"gray\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "    \n",
    "visualise(model_q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Resources used to make this tutorial:\n",
    "\n",
    "1. [Pennylane: Quanvolutional Neural Networks](https://pennylane.ai/qml/demos/tutorial_quanvolution.html)\n",
    "2. Henderson, M., Shakya, S., Pradhan, S. et al. Quanvolutional neural networks: powering image recognition with quantum circuits. Quantum Mach. Intell. 2, 1–9 (2020). https://doi.org/10.1007/s42484-020-00012-y\n",
    "3. [Keras for Beginners: Implementing a Convolutional Neural Network. Victor Zhou](https://victorzhou.com/blog/keras-cnn-tutorial/).\n",
    "4. [CNNs, Part 1: An Introduction to Convolutional Neural Networks. Victor Zhou](https://victorzhou.com/blog/intro-to-cnns-part-1/).\n",
    "5. [How Convolutional Neural Networks work](https://www.youtube.com/watch?v=FmpDIaiMIeA)\n",
    "6. [Neural Networks and Deep Learning, chapter 6. Michael Nielsen](http://neuralnetworksanddeeplearning.com/chap6.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}