high-performance-computing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# High-Performance Computing with Norse and PyTorch Lightning\n",
    "\n",
    "![](https://raw.githubusercontent.com/norse/norse-hbp-workshop/main/images/plnorse.png)\n",
    "\n",
    "Norse provides two necessary requirements for scaling spiking neural network simulations to cluster-wide simulations: a solid infrastructure (PyTorch) and proper state handling. PyTorch permits us to apply a highly developed toolchain (as we will see in a minute) to solve our problems. This saves a dramatic amount of time. Proper state handling permits us to paralellize our simulations, which is practically impossible in many other neuron simulators.\n",
    "\n",
    "In this small tutorial, you will learn to use Norse with the PyTorch-based framework, [PyTorch Lightning](https://pytorch-lightning.readthedocs.io/en/latest/) (PL). PL makes it *lightning*-fast (pun intended I'm sure) to build and scale your networks.\n",
    "\n",
    "The workshop is structured as follows\n",
    "* Build our first PyTorch Lightning SNN model (~10 minutes)\n",
    "* Discuss relevant toy problems with your study mates (~5 minutes)\n",
    "* Try out your own ideas (~15 minutes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step -1: Change Runtime Type\n",
    "\n",
    "This step is specific to google collab and doesn't apply if you execute this notebook at home or somewhere else. Select \"Runtime\" above and choose \"GPU\" as the accelerator. This will make sure\n",
    "that all of the example code below will run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 0: Installations\n",
    "\n",
    "First of all, we will need to install Norse and PyTorch Lightning. Please run the cell below. Read on while it's running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install norse pytorch-lightning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Building our own model\n",
    "\n",
    "In this part we will get a lightning quick overview of PyTorch Lightning, train a model, and then try to accelerate it. We have little time, so try to just skim over it now and remember that the material is available after the workshop."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 What is PyTorch Lightning?\n",
    "\n",
    "![](https://pytorch-lightning.readthedocs.io/en/stable/_static/images/logo.svg)\n",
    "\n",
    "The primitives of PyTorch are designed as atoms that we can stitch together to form complicated models. On a more higher level there are some things that become *very* repetitive once you've built your first few models: preprocessing data, defining training loops, measuring loss, plotting your results. This is where PyTorch Lightning comes in. The help us to \"Spend more time on research, less on engineering.\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 PyTorch Lightning models\n",
    "\n",
    "In vanilla PyTorch we would use the `torch.nn.Module` as a base class. Here, we have to extend the `LightningModule` and implement (at least) three methods:\n",
    "\n",
    "* `__init__` - The constructor where you build your model\n",
    "* `configure_optimizers` - This is where you define how your model is optimized\n",
    "* `training_step` - This is where the model is being applied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'pytorch_lightning'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "Input \u001b[0;32mIn [1]\u001b[0m, in \u001b[0;36m<cell line: 2>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mtorch\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mpytorch_lightning\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mpl\u001b[39;00m\n\u001b[1;32m      3\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mnorse\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtorch\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mnorse\u001b[39;00m\n\u001b[1;32m      6\u001b[0m \u001b[38;5;28;01mclass\u001b[39;00m \u001b[38;5;21;01mSpikingModel\u001b[39;00m(pl\u001b[38;5;241m.\u001b[39mLightningModule):\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'pytorch_lightning'"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pytorch_lightning as pl\n",
    "import norse.torch as norse\n",
    "\n",
    "\n",
    "class SpikingModel(pl.LightningModule):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.model = norse.SequentialState(\n",
    "            norse.ConstantCurrentLIFEncoder(seq_length=32),  # Encode in time\n",
    "            norse.ConvNet(),  # Apply convolution\n",
    "        )\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        return torch.optim.Adam(self.model.parameters(), lr=1e-3)\n",
    "\n",
    "    def training_step(self, batch, batch_index):\n",
    "        x, y = batch\n",
    "        # Input has shape (32, 1, 28, 28): (batch, channel, x, y)\n",
    "        out, state = self.model(x)  # Note the output state; we ignore it for now\n",
    "        # Output has shape (32, 32, 10) because we encoded each input in 32 timesteps\n",
    "        # Here we sum up the time dimension and see which class got most spikes\n",
    "        out, _ = out.max(dim=0)\n",
    "        return torch.nn.functional.cross_entropy(out, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we are not specifiying anything about the data. We are simply assuming that it arrives through the `batch` variable that contains both the input (`x`) and the labels (`y`). Now we simply need to load in the data and start training:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Loading in data\n",
    "\n",
    "MNIST is a pretty boring example, so that's why we chose it. For now, we just want to get the data loading out of the way so we can start training and scaling our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision\n",
    "\n",
    "# We need to normalize the data so we get *some* response from the neurons\n",
    "data_transform = torchvision.transforms.Compose(\n",
    "    [\n",
    "        torchvision.transforms.ToTensor(),\n",
    "        torchvision.transforms.Normalize((0.1307,), (0.3081,)),\n",
    "    ]\n",
    ")\n",
    "dataloader = torch.utils.data.DataLoader(\n",
    "    torchvision.datasets.MNIST(root=\".\", download=True, transform=data_transform),\n",
    "    batch_size=32,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Training our model\n",
    "\n",
    "We are now ready to train our model! The only thing we need is a wrapper class to take care of loading in the data and feeding it to our `LightningModule` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = SpikingModel()  # Our model from before\n",
    "trainer = pl.Trainer()\n",
    "trainer.fit(model, dataloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5 Help! My model is taking forever!\n",
    "\n",
    "Yep it is. And this is even just a simple MNIST model! We don't have time to wait for this: go ahead and stop it by pressing the ■ icon above. Notice that PyTorchLightning helpfully warned us that we\n",
    "have an unused GPU!\n",
    "\n",
    "Let's try that again, only this time with a GPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = SpikingModel()  # Our model from before\n",
    "trainer = pl.Trainer(gpus=1)  # Notice the gpus flag\n",
    "trainer.fit(model, dataloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.6 Scaling to HPC with PyTorch Lightning\n",
    "\n",
    "To summarize what we saw so far, we managed to build a model, train it with a dataset, and scale it to 1 (but potentially many) GPUs with around 50 lines of code. Not bad!\n",
    "\n",
    "You may be wondering where the HPC element in this comes in, but we have actually achieved the most significant objective already: a scalable model. Because Norse is handling state correctly, and because PyTorch Lightning takes care of the synchronization of losses across an arbitrary number of machines, you are able to run this on multiple nodes and several GPUs if you wanted to!\n",
    "\n",
    "To be slightly more specific, PyTorch Lightning already features support for HPC clusters because your model is easy to scale. The different way to distribute models (e. g. in HPC) is called [`accelerators` in PyTorch Lightning](https://pytorch-lightning.readthedocs.io/en/stable/accelerators.html?highlight=hpc#). Here is an example that will run your model in an HPC (which won't work here for obvious reasons):\n",
    "\n",
    "```python\n",
    "model = SpikingModel() # Our model from before\n",
    "trainer = Trainer(accelerator=DDPHPCAccelerator())\n",
    "trainer.fit(model, dataloader)\n",
    "```\n",
    "\n",
    "You can read more about their [DDPHPCAccelerator here](https://pytorch-lightning.readthedocs.io/en/stable/accelerators.html?highlight=hpc#ddp-hpc-accelerator) or [see how the accelerator parameter works with the Tranier module here](https://pytorch-lightning.readthedocs.io/en/stable/trainer.html#accelerator)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.7 SLURM Support out of the Box\n",
    "\n",
    "You can run your PyTorch-Lightning model with almost no hassle on your favorite SLURM\n",
    "cluster (JUWELS anyone?) of choice! See https://pytorch-lightning.readthedocs.io/en/latest/slurm.html.\n",
    "It will helpfully save your progress and restart from the last checkpoint accross\n",
    "job submissions. You can get your model running accross 2 super computing\n",
    "nodes and 8 V100 GPU cards without almost no effort of your own."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Discussion\n",
    "\n",
    "Now that you know how PyTorch works, have seen how Norse builds models, and knows how PyTorch Lightning can accelerate your models, it's time time to discuss how to apply your knowledge!\n",
    "\n",
    "MNIST is a good example for non-trivial supervised learning. And, as you saw, Norse can indeed solve it.\n",
    "However, there are much more interesting datasets out there and they might not necessarily be solvable with supervised learning. Norse also supports [STDP](https://github.com/norse/norse/blob/master/norse/torch/functional/stdp.py) (although still in an early stage) and other learning rules.\n",
    "\n",
    "So, here is our challenge to you: consider your own work/domain/expertise and how spike-based learning (supervised/non-supervised, local/global, etc.) is be relevant. Specifically, ask yourself whether you can come up with neuron-based simulations that are sufficiently large to require HPC access. Could you model this with Norse? Discuss this with your discord group now. Here are some questions you can ask your group to kickstart the discussion:\n",
    "\n",
    "* Do you think idea X can be solved with spike-based learning?\n",
    "* How would the dataset look like?\n",
    "  * Can you learn that with biologically inspired learning algorithms?\n",
    "  * Would you need supervised learning?\n",
    "* What learning algorithm would you use to solve X?\n",
    "\n",
    "When you are done (don't spend more than ~5-10 minutes) move on to the final implementation section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Implementation\n",
    "\n",
    "In the final minutes of this workshop, we encourage you to boil down your problem to a really small toy example. You will spend the rest of the workshop trying to implement it, so we recommend that you recruit the aid of your workshop group and try to get something simple working. Please be realistic (in this workshop): we are severely time restricted.\n",
    "\n",
    "Here are a few recommendations:\n",
    "* Try to only modify the data and the neural network\n",
    "  * You can, for instance, modify the code above to solve a simple XOR problem: the network would be much simpler and the data could be generated\n",
    "* Look at [the examples in the PyTorch Lightning documentation](https://pytorch-lightning.readthedocs.io) for inspiration (in the left sidebar)\n",
    "* Loot at [the datasets available in Norse](https://github.com/norse/norse/tree/master/norse/dataset). They are event-based so you do not have to worry about encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Profit\n",
    "\n",
    "We only scratched the surface of what PyTorch Lightning provides. They also provide excellent logging/dashboarding with [Tensorboard](https://www.tensorflow.org/tensorboard/), allow model checkpointing, model discretization, and much more.\n",
    "\n",
    "We hope this was enlightening. The workshop material is available online at https://github.com/norse/norse-hbp-workshop, so you can revisit it any time.\n",
    "\n",
    "Thank you for your attention!\n",
    "\n",
    "![](https://raw.githubusercontent.com/norse/norse/master/logo.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.6 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}