Parallel Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a7fea9bc",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Parallel Python 🖥🖥🖥\n",
    "### Zbyszek & Jakob\n",
    "### ASPP 2022, Bilbao, Spain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21a5cdaf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Outline\n",
    "\n",
    "* Numba (moved over from yesterday)\n",
    "* Introduction\n",
    "* Nodes, processes, threads\n",
    "* The builtin `multiprocessing` module\n",
    "* Multi-node computation with `ipyparallel`\n",
    "* Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "667472bf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Introduction: we have to parallelize!\n",
    "\n",
    "![Dennard scaling](figures/scaling1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "830b12c2",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "- transistor count increases, but max frequency doesn't increase any more\n",
    "- rather: multiple cores on one chip\n",
    "- to benefit from these, we need to *parallelize*\n",
    "- also: for processing data larger than memory of one computer(?)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d74e1d9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Nodes, processes, threads"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc4ec5d7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- \"A processes runs on the CPU.\"\n",
    "- \"A process can consist of many threads.\"\n",
    "- \"Modern CPUs have many cores and threads (multicore, multithreaded).\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eca7f0b9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## How long does it take to provide 30 students with coffee?\n",
    "\n",
    "[<img src=\"figures/coffee_serial.jpg\" width=\"360\"/>](figures/coffee_serial.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56f269be",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## How long does it take to provide 30 students with coffee?\n",
    "\n",
    "[<img src=\"figures/coffee_parallel.jpg\" width=\"360\"/>](figures/coffee_parallel.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42e0e983",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## \"Embarassingly parallel\" problems\n",
    "\n",
    "- little/no effort to split work into parallel tasks\n",
    "- no serial dependencies\n",
    "- little/no communication required\n",
    "\n",
    "Examples:\n",
    "- compute the cubes of a set of numbers\n",
    "- eigendecomposition of a collection of matrices\n",
    "- performing the same analysis on a collection of images\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10a4a28d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The builtin `multiprocessing` module\n",
    "\n",
    "Now let's put theory intro practice. Jakob will introduce the `multiprocessing` module, which allows you to perform simple parallelization in Python.\n",
    "\n",
    "We will only consider embarassingly parallel problems, and investigate the difference between using **threads** or **processes** for parallelization.\n",
    "\n",
    "For exercises see the notebooks in the folder `./multiprocessing/`.\n",
    "\n",
    "(the notebook from the live lecture with additional comments and the solved exercises will be available in this repository after the lecture)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "742e8d38",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Multi-node computation with `ipyparallel`\n",
    "\n",
    "So far we have focus on parallelization on a *single machine*. However, you may have *multiple machines* available to perform work for you (e.g., serveral workstations in a lab). We will use the `IPython Parallel` (`ipyparallel`) package to build our own happy little ASPP cluster to solve embrassingly-parallel problems.\n",
    "\n",
    "First, we will briefly explain how `ipyparallel` works.\n",
    "\n",
    "Then, check out the instruction in the [ipyparallel-asppcluster.ipynb](ipyparallel-asppcluster.ipynb) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9126f375",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "- processors are not becomming faster, but machines have more and more cores available -> parallelization\n",
    "- use the builtin `multiprocessing` module for easy parallelization\n",
    "- **careful**: parallelization may be present at multiple levels (e.g., Python & C, careful with libraries like NumPy)\n",
    "- use `IPython Parallel` to leverage the power of multiple machines\n",
    "\n",
    "- going further:\n",
    "  - parallelization in Cython via `prange`\n",
    "  - MPI (\"Message Passing Interface\"): some problems are too big for one machine but still require processes to exchange information during runtime (e.g., weather simulations); MPI is a standard to allow processes to communicate (for example across a network connection); `mpi4py` is a Python library which allows you to approach problems like this (**WARNING**: involves significant cognitive overhead)\n",
    "  - dask(?)\n",
    "  - joblib(?)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}