Parallelism.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parallelism in Julia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Julia offers three main methods for parallelism:\n",
    "\n",
    "1. Coroutines (\"green threads\" or Tasks)\n",
    "2. Multi-Threading\n",
    "3. Multi-Core or Distributed Processing (Multi Processing)\n",
    "\n",
    "There is also an additional level that's relevant for MixedModels, but which we won't highlight here: concurrency in external libraries, such as BLAS.\n",
    "\n",
    "Note that the material presented here is very condensed and based on several Julia documents as well as my experience implementing multi-threading for `MixedModels.parametricbootstrap` and `MixedModelsSim.simulate_waldtests`.\n",
    "\n",
    "1. [Announcing composable multi-threaded parallelism in Julia](https://julialang.org/blog/2019/07/multithreading/)\n",
    "2. [Parallel Computing](https://docs.julialang.org/en/v1/manual/parallel-computing/)\n",
    "3. [Surprising capture boxing behavior in closure](https://discourse.julialang.org/t/surprising-capture-boxing-behavior-in-closure/20254).\n",
    "4. [`ProgressMeter.jl` documentation](https://github.com/timholy/ProgressMeter.jl)\n",
    "\n",
    "## Coroutines\n",
    "\n",
    "Coroutines don't depend on the particulars of the host operating system at all. They are handled completely within Julia. But they require a complete restructuring of your code.\n",
    "\n",
    "If you've ever written a generator in Python (i.e. written a function that uses `yield` instead of `return`), then you've used a form of coroutines. They are not impossible to right, but they're hard to strap on post hoc. \n",
    "\n",
    "That said, if you're using comprehensions or `map` with independent entries, you should check out [`asyncmap`](https://docs.julialang.org/en/v1/base/parallel/#Base.asyncmap).\n",
    "\n",
    "Also, there seems to be a serious restriction to these:\n",
    "\n",
    "> Currently, all tasks in Julia are executed in a single OS thread co-operatively. Consequently, asyncmap is beneficial only when the mapping function involves any I/O - disk, network, remote worker invocation, etc.\n",
    "\n",
    "Note that a lot of magic happens in the other two options that gives a surface appearance similar to coroutines (e.g. `fetch()` is defined on Tasks).\n",
    "\n",
    "\n",
    "## Multi-Threading\n",
    "\n",
    "Multi-Threading works by splitting the workload among threads on a single machine. This has implications for the cost of copying objects as well as how we can share memory.\n",
    "\n",
    "Note that this appears to fall somewhere between Python's `threading` and `multiprocessing` libraries. The Julia documentation consistently calls things \"threads\" or even \"operating system threads\", yet discusses the \"fork and join\" approach, which on Unix systems would be closer to \"processes\". Based on my examination of running processes on my Linux system, Julia appears to be using \"threads\", but I haven't taken the time to go through the source. This may also be operating system dependent, similar to `multicore` vs. `multisession` in R. In either case, Julia does not have a GIL (Global Interpreter Lock) like Python.\n",
    "\n",
    "If you have used the relatively new `future` package in R, then you will be relatively well preparing for multi-threading and distributed processing.\n",
    "\n",
    "## Distributed Processing\n",
    "\n",
    "Distributed processing creates additional \"worker\" processes that may or may not reside on a single machine.  This has implications for the cost of copying objects as well as how we can share memory.\n",
    "\n",
    "It is much simpler than R's `snow` package, yet also allows for processes to be moved across machines. Some functinality maps onto serial Julia much the way functionality in Python's `multiprocessing` maps onto base Python.\n",
    "\n",
    "\n",
    "# A word of warning\n",
    "\n",
    "Parellization and concurrency are hard problems. Really hard. They can introduce subtle bugs that are hard to detect because of the stochastic nature of race conditions. They can also make it nearly impossible to kill 'a' process because you actually have lots of processes. So, as the saying goes, \"measure twice, cut once\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-Threading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For several of the examples here, we will use rather naive/inefficient algorithms and then parallelize them. We use the naive version because they make easy pedagogical examples, both in terms of understanding the individual examples and in terms of highlighting that concurrency isn't a replacemant for good algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that multi-threading in Julia is still considered 'experimental', but Julia respects semantic versioning and so the examples presented here will continue working will future 1.x releases. Some of the functionality I present here was introduced 1.3, but that's what we require for MixedModels.jl, so you're all using it anyway."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number and availability of threads depends on the environment variable `JULIA_NUM_THREADS`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Threads.nthreads()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Threads.threadid()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "using BenchmarkTools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our purposes, we consider the Fibonnaci numbers to be the following sequence:\n",
    "\n",
    "$ 0, 1, 2, 3, 5, 8, \\ldots $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Single-Threading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "function fib(n::BigInt)\n",
    "    if n < 2\n",
    "        return n\n",
    "    end\n",
    "   \n",
    "    return fib(n - 1) + fib(n - 2)\n",
    "end\n",
    "\n",
    "fib(n::Int) = fib(BigInt(n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@benchmark fib(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Multi-Threading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "function fib_mt(n::BigInt)\n",
    "    if n < 2\n",
    "        return n\n",
    "    end\n",
    "    t = Threads.@spawn fib(n - 2)\n",
    "    return fib(n - 1) + fetch(t)\n",
    "end\n",
    "\n",
    "fib_mt(n::Int) = fib_mt(BigInt(n))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note this is different in internal magic, but very similar in application to R's `future`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@benchmark fib_mt(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's not really a huge speed boost. In some places, it was even worse. \n",
    "\n",
    "Pedagogical moment: Why?\n",
    "\n",
    "\n",
    "There are two ways we could do much better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Efficient single-threaded implementation with different form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "function fib_loop(n::BigInt)\n",
    "    if n < 2\n",
    "        return n\n",
    "    end\n",
    "    prev_val = 0\n",
    "    val = 1\n",
    "    for i in 2:n\n",
    "        val, prev_val = val + prev_val, val\n",
    "    end\n",
    "    return val\n",
    "end\n",
    "\n",
    "fib_loop(n::Int) = fib_loop(BigInt(n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fib(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fib_loop(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@benchmark fib_loop(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single-Threading with Memoizing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "let \n",
    "    global fib_memo\n",
    "    memo = Dict()\n",
    "    function fib_memo(n::BigInt)\n",
    "        get!(memo, n) do \n",
    "            fib(n)\n",
    "        end\n",
    "    end\n",
    "end\n",
    "\n",
    "fib_memo(n::Int) = fib_memo(BigInt(n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@benchmark fib_memo(12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-Threading with Memoizing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "let \n",
    "    global fib_memo_mt\n",
    "    global fib_mt\n",
    "    memo = Dict()\n",
    "    function fib_memo_mt(n::BigInt)\n",
    "        get!(memo, n) do \n",
    "            fib_mt(n)\n",
    "        end\n",
    "    end\n",
    "end\n",
    "\n",
    "fib_memo_mt(n::Int) = fib_memo_mt(BigInt(n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@benchmark fib_memo_mt(12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looping with Multi-Threading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in 1:10\n",
    "    println(Threads.threadid())\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For loops where each iteration depends on no other iteration, we can easily parallelize with `Threads.@threads`. Note that the execution order is not guaranteed and simultaneously modifying the same objects can lead to inconsistent state and nasty race conditions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Threads.@threads for i in 1:10\n",
    "    println(Threads.threadid())\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Random.seed!(42)\n",
    "for i in 1:10\n",
    "    print(randstring(1))\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Random.seed!(42)\n",
    "Threads.@threads for i in 1:10\n",
    "    print(randstring(1))\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that both the sequence and character differ! In order to guarantee thread safety, Julia creates a freshly seeded copy of the default RNG for every thread, which means that we're drawing different random numbers for the extra threads. \n",
    "\n",
    "Let's wrap this up in a function and show how to make it multithreaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "function argle_bargle(draws; seed=42, use_threads=false)\n",
    "    val = []\n",
    "    Random.seed!(seed)\n",
    "    for i in 1:draws\n",
    "        append!(val,randstring(1))\n",
    "    end\n",
    "    val\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "join(argle_bargle(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default Julia RNG is the MersenneTwister, so we can just create our own RNG and share it among threads. However, random number generation is not *atomic* (roughly \"an instruction/operation with no intermediate states\") because a lot of calculations go into generating the next draw. In other words, we cannot allow two threads to access an RNG at the same time because they may leave the RNG in an inconsistent state. Moreover, when we access an RNG, we often ask it to generate lots of numbers (e.g. the random noise for all observations in a simulation), and so there is an additional level of not being atomic -- each draw is not atomic and we're asking for lots of draws. So we create a *lock* that blocks two threads from accessing the RNG at once.\n",
    "\n",
    "Anything that has *side effects* -- in Julia indicated by convention with a `!` in the name is by definition not atomic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "function argle_bargle(draws; seed=42, use_threads=false)\n",
    "    val = []\n",
    "    rng = MersenneTwister(seed)\n",
    "    if use_threads\n",
    "        rnglock = ReentrantLock()\n",
    "        Threads.@threads for i in 1:draws\n",
    "            lock(rnglock)\n",
    "            s = randstring(rng,1)\n",
    "            append!(val,s)\n",
    "            unlock(rnglock)\n",
    "        end\n",
    "    else\n",
    "        for i in 1:draws\n",
    "            s = randstring(rng,1)\n",
    "            append!(val,s)\n",
    "        end\n",
    "    end\n",
    "    val\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "join(argle_bargle(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "join(argle_bargle(10, use_threads=true))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is so simple compared to the granularity of the locking that the threads are largely scheduled sequentially, but it does highlight how to do the locking."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to this approach, Julia allows \"fast-forwarding\" the MersenneTwister. This can be useful, if you know how much to fast forward by."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Multi-Threading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a more advanced example, this is the naive version of `simulate_waldtests` without multi-threading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```julia\n",
    "function simulate_waldtests(\n",
    "    rng::AbstractRNG,\n",
    "    n::Integer,\n",
    "    morig::MixedModel{T};\n",
    "    β = morig.β,\n",
    "    σ = morig.σ,\n",
    "    θ = morig.θ,\n",
    ") where {T}\n",
    "    β = convert(Vector{T},β)\n",
    "    σ = T(σ)\n",
    "    θ = convert(Vector{T},θ)\n",
    "\n",
    "    nβ, mod = length(β), deepcopy(morig)\n",
    "    replicate(n, use_threads=false) do\n",
    "        mod = simulate!(rng, mod, β = β, σ = σ, θ = θ)\n",
    "        refit!(mod)\n",
    "        ct = coeftable(mod)\n",
    "        names = Tuple(Symbol.(ct.rownms))\n",
    "        (\n",
    "         β = NamedTuple{names}(ct.cols[1]),\n",
    "         se = NamedTuple{names}(ct.cols[2]),\n",
    "         z = NamedTuple{names}(ct.cols[3]),\n",
    "         p = NamedTuple{names}(ct.cols[4]),\n",
    "        )\n",
    "    end\n",
    "end\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And this is the version with threading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```julia\n",
    "function simulate_waldtests(\n",
    "    rng::AbstractRNG,\n",
    "    n::Integer,\n",
    "    morig::MixedModel{T};\n",
    "    β = morig.β,\n",
    "    σ = morig.σ,\n",
    "    θ = morig.θ,\n",
    "    use_threads = false,\n",
    ") where {T}\n",
    "    β = convert(Vector{T},β)\n",
    "    σ = T(σ)\n",
    "    θ = convert(Vector{T},θ)\n",
    "\n",
    "    nβ, m = length(β), deepcopy(morig)\n",
    "    # we need to do for in-place operations to work across threads\n",
    "    m_threads = [m]\n",
    "\n",
    "    if use_threads\n",
    "        Threads.resize_nthreads!(m_threads)\n",
    "    end\n",
    "\n",
    "    rnglock = ReentrantLock()\n",
    "    replicate(n, use_threads=use_threads) do\n",
    "        mod = m_threads[Threads.threadid()]\n",
    "        lock(rnglock)\n",
    "        mod = simulate!(rng, mod, β = β, σ = σ, θ = θ)\n",
    "        unlock(rnglock)\n",
    "        refit!(mod)\n",
    "        ct = coeftable(mod)\n",
    "        names = Tuple(Symbol.(ct.rownms))\n",
    "        (\n",
    "        # ct.testvalcol\n",
    "         se = NamedTuple{names}(ct.cols[2]),\n",
    "         z = NamedTuple{names}(ct.cols[3]),\n",
    "         p = NamedTuple{names}(ct.cols[4]),\n",
    "        )\n",
    "    end\n",
    "end\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributed Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quoting from the documentation,\n",
    "\n",
    ">  Julia provides a multiprocessing environment based on message passing ... Distributed programming in Julia is built on two primitives: remote references and remote calls. \n",
    "\n",
    "Distributed computing is more \"mainline\" / less experimental than multi-threading, but it does take a fair amount of finesse to really maximize its potential."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of workers available is determined by the startup flag `-p` in Julia. We can check it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import Distributed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Distributed.workers()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also add workers from within Julia, unlike with threads. Note that the default cluster is simply the local machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Distributed.addprocs()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Distributed.workers()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Distributed.rmprocs(Distributed.workers())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Distributed.workers()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Distributed.addprocs()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Distributed.nprocs(), Distributed.nworkers()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Distributed processing is bit more complex than multithreading with a lot more options (especially if you want to use a proper cluster), but the basics look a lot like multi-threading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "function fib_dp(n::BigInt)\n",
    "    if n < 2\n",
    "        return n\n",
    "    end\n",
    "    t = Distributed.@spawn fib(n - 2)\n",
    "    return fib(n - 1) + fetch(t)\n",
    "end\n",
    "\n",
    "fib_dp(n::Int) = fib_mt(BigInt(n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@benchmark fib_dp(15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fib(15), fib_mt(15),fib_dp(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loops are a bit more tricky because memory isn't automatically shared."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Distributed.@distributed for i in 1:10\n",
    "    println(Distributed.myid())\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val = zeros(Int, 20);\n",
    "for i = 1:length(val)\n",
    "    val[i] = i\n",
    "end\n",
    "show(val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val = zeros(Int, 20);\n",
    "Threads.@threads for i = 1:length(val)\n",
    "    val[i] = Threads.threadid()\n",
    "end\n",
    "show(val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val = zeros(Int, 20);\n",
    "Distributed.@distributed for i in 1:10\n",
    "    val[i] = Distributed.myid()\n",
    "end\n",
    "show(val)\n",
    "println()\n",
    "sleep(1)\n",
    "show(val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "using SharedArrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val = SharedArray{Int}(20);\n",
    "Distributed.@distributed for i in 1:10\n",
    "    val[i] = Distributed.myid()\n",
    "end\n",
    "\n",
    "show(val)\n",
    "println()\n",
    "sleep(1)\n",
    "show(val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val = SharedArray{Int}(20);\n",
    "Distributed.@sync Distributed.@distributed for i in 1:10\n",
    "    val[i] = Distributed.myid()\n",
    "end\n",
    "\n",
    "show(val)\n",
    "println()\n",
    "sleep(1)\n",
    "show(val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although this is very similar to threading, there are some differences that matter when you really get into performance tuning. For example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Threads.@spawn(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Distributed.@spawn(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're comfortable using `map`, then you should checkout [`pmap`](https://docs.julialang.org/en/v1/stdlib/Distributed/#Distributed.pmap), especially combined with [ProgressMeter](https://github.com/timholy/ProgressMeter.jl)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "using ProgressMeter\n",
    "using LinearAlgebra\n",
    "import Distributed.pmap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "M = Matrix{Float64}[rand(1000,1000) for i = 1:10];\n",
    "@showprogress pmap(svdvals, M);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More advanced features in distributed processing include `@async`, `@sync`,`@spawnat`, `fetchfrom`, `@everywhere`, `remotecall_wait`, `wait`, `put!`, `take!`, `isready`, `SharedArray`s, `DArrays`, and `ClusterManager`s. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I haven't yet gotten too far into how locking and synchronization works in distributed processing, so I don't have a good, optimized RNG example for you, sorry. In the future, I might add an example of distributed processing across a cluster for `parametricbootstrap` or `simulate_waldtests`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.3.1",
   "language": "julia",
   "name": "julia-1.3"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.3.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}