completed_tutorials/04-Electrophysiology Imported Tables.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with automated computations: Imported tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome back! In this session, we are going to continue working with the pipeline for the mouse electrophysiology example. \n",
    "\n",
    "In this session, we will learn to:\n",
    "\n",
    "* import neuron activity data from data files into an `Imported` table\n",
    "* automatically trigger computations for all missing entries with `populate`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First thing first, let's import `datajoint` again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datajoint as dj"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we are going to perform some computations, let's go ahead and import NumPy as well as Matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we would like to continue working with the tables we defined in the previous notebook. To do so, we would need the classes for each table: `Mouse` and `Session`. We can either redefine it here, but for your convenience, we have included the schema and table class definitions in a package called `tutorial_pipeline.mouse_session`, from which you can import the classes as well as the schema object. We will use the schema object again to define more tables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please enter DataJoint username: root\n",
      "Please enter DataJoint password: ········\n",
      "Connecting root@localhost:3306\n"
     ]
    }
   ],
   "source": [
    "from tutorial_pipeline.mouse_session import schema, Mouse, Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        \n",
       "        <style type=\"text/css\">\n",
       "            .Relation{\n",
       "                border-collapse:collapse;\n",
       "            }\n",
       "            .Relation th{\n",
       "                background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "                font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "            }\n",
       "            .Relation td{\n",
       "                padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "            }\n",
       "            .Relation tr:nth-child(odd){\n",
       "                background: #ffffff;\n",
       "            }\n",
       "            .Relation tr:nth-child(even){\n",
       "                background: #f3f1ff;\n",
       "            }\n",
       "            /* Tooltip container */\n",
       "            .djtooltip {\n",
       "            }\n",
       "            /* Tooltip text */\n",
       "            .djtooltip .djtooltiptext {\n",
       "                visibility: hidden;\n",
       "                width: 120px;\n",
       "                background-color: black;\n",
       "                color: #fff;\n",
       "                text-align: center;\n",
       "                padding: 5px 0;\n",
       "                border-radius: 6px;\n",
       "                /* Position the tooltip text - see examples below! */\n",
       "                position: absolute;\n",
       "                z-index: 1;\n",
       "            }\n",
       "            #primary {\n",
       "                font-weight: bold;\n",
       "                color: black;\n",
       "            }\n",
       "\n",
       "            #nonprimary {\n",
       "                font-weight: normal;\n",
       "                color: white;\n",
       "            }\n",
       "\n",
       "            /* Show the tooltip text when you mouse over the tooltip container */\n",
       "            .djtooltip:hover .djtooltiptext {\n",
       "                visibility: visible;\n",
       "            }\n",
       "        </style>\n",
       "        \n",
       "        <b>Experimental animals</b>\n",
       "            <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "            <table border=\"1\" class=\"Relation\">\n",
       "                <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">mouse_id</p>\n",
       "                                <span class=\"djtooltiptext\">Unique animal ID</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">dob</p>\n",
       "                                <span class=\"djtooltiptext\">date of birth</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">sex</p>\n",
       "                                <span class=\"djtooltiptext\">sex</span>\n",
       "                            </div> </th> </tr> </thead>\n",
       "                <tbody> <tr> <td>0</td>\n",
       "<td>2017-03-01</td>\n",
       "<td>M</td></tr><tr><td>1</td>\n",
       "<td>2016-11-19</td>\n",
       "<td>M</td></tr><tr><td>2</td>\n",
       "<td>2016-11-20</td>\n",
       "<td>unknown</td></tr><tr><td>5</td>\n",
       "<td>2016-12-25</td>\n",
       "<td>F</td></tr><tr><td>10</td>\n",
       "<td>2017-01-01</td>\n",
       "<td>F</td></tr><tr><td>11</td>\n",
       "<td>2017-01-03</td>\n",
       "<td>F</td></tr><tr><td>100</td>\n",
       "<td>2017-05-12</td>\n",
       "<td>F</td> </tr> </tbody>\n",
       "            </table>\n",
       "            \n",
       "            <p>Total: 7</p></div>\n",
       "            "
      ],
      "text/plain": [
       "*mouse_id    dob            sex        \n",
       "+----------+ +------------+ +---------+\n",
       "0            2017-03-01     M          \n",
       "1            2016-11-19     M          \n",
       "2            2016-11-20     unknown    \n",
       "5            2016-12-25     F          \n",
       "10           2017-01-01     F          \n",
       "11           2017-01-03     F          \n",
       "100          2017-05-12     F          \n",
       " (Total: 7)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Mouse()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        \n",
       "        <style type=\"text/css\">\n",
       "            .Relation{\n",
       "                border-collapse:collapse;\n",
       "            }\n",
       "            .Relation th{\n",
       "                background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "                font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "            }\n",
       "            .Relation td{\n",
       "                padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "            }\n",
       "            .Relation tr:nth-child(odd){\n",
       "                background: #ffffff;\n",
       "            }\n",
       "            .Relation tr:nth-child(even){\n",
       "                background: #f3f1ff;\n",
       "            }\n",
       "            /* Tooltip container */\n",
       "            .djtooltip {\n",
       "            }\n",
       "            /* Tooltip text */\n",
       "            .djtooltip .djtooltiptext {\n",
       "                visibility: hidden;\n",
       "                width: 120px;\n",
       "                background-color: black;\n",
       "                color: #fff;\n",
       "                text-align: center;\n",
       "                padding: 5px 0;\n",
       "                border-radius: 6px;\n",
       "                /* Position the tooltip text - see examples below! */\n",
       "                position: absolute;\n",
       "                z-index: 1;\n",
       "            }\n",
       "            #primary {\n",
       "                font-weight: bold;\n",
       "                color: black;\n",
       "            }\n",
       "\n",
       "            #nonprimary {\n",
       "                font-weight: normal;\n",
       "                color: white;\n",
       "            }\n",
       "\n",
       "            /* Show the tooltip text when you mouse over the tooltip container */\n",
       "            .djtooltip:hover .djtooltiptext {\n",
       "                visibility: visible;\n",
       "            }\n",
       "        </style>\n",
       "        \n",
       "        <b>Experiment session</b>\n",
       "            <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "            <table border=\"1\" class=\"Relation\">\n",
       "                <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">mouse_id</p>\n",
       "                                <span class=\"djtooltiptext\">Unique animal ID</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">session_date</p>\n",
       "                                <span class=\"djtooltiptext\">date</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">experiment_setup</p>\n",
       "                                <span class=\"djtooltiptext\">experiment setup ID</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">experimenter</p>\n",
       "                                <span class=\"djtooltiptext\">experimenter name</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">data_path</p>\n",
       "                                <span class=\"djtooltiptext\"></span>\n",
       "                            </div> </th> </tr> </thead>\n",
       "                <tbody> <tr> <td>0</td>\n",
       "<td>2017-05-15</td>\n",
       "<td>0</td>\n",
       "<td>Edgar Y. Walker</td>\n",
       "<td></td></tr><tr><td>0</td>\n",
       "<td>2017-05-19</td>\n",
       "<td>0</td>\n",
       "<td>Edgar Y. Walker</td>\n",
       "<td></td></tr><tr><td>5</td>\n",
       "<td>2017-01-05</td>\n",
       "<td>1</td>\n",
       "<td>Fabian Sinz</td>\n",
       "<td></td></tr><tr><td>100</td>\n",
       "<td>2017-05-25</td>\n",
       "<td>100</td>\n",
       "<td>Jacob Reimer</td>\n",
       "<td></td> </tr> </tbody>\n",
       "            </table>\n",
       "            \n",
       "            <p>Total: 4</p></div>\n",
       "            "
      ],
      "text/plain": [
       "*mouse_id    *session_date  experiment_set experimenter   data_path    \n",
       "+----------+ +------------+ +------------+ +------------+ +-----------+\n",
       "0            2017-05-15     0              Edgar Y. Walke              \n",
       "0            2017-05-19     0              Edgar Y. Walke              \n",
       "5            2017-01-05     1              Fabian Sinz                 \n",
       "100          2017-05-25     100            Jacob Reimer                \n",
       " (Total: 4)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `mouse_session.py` also fills each table with data to make sure we are all on the same page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing data from data files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall from the project description\n",
    "> * In each experimental session, you record electrical activity from a single neuron. You use recording equipment that produces separate data files for each neuron you recorded.\n",
    "\n",
    "Our recording equipment produces a data file for each neuron recorded. Since we record from one neuron per session, there should be one data file for each session."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the `data` directory, you will find `.npy` (saved NumPy array) files with names like `data_100_2017-05-25.npy`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you might have guessed, these are the data for the recording sessions in the `Session` table, and each file are named according to the `mouse_id` and `session_date` - the attributes of the primary keys - in the format `data_{mouse_id}_{session_date}.npy`.\n",
    "\n",
    "So `data_100_2017-05-25.npy` is the data for session identified by `mouse_id = 100` and `session_date = \"2017-05-25\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tutorial_pipeline import data_dir\n",
    "data_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looking at the data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a quick peak at the data file content."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's pick a session to load the data for. To do this we are going to first fetch the **primary key attributes** of `Session` as a list of dictionaries. We make use of the special `fetch('KEY')` syntax to achieve this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'mouse_id': 0, 'session_date': datetime.date(2017, 5, 15)},\n",
       " {'mouse_id': 0, 'session_date': datetime.date(2017, 5, 19)},\n",
       " {'mouse_id': 5, 'session_date': datetime.date(2017, 1, 5)},\n",
       " {'mouse_id': 100, 'session_date': datetime.date(2017, 5, 25)}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keys = Session.fetch('KEY')\n",
    "keys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any item in this list of keys can be used to uniquely identify a single session!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        \n",
       "        <style type=\"text/css\">\n",
       "            .Relation{\n",
       "                border-collapse:collapse;\n",
       "            }\n",
       "            .Relation th{\n",
       "                background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "                font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "            }\n",
       "            .Relation td{\n",
       "                padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "            }\n",
       "            .Relation tr:nth-child(odd){\n",
       "                background: #ffffff;\n",
       "            }\n",
       "            .Relation tr:nth-child(even){\n",
       "                background: #f3f1ff;\n",
       "            }\n",
       "            /* Tooltip container */\n",
       "            .djtooltip {\n",
       "            }\n",
       "            /* Tooltip text */\n",
       "            .djtooltip .djtooltiptext {\n",
       "                visibility: hidden;\n",
       "                width: 120px;\n",
       "                background-color: black;\n",
       "                color: #fff;\n",
       "                text-align: center;\n",
       "                padding: 5px 0;\n",
       "                border-radius: 6px;\n",
       "                /* Position the tooltip text - see examples below! */\n",
       "                position: absolute;\n",
       "                z-index: 1;\n",
       "            }\n",
       "            #primary {\n",
       "                font-weight: bold;\n",
       "                color: black;\n",
       "            }\n",
       "\n",
       "            #nonprimary {\n",
       "                font-weight: normal;\n",
       "                color: white;\n",
       "            }\n",
       "\n",
       "            /* Show the tooltip text when you mouse over the tooltip container */\n",
       "            .djtooltip:hover .djtooltiptext {\n",
       "                visibility: visible;\n",
       "            }\n",
       "        </style>\n",
       "        \n",
       "        <b>Experiment session</b>\n",
       "            <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "            <table border=\"1\" class=\"Relation\">\n",
       "                <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">mouse_id</p>\n",
       "                                <span class=\"djtooltiptext\">Unique animal ID</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">session_date</p>\n",
       "                                <span class=\"djtooltiptext\">date</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">experiment_setup</p>\n",
       "                                <span class=\"djtooltiptext\">experiment setup ID</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">experimenter</p>\n",
       "                                <span class=\"djtooltiptext\">experimenter name</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">data_path</p>\n",
       "                                <span class=\"djtooltiptext\"></span>\n",
       "                            </div> </th> </tr> </thead>\n",
       "                <tbody> <tr> <td>0</td>\n",
       "<td>2017-05-19</td>\n",
       "<td>0</td>\n",
       "<td>Edgar Y. Walker</td>\n",
       "<td></td> </tr> </tbody>\n",
       "            </table>\n",
       "            \n",
       "            <p>Total: 1</p></div>\n",
       "            "
      ],
      "text/plain": [
       "*mouse_id    *session_date  experiment_set experimenter   data_path    \n",
       "+----------+ +------------+ +------------+ +------------+ +-----------+\n",
       "0            2017-05-19     0              Edgar Y. Walke              \n",
       " (Total: 1)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ENTER YOUR CODE! - restrict session by an element of keys\n",
    "Session & keys[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take the first key, and generate the file name that corresponds to this session. Remember the `data_{mouse_id}_{session_date}.npy` filename convention!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mouse_id': 0, 'session_date': datetime.date(2017, 5, 15)}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key = keys[0]\n",
    "key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'data/data_0_2017-05-15.npy'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filename = '{data_dir}/data_{mouse_id}_{session_date}.npy'.format(**key, data_dir=data_dir)\n",
    "filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we have made use of Python's dictionary unpacking and `format` method on strings to generate the filename from the `key`.\n",
    "\n",
    "Finally, let's load the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.load(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at its content..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 3.57887412e-01,  4.47531558e-01,  1.96412988e-01,\n",
       "         3.91114491e-01,  1.76695176e-01,  3.80088116e-01,\n",
       "         9.36353656e-02,  1.10668001e-02,  1.14509759e-01,\n",
       "         1.65947705e-01,  1.02058476e-01,  1.15037690e-01,\n",
       "         2.18115225e-01,  1.34076355e-01,  7.74560696e-02,\n",
       "         9.19639027e-02, -7.60109670e-03,  1.12814011e+00,\n",
       "         9.48711628e-01,  7.77781245e-01,  5.46847747e-01,\n",
       "         3.91930226e-01,  2.19542222e-01,  3.92001858e-01,\n",
       "         3.17729202e-01,  2.97934834e-01,  1.85589403e-01,\n",
       "         3.04215179e-01,  8.75626287e-02, -1.76018214e-02,\n",
       "         1.43719730e-01, -1.28348781e-01, -4.38344653e-02,\n",
       "         1.57370375e-01,  1.70233992e-01,  8.98556797e-01,\n",
       "         9.36896773e-01,  1.67418776e+00,  2.48160598e+00,\n",
       "         1.89737445e+00,  1.45746470e+00,  1.32137967e+00,\n",
       "         1.05349900e+00,  1.88503636e+00,  1.53324759e+00,\n",
       "         1.34566597e+00,  1.93924029e+00,  1.58839106e+00,\n",
       "         1.26074176e+00,  1.21973014e+00,  9.80495008e-01,\n",
       "         7.66163859e-01,  1.59531686e+00,  1.10087015e+00,\n",
       "         1.15513304e+00,  8.56314992e-01,  7.37593259e-01,\n",
       "         6.50912336e-01,  3.78222011e-01,  3.33381961e-01,\n",
       "         3.36530722e-01,  2.72070063e-01,  2.21307773e-02,\n",
       "         1.27166159e-01,  1.02442286e-01,  7.17979754e-02,\n",
       "         1.24111799e-01,  2.95003486e-02,  5.81101732e-02,\n",
       "         7.83296477e-02,  1.07236342e-01, -7.71500819e-02,\n",
       "        -4.03102332e-02,  9.32218744e-02, -1.39751021e-01,\n",
       "         4.34990973e-02, -3.62238166e-02,  5.65840332e-02,\n",
       "         5.83848056e-03, -1.86444246e-04,  5.48341851e-02,\n",
       "         5.23098255e-03, -3.18617746e-02, -1.09971236e-01,\n",
       "        -1.19861696e-01,  8.11585903e-02,  5.25812482e-02,\n",
       "        -5.30601889e-02,  5.57417328e-02,  1.53998941e-02,\n",
       "        -3.64440351e-02, -2.59108524e-01,  1.20934701e-02,\n",
       "        -7.09599743e-02, -5.51095762e-02, -1.35461062e-01,\n",
       "         4.75431948e-02,  6.44133300e-02,  1.50885861e-01,\n",
       "        -1.46488851e-01, -9.53648657e-02,  7.59548599e-03,\n",
       "        -1.44054364e-01, -8.78515921e-02, -1.42428227e-01,\n",
       "        -3.45118278e-03, -1.26940714e-01, -9.20832554e-02,\n",
       "         3.68553737e-02, -8.35685105e-02,  1.60588415e-02,\n",
       "        -1.11976400e-01, -1.07021363e-01, -1.43319906e-02,\n",
       "        -2.41195390e-02, -3.78984606e-03,  6.26503649e-02,\n",
       "         6.73261037e-02,  2.42722828e-02,  4.78000884e-02,\n",
       "         1.87360494e-02,  3.77974599e-02, -1.59249774e-01,\n",
       "        -1.14799869e-01,  4.07530128e-02,  1.17039319e-02,\n",
       "         6.04455289e-02,  1.36546722e-01,  3.24868542e-02,\n",
       "        -2.02990620e-01, -1.85726063e-01,  5.95638848e-03,\n",
       "        -1.62046808e-01,  7.36242620e-02, -7.53509144e-02,\n",
       "         7.23394806e-02,  1.24649397e-01,  1.33367848e-01,\n",
       "         1.49916663e-01,  7.09169055e-02,  5.54171809e-02,\n",
       "        -1.70262178e-02,  1.51023527e-01, -5.86649941e-02,\n",
       "         2.07455481e-01,  2.43959267e-01,  8.83507209e-01,\n",
       "         8.55214006e-01,  5.76237481e-01,  5.52699744e-01,\n",
       "         5.67923229e-01,  5.26204825e-01,  1.51617555e+00,\n",
       "         9.54140744e-01,  1.00554631e+00,  7.66225147e-01,\n",
       "         6.92448058e-01,  5.16811992e-01,  3.90037626e-01,\n",
       "         1.58590830e-01,  3.30612987e-01,  1.72983877e-01,\n",
       "         1.28256029e-01,  8.26691865e-02, -4.49560014e-02,\n",
       "         2.43301325e-01,  2.29370998e-01,  1.06948655e+00,\n",
       "         8.22995772e-01,  8.25181132e-01,  6.73353030e-01,\n",
       "         4.99403896e-01,  4.65513637e-01,  3.37727245e-01,\n",
       "         2.48252812e-01,  1.65315878e-01,  2.74795374e-01,\n",
       "         5.83060391e-02,  1.27371795e-01, -1.17808927e-02,\n",
       "        -5.55786029e-03,  2.11406405e-01,  5.06522488e-02,\n",
       "        -1.32739957e-01,  1.14380044e+00,  1.89665344e+00,\n",
       "         1.40624390e+00,  1.24860249e+00,  9.67295548e-01,\n",
       "         9.02500744e-01,  5.92321135e-01,  4.75769619e-01,\n",
       "         4.87856821e-01,  4.38424136e-01,  2.88419750e-01,\n",
       "         4.40335548e-01,  1.33597568e-01,  1.91711553e-01,\n",
       "         1.22081460e-01,  2.06724114e-01, -1.55619770e-02,\n",
       "         1.71480833e-01,  3.83268407e-02,  1.94964042e-01,\n",
       "         5.51857012e-02,  1.97124745e-02,  8.21195929e-02,\n",
       "         1.03844219e-01, -3.18097486e-01,  7.02687913e-02,\n",
       "        -7.25411944e-02,  2.14994159e-01, -2.28972464e-01,\n",
       "        -2.18239963e-03,  4.52749496e-02,  1.23876523e-01,\n",
       "        -1.12786726e-01, -1.68840688e-01,  1.09262059e-02,\n",
       "         8.87628064e-04, -7.79933396e-02, -1.19261199e-02,\n",
       "        -2.80171894e-02,  1.68434803e-01, -2.48579065e-02,\n",
       "         1.02382818e-01,  1.53459838e-01, -6.62972617e-02,\n",
       "         1.57506657e-01, -9.38050260e-02, -3.86519361e-03,\n",
       "         5.79287013e-02, -1.32487716e-01,  1.07614132e-02,\n",
       "         1.06144573e+00,  1.78822809e+00,  1.42992074e+00,\n",
       "         1.20228847e+00,  2.01093206e+00,  1.56718338e+00,\n",
       "         1.40102371e+00,  1.03295413e+00,  7.41122788e-01,\n",
       "         6.61526796e-01,  4.54307411e-01,  5.88944641e-01,\n",
       "         4.42579987e-01,  5.34835898e-01,  3.91442415e-01,\n",
       "         5.80988295e-02,  1.56439534e-01,  2.58719019e-01,\n",
       "        -1.15456059e-01,  6.57661377e-02,  1.17996440e-01,\n",
       "        -6.02754374e-02,  6.95786749e-02,  1.14241287e-01,\n",
       "         1.53877204e-01, -3.50059527e-02,  1.55834390e-01,\n",
       "        -1.68767662e-02,  1.23508887e-01, -1.02828127e-01,\n",
       "         1.03055306e+00,  9.56828891e-01,  6.26103277e-01,\n",
       "         4.99778832e-01,  4.08081478e-01,  4.30054527e-01,\n",
       "         3.02228063e-01,  3.69730534e-01,  2.40954326e-01,\n",
       "         4.56641352e-02,  1.63000709e-01,  1.18606679e+00,\n",
       "         9.61793771e-01,  8.96002396e-01,  5.34468548e-01,\n",
       "         1.57483007e+00,  2.09870766e+00,  1.82291133e+00,\n",
       "         1.63844619e+00,  1.21440231e+00,  1.05446930e+00,\n",
       "         8.93141567e-01,  6.60007561e-01,  6.71775232e-01,\n",
       "         1.50441168e+00,  1.14440896e+00,  1.02966220e+00,\n",
       "         7.52415220e-01,  6.67011874e-01,  3.46874670e-01,\n",
       "         2.68413022e-01,  2.77949691e-01,  3.34664046e-01,\n",
       "         4.03215085e-01,  2.53651907e-01,  1.07638536e-01,\n",
       "         1.05343159e-01,  8.88901142e-02,  1.80015728e-01,\n",
       "         1.98767886e-03,  2.02111090e-02, -9.31692393e-02,\n",
       "        -4.47384663e-02,  1.56458083e-01,  2.01720297e-01,\n",
       "         2.03542534e-01, -2.14269428e-01,  2.80797183e-02,\n",
       "         3.25476819e-02,  5.35097248e-02, -1.46752968e-01,\n",
       "        -2.77763791e-01,  9.39191325e-02, -8.00522257e-02,\n",
       "         7.86411156e-02, -1.31305393e-01, -1.18440907e-01,\n",
       "        -2.11471791e-01,  1.83345846e-01,  1.14036980e-02,\n",
       "         3.12572490e-02,  1.67251288e-01,  4.50737025e-02,\n",
       "         8.50116891e-02, -1.05570136e-01, -3.91225599e-02,\n",
       "         4.19435506e-02,  9.18295673e-02,  1.00948351e-01,\n",
       "        -2.69381523e-02, -5.80990336e-02,  1.78913698e-02,\n",
       "        -2.19932511e-01,  1.00536095e-02,  7.50087514e-02,\n",
       "         5.71439648e-02, -7.05402068e-02, -5.04720874e-02,\n",
       "        -2.02845248e-01,  7.49185791e-02, -5.45357043e-03,\n",
       "         1.27005895e-02, -3.52048059e-02,  7.63334673e-02,\n",
       "         1.13931022e-01,  2.21661258e-02,  2.64159696e-01,\n",
       "         1.08020316e-01, -1.65623301e-01,  2.70601016e-02,\n",
       "         4.63893087e-02,  5.42843157e-02, -3.47460497e-02,\n",
       "         5.83374400e-02,  3.20027719e-02,  9.75737361e-02,\n",
       "        -5.17204965e-02,  2.49865141e-02,  4.30739172e-02,\n",
       "        -1.55561967e-01, -4.40168629e-02, -1.25899451e-01,\n",
       "        -5.39137396e-02,  6.85689395e-02, -7.06141866e-03,\n",
       "        -2.29774809e-01, -1.22652670e-01,  4.91485274e-02,\n",
       "        -1.17181342e-01, -4.67479206e-02,  1.31136674e-01,\n",
       "        -2.16766438e-01,  9.24738157e-01,  8.30514746e-01,\n",
       "         8.36875466e-01,  5.89253038e-01,  4.09113568e-01,\n",
       "         2.66889234e-01,  3.18329291e-01,  2.90857104e-01,\n",
       "         3.99022616e-01,  1.29407638e-01,  2.26561258e-01,\n",
       "         1.40736112e-01,  1.74887173e-01,  1.73266324e-01,\n",
       "         3.76184057e-02, -6.00556778e-02, -2.01416090e-01,\n",
       "        -1.67598812e-03, -7.12499296e-03,  1.38259933e-01,\n",
       "        -1.34984095e-01,  1.14370726e+00,  7.05035228e-01,\n",
       "         6.93680054e-01,  4.93018199e-01,  3.01884856e-01,\n",
       "         3.21853060e-01,  3.21720921e-01,  1.89495940e-01,\n",
       "         4.79519880e-01,  1.29843248e-01,  6.39370486e-02,\n",
       "         1.80744947e-01,  8.32444042e-02,  7.96862651e-02,\n",
       "         8.80065818e-02,  9.51188930e-02,  1.01090543e-01,\n",
       "         2.05419796e-01, -2.98784768e-02,  1.45803916e-01,\n",
       "        -2.98838255e-02,  1.35847269e-01, -3.03913537e-02,\n",
       "         4.96040245e-02, -1.03077876e-01, -1.22083976e-01,\n",
       "        -9.84669206e-02, -3.80553913e-02, -1.28106091e-01,\n",
       "         2.30966517e-02, -1.11285441e-01,  3.60091188e-03,\n",
       "         1.99334515e-02, -2.56325565e-02, -1.01025035e-01,\n",
       "        -7.75270782e-02,  1.04298520e-01,  2.41849780e-03,\n",
       "         4.48712026e-02,  2.52274241e-01, -6.00374401e-02,\n",
       "         5.40367599e-02,  1.73639411e-01, -7.86343607e-03,\n",
       "         6.22913351e-02,  2.53208560e-02, -1.69087628e-01,\n",
       "        -1.65132452e-01,  2.86023053e-02, -9.29146792e-02,\n",
       "        -6.11378213e-02,  2.04881973e-01,  1.88225440e-01,\n",
       "        -5.91635886e-02, -5.79665004e-02,  1.44294351e-01,\n",
       "         4.46050806e-02,  7.94540969e-03, -6.35426095e-02,\n",
       "        -1.15997227e-01,  1.83191753e-02,  5.28225263e-02,\n",
       "        -1.08532350e-01,  1.44496792e-01,  1.43302657e-01,\n",
       "        -1.46524350e-01,  1.18188356e+00,  9.03639320e-01,\n",
       "         6.04454661e-01,  4.49513478e-01,  3.83200563e-01,\n",
       "         5.06436477e-01,  3.49265581e-01,  3.99738778e-01,\n",
       "         2.45887692e-01,  2.81484301e-01,  1.77785305e-01,\n",
       "         1.09703265e-02,  3.70530398e-02,  1.31330082e-01,\n",
       "         8.71425640e-02,  3.33857832e-01, -7.12413968e-02,\n",
       "        -8.97154629e-02,  7.40730721e-02,  5.97128245e-02,\n",
       "         2.34869128e-01, -3.20819416e-02,  2.65632944e-02,\n",
       "        -6.07013335e-02, -1.27002758e-01,  2.16984986e-02,\n",
       "         9.03023410e-03,  2.20839362e-01, -8.91938650e-02,\n",
       "         2.33953850e-01,  1.43573346e-01,  2.75152230e-01,\n",
       "         1.28656643e-01, -1.16480260e-02,  9.03293415e-04,\n",
       "        -1.18839774e-01,  7.86330161e-02,  4.00007325e-02,\n",
       "        -9.80397383e-02, -4.42695682e-02, -1.49883960e-02,\n",
       "        -1.69976818e-01, -7.78312668e-03, -1.94404123e-02,\n",
       "        -8.03773902e-02, -6.05579523e-02,  1.95069803e-02,\n",
       "        -4.52126912e-02,  1.74422122e-01, -1.99238260e-01,\n",
       "        -3.91236052e-02,  5.89841644e-02, -6.67758085e-02,\n",
       "        -2.73644241e-02,  2.19098411e-01,  4.46983268e-02,\n",
       "        -3.66252078e-02,  4.43283833e-02,  2.25697557e-02,\n",
       "         1.95433005e-01,  4.09668282e-02, -5.43391320e-02,\n",
       "        -4.33887655e-02,  1.47436220e-01,  1.65408829e-01,\n",
       "        -9.09196975e-02,  1.07774571e-01,  8.58332038e-02,\n",
       "         4.33849188e-03,  1.53411792e-01, -3.97305805e-02,\n",
       "        -2.14871074e-02, -6.97095042e-02,  3.00432966e-02,\n",
       "        -3.16463544e-02,  3.07840191e-03, -2.16733667e-02,\n",
       "        -3.18560339e-02,  3.66698288e-02, -3.43363193e-02,\n",
       "         2.06746714e-01, -7.60502887e-02, -1.71189864e-01,\n",
       "         1.06735423e-01, -3.61907100e-02, -1.88581197e-01,\n",
       "         1.55249578e-01,  6.59242205e-03, -4.46976133e-02,\n",
       "         1.86226099e-01, -1.49095939e-01, -2.47424070e-02,\n",
       "         1.07958794e-01,  7.10083147e-02,  5.70557055e-02,\n",
       "         8.96181297e-03, -2.11594738e-01, -5.03600676e-02,\n",
       "        -2.56128830e-02, -4.02507671e-02, -1.99935220e-01,\n",
       "         1.62758261e-02, -2.10041565e-02,  3.83952841e-02,\n",
       "        -1.20534363e-02, -5.10847887e-02, -4.33253439e-02,\n",
       "        -1.88213705e-02,  4.58028139e-02, -1.80158629e-02,\n",
       "         1.43620301e-01, -1.70544931e-01,  1.77761139e-03,\n",
       "        -2.08095349e-02,  3.34326742e-02,  4.40900132e-02,\n",
       "         6.77012418e-02, -9.25022511e-02,  8.56213543e-02,\n",
       "        -1.64681050e-02,  1.51338327e-02,  5.40416644e-02,\n",
       "         1.33405086e-02, -6.50942134e-02, -4.35238219e-03,\n",
       "         6.74543989e-02,  1.83795805e-02, -8.89084565e-03,\n",
       "        -8.38116481e-02, -8.85186574e-02, -8.22233688e-02,\n",
       "         1.63380941e-01,  1.93323312e-01, -2.52080190e-03,\n",
       "         1.55618202e-01, -3.48718916e-02, -4.39154177e-03,\n",
       "        -1.50650829e-01, -3.85879559e-02,  1.23910600e-01,\n",
       "         1.26416145e-01,  4.57916224e-02,  8.45197602e-02,\n",
       "         6.91956186e-02,  1.29578141e-01, -3.42267972e-02,\n",
       "         6.38280360e-02, -1.71677496e-02, -1.92718154e-01,\n",
       "         1.16080879e+00,  9.21546435e-01,  5.16635067e-01,\n",
       "         7.23468589e-01,  3.09147097e-01,  4.49592539e-01,\n",
       "         2.62198787e-01,  1.89167293e-01,  6.80711173e-02,\n",
       "         1.16969270e-01,  1.50577864e-01,  3.10725745e-02,\n",
       "        -9.74933170e-03, -1.04268235e-01,  6.56952998e-02,\n",
       "         1.40981993e-01, -5.38736937e-02,  9.89558899e-02,\n",
       "        -4.52553764e-02, -7.47028216e-02, -2.45633731e-02,\n",
       "         4.91456086e-03,  2.24777465e-02,  5.19085415e-02,\n",
       "        -7.12450557e-02,  8.96366331e-02,  5.53691200e-02,\n",
       "        -7.74717503e-02, -6.71054350e-02, -6.39028129e-02,\n",
       "         3.12679383e-02, -1.84819956e-01,  2.07870708e-02,\n",
       "        -1.31116132e-02,  2.76283840e-02,  4.87966338e-02,\n",
       "         9.84081330e-02, -6.95627406e-02,  2.36105975e-01,\n",
       "         1.45560427e-02,  8.64798612e-02, -3.32754493e-03,\n",
       "         4.40784273e-02,  8.92873175e-03, -7.64205487e-03,\n",
       "         6.77330030e-02, -6.38034776e-02,  2.32453845e-02,\n",
       "        -1.76792583e-02, -1.22653199e-01,  2.60944692e-01,\n",
       "        -8.19576302e-03,  1.21502703e-01, -1.17436798e-01,\n",
       "         5.91152919e-02,  1.45069334e-02, -2.77748199e-02,\n",
       "         8.70198108e-02,  1.09750973e+00,  7.82291035e-01,\n",
       "         7.21941782e-01,  4.34205229e-01,  4.95265468e-01,\n",
       "         4.64622108e-01,  2.74796291e-01,  2.18068847e-01,\n",
       "         3.26694544e-01,  1.82281257e-01,  5.71407916e-02,\n",
       "         1.35026995e-01,  6.95359173e-02,  1.19570987e-01,\n",
       "         2.85917443e-02,  1.68845114e-01,  2.88802329e-02,\n",
       "         1.38069548e-01,  4.18650358e-02, -1.33298268e-01,\n",
       "        -5.40673322e-02, -2.12729776e-02, -6.04156952e-02,\n",
       "         1.01418967e-01,  6.41443313e-02,  3.63175454e-02,\n",
       "         8.56384845e-02, -1.15120677e-02,  1.10539191e-01,\n",
       "        -1.24283472e-01, -1.81287322e-01, -1.05492936e-01,\n",
       "        -1.39418774e-01, -8.96223894e-02,  5.29831286e-02,\n",
       "        -9.06341080e-02,  1.11775639e-01,  2.52898789e-01,\n",
       "         1.37988335e-01, -1.99189622e-03,  7.86036381e-02,\n",
       "         6.69568198e-02, -1.39980384e-02, -8.60498166e-02,\n",
       "         9.52417978e-02,  1.19438557e-02,  3.25813543e-02,\n",
       "         6.52716977e-02, -1.01625274e-02,  1.38016286e-01,\n",
       "         1.68098374e-01,  2.08439386e-02,  5.07228859e-02,\n",
       "         1.64968265e-01, -1.94238088e-01, -2.26970816e-01,\n",
       "         1.90859273e-01,  1.55852829e-02,  9.92850491e-02,\n",
       "         4.34517009e-02, -6.93774354e-03,  2.37587998e-02,\n",
       "        -1.10832949e-01,  1.28747269e-01, -8.28914026e-02,\n",
       "         1.84212900e-01,  3.88899986e-02,  2.75027529e-02,\n",
       "        -4.41476612e-02,  9.83779242e-03, -7.59881145e-02,\n",
       "        -5.62181424e-02,  4.71012725e-02, -2.83273040e-02,\n",
       "        -8.95039250e-02, -1.73206060e-01,  8.04575793e-02,\n",
       "         4.89516889e-02, -5.36636280e-02,  1.11401613e-01,\n",
       "         7.41568054e-02, -1.58587913e-02,  1.38660466e-02,\n",
       "         9.15840051e-01,  9.35520924e-01,  8.20295289e-01,\n",
       "         4.16726598e-01,  4.10924305e-01,  4.29348959e-01,\n",
       "         1.88910731e-01,  2.71345351e-01,  3.36252176e-01,\n",
       "         7.99709853e-02,  1.18817313e+00,  1.12547896e+00,\n",
       "         7.10084698e-01,  5.19828662e-01,  4.90804498e-01,\n",
       "         3.38881990e-01,  3.76448904e-01,  1.18645994e+00,\n",
       "         1.06435236e+00,  1.94919081e+00,  1.41683039e+00,\n",
       "         1.43564448e+00,  1.04081010e+00,  7.90455203e-01,\n",
       "         5.63441762e-01,  4.12188514e-01,  4.14429943e-01,\n",
       "         3.94528788e-01,  1.34387624e+00,  9.47544414e-01,\n",
       "         1.02847705e+00,  7.18964295e-01,  5.60308504e-01,\n",
       "         5.55145624e-01,  3.72224551e-01,  2.14687813e-01,\n",
       "         3.13142646e-01,  2.50581325e-01,  2.09818408e-02,\n",
       "         1.49008637e-01,  2.03570380e-01,  5.05068803e-02,\n",
       "         1.20428405e-01,  1.11461852e+00,  8.81700992e-01,\n",
       "         1.81056078e+00,  1.52936936e+00,  1.16525616e+00,\n",
       "         8.56957933e-01,  9.04530004e-01,  5.83086508e-01,\n",
       "         6.42397646e-01,  3.49120161e-01,  3.95190201e-01,\n",
       "         3.04481134e-01,  3.28985466e-01,  3.22194130e-01,\n",
       "        -4.78729019e-02,  3.06232461e-01, -4.02626838e-02,\n",
       "         1.23166186e-01,  4.93381103e-02,  1.10558060e-01,\n",
       "         2.07875455e-02,  2.31902493e-01, -1.12148445e-01,\n",
       "        -1.28066781e-02, -4.42326632e-02, -2.45775081e-02,\n",
       "        -1.04753587e-01,  8.97585073e-02, -2.27588842e-02,\n",
       "         8.57118601e-02,  5.94292811e-03,  1.68235395e-01,\n",
       "         1.74473514e-01,  8.15705527e-02, -1.62542119e-01,\n",
       "        -1.50153441e-01, -3.82194543e-01,  7.89967120e-02,\n",
       "         1.24402821e-01, -7.45099363e-02,  1.02042814e-01,\n",
       "        -1.54368942e-02,  1.51022438e-01,  5.45528677e-02,\n",
       "         2.11612947e-01,  3.05863774e-02,  1.13577596e-01,\n",
       "        -9.28138132e-02,  1.05944320e+00,  8.22694192e-01,\n",
       "         6.86775834e-01,  5.26383175e-01,  4.87463624e-01,\n",
       "         3.59415382e-01,  1.79998644e-01,  3.03695719e-01,\n",
       "         1.62269198e-01,  1.88029365e-01,  7.70099213e-02,\n",
       "         1.03689795e+00,  8.59575605e-01,  6.84051568e-01,\n",
       "         4.93971211e-01,  5.43776616e-01,  3.61937201e-01,\n",
       "         2.07496283e-01,  2.38354097e-01,  2.53068762e-01,\n",
       "         1.52493846e-01,  3.84866267e-01,  8.58807488e-02,\n",
       "         6.02818296e-02,  1.24288796e-01,  2.27447449e-01,\n",
       "         2.01378209e-02, -7.01849363e-02,  1.23180928e-01,\n",
       "        -4.31365196e-02, -1.05023995e-01,  1.57140085e-02,\n",
       "         7.78643730e-02,  1.16315021e-01, -1.95906107e-01,\n",
       "        -9.75358646e-03, -1.03878208e-01,  3.84609155e-02,\n",
       "         1.72760630e-01, -3.90920446e-02,  2.92722474e-03,\n",
       "        -1.35533373e-02,  1.13720826e-02, -5.35428824e-02,\n",
       "        -1.76178376e-01, -7.24951113e-02, -1.95280857e-02,\n",
       "         1.25993789e-01,  9.37632205e-02,  8.89435244e-02,\n",
       "         6.00251479e-02, -4.69134844e-02, -4.78302608e-02,\n",
       "        -9.85106634e-02,  2.43915467e-01,  1.59116000e-01,\n",
       "        -1.77494253e-01,  7.37983421e-02, -8.42027434e-02,\n",
       "        -1.41031791e-01, -1.17614325e-01, -1.72387392e-01,\n",
       "         1.13488435e+00,  7.94030330e-01,  7.05362490e-01,\n",
       "         4.25140173e-01,  3.77941998e-01,  4.51229738e-01,\n",
       "         3.91577648e-01,  2.21325503e-01,  3.04573167e-01,\n",
       "         1.56762722e-01,  1.35208947e-01,  1.72394809e-01,\n",
       "         8.77570143e-04, -4.35287122e-02, -1.72645376e-01,\n",
       "         1.35352750e-01,  2.53995982e-02,  3.90884089e-02,\n",
       "         5.71430012e-02,  1.80028326e-01,  2.13436539e-02,\n",
       "         1.80930730e-01,  1.03541707e-01,  8.41700237e-01,\n",
       "         8.41336519e-01,  6.15004166e-01,  1.66401572e+00,\n",
       "         1.24331519e+00,  1.10549354e+00,  7.18672227e-01,\n",
       "         7.10294950e-01,  4.68031444e-01,  3.07723842e-01,\n",
       "         3.85116194e-01,  3.39574633e-01,  8.18889780e-02,\n",
       "         1.33506454e-01,  1.14196508e-02, -5.71529176e-02,\n",
       "         6.48490146e-02, -1.23334789e-02,  2.30623539e-01,\n",
       "         5.12007823e-02,  8.73800247e-02, -9.04408350e-03,\n",
       "        -6.99598011e-02,  6.78200687e-03,  1.15865457e-01,\n",
       "        -4.18379703e-02, -1.09293334e-01,  3.59053268e-02,\n",
       "        -8.22174747e-02,  5.11019451e-03, -6.16000717e-02,\n",
       "         5.44727222e-03, -9.78321940e-02,  7.70249179e-02,\n",
       "        -5.88816408e-02, -6.53482040e-02, -4.06193822e-02,\n",
       "        -1.53111574e-01,  6.09304478e-02,  1.30095832e-01,\n",
       "         3.93270237e-02, -1.36006509e-02,  1.24505657e-01,\n",
       "        -8.51880316e-02, -1.04119637e-02,  6.79276317e-02,\n",
       "         4.43370379e-02, -3.44310185e-02, -1.46022386e-01,\n",
       "         7.97515407e-02, -1.55983167e-02,  2.59952844e-02,\n",
       "        -1.92743149e-01, -1.14546115e-01,  1.68661011e-01,\n",
       "         7.21492352e-02, -8.15528413e-02, -1.26248020e-01,\n",
       "         5.42392992e-04, -3.40266491e-02, -2.67852136e-02,\n",
       "        -1.86944094e-02,  1.51646655e-01, -2.88587981e-02,\n",
       "         7.91561911e-02, -1.49274034e-01, -7.83390380e-02,\n",
       "        -6.39837999e-02,  7.98993185e-03,  5.00516600e-02,\n",
       "        -8.89976239e-02]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and check the shape of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 1000)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So this particular file contains a NumPy array of size 1 x 1000. This represents a (simulated) recording of raw electric activity from neuron(s) (1st dimension) over 1000 time bins (2nd dimension)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the table for recorded neurons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now would like to have all these recorded `Neuron` represented and stored in our data pipeline.\n",
    "\n",
    "Since there may be multiple neurons recorded from each session, a `Neuron` can be uniquely identified by knowing the `Session` it was recorded in, as well as its `neuron_id`. For each `Neuron`, we want to store the neural activity found in the data file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "@schema\n",
    "class Neuron(dj.Imported):\n",
    "    definition = \"\"\"\n",
    "    -> Session\n",
    "    neuron_id: int\n",
    "    ---\n",
    "    activity: longblob    # electric activity of the neuron\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the state of our pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<svg height=\"185pt\" viewBox=\"0.00 0.00 75.50 185.00\" width=\"76pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 181)\">\n<title>%3</title>\n<polygon fill=\"white\" points=\"-4,4 -4,-181 71.5,-181 71.5,4 -4,4\" stroke=\"none\"/>\n<!-- Mouse -->\n<g class=\"node\" id=\"node1\"><title>Mouse</title>\n<g id=\"a_node1\"><a xlink:title=\"mouse_id             \r\n------------------------------\r\ndob=null             \r\nsex=&quot;unknown&quot;        \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"59.75,-177 7.75,-177 7.75,-142 59.75,-142 59.75,-177\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"darkgreen\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"15.75\" y=\"-157.4\">Mouse</text>\n</a>\n</g>\n</g>\n<!-- Session -->\n<g class=\"node\" id=\"node3\"><title>Session</title>\n<g id=\"a_node3\"><a xlink:title=\"→ Mouse\r\nsession_date         \r\n------------------------------\r\nexperiment_setup     \r\nexperimenter         \r\ndata_path=&quot;&quot;         \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"63.25,-106 4.25,-106 4.25,-71 63.25,-71 63.25,-106\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"darkgreen\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"12.25\" y=\"-86.4\">Session</text>\n</a>\n</g>\n</g>\n<!-- Mouse&#45;&gt;Session -->\n<g class=\"edge\" id=\"edge1\"><title>Mouse-&gt;Session</title>\n<path d=\"M33.75,-141.797C33.75,-130.949 33.75,-116.867 33.75,-106.049\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- Neuron -->\n<g class=\"node\" id=\"node2\"><title>Neuron</title>\n<g id=\"a_node2\"><a xlink:title=\"→ Session\r\nneuron_id            \r\n------------------------------\r\nactivity             \r\n\">\n<ellipse cx=\"33.75\" cy=\"-17.5\" fill=\"#00007f\" fill-opacity=\"0.250980\" rx=\"33.5018\" ry=\"17.5\" stroke=\"#00007f\" stroke-opacity=\"0.250980\"/>\n<text fill=\"#00007f\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"14.75\" y=\"-15.4\">Neuron</text>\n</a>\n</g>\n</g>\n<!-- Session&#45;&gt;Neuron -->\n<g class=\"edge\" id=\"edge2\"><title>Session-&gt;Neuron</title>\n<path d=\"M33.75,-70.797C33.75,-59.9485 33.75,-45.8669 33.75,-35.0492\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n</g>\n</svg>",
      "text/plain": [
       "<datajoint.diagram.Diagram at 0x21e2c9bdb00>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ENTER YOUR CODE! - plot Diagram of the schema\n",
    "dj.Diagram(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We defined `activity` as a `longblob` so that it can store a NumPy array holding the electric activity over time. This NumPy array will be imported from the file corresponding to each neuron."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that our `Neuron` class inherits from `dj.Imported` instead of `dj.Manual` like others. This is because **this table's content will depend on data imported from an external file**. The `Manual` vs `Imported` are said to specify the **tier of the table**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataJoint table tiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In DataJoint, the tier of the table indicates **the nature of the data and the data source for the table**. So far we have encountered two table tiers: `Manual` and `Imported`, and we will encounter the two other major tiers in this session. \n",
    "\n",
    "DataJoint tables in `Manual` tier, or simply **Manual tables** indicate that its contents are **manually** entered by either experimenters or a recording system, and its content **do not depend on external data files or other tables**. This is the most basic table type you will encounter, especially as the tables at the beginning of the pipeline. In the Diagram, `Manual` tables are depicted by green rectangles.\n",
    "\n",
    "On the other hand, **Imported tables** are understood to pull data (or *import* data) from external data files, and come equipped with functionalities to perform this importing process automatically, as we will see shortly! In the Diagram, `Imported` tables are depicted by blue ellipses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<svg height=\"185pt\" viewBox=\"0.00 0.00 75.50 185.00\" width=\"76pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 181)\">\n<title>%3</title>\n<polygon fill=\"white\" points=\"-4,4 -4,-181 71.5,-181 71.5,4 -4,4\" stroke=\"none\"/>\n<!-- Mouse -->\n<g class=\"node\" id=\"node1\"><title>Mouse</title>\n<g id=\"a_node1\"><a xlink:title=\"mouse_id             \r\n------------------------------\r\ndob=null             \r\nsex=&quot;unknown&quot;        \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"59.75,-177 7.75,-177 7.75,-142 59.75,-142 59.75,-177\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"darkgreen\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"15.75\" y=\"-157.4\">Mouse</text>\n</a>\n</g>\n</g>\n<!-- Session -->\n<g class=\"node\" id=\"node3\"><title>Session</title>\n<g id=\"a_node3\"><a xlink:title=\"→ Mouse\r\nsession_date         \r\n------------------------------\r\nexperiment_setup     \r\nexperimenter         \r\ndata_path=&quot;&quot;         \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"63.25,-106 4.25,-106 4.25,-71 63.25,-71 63.25,-106\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"darkgreen\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"12.25\" y=\"-86.4\">Session</text>\n</a>\n</g>\n</g>\n<!-- Mouse&#45;&gt;Session -->\n<g class=\"edge\" id=\"edge1\"><title>Mouse-&gt;Session</title>\n<path d=\"M33.75,-141.797C33.75,-130.949 33.75,-116.867 33.75,-106.049\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- Neuron -->\n<g class=\"node\" id=\"node2\"><title>Neuron</title>\n<g id=\"a_node2\"><a xlink:title=\"→ Session\r\nneuron_id            \r\n------------------------------\r\nactivity             \r\n\">\n<ellipse cx=\"33.75\" cy=\"-17.5\" fill=\"#00007f\" fill-opacity=\"0.250980\" rx=\"33.5018\" ry=\"17.5\" stroke=\"#00007f\" stroke-opacity=\"0.250980\"/>\n<text fill=\"#00007f\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"14.75\" y=\"-15.4\">Neuron</text>\n</a>\n</g>\n</g>\n<!-- Session&#45;&gt;Neuron -->\n<g class=\"edge\" id=\"edge2\"><title>Session-&gt;Neuron</title>\n<path d=\"M33.75,-70.797C33.75,-59.9485 33.75,-45.8669 33.75,-35.0492\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n</g>\n</svg>",
      "text/plain": [
       "<datajoint.diagram.Diagram at 0x21e2c9bd400>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dj.Diagram(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing data into the `Imported` table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than filling out the content of the table manually using `insert1` or `insert` methods, we are going to make use of the `make` and `populate` logic that comes with `Imported` tables to automatically figure out what needs to be imported and perform the import!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `make` and `populate` methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Imported` table comes with a special method called `populate`. Let's try calling it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "ename": "NotImplementedError",
     "evalue": "Subclasses of AutoPopulate must implement the method `make`",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNotImplementedError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-16-effb7edceadd>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# ENTER YOUR CODE! - call `populate` on the table\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mNeuron\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpopulate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mC:\\ProgramData\\Anaconda3\\lib\\site-packages\\datajoint\\autopopulate.py\u001b[0m in \u001b[0;36mpopulate\u001b[1;34m(self, suppress_errors, return_exception_objects, reserve_jobs, order, limit, max_calls, display_progress, *restrictions)\u001b[0m\n\u001b[0;32m    157\u001b[0m                     \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_allow_insert\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    158\u001b[0m                     \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 159\u001b[1;33m                         \u001b[0mmake\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdict\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    160\u001b[0m                     \u001b[1;32mexcept\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mKeyboardInterrupt\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mSystemExit\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mException\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0merror\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    161\u001b[0m                         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\ProgramData\\Anaconda3\\lib\\site-packages\\datajoint\\autopopulate.py\u001b[0m in \u001b[0;36mmake\u001b[1;34m(self, key)\u001b[0m\n\u001b[0;32m     60\u001b[0m         \u001b[0mattributes\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;32mand\u001b[0m \u001b[0minserts\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mnew\u001b[0m \u001b[0mtuples\u001b[0m \u001b[0minto\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     61\u001b[0m         \"\"\"\n\u001b[1;32m---> 62\u001b[1;33m         \u001b[1;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'Subclasses of AutoPopulate must implement the method `make`'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     63\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     64\u001b[0m     \u001b[1;33m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNotImplementedError\u001b[0m: Subclasses of AutoPopulate must implement the method `make`"
     ]
    }
   ],
   "source": [
    "# ENTER YOUR CODE! - call `populate` on the table\n",
    "Neuron.populate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that `populate` call complained that a method called `make` is not implemented. Let me show a simple `make` method that will help elucidate what this is all about."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "@schema\n",
    "class Neuron(dj.Imported):\n",
    "    definition = \"\"\"\n",
    "    -> Session\n",
    "    neuron_id: int\n",
    "    ---\n",
    "    activity: longblob    # electric activity of the neuron\n",
    "    \"\"\"\n",
    "    def make(self, key): # `make` takes a single argument `key`\n",
    "        print('key is', key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's call `populate` again!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "key is {'mouse_id': 0, 'session_date': datetime.date(2017, 5, 15)}\n",
      "key is {'mouse_id': 0, 'session_date': datetime.date(2017, 5, 19)}\n",
      "key is {'mouse_id': 5, 'session_date': datetime.date(2017, 1, 5)}\n",
      "key is {'mouse_id': 100, 'session_date': datetime.date(2017, 5, 25)}\n"
     ]
    }
   ],
   "source": [
    "# ENTER YOUR CODE! - call `populate` on the table\n",
    "Neuron.populate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you call `populate` on an `Imported` table, this triggers DataJoint to look up all tables that the `Imported` table depends on.\n",
    "\n",
    "For **every unique combination of entries in the depended or \"parent\" tables**, DataJoint calls `make` function, passing in the primary key of the parent(s)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because `Neuron` depends on `Session`, `Neuron`'s `make` method was called for each entry of `Session`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        \n",
       "        <style type=\"text/css\">\n",
       "            .Relation{\n",
       "                border-collapse:collapse;\n",
       "            }\n",
       "            .Relation th{\n",
       "                background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "                font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "            }\n",
       "            .Relation td{\n",
       "                padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "            }\n",
       "            .Relation tr:nth-child(odd){\n",
       "                background: #ffffff;\n",
       "            }\n",
       "            .Relation tr:nth-child(even){\n",
       "                background: #f3f1ff;\n",
       "            }\n",
       "            /* Tooltip container */\n",
       "            .djtooltip {\n",
       "            }\n",
       "            /* Tooltip text */\n",
       "            .djtooltip .djtooltiptext {\n",
       "                visibility: hidden;\n",
       "                width: 120px;\n",
       "                background-color: black;\n",
       "                color: #fff;\n",
       "                text-align: center;\n",
       "                padding: 5px 0;\n",
       "                border-radius: 6px;\n",
       "                /* Position the tooltip text - see examples below! */\n",
       "                position: absolute;\n",
       "                z-index: 1;\n",
       "            }\n",
       "            #primary {\n",
       "                font-weight: bold;\n",
       "                color: black;\n",
       "            }\n",
       "\n",
       "            #nonprimary {\n",
       "                font-weight: normal;\n",
       "                color: white;\n",
       "            }\n",
       "\n",
       "            /* Show the tooltip text when you mouse over the tooltip container */\n",
       "            .djtooltip:hover .djtooltiptext {\n",
       "                visibility: visible;\n",
       "            }\n",
       "        </style>\n",
       "        \n",
       "        <b>Experiment session</b>\n",
       "            <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "            <table border=\"1\" class=\"Relation\">\n",
       "                <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">mouse_id</p>\n",
       "                                <span class=\"djtooltiptext\">Unique animal ID</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">session_date</p>\n",
       "                                <span class=\"djtooltiptext\">date</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">experiment_setup</p>\n",
       "                                <span class=\"djtooltiptext\">experiment setup ID</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">experimenter</p>\n",
       "                                <span class=\"djtooltiptext\">experimenter name</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">data_path</p>\n",
       "                                <span class=\"djtooltiptext\"></span>\n",
       "                            </div> </th> </tr> </thead>\n",
       "                <tbody> <tr> <td>0</td>\n",
       "<td>2017-05-15</td>\n",
       "<td>0</td>\n",
       "<td>Edgar Y. Walker</td>\n",
       "<td></td></tr><tr><td>0</td>\n",
       "<td>2017-05-19</td>\n",
       "<td>0</td>\n",
       "<td>Edgar Y. Walker</td>\n",
       "<td></td></tr><tr><td>5</td>\n",
       "<td>2017-01-05</td>\n",
       "<td>1</td>\n",
       "<td>Fabian Sinz</td>\n",
       "<td></td></tr><tr><td>100</td>\n",
       "<td>2017-05-25</td>\n",
       "<td>100</td>\n",
       "<td>Jacob Reimer</td>\n",
       "<td></td> </tr> </tbody>\n",
       "            </table>\n",
       "            \n",
       "            <p>Total: 4</p></div>\n",
       "            "
      ],
      "text/plain": [
       "*mouse_id    *session_date  experiment_set experimenter   data_path    \n",
       "+----------+ +------------+ +------------+ +------------+ +-----------+\n",
       "0            2017-05-15     0              Edgar Y. Walke              \n",
       "0            2017-05-19     0              Edgar Y. Walke              \n",
       "5            2017-01-05     1              Fabian Sinz                 \n",
       "100          2017-05-25     100            Jacob Reimer                \n",
       " (Total: 4)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `make` only receives the *primary key attributes* of `Session` (`mouse_id` and `session_date`) but not the other attributes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing `make`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a better understanding of `make`, let's implement `make` to perform the importing of data from file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "@schema\n",
    "class Neuron(dj.Imported):\n",
    "    definition = \"\"\"\n",
    "    -> Session\n",
    "    neuron_id: int\n",
    "    ---\n",
    "    activity: longblob    # electric activity of the neuron\n",
    "    \"\"\"\n",
    "    def make(self, key):\n",
    "        # use key dictionary to determine the data file path\n",
    "        data_file = '{data_dir}/data_{mouse_id}_{session_date}.npy'.format(**key, data_dir=data_dir)\n",
    "\n",
    "        # load the data\n",
    "        data = np.load(data_file)\n",
    "        \n",
    "        print('Populating neuron(s) for mouse_id={mouse_id} on session_date={session_date}'.format(**key))\n",
    "        for idx, d in enumerate(data):\n",
    "            # add the index of the 1st dimension as neuron_id\n",
    "            key['neuron_id'] = idx\n",
    "            \n",
    "            # add the loaded data as the \"activity\" column\n",
    "            key['activity'] = d\n",
    "\n",
    "            # insert the key into self\n",
    "            self.insert1(key)\n",
    "\n",
    "            print('\\tPopulated neuron {neuron_id}'.format(**key))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we added the missing attribute information `activity` into the `key` dictionary, and finally **inserted the entry** into `self` = `Neuron` table. The `make` method's job is to create and insert a new entry corresponding to the `key` into this table!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's go ahead and call `populate` to actually populate the `Neuron` table, filling it with data loaded from data files!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating neuron(s) for mouse_id=0 on session_date=2017-05-15\n",
      "\tPopulated neuron 0\n",
      "Populating neuron(s) for mouse_id=0 on session_date=2017-05-19\n",
      "\tPopulated neuron 0\n",
      "Populating neuron(s) for mouse_id=5 on session_date=2017-01-05\n",
      "\tPopulated neuron 0\n",
      "Populating neuron(s) for mouse_id=100 on session_date=2017-05-25\n",
      "\tPopulated neuron 0\n"
     ]
    }
   ],
   "source": [
    "Neuron.populate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        \n",
       "        <style type=\"text/css\">\n",
       "            .Relation{\n",
       "                border-collapse:collapse;\n",
       "            }\n",
       "            .Relation th{\n",
       "                background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "                font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "            }\n",
       "            .Relation td{\n",
       "                padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "            }\n",
       "            .Relation tr:nth-child(odd){\n",
       "                background: #ffffff;\n",
       "            }\n",
       "            .Relation tr:nth-child(even){\n",
       "                background: #f3f1ff;\n",
       "            }\n",
       "            /* Tooltip container */\n",
       "            .djtooltip {\n",
       "            }\n",
       "            /* Tooltip text */\n",
       "            .djtooltip .djtooltiptext {\n",
       "                visibility: hidden;\n",
       "                width: 120px;\n",
       "                background-color: black;\n",
       "                color: #fff;\n",
       "                text-align: center;\n",
       "                padding: 5px 0;\n",
       "                border-radius: 6px;\n",
       "                /* Position the tooltip text - see examples below! */\n",
       "                position: absolute;\n",
       "                z-index: 1;\n",
       "            }\n",
       "            #primary {\n",
       "                font-weight: bold;\n",
       "                color: black;\n",
       "            }\n",
       "\n",
       "            #nonprimary {\n",
       "                font-weight: normal;\n",
       "                color: white;\n",
       "            }\n",
       "\n",
       "            /* Show the tooltip text when you mouse over the tooltip container */\n",
       "            .djtooltip:hover .djtooltiptext {\n",
       "                visibility: visible;\n",
       "            }\n",
       "        </style>\n",
       "        \n",
       "        <b></b>\n",
       "            <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "            <table border=\"1\" class=\"Relation\">\n",
       "                <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">mouse_id</p>\n",
       "                                <span class=\"djtooltiptext\">Unique animal ID</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">session_date</p>\n",
       "                                <span class=\"djtooltiptext\">date</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">neuron_id</p>\n",
       "                                <span class=\"djtooltiptext\"></span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">activity</p>\n",
       "                                <span class=\"djtooltiptext\">electric activity of the neuron</span>\n",
       "                            </div> </th> </tr> </thead>\n",
       "                <tbody> <tr> <td>0</td>\n",
       "<td>2017-05-15</td>\n",
       "<td>0</td>\n",
       "<td>=BLOB=</td></tr><tr><td>0</td>\n",
       "<td>2017-05-19</td>\n",
       "<td>0</td>\n",
       "<td>=BLOB=</td></tr><tr><td>5</td>\n",
       "<td>2017-01-05</td>\n",
       "<td>0</td>\n",
       "<td>=BLOB=</td></tr><tr><td>100</td>\n",
       "<td>2017-05-25</td>\n",
       "<td>0</td>\n",
       "<td>=BLOB=</td> </tr> </tbody>\n",
       "            </table>\n",
       "            \n",
       "            <p>Total: 4</p></div>\n",
       "            "
      ],
      "text/plain": [
       "*mouse_id    *session_date  *neuron_id    activity  \n",
       "+----------+ +------------+ +-----------+ +--------+\n",
       "0            2017-05-15     0             =BLOB=    \n",
       "0            2017-05-19     0             =BLOB=    \n",
       "5            2017-01-05     0             =BLOB=    \n",
       "100          2017-05-25     0             =BLOB=    \n",
       " (Total: 4)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Neuron()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can obviously see, in these example datasets, we only have data for one neuron per session. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if we call `Neuron.populate` again?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "Neuron.populate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's right - nothing! This makes sense, because we have imported `Neuron` for all entries in `Session` and nothing is left to be imported."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now what happens if we insert a new entry into `Session`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "Session.insert1({\n",
    "    \"mouse_id\": 100,\n",
    "    \"session_date\": \"2017-06-01\",\n",
    "    \"experiment_setup\": 1,\n",
    "    \"experimenter\": \"Jacob Reimer\"\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can find all `Session` without corresponding `Neuron` entry with the **negative restriction operator** `-`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        \n",
       "        <style type=\"text/css\">\n",
       "            .Relation{\n",
       "                border-collapse:collapse;\n",
       "            }\n",
       "            .Relation th{\n",
       "                background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "                font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "            }\n",
       "            .Relation td{\n",
       "                padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "            }\n",
       "            .Relation tr:nth-child(odd){\n",
       "                background: #ffffff;\n",
       "            }\n",
       "            .Relation tr:nth-child(even){\n",
       "                background: #f3f1ff;\n",
       "            }\n",
       "            /* Tooltip container */\n",
       "            .djtooltip {\n",
       "            }\n",
       "            /* Tooltip text */\n",
       "            .djtooltip .djtooltiptext {\n",
       "                visibility: hidden;\n",
       "                width: 120px;\n",
       "                background-color: black;\n",
       "                color: #fff;\n",
       "                text-align: center;\n",
       "                padding: 5px 0;\n",
       "                border-radius: 6px;\n",
       "                /* Position the tooltip text - see examples below! */\n",
       "                position: absolute;\n",
       "                z-index: 1;\n",
       "            }\n",
       "            #primary {\n",
       "                font-weight: bold;\n",
       "                color: black;\n",
       "            }\n",
       "\n",
       "            #nonprimary {\n",
       "                font-weight: normal;\n",
       "                color: white;\n",
       "            }\n",
       "\n",
       "            /* Show the tooltip text when you mouse over the tooltip container */\n",
       "            .djtooltip:hover .djtooltiptext {\n",
       "                visibility: visible;\n",
       "            }\n",
       "        </style>\n",
       "        \n",
       "        <b>Experiment session</b>\n",
       "            <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "            <table border=\"1\" class=\"Relation\">\n",
       "                <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">mouse_id</p>\n",
       "                                <span class=\"djtooltiptext\">Unique animal ID</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">session_date</p>\n",
       "                                <span class=\"djtooltiptext\">date</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">experiment_setup</p>\n",
       "                                <span class=\"djtooltiptext\">experiment setup ID</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">experimenter</p>\n",
       "                                <span class=\"djtooltiptext\">experimenter name</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">data_path</p>\n",
       "                                <span class=\"djtooltiptext\"></span>\n",
       "                            </div> </th> </tr> </thead>\n",
       "                <tbody> <tr> <td>100</td>\n",
       "<td>2017-06-01</td>\n",
       "<td>1</td>\n",
       "<td>Jacob Reimer</td>\n",
       "<td></td> </tr> </tbody>\n",
       "            </table>\n",
       "            \n",
       "            <p>Total: 1</p></div>\n",
       "            "
      ],
      "text/plain": [
       "*mouse_id    *session_date  experiment_set experimenter   data_path    \n",
       "+----------+ +------------+ +------------+ +------------+ +-----------+\n",
       "100          2017-06-01     1              Jacob Reimer                \n",
       " (Total: 1)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select all Session entries *without* a corresponding entry in Neuron\n",
    "Session - Neuron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating neuron(s) for mouse_id=100 on session_date=2017-06-01\n",
      "\tPopulated neuron 0\n"
     ]
    }
   ],
   "source": [
    "Neuron.populate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        \n",
       "        <style type=\"text/css\">\n",
       "            .Relation{\n",
       "                border-collapse:collapse;\n",
       "            }\n",
       "            .Relation th{\n",
       "                background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "                font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "            }\n",
       "            .Relation td{\n",
       "                padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "            }\n",
       "            .Relation tr:nth-child(odd){\n",
       "                background: #ffffff;\n",
       "            }\n",
       "            .Relation tr:nth-child(even){\n",
       "                background: #f3f1ff;\n",
       "            }\n",
       "            /* Tooltip container */\n",
       "            .djtooltip {\n",
       "            }\n",
       "            /* Tooltip text */\n",
       "            .djtooltip .djtooltiptext {\n",
       "                visibility: hidden;\n",
       "                width: 120px;\n",
       "                background-color: black;\n",
       "                color: #fff;\n",
       "                text-align: center;\n",
       "                padding: 5px 0;\n",
       "                border-radius: 6px;\n",
       "                /* Position the tooltip text - see examples below! */\n",
       "                position: absolute;\n",
       "                z-index: 1;\n",
       "            }\n",
       "            #primary {\n",
       "                font-weight: bold;\n",
       "                color: black;\n",
       "            }\n",
       "\n",
       "            #nonprimary {\n",
       "                font-weight: normal;\n",
       "                color: white;\n",
       "            }\n",
       "\n",
       "            /* Show the tooltip text when you mouse over the tooltip container */\n",
       "            .djtooltip:hover .djtooltiptext {\n",
       "                visibility: visible;\n",
       "            }\n",
       "        </style>\n",
       "        \n",
       "        <b></b>\n",
       "            <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "            <table border=\"1\" class=\"Relation\">\n",
       "                <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">mouse_id</p>\n",
       "                                <span class=\"djtooltiptext\">Unique animal ID</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">session_date</p>\n",
       "                                <span class=\"djtooltiptext\">date</span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"primary\">neuron_id</p>\n",
       "                                <span class=\"djtooltiptext\"></span>\n",
       "                            </div></th><th><div class=\"djtooltip\">\n",
       "                                <p id=\"nonprimary\">activity</p>\n",
       "                                <span class=\"djtooltiptext\">electric activity of the neuron</span>\n",
       "                            </div> </th> </tr> </thead>\n",
       "                <tbody> <tr> <td>0</td>\n",
       "<td>2017-05-15</td>\n",
       "<td>0</td>\n",
       "<td>=BLOB=</td></tr><tr><td>0</td>\n",
       "<td>2017-05-19</td>\n",
       "<td>0</td>\n",
       "<td>=BLOB=</td></tr><tr><td>5</td>\n",
       "<td>2017-01-05</td>\n",
       "<td>0</td>\n",
       "<td>=BLOB=</td></tr><tr><td>100</td>\n",
       "<td>2017-05-25</td>\n",
       "<td>0</td>\n",
       "<td>=BLOB=</td></tr><tr><td>100</td>\n",
       "<td>2017-06-01</td>\n",
       "<td>0</td>\n",
       "<td>=BLOB=</td> </tr> </tbody>\n",
       "            </table>\n",
       "            \n",
       "            <p>Total: 5</p></div>\n",
       "            "
      ],
      "text/plain": [
       "*mouse_id    *session_date  *neuron_id    activity  \n",
       "+----------+ +------------+ +-----------+ +--------+\n",
       "0            2017-05-15     0             =BLOB=    \n",
       "0            2017-05-19     0             =BLOB=    \n",
       "5            2017-01-05     0             =BLOB=    \n",
       "100          2017-05-25     0             =BLOB=    \n",
       "100          2017-06-01     0             =BLOB=    \n",
       " (Total: 5)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Neuron()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have successfully extended your pipeline with a table to represent recorded data (`Neuron` as `Imported` table), learned and implemented the `make()` and `populate()` call to load external data to your tables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<svg height=\"185pt\" viewBox=\"0.00 0.00 75.50 185.00\" width=\"76pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 181)\">\n<title>%3</title>\n<polygon fill=\"white\" points=\"-4,4 -4,-181 71.5,-181 71.5,4 -4,4\" stroke=\"none\"/>\n<!-- Mouse -->\n<g class=\"node\" id=\"node1\"><title>Mouse</title>\n<g id=\"a_node1\"><a xlink:title=\"mouse_id             \r\n------------------------------\r\ndob=null             \r\nsex=&quot;unknown&quot;        \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"59.75,-177 7.75,-177 7.75,-142 59.75,-142 59.75,-177\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"darkgreen\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"15.75\" y=\"-157.4\">Mouse</text>\n</a>\n</g>\n</g>\n<!-- Session -->\n<g class=\"node\" id=\"node3\"><title>Session</title>\n<g id=\"a_node3\"><a xlink:title=\"→ Mouse\r\nsession_date         \r\n------------------------------\r\nexperiment_setup     \r\nexperimenter         \r\ndata_path=&quot;&quot;         \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"63.25,-106 4.25,-106 4.25,-71 63.25,-71 63.25,-106\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"darkgreen\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"12.25\" y=\"-86.4\">Session</text>\n</a>\n</g>\n</g>\n<!-- Mouse&#45;&gt;Session -->\n<g class=\"edge\" id=\"edge1\"><title>Mouse-&gt;Session</title>\n<path d=\"M33.75,-141.797C33.75,-130.949 33.75,-116.867 33.75,-106.049\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- Neuron -->\n<g class=\"node\" id=\"node2\"><title>Neuron</title>\n<g id=\"a_node2\"><a xlink:title=\"→ Session\r\nneuron_id            \r\n------------------------------\r\nactivity             \r\n\">\n<ellipse cx=\"33.75\" cy=\"-17.5\" fill=\"#00007f\" fill-opacity=\"0.250980\" rx=\"33.5018\" ry=\"17.5\" stroke=\"#00007f\" stroke-opacity=\"0.250980\"/>\n<text fill=\"#00007f\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"14.75\" y=\"-15.4\">Neuron</text>\n</a>\n</g>\n</g>\n<!-- Session&#45;&gt;Neuron -->\n<g class=\"edge\" id=\"edge2\"><title>Session-&gt;Neuron</title>\n<path d=\"M33.75,-70.797C33.75,-59.9485 33.75,-45.8669 33.75,-35.0492\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n</g>\n</svg>",
      "text/plain": [
       "<datajoint.diagram.Diagram at 0x21e2cbe7358>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dj.Diagram(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, our pipeline contains the core elements with data populated, ready for further downstream analysis.\n",
    "\n",
    "In the next session, we are going to introduce the concept of `Computed` table, and `Lookup` table, as well as learning to set up a automated computation routine."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.16"
  },
  "vscode": {
   "interpreter": {
    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}