Datalad.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [git-annex](https://git-annex.branchable.com/) and [Datalad](datalad.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Git\n",
    "You probably already know Git, it is:\n",
    "- **distributed** version control system\n",
    "- GitHub, Bitbucket &ndash; a web-based hosting services for version control using git\n",
    "- very efficient for managing textual files (code, text, configuration, etc.)\n",
    "- **inefficient for storing data**\n",
    "    - it’s not design to handle big files\n",
    "    - you don’t want to upload your big files to GitHub and keep a copy of them in every single location"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### git-annex\n",
    "- built on top of Git\n",
    "- allows managing files with git without checking the file contents into git (it commits only symbolink links to the files)\n",
    "- adds \"special remotes\" to provide access to data content from various sources: web, Amazon S3, Dropbox, Google Drive, etc.\n",
    "- both Git and git-annex largely work on a single repository level"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Datalad\n",
    "- relies on Git and git-annex\n",
    "- **manages multiple repositories organized into “super-datasets”**\n",
    "- can crawl external online data sources, and update git/annex repositories upon changes\n",
    "- is scalable since data stays with original data providers\n",
    "- unifies access to data regardless of its origin (custom portals with authentication, S3, etc.) or serialization (e.g., tarballs)\n",
    "- **aggregates datasets’ meta-data and allows for quick search**\n",
    "- can publish original or derived datasets publicly (a web server, github) or for internal use (e.g., via ssh)\n",
    "- **comes with command line and Python interfaces**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In your scientific work Datalad can used to:**\n",
    "- **Discover Data**: built-in support for metadata extraction and search\n",
    "- **Consume Data**: direct access to individual files (great when you only need a few files from a large datasets)\n",
    "- **Publish Data**: support for sharing datasets with the public or your lab on platforms that you are using already \n",
    "- **Reproducibility**: support for joint management of analysis code and data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example of Data Consumption using Python API**:\n",
    "\n",
    "You can install `git-annex` and `Datalad` locally as described in [Datalad GitHub](https://github.com/datalad/datalad#other-linuxes-osx-windows-yet-todo-via-pip).  \n",
    "You can also use the Docker Image described in Docker notebook (you can pull from DockerHub: `docker pull djarecka/repropython`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] Cloning http://datasets.datalad.org/openfmri/ds000001 to '/Users/dorota/trainings/pycon2018/ReproduciblePython/ds000001' \n",
      "[INFO] access to dataset sibling \"datalad\" not auto-enabled, enable with:\n",
      "| \t\tdatalad siblings -d \"/Users/dorota/trainings/pycon2018/ReproduciblePython/ds000001\" enable -s datalad \n"
     ]
    }
   ],
   "source": [
    "# First we will install a publicly available dataset\n",
    "from datalad.api import install\n",
    "ds = install('///openfmri/ds000001')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 40\n",
      "-rw-r--r--  1 dorota  staff   578 May  7 23:26 CHANGES\n",
      "-rw-r--r--  1 dorota  staff  2141 May  7 23:26 README\n",
      "-rw-r--r--  1 dorota  staff   995 May  7 23:26 dataset_description.json\n",
      "-rw-r--r--  1 dorota  staff   215 May  7 23:26 participants.tsv\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-01\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-02\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-03\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-04\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-05\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-06\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-07\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-08\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-09\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-10\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-11\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-12\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-13\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-14\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-15\n",
      "drwxr-xr-x  4 dorota  staff   136 May  7 23:26 sub-16\n",
      "-rw-r--r--  1 dorota  staff    73 May  7 23:26 task-balloonanalogrisktask_bold.json\n"
     ]
    }
   ],
   "source": [
    "# As you can see datalad cloned the dataset to local directory\n",
    "# (you might have some warnings if your git is not configured)\n",
    "\n",
    "# We can check the content of the new directory\n",
    "!ls -l ds000001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 8\n",
      "lrwxr-xr-x 1 repropython users 138 May  7 22:09 sub-10_inplaneT2.nii.gz -> ../../.git/annex/objects/FJ/GK/MD5E-s684908--ce1ce3e4fb6be073bf5e03280ea14acb.nii.gz/MD5E-s684908--ce1ce3e4fb6be073bf5e03280ea14acb.nii.gz\n",
      "lrwxr-xr-x 1 repropython users 140 May  7 22:09 sub-10_T1w.nii.gz -> ../../.git/annex/objects/Qk/mv/MD5E-s5481518--675ca482ce805a7dd359e52a576a9206.nii.gz/MD5E-s5481518--675ca482ce805a7dd359e52a576a9206.nii.gz\n"
     ]
    }
   ],
   "source": [
    "# we can check files in one of the subdirectories\n",
    "!ls -l ds000001/sub-10/anat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lrwxr-xr-x  1 dorota  staff  140 May  7 23:26 ds000001/sub-10/anat/sub-10_T1w.nii.gz -> ../../.git/annex/objects/Qk/mv/MD5E-s5481518--675ca482ce805a7dd359e52a576a9206.nii.gz/MD5E-s5481518--675ca482ce805a7dd359e52a576a9206.nii.gz\n",
      "lrwxr-xr-x  1 dorota  staff  138 May  7 23:26 ds000001/sub-10/anat/sub-10_inplaneT2.nii.gz -> ../../.git/annex/objects/FJ/GK/MD5E-s684908--ce1ce3e4fb6be073bf5e03280ea14acb.nii.gz/MD5E-s684908--ce1ce3e4fb6be073bf5e03280ea14acb.nii.gz\n",
      "ls: sub-10_T1w.nii.gz: No such file or directory\n",
      "ls: sub-10_inplaneT2.nii.gz: No such file or directory\n"
     ]
    }
   ],
   "source": [
    "# so we have a list symbolic links that points to the files, but the content was not downloaded\n",
    "\n",
    "# you can check the size of the files\n",
    "!ls -lL ds000001/sub-10/anat/*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] Actually getting 2 files \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total:   0%|          | 0.00/6.17M [00:00<?, ?B/s]\n",
      "Total:   4%|▎         | 225k/6.17M [00:01<00:32, 186kB/s]\n",
      "Total:  11%|█         | 684k/6.17M [00:01<00:22, 244kB/s]\n",
      "Total:  16%|█▋        | 1.02M/6.17M [00:05<00:31, 165kB/s]\n",
      "Total:  24%|██▍       | 1.50M/6.17M [00:05<00:21, 220kB/s]\n",
      "Total:  29%|██▉       | 1.81M/6.17M [00:06<00:18, 241kB/s]\n",
      "Total:  37%|███▋      | 2.27M/6.17M [00:09<00:17, 221kB/s]\n",
      "sub-10/anat .. _T1w.nii.gz:  41%|████▏     | 2.27M/5.48M [00:08<00:12, 250kB/s]\u001b[A"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/dorota/anaconda/envs/reproPython/lib/python3.6/site-packages/tqdm/_monitor.py:89: TqdmSynchronisationWarning: Set changed size during iteration (see https://github.com/tqdm/tqdm/issues/481)\n",
      "  TqdmSynchronisationWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total:  50%|████▉     | 3.06M/6.17M [00:10<00:11, 267kB/s]\n",
      "Total:  66%|██████▋   | 4.10M/6.17M [00:11<00:05, 361kB/s]\n",
      "Total:  85%|████████▌ | 5.26M/6.17M [00:11<00:01, 483kB/s]\n",
      "Total (1 ok out of 2):  89%|████████▉ | 5.48M/6.17M [00:11<00:01, 483kB/s]\n",
      "                                                                               \u001b[A\n",
      "sub-10/anat .. neT2.nii.gz:   0%|          | 0.00/685k [00:00<?, ?B/s]\u001b[A\n",
      "Total (1 ok out of 2): 100%|██████████| 6.17M/6.17M [00:12<00:00, 541kB/s]\n",
      "Total (2 ok out of 2): 100%|██████████| 6.17M/6.17M [00:13<00:00, 541kB/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'type': 'file',\n",
       "  'refds': '/Users/dorota/trainings/pycon2018/ReproduciblePython/ds000001',\n",
       "  'status': 'ok',\n",
       "  'path': '/Users/dorota/trainings/pycon2018/ReproduciblePython/ds000001/sub-10/anat/sub-10_T1w.nii.gz',\n",
       "  'action': 'get',\n",
       "  'annexkey': 'MD5E-s5481518--675ca482ce805a7dd359e52a576a9206.nii.gz'},\n",
       " {'type': 'file',\n",
       "  'refds': '/Users/dorota/trainings/pycon2018/ReproduciblePython/ds000001',\n",
       "  'status': 'ok',\n",
       "  'path': '/Users/dorota/trainings/pycon2018/ReproduciblePython/ds000001/sub-10/anat/sub-10_inplaneT2.nii.gz',\n",
       "  'action': 'get',\n",
       "  'annexkey': 'MD5E-s684908--ce1ce3e4fb6be073bf5e03280ea14acb.nii.gz'},\n",
       " {'action': 'get',\n",
       "  'path': '/Users/dorota/trainings/pycon2018/ReproduciblePython/ds000001/sub-10/anat',\n",
       "  'type': 'directory',\n",
       "  'refds': '/Users/dorota/trainings/pycon2018/ReproduciblePython/ds000001',\n",
       "  'status': 'ok'}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# you should see the message that the files are not present\n",
    "\n",
    "# let's now download the files for subject 10 using the get method\n",
    "ds.get('sub-10/anat')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-r--r--r--  1 dorota  staff  5481518 May  7 23:31 ds000001/sub-10/anat/sub-10_T1w.nii.gz\n",
      "-r--r--r--  1 dorota  staff   684908 May  7 23:31 ds000001/sub-10/anat/sub-10_inplaneT2.nii.gz\n"
     ]
    }
   ],
   "source": [
    "# let's check the size of the files again\n",
    "!ls -lL ds000001/sub-10/anat/*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now the content of the files is in your local directory\n",
    "# but you didn't have to download the content of entire repository to get only these two files!! :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More examples and videos can be found [here](http://datalad.org/features.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<link href=\"https://fonts.googleapis.com/css?family=Didact+Gothic|Dosis:400,500,700\" rel=\"stylesheet\"><style>\n",
       "@font-face {\n",
       "  font-family: \"Computer Modern\";\n",
       "  src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
       "}\n",
       "/* div.cell{\n",
       "width:800px;\n",
       "margin-left:16% !important;\n",
       "margin-right:auto;\n",
       "} */\n",
       "h1 {\n",
       "  font-family: 'Dosis', \"Helvetica Neue\", Arial, sans-serif;\n",
       "  color: #0B132B;\n",
       "}\n",
       "h2 {\n",
       "  font-family: 'Dosis', sans-serif;\n",
       "  color: #1C2541;\n",
       "}\n",
       "h3{\n",
       "  font-family: 'Dosis', sans-serif;\n",
       "  margin-top:12px;\n",
       "  margin-bottom: 3px;\n",
       "  color: #40a8a6;\n",
       "}\n",
       "h4{\n",
       "  font-family: 'Dosis', sans-serif;\n",
       "  color: #40a8a6;\n",
       "}\n",
       "h5 {\n",
       "  font-family: 'Dosis', sans-serif;\n",
       "  color: #40a8a6;\n",
       "}\n",
       "div.text_cell_render{\n",
       "  font-family: 'Didact Gothic',Computer Modern, \"Helvetica Neue\", Arial, Helvetica,\n",
       "  Geneva, sans-serif;\n",
       "  line-height: 130%;\n",
       "  font-size: 110%;\n",
       "  /* width:600px; */\n",
       "  /* margin-left:auto;\n",
       "  margin-right:auto; */\n",
       "}\n",
       "\n",
       ".text_cell_render h1 {\n",
       "  font-weight: 200;\n",
       "  font-size: 30pt;\n",
       "  /* font-size: 50pt */\n",
       "  line-height: 100%;\n",
       "  color:#0B132B;\n",
       "  margin-bottom: 0.5em;\n",
       "  margin-top: 0.5em;\n",
       "  display: block;\n",
       "}\n",
       "\n",
       ".text_cell_render h2{\n",
       "  font-weight: 500;\n",
       "}\n",
       "\n",
       ".text_cell_render h3{\n",
       "  font-weight: 500;\n",
       "}\n",
       "\n",
       "\n",
       ".warning{\n",
       "  color: rgb( 240, 20, 20 )\n",
       "}\n",
       "\n",
       "div.warn {\n",
       "  background-color: #FF5A5F;\n",
       "  border-color: #FF5A5F;\n",
       "  border-left: 5px solid #C81D25;\n",
       "  padding: 0.5em;\n",
       "\n",
       "  color: #fff;\n",
       "  opacity: 0.8;\n",
       "}\n",
       "\n",
       "div.info {\n",
       "  background-color: #087E8B;\n",
       "  border-color: #087E8B;\n",
       "  border-left: 5px solid #0B3954;\n",
       "  padding: 0.5em;\n",
       "  color: #fff;\n",
       "  opacity: 0.8;\n",
       "}\n",
       "\n",
       "</style>\n",
       "<script>\n",
       "MathJax.Hub.Config({\n",
       "  TeX: {\n",
       "    extensions: [\"AMSmath.js\"]\n",
       "    },\n",
       "    tex2jax: {\n",
       "      inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
       "      displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
       "      },\n",
       "      displayAlign: 'center', // Change this to 'center' to center equations.\n",
       "      \"HTML-CSS\": {\n",
       "        styles: {'.MathJax_Display': {\"margin\": 4}}\n",
       "      }\n",
       "      });\n",
       "      </script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "\n",
    "\n",
    "def css_styling():\n",
    "    styles = open(\"styles/custom.css\", \"r\").read()\n",
    "    return HTML(styles)\n",
    "css_styling()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}