BIDS_Intro_Heudiconv.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to BIDS\n",
    "> \"A simple and intuitive way to organize and describe your neuroimaging and behavioral data.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is BIDS\n",
    "See above (heh).\n",
    "\n",
    "If you really wanna dig into what BIDS is, I would recommend reading the original paper that pushes for more investigators to use BIDS.\n",
    "\n",
    "From the creators original paper: \"The BIDS standard uses file formats compatible with existing software, unifies the majority of practices already common in the field, and captures the metadata necessary for most common data processing operations.\" [^1]\n",
    "\n",
    "A BIDS filestructure looks something like:\n",
    "![bids2](misc/bids.png)\n",
    "\n",
    "[^1]: See https://www.nature.com/articles/sdata201644"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why do we need BIDS?\n",
    "Essentially it boils down to this:\n",
    "* different researchers often use different processing pipelines and data/file structures (in neuroscience and science more generally)\n",
    "* having different file structures makes it difficult to:\n",
    "    * work with other peoples code\n",
    "    * develop applications and programs that can work with many people's data\n",
    "* BIDS is (ONE) proposed solution to help make data\n",
    "    * more accessible (and thus more reproducible) for other researchers\n",
    "    * usable with different apps developed by different researchers\n",
    "    * more accessible for newer researchers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BIDS apps\n",
    "With data that are structured using the BIDS format, developers and (very generous) researchers can create and share apps that can easily work with data in a specific format. It also allows these apps to be deployed interactively and online on sites like [openneuro.org](https://openneuro.org/datasets/ds001399/versions/00002). Some examples of popular BIDS apps include:\n",
    "\n",
    "### mriqc\n",
    "[mriqc](https://mriqc.readthedocs.io/en/stable/) was developed by the Poldrack Lab at Stanford University. In essence, it generates IQMs (image quality metrics) and produces an [easy to read report](https://s3.amazonaws.com/openneuro.outputs/94c2aca4b5cc09816ea88c73b2d593e1/3ab385a2-0d22-4636-8e72-c274a42911d5/reports/sub-MOCCAG001_ses-01_task-reward_acq-multiband2p4tiltPA_run-01_bold.html) to determine the quality of your MRI data.\n",
    "\n",
    "### fmriprep\n",
    "[fmriprep](https://github.com/poldracklab/fmriprep) was *also* developed by the Poldrack Lab at Stanford (you'll notice a trend here). From their github:\n",
    "> `fmriprep` is a functional magnetic resonance imaging (fMRI) data preprocessing pipeline that is designed to provide an easily accessible, state-of-the-art interface that is robust to variations in scan acquisition protocols and that requires minimal user input, while providing easily interpretable and comprehensive error and output reporting. It performs basic processing steps (coregistration, normalization, unwarping, noise component extraction, segmentation, skullstripping etc.) providing outputs that can be easily submitted to a variety of group level analyses, including task-based or resting-state fMRI, graph theory measures, surface or volume-based statistics, etc.\n",
    "\n",
    "### some important caveats\n",
    "It's important to keep in mind that BIDS was initially introduced to the field just a few years ago. It is *very new* and should be used ***carefully***. Because it is still very much an idea and set of tools in development, it is important to stay up to date with the changes made to the apps and what versions of software you are working with."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HEUDICONV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Docker\n",
    "[Docker](https://www.docker.com) is a developer tool that makes it easy for multiple users to create, deploy, and run applications using 'containers'. Docker can be a tricky concept to understand but essentially, it provides users and developers with a standard unit of software that packages code and all of its dependencies so that applications can run quickly and reliably across computing environments. There are a lot of resources for understanding containers and how Docker works, here are a few I've found helpful:\n",
    "* https://www.docker.com/resources/what-container\n",
    "* https://ropenscilabs.github.io/r-docker-tutorial/01-what-and-why.html\n",
    "* https://opensource.com/resources/what-docker\n",
    "* https://docs.docker.com/engine/docker-overview/\n",
    "\n",
    "For me, I think of Docker similarly to the way one would load libraries in R, except that for a single app, you would only load a single library that would include all of the necessary dependencies.\n",
    "\n",
    "### Installing Docker\n",
    "To install Docker, follow the instructions [here](https://www.docker.com/products/docker-desktop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running HEUDICONV: an example\n",
    "### set up\n",
    "Before we do anything else, we'll need to download Heudiconv. Because many of the BIDS apps pull from a number of different languages/packages, I find it's easiest to use docker (as they recommend) so that you are always working with the dependencies the developers use.\n",
    "\n",
    "In your command terminal, type `docker pull nipy/heudiconv:debian`.\n",
    "\n",
    "***Note:*** In many of the other online tutorials, people use `docker pull nipy/heudiconv:latest`. I couldn't get this to work for me and when I checked heudiconv's [docker tags](https://hub.docker.com/r/nipy/heudiconv/tags), I couldn't find them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### creating the .heudiconv directory\n",
    "The first thing we'll do is essentially run a blank version of the command in order to generate the heuristic file. To understand what each of the flags in the command, check out [heudiconv's documentation](https://heudiconv.readthedocs.io/en/latest/usage.html)\n",
    "\n",
    "    docker run --rm -it \\\n",
    "    -v /Users/lizbeard/Desktop/DEAL:/base \\\n",
    "    nipy/heudiconv:debian \\\n",
    "    -d /base/DEAL-pilot-{subject}/scans/*/DICOM/*.dcm \\\n",
    "    -o /base/BIDS/ \\\n",
    "    -f convertall \\\n",
    "    -s 001 \\\n",
    "    -c none\n",
    "\n",
    "And it should output something that looks like:\n",
    "\n",
    "![screenshot01](misc/01.png)\n",
    "\n",
    "Digging into that .heudiconv directory you can see that the convertall has created an info folder in the new output directory.\n",
    "\n",
    "![screenshot02](misc/02.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two important items that were created as a result of this dry run:\n",
    "* `convertall.py`: aka the heuristics file which will let heudiconv accurately comb through our files to better convert them into BIDS format\n",
    "* `dicominfo_*.tsv`: a tsv file that contains the necessary information we'll use to update the heuristics file "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### setting up your heuristics file\n",
    "Next, we'll open the convertall.py script to put in our necessary heuristics. \n",
    "\n",
    "We'll be editing a minimal amount of info within the script, but you would presumably have one script for each study you run. Below is what the heudiconv `convertall.py` script provides for you:\n",
    "\n",
    "![screenshot03](misc/03.png)\n",
    "\n",
    "`data` is an example for how to create a key. Keys define the type of scan you want your heuristic file to look for and will format your filenames according BIDS specifications. (For more info about BIDS naming conventions, look [here](https://bids.neuroimaging.io/bids_spec.pdf) So for our purposes, we may include objects such as:\n",
    "\n",
    "    T1w = create_key('sub-{subject}/anat/sub-{subject}_T1w')\n",
    "    rest = create_key('sub-{subject}/func/sub-{subject}_task-rest_rec-{rec}_run-{item:01d}_bold')\n",
    "    \n",
    "This is where it helps to be at least somewhat familiar with python constructs but essentially, you are using logic statements that will parse through the `dicominfo_*.tsv` file and re-name your files in the proper format (that YOU specify). [This tutorial](https://nipy.org/heudiconv/#17) has great examples for how to use multiple logical statements and also how to account for multiple runs in your files.\n",
    "\n",
    "    if 't1' in s.protocol_name:\n",
    "        info[T1w] = [s.series_id]\n",
    "    if 't2' in s.protocol_name:\n",
    "        info[T2w] = [s.series_id]\n",
    "\n",
    "After you've updated your `convertall.py` script, save it with the rest of your code (generally it's recommended to have a `project/BIDS/code` folder).\n",
    "\n",
    "***Note:*** Newer versions of heudiconv will break if you re-name your `convertall.py` script. Be sure to save your updated file as `convertall.py` in a new location instead of renaming the file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LET'S DO THIS THING!\n",
    "OKAY, so after you've gone through *all* this setup, we should finally be ready to re-run the heudiconv app to convert our files to .nii format and shift them into BIDS format.\n",
    "\n",
    "To do this, we'll have to change some of the initial code from generating the heuristic file:\n",
    "\n",
    "    docker run --rm -it \\\n",
    "    -v /Users/lizbeard/Desktop/DEAL:/base \\\n",
    "    nipy/heudiconv:debian \\\n",
    "    -d /base/DEAL-pilot-{subject}/scans/*/DICOM/*.dcm \\\n",
    "    -o /base/BIDS/ \\\n",
    "    -f /base/BIDS/code/convertall.py \\\n",
    "    -s 001 \\\n",
    "    -c dcm2niix \\\n",
    "    -b\n",
    "\n",
    "***Note:*** There are a number of different converters that work with [BIDS apps](http://bids.neuroimaging.io/). Here I'm using dcm2niix since we're also (likely) looking to convert our dicoms to .nii files. It's also the converter that works specifically with heudiconv.\n",
    "\n",
    "Once we've run the command (correctly), we should see that our `BIDS` directory has been populated. It should look something like this:\n",
    "\n",
    "![screenshot04](misc/04.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confirming BIDS format \n",
    "There are a number of BIDS validation tools that you can use to confirm your data is indeed in BIDS format:\n",
    "* [this program](https://github.com/bids-standard/bids-validator) runs locally and instructions are included on their webpage, it's probably better for checking large batches of data\n",
    "* [this program](http://bids-standard.github.io/bids-validator/) allows you to validate a directory/file structure online\n",
    "\n",
    "Let's use the online tool to see if our conversion worked! At first pass, it looks like heudiconv did at least *something*.\n",
    "\n",
    "All in all, looks pretty good! I think we can delve into those Warnings at a later point.\n",
    "\n",
    "![screenshot06](misc/05.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RESOURCES\n",
    "## BIDS General Resources\n",
    "* [bids.neuroimaging.io](http://bids.neuroimaging.io/): has a good overview and list of resources for BIDS some preliminary apps\n",
    "* [BIDS App](http://bids-apps.neuroimaging.io/apps/): container image capturing a neuroimaging pipeline that takes a BIDS formatted dataset as input (aka: a list of apps you can use with BIDS formatted data -- this list is not comprehensive)\n",
    "* [BIDS Starter Kit](https://github.com/bids-standard/bids-starter-kit): a community-curated collection of tutorials, wikis, and templates to get you started with creating BIDS compliant datasets\n",
    "\n",
    "## HEUDICONV Resources\n",
    "* [Heudiconv's readthedocs](https://heudiconv.readthedocs.io/en/latest/index.html): Heudiconv recently switched from having the majority of their info on GitHub to readthedocs. It provides a general overview of how to use the converter and also has a number of resources/user tutorials on their page that I've included here.\n",
    "* [Heudiconv GitHub](https://github.com/nipy/heudiconv): the GitHub page for Heudiconv. A good place to look for updates/bugs with the code. Sometimes their documentation/explanation isn't always comprehensive.\n",
    "* [This YouTube tutorial created by James Kent](https://www.youtube.com/watch?v=O1kZAuR7E00): I watched this one a few times and James' thorough walkthrough of the terminal commands was really helpful.\n",
    "* [This tutorial from Stanford Center for Reproducible Neuroscience](http://reproducibility.stanford.edu/bids-tutorial-series-part-2a/#heuman3): I found this tutorial also helpful to refer back to.\n",
    "* [This sample conversion from the Heudiconv team](https://nipy.org/heudiconv/#1): This tutorial has helpful slides on developing your heuristics script. It only references a specific example though and thus can be tricky to follow if you're not familiar with how the syntax really works.\n",
    "* [This is an older version of the sample conversion](http://nipy.org/workshops/2017-03-boston/lectures/bids-heudiconv/#1): and has some additional interesting tools/tidbits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}