Notebooks/A0-Preparing your computer for HiC data analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "* [Hardware](#Hardware)\n",
    "\t* [Disk space](#Disk-space)\n",
    "\t* [RAM memory](#RAM-memory)\n",
    "\t* [CPUs](#CPUs)\n",
    "* [Software](#Software)\n",
    "\t* [GEM Mapper](#GEM-Mapper)\n",
    "\t* [Conda](#Conda)\n",
    "\t* [Python dependencies](#Python-dependencies)\n",
    "\t* [DSRC FASTQ compressor](#DSRC-FASTQ-compressor)\n",
    "\t* [TADbit](#TADbit)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hardware"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Disk space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The processing of a \"typical\" Hi-C experiment of about 200 M reads will occupy a space of around __100 GB per experiment__. After the anaysis many of the intermediate files can be compressed or erased, but at it is probable that at each of the experiment/replicate will at least 50 Gb in disk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RAM memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The more the better. RAM is specially important to load matrices at high resolution, but usually __32 Gb__ of RAM should be enough to deal with 50 kb resolution matrices on a human genome."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CPUs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No limitations here, just time. A __8 core computer__ should be abble to process a single Hi-C experiment (200 M reads, analyzed at 50 kb) in __3-4 days__. This includes all the steps of the mapping, filtering, normalization and detection of TADs and compartments.\n",
    "\n",
    "The 3D modeling will depend on the size of the regions to be modeled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Software"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conda (http://conda.pydata.org/docs/index.html) is a package manager, mainly hosting python programs, that is very usefull when no root access is available and the softwares have complicated dependencies.\n",
    "\n",
    "To install is just download the installer from http://conda.pydata.org/miniconda.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "! wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And run it with all the default options. The installer will create a `miniconda2` folder in your home directory where all the programs that you need will be stored (including python)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With conda you can install the needed dependencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fetching package metadata .......\n",
      "Solving package specifications: ..........\n",
      "\n",
      "Package plan for installation in environment /home/fransua/.miniconda2:\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    scipy-0.18.1               |      np111py27_0        30.9 MB\n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "    scipy: 0.17.1-np111py27_1 --> 0.18.1-np111py27_0\n",
      "\n",
      "Fetching packages ...\n",
      "scipy-0.18.1-n 100% |################################| Time: 0:00:08   4.05 MB/s\n",
      "Extracting packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Unlinking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Linking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Fetching package metadata .......\n",
      "Solving package specifications: ..........\n",
      "\n",
      "Package plan for installation in environment /home/fransua/.miniconda2:\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    numpy-1.11.2               |           py27_0         6.2 MB\n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "    numpy: 1.11.1-py27_0 --> 1.11.2-py27_0\n",
      "\n",
      "Fetching packages ...\n",
      "numpy-1.11.2-p 100% |################################| Time: 0:00:04   1.50 MB/s\n",
      "Extracting packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Unlinking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Linking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Fetching package metadata .......\n",
      "Solving package specifications: ..........\n",
      "\n",
      "Package plan for installation in environment /home/fransua/.miniconda2:\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    expat-2.1.0                |                0         365 KB\n",
      "    libxcb-1.12                |                0         1.5 MB\n",
      "    dbus-1.10.10               |                0         2.4 MB\n",
      "    libpng-1.6.22              |                0         214 KB\n",
      "    gstreamer-1.8.0            |                0         2.6 MB\n",
      "    sip-4.18                   |           py27_0         264 KB\n",
      "    fontconfig-2.11.1          |                6         405 KB\n",
      "    gst-plugins-base-1.8.0     |                0         3.1 MB\n",
      "    qt-5.6.0                   |                0        43.9 MB\n",
      "    pyqt-5.6.0                 |           py27_0         5.3 MB\n",
      "    matplotlib-1.5.3           |      np111py27_0         8.2 MB\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:        68.2 MB\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    dbus:             1.10.10-0        \n",
      "    expat:            2.1.0-0          \n",
      "    gst-plugins-base: 1.8.0-0          \n",
      "    gstreamer:        1.8.0-0          \n",
      "    libxcb:           1.12-0           \n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "    fontconfig:       2.11.1-5          --> 2.11.1-6         \n",
      "    libpng:           1.6.17-0          --> 1.6.22-0         \n",
      "    matplotlib:       1.5.1-np110py27_0 --> 1.5.3-np111py27_0\n",
      "    pyqt:             4.11.4-py27_1     --> 5.6.0-py27_0     \n",
      "    qt:               4.8.7-1           --> 5.6.0-0          \n",
      "    sip:              4.16.9-py27_0     --> 4.18-py27_0      \n",
      "\n",
      "Fetching packages ...\n",
      "expat-2.1.0-0. 100% |################################| Time: 0:00:01 372.01 kB/s\n",
      "libxcb-1.12-0. 100% |################################| Time: 0:00:01 887.07 kB/s\n",
      "dbus-1.10.10-0 100% |################################| Time: 0:00:03 818.91 kB/s\n",
      "libpng-1.6.22- 100% |################################| Time: 0:00:00 287.09 kB/s\n",
      "gstreamer-1.8. 100% |################################| Time: 0:00:03 836.25 kB/s\n",
      "sip-4.18-py27_ 100% |################################| Time: 0:00:00 286.39 kB/s\n",
      "fontconfig-2.1 100% |################################| Time: 0:00:01 349.04 kB/s\n",
      "gst-plugins-ba 100% |################################| Time: 0:00:06 539.21 kB/s\n",
      "qt-5.6.0-0.tar 100% |################################| Time: 0:00:06   6.97 MB/s\n",
      "pyqt-5.6.0-py2 100% |################################| Time: 0:00:03   1.76 MB/s\n",
      "matplotlib-1.5 100% |################################| Time: 0:00:04   1.96 MB/s\n",
      "Extracting packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Unlinking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Linking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Fetching package metadata .........\n",
      "Solving package specifications: ..........\n",
      "\n",
      "# All requested packages already installed.\n",
      "# packages in environment at /home/fransua/.miniconda2:\n",
      "#\n",
      "pysam                     0.8.4pre0                py27_0    bcbio\n",
      "Fetching package metadata .........\n",
      "Solving package specifications: ..........\n",
      "\n",
      "Package plan for installation in environment /home/fransua/.miniconda2:\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    numpy-1.10.4               |           py27_2         6.0 MB\n",
      "    numexpr-2.6.0              |      np110py27_0         351 KB\n",
      "    scipy-0.17.1               |      np110py27_1        30.1 MB\n",
      "    scikit-learn-0.17.1        |      np110py27_2         8.6 MB\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:        45.0 MB\n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "    hdf5:         1.8.15.1-3         --> 1.8.16-0          \n",
      "    numexpr:      2.6.0-np111py27_0  --> 2.6.0-np110py27_0 \n",
      "    scikit-learn: 0.17.1-np111py27_2 --> 0.17.1-np110py27_2\n",
      "\n",
      "The following packages will be DOWNGRADED due to dependency conflicts:\n",
      "\n",
      "    libpng:       1.6.22-0           --> 1.6.17-0          \n",
      "    numpy:        1.11.2-py27_0      --> 1.10.4-py27_2     \n",
      "    scipy:        0.18.1-np111py27_0 --> 0.17.1-np110py27_1\n",
      "\n",
      "Fetching packages ...\n",
      "numpy-1.10.4-p 100% |################################| Time: 0:00:04   1.39 MB/s\n",
      "numexpr-2.6.0- 100% |################################| Time: 0:00:01 309.06 kB/s\n",
      "scipy-0.17.1-n 100% |################################| Time: 0:00:04   7.09 MB/s\n",
      "scikit-learn-0 100% |################################| Time: 0:00:04   2.09 MB/s\n",
      "Extracting packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Unlinking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Linking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Fetching package metadata .......\n",
      "Solving package specifications: ..........\n",
      "\n",
      "# All requested packages already installed.\n",
      "# packages in environment at /home/fransua/.miniconda2:\n",
      "#\n",
      "pip                       8.1.2                    py27_0  \n",
      "Fetching package metadata .........\n",
      "Solving package specifications: ..........\n",
      "\n",
      "Package plan for installation in environment /home/fransua/.miniconda2:\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    mcl-14.137                 |                1         6.4 MB  bioconda\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    mcl: 14.137-1 bioconda\n",
      "\n",
      "Fetching packages ...\n",
      "mcl-14.137-1.t 100% |################################| Time: 0:00:04   1.61 MB/s\n",
      "Extracting packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Linking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n"
     ]
    }
   ],
   "source": [
    "! conda install -y scipy # scientific computing in python\n",
    "! conda install -y numpy # scientific computing in python\n",
    "! conda install -y matplotlib # to produce plots\n",
    "! conda install -y jupyter # this notebook :)\n",
    "! conda install -y -c https://conda.anaconda.org/bcbio pysam # to deal with SAM/BAM files\n",
    "! conda install -y -c https://conda.anaconda.org/salilab imp # for 3D modeling\n",
    "! conda install -y pip # yet another python package manager\n",
    "! conda install -y -c bioconda mcl # for clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GEM Mapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this course we will use GEM, but any other alternative is just fine.\n",
    "\n",
    "To install GEM, go to the download page: https://sourceforge.net/projects/gemlibrary/files/gem-library/Binary%20pre-release%202/\n",
    "and download the `i3` version (the other version is for older computers, and you usually won't have to use it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2018-09-17 12:28:40--  https://sourceforge.net/projects/gemlibrary/files/gem-library/Binary%20pre-release%202/GEM-binaries-Linux-x86_64-core_i3-20121106-022124.tbz2/download\n",
      "Resolving sourceforge.net... 216.105.38.13\n",
      "Connecting to sourceforge.net|216.105.38.13|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://downloads.sourceforge.net/project/gemlibrary/gem-library/Binary%20pre-release%202/GEM-binaries-Linux-x86_64-core_i3-20121106-022124.tbz2?r=&ts=1537183721&use_mirror=netcologne [following]\n",
      "--2018-09-17 12:28:41--  https://downloads.sourceforge.net/project/gemlibrary/gem-library/Binary%20pre-release%202/GEM-binaries-Linux-x86_64-core_i3-20121106-022124.tbz2?r=&ts=1537183721&use_mirror=netcologne\n",
      "Resolving downloads.sourceforge.net... 216.105.38.13\n",
      "Connecting to downloads.sourceforge.net|216.105.38.13|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://netcologne.dl.sourceforge.net/project/gemlibrary/gem-library/Binary%20pre-release%202/GEM-binaries-Linux-x86_64-core_i3-20121106-022124.tbz2 [following]\n",
      "--2018-09-17 12:28:42--  https://netcologne.dl.sourceforge.net/project/gemlibrary/gem-library/Binary%20pre-release%202/GEM-binaries-Linux-x86_64-core_i3-20121106-022124.tbz2\n",
      "Resolving netcologne.dl.sourceforge.net... 78.35.24.46, 2001:4dd0:1234:6::5f\n",
      "Connecting to netcologne.dl.sourceforge.net|78.35.24.46|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 3999929 (3.8M) [application/octet-stream]\n",
      "Saving to: ‘GEM.tbz2’\n",
      "\n",
      "GEM.tbz2            100%[===================>]   3.81M   316KB/s    in 14s     \n",
      "\n",
      "2018-09-17 12:28:57 (270 KB/s) - ‘GEM.tbz2’ saved [3999929/3999929]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! wget -O GEM.tbz2 https://sourceforge.net/projects/gemlibrary/files/gem-library/Binary%20pre-release%202/GEM-binaries-Linux-x86_64-core_i3-20121106-022124.tbz2/download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncompress the archive: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/\n",
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-indexer\n",
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-indexer.man\n",
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-map-2-map.man\n",
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-mapper.man\n",
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-indexer_fasta2meta+cont\n",
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-2-sam\n",
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-map-2-map\n",
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-indexer_generate\n",
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-2-sam.man\n",
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-mapper\n",
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-indexer_bwt-dna\n",
      "GEM-binaries-Linux-x86_64-core_i3-20121106-022124/LICENSE\n"
     ]
    }
   ],
   "source": [
    "! tar -xjvf GEM.tbz2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And copy the needed binaries to somewhere in your PATH, like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "! rm -f GEM-binaries-Linux-x86_64-core_i3-20121106-022124/*man\n",
    "! rm -f GEM-binaries-Linux-x86_64-core_i3-20121106-022124/LICENCE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "! cp GEM-binaries-Linux-x86_64-core_i3-20121106-022124/* ~/miniconda2/bin/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DSRC FASTQ compressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DSRC is a FASTQ compressor, it's not needed, but we use it as the size of the files is significantly smaller than using gunzip (>30%), and, more importantly, the access to them can be parallelized, and is much faster than any other alternative.\n",
    "\n",
    "It can be downloaded from https://github.com/lrog/dsrc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "wget: /home/dcastillo/miniconda2/lib/libuuid.so.1: no version information available (required by wget)\n",
      "--2018-09-16 17:44:10--  http://sun.aei.polsl.pl/dsrc/download/2.0rc/dsrc\n",
      "Resolving sun.aei.polsl.pl (sun.aei.polsl.pl)... 157.158.77.3\n",
      "Connecting to sun.aei.polsl.pl (sun.aei.polsl.pl)|157.158.77.3|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1761768 (1.7M) [text/plain]\n",
      "Saving to: ‘dsrc’\n",
      "\n",
      "100%[======================================>] 1,761,768   1.95MB/s   in 0.9s   \n",
      "\n",
      "2018-09-16 17:44:11 (1.95 MB/s) - ‘dsrc’ saved [1761768/1761768]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! wget http://sun.aei.polsl.pl/dsrc/download/2.0rc/dsrc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "! chmod +x dsrc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And copy to miniconda bin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mv dsrc ~/miniconda2/bin/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TADbit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For now TADbit is not available through conda or pip package manager, so to install it we will have to clone the repository, and compile the binaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "! git clone git@github.com:3DGenomes/TADbit.git -b dev\n",
    "! cd TADbit; python setup.py install"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}