IPFS_Walkthrough.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intro To IPFS\n",
    "\n",
    "IPFS is a a distributed file system that is focused on content addressing the files instead addressing files by location.  This is accomplished by running the content of a file through a hashing algorithm, like SHA256, and using that as the identifier.  You will need to install and intialize IPFS for the following commands to work for you.  A link to the website is below the image.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IPFS Website"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://ipfs.io/images/video-still-demo.png\"></img>\n",
    "<a href=\"http://ipfs.io\" target=\"_blank\">IPFS</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview\n",
    "\n",
    "A walk through of an example use for IPFS will be considered.  This will assume you have IPFS installed already.  \n",
    "\n",
    "If you do not want to install IPFS, a web app was created.  The web app is able to be downloaded from github.  Following along with the README for <a href=\"https://github.com/kdgosik/WDL_Tasks\" target=\"_blank\">this repository</a> will get the app up and running with a few commands. (assuming that you have npm installed already)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example\n",
    "\n",
    "First we will start off by a simple bash command to look at the contents of a wdl file.  This file will then be used to put into the IPFS file system.  The hash will be displayed.  We can then use that hash to retrieve the content of the wdl file again.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "workflow preprocess_depth {\n",
      "  Array[File] beds\n",
      "  String batch\n",
      "\n",
      "  call concat_batch as preprocess_DELs {\n",
      "    input:\n",
      "      beds=beds,\n",
      "      batch=batch,\n",
      "      svtype=\"DEL\"\n",
      "  }\n",
      "\n",
      "  call concat_batch as preprocess_DUPs {\n",
      "    input:\n",
      "      beds=beds,\n",
      "      batch=batch,\n",
      "      svtype=\"DUP\"\n",
      "  }\n",
      "\n",
      "  output {\n",
      "    File del_bed = preprocess_DELs.bed\n",
      "    File dup_bed = preprocess_DUPs.bed\n",
      "    File del_bed_idx = preprocess_DELs.bed_idx\n",
      "    File dup_bed_idx = preprocess_DUPs.bed_idx\n",
      "  }\n",
      "}\n",
      "\n",
      "task concat_batch {\n",
      "  Array[File] beds\n",
      "  String svtype\n",
      "  String batch\n",
      "\n",
      "  command <<<\n",
      "    zcat ${sep=' ' beds} \\\n",
      "      | sed -e '/^#chr/d' -e 's/cn.MOPS/cnmops/g' \\\n",
      "      | awk -v svtype=${svtype} '($6==svtype)' \\\n",
      "      | sort -k1,1V -k2,2n \\\n",
      "      | awk -v OFS=\"\\t\" -v svtype=${svtype} -v batch=${batch} '{$4=batch\"_\"svtype\"_\"NR; print}' \\\n",
      "      | cat <(echo -e \"#chr\\tstart\\tend\\tname\\tsample\\tsvtype\\tsources\") - \\\n",
      "      | bgzip -c \\\n",
      "      > ${batch}.${svtype}.bed.gz;\n",
      "  tabix -p bed ${batch}.${svtype}.bed.gz\n",
      "  >>>\n",
      "\n",
      "  output {\n",
      "    File bed=\"${batch}.${svtype}.bed.gz\"\n",
      "    File bed_idx=\"${batch}.${svtype}.bed.gz.tbi\"\n",
      "  }\n",
      "  \n",
      "  runtime {\n",
      "    docker: \"msto/sv-pipeline\"\n",
      "  }\n",
      "}"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cat /Users/kgosik/Documents/Projects/WebApps/WDL_Tasks/ParseWDLs/ValidatedWDLs/00_depth_preprocessing.wdl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding the file to IPFS\n",
    "\n",
    "Make sure to have IPFS installed and intialized before attempting to run the following commands.  Ths first command is to add the file to the IPFS file system. This will result in a message with the added hash for the content of the wdl file.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "added QmddfhXwGhxEWVfze5GUXBy4BaxeRpR7gBAUQkRV6TmZu6 00_depth_preprocessing.wdl\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 1.12 KB / 1.12 KB  100.00% 0s\u001b[2K\r"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "ipfs add ParseWDLs/ValidatedWDLs/00_depth_preprocessing.wdl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieving the file content by hash\n",
    "\n",
    "Once you have the above hash, this corresponds to the content of the wdl file.  If you add the same content of the file, you will get the same hash.  Even if the name of the file is different or it is added from another location, you will always get the same hash.  To retrieve the content you run the following command. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "workflow preprocess_depth {\n",
      "  Array[File] beds\n",
      "  String batch\n",
      "\n",
      "  call concat_batch as preprocess_DELs {\n",
      "    input:\n",
      "      beds=beds,\n",
      "      batch=batch,\n",
      "      svtype=\"DEL\"\n",
      "  }\n",
      "\n",
      "  call concat_batch as preprocess_DUPs {\n",
      "    input:\n",
      "      beds=beds,\n",
      "      batch=batch,\n",
      "      svtype=\"DUP\"\n",
      "  }\n",
      "\n",
      "  output {\n",
      "    File del_bed = preprocess_DELs.bed\n",
      "    File dup_bed = preprocess_DUPs.bed\n",
      "    File del_bed_idx = preprocess_DELs.bed_idx\n",
      "    File dup_bed_idx = preprocess_DUPs.bed_idx\n",
      "  }\n",
      "}\n",
      "\n",
      "task concat_batch {\n",
      "  Array[File] beds\n",
      "  String svtype\n",
      "  String batch\n",
      "\n",
      "  command <<<\n",
      "    zcat ${sep=' ' beds} \\\n",
      "      | sed -e '/^#chr/d' -e 's/cn.MOPS/cnmops/g' \\\n",
      "      | awk -v svtype=${svtype} '($6==svtype)' \\\n",
      "      | sort -k1,1V -k2,2n \\\n",
      "      | awk -v OFS=\"\\t\" -v svtype=${svtype} -v batch=${batch} '{$4=batch\"_\"svtype\"_\"NR; print}' \\\n",
      "      | cat <(echo -e \"#chr\\tstart\\tend\\tname\\tsample\\tsvtype\\tsources\") - \\\n",
      "      | bgzip -c \\\n",
      "      > ${batch}.${svtype}.bed.gz;\n",
      "  tabix -p bed ${batch}.${svtype}.bed.gz\n",
      "  >>>\n",
      "\n",
      "  output {\n",
      "    File bed=\"${batch}.${svtype}.bed.gz\"\n",
      "    File bed_idx=\"${batch}.${svtype}.bed.gz.tbi\"\n",
      "  }\n",
      "  \n",
      "  runtime {\n",
      "    docker: \"msto/sv-pipeline\"\n",
      "  }\n",
      "}"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "ipfs cat QmddfhXwGhxEWVfze5GUXBy4BaxeRpR7gBAUQkRV6TmZu6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linking Files\n",
    "\n",
    "There is also the ability to link two files together.  This will result in a unique hash that represents the two files linked together.  The first thing that needs to be done is to add another file that we will link to the first file added.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "added QmYjBMLVeZTJ2RYHkVnVarkQ1qSv5xnqSzTsCbVZuzqL6w 00_pesr_processing_single_algorithm.wdl\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 1.17 KB / 1.17 KB  100.00% 0s\u001b[2K\r"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "ipfs add ParseWDLs/ValidatedWDLs/00_pesr_processing_single_algorithm.wdl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "workflow preprocess_algorithm {\n",
      "  File vcf\n",
      "  File contigs\n",
      "  String sample\n",
      "  String algorithm\n",
      "  Int min_svsize\n",
      "\n",
      "  call standardize_vcf {\n",
      "    input: \n",
      "      raw_vcf=vcf,\n",
      "      algorithm=algorithm,\n",
      "      group=sample,\n",
      "      contigs=contigs,\n",
      "      min_svsize=min_svsize\n",
      "  }\n",
      "\n",
      "  call sort_vcf {\n",
      "    input: \n",
      "      unsorted_vcf=standardize_vcf.std_vcf,\n",
      "      algorithm=algorithm,\n",
      "      group=sample\n",
      "  }\n",
      "\n",
      "  output {\n",
      "    File std_vcf = sort_vcf.sorted_vcf\n",
      "  }\n",
      "}\n",
      "\n",
      "task standardize_vcf {\n",
      "  File raw_vcf\n",
      "  File contigs\n",
      "  Int min_svsize\n",
      "  String algorithm\n",
      "  String group\n",
      "\n",
      "  command {\n",
      "    svtk standardize --prefix ${algorithm}_${group} --contigs ${contigs} --min-size ${min_svsize} ${raw_vcf} ${algorithm}.${group}.vcf ${algorithm}\n",
      "  }\n",
      "\n",
      "  output { \n",
      "    File std_vcf=\"${algorithm}.${group}.vcf\"\n",
      "    String group_=\"${group}\"\n",
      "  }\n",
      "  \n",
      "  runtime {\n",
      "    docker: \"msto/sv-pipeline\"\n",
      "  }\n",
      "}\n",
      "\n",
      "task sort_vcf {\n",
      "  File unsorted_vcf\n",
      "  String algorithm\n",
      "  String group\n",
      " \n",
      "  command {\n",
      "    vcf-sort -c ${unsorted_vcf} | bgzip -c > ${algorithm}.${group}.vcf.gz;\n",
      "    tabix -p vcf ${algorithm}.${group}.vcf.gz\n",
      "  }\n",
      "  \n",
      "  output {\n",
      "    File sorted_vcf=\"${algorithm}.${group}.vcf.gz\"\n",
      "  }\n",
      "  \n",
      "  runtime {\n",
      "    docker: \"msto/sv-pipeline\"\n",
      "  }\n",
      "}"
     ]
    }
   ],
   "source": [
    "%%bash \n",
    "\n",
    "ipfs cat QmYjBMLVeZTJ2RYHkVnVarkQ1qSv5xnqSzTsCbVZuzqL6w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have two files added to the file system, you are able to take the hashing of the two files and link them together into an object.  This will produce a single hash but you are able to retrieve the linked content of one or both of the files involved.  The following command is run to link the first wdl to the second wdl and returns a hash.  We can name the link between the files.  In this case we will just call it \"linked-wdls\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "QmTBCvac8akrk1nWBZxQcc1paRJ9iKGPYnjWD2XXfGbcuW\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "ipfs object patch add-link QmddfhXwGhxEWVfze5GUXBy4BaxeRpR7gBAUQkRV6TmZu6 linked-wdls QmYjBMLVeZTJ2RYHkVnVarkQ1qSv5xnqSzTsCbVZuzqL6w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the two wdl files linked together we can explore the hash to see what content we can get from it.  The first thing we can do is run the <code>ipfs cat</code> command to see the entire file content.  This will just be the two files' content concatenated into one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "workflow preprocess_depth {\n",
      "  Array[File] beds\n",
      "  String batch\n",
      "\n",
      "  call concat_batch as preprocess_DELs {\n",
      "    input:\n",
      "      beds=beds,\n",
      "      batch=batch,\n",
      "      svtype=\"DEL\"\n",
      "  }\n",
      "\n",
      "  call concat_batch as preprocess_DUPs {\n",
      "    input:\n",
      "      beds=beds,\n",
      "      batch=batch,\n",
      "      svtype=\"DUP\"\n",
      "  }\n",
      "\n",
      "  output {\n",
      "    File del_bed = preprocess_DELs.bed\n",
      "    File dup_bed = preprocess_DUPs.bed\n",
      "    File del_bed_idx = preprocess_DELs.bed_idx\n",
      "    File dup_bed_idx = preprocess_DUPs.bed_idx\n",
      "  }\n",
      "}\n",
      "\n",
      "task concat_batch {\n",
      "  Array[File] beds\n",
      "  String svtype\n",
      "  String batch\n",
      "\n",
      "  command <<<\n",
      "    zcat ${sep=' ' beds} \\\n",
      "      | sed -e '/^#chr/d' -e 's/cn.MOPS/cnmops/g' \\\n",
      "      | awk -v svtype=${svtype} '($6==svtype)' \\\n",
      "      | sort -k1,1V -k2,2n \\\n",
      "      | awk -v OFS=\"\\t\" -v svtype=${svtype} -v batch=${batch} '{$4=batch\"_\"svtype\"_\"NR; print}' \\\n",
      "      | cat <(echo -e \"#chr\\tstart\\tend\\tname\\tsample\\tsvtype\\tsources\") - \\\n",
      "      | bgzip -c \\\n",
      "      > ${batch}.${svtype}.bed.gz;\n",
      "  tabix -p bed ${batch}.${svtype}.bed.gz\n",
      "  >>>\n",
      "\n",
      "  output {\n",
      "    File bed=\"${batch}.${svtype}.bed.gz\"\n",
      "    File bed_idx=\"${batch}.${svtype}.bed.gz.tbi\"\n",
      "  }\n",
      "  \n",
      "  runtime {\n",
      "    docker: \"msto/sv-pipeline\"\n",
      "  }\n",
      "}workflow preprocess_algorithm {\n",
      "  File vcf\n",
      "  File contigs\n",
      "  String sample\n",
      "  String algorithm\n",
      "  Int min_svsize\n",
      "\n",
      "  call standardize_vcf {\n",
      "    input: \n",
      "      raw_vcf=vcf,\n",
      "      algorithm=algorithm,\n",
      "      group=sample,\n",
      "      contigs=contigs,\n",
      "      min_svsize=min_svsize\n",
      "  }\n",
      "\n",
      "  call sort_vcf {\n",
      "    input: \n",
      "      unsorted_vcf=standardize_vcf.std_vcf,\n",
      "      algorithm=algorithm,\n",
      "      group=sample\n",
      "  }\n",
      "\n",
      "  output {\n",
      "    File std_vcf = sort_vcf.sorted_vcf\n",
      "  }\n",
      "}\n",
      "\n",
      "task standardize_vcf {\n",
      "  File raw_vcf\n",
      "  File contigs\n",
      "  Int min_svsize\n",
      "  String algorithm\n",
      "  String group\n",
      "\n",
      "  command {\n",
      "    svtk standardize --prefix ${algorithm}_${group} --contigs ${contigs} --min-size ${min_svsize} ${raw_vcf} ${algorithm}.${group}.vcf ${algorithm}\n",
      "  }\n",
      "\n",
      "  output { \n",
      "    File std_vcf=\"${algorithm}.${group}.vcf\"\n",
      "    String group_=\"${group}\"\n",
      "  }\n",
      "  \n",
      "  runtime {\n",
      "    docker: \"msto/sv-pipeline\"\n",
      "  }\n",
      "}\n",
      "\n",
      "task sort_vcf {\n",
      "  File unsorted_vcf\n",
      "  String algorithm\n",
      "  String group\n",
      " \n",
      "  command {\n",
      "    vcf-sort -c ${unsorted_vcf} | bgzip -c > ${algorithm}.${group}.vcf.gz;\n",
      "    tabix -p vcf ${algorithm}.${group}.vcf.gz\n",
      "  }\n",
      "  \n",
      "  output {\n",
      "    File sorted_vcf=\"${algorithm}.${group}.vcf.gz\"\n",
      "  }\n",
      "  \n",
      "  runtime {\n",
      "    docker: \"msto/sv-pipeline\"\n",
      "  }\n",
      "}"
     ]
    }
   ],
   "source": [
    "%%bash \n",
    "\n",
    "ipfs cat QmTBCvac8akrk1nWBZxQcc1paRJ9iKGPYnjWD2XXfGbcuW"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also explore the created object piece by piece.  This we will use the command <code>ipfs object</code> set of commands.  We can look at the object itself, the hash of the parts of the objects or any links that were established in the object.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "messy version:\n",
      "{\"Links\":[{\"Name\":\"linked-wdls\",\"Hash\":\"QmYjBMLVeZTJ2RYHkVnVarkQ1qSv5xnqSzTsCbVZuzqL6w\",\"Size\":1205}],\"Data\":\"\\u0008\\u0002\\u0012\\ufffd\\tworkflow preprocess_depth {\\n  Array[File] beds\\n  String batch\\n\\n  call concat_batch as preprocess_DELs {\\n    input:\\n      beds=beds,\\n      batch=batch,\\n      svtype=\\\"DEL\\\"\\n  }\\n\\n  call concat_batch as preprocess_DUPs {\\n    input:\\n      beds=beds,\\n      batch=batch,\\n      svtype=\\\"DUP\\\"\\n  }\\n\\n  output {\\n    File del_bed = preprocess_DELs.bed\\n    File dup_bed = preprocess_DUPs.bed\\n    File del_bed_idx = preprocess_DELs.bed_idx\\n    File dup_bed_idx = preprocess_DUPs.bed_idx\\n  }\\n}\\n\\ntask concat_batch {\\n  Array[File] beds\\n  String svtype\\n  String batch\\n\\n  command \\u003c\\u003c\\u003c\\n    zcat ${sep=' ' beds} \\\\\\n      | sed -e '/^#chr/d' -e 's/cn.MOPS/cnmops/g' \\\\\\n      | awk -v svtype=${svtype} '($6==svtype)' \\\\\\n      | sort -k1,1V -k2,2n \\\\\\n      | awk -v OFS=\\\"\\\\t\\\" -v svtype=${svtype} -v batch=${batch} '{$4=batch\\\"_\\\"svtype\\\"_\\\"NR; print}' \\\\\\n      | cat \\u003c(echo -e \\\"#chr\\\\tstart\\\\tend\\\\tname\\\\tsample\\\\tsvtype\\\\tsources\\\") - \\\\\\n      | bgzip -c \\\\\\n      \\u003e ${batch}.${svtype}.bed.gz;\\n  tabix -p bed ${batch}.${svtype}.bed.gz\\n  \\u003e\\u003e\\u003e\\n\\n  output {\\n    File bed=\\\"${batch}.${svtype}.bed.gz\\\"\\n    File bed_idx=\\\"${batch}.${svtype}.bed.gz.tbi\\\"\\n  }\\n  \\n  runtime {\\n    docker: \\\"msto/sv-pipeline\\\"\\n  }\\n}\\u0018\\ufffd\\t\"}\n",
      "\n",
      "\n",
      "pretty print:\n",
      "{\n",
      "    \"Data\": \"\\b\\u0002\\u0012\\ufffd\\tworkflow preprocess_depth {\\n  Array[File] beds\\n  String batch\\n\\n  call concat_batch as preprocess_DELs {\\n    input:\\n      beds=beds,\\n      batch=batch,\\n      svtype=\\\"DEL\\\"\\n  }\\n\\n  call concat_batch as preprocess_DUPs {\\n    input:\\n      beds=beds,\\n      batch=batch,\\n      svtype=\\\"DUP\\\"\\n  }\\n\\n  output {\\n    File del_bed = preprocess_DELs.bed\\n    File dup_bed = preprocess_DUPs.bed\\n    File del_bed_idx = preprocess_DELs.bed_idx\\n    File dup_bed_idx = preprocess_DUPs.bed_idx\\n  }\\n}\\n\\ntask concat_batch {\\n  Array[File] beds\\n  String svtype\\n  String batch\\n\\n  command <<<\\n    zcat ${sep=' ' beds} \\\\\\n      | sed -e '/^#chr/d' -e 's/cn.MOPS/cnmops/g' \\\\\\n      | awk -v svtype=${svtype} '($6==svtype)' \\\\\\n      | sort -k1,1V -k2,2n \\\\\\n      | awk -v OFS=\\\"\\\\t\\\" -v svtype=${svtype} -v batch=${batch} '{$4=batch\\\"_\\\"svtype\\\"_\\\"NR; print}' \\\\\\n      | cat <(echo -e \\\"#chr\\\\tstart\\\\tend\\\\tname\\\\tsample\\\\tsvtype\\\\tsources\\\") - \\\\\\n      | bgzip -c \\\\\\n      > ${batch}.${svtype}.bed.gz;\\n  tabix -p bed ${batch}.${svtype}.bed.gz\\n  >>>\\n\\n  output {\\n    File bed=\\\"${batch}.${svtype}.bed.gz\\\"\\n    File bed_idx=\\\"${batch}.${svtype}.bed.gz.tbi\\\"\\n  }\\n  \\n  runtime {\\n    docker: \\\"msto/sv-pipeline\\\"\\n  }\\n}\\u0018\\ufffd\\t\",\n",
      "    \"Links\": [\n",
      "        {\n",
      "            \"Hash\": \"QmYjBMLVeZTJ2RYHkVnVarkQ1qSv5xnqSzTsCbVZuzqL6w\",\n",
      "            \"Name\": \"linked-wdls\",\n",
      "            \"Size\": 1205\n",
      "        }\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "echo messy version:\n",
    "## messy version\n",
    "ipfs object get QmTBCvac8akrk1nWBZxQcc1paRJ9iKGPYnjWD2XXfGbcuW\n",
    "echo \n",
    "echo\n",
    "echo pretty print:\n",
    "\n",
    "## pretty print with python\n",
    "ipfs object get QmTBCvac8akrk1nWBZxQcc1paRJ9iKGPYnjWD2XXfGbcuW | python -m json.tool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\u0002\u0012�\tworkflow preprocess_depth {\n",
      "  Array[File] beds\n",
      "  String batch\n",
      "\n",
      "  call concat_batch as preprocess_DELs {\n",
      "    input:\n",
      "      beds=beds,\n",
      "      batch=batch,\n",
      "      svtype=\"DEL\"\n",
      "  }\n",
      "\n",
      "  call concat_batch as preprocess_DUPs {\n",
      "    input:\n",
      "      beds=beds,\n",
      "      batch=batch,\n",
      "      svtype=\"DUP\"\n",
      "  }\n",
      "\n",
      "  output {\n",
      "    File del_bed = preprocess_DELs.bed\n",
      "    File dup_bed = preprocess_DUPs.bed\n",
      "    File del_bed_idx = preprocess_DELs.bed_idx\n",
      "    File dup_bed_idx = preprocess_DUPs.bed_idx\n",
      "  }\n",
      "}\n",
      "\n",
      "task concat_batch {\n",
      "  Array[File] beds\n",
      "  String svtype\n",
      "  String batch\n",
      "\n",
      "  command <<<\n",
      "    zcat ${sep=' ' beds} \\\n",
      "      | sed -e '/^#chr/d' -e 's/cn.MOPS/cnmops/g' \\\n",
      "      | awk -v svtype=${svtype} '($6==svtype)' \\\n",
      "      | sort -k1,1V -k2,2n \\\n",
      "      | awk -v OFS=\"\\t\" -v svtype=${svtype} -v batch=${batch} '{$4=batch\"_\"svtype\"_\"NR; print}' \\\n",
      "      | cat <(echo -e \"#chr\\tstart\\tend\\tname\\tsample\\tsvtype\\tsources\") - \\\n",
      "      | bgzip -c \\\n",
      "      > ${batch}.${svtype}.bed.gz;\n",
      "  tabix -p bed ${batch}.${svtype}.bed.gz\n",
      "  >>>\n",
      "\n",
      "  output {\n",
      "    File bed=\"${batch}.${svtype}.bed.gz\"\n",
      "    File bed_idx=\"${batch}.${svtype}.bed.gz.tbi\"\n",
      "  }\n",
      "  \n",
      "  runtime {\n",
      "    docker: \"msto/sv-pipeline\"\n",
      "  }\n",
      "}\u0018�\t"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "ipfs object data QmTBCvac8akrk1nWBZxQcc1paRJ9iKGPYnjWD2XXfGbcuW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "QmYjBMLVeZTJ2RYHkVnVarkQ1qSv5xnqSzTsCbVZuzqL6w 1205 linked-wdls \n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "ipfs object links QmTBCvac8akrk1nWBZxQcc1paRJ9iKGPYnjWD2XXfGbcuW"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Web App\n",
    "\n",
    "A web app was created to accomplish similar task as what was just shown.  It may take a few seconds for the drop down menus to be filled.  The drop down menus are filled with the WDL ids from using the GA4GH Tool (get ga4gh/v1/tools) registry from this <a href=\"https://api.firecloud.org/\" target=\"_blank\">api</a>.  You could also skip calling a tool from the api and type any text you wanted into the text area.  This content will be hashed instead and will be able to be retrieved later.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"AppScreen.png\"></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of adding the file via <code>ipfs add</code> command, you are able to select from a drop down menu.  Once you have selected the WDL you would like to had click the submit button.  You can check the content of the file in the text area.  Once you are satisfied you can use the 'add to ipfs' button to add the content and it will render the hash for you.  The blue box is identical to the yellow box.  The yellow is for the first WDL and the blue is for the second WDL to for it to be linked to.  Make sure you have added both WDLs selected to IPFS before moving on to the green box labeled 'Linked Data'.  Once you see the hash of each file in both the yellow and blue box, you can move down to the green box and push the 'Link Data' button.  This will hash the two WDL files together and output the resulting hash. This hash is a signature of the two linked files.  This can be used to retrieve the two WDLs in the future."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"AppBottomScreen.png\"></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scrolling down you will see a box for a hash to be entered.  This could be any hash but is intended to be for linked WDLs like above. You can copy and paste the new hash that was created by the 'Linked Data' section.  Once pasted into the text box you can select the 'Submit' button.  This should be able to retrieve the same content from the above two WDLs.  WDL 1 would be from the yellow box and WDL 2 would be from the blue box.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}