Cohort1_Community_Analyses_Through_QIIME.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Community Analyses through QIIME (Cohort 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "11/09/2020<br>\n",
    "Author: Robert Mills<br>\n",
    "Environment: Qiime2-2019.7 & Qiime1 when specified<br> \n",
    "<br>\n",
    "<i>This notebook contains code for reproducing several figures related to microbial community  relationships to IBD mediated through Qiime2 from the manuscript, <b> \"Meta–omics Reveals Microbiome Driven Proteolysis as a Contributing Factor to Severity of Ulcerative Colitis Disease Activity\" </b>by Mills et al. <br><br>\n",
    "More specifically this notebook is a companion to a separate notebook for Cohort 2 QIIME analyses and has code for recreating analyses performed related to the UC cohort 1 as well as compiling beta-diversity associations to clinical metrics. <br><br>\n",
    "Data sheets used here will be made available on Massive.ucsd.edu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Import dependencies\n",
    "%matplotlib inline\n",
    "\n",
    "from os import mkdir\n",
    "import os\n",
    "import copy\n",
    "from os.path import abspath, join as pjoin, exists\n",
    "from shutil import copy2, move\n",
    "from time import strftime, strptime\n",
    "from numpy import nan, isnan, arange\n",
    "from pandas import read_csv, Series, DataFrame\n",
    "from IPython.display import Image\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Biom Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Make subset of bacterial proteins only:\n",
    "df = pd.read_csv(\"./2Search/CSVs/NormalizedCommonReps.txt\", \n",
    "                 sep = '\\t', index_col= \"datarest$ProteinID\")\n",
    "\n",
    "#Remove any human derived proteins\n",
    "df = df[df.index.str.contains('k99_') != False]\n",
    "#df.to_csv('./2Search/CSVs/NormalizedCommonReps_nohuman.txt', sep = '\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#2 Search pDB Approach - no human\n",
    "#Convert tab-separated file to biom file\n",
    "!biom convert -i ./2Search/CSVs/NormalizedCommonReps_nohuman.txt \\\n",
    "-o ./2Search/NormalizedCommonReps2_nohuman.biom \\\n",
    "-m ../UC_MP_Emperor_Map_1.txt \\\n",
    "--table-type=\"OTU table\" --to-hdf5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#2 Search pDB Approach\n",
    "#Convert tab-separated file to biom file\n",
    "!biom convert -i ./2Search/CSVs/NormalizedCommonReps.txt \\\n",
    "-o ./2Search/NormalizedCommonReps2.biom \\\n",
    "-m ../UC_MP_Emperor_Map_1.txt \\\n",
    "--table-type=\"OTU table\" --to-hdf5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Metabolomics\n",
    "#Convert tab-separated file to biom file\n",
    "!biom convert -i ../Metabolomics/Feature_table.txt \\\n",
    "-o ./Metabolomics.biom \\\n",
    "-m ../UC_MP_Emperor_Map_1.txt \\\n",
    "--table-type=\"OTU table\" --to-hdf5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#16S\n",
    "#Convert tab-separated file to biom file\n",
    "!biom convert -i ../Genomics/16S/reference-hit_idswap2_blankremove.txt \\\n",
    "-o ./16S.biom \\\n",
    "-m ../UC_MP_Emperor_Map_1.txt \\\n",
    "--table-type=\"OTU table\" --to-hdf5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Serum\n",
    "!biom convert -i ../Serum/CSVs/NormalizedCommonReps_ids.txt \\\n",
    "-o ./Serum_Common.biom \\\n",
    "-m ../UC_MP_Emperor_Map_1.txt \\\n",
    "--table-type=\"OTU table\" --to-hdf5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#MG\n",
    "!biom convert -i ../Genomics/Shotgun/Salmon_CPMs_0s.txt \\\n",
    "-o ./Salmon_CPMs.biom \\\n",
    "-m ../UC_MP_Emperor_Map_1.txt \\\n",
    "--table-type=\"OTU table\" --to-hdf5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import all as Qiime2 artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime tools import \\\n",
    "  --input-path ./Metabolomics.biom \\\n",
    "  --type 'FeatureTable[Frequency]' \\\n",
    "  --output-path Metabolomics_biom.qza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime tools import \\\n",
    "  --input-path ./2Search/NormalizedCommonReps2.biom\\\n",
    "  --type 'FeatureTable[Frequency]' \\\n",
    "  --output-path 2search_common_biom.qza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime tools import \\\n",
    "  --input-path ./16S.biom\\\n",
    "  --type 'FeatureTable[Frequency]' \\\n",
    "  --output-path 16S_biom.qza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime tools import \\\n",
    "  --input-path ./Salmon_CPMs.biom\\\n",
    "  --type 'FeatureTable[Frequency]' \\\n",
    "  --output-path ./Salmon_CPMs_biom.qza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime tools import \\\n",
    "  --input-path ./Serum_Common.biom\\\n",
    "  --type 'FeatureTable[Frequency]' \\\n",
    "  --output-path ./Serum_Common_biom.qza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!qiime tools import \\\n",
    "  --input-path ./2Search/NormalizedCommonReps2_nohuman.biom\\\n",
    "  --type 'FeatureTable[Frequency]' \\\n",
    "  --output-path ./pDB_Common_nohuman_biom.qza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature table summarize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ./pDB_Common_nohuman_biom.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-table summarize \\\n",
    "  --i-table ./pDB_Common_nohuman_biom.qza \\\n",
    "  --o-visualization ./pDB_Common_nohuman_biom.qzv \\\n",
    "  --m-sample-metadata-file ../UC_MP_Emperor_Map.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: Metabolomics_biom.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-table summarize \\\n",
    "  --i-table Metabolomics_biom.qza \\\n",
    "  --o-visualization Metabolomics_biom.qzv \\\n",
    "  --m-sample-metadata-file ../UC_MP_Emperor_Map.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: 2search_biom.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-table summarize \\\n",
    "  --i-table 2search_common_biom.qza \\\n",
    "  --o-visualization 2search_biom.qzv \\\n",
    "  --m-sample-metadata-file ../UC_MP_Emperor_Map.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: 16S_biom.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-table summarize \\\n",
    "  --i-table 16S_biom.qza \\\n",
    "  --o-visualization 16S_biom.qzv \\\n",
    "  --m-sample-metadata-file ../UC_MP_Emperor_Map.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ./Serum_Common_biom.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-table summarize \\\n",
    "  --i-table ./Serum_Common_biom.qza \\\n",
    "  --o-visualization ./Serum_Common_biom.qzv \\\n",
    "  --m-sample-metadata-file ../UC_MP_Emperor_Map.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "^C\r\n",
      "\r\n",
      "Aborted!\r\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-table summarize \\\n",
    "  --i-table ./Salmon_CPMs_biom.qza \\\n",
    "  --o-visualization Salmon_CPMs_biom.qzv \\\n",
    "  --m-sample-metadata-file ../UC_MP_Emperor_Map.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "qiime feature-table summarize \\\n",
    "  --i-table ./Serum_biom.qza \\\n",
    "  --o-visualization Serum_biom.qzv \\\n",
    "  --m-sample-metadata-file ../../UC_MP_Emperor_Map.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ../Genomics/gOTU_Table_UC40_Summary.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-table summarize \\\n",
    "  --i-table ../Genomics/Shotgun/gOTU_table_UC40.qza \\\n",
    "  --o-visualization ../Genomics/gOTU_Table_UC40_Summary.qzv \\\n",
    "  --m-sample-metadata-file ../Genomics/16S/UC_Severity_1_MF_Idswap6.21.18.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filter samples without histology data for quantitative assessment of the relationship between beta-diversity and histology"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/sh: qiime: command not found\r\n"
     ]
    }
   ],
   "source": [
    "## remove samples without histology information \n",
    "!qiime feature-table filter-samples \\\n",
    "  --i-table ./gOTU_table_UC40_2.qza  \\\n",
    "  --m-metadata-file ../UC_Severity/UC_MP_Emperor_Map_v2_2020.txt \\\n",
    "  --p-where \"Geboes_Grade_5_Numeric!='not applicable'\" \\\n",
    "  --o-filtered-table ./gOTU_table_UC40_ForHist.qza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Qiime 1 Beta-Diversity Statistics Adonis & PERMANOVA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../Cohort1_FinalTable_Amp.biom \\\n",
    "-m ../UC_MP_Emperor_Map_v2_2020_q2.txt \\\n",
    "-o ./Qiime/Cohort1_16S -p ../smallDB_correct/PCoA_Plots/paramaters2_16S.txt -t ../Genomics/16S/insertion_tree.relabelled.tre "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../Cohort1_FinalTable_MG.biom \\\n",
    "-m ../UC_MP_Emperor_Map_v2_2020_q2.txt \\\n",
    "-o ./Qiime/Cohort1_MG -p ../smallDB_correct/PCoA_Plots/paramaters2_16S.txt -t ../Genomics/Shotgun/tree.nwk "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../Cohort1_FinalTable_Mb.biom \\\n",
    "-m ../UC_MP_Emperor_Map_v2_2020_q2.txt \\\n",
    "-o ./Qiime/Cohort1_Mb -p ../smallDB_correct/PCoA_Plots/paramaters2.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../Cohort1_FinalTable_Mp.biom \\\n",
    "-m ../UC_MP_Emperor_Map_v2_2020_q2.txt \\\n",
    "-o ./Qiime/Cohort1_Mp -p ../smallDB_correct/PCoA_Plots/paramaters2.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../Cohort1_FinalTable_Ser.biom \\\n",
    "-m ../UC_MP_Emperor_Map_v2_2020_q2.txt \\\n",
    "-o ./Qiime/Cohort1_Ser -p ../smallDB_correct/PCoA_Plots/paramaters2.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For histology where some data was missing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../Cohort1_FinalTable_Amp_ForHist/feature-table.biom \\\n",
    "-m ../UC_MP_Emperor_Map_v2_2020_q2.txt \\\n",
    "-o ./Qiime/Cohort1_16S_ForHist -p ../smallDB_correct/PCoA_Plots/paramaters2_16S.txt -t ../Genomics/16S/insertion_tree.relabelled.tre "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../Cohort1_FinalTable_MG_ForHist/feature-table.biom \\\n",
    "-m ../UC_MP_Emperor_Map_v2_2020_q2.txt \\\n",
    "-o ./Qiime/Cohort1_MG_ForHist -p ../smallDB_correct/PCoA_Plots/paramaters2_16S.txt -t ../Genomics/Shotgun/tree.nwk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../Cohort1_FinalTable_Mp_ForHist/feature-table.biom \\\n",
    "-m ../UC_MP_Emperor_Map_v2_2020_q2.txt \\\n",
    "-o ./Qiime/Cohort1_Mp_ForHist -p ../smallDB_correct/PCoA_Plots/paramaters2.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../Cohort1_FinalTable_Mb_ForHist/feature-table.biom \\\n",
    "-m ../UC_MP_Emperor_Map_v2_2020_q2.txt \\\n",
    "-o ./Qiime/Cohort1_Mb_ForHist -p ../smallDB_correct/PCoA_Plots/paramaters2.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../Cohort1_FinalTable_Ser_ForHist/feature-table.biom \\\n",
    "-m ../UC_MP_Emperor_Map_v2_2020_q2.txt \\\n",
    "-o ./Qiime/Cohort1_Ser_ForHist -p ../smallDB_correct/PCoA_Plots/paramaters2.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For second cohort:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../../IBD200/16S_CommonSamples_tubeid_UC/feature-table.biom \\\n",
    "-m ../../IBD200/Combined_metadata_inallomics_hist.txt \\\n",
    "-o ./Qiime/Cohort2_16S -p ../smallDB_correct/PCoA_Plots/paramaters2_16S.txt -t ../Genomics/16S/insertion_tree.relabelled.tre "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../../IBD200/MB_CommonSamples_tubeid_UC/feature-table.biom \\\n",
    "-m ../../IBD200/Combined_metadata_inallomics_hist.txt \\\n",
    "-o ./Qiime/Cohort2_MB -p ../smallDB_correct/PCoA_Plots/paramaters2.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../../IBD200/MP_CommonSamples_tubeid_UC/feature-table.biom \\\n",
    "-m ../../IBD200/Combined_metadata_inallomics_hist.txt \\\n",
    "-o ./Qiime/Cohort2_MP -p ../smallDB_correct/PCoA_Plots/paramaters2.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../../IBD200/MG_CommonSamples_tubeid_UC/feature-table.biom \\\n",
    "-m ../../IBD200/Combined_metadata_inallomics_hist.txt \\\n",
    "-o ./Qiime/Cohort2_MG -p ../smallDB_correct/PCoA_Plots/paramaters2_16S.txt -t ../Genomics/Shotgun/tree.nwk "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subset to only the UC samples with all quantitative variables filled out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../../IBD200/16S_CommonSamples_tubeid_UC_quant/feature-table.biom \\\n",
    "-m ../../IBD200/Combined_metadata_inallomics_hist.txt \\\n",
    "-o ./Qiime/Cohort2_quant_16S -p ../smallDB_correct/PCoA_Plots/paramaters2_16S.txt -t ../Genomics/16S/insertion_tree.relabelled.tre "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../../IBD200/MB_CommonSamples_tubeid_UC_quant/feature-table.biom \\\n",
    "-m ../../IBD200/Combined_metadata_inallomics_hist.txt \\\n",
    "-o ./Qiime/Cohort2_quant_MB -p ../smallDB_correct/PCoA_Plots/paramaters2.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../../IBD200/MP_CommonSamples_tubeid_UC_quant/feature-table.biom \\\n",
    "-m ../../IBD200/Combined_metadata_inallomics_hist.txt \\\n",
    "-o ./Qiime/Cohort2_quant_MP -p ../smallDB_correct/PCoA_Plots/paramaters2.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!beta_diversity_through_plots.py -i ../../IBD200/MG_CommonSamples_tubeid_UC_quant/feature-table.biom \\\n",
    "-m ../../IBD200/Combined_metadata_inallomics_hist.txt \\\n",
    "-o ./Qiime/Cohort2_quant_MG -p ../smallDB_correct/PCoA_Plots/paramaters2_16S.txt -t ../Genomics/Shotgun/tree.nwk "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Data types which require PERMANOVA or Adonis for quantitative data\n",
    "Perma = ['sex','race','historic_extent','ASA_exposure', 'current_5ASA', 'steroid_exposure', 'current_steroids', 'IM_exposure', 'IM_type', 'biologic_exposure',\n",
    "        'biologic_exposure_type','current_biologic','current_biologic_type','Experiment','TMT_Label']\n",
    "Adonis = ['CRP','Calprotectin', 'partial_Mayo','age','age_diagnosis','disease_duration','height','stool_frequency','rectal_bleeding','PGA','mayo_endoscopic_score','UCEIS_endoscopic_score','COLLECTION_TIMESTAMP','Endoscopy_date']\n",
    "Hist = ['Geboes_Grade_0_Numeric','Geboes_Grade_1_Numeric','Geboes_Grade_2A_Numeric','Geboes_Grade_2B_Numeric','Geboes_Grade_3_Numeric','Geboes_Grade_4_Numeric','Geboes_Grade_5_Numeric']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#For bray curtis metric:\n",
    "dtypes=['Mp','Mb','16S','MG','Ser']\n",
    "for i in dtypes:\n",
    "    for j in Perma:\n",
    "        !compare_categories.py --method permanova -i ./Qiime/Cohort1_$i/bray_curtis_dm.txt -m ../UC_MP_Emperor_Map_v2_2020_q2.txt -c $j -o ./Qiime/Permanova/Cohort1_$i/$j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#For unweighted unifrac metric:\n",
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    for j in Perma:\n",
    "        !compare_categories.py --method permanova -i ./Qiime/Cohort1_$i/unweighted_unifrac_dm.txt -m ../UC_MP_Emperor_Map_v2_2020_q2.txt -c $j -o ./Qiime/Permanova/Cohort1_unweighted_unifrac_$i/$j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Make subsets of the metadata to have only the columns of interest (Quantitative)\n",
    "meta = pd.read_csv('../UC_MP_Emperor_Map_v2_2020_q2.txt',sep='\\t',index_col='#SampleID')\n",
    "meta[Adonis].to_csv('../UC_MP_Emperor_Map_v2_2020_q2_adonis.txt',sep='\\t')\n",
    "meta[Hist].to_csv('../UC_MP_Emperor_Map_v2_2020_q2_hist.txt',sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#For adonis using bray curtis metric:\n",
    "dtypes=['Mp','Mb','16S','MG','Ser']\n",
    "for i in dtypes:\n",
    "    for j in Adonis:\n",
    "        !compare_categories.py --method adonis -i ./Qiime/Cohort1_$i/bray_curtis_dm.txt -m ../UC_MP_Emperor_Map_v2_2020_q2_adonis.txt -c $j -o ./Qiime/Adonis/Cohort1_$i/$j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#For adonis using unweighted unifrac metric:\n",
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    for j in Adonis:\n",
    "        !compare_categories.py --method adonis -i ./Qiime/Cohort1_$i/unweighted_unifrac_dm.txt -m ../UC_MP_Emperor_Map_v2_2020_q2_adonis.txt -c $j -o ./Qiime/Adonis/Cohort1_unweightedunifrac_$i/$j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#For bray curtis metric & histology:\n",
    "dtypes=['Mp','Mb','16S','MG','Ser']\n",
    "for i in dtypes:\n",
    "    for j in Hist:\n",
    "        !compare_categories.py --method adonis -i ./Qiime/Cohort1_ForHist_$i/bray_curtis_dm.txt -m ../UC_MP_Emperor_Map_v2_2020_q2_hist.txt -c $j -o ./Qiime/Adonis/Cohort1_$i/$j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#For unweighted unifrac metric & histology:\n",
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    for j in Hist:\n",
    "        !compare_categories.py --method adonis -i ./Qiime/Cohort1_ForHist_$i/unweighted_unifrac_dm.txt -m ../UC_MP_Emperor_Map_v2_2020_q2_hist.txt -c $j -o ./Qiime/Adonis/Cohort1_unweightedunifrac_$i/$j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Create a dictionary for all of the f and p-values\n",
    "Dictionary={}\n",
    "dtypes=['Mp','Mb','16S','MG','Ser']\n",
    "for i in dtypes:\n",
    "    pvalues=[]\n",
    "    fvalues=[]\n",
    "    cats=[]\n",
    "    for j in Perma:\n",
    "        df=pd.read_csv('./Qiime/Permanova/Cohort1_%s/%s/permanova_results.txt' % (i,j),sep='\\t',index_col='method name')\n",
    "        pval=df.loc['p-value'][0][0:6]\n",
    "        fval=df.loc['test statistic'][0][0:6]\n",
    "        pvalues.append(pval)\n",
    "        fvalues.append(fval)\n",
    "        cats.append(j)\n",
    "        newdf=pd.DataFrame({'Variable':cats,\n",
    "                            'pval_%s_BrayCurtis' % i:pvalues,\n",
    "                           'FStatistic_%s_BrayCurtis' % i :fvalues})\n",
    "        Dictionary[i]=newdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Create a dictionary for all of the f and p-values (Unweighted UniFrac)\n",
    "Dictionary2={}\n",
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    pvalues=[]\n",
    "    fvalues=[]\n",
    "    cats=[]\n",
    "    for j in Perma:\n",
    "        df=pd.read_csv('./Qiime/Permanova/Cohort1_unweighted_unifrac_%s/%s/permanova_results.txt' % (i,j),sep='\\t',index_col='method name')\n",
    "        pval=df.loc['p-value'][0][0:6]\n",
    "        fval=df.loc['test statistic'][0][0:6]\n",
    "        pvalues.append(pval)\n",
    "        fvalues.append(fval)\n",
    "        cats.append(j)\n",
    "        newdf=pd.DataFrame({'Variable':cats,\n",
    "                            'pval_%s_UnweightedUniFrac' % i:pvalues,\n",
    "                           'FStatistic_%s_UnweightedUniFrac' % i :fvalues})\n",
    "        Dictionary2[i]=newdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Categorical_df = Dictionary['Mp'].merge(Dictionary['16S'],left_on='Variable',right_on='Variable')\n",
    "Categorical_df=Categorical_df.merge(Dictionary['Mb'],left_on='Variable',right_on='Variable')\n",
    "Categorical_df=Categorical_df.merge(Dictionary['MG'],left_on='Variable',right_on='Variable')\n",
    "Categorical_df=Categorical_df.merge(Dictionary['Ser'],left_on='Variable',right_on='Variable')\n",
    "Categorical_df=Categorical_df.merge(Dictionary2['MG'],left_on='Variable',right_on='Variable')\n",
    "Categorical_df=Categorical_df.merge(Dictionary2['16S'],left_on='Variable',right_on='Variable')\n",
    "Categorical_df.index = Categorical_df['Variable']\n",
    "Categorical_df.drop(columns='Variable',inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Create a dictionary for all of the f and p-values\n",
    "Quantitative = Hist+Adonis\n",
    "Dictionary={}\n",
    "dtypes=['Mp','Mb','16S','MG','Ser']\n",
    "for i in dtypes:\n",
    "    pvalues=[]\n",
    "    fvalues=[]\n",
    "    r2values=[]\n",
    "    cats=[]\n",
    "    for j in Quantitative:\n",
    "        df=pd.read_csv('./Qiime/Adonis/Cohort1_%s/%s/adonis_results.txt' % (i,j),sep='\\t')\n",
    "        a=list(df.loc[5][0].split(' '))\n",
    "        alist = ' '.join(a).split()\n",
    "        pval=alist[6]\n",
    "        fval=alist[4]\n",
    "        r2=alist[5]\n",
    "        pvalues.append(pval)\n",
    "        fvalues.append(fval)\n",
    "        r2values.append(r2)\n",
    "        cats.append(j)\n",
    "        newdf=pd.DataFrame({'Variable':cats,\n",
    "                            'pval_%s_BrayCurtis' % i:pvalues,\n",
    "                            'R2_%s_BrayCurtis' % i:r2values,\n",
    "                           'FStatistic_%s_BrayCurtis' % i :fvalues})\n",
    "        Dictionary[i]=newdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Create a dictionary for all of the f and p-values\n",
    "Quantitative = Hist+Adonis\n",
    "Dictionary2={}\n",
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    pvalues=[]\n",
    "    fvalues=[]\n",
    "    r2values=[]\n",
    "    cats=[]\n",
    "    for j in Quantitative:\n",
    "        df=pd.read_csv('./Qiime/Adonis/Cohort1_unweightedunifrac_%s/%s/adonis_results.txt' % (i,j),sep='\\t')\n",
    "        a=list(df.loc[5][0].split(' '))\n",
    "        alist = ' '.join(a).split()\n",
    "        pval=alist[6]\n",
    "        fval=alist[4]\n",
    "        r2=alist[5]\n",
    "        pvalues.append(pval)\n",
    "        fvalues.append(fval)\n",
    "        r2values.append(r2)\n",
    "        cats.append(j)\n",
    "        newdf=pd.DataFrame({'Variable':cats,\n",
    "                            'pval_%s_UnweightedUniFrac' % i:pvalues,\n",
    "                            'R2_%s_UnweightedUniFrac' % i:r2values,\n",
    "                           'FStatistic_%s_UnweightedUniFrac' % i :fvalues})\n",
    "        Dictionary2[i]=newdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Quant_df = Dictionary['Mp'].merge(Dictionary['16S'],left_on='Variable',right_on='Variable')\n",
    "Quant_df=Quant_df.merge(Dictionary['Mb'],left_on='Variable',right_on='Variable')\n",
    "Quant_df=Quant_df.merge(Dictionary['MG'],left_on='Variable',right_on='Variable')\n",
    "Quant_df=Quant_df.merge(Dictionary['Ser'],left_on='Variable',right_on='Variable')\n",
    "Quant_df=Quant_df.merge(Dictionary2['MG'],left_on='Variable',right_on='Variable')\n",
    "Quant_df=Quant_df.merge(Dictionary2['16S'],left_on='Variable',right_on='Variable')\n",
    "Quant_df.index = Quant_df['Variable']\n",
    "Quant_df.drop(columns='Variable',inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dtypes=['Mp','Mb','16S','MG','Ser']\n",
    "for i in dtypes:\n",
    "    Quant_df['%s'%i] = Quant_df['pval_%s_BrayCurtis'%i]+', '+Quant_df['FStatistic_%s_BrayCurtis'%i] +', '+Quant_df['R2_%s_BrayCurtis'%i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    Quant_df['%s Unweighted UniFrac'%i] = Quant_df['pval_%s_UnweightedUniFrac'%i]+', '+Quant_df['FStatistic_%s_UnweightedUniFrac'%i] +', '+Quant_df['R2_%s_UnweightedUniFrac'%i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dtypes=['Mp','Mb','16S','MG','Ser']\n",
    "for i in dtypes:\n",
    "    Categorical_df['%s'%i] = Categorical_df['pval_%s_BrayCurtis'%i]+', '+Categorical_df['FStatistic_%s_BrayCurtis'%i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    Categorical_df['%s Unweighted UniFrac'%i] = Categorical_df['pval_%s_UnweightedUniFrac'%i]+', '+Categorical_df['FStatistic_%s_UnweightedUniFrac'%i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "totaldf=pd.concat([Quant_df,Categorical_df])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "totaldf.to_csv('../UC_Cohort1_AdonisPermanova_Stats.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Perform analyses for Second Cohort"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Data types which require PERMANOVA categorical significance.\n",
    "Categorical=['sex','Race','ASA_Exposure','Biologic_Exposure',\n",
    "             'Biologic_Exposure_Type_IFX1_ADA2_GOL3_VDZ4_SIM5_UST6_TOF7',\n",
    "             'Current_5ASA','Current_Antidepressents','Current_Biologic','Current_Biologic_Type_IFX1_ADA2_GOL3_VDZ4_SIM5_UST6_TOF7',\n",
    "            'Current_IM','Current_IM_Type_1AZA_26MP_3MTX','Current_Steroids','Historic_Extent',\n",
    "             'IM_Exposure','IM_Exposure_Type_1AZA_26MP_3MTX','Smoker','Steroid_Exposure']\n",
    "\n",
    "Quantitative=['PGA','host_height','Rectal_Bleeding','Mayo_Endoscopic_Score','UCEIS','partial_Mayo','Age_at_Diagnosis','Disease_Duration',\n",
    "             'Stool_Frequency','host_age','Geboes_Grade_0_Numeric','Geboes_Grade_1_Numeric','Geboes_Grade_2A_Numeric','Geboes_Grade_2B_Numeric','Geboes_Grade_3_Numeric','Geboes_Grade_4_Numeric','Geboes_Grade_5_Numeric']\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#For bray curtis metric:\n",
    "dtypes=['MP','MB','16S','MG']\n",
    "for i in dtypes:\n",
    "    for j in Categorical:\n",
    "        !compare_categories.py --method permanova -i ./Qiime/Cohort2_$i/bray_curtis_dm.txt -m ../../IBD200/Combined_metadata_inallomics_hist.txt -c $j -o ./Qiime/Permanova/Cohort2_$i/$j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#For unifrac curtis metric:\n",
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    for j in Categorical:\n",
    "        !compare_categories.py --method permanova -i ./Qiime/Cohort2_$i/unweighted_unifrac_dm.txt -m ../../IBD200/Combined_metadata_inallomics_hist.txt -c $j -o ./Qiime/Permanova/Cohort2_unweighted_unifrac_$i/$j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create a dictionary for all of the f and p-values\n",
    "Dictionary={}\n",
    "dtypes=['MP','MB','16S','MG']\n",
    "for i in dtypes:\n",
    "    pvalues=[]\n",
    "    fvalues=[]\n",
    "    cats=[]\n",
    "    for j in Categorical:\n",
    "        df=pd.read_csv('./Qiime/Permanova/Cohort2_%s/%s/permanova_results.txt' % (i,j),sep='\\t',index_col='method name')\n",
    "        pval=df.loc['p-value'][0][0:6]\n",
    "        fval=df.loc['test statistic'][0][0:6]\n",
    "        pvalues.append(pval)\n",
    "        fvalues.append(fval)\n",
    "        cats.append(j)\n",
    "        newdf=pd.DataFrame({'Variable':cats,\n",
    "                            'pval_%s_BrayCurtis' % i:pvalues,\n",
    "                           'FStatistic_%s_BrayCurtis' % i :fvalues})\n",
    "        Dictionary[i]=newdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Categorical_df = Dictionary['MP'].merge(Dictionary['16S'],left_on='Variable',right_on='Variable')\n",
    "Categorical_df=Categorical_df.merge(Dictionary['MB'],left_on='Variable',right_on='Variable')\n",
    "Categorical_df=Categorical_df.merge(Dictionary['MG'],left_on='Variable',right_on='Variable')\n",
    "Categorical_df.index=Categorical_df['Variable']\n",
    "Categorical_df=Categorical_df.drop(columns='Variable')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create a dictionary for all of the f and p-values (Unweighted UniFrac)\n",
    "Dictionary={}\n",
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    pvalues=[]\n",
    "    fvalues=[]\n",
    "    cats=[]\n",
    "    for j in Categorical:\n",
    "        df=pd.read_csv('./Qiime/Permanova/Cohort2_unweighted_unifrac_%s/%s/permanova_results.txt' % (i,j),sep='\\t',index_col='method name')\n",
    "        pval=df.loc['p-value'][0][0:6]\n",
    "        fval=df.loc['test statistic'][0][0:6]\n",
    "        pvalues.append(pval)\n",
    "        fvalues.append(fval)\n",
    "        cats.append(j)\n",
    "        newdf=pd.DataFrame({'Variable':cats,\n",
    "                            'pval_%s_UnweightedUniFrac' % i:pvalues,\n",
    "                           'FStatistic_%s_UnweightedUniFrac' % i :fvalues})\n",
    "        Dictionary[i]=newdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Categorical_df2 = Dictionary['MG'].merge(Dictionary['16S'],left_on='Variable',right_on='Variable')\n",
    "Categorical_df2.index=Categorical_df2['Variable']\n",
    "Categorical_df2=Categorical_df2.drop(columns='Variable')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Save a metadata file that is all quantitative\n",
    "df=pd.read_csv('../../IBD200/Combined_metadata_inallomics_hist.txt',sep='\\t')\n",
    "df.index=df['#SampleID']\n",
    "df=df[df['Diagnosis']=='UC']\n",
    "df=df[Quantitative]\n",
    "for i in Quantitative:\n",
    "    df=df[pd.to_numeric(df[i], errors='coerce').notnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#For bray curtis metric:\n",
    "dtypes=['MP','MB','16S','MG']\n",
    "for i in dtypes:\n",
    "    for j in Quantitative:\n",
    "        !compare_categories.py --method adonis -i ./Qiime/Cohort2_quant_$i/bray_curtis_dm.txt -m ../../IBD200/Combined_metadata_inallomics_hist_UC_quantitative.txt -c $j -o ./Qiime/Adonis/Cohort2_quant_$i/$j -n 999"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#For unweighted_unifrac metric:\n",
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    for j in Quantitative:\n",
    "        !compare_categories.py --method adonis -i ./Qiime/Cohort2_quant_$i/unweighted_unifrac_dm.txt -m ../../IBD200/Combined_metadata_inallomics_hist_UC_quantitative.txt -c $j -o ./Qiime/Adonis/Cohort2_quant_unweightedunifrac_$i/$j -n 999"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create a dictionary for all of the f and p-values\n",
    "Dictionary2={}\n",
    "dtypes=['MP','MB','16S','MG']\n",
    "for i in dtypes:\n",
    "    pvalues=[]\n",
    "    fvalues=[]\n",
    "    r2values=[]\n",
    "    cats=[]\n",
    "    for j in Quantitative:\n",
    "        df=pd.read_csv('./Qiime/Adonis/Cohort2_quant_%s/%s/adonis_results.txt' % (i,j),sep='\\t')\n",
    "        a=list(df.loc[5][0].split(' '))\n",
    "        alist = ' '.join(a).split()\n",
    "        pval=alist[6]\n",
    "        fval=alist[4]\n",
    "        r2=alist[5]\n",
    "        pvalues.append(pval)\n",
    "        fvalues.append(fval)\n",
    "        r2values.append(r2)\n",
    "        cats.append(j)\n",
    "        newdf=pd.DataFrame({'Variable':cats,\n",
    "                            'pval_%s_BrayCurtis' % i:pvalues,\n",
    "                            'R2_%s_BrayCurtis' % i:r2values,\n",
    "                           'FStatistic_%s_BrayCurtis' % i :fvalues})\n",
    "        Dictionary2[i]=newdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Quantitative_df = Dictionary2['MP'].merge(Dictionary2['16S'],left_on='Variable',right_on='Variable')\n",
    "Quantitative_df=Quantitative_df.merge(Dictionary2['MB'],left_on='Variable',right_on='Variable')\n",
    "Quantitative_df=Quantitative_df.merge(Dictionary2['MG'],left_on='Variable',right_on='Variable')\n",
    "Quantitative_df.index = Quantitative_df['Variable']\n",
    "Quantitative_df=Quantitative_df.drop(columns='Variable')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create a dictionary for all of the f and p-values\n",
    "Dictionary2={}\n",
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    pvalues=[]\n",
    "    fvalues=[]\n",
    "    r2values=[]\n",
    "    cats=[]\n",
    "    for j in Quantitative:\n",
    "        df=pd.read_csv('./Qiime/Adonis/Cohort2_quant_unweightedunifrac_%s/%s/adonis_results.txt' % (i,j),sep='\\t')\n",
    "        a=list(df.loc[5][0].split(' '))\n",
    "        alist = ' '.join(a).split()\n",
    "        pval=alist[6]\n",
    "        fval=alist[4]\n",
    "        r2=alist[5]\n",
    "        pvalues.append(pval)\n",
    "        fvalues.append(fval)\n",
    "        r2values.append(r2)\n",
    "        cats.append(j)\n",
    "        newdf=pd.DataFrame({'Variable':cats,\n",
    "                            'pval_%s_UnweightedUniFrac' % i:pvalues,\n",
    "                            'R2_%s_UnweightedUniFrac' % i:r2values,\n",
    "                           'FStatistic_%s_UnweightedUniFrac' % i :fvalues})\n",
    "        Dictionary2[i]=newdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Combine the tables together\n",
    "Quantitative_df2 = Dictionary2['MG'].merge(Dictionary2['16S'],left_on='Variable',right_on='Variable')\n",
    "Quantitative_df2.index = Quantitative_df2['Variable']\n",
    "Quantitative_df2=Quantitative_df2.drop(columns='Variable')\n",
    "Quantitative_df3=Quantitative_df.merge(Quantitative_df2,left_index=True,right_index=True)\n",
    "Categorical_df3=Categorical_df.merge(Categorical_df2,left_index=True,right_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Relabel them to be consistent between analyses\n",
    "dtypes=['MP','MB','16S','MG']\n",
    "for i in dtypes:\n",
    "    Quantitative_df3['%s'%i] = Quantitative_df3['pval_%s_BrayCurtis'%i]+', '+Quantitative_df3['FStatistic_%s_BrayCurtis'%i] +', '+Quantitative_df3['R2_%s_BrayCurtis'%i]\n",
    "\n",
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    Quantitative_df3['%s Unweighted UniFrac'%i] = Quantitative_df3['pval_%s_UnweightedUniFrac'%i]+', '+Quantitative_df3['FStatistic_%s_UnweightedUniFrac'%i] +', '+Quantitative_df3['R2_%s_UnweightedUniFrac'%i]\n",
    "\n",
    "\n",
    "dtypes=['MP','MB','16S','MG']\n",
    "for i in dtypes:\n",
    "    Categorical_df3['%s'%i] = Categorical_df3['pval_%s_BrayCurtis'%i]+', '+Categorical_df3['FStatistic_%s_BrayCurtis'%i]\n",
    "\n",
    "\n",
    "dtypes=['16S','MG']\n",
    "for i in dtypes:\n",
    "    Categorical_df3['%s Unweighted UniFrac'%i] = Categorical_df3['pval_%s_UnweightedUniFrac'%i]+', '+Categorical_df3['FStatistic_%s_UnweightedUniFrac'%i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Concatenate and save the statistics\n",
    "totaldf=pd.concat([Quantitative_df3,Categorical_df3])\n",
    "totaldf.to_csv('../UC_Cohort2_AdonisPermanova_Stats.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Core metrics - Qiime2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved FeatureTable[Frequency] to: core-metrics-results_MGgOTU/rarefied_table.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] % Properties('phylogenetic') to: core-metrics-results_MGgOTU/faith_pd_vector.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] to: core-metrics-results_MGgOTU/observed_otus_vector.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] to: core-metrics-results_MGgOTU/shannon_vector.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] to: core-metrics-results_MGgOTU/evenness_vector.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix % Properties('phylogenetic') to: core-metrics-results_MGgOTU/unweighted_unifrac_distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix % Properties('phylogenetic') to: core-metrics-results_MGgOTU/weighted_unifrac_distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix to: core-metrics-results_MGgOTU/jaccard_distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix to: core-metrics-results_MGgOTU/bray_curtis_distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults to: core-metrics-results_MGgOTU/unweighted_unifrac_pcoa_results.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults to: core-metrics-results_MGgOTU/weighted_unifrac_pcoa_results.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults to: core-metrics-results_MGgOTU/jaccard_pcoa_results.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults to: core-metrics-results_MGgOTU/bray_curtis_pcoa_results.qza\u001b[0m\n",
      "\u001b[32mSaved Visualization to: core-metrics-results_MGgOTU/unweighted_unifrac_emperor.qzv\u001b[0m\n",
      "\u001b[32mSaved Visualization to: core-metrics-results_MGgOTU/weighted_unifrac_emperor.qzv\u001b[0m\n",
      "\u001b[32mSaved Visualization to: core-metrics-results_MGgOTU/jaccard_emperor.qzv\u001b[0m\n",
      "\u001b[32mSaved Visualization to: core-metrics-results_MGgOTU/bray_curtis_emperor.qzv\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "#gOTUs\n",
    "!qiime diversity core-metrics-phylogenetic \\\n",
    "  --i-phylogeny ../../IBD200/Metagenomics/tree.qza \\\n",
    "  --i-table ../Genomics/Shotgun/gOTU_table_UC40.qza \\\n",
    "  --p-sampling-depth 16635 \\\n",
    "  --m-metadata-file ../Genomics/16S/UC_Severity_1_MF_Idswap6.21.18.txt \\\n",
    "  --output-dir core-metrics-results_MGgOTU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ./Metagenome_gOTU_UC.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-table summarize \\\n",
    "  --i-table ../Genomics/Shotgun/gOTU_table_UC40.qza \\\n",
    "  --o-visualization ./Metagenome_gOTU_UC.qzv \\\n",
    "  --m-sample-metadata-file ./UC_Severity_gOTU_metadata_2020.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#MG\n",
    "!biom convert -i ./gOTU_table_UC40_2.txt \\\n",
    "-o ./gOTU_table_UC40_2.biom \\\n",
    "-m ./UC_Severity_gOTUreg_metadata_2020_2.txt \\\n",
    "--table-type=\"OTU table\" --to-hdf5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mImported ./gOTU_table_UC40_2.biom as BIOMV210DirFmt to ./gOTU_table_UC40_2.qza\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "#Metagenomics using UniFrac. Based on centrifuge counts.\n",
    "!qiime tools import \\\n",
    "  --input-path ./gOTU_table_UC40_2.biom \\\n",
    "  --type 'FeatureTable[Frequency]' \\\n",
    "  --output-path ./gOTU_table_UC40_2.qza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved FeatureTable[Frequency] to: core-metrics-results_pDB_noHuman/rarefied_table.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] to: core-metrics-results_pDB_noHuman/observed_otus_vector.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] to: core-metrics-results_pDB_noHuman/shannon_vector.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] to: core-metrics-results_pDB_noHuman/evenness_vector.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix to: core-metrics-results_pDB_noHuman/jaccard_distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix to: core-metrics-results_pDB_noHuman/bray_curtis_distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults to: core-metrics-results_pDB_noHuman/jaccard_pcoa_results.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults to: core-metrics-results_pDB_noHuman/bray_curtis_pcoa_results.qza\u001b[0m\n",
      "\u001b[32mSaved Visualization to: core-metrics-results_pDB_noHuman/jaccard_emperor.qzv\u001b[0m\n",
      "\u001b[32mSaved Visualization to: core-metrics-results_pDB_noHuman/bray_curtis_emperor.qzv\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "#2Searh pDB using the no-human proteins\n",
    "!qiime diversity core-metrics \\\n",
    "  --i-table pDB_Common_nohuman_biom.qza \\\n",
    "    --p-sampling-depth 539826 \\\n",
    "--m-metadata-file ../UC_MP_Emperor_Map.txt \\\n",
    "--output-dir core-metrics-results_pDB_noHuman"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved FeatureTable[Frequency] to: core-metrics-results_2search/rarefied_table.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] to: core-metrics-results_2search/observed_otus_vector.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] to: core-metrics-results_2search/shannon_vector.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] to: core-metrics-results_2search/evenness_vector.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix to: core-metrics-results_2search/jaccard_distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix to: core-metrics-results_2search/bray_curtis_distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults to: core-metrics-results_2search/jaccard_pcoa_results.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults to: core-metrics-results_2search/bray_curtis_pcoa_results.qza\u001b[0m\n",
      "\u001b[32mSaved Visualization to: core-metrics-results_2search/jaccard_emperor.qzv\u001b[0m\n",
      "\u001b[32mSaved Visualization to: core-metrics-results_2search/bray_curtis_emperor.qzv\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime diversity core-metrics \\\n",
    "  --i-table 2search_common_biom.qza \\\n",
    "    --p-sampling-depth 694193 \\\n",
    "--m-metadata-file ../UC_MP_Emperor_Map.txt \\\n",
    "--output-dir core-metrics-results_2search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For metabolomics data with 'phylogenetic' assessment of metabolites through Qemistree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mImported ../Metabolomics/StandardizedMzMIne2020Rerun/FBMN-download_qza_table_data-mzminev20/clusterinfo_summary/library_identifications.tsv as TSVMoleculesFormat to ../Metabolomics/StandardizedMzMIne2020Rerun/FBMN-download_qza_table_data-mzminev20/library_identifications.qza\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime tools import \\\n",
    "  --input-path ../Metabolomics/StandardizedMzMIne2020Rerun/FBMN-download_qza_table_data-mzminev20/clusterinfo_summary/library_identifications.tsv \\\n",
    "  --output-path ../Metabolomics/StandardizedMzMIne2020Rerun/FBMN-download_qza_table_data-mzminev20/library_identifications.qza \\\n",
    "  --type FeatureData[Molecules]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Phylogeny[Rooted] to: ../Metabolomics/StandardizedMzMIne2020Rerun/qemistree.qza\u001b[0m\n",
      "\u001b[32mSaved FeatureTable[Frequency] to: ../Metabolomics/StandardizedMzMIne2020Rerun/feature-table-hashed.qza\u001b[0m\n",
      "\u001b[32mSaved FeatureData[Molecules] to: ../Metabolomics/StandardizedMzMIne2020Rerun/feature-data.qza\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime qemistree make-hierarchy \\\n",
    "  --i-csi-results ../Metabolomics/StandardizedMzMIne2020Rerun/GNPS_QEMISTREE/output_folder/fingerprints.qza \\\n",
    "  --i-feature-tables ../Metabolomics/StandardizedMzMIne2020Rerun/FBMN-download_qza_table_data-mzminev20/qiime2_output/qiime2_table.qza \\\n",
    "--i-library-matches ../Metabolomics/StandardizedMzMIne2020Rerun/FBMN-download_qza_table_data-mzminev20/library_identifications.qza \\\n",
    "--o-tree ../Metabolomics/StandardizedMzMIne2020Rerun/qemistree.qza \\\n",
    "  --o-feature-table ../Metabolomics/StandardizedMzMIne2020Rerun/feature-table-hashed.qza \\\n",
    "  --o-feature-data ../Metabolomics/StandardizedMzMIne2020Rerun/feature-data.qza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ../Metabolomics/StandardizedMzMIne2020Rerun/feature-table-hashed.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-table summarize \\\n",
    "  --i-table ../Metabolomics/StandardizedMzMIne2020Rerun/feature-table-hashed.qza \\\n",
    "  --o-visualization ../Metabolomics/StandardizedMzMIne2020Rerun/feature-table-hashed.qzv \\\n",
    "  --m-sample-metadata-file ../UC_MB_Emperor_Map.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved FeatureTable[Frequency] to: core-metrics-results-MB-phylogenetic/rarefied_table.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] % Properties('phylogenetic') to: core-metrics-results-MB-phylogenetic/faith_pd_vector.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] to: core-metrics-results-MB-phylogenetic/observed_otus_vector.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] to: core-metrics-results-MB-phylogenetic/shannon_vector.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[AlphaDiversity] to: core-metrics-results-MB-phylogenetic/evenness_vector.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix % Properties('phylogenetic') to: core-metrics-results-MB-phylogenetic/unweighted_unifrac_distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix % Properties('phylogenetic') to: core-metrics-results-MB-phylogenetic/weighted_unifrac_distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix to: core-metrics-results-MB-phylogenetic/jaccard_distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix to: core-metrics-results-MB-phylogenetic/bray_curtis_distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults to: core-metrics-results-MB-phylogenetic/unweighted_unifrac_pcoa_results.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults to: core-metrics-results-MB-phylogenetic/weighted_unifrac_pcoa_results.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults to: core-metrics-results-MB-phylogenetic/jaccard_pcoa_results.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults to: core-metrics-results-MB-phylogenetic/bray_curtis_pcoa_results.qza\u001b[0m\n",
      "\u001b[32mSaved Visualization to: core-metrics-results-MB-phylogenetic/unweighted_unifrac_emperor.qzv\u001b[0m\n",
      "\u001b[32mSaved Visualization to: core-metrics-results-MB-phylogenetic/weighted_unifrac_emperor.qzv\u001b[0m\n",
      "\u001b[32mSaved Visualization to: core-metrics-results-MB-phylogenetic/jaccard_emperor.qzv\u001b[0m\n",
      "\u001b[32mSaved Visualization to: core-metrics-results-MB-phylogenetic/bray_curtis_emperor.qzv\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime diversity core-metrics-phylogenetic \\\n",
    "  --i-phylogeny ../Metabolomics/StandardizedMzMIne2020Rerun/qemistree.qza \\\n",
    "  --i-table ../Metabolomics/StandardizedMzMIne2020Rerun/feature-table-hashed.qza \\\n",
    "  --p-sampling-depth 50000000 \\\n",
    "  --m-metadata-file ../UC_MB_Emperor_Map.txt \\\n",
    "  --output-dir core-metrics-results-MB-phylogenetic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved FeatureData[Molecules] to: ../Metabolomics/StandardizedMzMIne2020Rerun/classified-feature-data.qza\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime qemistree get-classyfire-taxonomy \\\n",
    "  --i-feature-data ../Metabolomics/StandardizedMzMIne2020Rerun/feature-data.qza \\\n",
    "  --o-classified-feature-data ../Metabolomics/StandardizedMzMIne2020Rerun/classified-feature-data.qza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Adonis statistics for main fig. Qiime2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##Make a subset of the metadata to contain only numerics\n",
    "df = pd.read_csv('../UC_MP_Emperor_Map.txt',sep='\\t', index_col='id')\n",
    "df['partial_Mayo'] = df['partial_Mayo'].astype('float')\n",
    "numeric_cols = ['pielou_e','partial_Mayo']\n",
    "df = df[numeric_cols]\n",
    "df.to_csv('../UC_MP_Emperor_Map_numerics.txt',sep='\\t')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ./core-metrics-results_2search/bray_curtis_adonis_pmayo.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime diversity adonis \\\n",
    "  --i-distance-matrix ./core-metrics-results_2search/bray_curtis_distance_matrix.qza \\\n",
    "  --m-metadata-file ../UC_MP_Emperor_Map_numerics.txt \\\n",
    "  --o-visualization ./core-metrics-results_2search/bray_curtis_adonis_pmayo.qzv \\\n",
    "  --p-formula partial_Mayo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ../core-metrics-results_Serum/bray_curtis_adonis_pmayo.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime diversity adonis \\\n",
    "  --i-distance-matrix ../core-metrics-results_Serum/bray_curtis_distance_matrix.qza \\\n",
    "  --m-metadata-file ../UC_MP_Emperor_Map_numerics.txt \\\n",
    "  --o-visualization ../core-metrics-results_Serum/bray_curtis_adonis_pmayo.qzv \\\n",
    "  --p-formula partial_Mayo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "##Make a subset of the metadata to contain only numerics\n",
    "df = pd.read_csv('../Genomics/16S/UC_Severity_1_MF_Idswap6.21.18.txt',sep='\\t')\n",
    "df.rename(columns={'#SampleID':'id'},inplace=True)\n",
    "df.index = df['id']\n",
    "df = df[df['partial_mayo']!='not applicable']\n",
    "\n",
    "df['partial_mayo'] = df['partial_mayo'].astype('float')\n",
    "numeric_cols = ['mayo_endoscopic_score','partial_mayo']\n",
    "df = df[numeric_cols]\n",
    "df.to_csv('../UC_MP_Emperor_Map_16Snumerics.txt',sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ../Genomics/16S/core-metrics-results_idswap2_newmetadata_allsamples/unweighted_unifrac_adonis_pmayo.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime diversity adonis \\\n",
    "  --i-distance-matrix ../Genomics/16S/core-metrics-results_idswap2_newmetadata_allsamples/unweighted_unifrac_distance_matrix.qza \\\n",
    "  --m-metadata-file ../UC_MP_Emperor_Map_16Snumerics.txt \\\n",
    "  --o-visualization ../Genomics/16S/core-metrics-results_idswap2_newmetadata_allsamples/unweighted_unifrac_adonis_pmayo.qzv \\\n",
    "  --p-formula partial_mayo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ../Genomics/16S/core-metrics-results_idswap2_newmetadata_allsamples/weighted_unifrac_adonis_pmayo.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime diversity adonis \\\n",
    "  --i-distance-matrix ../Genomics/16S/core-metrics-results_idswap2_newmetadata_allsamples/weighted_unifrac_distance_matrix.qza \\\n",
    "  --m-metadata-file ../UC_MP_Emperor_Map_16Snumerics.txt \\\n",
    "  --o-visualization ../Genomics/16S/core-metrics-results_idswap2_newmetadata_allsamples/weighted_unifrac_adonis_pmayo.qzv \\\n",
    "  --p-formula partial_mayo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "##Make a subset of the metadata to contain only numerics\n",
    "df = pd.read_csv('../UC_MB_Emperor_Map.txt',sep='\\t', index_col='id')\n",
    "df['partial_Mayo'] = df['partial_Mayo'].astype('float')\n",
    "numeric_cols = ['pielou_e','partial_Mayo']\n",
    "df = df[numeric_cols]\n",
    "df.to_csv('../UC_MB_Emperor_Map_numerics.txt',sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ./core-metrics-results-MB-phylogenetic/weighted_unifrac_adonis_pmayo.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime diversity adonis \\\n",
    "  --i-distance-matrix ./core-metrics-results-MB-phylogenetic/weighted_unifrac_distance_matrix.qza \\\n",
    "  --m-metadata-file ../UC_MB_Emperor_Map_numerics.txt \\\n",
    "  --o-visualization ./core-metrics-results-MB-phylogenetic/weighted_unifrac_adonis_pmayo.qzv \\\n",
    "  --p-formula partial_Mayo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ./core-metrics-results-MB-phylogenetic/unweighted_unifrac_adonis_pmayo.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime diversity adonis \\\n",
    "  --i-distance-matrix ./core-metrics-results-MB-phylogenetic/unweighted_unifrac_distance_matrix.qza \\\n",
    "  --m-metadata-file ../UC_MB_Emperor_Map_numerics.txt \\\n",
    "  --o-visualization ./core-metrics-results-MB-phylogenetic/unweighted_unifrac_adonis_pmayo.qzv \\\n",
    "  --p-formula partial_Mayo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ./core-metrics-results-MB-phylogenetic/bray_curtis_adonis_pmayo.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime diversity adonis \\\n",
    "  --i-distance-matrix ./core-metrics-results-MB-phylogenetic/bray_curtis_distance_matrix.qza \\\n",
    "  --m-metadata-file ../UC_MB_Emperor_Map_numerics.txt \\\n",
    "  --o-visualization ./core-metrics-results-MB-phylogenetic/bray_curtis_adonis_pmayo.qzv \\\n",
    "  --p-formula partial_Mayo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ./core-metrics-results_MGgOTU/weighted_unifrac_adonis_pmayo.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime diversity adonis \\\n",
    "  --i-distance-matrix ./core-metrics-results_MGgOTU/weighted_unifrac_distance_matrix.qza \\\n",
    "  --m-metadata-file ../UC_MP_Emperor_Map_16Snumerics.txt \\\n",
    "  --o-visualization ./core-metrics-results_MGgOTU/weighted_unifrac_adonis_pmayo.qzv \\\n",
    "  --p-formula partial_mayo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ./core-metrics-results_MGgOTU/unweighted_unifrac_adonis_pmayo.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime diversity adonis \\\n",
    "  --i-distance-matrix ./core-metrics-results_MGgOTU/unweighted_unifrac_distance_matrix.qza \\\n",
    "  --m-metadata-file ../UC_MP_Emperor_Map_16Snumerics.txt \\\n",
    "  --o-visualization ./core-metrics-results_MGgOTU/unweighted_unifrac_adonis_pmayo.qzv \\\n",
    "  --p-formula partial_mayo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: ./core-metrics-results_MGgOTU/bray_curtis_adonis_pmayo.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime diversity adonis \\\n",
    "  --i-distance-matrix ./core-metrics-results_MGgOTU/bray_curtis_distance_matrix.qza \\\n",
    "  --m-metadata-file ../UC_MP_Emperor_Map_16Snumerics.txt \\\n",
    "  --o-visualization ./core-metrics-results_MGgOTU/bray_curtis_adonis_pmayo.qzv \\\n",
    "  --p-formula partial_mayo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Create a combined table for random forest analyses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>H8</th>\n",
       "      <th>H11</th>\n",
       "      <th>H12</th>\n",
       "      <th>L7</th>\n",
       "      <th>H14</th>\n",
       "      <th>H16</th>\n",
       "      <th>L22</th>\n",
       "      <th>H7</th>\n",
       "      <th>L9</th>\n",
       "      <th>...</th>\n",
       "      <th>H5</th>\n",
       "      <th>H1</th>\n",
       "      <th>L12</th>\n",
       "      <th>H18</th>\n",
       "      <th>L20</th>\n",
       "      <th>L14</th>\n",
       "      <th>L11</th>\n",
       "      <th>H17</th>\n",
       "      <th>H3</th>\n",
       "      <th>H19</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>46909.498401</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>179.819380</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>531.088310</td>\n",
       "      <td>1271.684763</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>599.695089</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6128.969926</td>\n",
       "      <td>6013.327664</td>\n",
       "      <td>320.286032</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>21456.423961</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3209.658048</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1443.415289</td>\n",
       "      <td>4484.421327</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>707.640558</td>\n",
       "      <td>26628.566164</td>\n",
       "      <td>614.571046</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>10999.273694</td>\n",
       "      <td>628.479395</td>\n",
       "      <td>3588.391143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>17150.358235</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>508.676267</td>\n",
       "      <td>552.002760</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3055.094548</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>360.907039</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>18506.034913</td>\n",
       "      <td>12964.227060</td>\n",
       "      <td>50320.039626</td>\n",
       "      <td>24622.798409</td>\n",
       "      <td>8791.566438</td>\n",
       "      <td>48346.760347</td>\n",
       "      <td>10000.295012</td>\n",
       "      <td>46644.925805</td>\n",
       "      <td>25199.698602</td>\n",
       "      <td>...</td>\n",
       "      <td>14888.980413</td>\n",
       "      <td>13836.875906</td>\n",
       "      <td>8250.907422</td>\n",
       "      <td>18713.108934</td>\n",
       "      <td>26692.233282</td>\n",
       "      <td>33902.363004</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3862.980372</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>29333.622313</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6</td>\n",
       "      <td>14764.662583</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>45.464297</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>760.312761</td>\n",
       "      <td>400.923157</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>250.920502</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>936.729145</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>7</td>\n",
       "      <td>11933.452966</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>934.435196</td>\n",
       "      <td>308.408944</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>224.047768</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>10731.423898</td>\n",
       "      <td>4938.744773</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>8</td>\n",
       "      <td>23509.327949</td>\n",
       "      <td>238.960875</td>\n",
       "      <td>2872.267196</td>\n",
       "      <td>21287.137541</td>\n",
       "      <td>751.349323</td>\n",
       "      <td>1674.558384</td>\n",
       "      <td>2593.909401</td>\n",
       "      <td>23874.233192</td>\n",
       "      <td>1004.766215</td>\n",
       "      <td>...</td>\n",
       "      <td>1005.563649</td>\n",
       "      <td>6042.383248</td>\n",
       "      <td>12235.184253</td>\n",
       "      <td>14661.388796</td>\n",
       "      <td>10894.567164</td>\n",
       "      <td>15239.497263</td>\n",
       "      <td>13510.629037</td>\n",
       "      <td>114.861375</td>\n",
       "      <td>15047.233221</td>\n",
       "      <td>1153.853265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>9</td>\n",
       "      <td>14812.638646</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5070.186710</td>\n",
       "      <td>10615.555566</td>\n",
       "      <td>245.209067</td>\n",
       "      <td>13709.412682</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>429.369600</td>\n",
       "      <td>...</td>\n",
       "      <td>6358.672761</td>\n",
       "      <td>36255.702427</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>9214.322778</td>\n",
       "      <td>96.097618</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>222.506894</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>39531.430739</td>\n",
       "      <td>5617.084413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>10</td>\n",
       "      <td>14547.208020</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4735.698207</td>\n",
       "      <td>1727.315911</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6699.919421</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>536.245370</td>\n",
       "      <td>...</td>\n",
       "      <td>670.011213</td>\n",
       "      <td>19070.538821</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8106.818511</td>\n",
       "      <td>36.107493</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>43.697930</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>16210.632852</td>\n",
       "      <td>892.785097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>11</td>\n",
       "      <td>11838.586875</td>\n",
       "      <td>28659.377981</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>241.303695</td>\n",
       "      <td>781.786822</td>\n",
       "      <td>2326.596671</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1510.519681</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>664.197731</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1689.391530</td>\n",
       "      <td>15051.337939</td>\n",
       "      <td>1171.765579</td>\n",
       "      <td>409.760146</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1447.331765</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>12</td>\n",
       "      <td>6964.462488</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>864.119028</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>14</td>\n",
       "      <td>9837.428525</td>\n",
       "      <td>4557.090682</td>\n",
       "      <td>14604.204938</td>\n",
       "      <td>3318.455564</td>\n",
       "      <td>33535.145233</td>\n",
       "      <td>10837.205585</td>\n",
       "      <td>5542.525889</td>\n",
       "      <td>9418.661954</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>13701.558527</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>48191.792871</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3494.344375</td>\n",
       "      <td>13011.533439</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>15</td>\n",
       "      <td>8882.742635</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>50842.322744</td>\n",
       "      <td>31779.962761</td>\n",
       "      <td>1161.677011</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>155.194793</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>32.258684</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>149.732179</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>16</td>\n",
       "      <td>7396.782709</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>10428.043216</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1371.123123</td>\n",
       "      <td>4077.793492</td>\n",
       "      <td>5449.741722</td>\n",
       "      <td>2920.145386</td>\n",
       "      <td>6561.707220</td>\n",
       "      <td>18633.917550</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7183.644226</td>\n",
       "      <td>104.164601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>17</td>\n",
       "      <td>7426.684970</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>11287.572447</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>18</td>\n",
       "      <td>5774.084522</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>82.045990</td>\n",
       "      <td>639.547601</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>179.648330</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>472.031371</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2299.183025</td>\n",
       "      <td>648.401932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>19</td>\n",
       "      <td>4338.213574</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>17953.527985</td>\n",
       "      <td>10325.075259</td>\n",
       "      <td>236.211043</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>25556.447288</td>\n",
       "      <td>5955.463730</td>\n",
       "      <td>...</td>\n",
       "      <td>2417.216583</td>\n",
       "      <td>3743.767235</td>\n",
       "      <td>1275.264190</td>\n",
       "      <td>6356.494265</td>\n",
       "      <td>9030.695682</td>\n",
       "      <td>19542.407141</td>\n",
       "      <td>56827.646100</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>14356.260182</td>\n",
       "      <td>16078.163640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>20</td>\n",
       "      <td>5700.525882</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>319.162251</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1582.225983</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7250.397641</td>\n",
       "      <td>1783.955230</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>21</td>\n",
       "      <td>4625.940096</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>9437.776518</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6612.928091</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3491.363756</td>\n",
       "      <td>1509.448928</td>\n",
       "      <td>5212.167311</td>\n",
       "      <td>26116.094061</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6139.453970</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>22</td>\n",
       "      <td>17707.595180</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>10454.424130</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3550.583501</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4527.415381</td>\n",
       "      <td>22175.294333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2149.149314</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8532.012408</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>23</td>\n",
       "      <td>4053.997414</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>186.452524</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>24</td>\n",
       "      <td>5428.524546</td>\n",
       "      <td>2342.050692</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>554.209273</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>133.566268</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>180.609743</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>526.865384</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>715.622238</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>25</td>\n",
       "      <td>5674.088565</td>\n",
       "      <td>950.576696</td>\n",
       "      <td>1473.303188</td>\n",
       "      <td>8368.254666</td>\n",
       "      <td>9904.222633</td>\n",
       "      <td>1887.644011</td>\n",
       "      <td>8073.800624</td>\n",
       "      <td>12917.554890</td>\n",
       "      <td>1118.698123</td>\n",
       "      <td>...</td>\n",
       "      <td>733.217610</td>\n",
       "      <td>2345.865348</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5750.849733</td>\n",
       "      <td>18004.574636</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>16993.722790</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4986.843158</td>\n",
       "      <td>1655.038404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>26</td>\n",
       "      <td>5207.443335</td>\n",
       "      <td>231.233267</td>\n",
       "      <td>1084.774169</td>\n",
       "      <td>5761.957965</td>\n",
       "      <td>5560.418503</td>\n",
       "      <td>1472.439711</td>\n",
       "      <td>8621.794535</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>798.532404</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1329.395421</td>\n",
       "      <td>5560.511497</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>13467.211955</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4355.244781</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1077.866393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>27</td>\n",
       "      <td>4603.139320</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>68.956728</td>\n",
       "      <td>157.494274</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1678.192409</td>\n",
       "      <td>137.879061</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>415.689208</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>453.503868</td>\n",
       "      <td>3920.000502</td>\n",
       "      <td>3461.956970</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>28</td>\n",
       "      <td>4467.682423</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5324.740055</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>30.860002</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>29</td>\n",
       "      <td>5437.998922</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>60.829782</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>781.012442</td>\n",
       "      <td>723.696754</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>30</td>\n",
       "      <td>4830.906239</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>144.744521</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>31</td>\n",
       "      <td>3333.260334</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>35.354940</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>100.786812</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>79.405130</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>254.207421</td>\n",
       "      <td>693.359422</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>32</td>\n",
       "      <td>3456.048357</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>718.150247</td>\n",
       "      <td>2283.941330</td>\n",
       "      <td>25070.791642</td>\n",
       "      <td>6482.814156</td>\n",
       "      <td>4223.909911</td>\n",
       "      <td>3684.580836</td>\n",
       "      <td>1303.911257</td>\n",
       "      <td>...</td>\n",
       "      <td>1533.997307</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4208.663376</td>\n",
       "      <td>21587.580850</td>\n",
       "      <td>5732.773120</td>\n",
       "      <td>44640.086603</td>\n",
       "      <td>13240.448686</td>\n",
       "      <td>4932.891827</td>\n",
       "      <td>1383.464912</td>\n",
       "      <td>1060.578277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4237</th>\n",
       "      <td>6037</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>308.552153</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>594.152665</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4238</th>\n",
       "      <td>6038</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>174.932400</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>655.375765</td>\n",
       "      <td>139.549755</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4239</th>\n",
       "      <td>6039</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>167.185584</td>\n",
       "      <td>46.576215</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>273.271648</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4240</th>\n",
       "      <td>6042</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>207.914956</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>220.734930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4241</th>\n",
       "      <td>6044</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>231.195394</td>\n",
       "      <td>445.540665</td>\n",
       "      <td>361.528504</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4242</th>\n",
       "      <td>6050</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>65.151013</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>134.191947</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>89.964140</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4243</th>\n",
       "      <td>6051</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>97.878679</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>398.584881</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4244</th>\n",
       "      <td>6053</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>57.471875</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>420.575495</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>298.448399</td>\n",
       "      <td>37.407481</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4245</th>\n",
       "      <td>6054</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>513.695357</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>243.444340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4246</th>\n",
       "      <td>6056</td>\n",
       "      <td>73.665229</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>65.131468</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>140.860448</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>57.859199</td>\n",
       "      <td>421.580719</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>151.480262</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4247</th>\n",
       "      <td>6057</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>132.521987</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>116.107001</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4248</th>\n",
       "      <td>6059</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>71.254791</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>67.508383</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>102.669023</td>\n",
       "      <td>1222.503964</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>76.286250</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4249</th>\n",
       "      <td>6061</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>694.097357</td>\n",
       "      <td>744.590657</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4250</th>\n",
       "      <td>6062</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>37.019691</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>332.012666</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>210.531808</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4251</th>\n",
       "      <td>6063</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>53.972953</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>208.266417</td>\n",
       "      <td>358.340469</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>167.377087</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4252</th>\n",
       "      <td>6065</td>\n",
       "      <td>92.759783</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>56.636866</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>54.541088</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>291.463516</td>\n",
       "      <td>592.387001</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4253</th>\n",
       "      <td>6068</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>99.837808</td>\n",
       "      <td>208.920654</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4254</th>\n",
       "      <td>6072</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>57.043119</td>\n",
       "      <td>47.569636</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>213.991098</td>\n",
       "      <td>185.242873</td>\n",
       "      <td>27.423263</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4255</th>\n",
       "      <td>6073</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>158.079195</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>60.643489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4256</th>\n",
       "      <td>6074</td>\n",
       "      <td>277.914814</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>197.993595</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>194.966902</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>105.004610</td>\n",
       "      <td>245.442379</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>322.269170</td>\n",
       "      <td>410.583302</td>\n",
       "      <td>62.286004</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>74.845694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4257</th>\n",
       "      <td>6075</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>115.532055</td>\n",
       "      <td>123.378458</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4258</th>\n",
       "      <td>6076</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>93.788450</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>381.062182</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>135.329314</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>155.194780</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>204.208036</td>\n",
       "      <td>120.118669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4259</th>\n",
       "      <td>6077</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>208.312875</td>\n",
       "      <td>219.854160</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4260</th>\n",
       "      <td>6080</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>146.974539</td>\n",
       "      <td>121.112299</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4261</th>\n",
       "      <td>6081</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>171.216093</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>694.472508</td>\n",
       "      <td>71.777762</td>\n",
       "      <td>153.700818</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>364.945836</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>355.008292</td>\n",
       "      <td>345.002713</td>\n",
       "      <td>173.664210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4262</th>\n",
       "      <td>6084</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>58.807917</td>\n",
       "      <td>250.064831</td>\n",
       "      <td>...</td>\n",
       "      <td>227.617539</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>52.990109</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>587.478889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4263</th>\n",
       "      <td>6085</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>464.075305</td>\n",
       "      <td>140.491605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4264</th>\n",
       "      <td>6086</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>136.242267</td>\n",
       "      <td>200.581304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4265</th>\n",
       "      <td>6087</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>124.166995</td>\n",
       "      <td>108.672217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4266</th>\n",
       "      <td>6088</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>93.595621</td>\n",
       "      <td>242.466789</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4267 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0            H8          H11           H12            L7  \\\n",
       "0              1  46909.498401      0.00000      0.000000    179.819380   \n",
       "1              2  21456.423961      0.00000   3209.658048      0.000000   \n",
       "2              4  17150.358235      0.00000      0.000000      0.000000   \n",
       "3              5  18506.034913  12964.22706  50320.039626  24622.798409   \n",
       "4              6  14764.662583      0.00000      0.000000     45.464297   \n",
       "...          ...           ...          ...           ...           ...   \n",
       "4262        6084      0.000000      0.00000      0.000000      0.000000   \n",
       "4263        6085      0.000000      0.00000      0.000000      0.000000   \n",
       "4264        6086      0.000000      0.00000      0.000000      0.000000   \n",
       "4265        6087      0.000000      0.00000      0.000000      0.000000   \n",
       "4266        6088      0.000000      0.00000      0.000000      0.000000   \n",
       "\n",
       "              H14           H16           L22            H7            L9  \\\n",
       "0        0.000000      0.000000      0.000000    531.088310   1271.684763   \n",
       "1     1443.415289   4484.421327      0.000000      0.000000      0.000000   \n",
       "2        0.000000      0.000000      0.000000    508.676267    552.002760   \n",
       "3     8791.566438  48346.760347  10000.295012  46644.925805  25199.698602   \n",
       "4        0.000000      0.000000      0.000000    760.312761    400.923157   \n",
       "...           ...           ...           ...           ...           ...   \n",
       "4262     0.000000      0.000000      0.000000     58.807917    250.064831   \n",
       "4263     0.000000      0.000000      0.000000      0.000000      0.000000   \n",
       "4264     0.000000      0.000000      0.000000      0.000000      0.000000   \n",
       "4265     0.000000      0.000000      0.000000      0.000000      0.000000   \n",
       "4266     0.000000      0.000000      0.000000      0.000000      0.000000   \n",
       "\n",
       "      ...            H5            H1           L12           H18  \\\n",
       "0     ...      0.000000    599.695089      0.000000      0.000000   \n",
       "1     ...      0.000000    707.640558  26628.566164    614.571046   \n",
       "2     ...      0.000000   3055.094548      0.000000      0.000000   \n",
       "3     ...  14888.980413  13836.875906   8250.907422  18713.108934   \n",
       "4     ...      0.000000    250.920502      0.000000      0.000000   \n",
       "...   ...           ...           ...           ...           ...   \n",
       "4262  ...    227.617539      0.000000      0.000000      0.000000   \n",
       "4263  ...      0.000000      0.000000      0.000000      0.000000   \n",
       "4264  ...      0.000000      0.000000      0.000000      0.000000   \n",
       "4265  ...      0.000000      0.000000      0.000000      0.000000   \n",
       "4266  ...      0.000000      0.000000      0.000000      0.000000   \n",
       "\n",
       "               L20           L14         L11           H17          H3  \\\n",
       "0      6128.969926   6013.327664  320.286032      0.000000    0.000000   \n",
       "1         0.000000      0.000000    0.000000  10999.273694  628.479395   \n",
       "2       360.907039      0.000000    0.000000      0.000000    0.000000   \n",
       "3     26692.233282  33902.363004    0.000000   3862.980372    0.000000   \n",
       "4         0.000000    936.729145    0.000000      0.000000    0.000000   \n",
       "...            ...           ...         ...           ...         ...   \n",
       "4262      0.000000      0.000000    0.000000     52.990109    0.000000   \n",
       "4263      0.000000      0.000000    0.000000      0.000000  464.075305   \n",
       "4264      0.000000      0.000000    0.000000      0.000000  136.242267   \n",
       "4265      0.000000      0.000000    0.000000      0.000000  124.166995   \n",
       "4266      0.000000      0.000000    0.000000      0.000000   93.595621   \n",
       "\n",
       "               H19  \n",
       "0         0.000000  \n",
       "1      3588.391143  \n",
       "2         0.000000  \n",
       "3     29333.622313  \n",
       "4         0.000000  \n",
       "...            ...  \n",
       "4262    587.478889  \n",
       "4263    140.491605  \n",
       "4264    200.581304  \n",
       "4265    108.672217  \n",
       "4266    242.466789  \n",
       "\n",
       "[4267 rows x 41 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Mb = pd.read_csv('../Metabolomics/StandardizedMzMIne2020Rerun/UC40_MB_Table_normalized.csv')\n",
    "Mb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rhmills/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/ipykernel_launcher.py:16: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version\n",
      "of pandas will change to not sort by default.\n",
      "\n",
      "To accept the future behavior, pass 'sort=False'.\n",
      "\n",
      "To retain the current behavior and silence the warning, pass 'sort=True'.\n",
      "\n",
      "  app.launch_new_instance()\n"
     ]
    }
   ],
   "source": [
    "#Create combined table with all data\n",
    "Mb = pd.read_csv('../Metabolomics/StandardizedMzMIne2020Rerun/UC40_MB_Table_normalized.csv', index_col = '#OTU ID')\n",
    "Mp = pd.read_csv('../pDB_Proteomics/2Search/CSVs/NormalizedCommonReps.txt', sep = '\\t', index_col = 'datarest$ProteinID')\n",
    "Ser = pd.read_csv('../Serum/CSVs/NormalizedCommonReps_ids.txt', sep = '\\t', index_col = 'datarest$ProteinID')\n",
    "MG = pd.read_csv('./gOTU_table_UC40_2.txt', sep = '\\t', index_col = '#OTU ID')\n",
    "Amp = pd.read_csv('../Genomics/16S/reference-hit_idswap2_noblanks.txt', sep = '\\t', index_col = '#OTU ID')\n",
    "\n",
    "\n",
    "#Overlapping IDS are a problem, so append the metaproteome ids with _MP to signify metaproteome.\n",
    "Mp.index = Mp.index + '_MP'\n",
    "\n",
    "#Concatenate data\n",
    "Objs = [Mb, Mp, Ser, MG, Amp]\n",
    "alldf = pd.concat(Objs)\n",
    "\n",
    "alldf.index.rename('Features', inplace = True)\n",
    "\n",
    "#Save the new feature table\n",
    "alldf.to_csv('./allfeaturesconcat3.txt', sep = '\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Create biom file\n",
    "!biom convert -i ./allfeaturesconcat3.txt \\\n",
    "-o ../Allfeaturesconcat3.biom \\\n",
    "-m ../UC_MP_Emperor_Map_1.txt \\\n",
    "--table-type=\"OTU table\" --to-hdf5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mImported ../Allfeaturesconcat3.biom as BIOMV210DirFmt to ../Allfeaturesconcat3.qza\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime tools import \\\n",
    "  --input-path ../Allfeaturesconcat3.biom \\\n",
    "  --type 'FeatureTable[Frequency]' \\\n",
    "  --output-path ../Allfeaturesconcat3.qza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i> Performed random forest analyses on supercomputer using the Allfeaturesconcat.biom file </i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 16S analysis - Qiime2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initializes the notebook with inline display\n",
    "%matplotlib inline\n",
    "\n",
    "from os import mkdir\n",
    "import os\n",
    "import copy\n",
    "from os.path import abspath, join as pjoin, exists\n",
    "from shutil import copy2, move\n",
    "from time import strftime, strptime\n",
    "from numpy import nan, isnan, arange\n",
    "from pandas import read_csv, Series, DataFrame\n",
    "from IPython.display import Image\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import FileLinks, FileLink"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import all files as Qiime2 artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mImported ../Genomics/16S/reference-hit_idswap2.biom as BIOMV210DirFmt to biom_id2.qza\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime tools import \\\n",
    "  --input-path ../Genomics/16S/reference-hit_idswap2.biom \\\n",
    "  --type 'FeatureTable[Frequency]' \\\n",
    "  --output-path biom_id2.qza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime feature-table summarize \\\n",
    "  --i-table biom_id2.qza \\\n",
    "  --o-visualization biom_id2.qzv \\\n",
    "  --m-sample-metadata-file ../Genomics/16S/UC_Severity_1_MF_Idswap6.21.18.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## rep-seqs \n",
    "!qiime tools import \\\n",
    "  --input-path reference-hit.seqs.fa \\\n",
    "  --output-path sequences.qza \\\n",
    "  --type 'FeatureData[Sequence]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime alignment mafft \\\n",
    "  --i-sequences sequences.qza \\\n",
    "  --o-alignment aligned-rep-seqs.qza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime alignment mask \\\n",
    "  --i-alignment aligned-rep-seqs.qza \\\n",
    "  --o-masked-alignment masked-aligned-rep-seqs.qza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime phylogeny fasttree \\\n",
    "  --i-alignment masked-aligned-rep-seqs.qza \\\n",
    "  --o-tree unrooted-tree.qza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime phylogeny midpoint-root \\\n",
    "  --i-tree unrooted-tree.qza \\\n",
    "  --o-rooted-tree rooted-tree.qza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Core Diversity Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime diversity core-metrics-phylogenetic \\\n",
    "  --i-phylogeny rooted-tree.qza \\\n",
    "  --i-table biom_id2.qza \\\n",
    "  --p-sampling-depth 4166 \\\n",
    "  --m-metadata-file UC_Severity_1_MF_Idswap6.21.18.txt \\\n",
    "  --output-dir core-metrics-results_idswap2_newmetadata_allsamples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Alpha diversity analyses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime diversity alpha-group-significance \\\n",
    "  --i-alpha-diversity core-metrics-results_idswap2/faith_pd_vector.qza \\\n",
    "  --m-metadata-file UC_Severity_1_MF_Idswap6.21.18.txt \\\n",
    "  --o-visualization core-metrics-results_idswap2_newmetadata/faith-pd-group-significance.qzv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime diversity alpha-group-significance \\\n",
    "  --i-alpha-diversity core-metrics-results_idswap2/evenness_vector.qza \\\n",
    "  --m-metadata-file UC_Severity_1_MF_Idswap6.21.18.txt \\\n",
    "  --o-visualization core-metrics-results_idswap2_newmetadata/evenness-group-significance.qzv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Taxanomic analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime feature-classifier classify-sklearn \\\n",
    "  --i-classifier gg-13-8-99-515-806-nb-classifier.qza \\\n",
    "  --i-reads sequences.qza \\\n",
    "  --o-classification taxonomy.qza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime metadata tabulate \\\n",
    "  --m-input-file taxonomy.qza \\\n",
    "  --o-visualization taxonomy.qzv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!qiime taxa barplot \\\n",
    "  --i-table biom_id2.qza \\\n",
    "  --i-taxonomy taxonomy.qza \\\n",
    "  --m-metadata-file UC_Severity_1_MF_Idswap6.21.18.txt \\\n",
    "  --o-visualization taxa-bar-plots_idswap2.qzv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}