pandas_workshop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas SciProg Workshop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For reference:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://github.com/jakevdp/PythonDataScienceHandbook - Data Science textbook all in iPython notebooks; much of the text borrowed from here\n",
    "\n",
    "Extremely useful, and completely free. Great for beginners.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas is a newer package built on top of NumPy, a powerful N-dimensional array object, and provides an efficient implementation of a ``DataFrame``.\n",
    "``DataFrame``s are essentially multidimensional arrays with attached row and column labels, and often with heterogeneous types and/or missing data.\n",
    "As well as offering a convenient storage interface for labeled data, Pandas implements a number of powerful data operations familiar to users of both database frameworks and spreadsheet programs.\n",
    "\n",
    "NumPy's ``ndarray`` data structure provides essential features for the type of clean, well-organized data typically seen in numerical computing tasks.\n",
    "While it serves this purpose very well, its limitations become clear when we need more flexibility (e.g., attaching labels to data, working with missing data, etc.) and when attempting operations that do not map well to element-wise broadcasting (e.g., groupings, pivots, etc.), each of which is an important piece of analyzing the less structured data available in many forms in the world around us.\n",
    "Pandas, and in particular its ``Series`` and ``DataFrame`` objects, builds on the NumPy array structure and provides efficient access to these sorts of \"data munging\" tasks that occupy much of a data scientist's time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd # Pandas!\n",
    "import numpy as np # NumPy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Pandas ``DataFrame``\n",
    "\n",
    "The fundamental structure in Pandas is the ``DataFrame``. The ``DataFrame`` can be thought of either as a generalization of a NumPy array, or as a specialization of a Python dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data1 = pd.DataFrame({'a': [1,2,3],\n",
    "                      'b': [2,3,4],\n",
    "                      'c': })# Fill in here\n",
    "# What happens if there are more than 3 elements in c?\n",
    "data1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data2 = pd.DataFrame([{'a': 1, 'b': 2, 'c': 4},\n",
    "                      {'a': 2, 'b': 3, 'c': 5},\n",
    "                      {}]) # Fill in here\n",
    "# What happens if there are more than 3 elements in the last row above?\n",
    "data2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The ``Series`` object\n",
    "Each column of the ``DataFrame`` corresponds to a ``Series`` object.\n",
    "\n",
    "The ``Series`` object is basically interchangeable with a one-dimensional NumPy array.\n",
    "The essential difference is the presence of the index: while the Numpy Array has an *implicitly defined* integer index used to access the values, the Pandas ``Series`` has an *explicitly defined* index associated with the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data1['a']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data1.a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data1['a'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data1['a'].index = ['First', 'Second', 'Third']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data1['a']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, ``DataFrames`` are usually created from tabulated data. In this case, let's read some normalized transcriptomic data from TCGA. Each file is compressed using gzip and is tab delimited."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cancer_data = pd.read_csv('TCGA_sample.tsv.gz', compression='gzip', sep='\\t', index_col=0)\n",
    "cancer_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing and Selecting Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexers: loc, iloc, and ix\n",
    "\n",
    "``loc``: Selection By Label\n",
    "\n",
    "``iloc``: Selection By Position (more pythonic)\n",
    "\n",
    "``ix``: Label-based, but falls back to position-based if label doesn't exist. Good for mixed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Examples\n",
    "# Note: loc is indexed at 1, whereas iloc is indexed at 0\n",
    "cancer_data.loc['TCGA-S9-A7J2-01':'TCGA-EE-A17X-06', ['PTEN', 'BRCA1']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cancer_data.iloc[1:10:5, 0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cancer_data.ix[3:5, 'PTEN': 'BRCA2']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to ``NumPy`` arrays, vector calculations can be done on ``Series`` objects. Try finding the sum of gene expression values for the ``BRCA1`` and ``BRCA2`` genes for each patient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cancer_data['BRCA1'] + cancer_data['BRCA2']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Your turn\n",
    "\n",
    "1) Try to find out whether the first half of patients have greater PTEN expression or the last half (Hint: np.sum()).\n",
    "\n",
    "2) Despite it not making sense to do so, try adding expression values of the BRCA genes for every other patient.\n",
    "\n",
    "3) Other suggestions?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Masking\n",
    "\n",
    "You can use an array of ``True`` or ``False`` values to mask values that you do not want to see. For example, what if we're only interested in patients whose ``BRCA1`` expression exceeds 8."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cancer_data['BRCA1'] > 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cancer_data[cancer_data['BRCA1'] > 8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### All NumPy functions can be used on Pandas Series, including ufuncs (vectorized functions)\n",
    "https://docs.scipy.org/doc/numpy-1.13.0/reference/ufuncs.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using DataFrame.apply()\n",
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.apply.html\n",
    "\n",
    "Let's find the mean expression for each gene.\n",
    "\n",
    "For the programmers: Do not use for-loops\n",
    "\n",
    "http://stupidpythonideas.blogspot.ca/2015/09/going-faster-with-numpy.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "numerical_cancer = cancer_data.loc[:, 'PTEN': 'TP53']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "numerical_cancer.apply(np.mean, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about the sum of gene expression across each patient?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "numerical_cancer.apply(np.sum, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also define any arbitrary function using Python `lambda` functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Lambda example:\n",
    "numerical_cancer.apply(lambda col: np.sum(col) + np.mean(col), axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenge task\n",
    "\n",
    "Using the ``Pandas`` and ``NumPy`` documentation, try to attempt this challenge. In this hypothetical scenario, we want to find patients whose expression of ``BRCA1`` and ``BRCA2`` are greater than the mean expression for those genes. Then, find the standard deviation of ``PTEN`` expression for those patients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}