python-6-multiple-datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "## Analyzing Multiple Data Sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "We have created a function called `analyze` that creates graphs of the minimum, average, and maximum daily inflammation rates\n",
    "for a single data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "def analyze(filename):\n",
    "    data = np.loadtxt(fname=filename, delimiter=',')\n",
    "    \n",
    "    plt.figure(figsize=(10.0, 3.0))\n",
    "    \n",
    "    plt.subplot(1, 3, 1)\n",
    "    plt.ylabel('average')\n",
    "    plt.plot(data.mean(0))\n",
    "    \n",
    "    plt.subplot(1, 3, 2)\n",
    "    plt.ylabel('max')\n",
    "    plt.plot(data.max(0))\n",
    "    \n",
    "    plt.subplot(1, 3, 3)\n",
    "    plt.ylabel('min')\n",
    "    plt.plot(data.min(0))\n",
    "    \n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "analyze('inflammation-01.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "We can use it to analyze other data sets one by one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "analyze('inflammation-02.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "but we have a dozen data sets right now and more on the way.\n",
    "We want to create plots for all our data sets with a single statement.\n",
    "To do that,\n",
    "we'll have to teach the computer how to repeat things."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": [
     "objectives"
    ]
   },
   "source": [
    "#### Objectives\n",
    "\n",
    "*   Explain what a for loop does.\n",
    "*   Correctly write for loops to repeat simple calculations.\n",
    "*   Trace changes to a loop variable as the loop runs.\n",
    "*   Trace changes to other variables as they are updated by a for loop.\n",
    "*   Explain what a list is.\n",
    "*   Create and index lists of simple values.\n",
    "*   Use a library function to get a list of filenames that match a simple wildcard pattern.\n",
    "*   Use a for loop to process multiple files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "### Lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "Just as a `for` loop is a way to do operations many times,\n",
    "a list is a way to store many values.\n",
    "Unlike NumPy arrays,\n",
    "there are built into the language.\n",
    "We create a list by putting values inside square brackets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "odds = [1, 3, 5, 7]\n",
    "print 'odds are:', odds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "We select individual elements from lists by indexing them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print 'first and last:', odds[0], odds[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "and if we loop over a list,\n",
    "the loop variable is assigned elements one at a time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for number in odds:\n",
    "    print number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "There is one important difference between lists and strings:\n",
    "we can change the values in a list,\n",
    "but we cannot change the characters in a string.\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "names = ['Newton', 'Darwing', 'Turing'] # typo in Darwin's name\n",
    "print 'names is originally:', names\n",
    "names[1] = 'Darwin' # correct the name\n",
    "print 'final value of names:', names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "works, but:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "name = 'Bell'\n",
    "name[0] = 'b'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "does not.\n",
    "\n",
    "> #### Ch-Ch-Ch-Changes\n",
    ">\n",
    "> Data that can be changed is called [mutable](../../gloss.html#mutable),\n",
    "> while data that cannot be is called [immutable](../../gloss.html#immutable).\n",
    "> Like strings,\n",
    "> numbers are immutable:\n",
    "> there's no way to make the number 0 have the value 1 or vice versa\n",
    "> (at least, not in Python&mdash;there actually *are* languages that will let people do this,\n",
    "> with predictably confusing results).\n",
    "> Lists and arrays,\n",
    "> on the other hand,\n",
    "> are mutable:\n",
    "> both can be modified after they have been created.\n",
    ">\n",
    "> Programs that modify data in place can be harder to understand than ones that don't\n",
    "> because readers may have to mentally sum up many lines of code\n",
    "> in order to figure out what the value of something actually is.\n",
    "> On the other hand,\n",
    "> programs that modify data in place instead of creating copies that are almost identical to the original\n",
    "> every time they want to make a small change\n",
    "> are much more efficient.\n",
    "\n",
    "There are many ways to change the contents of in lists besides assigning to elements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "odds.append(11)\n",
    "print 'odds after adding a value:', odds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "del odds[0]\n",
    "print 'odds after removing the first element:', odds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "odds.reverse()\n",
    "print 'odds after reversing:', odds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "### Processing Multiple Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "We now have almost everything we need to process all our data files.\n",
    "The only thing that's missing is a library with a rather unpleasant name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import glob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "The `glob` library contains a single function, also called `glob`,\n",
    "that finds files whose names match a pattern.\n",
    "We provide those patterns as strings:\n",
    "the character `*` matches zero or more characters,\n",
    "while `?` matches any one character.\n",
    "We can use this to get the names of all the IPython Notebooks we have created so far:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print glob.glob('*.ipynb')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "or to get the names of all our CSV data files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print glob.glob('*.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "As these examples show,\n",
    "`glob.glob`'s result is a list of strings,\n",
    "which means we can loop over it\n",
    "to do something with each filename in turn.\n",
    "In our case,\n",
    "the \"something\" we want is our `analyze` function.\n",
    "Let's test it by analyzing the first three files in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_tags": [],
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filenames = glob.glob('*.csv')\n",
    "filenames = filenames[0:3]\n",
    "for f in filenames:\n",
    "    print f\n",
    "    analyze(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "Sure enough,\n",
    "the maxima of these data sets show exactly the same ramp as the first,\n",
    "and their minima show the same staircase structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": [
     "challenges"
    ]
   },
   "source": [
    "#### Challenges\n",
    "\n",
    "1.  Write a function called `analyze_all` that takes a filename pattern as its sole argument\n",
    "    and runs `analyze` for each file whose name matches the pattern."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "#### Key Points\n",
    "\n",
    "*   Use `for variable in collection` to process the elements of a collection one at a time.\n",
    "*   The body of a for loop must be indented.\n",
    "*   Use `len(thing)` to determine the length of something that contains other values.\n",
    "*   `[value1, value2, value3, ...]` creates a list.\n",
    "*   Lists are indexed and sliced in the same way as strings and arrays.\n",
    "*   Lists are mutable (i.e., their values can be changed in place).\n",
    "*   Strings are immutable (i.e., the characters in them cannot be changed).\n",
    "*   Use `glob.glob(pattern)` to create a list of files whose names match a pattern.\n",
    "*   Use `*` in a pattern to match zero or more characters, and `?` to match any single character."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_tags": []
   },
   "source": [
    "#### Next Steps\n",
    "\n",
    "We have now solved our original problem:\n",
    "we can analyze any number of data files with a single command.\n",
    "More importantly,\n",
    "we have met two of the most important ideas in programming:\n",
    "\n",
    "1.  Use functions to make code easier to re-use and easier to understand.\n",
    "1.  Use lists and arrays to store related values, and loops to repeat operations on them.\n",
    "\n",
    "We have one more big idea to introduce,\n",
    "and then we will be able to go back and create a heat map\n",
    "like the one we initially used to display our first data set."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}