2016_DePy_conda_talk/DePy_conda_talk.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "<center>\n",
    "<img class=\"logo\" src=\"images/anaconda_logo_web.png\" height=\"20%\" width=\"20%\" align=\"center\"/>\n",
    "<h1>Building and Distributing Python Software with Conda</h1>\n",
    "<br/><br/>\n",
    "<h2>Jonathan J. Helmus\n",
    "</h2>\n",
    "<br/><br/>\n",
    "<h3>DePy 2016</h3>\n",
    "<h3>\n",
    "Chicago, IL</h3>\n",
    "<h3>May 7, 2016</h3>\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "### Abstract\n",
    "    \n",
    "Conda is a cross platform, package management system widely used in the scientific and data science Python communities. Conda can be used to package and distribute software written in any language but has first class support for Python packages. This talk will briefly cover how to use conda to install and manage data science packages as well as how conda can be used to create isolated computing environments. The main focus of the talk will be an in-depth look at how to easily and reproducibly create conda packages for your own Python software, and options for how to share these packages with others. Finally, combining a collection of conda packages into custom cross-platform installable conda-based Python distributions will be explored."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## About Me\n",
    "\n",
    "<img src=\"images/argonne_logo.png\" height=\"25%\" width=\"25%\" align=\"right\"/>\n",
    "\n",
    "* Scientist and software developer at Argonne National Laboratory\n",
    "* User of the SciPy stack and PyData tools\n",
    "    * NumPy, SciPy, matplotlib, Jupyter, pandas, ...\n",
    "* Contributor to a number of open source projects\n",
    "* conda enthusiast\n",
    "<img src=\"images/pydatalogo.png\" height=\"30%\" width=\"30%\" align=\"right\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The Problem\n",
    "\n",
    "A number of **powerful** Python data science libraries exist.  Installing and managing these libraries can be **difficult and time consuming**.\n",
    "\n",
    "\n",
    "* Traditional **package managers** (apt-get, brew, etc)\n",
    "    * out of date or missing packages\n",
    "    * operating system specific\n",
    "\n",
    "<img src=\"images/build_from_source_meme.jpg\" height=\"40%\" width=\"40%\" align=\"right\"/>  \n",
    "\n",
    "* **pip**\n",
    "    * often builds from source\n",
    "    * limited to Python packages\n",
    "<br/><br/> \n",
    "* Install everything from **scratch**\n",
    "    * works, but really?!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## [conda](http://conda.pydata.org/)\n",
    "\n",
    "<img src=\"images/keep-calm-and-conda-install.png\" height=\"40%\" width=\"40%\" align=\"right\"/>  \n",
    "* Cross-platform **package** and **environment** management system \n",
    "* Created by Continuum Analytics\n",
    "* Open source, BSD license\n",
    "* Created for Python programs but can package and manage any software.\n",
    "* Does not require administator privileges.\n",
    "* Available by installing **Anaconda** or **Miniconda** from Continuum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## conda : the package manager\n",
    "\n",
    "Conda is a cross platform package manager which installs binary conda packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "# Before talk\n",
    "conda create -n depy python=3.5\n",
    "source activate depy\n",
    "\n",
    "# In terminal during talk\n",
    "\n",
    "python\n",
    ">>> import pandas   # No module named pandas...what!\n",
    "\n",
    "conda list   # not much installed\n",
    "conda install pandas\n",
    "conda install matplotlib ipython\n",
    "ipython \n",
    ">>> import pandas\n",
    ">>> pandas.__version__   # 0.18.1\n",
    "\n",
    "conda list\n",
    "conda list pandas\n",
    "conda remove pandas\n",
    "conda install pandas=0.16\n",
    "python -c \"import pandas; print(pandas.__version__)\" # 0.16.2\n",
    "\n",
    "conda update pandas\n",
    "python -c \"import pandas; print(pandas.__version__)\" # 0.18.1\n",
    "\n",
    "# lots of other commands\n",
    "conda --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "* **conda install** : Install a package\n",
    "* **conda remove** : Remove a package\n",
    "* **conda update** : Update a package\n",
    "* **conda list** : List packages installed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## conda : the environment manager\n",
    "\n",
    "Conda can create isolated environments that have their own \n",
    "set of installed and managed packages.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "source deactivate\n",
    "\n",
    "# say we want to more easily go between a pandas 0.16 and 0.18, make seperate environments\n",
    "conda create\n",
    "\n",
    "conda create -n depy_pandas16 pandas=0.16 python=2.7  # also want to use Python 2.7\n",
    "source activate depy_pandas16\n",
    "python  # notice that we have python 2.7\n",
    ">>> print \"Hi DePy\"\n",
    ">>> import pandas\n",
    ">>> pandas.__version.__\n",
    "0.16.2\n",
    "\n",
    "source deactivate\n",
    "\n",
    "conda create -n depy_pandas18 pandas=0.18 python # defaults to latest Python, 3.5\n",
    "source activate depy_pandas18\n",
    "python\n",
    ">>> import pandas\n",
    ">>> pandas.__version.__\n",
    "\n",
    "source deactivate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* **conda create** : Create a new conda environment\n",
    "* **source activate** : Activate a conda enviroment\n",
    "* **source deactivate** : Deactivate the current conda enviroment\n",
    "\n",
    "Packages are hard linked into the enviroment to save disk space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Finding conda packages\n",
    "\n",
    "Conda can search for packages from the repository provided by Continuum as well as packages created by users and shared on the [Anaconda cloud](https://anaconda.org)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "conda search scikit-learn\n",
    "conda search tensorflow\n",
    "\n",
    "anaconda search tensorflow\n",
    "anaconda show jjhelmus/tensorflow\n",
    "\n",
    "# Search on Anaconda.org\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* **conda search** : Search for packages in the default repo\n",
    "* **anaconda search** : Search Anaconda.org for packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Creating and sharing conda packages\n",
    "\n",
    "Conda packages can be built from a recipe and shared on Anaconda.org"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "cd recipe/nmrglue\n",
    "\n",
    "cat meta.yaml\n",
    "cat build.sh\n",
    "cat bld.sh\n",
    "\n",
    "cd ..\n",
    "conda build nmrglue\n",
    "<lots of text>\n",
    "\n",
    "anaconda upload ...\n",
    "\n",
    "cd from_skeleton\n",
    "conda skeleton pypi nmrglue\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* **conda build** : Build a conda package from a recipe\n",
    "* **anaconda upload** : Upload a package to Anaconda.org\n",
    "* **conda skeleton** : Generate a boilderplate recipe from PyPI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## [conda-forge](https://conda-forge.github.io/)\n",
    "\n",
    "A community led collection of recipes, build infrastructure and distributions for the conda package manager.\n",
    "\n",
    "* reproducable method for building package on all platforms\n",
    "* recipes are submitted to the [**staged-recipes** repo](https://github.com/conda-forge/staged-recipes)\n",
    "* once working, a **feedstock** repository is created\n",
    "* bug fixes and new versions done in the **feedstock** repo\n",
    "* packages are uploaded to the [**conda-forge channel**](https://anaconda.org/conda-forge)\n",
    "<center>\n",
    "<img src=\"images/condaforge_logo.png\" height=\"40%\" width=\"40%\" align=\"center\"/>\n",
    "</center> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "# Show conda-forge webpage\n",
    "# Show staged-recipes repo\n",
    "\n",
    "cd ..\n",
    "cd staged-recipes\n",
    "cd recipes\n",
    "ls\n",
    "cat meta.yaml\n",
    "\n",
    "git checkout -b nmrglue\n",
    "git add .\n",
    "git commit -m \"Add nmrglue recipe\"\n",
    "git pull-request\n",
    "\n",
    "# Show staged-recipe repo\n",
    "# Show Py-ART repo\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## (conda) constructor\n",
    "\n",
    "[constructor](https://github.com/conda/constructor) is a tool for creating an installer from a collection of conda packages.\n",
    "\n",
    "* open source software by Continuum Analytics\n",
    "* install using, `conda install constructor`\n",
    "* Installer settings are specified in a `construct.yaml` file\n",
    "* cross-platform, create installers for Linux, OS X, and Windows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## conda from scratch?\n",
    "\n",
    "conda-forge packages + constructor == custom Python distribution?\n",
    "\n",
    "[**acpd**](https://github.com/acpd/acpd) : Another conda-based Python distribution\n",
    "* Proof of concept, builds installer which is self-hosting\n",
    "* repository include all conda recipes and scripts\n",
    "* Currently limited to linux-64 and Python 3.5\n",
    "* more to come...\n",
    "\n",
    "<center>\n",
    "<img src=\"images/acpd_logo.png\" height=\"30%\" width=\"30%\" align=\"center\"/>\n",
    "</center> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center>\n",
    "\n",
    "## Thanks\n",
    "<br/> <br/>\n",
    "## Questions?\n",
    "<br/> <br/>\n",
    "\n",
    "<center>\n",
    "<img src=\"images/jhelmus_square_headshot_medium.jpg\" height=\"25%\" width=\"25%\" align=\"center\"/>\n",
    "\n",
    "<center>\n",
    "Jonathan J. Helmus\n",
    "<br/><br/>\n",
    "jjhelmus@gmail.com \n",
    "<br/><br/>\n",
    "GitHub: jjhelmus \n",
    "<br/><br/>\n",
    "http://nmrglue.com/jhelmus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}