Embrace conda packages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![PyData_logo](./static/pydata-logo-madrid-2016.png)\n",
    "\n",
    "# Embrace conda packages\n",
    "\n",
    "## The build system we always needed, but never deserved\n",
    "\n",
    "###### Juan Luis Cano Rodríguez\n",
    "###### Madrid, 2016-04-08"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Outline\n",
    "\n",
    "* Introduction\n",
    "* Motivation: What brought us here?\n",
    "* Our first conda package\n",
    "* Some more tricks\n",
    "* Working with other languages\n",
    "* conda-forge: a community repository\n",
    "* Limitations and future work\n",
    "* Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Who is this guy?\n",
    "\n",
    "<img src=\"static/juanlu.jpg\" width=\"350px\" style=\"float: right\" />\n",
    "\n",
    "* _Almost_ **Aerospace Engineer**\n",
    "* Quant Developer for BBVA at Indizen (yeah, lots of Python there!)\n",
    "* Writer and furious tweeter at **Pybonacci**\n",
    "* Chair ~~and BDFL~~ of **Python España**\n",
    "* Co-creator and charismatic leader of **AeroPython** (\\*not the Lorena Barba course)\n",
    "* _When time permits (rare) [writes some open source Python code](https://github.com/Juanlu001/)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "You know, I've been giving talks on Python and its scientific ecosystem for about three years now... And I always write this bit there, that \"Almost\" word in italics before my background. You may reasonably wonder now what the heck I've been doing all these years to always introduce myself as an \"almost\" Aerospace Engineer, right? Well, I promise that I'm taking the required steps to graduate not later than this Autumn, but anyway this talk reflects one of the severe pains I've been going through while carrying my final project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Motivation: What brought us here?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Let's begin with some questions:\n",
    "\n",
    "* Who writes Python code here, either for a living or for fun?\n",
    "* Who can write a `setup.py`... without copying a working one from the Internet?\n",
    "* How many Linux users... can configure a Visual Studio project properly?\n",
    "* How many of you are using Anaconda... because it was the only way to survive?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### _...or: \"The sad state of scientific software\"_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* [The scientific Python community was told to \"fix the packaging problem themselves\" in 2014](https://speakerdeck.com/teoliphant/building-the-pydata-community), Christoph Gohlke packages were the only practical way to use Python on Windows for years before Python(x,y), Canopy and Anaconda were born"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* One of the FAQ items of the Sage project: [_\"Wouldn’t it be way better if Sage did not ship as a gigantic bundle?\"_](http://doc.sagemath.org/html/en/faq/faq-general.html), [they started a SaaS to end the pain](http://sagemath.blogspot.com.es/2014/10/a-non-technical-overview-of.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* PETSc (solution of PDEs): They are forced to maintain their own forks because [upstream projects won't fix bugs, even with patches and reproducible tests](http://scisoftdays.org/pdf/2016_slides/brown.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* DOLFIN (part of the FEniCS project): Extremely difficult to make it work outside Ubuntu, pure Python alternatives are being developed, [my fenics-recipes project has at least 7 meaningful forks already](http://firedrakeproject.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"static/STAHP.jpg\" style=\"margin: 0 auto;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Some inconvenient truths: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# Portability is hard (unless you stick to pure Python)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# Properly distributing software libraries is very hard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Result:\n",
    "\n",
    "<img src=\"static/keep-calm-it-works-on-my-machine.png\" width=\"400px\" style=\"margin: 0 auto;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## What horror have we created\n",
    "\n",
    "> If you’re missing a library or program, and that library or program happens to be written in C, **you either need root to install it from your package manager, or you will descend into a lovecraftian nightmare of attempted local builds from which there is no escape**. You say you need lxml on shared hosting and they don’t have libxml2 installed? Well, fuck you.\n",
    ">\n",
    "> — Eevee, [\"The sad state of web app deployment\"](https://eev.ee/blog/2015/09/17/the-sad-state-of-web-app-deployment/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Are virtual machines and containers the solution?\n",
    "\n",
    "> _\"It's easy to build a VM if you automate the install process, and providing that install script for even one OS can demystify the install process for others; conversely, **just because you provide a VM doesn't mean that anyone other than you can install your software**\"_\n",
    ">\n",
    "> — C. Titus Brown, [\"Virtual machines considered harmful for reproducibility\"](http://ivory.idyll.org/blog/vms-considered-harmful.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Our first conda package\n",
    "\n",
    "Let's install `conda-build`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Anaconda Cloud api site https://api.anaconda.org\n",
      "Fetching package metadata: ........\n",
      "Solving package specifications: .........\n",
      "\n",
      "# All requested packages already installed.\n",
      "# packages in environment at /home/juanlu/.miniconda3:\n",
      "#\n",
      "conda-build               1.20.0                   py34_0  \n"
     ]
    }
   ],
   "source": [
    "!conda install -y conda-build -q -n root"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "conda packages are created from conda recipes. We can create a bare recipe using `conda skeleton` to build it from a PyPI package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Anaconda Cloud api site https://api.anaconda.org\r\n"
     ]
    }
   ],
   "source": [
    "!conda skeleton pypi pytest-benchmark > /dev/null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bld.bat  build.sh  meta.yaml\r\n"
     ]
    }
   ],
   "source": [
    "!ls pytest-benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "These are the minimum files for the recipe:\n",
    "\n",
    "* `meta.yaml` contains all the metadata\n",
    "* `build.sh` and `bld.bat` are the build scripts for Linux/OS X and Windows respectively"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### The `meta.yaml` file\n",
    "\n",
    "It contains the metadata in YAML format.\n",
    "\n",
    "* `package`, `source` and `build` specify the name, version and source of the package\n",
    "* `requirements` specify the build (install time) and run (runtime) requirements\n",
    "* `test` specify imports, commands and scripts to test\n",
    "* `about` adds some additional data for the package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "package:\r\n",
      "  name: pytest-benchmark\r\n",
      "  version: \"3.0.0\"\r\n",
      "\r\n",
      "source:\r\n",
      "  fn: pytest-benchmark-3.0.0.zip\r\n",
      "  url: https://pypi.python.org/packages/source/p/pytest-benchmark/pytest-benchmark-3.0.0.zip\r\n",
      "  md5: f8ab8e438f039366e3765168ad831b4c\r\n",
      "\r\n",
      "build:\r\n",
      "  preserve_egg_dir: True\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "requirements:\r\n",
      "  build:\r\n",
      "    - python\r\n",
      "    - setuptools\r\n",
      "    - pytest >=2.6\r\n",
      "\r\n",
      "  run:\r\n",
      "    - python\r\n",
      "    - setuptools\r\n",
      "    - pytest >=2.6\r\n"
     ]
    }
   ],
   "source": [
    "!grep -v \"#\" pytest-benchmark/meta.yaml | head -n24"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## The `build.sh` and `bld.bat` files\n",
    "\n",
    "They specify how to build the package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#!/bin/bash\r\n",
      "\r\n",
      "$PYTHON setup.py install\r\n",
      "\r\n",
      "# Add more build steps here, if they are necessary.\r\n",
      "\r\n",
      "# See\r\n",
      "# http://docs.continuum.io/conda/build.html\r\n",
      "# for a list of environment variables that are set during the build process.\r\n"
     ]
    }
   ],
   "source": [
    "!cat pytest-benchmark/build.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"%PYTHON%\" setup.py install\r\n",
      "if errorlevel 1 exit 1\r\n",
      "\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!grep -v \"::\" pytest-benchmark/bld.bat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### The build process\n",
    "\n",
    "Adapted from http://conda.pydata.org/docs/building/recipe.html#conda-recipe-files-overview\n",
    "\n",
    "1. Downloads the source\n",
    "2. Applies patches (if any)\n",
    "3. Install build dependencies\n",
    "4. Runs the build script\n",
    "5. Packages new files\n",
    "6. Run tests against newly created package\n",
    "\n",
    "Seems legit!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Anaconda Cloud api site https://api.anaconda.org\n",
      "+ /home/juanlu/.miniconda3/envs/_build/bin/python setup.py install\n",
      "warning: no directories found matching 'examples'\n",
      "warning: no files found matching '.isort.cfg'\n",
      "warning: no files found matching '.pylintrc'\n",
      "warning: no previously-included files matching '*.py[cod]' found anywhere in distribution\n",
      "warning: no previously-included files matching '__pycache__' found anywhere in distribution\n",
      "warning: no previously-included files matching '*.so' found anywhere in distribution\n",
      "warning: no previously-included files matching '*.dylib' found anywhere in distribution\n"
     ]
    }
   ],
   "source": [
    "!conda build pytest-benchmark --python 3.5 > /dev/null  # It works!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/juanlu/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2\r\n"
     ]
    }
   ],
   "source": [
    "!ls ~/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"static/conda-names.png\" style=\"margin: 0 auto;\" />\n",
    "\n",
    "(From http://conda.pydata.org/docs/building/pkg-name-conv.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Anaconda Cloud api site https://api.anaconda.org\n",
      "Fetching package metadata: ..........\n",
      "Solving package specifications: .........\n",
      "\n",
      "Package plan for installation in environment /home/juanlu/.miniconda3/envs/py35:\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    pytest-benchmark: 3.0.0-py35_0\n",
      "\n",
      "Linking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n"
     ]
    }
   ],
   "source": [
    "!conda install pytest-benchmark --use-local --yes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Build, test, upload, repeat\n",
    "\n",
    "* Custom packages can be uploaded to Anaconda Cloud https://anaconda.org/\n",
    "* This process can be automated through Anaconda Build http://docs.anaconda.org/build.html\n",
    "* Later on we can use our custom **channels** to install non-official packages\n",
    "\n",
    "<img src=\"static/anaconda-cloud.png\" style=\"margin: 0 auto;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's upload the package first using `anaconda-client`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Anaconda Cloud api site https://api.anaconda.org\n",
      "Fetching package metadata: ........\n",
      "Solving package specifications: .........\n",
      "\n",
      "# All requested packages already installed.\n",
      "# packages in environment at /home/juanlu/.miniconda3/envs/py35:\n",
      "#\n",
      "anaconda-client           1.4.0                    py35_0  \n"
     ]
    }
   ],
   "source": [
    "!conda install anaconda-client --quiet --yes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Anaconda Cloud api site https://api.anaconda.org\n",
      "detecting package type ...\n",
      "conda\n",
      "extracting package attributes for upload ...\n",
      "done\n",
      "\n",
      "Uploading file Juanlu001/pytest-benchmark/3.0.0/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 ... \n",
      " uploaded 54 of 54Kb: 100.00% ETA: 0.0 minutes\n",
      "\n",
      "\n",
      "Upload(s) Complete\n",
      "\n",
      "Package located at:\n",
      "https://anaconda.org/juanlu001/pytest-benchmark\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!anaconda upload ~/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "And now, let's install it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Anaconda Cloud api site https://api.anaconda.org\r\n"
     ]
    }
   ],
   "source": [
    "!conda remove pytest-benchmark --yes > /dev/null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Anaconda Cloud api site https://api.anaconda.org\n",
      "Fetching package metadata: ..........\n",
      "Solving package specifications: .........\n",
      "\n",
      "Package plan for installation in environment /home/juanlu/.miniconda3/envs/py35:\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    pytest-benchmark: 3.0.0-py35_0\n",
      "\n",
      "Linking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n"
     ]
    }
   ],
   "source": [
    "!conda install pytest-benchmark --channel juanlu001 --yes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Some more tricks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Running the tests\n",
    "\n",
    "You can run your tests with Python, Perl or shell scripts (`run_test.[py,pl,sh,bat]`)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# run_test.sh\n",
    "\n",
    "cd $SRC_DIR/test\n",
    "cmake .\n",
    "make"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Convert pure Python packages to other platforms\n",
    "\n",
    "Using `conda convert` for pure Python packages, we can quickly provide packages for other platforms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converting /home/juanlu/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 from linux to osx-64\n",
      "Converting /home/juanlu/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 from linux to linux-32\n",
      "Converting /home/juanlu/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 from linux to linux-64\n",
      "Converting /home/juanlu/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 from linux to win-32\n",
      "Converting /home/juanlu/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 from linux to win-64\n"
     ]
    }
   ],
   "source": [
    "!conda convert ~/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 --platform all | grep Converting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Platform-specific metadata"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# from glpk\n",
    "\n",
    "build:\n",
    "  features:\n",
    "    - vc9      [win and py27]\n",
    "    - vc10     [win and py34]\n",
    "    - vc14     [win and py35]\n",
    "\n",
    "requirements:\n",
    "  build:\n",
    "    - gmp      [linux or osx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Templating for `meta.yaml`\n",
    "\n",
    "Metadata files support templating using Jinja2!"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# from glpk\n",
    "\n",
    "build:\n",
    "  number: {{ environ.get(\"APPVEYOR_BUILD_NUMBER\", 1) }} [win]\n",
    "\n",
    "# from poliastro at conda-forge\n",
    "\n",
    "{% set version = \"0.5.0\" %}\n",
    "\n",
    "package:\n",
    "  name: poliastro\n",
    "  version: {{ version }}\n",
    "\n",
    "source:\n",
    "  fn: v{{ version }}.tar.gz\n",
    "  url: https://github.com/poliastro/poliastro/archive/v{{ version }}.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Working with other languages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### _or: conda as a cross-platform package manager_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* conda can be used to build software written in any language\n",
    "* Just don't include `python` as a build or run dependency!\n",
    "* It's already being used to distribute pure C and C++ libraries, R packages..."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# build.sh from glpk\n",
    "\n",
    "export CFLAGS=\"-O3\"\n",
    "./configure --prefix=$PREFIX --with-gmp\n",
    "\n",
    "make check install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Important caveat:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## The burden is on _you_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### _There be dragons_\n",
    "\n",
    "* _conda-build does not solve cross-compiling_ so you will need to build compiled packages on each platform\n",
    "* Regarding Linux, there are [a lot of sources of binary incompatibility](https://www.python.org/dev/peps/pep-0513/#key-causes-of-inter-linux-binary-incompatibility)\n",
    "  - Building on a clean operative system is key\n",
    "  - Using an old version of Linux (CentOS 5?) also helps, because many core system libraries have strict backwards compatibility policies\n",
    "  - **Packages that assume everything is on root locations will fail to compile**\n",
    "  - Sometimes careful editing of compiler flags and event patching is necessary\n",
    "\n",
    "If the recipe builds on a fresh, headless, old Linux it will work everywhere"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## conda-forge: a community repository\n",
    "\n",
    "<img src=\"static/conda-forge.png\" style=\"margin: 0 auto;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "> [**conda-forge**](https://github.com/conda-forge) is a github organization containing repositories of conda recipes. Thanks to some awesome continuous integration providers (AppVeyor, CircleCI and TravisCI), each repository, also known as a feedstock, automatically builds its own recipe in a clean and repeatable way on Windows, Linux and OSX."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Features:\n",
    "\n",
    "* Automatic linting of recipes\n",
    "* Continuous integration of recipes in Linux, OS X and Windows\n",
    "* Automatic upload of packages\n",
    "\n",
    "What I love:\n",
    "\n",
    "* Having a blessed community channel (like Arch Linux AUR)\n",
    "* Ensuring recipes run everywhere\n",
    "* High quality standards!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Limitations and future work\n",
    "\n",
    "conda (2012?) and conda-build (2013) are very young projects and still have some pain points that ought to be addressed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Support for [gcc](https://github.com/conda/conda-recipes/pull/279) and [libgfortran](https://github.com/ContinuumIO/anaconda-issues/issues/686) is not yet polished in Anaconda and there are still some portability issues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* [No way to include custom channels on a `meta.yaml`](https://github.com/conda/conda-build/issues/532), the only option is to keep a copy of all dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* [Pinning NumPy versions on `meta.yaml` can be a mess](https://github.com/conda/conda-build/pull/650)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The state of Python packaging is improving upstream too!\n",
    "\n",
    "* pip builds and caches wheels locally - the problem of compiling NumPy over and over again was addressed a while ago\n",
    "* Windows and OS X wheels are easy to build and widely available for many scientific packages\n",
    "* [PEP 0513](https://www.python.org/dev/peps/pep-0513/) provides a way to **finally** upload Linux wheels to PyPI which are compatible with _many_ Linux distributions\n",
    "* [PEP 0516](https://www.python.org/dev/peps/pep-0516/) proposes \"a simple and standard sdist format **that isn't intertwined with distutils**\"!!!1!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Still, there are some remaining irks:\n",
    "\n",
    "* [pip does not have a dependency solver](https://github.com/pypa/pip/issues/988)\n",
    "* conda-build has a more streamlined process to build and test packages in an isolated way"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "<img src=\"static/keep-calm-and-conda-install.png\" style=\"margin: 0 auto;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"static/pydata-logo-madrid-2016.png\" style=\"margin: 0 auto;\" />\n",
    "\n",
    "* This talk: https://github.com/AeroPython/embrace-conda-packages\n",
    "* My GitHub: https://github.com/Juanlu001/\n",
    "* Me on Twitter: @astrojuanlu, @Pybonacci, @PyConES, @AeroPython\n",
    "\n",
    "### Approach me during the conference, interrupt me while I'm on a conversation, ask me questions, let's talk about your ideas and projects! 😊\n",
    "\n",
    "# Thanks for yor attention!"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}