19_neural/19_01_tensor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to Neural Networks through SKFlow\n",
    "\n",
    "<img src=\"https://pbs.twimg.com/media/CbccPCBWAAErTII.jpg\"/>\n",
    "\n",
    "By now, you've installed tensorflow as well as the sklearn-centric wrapper, SKFlow.\n",
    "\n",
    "- To install tensorflow:\n",
    "\n",
    "`conda install -c https://conda.anaconda.org/jjhelmus tensorflow=0.6.0`\n",
    "\n",
    "- To install skflow afterwards:\n",
    "\n",
    "`pip install skflow`\n",
    "\n",
    "- Check that both components have correctly been installed in your ipython notebook:\n",
    "\n",
    "`import skflow`\n",
    "\n",
    "`import tensorflow as tf`\n",
    "\n",
    "To adequately use TensorFlow would require a number of class sessions, but let's explore some of the basics through SKFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn import metrics\n",
    "from sklearn import cross_validation\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import skflow\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Digit Recognition\n",
    "\n",
    "We're going to use the digits dataset again. In reality, this is a subset of the classic MNIST database of handwritten digits. Overall, MNIST has 60,000 training examples, and a test set of 10,000 examples. \n",
    "\n",
    "Example algorithm performances (test error rate %):\n",
    "- K Nearest Neighbors: 3%\n",
    "- SVM with RBF kernel: 1.4%\n",
    "- Boosted Stumps (depth=1 decision trees): 7.7%\n",
    "- Boosted Trees: 1.53%\n",
    "\n",
    "- 2 Layer NN: 4.7%\n",
    "- Modified Convolutional Nets: < 1%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "digits = datasets.load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABKUAAAEcCAYAAAAWQAdRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAG11JREFUeJzt3XuMpeddH/Dvk2xEruysU3EL2LObNq1o2h3HVtSIlozb\nNQKUshNaWyAuHhVk80cLY6Gy7gU8hoJsqKo1AoSR3GySQkgMeAIoEMWFcUQQtF4ySwOhCLJjQhua\nBO9sLsQikKd/nNkw2Xiyzzt7znNmzn4+0kp7+Z73/M6x57fn/e57zpRaawAAAACgp2dNewAAAAAA\nrj1KKQAAAAC6U0oBAAAA0J1SCgAAAIDulFIAAAAAdKeUAgAAAKC7A19KlVLeVkr5lnFngdllbwBD\n2BnAUPYGMMS1vDNKrbX/nZbysSSX7vgFSZ5O8tfbv76z1vqm7kONWSnlnyX58SRfluS3kyzXWv9k\nulPBwTXre6OU8pwkb0pyU5IbktxSa318ulPBwXUN7Ix/lOQHkrwio8e1nuQ7a61/Ns254CC7BvbG\nlyd5Q5Jj2791NqO98d7pTQUH16zvjJ1KKd+XZDXJiVrrr43z2FO5UqrW+sJa64tqrS9K8mSS11z6\n9c7/cKWUQ9OY72qVUv5Wkp9P8h+SHEnyRJI3T3UoOOBmfW9se2eSb07yZ/mbv+CAPbgGdsZckp/M\nqMS+IclHk7xuqhPBAXcN7I3/k+S2JC/e/vGLSX52qhPBAXYN7IwkSSnlpUn+ZZL/O4nj76u375VS\nFkspf1pK+Z5SygeSPFxKmSul/HIp5YOllKdKKb9USnnJjtusl1K+bfvny6WU3yil/Mh29n2llK/e\nY/ZoKeWdpZSPlFLeUUr58VLKGxsfytcneU+t9edrrX+ZUaN4vJTysqt/loCdZmVv1Fo/WWv90Vrr\nu/I3/8ICjNkM7Yxf3X6d8bFa6ycyujr7K8b0NAE7zNDeuFhrPV9Hb5V5dpJPJXnpeJ4l4JJZ2Rk7\n/FiSU0k+eTXPy272VSm17Qszurro+iR3ZTTjw9u/vj7JJzJ6Ui6p+cwrCl6Z5A8yav9/ePu2e8n+\nTJLfSnJdRqXSN++8bSnlXCnlG3Z5DH8/yblP32mtf5Hkj5K8fLcHDVyVWdgbQD+zuDO+Msl7GrPA\ncDOzN0opW9vz/miSH/pcWWDPZmJnlFJuS/J0rfVXrvB492w/Xkb2qST31lo/mVET93SSRy/9YSnl\nh5J8rvcwPllrfXg7+4YkP1FK+YJa6wdbs0mem+TmjD7T5a+SvKuU8otJyqUb1lqPf44ZXpDkQ5f9\n3keSvPBz3AbYu1nYG0A/M7UzSin/MMn3Jvm6ljywJzOzN2qtc6WU5ye5I6O3HAHjd+B3RinlRUl+\nMMmJlge8V/vxSqkPbb/lLUlSSnl+KeWhUspmKeVikseTHC6llF1u/+kP+Ny+QinZvQzaLfslSZ6q\ntT69I/v+AY/hY0k+/7LfO5zR5z0A4zcLewPoZ2Z2Rinlbyd5W0YfVvyuobcHms3M3thx3J9M8oYy\n+jxcYLxmYWesJnnjZd+wbbd592w/llKXf7jvdyd5WZJX1loPJ3l1Rk/E2J+MHT6Q5LpSyvN2/N71\nA27/e0k+3TiWUl6Q0fu1f2884wGXmYW9AfQzEzujlHJDknck+f5a60+Pczjgs8zE3rjMs5M8P8lL\nrhQEBpuFnfFPk3xnKeUD25+N9WVJ3lJK+bfjHHI/llKXe2FG77e8WEq5Lsm9k77DWuuTGX3HvNVS\nynNKKa9K8pq0fzesR5O8vJTy9aWU52Y080at9Q8nMzFwmYO4N1JK+bztnZEkO38OTNaB2xnbH476\na0l+rNb6U5ObFNjFQdwbJ0opC6WUZ5dSPj/Jf0nyVJL3Tmxo4JIDtzOS/LOMPi/7eJKFjL773p1J\nfmKcc+7HUuryJ+h0kucl+XCS30zyK8+Q2Xnby/9sr9lvSvKqJH+e5AeSvDnJzsvv3lNK+cZnPHCt\nH07yLzJ6/+VTGb2P04cbw+Qc+L2x7X8n+YuMLrV9e5KPl1JcbQXjNws749uTHM3oheZHt398ZJcs\ncPVmYW/MJXlTkq2MvgnT0SRfvfMtRsDYHPidUWt9qtb6we0f/y+j7xB+odb68V1m2ZMy+o6gXEkp\n5c1Jfr/Wet+0ZwEOBnsDGMLOAIayN4Ah9uPO2I9XSu0LpZSbSykvLaU8q5TyNRl9R5u1ac8F7F/2\nBjCEnQEMZW8AQxyEnXFo2gPsY1+U5BeSvDijT6j/jlrruemOBOxz9gYwhJ0BDGVvAEPs+53h7XsA\nAAAAdOftewAAAAB0p5QCAAAAoDulFAAAAADdKaUAAAAA6E4pBQAAAEB3SikAAAAAulNKAQAAANCd\nUgoAAACA7pRSAAAAAHSnlAIAAACgO6UUAAAAAN0ppQAAAADoTikFAAAAQHdKKQAAAAC6U0oBAAAA\n0J1SCgAAAIDuDo3hGHUMx/gsjzzySHP21KlTzdlbb7110Bz3339/c/bIkSODjj1AmdSBYUomsjeG\nWFxcbM5ubW01Z1dXV5uzS0tLzdmB7Axm0dT3xvr6enN2yNf3wsLCRGYYyN5g1kxkZzzwwAPN2Xvu\nuac5e/To0ebs2bNnm7POT2CQqb/WGHLesby83JxdW1vbwzRj94x7w5VSAAAAAHSnlAIAAACgO6UU\nAAAAAN0ppQAAAADoTikFAAAAQHdKKQAAAAC6U0oBAAAA0J1SCgAAAIDulFIAAAAAdHdo2gPs5tSp\nU83Z8+fPN2cvXLgwaI7rrruuOfuWt7ylOXvbbbcNmgMYr7m5uebs448/3pxdX19vzi4tLTVngcnY\n2Nhozt5yyy3N2cOHDzdnNzc3m7PA+N1zzz3N2SGv9x966KHm7F133dWcPXv2bHP2xIkTzVlg+s6c\nOdOcXVhYmNwgHblSCgAAAIDulFIAAAAAdKeUAgAAAKA7pRQAAAAA3SmlAAAAAOhOKQUAAABAd0op\nAAAAALpTSgEAAADQnVIKAAAAgO6UUgAAAAB0d6jnnZ09e7Y5e/78+ebsH//xHzdnjx071pxNkltv\nvbU5O+Tx3XbbbYPmAK5sY2OjObu+vj6RGRYWFiZyXGAy1tbWmrPHjx9vzi4tLTVn77vvvuYsMH53\n3nlnc/bUqVPN2Ztuuqk5e/To0ebsiRMnmrPA9G1tbTVnz5w505xdWVlpzm5ubjZnh5ifn7/qY7hS\nCgAAAIDulFIAAAAAdKeUAgAAAKA7pRQAAAAA3SmlAAAAAOhOKQUAAABAd0opAAAAALpTSgEAAADQ\nnVIKAAAAgO6UUgAAAAB0d6jnnV24cKE5+4pXvKI5e+zYsb2M0+Smm26a2LGBKzt9+nRzdnV1tTl7\n8eLFPUxzZYuLixM5LjAZKysrzdn5+fmJHPfkyZPNWWD8hpxLvO9972vOnj9/vjl74sSJ5uyQc6oj\nR440Z4HJOHPmTHN2c3OzObu8vNycHfK6ZG5urjk75PxrN66UAgAAAKA7pRQAAAAA3SmlAAAAAOhO\nKQUAAABAd0opAAAAALpTSgEAAADQnVIKAAAAgO6UUgAAAAB0p5QCAAAAoDulFAAAAADdHep5Zxcu\nXGjO3nrrrROcpN2QmY8cOTLBSeDatLKy0pxdXl5uzk7q63Vra2sixwXaDfk6PH36dHN2bW1tL+Nc\n0ZkzZyZyXGD8jh071px96qmnmrMnTpyYSPaxxx5rzjqXgXZDXhPcfffdzdk77rhjL+Nc0YMPPtic\nfd3rXjeRGXbjSikAAAAAulNKAQAAANCdUgoAAACA7pRSAAAAAHSnlAIAAACgO6UUAAAAAN0ppQAA\nAADoTikFAAAAQHdKKQAAAAC6U0oBAAAA0N2hnnd25MiR5uzZs2cnMsOFCxcG5Z944onm7O233z50\nHGDGbGxsNGcXFhYmOAlcu1ZXV5uzDz744ERmePTRR5uzc3NzE5kBmK4h5z6PPfZYc/auu+5qzj7w\nwAPN2fvvv785C9e6IX93Hz58uDn7+te/vjk75LxjiKWlpYkcdzeulAIAAACgO6UUAAAAAN0ppQAA\nAADoTikFAAAAQHdKKQAAAAC6U0oBAAAA0J1SCgAAAIDulFIAAAAAdKeUAgAAAKA7pRQAAAAA3R3q\neWfHjh1rzj7xxBPN2UceeWQi2aFOnTo1sWMDAG2Wl5ebs+vr683Zc+fONWdf+9rXNmdPnjzZnB3y\n2JaWlpqzQJt77rmnOXvixInm7IULF5qz73jHO5qzt99+e3MWaLe4uNic3draas5ubGxMZIY77rij\nOTs3N9ecHQdXSgEAAADQnVIKAAAAgO6UUgAAAAB0p5QCAAAAoDulFAAAAADdKaUAAAAA6E4pBQAA\nAEB3SikAAAAAulNKAQAAANCdUgoAAACA7g71vLNjx441Zx944IHm7KlTp5qzN998c3M2Sc6ePTso\nD0zP3Nxcc/bkyZPN2be+9a3N2fX19ebs8vJycxZot7Cw0Jzd2NiYSHZ1dbU5O2THzM/PN2eXlpaa\ns0CbI0eONGfvvPPOicxw++23N2cfeuihicwATMaQ85mLFy82Z/fzeYcrpQAAAADoTikFAAAAQHdK\nKQAAAAC6U0oBAAAA0J1SCgAAAIDulFIAAAAAdKeUAgAAAKA7pRQAAAAA3SmlAAAAAOhOKQUAAABA\nd6XWOu0ZAAAAALjGuFIKAAAAgO6UUgAAAAB0p5QCAAAAoDulFAAAAADdKaUAAAAA6E4pBQAAAEB3\nSikAAAAAulNKAQAAANCdUgoAAACA7pRSAAAAAHSnlAIAAACgO6UUAAAAAN0ppQAAAADoTikFAAAA\nQHdKKQAAAAC6U0oBAAAA0J1SCgAAAIDulFIAAAAAdKeUAgAAAKA7pRQAAAAA3SmlAAAAAOjuwJdS\npZS3lVK+ZdxZYDbZGcBQ9gYwhJ0BDHUt741Sa+1/p6V8LMmlO35BkqeT/PX2r++stb6p+1BjVEqZ\nT/K+JB/f8dv311p/cCoDwQE36zsjSUopz0/yn5PcluQ5Sc7VWl893ang4Jr1vVFK+aYkP7njt56V\n5HlJbqq1vns6U8HBNes7I0lKKbcnuS/JS5K8P8m/r7W+dbpTwcF1jeyNb09yKskXJfmNJP+q1vqB\nsd7HNEqpzxiglPNJvq3W+mvP8GeHaq1/NYWxrsqOUurZddpPMMyYWdwZSVJK+W8ZnVT+myRPJVlw\nYgnjMat7Y6dSyh1J/mOt9e9MexY46GZxZ5RSXpLR+cnX1VrfXkr52iSPJLmh1vrh6U4HB9+M7o3F\nJG9Ospjkj5I8mOTLa62L47yfffX2vVLKYinlT0sp31NK+UCSh0spc6WUXy6lfLCU8lQp5Ze2l+ql\n26yXUr5t++fLpZTfKKX8yHb2faWUr95j9mgp5Z2llI+UUt5RSvnxUsobBz6kffX8wqyZlZ1RSvl7\nSf55Rv+i8ud1RCEFEzAre+MZLCd5wx5vC+xihnbGlybZqrW+PUlqrW/L6F0dL73qJwn4DDO0N16T\n5JFa63trrZ9M8gNJvrKUcnQMT9On7cfS5AuTHElyfZK7Mprx4e1fX5/kE0l+bEe+5m8umUuSVyb5\ngyQvTvLD27fdS/ZnkvxWkuuSrCb55p23LaWcK6V8wxUey5OllPeXUv5rKeXFV8gCezMLO+OVSZ5M\n8v2llA+VUn63lPL1V3jcwN7Nwt7IjtwNSf5JlFIwKbOwM/5nkveWUl5TSnl2KWUpo7ca/e7nfujA\nHs3C3qhJyo5fX+qPXr5Lfk/2Yyn1qST31lo/WWt9utb6VK310e2ffyzJDyX5XJ+z8mSt9eHtt829\nIckXl1K+YEi2lHJ9kpuTfF+t9a9qre9K8ovZ8R+k1nq81vqzuxz3Q9u3vz7JTUlelOSnm58BYIhZ\n2BlfmtFy30ryxUn+dZLXl9EVVMD4zcLe2Olbk7yz1vpkQxYY7sDvjFrrp5K8McmbMiqjfjrJXbXW\nTwx5IoBmB35vJPnVJLeVUv5BKeV5Sb4vo6Lq+e1Pw5Xtx1LqQ7XWv7z0i1LK80spD5VSNkspF5M8\nnuRwKaXscvs/u/STWutfbP/0hQOzX5LkqVrr0zuy7299ALXWj9daf6fW+qla6wczOsH8qlLKC1qP\nATQ78Dsjo38p+WSS/7T9F8Y7k/x6kq8acAyg3SzsjZ2+Ncnr93hb4MoO/M4opZxI8kCSV9dan5PR\nyfDDpZTjrccABjnwe6PW+t8zurrq55Oc3/7x0SR/2nqMFvuxlLr8g8G/O8nLkryy1no4owVa8pmX\nkY3bB5Jct90GXnL9GI67H59vOOhmYWdcunT+8hl9owSYjFnYG0mSUspXZHSF5c+NazDgs8zCzljI\n6IrK30mSWusTSX47yYnxjQjsMAt7I7XWn6i1vqzW+kVJfiHJoSTvGeOMB6IkeWFGVxFcLKVcl+Te\nSd/h9uXvTyRZLaU8p5Tyqow+5KvpBLGU8spSyt8tpTxr+7OkfjTJr9daPzq5qYFtB25nZPQvJX+S\n5N+VUg5tn2QuJnn7JOYFPstB3BuX3JHk52qtHx/3jMCuDuLO+B9J/vGlK6NKKTdm9Fl05yYxL/BZ\nDtzeKKV8Xinl5WXk+iQ/leR0rfXiOOfcj6XU5U/Q6STPS/LhJL+Z5FeeIbPztpf/2V6z35TkVUn+\nPKNPmX9zkp2X372nlPKNuxz72PacH0nyvzL6n2+3LHB1DvzOqKNvEXsyyddm9LlSDyX5llrrH+4y\nC3B1Dvze2P7z5ya5Ld66B5N24HfG9kcD3Jfk50opH8no6sofrLU+tssswNU58HsjyXMz+vy5j2Z0\nZeW7knzvLtk9K6PPwuJKSilvTvL7tdb7pj0LsP/ZGcBQ9gYwhJ0BDLUf98Z+vFJqXyil3FxKeen2\nW/C+JsnXJVmb9lzA/mRnAEPZG8AQdgYw1EHYG4emPcA+dumDvF6c0SfUf0et1Xuugd3YGcBQ9gYw\nhJ0BDLXv94a37wEAAADQnbfvAQAAANCdUgoAAACA7pRSAAAAAHSnlAIAAACgO6UUAAAAAN0ppQAA\nAADoTikFAAAAQHdKKQAAAAC6U0oBAAAA0J1SCgAAAIDulFIAAAAAdKeUAgAAAKA7pRQAAAAA3Sml\nAAAAAOhOKQUAAABAd0opAAAAALo7NIZj1DEc47MsLi42Z+fn55uzZ86cGTzLPlCmPQCM2UT2xhBD\ndszW1lZzdmNjYw/TjJ2dwSyayN44ffp0c3bILlhbW2vOnjt3rjl7+PDh5uzm5mZzdm5uzt5g1kxk\nZ6ysrDRnh+yB5eXlicwwNzfXnB3IzmAWTWRvLC0tNWeHvNZYX1/fwzRT9Yx7w5VSAAAAAHSnlAIA\nAACgO6UUAAAAAN0ppQAAAADoTikFAAAAQHdKKQAAAAC6U0oBAAAA0J1SCgAAAIDuSq31ao9x1Qd4\nJvPz883ZJ598chIjJEluuOGG5uzm5uakxiiTOjBMyUT2xtraWnP2ta99bXP23nvvbc6urq42ZyfI\nzmAWTWRvnD59ehKHzcLCwkRm2Nraas6ur683Z2NvMHsmsjMWFxebs5M6NxhynjRwDwxhZzCLmvfG\nkK/vo0eP7mWWsTp+/HhzdmNjY1JjPOPecKUUAAAAAN0ppQAAAADoTikFAAAAQHdKKQAAAAC6U0oB\nAAAA0J1SCgAAAIDulFIAAAAAdKeUAgAAAKA7pRQAAAAA3SmlAAAAAOju0LQH2M3c3Fxz9sknn2zO\nHj58eNAci4uLzdmtra3m7JDHB7RZXV2dyHGXlpYmclxg+lZWViZy3CH7aHNzszm7vr4+eBZgfBYW\nFpqz8/PzzdkzZ840Z4ecRwzZGUPOe+BaN+Tcf4hXv/rVzdkhO2Y/v35wpRQAAAAA3SmlAAAAAOhO\nKQUAAABAd0opAAAAALpTSgEAAADQnVIKAAAAgO6UUgAAAAB0p5QCAAAAoDulFAAAAADdKaUAAAAA\n6O7QtAfYzfz8fHP23LlzzdmLFy8OmmNhYaE5Ozc3N+jYwHhtbW01Z48fP96cHbIHgOlbX1+fSHaI\n06dPT+S4a2trzdnl5eWJzADXsiFfVzfeeGNzdnNzszk75JxjyDkV0G5SX1tD/p5fWlpqzg45T+rN\nlVIAAAAAdKeUAgAAAKA7pRQAAAAA3SmlAAAAAOhOKQUAAABAd0opAAAAALpTSgEAAADQnVIKAAAA\ngO6UUgAAAAB0p5QCAAAAoDulFAAAAADdHZr2ALtZW1trzq6vrzdnNzY2Bs1x9913D8q3WllZmchx\n4Vq2tbXVnJ2fn2/Onj59ujm7tLQ0kRmAdkO+toa8LhjyemOIIa95FhcXJzID0GbIa40hHn/88ebs\n+fPnm7Nea8BkzM3NNWePHz/enD1y5Ehz9ru+67uas0Ne72xubjZnx7FjXCkFAAAAQHdKKQAAAAC6\nU0oBAAAA0J1SCgAAAIDulFIAAAAAdKeUAgAAAKA7pRQAAAAA3SmlAAAAAOhOKQUAAABAd0opAAAA\nALo7NO0BxmFxcXHaIyRJNjc3pz0CXNPm5+ebs48//nhzdmtrqzl79913N2ff/e53N2cXFhaas3Ct\nG7IL1tbWmrOllObso48+2pzdL69j4Fq1sbHRnL3llluas/fee29zdsh5xNLSUnN2yI4bsjuBdkN2\nzJDspM4PVlZWmrNDdsxuXCkFAAAAQHdKKQAAAAC6U0oBAAAA0J1SCgAAAIDulFIAAAAAdKeUAgAA\nAKA7pRQAAAAA3SmlAAAAAOhOKQUAAABAd0opAAAAALo7NO0BdrO2ttacnZuba86urq7uYZo2S0tL\nEzs2cGXLy8vN2bvvvrs5Oz8/35zd3Nxszg7ZcwsLC81ZoN3Kykpz9vDhw83ZxcXFPUwDTMOQv+eH\n7IEh+2XI64cbb7yxOXvmzJnm7CTPk4A2Q17zD9kxQ3bBkHOUcXClFAAAAADdKaUAAAAA6E4pBQAA\nAEB3SikAAAAAulNKAQAAANCdUgoAAACA7pRSAAAAAHSnlAIAAACgO6UUAAAAAN0ppQAAAADo7tC0\nB9jN+vp6c/bBBx+c2Bx33HFHc3ZxcXFicwBXtry83Jzd3Nxszp45c6Y5O2QPLC0tNWeByRjyemPI\nLpibmxs+DDAVQ75eh/w9f+TIkebs4cOHm7MnT55szq6srDRngckY8nW4sbHRnN3a2mrODnm9s7Cw\n0JwdB1dKAQAAANCdUgoAAACA7pRSAAAAAHSnlAIAAACgO6UUAAAAAN0ppQAAAADoTikFAAAAQHdK\nKQAAAAC6U0oBAAAA0J1SCgAAAIDuSq112jMAAAAAcI1xpRQAAAAA3SmlAAAAAOhOKQUAAABAd0op\nAAAAALpTSgEAAADQnVIKAAAAgO6UUgAAAAB0p5QCAAAAoDulFAAAAADdKaUAAAAA6E4pBQAAAEB3\nSikAAAAAulNKAQAAANCdUgoAAACA7pRSAAAAAHSnlAIAAACgO6UUAAAAAN0ppQAAAADoTikFAAAA\nQHdKKQAAAAC6+/+V0udzc/LJEgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11229a790>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "images_and_labels = zip( digits.images, digits.target )\n",
    "plt.figure(figsize=(20,4))\n",
    "for idx,(img,label) in enumerate(images_and_labels[:10]):\n",
    "    plt.subplot(2,5,idx+1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(img,cmap=plt.cm.gray_r,interpolation='nearest')\n",
    "    plt.title(\"Training: %i\\n\" % label)\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To do classification using one of our familiar algorithms, we might run the following code (as well as splitting the data into training and test sets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Load dataset, split into train and test set\n",
    "# Using half the data for test to make it harder\n",
    "\n",
    "X_train, X_test, y_train, y_test = cross_validation.train_test_split(digits.data,digits.target,test_size=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Going to scale all my data first\n",
    "from sklearn import preprocessing\n",
    "\n",
    "scaler = preprocessing.StandardScaler(with_mean=True,with_std=True)\n",
    "scaler.fit(X_train)\n",
    "\n",
    "X_train = scaler.transform(X_train)\n",
    "\n",
    "X_test  = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_neighbors=3, p=2, weights='uniform')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import neighbors\n",
    "\n",
    "model_knn = neighbors.KNeighborsClassifier(n_neighbors=3)\n",
    "model_knn.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.961068\n"
     ]
    }
   ],
   "source": [
    "score = metrics.accuracy_score(y_test, model_knn.predict(X_test))\n",
    "print('Accuracy: {0:f}'.format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TensorFlowClassifier\n",
    "Now, let's ease into SKFlow/Tensorflow by using a regular linear classifier\n",
    "\n",
    "(Network diagram on the board)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step #1, avg. loss: 2.61099\n",
      "Step #1001, epoch #34, avg. loss: 0.18512\n",
      "Step #2001, epoch #69, avg. loss: 0.05825\n",
      "Step #3001, epoch #103, avg. loss: 0.03986\n",
      "Step #4001, epoch #137, avg. loss: 0.02909\n",
      "Step #5001, epoch #172, avg. loss: 0.02446\n",
      "Step #6001, epoch #206, avg. loss: 0.02029\n",
      "Step #7001, epoch #241, avg. loss: 0.01742\n",
      "Step #8001, epoch #275, avg. loss: 0.01524\n",
      "Step #9001, epoch #310, avg. loss: 0.01381\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TensorFlowLinearClassifier(batch_size=32, class_weight=None,\n",
       "              continue_training=False, early_stopping_rounds=None,\n",
       "              keep_checkpoint_every_n_hours=10000, learning_rate=0.1,\n",
       "              max_to_keep=5, n_classes=10, num_cores=4, optimizer='SGD',\n",
       "              steps=10000, tf_master='', tf_random_seed=42, verbose=1)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_tfl = skflow.TensorFlowClassifier(n_classes=10,steps=10000)\n",
    "model_tfl.fit(X_train,y_train) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.963293\n"
     ]
    }
   ],
   "source": [
    "score = metrics.accuracy_score(y_test, model_tfl.predict(X_test))\n",
    "print('Accuracy: {0:f}'.format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TensorFlowDNNClassifier\n",
    "Great, let's see what the same code looks like using a Deep Neural Network  with 2 hidden layers\n",
    "\n",
    "(Network diagram on the board)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step #1, avg. loss: 2.91747\n",
      "Step #1001, epoch #34, avg. loss: 0.17692\n",
      "Step #2001, epoch #69, avg. loss: 0.00554\n",
      "Step #3001, epoch #103, avg. loss: 0.00220\n",
      "Step #4001, epoch #137, avg. loss: 0.00128\n",
      "Step #5001, epoch #172, avg. loss: 0.00091\n",
      "Step #6001, epoch #206, avg. loss: 0.00067\n",
      "Step #7001, epoch #241, avg. loss: 0.00054\n",
      "Step #8001, epoch #275, avg. loss: 0.00044\n",
      "Step #9001, epoch #310, avg. loss: 0.00038\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TensorFlowDNNClassifier(batch_size=32, class_weight=None,\n",
       "            continue_training=False, early_stopping_rounds=None,\n",
       "            hidden_units=[20, 15], keep_checkpoint_every_n_hours=10000,\n",
       "            learning_rate=0.1, max_to_keep=5, n_classes=10, num_cores=4,\n",
       "            optimizer='SGD', steps=10000, tf_master='', tf_random_seed=42,\n",
       "            verbose=1)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_dnn = skflow.TensorFlowDNNClassifier(hidden_units=[20,15],\n",
    "                                           n_classes=10,\n",
    "                                           steps=10000)\n",
    "model_dnn.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 1.000000\n"
     ]
    }
   ],
   "source": [
    "score = metrics.accuracy_score(y_train, model_dnn.predict(X_train))\n",
    "print('Accuracy: {0:f}'.format(score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.944383\n"
     ]
    }
   ],
   "source": [
    "score = metrics.accuracy_score(y_test, model_dnn.predict(X_test))\n",
    "print('Accuracy: {0:f}'.format(score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step #1, avg. loss: 2.91747\n",
      "Step #1001, epoch #34, avg. loss: 0.17692\n",
      "Step #2001, epoch #69, avg. loss: 0.00554\n",
      "Step #3001, epoch #103, avg. loss: 0.00220\n",
      "Step #4001, epoch #137, avg. loss: 0.00128\n",
      "Step #5001, epoch #172, avg. loss: 0.00091\n",
      "Step #6001, epoch #206, avg. loss: 0.00067\n",
      "Step #7001, epoch #241, avg. loss: 0.00054\n",
      "Step #8001, epoch #275, avg. loss: 0.00044\n",
      "Step #9001, epoch #310, avg. loss: 0.00038\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TensorFlowDNNClassifier(batch_size=32, class_weight=None,\n",
       "            continue_training=False, early_stopping_rounds=None,\n",
       "            hidden_units=[20, 15], keep_checkpoint_every_n_hours=10000,\n",
       "            learning_rate=0.1, max_to_keep=5, n_classes=10, num_cores=4,\n",
       "            optimizer='SGD', steps=10000, tf_master='', tf_random_seed=42,\n",
       "            verbose=1)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In TensorFlow, when we run and save the fitting to a directory\n",
    "# We can call in a terminal\n",
    "#   tensorboard --logdir=<THE_DIRECTORY>\n",
    "\n",
    "model_dnn.fit(X_train,y_train,logdir=\"/tmp/blah\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see a few of the examples where we were wrong with the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABZgAAACmCAYAAABX5eq+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGVVJREFUeJzt3XuQnXWZJ/Dnl3QSAiRAgpgUSoJcgoJEUBgRBdTCCSyL\nJevCoozAZNfVWaAsLZypUaYo3HEYKSnxVjIqi0A5DuOwI1jImnLXXSQIIytQgomyhssgkAC5kAt0\nLu/+cZqdXiWd9PM0dKf786lKVS79Pd+3+zz9nvc8fdLduq4LAAAAAAAYrkmjfQAAAAAAAOyaLJgB\nAAAAAEixYAYAAAAAIMWCGQAAAACAFAtmAAAAAABSLJgBAAAAAEixYB7DWmvntdZuH+3jYNdjdsgw\nN1SYH7LMDhnmhiyzQ5bZIcvskLUrzc6YWzC31ta31p4b+LWttbZx0J/PTtzej1tri4eZ+dettV8M\ndN7RWnv9EG97bWvthYG3faa19sPW2oLhHmdVa+3Q1tr3WmsrB47jttbaoa/0cYwms5PTWps9cKxP\nt9bWtNaWttbe9kofx2gxN3WttQ8NfOyG9X6PB+Ynr7X2N621Za21ra21c0fjGEaT2cmZ6Nc75ian\ntfaOQR+nwR+/973SxzJazE7ORD/nRJidkdAm6LWy2ckbznGPR2YnZ7Svd8bcgrnruj27rpvRdd2M\niHgkIk578c9d1/1t5iaH88attUMi4oaI+HBE7BURt0TEza21yUPc/l8PHO9rImJlRFz7ErfbWmtt\nOMcyTHtFxD9GxKER8eqIuDsivvcy9o05ZidtfUT8cUTs13Xd3hHx1xFxS2ttzJ0fXg7mpqa1tk9E\n/HlE/CKG+b6PB+an5N6I+JOI+N9hdszOzpvQ1zvmJqfrutsHfZxmRMRp0bv+ue3l6hxrzE7ahD7n\nRJidqol8rWx2chLHPe6YnZzRvt7ZZRZIrbVJrbU/a6091HqvtPy7gZN1tNZ2a63dMPD3q1trd7fW\n9mut/WVEvCMivjywuf/iTlT9YUTc3nXd0q7rtkVv2bZ/RJy4o2DXdZsi4m8j4oiB4/pxa+0/t9bu\niIgNEXFga+2w1tqSga9qLGut/dtB7+Ps1trNrbW1rbW7IuKgnf34dF33T13X/Zeu69Z0XbclIr4Q\nEQte/BhNZGZnh90vdF23vOu6bQNL5W0RsU9EzNrZ2xiPzM1O+6uIuCoinklkxy3zs2Nd132167r/\nHhHPDyc33pmdHXa73nkJ5mbYzouIvx84pgnN7Oyw2zlnO8zOTnOt/DvMzst33OOd2Rm28+IVvN7Z\nZRbMEXFhRJweESdExNyIWB0RXxn4t3MjYmb0vlIwKyL+Y0Rs6rruUxFxe0T8p4EN/kUREa21W1pr\nn9xOTxcRg7+iMGngz4cPcWxt4Hb3jIgPRu/VWC86JyL+fUTsGb0HlSXR+0rIqyLi30XEV9u/vNT+\nKxGxMSLmRO8VpefHoK+07OC4f9cJEfFE13Wrd/LtxzOzsxOz01q7PyI2Re9VGV/vuu7pod5+AjA3\nO5ib1tqxEXF0RHxtiGOdqMzP8B6z+Bdmx/VOhrnZyblpre0REf8mIr61o7edIMyOc06W2XGtnGV2\nRv64JwqzM5avd7quG7O/ImJFRLxr4PcPvvj7gT/PjYj+iJgcvQ/4HRHxxpe4jf8REYuH0bkgei8h\nPzEipkbEJRGxNSL+dDtvf230lnKrI+KJ6P0XqgMHdV866G3Pioj/9Tv5qyPiLwbej/6IOHTQv/1l\n9L5qMtyP22si4p8j4qzRvg/Nzi43O1Ojd4L70Gjfh+ZmbM/NQP6fIuLYQd1/PNr3ofnZNebnd27z\n9ol6zjE75dmZ0Nc75iY9N38UEf9ntO8/s7NLzs6EPueYHdfKZucVm51hHfd4/2V2dp3rnb7YdcyP\niP/aWts26O+2RMR+EXF9RLw2Ir7TWts7el8J+FTX+29MEcP4fitd1y1vvR829OXoDesN0Rvif95e\nJCKu6LruL7bz748N+v28iPiD1trgr3j3RcR1EbHvwO8Hv/2jO3vcL2qtvSoifhgRX+m67u+Gmx+n\n5ofZ2Sld1/VH72PxYGvt3q7r7s/czjgxP8zNUP4kIu7vuu7uQX/3sn8fu13I/DA/5MwPs7NDrnd+\nz/wwNzvr3IHbpGd+mJ0dcs55SfPD7AzFtfL2zQ+zM5LHPZHMD7Ozs17x651dacH8aESc33Xdndv5\n98si4rLW2ryIuDUilkfENZH4Rvpd1/1DRPxDRMTAYC6O3lcft2eoB4rB/Y9GxP/suu49v3cDvW8W\nviUiDojescfA73da633vmR9GxD92XfdXw8mOc2Zn+KZExIERMZEXzOZmaO+KiBNba6cO/HlWRBzV\nWlvYDfy3ownO/JBldnbA9c5LMjc7obX22ui9Guk/DDc7jpmdHXDO2S6zMzTXyttndnYgcdwThdnZ\nCaN1vbMrfQ/mr0XEZ1trB0T0vorcWjt94PcntdbeOHBnPBcRm6P38vWIiKdimN8Uu7X25tba5IGv\nVP9NRHyv67pfbe/Nd3Rzg37//Yg4tLV2TmttysCvY1prh3VdtzUiboqIS1tr01trb4jeVxx26hOh\ntTYzIv5bRPyk67o/35nMBGJ2hj7mP2itvb21NnUg/6fR+wrgXTuTH8fMzdDOi4jDImJhRLwpIn4W\nEZdGxKd2Mj/emZ8dH/eU1tpu0bsWmdp6P5jDK3vMzo6O2fXOSzM3O+ePIuKOrutWDDM3npmdoY/Z\nOWf7zM7QzgvXyttjdkb2uCcSs7NzRuV6Z1daMF8VETdHxA9ba+si4s6IOHbg3+ZExN9HxNrovWz9\nx9F7efyLufe31p5trX0hIqK1dmtr7c+G6PpC9L53yrLofQPuobb+XQx9Z/+/f+u6bn1EvCd63+P2\n8eh9b5a/it73dImIuCB63/T7yeh9leWawTe0g+N+X0S8JSLOb72fjPlca21da+01QxzbRGF2hj7u\nadH7rx9PR++/fCyKiH/Vdd2TQxzbRGBuhjjuruvWdl23cuDXU9H7XlHruq57bohjm0jMz46Pe0n0\nfoDFW6N30bYxej/heaIzO653MszNjo87oveEyw/3+/+ZHeecLLPjWjnL7IzscU8kZmcMX++0rhv2\nK8UBAAAAAGCXegUzAAAAAABjiAUzAAAAAAApFswAAAAAAKRYMAMAAAAAkNJXvYFf//rX6Z8SOG3a\ntFL3+vXr09k1a9aUuvfdd990dvbs2aXuTZs2pbP9/f2l7te97nWtdAODfPazn03Pzvz580vd06dP\nT2evv/76Hb/REObMmZPOfvrTny5133nnnens448/Xuq+6KKLRmx2Fi9enJ6dyuduRMR1112Xzl5+\n+eWl7nPPPTedveaaa3b8RkN46KGH0tnq+farX/3qiM3Oxz/+8fTsvOY1tR8Yv3HjxnT2ySefLHU/\n++yz6exJJ51U6q487qxcubLUfdlll43Y7Fx33XXp2Zk1a1ap+xe/+EU6u2LFilJ3Ze6PP/74UveW\nLVvS2dWrV5e6zzrrrBGbnc9//vPp2dlrr71K3XfddVc629dXe5pQOeeddtpppe4NGzaks88991yp\n+8ILLxyx2bnxxhvTs3PFFVeUut/4xjems3vvvXepe+bMmens6aefXupet25dOrt27dpS93vf+94R\nm50VK1aM2nP0yv1fOW9ERDzzzDPpbPU675577klnL7744lL3XXfdNWKzc84556Rn51WvelWpu/J8\no/L8PqL2uV895x1wwAHpbGXmIyK++c1vjtjsfOlLX0rPzkUXXVTqfs973pPOVndL8+bNS2ePO+64\nUndlR1PZp0ZEfOQjH/m92fEKZgAAAAAAUiyYAQAAAABIsWAGAAAAACDFghkAAAAAgBQLZgAAAAAA\nUiyYAQAAAABIsWAGAAAAACDFghkAAAAAgBQLZgAAAAAAUiyYAQAAAABIsWAGAAAAACDFghkAAAAA\ngBQLZgAAAAAAUiyYAQAAAABI6aveQGstnZ06dWqpe+7cuenss88+W+resGFDOtvf31/qnjVrVik/\nVlTu/0ceeaTU/fWvfz2d/e53v1vqXrRoUTp7ySWXlLp33333Un6s6OvLn7puu+22Uvdee+2Vzp58\n8sml7rvuuiudvfLKK0vdhx56aDo7Z86cUvdIqpx3qu/Hli1b0tmlS5eWun/605+mszNnzix1H3PM\nMaX8WFE571QftxcsWJDOVuYuImK//fZLZ6+77rpS93nnnVfKjxWTJ09OZ2fMmFHq3meffdLZt73t\nbaXub3/72+ls5flFRO3zdSypvB8/+9nPSt3HH398Olt93HjyySfT2UmTaq+fmjZtWik/Vozmc/Qp\nU6aks4899lip+6ijjkpnq4+X9957bzp79913l7pHUuW8c/XVV5e6K495n/zkJ0vdleeIN910U6n7\n/e9/fzo7ffr0UvdIqlzvvOUtbyl1H3vssensu971rlL3tddem84efPDBpe4XXnihlB9pXsEMAAAA\nAECKBTMAAAAAACkWzAAAAAAApFgwAwAAAACQYsEMAAAAAECKBTMAAAAAACkWzAAAAAAApFgwAwAA\nAACQYsEMAAAAAECKBTMAAAAAACkWzAAAAAAApFgwAwAAAACQYsEMAAAAAECKBTMAAAAAACl9o1k+\nefLkUn6fffZJZ+fNm1fqrlixYkUpP3v27BE6kl3XQw89VMqfdNJJ6Wx/f3+pe9WqVens3LlzS90P\nPPBAOtt1Xal7rLj//vtL+YULF6azP/nJT0rdV155ZTq7fv36UvfRRx+dzv72t78tdY8Vu+22Wyl/\nxhlnpLOf+9znSt2rV69OZ6dPn17q7uvLX2ps27at1D2SKscyderUUvfpp5+ezk6bNq3U/aY3vSmd\nvf3220vdN954Yzp7wgknlLpHUuXxc8qUKaXus846K5298847S90bN25MZ6+//vpS980335zOXnXV\nVaXukVSZnVtuuaXUfdxxx6WzzzzzTKn7sssuS2e3bt1a6p40Kf/6q/FyrVz9GC5btmzUuleuXJnO\nPv/886XuSv6UU04pdY+kyhxXrvkiIrZs2ZLO3nrrraXupUuXprOvfe1rS90LFixIZx955JFS90iq\nXCtfcsklpe7Kc51rrrmm1F353L/ppptK3bNmzUpnjzzyyFL3S/EKZgAAAAAAUiyYAQAAAABIsWAG\nAAAAACDFghkAAAAAgBQLZgAAAAAAUiyYAQAAAABIsWAGAAAAACDFghkAAAAAgBQLZgAAAAAAUiyY\nAQAAAABIsWAGAAAAACDFghkAAAAAgBQLZgAAAAAAUiyYAQAAAABIsWAGAAAAACClbzTLW2ul/ObN\nm9PZOXPmlLqfeOKJdHbr1q2l7kmT8l8X2LZtW6l7JHVdl84uXLiw1L1p06Z09uyzzy51v/71r09n\nzz333FL3qaeems729/eXuseKefPmlfIbNmxIZy+55JJS95QpU9LZD33oQ6PWXT3njaTKeaevr/aQ\nuWrVqnT2oosuKnVfccUV6ezcuXNL3VOnTk1nK4/zY8nkyZNL+crH4d3vfnepe9q0aens6tWrS937\n779/Ortly5ZS91hRueaLiPjGN76Rzq5cubLUfdBBB6Wzlfs+ovY5N5Yesyrmz59fyu+1117p7M03\n31zqPvzww9PZBQsWlLqXL1+ezo6X2XnwwQdL+enTp6ez1XP3ww8/nM5u3Lix1F15jnfmmWeWuseK\nSy+9tJRfsmRJOnvPPfeUumfPnp3Onn/++aXuivFy3qle74zmrqJyzXL66aeXuteuXZvOPv7446Xu\nl+IVzAAAAAAApFgwAwAAAACQYsEMAAAAAECKBTMAAAAAACkWzAAAAAAApFgwAwAAAACQYsEMAAAA\nAECKBTMAAAAAACkWzAAAAAAApFgwAwAAAACQYsEMAAAAAECKBTMAAAAAACkWzAAAAAAApFgwAwAA\nAACQ0le9ga7r0tlt27aVujdv3pzO9vXV3vXVq1enszNnzix1V469cn+NtMr9d8IJJ5S6jznmmHT2\nN7/5Tan75JNPTmc/+tGPlroffPDBdPZ973tfqXskbdmyJZ294IILSt0rV65MZ2+99dZS91FHHZXO\nnnTSSaXuyuxU7q+RVjnv9Pf3l7orjxvLli0rdbfW0tl999231L3PPvuks5X7a6RVrlmqnwPr169P\nZ3ffffdS96pVq9LZvffeu9Q9d+7cdLZ6jTmSKseydevWUveiRYvS2R/84Ael7sMOOyydfcc73lHq\nfsMb3pDOLl++vNQ9kir3f3V27rvvvnT2nnvuKXV/7GMfS2f33HPPUnclX/2Yj6TKc77p06eXuiv5\n6mPWpk2b0tnJkyeXug855JB0tnqNOZIqc1y93vnwhz+czl5xxRWl7sr9d+KJJ5a677333nR2vJx3\nqu/HgQcemM4uXbq01P32t789na1c60bUzh0vx27QK5gBAAAAAEixYAYAAAAAIMWCGQAAAACAFAtm\nAAAAAABSLJgBAAAAAEixYAYAAAAAIMWCGQAAAACAFAtmAAAAAABSLJgBAAAAAEixYAYAAAAAIMWC\nGQAAAACAFAtmAAAAAABSLJgBAAAAAEixYAYAAAAAIMWCGQAAAACAlL7qDbTW0tmtW7eWuiv5p556\nqtT9zDPPpLP7779/qfvpp59OZ5csWVLqPuSQQ0r5waZOnZrOrlu3rtR99dVXp7OXX355qXvmzJnp\n7Le+9a1S97Jly9LZ6dOnl7pHUl9f/tT1/PPPl7oPPvjgdPaJJ54odX/gAx9IZ7ds2VLq3rZtWzpb\n+VwfaZVj2bBhQ6m7ct66/fbbS90rVqxIZyszH1F7v3fbbbdS90iaNCn/NfnqeWfjxo3pbOV6JSLi\n2muvTWdnzJhR6n7zm9+czj722GOl7pE0efLkdLa/v7/UXblWfvzxx0vdixYtSmcrjznVfOUaY6RV\nZqd63rntttvS2X333bfUXTn2L37xi6Xu++67L5097bTTSt0jqfIc/aCDDip177nnnqV8xQMPPJDO\nVp6jRdTe7zVr1pS6R1LlHFi5Xomo3X933nlnqfuUU05JZzdv3lzqHi/Psyrnner1TuW6r/ocbzR3\nomPtescrmAEAAAAASLFgBgAAAAAgxYIZAAAAAIAUC2YAAAAAAFIsmAEAAAAASLFgBgAAAAAgxYIZ\nAAAAAIAUC2YAAAAAAFIsmAEAAAAASLFgBgAAAAAgxYIZAAAAAIAUC2YAAAAAAFIsmAEAAAAASLFg\nBgAAAAAgpW80y7du3VrKv/DCC+nsI488UuqeNCm/m3/qqadK3Y8++mg6e+GFF5a6L7jgglJ+sK7r\n0tm1a9eWuj/zmc+ks0888USp+5xzzkln77777lL3UUcdlc7usccepe6xYuPGjaX8kiVL0tljjjmm\n1L1o0aJ09umnny51b9u2LZ2dNm1aqXskVc47zz33XKl79uzZ6exuu+1W6p4+fXo6W3m8i6g91u++\n++6l7rGiet751a9+lc5WHzcefvjhdLavr3aZuXjx4nT21FNPLXWfffbZpfxglfPO888/X+quzM6U\nKVNK3UcffXQ6u2bNmlJ35TGr+n6PFZs2bSrlf/nLX6azM2bMKHVfddVV6Wz1vHPkkUems2Ppeqei\n+hy98jm0bt26UnflnHnEEUeMWvfkyZNL3SOp8phVvd6pPG5Ur5Xf+ta3lvIVlc+58XLeqez2IiLu\nuOOOdPad73xnqfvYY49NZ6t7rbF2veMVzAAAAAAApFgwAwAAAACQYsEMAAAAAECKBTMAAAAAACkW\nzAAAAAAApFgwAwAAAACQYsEMAAAAAECKBTMAAAAAACkWzAAAAAAApFgwAwAAAACQYsEMAAAAAECK\nBTMAAAAAACkWzAAAAAAApFgwAwAAAACQYsEMAAAAAEBK32iWb926tZRfs2ZNOjtt2rRS95w5c9LZ\n9evXl7r7+vJ329KlS0vdI6nrunR248aNpe6vfe1r6eyPfvSjUvcHP/jBdPbwww8vdZ955pnpbH9/\nf6l7JFVmZ/PmzaXu73//++nsJz7xiVL3oYcems4uX7681L127dp0tnJ/jbTKsWzatKnUvWrVqnS2\n+ng5f/78dPbiiy8udS9evDidHUuzU1E9f/785z9PZ1euXFnqXrhwYTpbfb9vuOGGdPaQQw4pdY+k\n0XzMeuCBB9LZRYsWlboPOOCAdPbhhx8udVevtceKyuxUP/8qj/tHHHFEqbvy+Ttr1qxS9x577JHO\nPvvss6XusaJ6zVGZvcrcRUTMmzevlK+ovN9j6XqnciwvvPBCqfvGG29MZ88777xS98knn5zOfuc7\n3yl1r169Op0dS7NT8eUvf7mUnzJlSjp7xhlnlLpf/epXp7OPPfZYqXvDhg3p7MsxO17BDAAAAABA\nigUzAAAAAAApFswAAAAAAKRYMAMAAAAAkGLBDAAAAABAigUzAAAAAAApFswAAAAAAKRYMAMAAAAA\nkGLBDAAAAABAigUzAAAAAAApFswAAAAAAKRYMAMAAAAAkGLBDAAAAABAigUzAAAAAAApreu60T4G\nAAAAAAB2QV7BDAAAAABAigUzAAAAAAApFswAAAAAAKRYMAMAAAAAkGLBDAAAAABAigUzAAAAAAAp\nFswAAAAAAKRYMAMAAAAAkGLBDAAAAABAigUzAAAAAAApFswAAAAAAKRYMAMAAAAAkGLBDAAAAABA\nigUzAAAAAAApFswAAAAAAKRYMAMAAAAAkGLBDAAAAABAigUzAAAAAAApFswAAAAAAKRYMAMAAAAA\nkPJ/AdOdhxGvPmBzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x114b38510>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "plt.figure(figsize=(20,4))\n",
    "cnt = 1\n",
    "for img_idx in np.nditer( np.where( y_test != model_dnn.predict(X_test))[0][:10]):\n",
    "    plt.subplot(1,10,cnt)\n",
    "    plt.imshow(X_test[img_idx,:].reshape((8,8)),cmap=plt.cm.gray_r,interpolation='nearest')\n",
    "    plt.axis('off')\n",
    "    plt.title(\"Test: %i Pred: %i\" % (label,y_test[img_idx]))\n",
    "    cnt = cnt + 1\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The true power comes in with custom models. TensorFlowEstimator allows you to specify your own conditions:\n",
    "* activation function (sigmoid,tanh - hyperbolic tangent, relu - rectified linear unit, etc.)\n",
    "* dropout (i.e. randomly drop out weights, should reduce overfitting)\n",
    "* modified hidden layers\n",
    "\n",
    "Example:\n",
    "https://medium.com/@ilblackdragon/tensorflow-tutorial-part-2-9ffe47049c92#.l3ig75mjv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step #1, avg. loss: 3.28346\n",
      "Step #401, epoch #13, avg. loss: 0.31768\n",
      "Step #801, epoch #27, avg. loss: 0.04370\n",
      "Step #1201, epoch #41, avg. loss: 0.02460\n",
      "Step #1601, epoch #55, avg. loss: 0.01851\n",
      "Step #2001, epoch #69, avg. loss: 0.01458\n",
      "Step #2401, epoch #82, avg. loss: 0.01170\n",
      "Step #2801, epoch #96, avg. loss: 0.00907\n",
      "Step #3201, epoch #110, avg. loss: 0.01123\n",
      "Step #3601, epoch #124, avg. loss: 0.00810\n",
      "Accuracy: 0.958843\n"
     ]
    }
   ],
   "source": [
    "def my_model(X, y):\n",
    "    \"\"\"This is DNN with 20, 15 hidden layers, and dropout of 0.1 probability.\"\"\"\n",
    "    layers = skflow.ops.dnn(X, [20, 15],tf.tanh, keep_prob=0.9)\n",
    "    return skflow.models.logistic_regression(layers, y)\n",
    "\n",
    "model_cdnn = skflow.TensorFlowEstimator(model_fn=my_model, n_classes=10,steps=4000,learning_rate=0.3)\n",
    "model_cdnn.fit(X_train,y_train)\n",
    "score = metrics.accuracy_score(y_test, model_cdnn.predict(X_test))\n",
    "print(\"Accuracy: %f\" % score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another example with a convolutional layer and a max pooling layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step #1, avg. loss: 3.56901\n",
      "Step #101, epoch #12, avg. loss: 2.38305\n",
      "Step #201, epoch #25, avg. loss: 1.82939\n",
      "Step #301, epoch #37, avg. loss: 1.59679\n",
      "Step #401, epoch #50, avg. loss: 1.31189\n",
      "Step #501, epoch #62, avg. loss: 1.27337\n",
      "Step #601, epoch #75, avg. loss: 1.23362\n",
      "Step #701, epoch #87, avg. loss: 1.05782\n",
      "Step #801, epoch #100, avg. loss: 1.03296\n",
      "Step #901, epoch #112, avg. loss: 0.86913\n",
      "Accuracy: 0.371524\n"
     ]
    }
   ],
   "source": [
    "def conv_model(X, y):\n",
    "    X = tf.expand_dims(X, 3)\n",
    "    features = tf.reduce_max(skflow.ops.conv2d(X, 12, [3, 3]), [1, 2])\n",
    "    features = tf.reshape(features, [-1, 12])\n",
    "    return skflow.models.logistic_regression(features, y)\n",
    "\n",
    "# Create a classifier, train and predict.\n",
    "classifier = skflow.TensorFlowEstimator(model_fn=conv_model, n_classes=10,\n",
    "                                        steps=1000, learning_rate=0.2,\n",
    "                                        batch_size=128)\n",
    "\n",
    "classifier.fit(X_train.reshape((898,8,8)), y_train)\n",
    "score = metrics.accuracy_score(classifier.predict(X_test.reshape((899,8,8))), y_test)\n",
    "print('Accuracy: %f' % score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Eventually, you will most likely want to just learn TensorFlow by itself, and then tie it into SKFlow as needed for fit/predict operations.\n",
    "\n",
    "\n",
    "For some practice on Skflow, try the following modifications and exercises:\n",
    "- (\\*) **Easy**\n",
    "- (\\*\\*) **Harder**\n",
    "- (\\*\\*\\*) **Hardest**\n",
    "\n",
    "------\n",
    "\n",
    "* (\\*) Try modifying the number of steps as well as the learning rate. How does this affect your results insample and out of sample? Try looking at the TensorBoard results as well. [TensorBoard](https://www.tensorflow.org/versions/r0.7/how_tos/summaries_and_tensorboard/index.html)\n",
    "* (\\*) Try modifying the number of hidden layers as well as the size of the hidden layers. [This link on hidden layers](http://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw) may help you get a good feel.\n",
    "* (\\*) Rerun your own version of the TensorFlowClassifier on a different dataset. Compare against techniques such as KNN, SVM, or Logistic Regression.\n",
    "* (\\*\\*) Try using a tf.relu activation function instead of sigmoid [List of various Activation Functions](https://www.tensorflow.org/versions/r0.7/api_docs/python/nn.html#activation-functions)\n",
    "* (\\*\\*\\*) Create a true TensorFlow model on the MNIST dataset [TensorFlow Example](https://www.tensorflow.org/versions/0.6.0/tutorials/mnist/beginners/index.html#mnist-for-ml-beginners)\n",
    "* (\\*\\*\\*\\*\\*\\*) Install and run through the [Theano tutorials](https://www.tensorflow.org/versions/0.6.0/tutorials/mnist/beginners/index.html#mnist-for-ml-beginners)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "* SKFlow tutorials: https://github.com/tensorflow/skflow\n",
    "* More SKFLow examples: https://medium.com/@ilblackdragon/tensorflow-tutorial-part-2-9ffe47049c92#.h7fhs7nup\n",
    "* Convolutional Neural Network guide: http://cs231n.github.io\n",
    "* Understanding CNN: http://timdettmers.com/2015/03/26/convolution-deep-learning/\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}