16_recommendation/16_03_branding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Who To Follow: Recommending Brands\n",
    "\n",
    "In this exercise, we consider a simple dataset: users following brands. We only know of a user follows a brand or not, but not how much he or she likes this brand.  Given the brands the user is following, we would like to recommend similar brands that s/he might be interested in.  \n",
    "\n",
    "\n",
    "This is another example of item-based collaborative filtering. Note! The original data comes in as a set of users following brands, so if we convert this data into a matrix of user x brand, we will encode each value as 0 (user does not follow brand) or 1 (user does follow brand). \n",
    "\n",
    "Given that we only have 1's and 0's, we probably should not use pearson correlation or vector cosine as a similarity metric.\n",
    "\n",
    "You could easily extend this to product recommendations (i.e. You purchase this product (1's and 0's). Here are some other products you might like).\n",
    "\n",
    "### Import code and data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sb\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>brand</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>80002</td>\n",
       "      <td>Target</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>80002</td>\n",
       "      <td>Home Depot</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>80010</td>\n",
       "      <td>Levi's</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>80010</td>\n",
       "      <td>Puma</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>80010</td>\n",
       "      <td>Cuisinart</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id       brand\n",
       "0  80002      Target\n",
       "1  80002  Home Depot\n",
       "2  80010      Levi's\n",
       "3  80010        Puma\n",
       "4  80010   Cuisinart"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv('user-brands.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (23804, 2) Unique User IDs: 3759 Unique Brands: 198\n"
     ]
    }
   ],
   "source": [
    "print \"Shape:\", data.shape, \"Unique User IDs:\", data.id.nunique(), \"Unique Brands:\", data.brand.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### User-by-brand matrix\n",
    "\n",
    "Note that our data above is in condensed format. We could make it into a sparse matrix, which might be easier to work with.  You could do this with `pd.pivot_table`:\n",
    "\n",
    "    M = pd.pivot_table(data, index='id', columns='brand', aggfunc='size', fill_value=0)\n",
    "\n",
    "We use a `groupby` statement, which gives us a multi-index series, and then we make an `unstack` call to transform it into a dataframe again.  \n",
    "\n",
    "Note that these steps are not necessary as you could complete this exercise in several different ways."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "M = pd.pivot_table(data, index=\"id\",columns=\"brand\",aggfunc=\"size\",fill_value=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>brand</th>\n",
       "      <th>6pm.com</th>\n",
       "      <th>Abercrombie &amp; Fitch</th>\n",
       "      <th>Adidas</th>\n",
       "      <th>Aeropostale</th>\n",
       "      <th>Aldo</th>\n",
       "      <th>All Saints</th>\n",
       "      <th>Amazon.com</th>\n",
       "      <th>American Apparel</th>\n",
       "      <th>American Eagle</th>\n",
       "      <th>Ann Taylor</th>\n",
       "      <th>...</th>\n",
       "      <th>Walgreens</th>\n",
       "      <th>Walk-Over</th>\n",
       "      <th>Wet Seal</th>\n",
       "      <th>Windsor</th>\n",
       "      <th>YSL</th>\n",
       "      <th>Yves Saint Laurent</th>\n",
       "      <th>ZOO</th>\n",
       "      <th>Zara</th>\n",
       "      <th>Zipcar</th>\n",
       "      <th>vineyard vines</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>80002</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80010</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80011</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80015</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80020</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 198 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "brand  6pm.com  Abercrombie & Fitch  Adidas  Aeropostale  Aldo  All Saints  \\\n",
       "id                                                                           \n",
       "80002        0                    0       0            0     0           0   \n",
       "80010        0                    0       0            0     0           0   \n",
       "80011        0                    0       0            0     0           0   \n",
       "80015        0                    0       0            0     0           0   \n",
       "80020        0                    0       0            0     0           0   \n",
       "\n",
       "brand  Amazon.com  American Apparel  American Eagle  Ann Taylor  \\\n",
       "id                                                                \n",
       "80002           0                 0               0           0   \n",
       "80010           0                 0               0           0   \n",
       "80011           0                 0               0           0   \n",
       "80015           0                 0               0           0   \n",
       "80020           0                 0               0           0   \n",
       "\n",
       "brand       ...        Walgreens  Walk-Over  Wet Seal  Windsor  YSL  \\\n",
       "id          ...                                                       \n",
       "80002       ...                0          0         0        0    0   \n",
       "80010       ...                0          0         0        0    0   \n",
       "80011       ...                0          0         0        0    0   \n",
       "80015       ...                0          0         0        0    0   \n",
       "80020       ...                0          0         0        0    0   \n",
       "\n",
       "brand  Yves Saint Laurent  ZOO  Zara  Zipcar  vineyard vines  \n",
       "id                                                            \n",
       "80002                   0    0     0       0               0  \n",
       "80010                   0    0     0       0               0  \n",
       "80011                   0    0     0       0               0  \n",
       "80015                   0    0     0       0               0  \n",
       "80020                   0    0     0       0               0  \n",
       "\n",
       "[5 rows x 198 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Jaccard distance\n",
    "\n",
    "Since we will use a neighborhood method, we need a definition of _distance_.  We'll use the _Jaccard distance_ for this. \n",
    "\n",
    "The [_Jaccard index_](https://en.wikipedia.org/wiki/Jaccard_index) is a similarity metric between two sets.  It measures how many elements two sets have in common, as a fraction of the total number of distinct elements in both sets.  \n",
    "\n",
    "$$\\text{Jaccard index} = \\frac{ |A \\cap B | }{ |A \\cup B| }$$\n",
    "\n",
    "We could make a Jaccard matrix $J$, with pairwise similarities $J_{ij}$ as entries.\n",
    "- `J[i, j]` = Jaccard similarity between doc _i_ and _j_ (between 0 and 1)\n",
    "- `J[i, i]` = 1, obviously, and\n",
    "- `J[i, j]` = `J[i, j]`, i.e., the matrix is symmetric.\n",
    "\n",
    "We could also define the _Jaccard distance_, which has $D_{ii} = 0$ for identical sets, and bigger values as the sets have less words in common.  We define: $D = 1 - J,$ which has values between 0 and 1.\n",
    "\n",
    "Common applications of the Jaccard index include text clustering, but we can use it for brand clustering as well, counting the number of followers they have in common."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "## Exercise\n",
    "\n",
    "\n",
    "- Create a brand-by-brand matrix, with the similarity distances between two brands in each entry.\n",
    "   - Obviously, you'd have $N_{ii} = 0$ for each brand $i$, and $N_{ij} = N_{ji}$ for each pair of brands.\n",
    "   - You can create a 2-dimenional `np.array` for this, or a nested dictionary `N = {i: {j: distance}}`, or anything you like.\n",
    "      \n",
    "      \n",
    "- For a few brands of your choice, show the top most similar brands.  \n",
    "   - Do your results make sense? Would you agree?\n",
    "   \n",
    "   \n",
    "- For a few users, make a few top recommendations.\n",
    "   - Per user, display the brands s/he's already following\n",
    "   - For each brand, compute the distance to all other brands\n",
    "   - Average all distances to find the few closest brands, with the shortest average distance\n",
    "   - Make sure you exclude the brands the user is already following from the recommendations\n",
    "   \n",
    "   \n",
    "**Hint: Remember that in this case, lower distances are closer matches!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def jaccard_distance(M):\n",
    "    n_brands = M.shape[1]\n",
    "    I = M.T.dot(M)  # number of users in common \n",
    "    n_users_per_brand = np.diag(I)\n",
    "    N = n_users_per_brand.reshape(n_brands, 1) * np.ones(n_brands)\n",
    "    U = N + N.T - I  # total unique followers = n_users_i + n_users_j - users in common\n",
    "    J = I / U.astype(float)  # similarity matrix\n",
    "    D = 1 - J  # distance\n",
    "    return D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 260.  260.  260. ...,  260.  260.  260.]\n",
      " [   8.    8.    8. ...,    8.    8.    8.]\n",
      " [   1.    1.    1. ...,    1.    1.    1.]\n",
      " ..., \n",
      " [   4.    4.    4. ...,    4.    4.    4.]\n",
      " [   1.    1.    1. ...,    1.    1.    1.]\n",
      " [   1.    1.    1. ...,    1.    1.    1.]]\n",
      "brand                  6pm.com  Abercrombie & Fitch  Adidas  Aeropostale  \\\n",
      "brand                                                                      \n",
      "6pm.com                    260                  267     260          266   \n",
      "Abercrombie & Fitch        267                    8       9           14   \n",
      "Adidas                     260                    9       1            8   \n",
      "Aeropostale                266                   14       8            7   \n",
      "Aldo                       261                    9       2            8   \n",
      "All Saints                 261                    9       2            8   \n",
      "Amazon.com                 261                    9       2            8   \n",
      "American Apparel           262                   11       4           10   \n",
      "American Eagle             281                   27      23           26   \n",
      "Ann Taylor                 266                   16       9           15   \n",
      "Anthropologie              266                   14       7           13   \n",
      "Apple                      262                   10       3            9   \n",
      "Armani                     262                   10       3            9   \n",
      "Armani Exchange            261                    9       2            8   \n",
      "Ashley Stewart             261                    9       2            8   \n",
      "BCBG                       261                    9       2            8   \n",
      "BCBGMAXAZRIA               597                  436     429          435   \n",
      "BCBGeneration              261                    9       2            8   \n",
      "BUCKLE                     264                   11       5           11   \n",
      "Bali                       261                    9       2            8   \n",
      "Banana Republic           1061                  937     932          938   \n",
      "Barneys                    261                    9       2            8   \n",
      "Barneys Warehouse          261                    9       2            8   \n",
      "Bed Bath & Beyond          266                   13       8           14   \n",
      "Betsey Johnson             439                  252     245          251   \n",
      "Billabong                  261                    9       2            8   \n",
      "Billionaire Boys Club      261                    9       2            8   \n",
      "Black Diamond              261                    9       2            8   \n",
      "Bloomingdale's             260                    8       2            7   \n",
      "Boden                      261                    9       2            8   \n",
      "...                        ...                  ...     ...          ...   \n",
      "Target                    1938                 1868    1866         1868   \n",
      "The Limited                261                   10       3            9   \n",
      "The North Face             261                    9       2            8   \n",
      "Tilly's                    261                    9       2            8   \n",
      "Tobi                       261                    9       2            8   \n",
      "Tommy Hilfiger             457                  264     257          261   \n",
      "Topshop                    261                    9       2            8   \n",
      "Torrid                     262                   10       3            9   \n",
      "Tory Burch                 262                   10       3            9   \n",
      "True Religion              262                   10       3            9   \n",
      "UGG Australia              262                   10       3            9   \n",
      "UNIQLO                     260                    9       2            8   \n",
      "Under Armour               264                   12       5           10   \n",
      "Urban Outfitters           269                   16      10           16   \n",
      "Vans                       261                    8       2            8   \n",
      "Vera Bradley               262                   10       3            9   \n",
      "Victoria's Secret          274                   21      16           20   \n",
      "Villeroy & Boch            301                   94      87           93   \n",
      "Vince Camuto               261                    9       2            8   \n",
      "Vitamix                    261                    9       2            8   \n",
      "Walgreens                  262                   10       3            9   \n",
      "Walk-Over                  261                    9       2            8   \n",
      "Wet Seal                   269                   16      10           16   \n",
      "Windsor                    261                    9       2            8   \n",
      "YSL                        261                    9       2            8   \n",
      "Yves Saint Laurent         261                    9       2            8   \n",
      "ZOO                        261                    9       2            8   \n",
      "Zara                       264                   11       5           11   \n",
      "Zipcar                     261                    9       2            8   \n",
      "vineyard vines             261                    9       2            8   \n",
      "\n",
      "brand                  Aldo  All Saints  Amazon.com  American Apparel  \\\n",
      "brand                                                                   \n",
      "6pm.com                 261         261         261               262   \n",
      "Abercrombie & Fitch       9           9           9                11   \n",
      "Adidas                    2           2           2                 4   \n",
      "Aeropostale               8           8           8                10   \n",
      "Aldo                      1           2           2                 4   \n",
      "All Saints                2           1           2                 4   \n",
      "Amazon.com                2           2           1                 4   \n",
      "American Apparel          4           4           4                 3   \n",
      "American Eagle           23          23          23                25   \n",
      "Ann Taylor                9           9           9                11   \n",
      "Anthropologie             7           7           7                 9   \n",
      "Apple                     3           3           3                 5   \n",
      "Armani                    3           3           3                 5   \n",
      "Armani Exchange           2           2           2                 4   \n",
      "Ashley Stewart            2           2           2                 4   \n",
      "BCBG                      2           2           2                 4   \n",
      "BCBGMAXAZRIA            430         430         430               432   \n",
      "BCBGeneration             2           2           2                 4   \n",
      "BUCKLE                    5           5           5                 7   \n",
      "Bali                      2           2           2                 4   \n",
      "Banana Republic         933         933         933               934   \n",
      "Barneys                   2           2           2                 4   \n",
      "Barneys Warehouse         2           2           2                 4   \n",
      "Bed Bath & Beyond         8           8           8                10   \n",
      "Betsey Johnson          246         246         246               248   \n",
      "Billabong                 2           2           2                 4   \n",
      "Billionaire Boys Club     2           2           2                 4   \n",
      "Black Diamond             2           2           2                 4   \n",
      "Bloomingdale's            2           2           2                 4   \n",
      "Boden                     2           2           2                 4   \n",
      "...                     ...         ...         ...               ...   \n",
      "Target                 1866        1867        1866              1866   \n",
      "The Limited               3           3           3                 5   \n",
      "The North Face            2           2           2                 4   \n",
      "Tilly's                   2           2           2                 4   \n",
      "Tobi                      2           2           2                 4   \n",
      "Tommy Hilfiger          258         258         257               260   \n",
      "Topshop                   2           2           2                 4   \n",
      "Torrid                    3           3           3                 5   \n",
      "Tory Burch                3           3           3                 5   \n",
      "True Religion             3           3           3                 5   \n",
      "UGG Australia             3           3           3                 5   \n",
      "UNIQLO                    2           2           2                 3   \n",
      "Under Armour              5           5           5                 7   \n",
      "Urban Outfitters         10          10          10                11   \n",
      "Vans                      2           2           2                 4   \n",
      "Vera Bradley              3           3           3                 5   \n",
      "Victoria's Secret        16          16          16                18   \n",
      "Villeroy & Boch          88          88          88                90   \n",
      "Vince Camuto              2           2           2                 4   \n",
      "Vitamix                   2           2           2                 4   \n",
      "Walgreens                 3           3           3                 5   \n",
      "Walk-Over                 2           2           2                 4   \n",
      "Wet Seal                 10          10          10                12   \n",
      "Windsor                   2           2           2                 4   \n",
      "YSL                       2           2           2                 4   \n",
      "Yves Saint Laurent        2           2           2                 4   \n",
      "ZOO                       2           2           2                 4   \n",
      "Zara                      5           5           5                 7   \n",
      "Zipcar                    2           2           2                 4   \n",
      "vineyard vines            2           2           2                 4   \n",
      "\n",
      "brand                  American Eagle  Ann Taylor       ...        Walgreens  \\\n",
      "brand                                                   ...                    \n",
      "6pm.com                           281         266       ...              262   \n",
      "Abercrombie & Fitch                27          16       ...               10   \n",
      "Adidas                             23           9       ...                3   \n",
      "Aeropostale                        26          15       ...                9   \n",
      "Aldo                               23           9       ...                3   \n",
      "All Saints                         23           9       ...                3   \n",
      "Amazon.com                         23           9       ...                3   \n",
      "American Apparel                   25          11       ...                5   \n",
      "American Eagle                     22          30       ...               24   \n",
      "Ann Taylor                         30           8       ...               10   \n",
      "Anthropologie                      28          14       ...                8   \n",
      "Apple                              24          10       ...                4   \n",
      "Armani                             24          10       ...                4   \n",
      "Armani Exchange                    23           9       ...                3   \n",
      "Ashley Stewart                     23           9       ...                3   \n",
      "BCBG                               23           9       ...                3   \n",
      "BCBGMAXAZRIA                      449         435       ...              431   \n",
      "BCBGeneration                      23           9       ...                3   \n",
      "BUCKLE                             25          12       ...                6   \n",
      "Bali                               23           9       ...                3   \n",
      "Banana Republic                   948         935       ...              934   \n",
      "Barneys                            23           9       ...                3   \n",
      "Barneys Warehouse                  23           9       ...                3   \n",
      "Bed Bath & Beyond                  28          15       ...                9   \n",
      "Betsey Johnson                    267         251       ...              247   \n",
      "Billabong                          23           9       ...                3   \n",
      "Billionaire Boys Club              23           9       ...                3   \n",
      "Black Diamond                      23           9       ...                3   \n",
      "Bloomingdale's                     23           9       ...                3   \n",
      "Boden                              23           9       ...                3   \n",
      "...                               ...         ...       ...              ...   \n",
      "Target                           1869        1867       ...             1866   \n",
      "The Limited                        24          10       ...                4   \n",
      "The North Face                     23           9       ...                3   \n",
      "Tilly's                            23           9       ...                3   \n",
      "Tobi                               22           9       ...                3   \n",
      "Tommy Hilfiger                    277         264       ...              259   \n",
      "Topshop                            23           9       ...                3   \n",
      "Torrid                             24          10       ...                4   \n",
      "Tory Burch                         23          10       ...                4   \n",
      "True Religion                      24          10       ...                4   \n",
      "UGG Australia                      24          10       ...                4   \n",
      "UNIQLO                             23           9       ...                3   \n",
      "Under Armour                       25          12       ...                6   \n",
      "Urban Outfitters                   30          17       ...               11   \n",
      "Vans                               22           9       ...                3   \n",
      "Vera Bradley                       24          10       ...                4   \n",
      "Victoria's Secret                  35          23       ...               17   \n",
      "Villeroy & Boch                   109          95       ...               89   \n",
      "Vince Camuto                       23           9       ...                3   \n",
      "Vitamix                            23           9       ...                3   \n",
      "Walgreens                          24          10       ...                2   \n",
      "Walk-Over                          23           9       ...                3   \n",
      "Wet Seal                           27          17       ...               11   \n",
      "Windsor                            23           9       ...                3   \n",
      "YSL                                23           9       ...                3   \n",
      "Yves Saint Laurent                 23           9       ...                3   \n",
      "ZOO                                23           9       ...                3   \n",
      "Zara                               26          12       ...                6   \n",
      "Zipcar                             23           9       ...                3   \n",
      "vineyard vines                     23           9       ...                3   \n",
      "\n",
      "brand                  Walk-Over  Wet Seal  Windsor   YSL  Yves Saint Laurent  \\\n",
      "brand                                                                           \n",
      "6pm.com                      261       269      261   261                 261   \n",
      "Abercrombie & Fitch            9        16        9     9                   9   \n",
      "Adidas                         2        10        2     2                   2   \n",
      "Aeropostale                    8        16        8     8                   8   \n",
      "Aldo                           2        10        2     2                   2   \n",
      "All Saints                     2        10        2     2                   2   \n",
      "Amazon.com                     2        10        2     2                   2   \n",
      "American Apparel               4        12        4     4                   4   \n",
      "American Eagle                23        27       23    23                  23   \n",
      "Ann Taylor                     9        17        9     9                   9   \n",
      "Anthropologie                  7        15        7     7                   7   \n",
      "Apple                          3        11        3     3                   3   \n",
      "Armani                         3        11        3     3                   3   \n",
      "Armani Exchange                2        10        2     1                   2   \n",
      "Ashley Stewart                 2        10        2     2                   2   \n",
      "BCBG                           2        10        2     2                   2   \n",
      "BCBGMAXAZRIA                 430       438      430   430                 430   \n",
      "BCBGeneration                  2        10        2     2                   2   \n",
      "BUCKLE                         5        12        5     5                   5   \n",
      "Bali                           2        10        2     1                   2   \n",
      "Banana Republic              933       940      933   932                 933   \n",
      "Barneys                        2        10        2     2                   2   \n",
      "Barneys Warehouse              2        10        2     2                   2   \n",
      "Bed Bath & Beyond              8        15        8     8                   8   \n",
      "Betsey Johnson               246       254      245   246                 246   \n",
      "Billabong                      2        10        2     2                   2   \n",
      "Billionaire Boys Club          2        10        2     2                   1   \n",
      "Black Diamond                  2        10        2     2                   2   \n",
      "Bloomingdale's                 2        10        2     2                   2   \n",
      "Boden                          2        10        2     2                   2   \n",
      "...                          ...       ...      ...   ...                 ...   \n",
      "Target                      1866      1867     1867  1866                1866   \n",
      "The Limited                    3        11        3     3                   3   \n",
      "The North Face                 2        10        2     2                   2   \n",
      "Tilly's                        2        10        2     2                   2   \n",
      "Tobi                           2        10        2     2                   2   \n",
      "Tommy Hilfiger               258       265      258   257                 258   \n",
      "Topshop                        2        10        2     2                   2   \n",
      "Torrid                         3        11        3     3                   3   \n",
      "Tory Burch                     3        11        3     3                   3   \n",
      "True Religion                  3        11        3     3                   3   \n",
      "UGG Australia                  3        10        3     3                   3   \n",
      "UNIQLO                         2        10        2     2                   2   \n",
      "Under Armour                   5        13        5     5                   5   \n",
      "Urban Outfitters              10        17       10    10                  10   \n",
      "Vans                           2         9        2     2                   2   \n",
      "Vera Bradley                   3        11        3     3                   3   \n",
      "Victoria's Secret             16        21       16    15                  16   \n",
      "Villeroy & Boch               88        96       88    87                  88   \n",
      "Vince Camuto                   2        10        2     2                   2   \n",
      "Vitamix                        2        10        2     2                   2   \n",
      "Walgreens                      3        11        3     3                   3   \n",
      "Walk-Over                      1        10        2     2                   2   \n",
      "Wet Seal                      10         9       10    10                  10   \n",
      "Windsor                        2        10        1     2                   2   \n",
      "YSL                            2        10        2     1                   2   \n",
      "Yves Saint Laurent             2        10        2     2                   1   \n",
      "ZOO                            2        10        2     2                   2   \n",
      "Zara                           5        13        5     5                   5   \n",
      "Zipcar                         2        10        2     2                   2   \n",
      "vineyard vines                 2        10        2     2                   2   \n",
      "\n",
      "brand                   ZOO  Zara  Zipcar  vineyard vines  \n",
      "brand                                                      \n",
      "6pm.com                 261   264     261             261  \n",
      "Abercrombie & Fitch       9    11       9               9  \n",
      "Adidas                    2     5       2               2  \n",
      "Aeropostale               8    11       8               8  \n",
      "Aldo                      2     5       2               2  \n",
      "All Saints                2     5       2               2  \n",
      "Amazon.com                2     5       2               2  \n",
      "American Apparel          4     7       4               4  \n",
      "American Eagle           23    26      23              23  \n",
      "Ann Taylor                9    12       9               9  \n",
      "Anthropologie             7    10       7               7  \n",
      "Apple                     3     6       3               3  \n",
      "Armani                    3     6       3               3  \n",
      "Armani Exchange           2     5       2               2  \n",
      "Ashley Stewart            2     5       2               2  \n",
      "BCBG                      2     5       2               2  \n",
      "BCBGMAXAZRIA            430   432     430             430  \n",
      "BCBGeneration             2     5       2               2  \n",
      "BUCKLE                    5     8       5               5  \n",
      "Bali                      2     5       2               2  \n",
      "Banana Republic         933   934     932             932  \n",
      "Barneys                   2     5       2               2  \n",
      "Barneys Warehouse         2     5       2               2  \n",
      "Bed Bath & Beyond         8    11       8               8  \n",
      "Betsey Johnson          246   249     246             246  \n",
      "Billabong                 2     5       2               2  \n",
      "Billionaire Boys Club     2     5       2               2  \n",
      "Black Diamond             2     5       2               2  \n",
      "Bloomingdale's            2     5       2               2  \n",
      "Boden                     2     5       2               2  \n",
      "...                     ...   ...     ...             ...  \n",
      "Target                 1867  1867    1867            1866  \n",
      "The Limited               3     6       3               3  \n",
      "The North Face            2     5       2               2  \n",
      "Tilly's                   2     5       2               2  \n",
      "Tobi                      2     5       2               2  \n",
      "Tommy Hilfiger          258   261     258             258  \n",
      "Topshop                   2     5       2               2  \n",
      "Torrid                    3     6       3               3  \n",
      "Tory Burch                3     6       3               3  \n",
      "True Religion             3     5       3               3  \n",
      "UGG Australia             3     6       3               3  \n",
      "UNIQLO                    2     5       2               2  \n",
      "Under Armour              5     8       5               5  \n",
      "Urban Outfitters         10    13      10              10  \n",
      "Vans                      2     5       2               2  \n",
      "Vera Bradley              3     6       3               3  \n",
      "Victoria's Secret        16    19      16              16  \n",
      "Villeroy & Boch          88    91      88              88  \n",
      "Vince Camuto              2     5       2               2  \n",
      "Vitamix                   2     5       2               2  \n",
      "Walgreens                 3     6       3               3  \n",
      "Walk-Over                 2     5       2               2  \n",
      "Wet Seal                 10    13      10              10  \n",
      "Windsor                   2     5       2               2  \n",
      "YSL                       2     5       2               2  \n",
      "Yves Saint Laurent        2     5       2               2  \n",
      "ZOO                       1     5       2               2  \n",
      "Zara                      5     4       5               5  \n",
      "Zipcar                    2     5       1               2  \n",
      "vineyard vines            2     5       2               1  \n",
      "\n",
      "[198 rows x 198 columns]\n"
     ]
    }
   ],
   "source": [
    "n_brands = M.shape[1]\n",
    "### Top part of fraction\n",
    "I = M.T.dot(M)  # number of users in common \n",
    "n_users_per_brand = np.diag(I)\n",
    "N = n_users_per_brand.reshape(n_brands, 1) * np.ones(n_brands)\n",
    "print N\n",
    "\n",
    "### Bottom part of fraction\n",
    "U = N + N.T - I  # total unique followers = n_users_i + n_users_j - users in common \n",
    "print U\n",
    "J = I / U.astype(float)  # similarity matrix\n",
    "D = 1 - J  # distance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>brand</th>\n",
       "      <th>6pm.com</th>\n",
       "      <th>Abercrombie &amp; Fitch</th>\n",
       "      <th>Adidas</th>\n",
       "      <th>Aeropostale</th>\n",
       "      <th>Aldo</th>\n",
       "      <th>All Saints</th>\n",
       "      <th>Amazon.com</th>\n",
       "      <th>American Apparel</th>\n",
       "      <th>American Eagle</th>\n",
       "      <th>Ann Taylor</th>\n",
       "      <th>...</th>\n",
       "      <th>Walgreens</th>\n",
       "      <th>Walk-Over</th>\n",
       "      <th>Wet Seal</th>\n",
       "      <th>Windsor</th>\n",
       "      <th>YSL</th>\n",
       "      <th>Yves Saint Laurent</th>\n",
       "      <th>ZOO</th>\n",
       "      <th>Zara</th>\n",
       "      <th>Zipcar</th>\n",
       "      <th>vineyard vines</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>brand</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6pm.com</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.996255</td>\n",
       "      <td>0.996154</td>\n",
       "      <td>0.996241</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.996183</td>\n",
       "      <td>0.996441</td>\n",
       "      <td>0.992481</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Abercrombie &amp; Fitch</th>\n",
       "      <td>0.996255</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.928571</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.888889</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.9375</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.909091</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Adidas</th>\n",
       "      <td>0.996154</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 198 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "brand                 6pm.com  Abercrombie & Fitch    Adidas  Aeropostale  \\\n",
       "brand                                                                       \n",
       "6pm.com              0.000000             0.996255  0.996154     0.996241   \n",
       "Abercrombie & Fitch  0.996255             0.000000  1.000000     0.928571   \n",
       "Adidas               0.996154             1.000000  0.000000     1.000000   \n",
       "\n",
       "brand                Aldo  All Saints  Amazon.com  American Apparel  \\\n",
       "brand                                                                 \n",
       "6pm.com                 1           1           1          0.996183   \n",
       "Abercrombie & Fitch     1           1           1          1.000000   \n",
       "Adidas                  1           1           1          1.000000   \n",
       "\n",
       "brand                American Eagle  Ann Taylor       ...        Walgreens  \\\n",
       "brand                                                 ...                    \n",
       "6pm.com                    0.996441    0.992481       ...                1   \n",
       "Abercrombie & Fitch        0.888889    1.000000       ...                1   \n",
       "Adidas                     1.000000    1.000000       ...                1   \n",
       "\n",
       "brand                Walk-Over  Wet Seal  Windsor  YSL  Yves Saint Laurent  \\\n",
       "brand                                                                        \n",
       "6pm.com                      1    1.0000        1    1                   1   \n",
       "Abercrombie & Fitch          1    0.9375        1    1                   1   \n",
       "Adidas                       1    1.0000        1    1                   1   \n",
       "\n",
       "brand                ZOO      Zara  Zipcar  vineyard vines  \n",
       "brand                                                       \n",
       "6pm.com                1  1.000000       1               1  \n",
       "Abercrombie & Fitch    1  0.909091       1               1  \n",
       "Adidas                 1  1.000000       1               1  \n",
       "\n",
       "[3 rows x 198 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "brand_distance = jaccard_distance(M)\n",
    "brand_distance.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this is a _distance_ matrix, so the lower, the closer, the more similar.  Hence we have zeros on the diagonal.\n",
    "\n",
    "Let's show the top most similar brands for some known brands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple               : Apple, Diesel, Levi's, 6pm.com, Nambe\n"
     ]
    }
   ],
   "source": [
    "top = 5\n",
    "for brand in ['Apple']:\n",
    "    print \"%-20s:\" % brand, \n",
    "    print \", \".join(brand_distance[brand].sort(ascending=True,inplace=False).index[:top]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's pick some other random brands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 5 similar brands for some random 10 brands\n",
      "Sunglass Hut        : Sunglass Hut, CB2, Diesel, Last Call by Neiman Marcus, Kenneth Cole\n",
      "Patagonia           : Patagonia, The North Face, Louis Vuitton, Ann Taylor, Coach\n",
      "Adidas              : Adidas, MAC, Louis Vuitton, Gucci, Pottery Barn\n",
      "Janie and Jack      : Janie and Jack, Pottery Barn, John Varvatos, Diesel, DKNY\n",
      "Journeys            : Journeys, Topshop, UGG Australia, Lululemon, Wet Seal\n",
      "New Balance         : New Balance, KitchenAid, Columbia, Levi's, Shoebuy\n",
      "Bali                : Bali, Armani Exchange, YSL, Charles David, Lancome\n",
      "Eddie Bauer         : Eddie Bauer, Life is good, Columbia, New Balance, Converse\n",
      "Vitamix             : Vitamix, Cuisinart, KitchenAid, Kohl's, Old Navy\n",
      "Children's Place    : Children's Place, Justice, Melissa & Doug, Puma, Old Navy\n"
     ]
    }
   ],
   "source": [
    "brands = M.columns\n",
    "\n",
    "n_show = 10  # show a few brands\n",
    "print \"Top %d similar brands for some random %d brands\" % (top, n_show)\n",
    "for brand in np.random.choice(brands, n_show, replace=False):\n",
    "    print \"%-20s:\" % brand, \n",
    "    print \", \".join(brand_distance[brand].sort(inplace=False).index[:top])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommendations\n",
    "Given a user, return recommended brands with scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def recommend_brands_for_user(user, M, top=5):\n",
    "    user_brands = M.loc[user][M.loc[user] > 0].index  # get brands of user\n",
    "    brand_distance = jaccard_distance(M)        \n",
    "    recs = brand_distance[user_brands].mean(axis=1).sort(ascending=True, inplace=False).index\n",
    "    # remove all top brands that are already on this user's list\n",
    "    recs = [rec for rec in recs if rec not in user_brands]\n",
    "    return recs[:top]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "User 82446\n",
      "Already following: Guess\n",
      "Recommended: Calvin Klein, Steve Madden, Express, DKNY, BCBGMAXAZRIA\n",
      "\n",
      "User 81989\n",
      "Already following: Crocs, Home Depot, Nordstrom, Old Navy, Target\n",
      "Recommended: Kohl's, Gap, Crate & Barrel, KitchenAid, Express\n",
      "\n",
      "User 80895\n",
      "Already following: Target\n",
      "Recommended: Old Navy, Kohl's, Home Depot, Gap, Crate & Barrel\n",
      "\n",
      "User 81320\n",
      "Already following: CB2, Crate & Barrel, Gap, J.Crew, Melissa & Doug, Nordstrom\n",
      "Recommended: Banana Republic, KitchenAid, Container Store, Restoration Hardware, Cuisinart\n",
      "\n",
      "User 82327\n",
      "Already following: New Balance\n",
      "Recommended: KitchenAid, Columbia, Levi's, Shoebuy, Converse\n",
      "\n"
     ]
    }
   ],
   "source": [
    "n_users = 5\n",
    "# for user in [90217, 86156, 89116, 89112]:\n",
    "for user in np.random.choice(M.index, n_users, replace=False):\n",
    "    print \"User %s\" % user\n",
    "    print \"Already following:\", \", \".join(brands[M.loc[user] > 0])\n",
    "    print \"Recommended:\", \", \".join(recommend_brands_for_user(user, M))\n",
    "    print"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}