urlexpander_quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fFjkHrN0HrJO"
   },
   "source": [
    "# urlExpander Quickstart\n",
    "\n",
    "[urlExpander](https://github.com/SMAPPNYU/urlExpander) is a Python package for quickly and thoroughly expanding URLs.\n",
    "\n",
    "urlExpander was created to address this challenge in a scalable and robust manner. It does so by providing utility functions to convert Tweets into link datasets, filter for known for link-shortening services (like bit.ly), resolve shortened links, and parse the title and meta description from webpages.\n",
    "\n",
    "This package differs from other approaches because it handles ad-based urls (like adf.ly, lnx.lu, linkbucks.com, and adfoc.us) thanks to the Unshortenit library, as well as resolves redirects to defunct websites (like blacktolive.com). \n",
    "\n",
    "Most importantly, urlExpander offers multithreaded url expansion. The multithreaded url expansion was created to overcome the bottleneck of mass link expansion through parallelization, minimizating http requests, caching results, and chunking the input into smaller pieces.\n",
    "\n",
    "\n",
    "You can download the software using pip:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "collapsed": true,
    "id": "Mwtz2YICIOxA",
    "jupyter": {
     "outputs_hidden": true
    },
    "outputId": "12669322-2b9d-4abc-fc42-c01bfaeb58fb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting urlexpander\n",
      "  Downloading urlexpander-0.0.37.tar.gz (11 kB)\n",
      "Collecting tldextract\n",
      "  Downloading tldextract-3.1.2-py2.py3-none-any.whl (87 kB)\n",
      "\u001b[K     |████████████████████████████████| 87 kB 2.5 MB/s \n",
      "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from urlexpander) (1.1.5)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from urlexpander) (1.19.5)\n",
      "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from urlexpander) (4.62.3)\n",
      "Collecting unshortenit\n",
      "  Downloading unshortenit-0.4.0.tar.gz (8.9 kB)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->urlexpander) (2018.9)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->urlexpander) (2.8.2)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->urlexpander) (1.15.0)\n",
      "Requirement already satisfied: idna in /usr/local/lib/python3.7/dist-packages (from tldextract->urlexpander) (2.10)\n",
      "Requirement already satisfied: filelock>=3.0.8 in /usr/local/lib/python3.7/dist-packages (from tldextract->urlexpander) (3.3.0)\n",
      "Collecting requests-file>=1.4\n",
      "  Downloading requests_file-1.5.1-py2.py3-none-any.whl (3.7 kB)\n",
      "Requirement already satisfied: requests>=2.1.0 in /usr/local/lib/python3.7/dist-packages (from tldextract->urlexpander) (2.23.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.1.0->tldextract->urlexpander) (2021.5.30)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.1.0->tldextract->urlexpander) (1.24.3)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.1.0->tldextract->urlexpander) (3.0.4)\n",
      "Requirement already satisfied: click>=6.7 in /usr/local/lib/python3.7/dist-packages (from unshortenit->urlexpander) (7.1.2)\n",
      "Requirement already satisfied: lxml>=4.1.1 in /usr/local/lib/python3.7/dist-packages (from unshortenit->urlexpander) (4.2.6)\n",
      "Building wheels for collected packages: urlexpander, unshortenit\n",
      "  Building wheel for urlexpander (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for urlexpander: filename=urlexpander-0.0.37-py3-none-any.whl size=11115 sha256=54dd180b4d884af3bf3486a5c62e139b4dfac56b6e4ed591377a7ea6b2d87406\n",
      "  Stored in directory: /root/.cache/pip/wheels/36/f2/aa/9319bc326946db050e310f27f18f63327cf083a8a80aff78c2\n",
      "  Building wheel for unshortenit (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for unshortenit: filename=unshortenit-0.4.0-py3-none-any.whl size=12418 sha256=313240919f44cfc7c5ccec5afaa1ca0ccd56e953a205b4a7ecfdcd22a9e35d96\n",
      "  Stored in directory: /root/.cache/pip/wheels/c6/93/bb/d3037d4200a03bf09725c786ecc28c8e1e3c78d66baadef7bf\n",
      "Successfully built urlexpander unshortenit\n",
      "Installing collected packages: requests-file, unshortenit, tldextract, urlexpander\n",
      "Successfully installed requests-file-1.5.1 tldextract-3.1.2 unshortenit-0.4.0 urlexpander-0.0.37\n"
     ]
    }
   ],
   "source": [
    "!pip install urlexpander"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bg2aPPy1HrJT",
    "outputId": "4a241073-c1b5-4afb-8c53-2eab02906717"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This notebook is using urlExpander v0.0.36\n"
     ]
    }
   ],
   "source": [
    "import urlexpander\n",
    "print(f\"This notebook is using urlExpander v{urlexpander.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H67bNRO1HrJV"
   },
   "source": [
    "Here is a toy example of some URLs taken from Congressional Twitter accounts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "PCJyrmzTHrJW"
   },
   "outputs": [],
   "source": [
    "urls = [\n",
    "    'https://trib.al/xXI5ruM',\n",
    "    'http://bit.ly/1Sv81cj',\n",
    "    'https://www.youtube.com/watch?v=8NwKcfXvGl4',\n",
    "    'https://t.co/zNU1eHhQRn',\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BEORdNXQHrJW"
   },
   "source": [
    "We can use the `expand` function (see the code) to unshorten any link:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vAk0SHgzHrJX",
    "outputId": "c3004b85-3d4f-47a9-a341-c2bec7a13c87"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://www.breitbart.com/video/2017/12/31/lindsey-graham-trump-just-cant-tweet-iran/'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urlexpander.expand(urls[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o4Cry4NgHrJX"
   },
   "source": [
    "It also works on any list of URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AXldCOJPHrJY",
    "outputId": "7b55454c-6633-4219-e0fa-00d7cf37c5ae"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://www.breitbart.com/video/2017/12/31/lindsey-graham-trump-just-cant-tweet-iran/',\n",
       " 'http://www.billshusterforcongress.com/__CONNECTIONPOOL_ERROR__',\n",
       " 'https://www.youtube.com/watch?v=8NwKcfXvGl4',\n",
       " 'http://www.nfib.com/content/press-release/elections/small-business-endorses-shuster-for-reelection-73730/?utm_campaign=Advocacy&utm_source=Twitter&utm_medium=Social']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urlexpander.expand(urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that URLs that resolve to defunct pages, still return the domain name -- followed by the type of error surrounded by two underscores IE `http://www.billshusterforcongress.com/__CONNECTIONPOOL_ERROR__`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FbNjVYv6HrJZ"
   },
   "source": [
    "### Checking for Shortened Links\n",
    "To save compute time, we can skip links that don't need to be expanded.<br>\n",
    "The `is_short` function takes any url and checks if the domain is from a known list of link shorteners"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0bNMQx3-HrJZ",
    "outputId": "7deda174-dced-4212-d96f-e55cabab832b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://bit.ly/1Sv81cj returns: True\n",
      "https://www.youtube.com/watch?v=8NwKcfXvGl4 returns: False\n"
     ]
    }
   ],
   "source": [
    "print(f\"{urls[1]} returns: {urlexpander.is_short(urls[1])}\")\n",
    "print(f\"{urls[2]} returns: {urlexpander.is_short(urls[2])}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mMhwsLrEHrJb"
   },
   "source": [
    "urlExpander takes advantage of a list of known domains that offer link shortening services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UnvIW0aOHrJb",
    "outputId": "df90d7b7-6c8a-4901-8e8b-5be7848bcdb2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['sh.st', 'adf.ly', 'lnx.lu', 'adfoc.us', 'j.gs', 'q.gs', 'u.bb', 'ay.gy', 'atominik.com', 'tinyium.com', 'microify.com', 'linkbucks.com', 'www.linkbucks.com', 'jzrputtbut.net', 'any.gs', 'cash4links.co', 'cache4files.co', 'dyo.gs', 'filesonthe.net', 'goneviral.com', 'megaline.co', 'miniurls.co', 'qqc.co', 'seriousdeals.net', 'theseblogs.com', 'theseforums.com', 'tinylinks.co', 'tubeviral.com', 'ultrafiles.net', 'urlbeat.net', 'whackyvidz.com', 'yyv.co', 'href.li', 'anonymz.com', 'festyy.com', 'ceesty.com', 'tiny.cc', 'dlvr.it', 'bit.ly', 'buff.ly', 'ow.ly', 'goo.gl', 'shar.es', 'ift.tt', 'fb.me', 'washex.am', 'smq.tc', 'trib.al', 'is.gd', 'paper.li', 'waa.ai', 'tinyurl.com', 'ht.ly', '1.usa.gov', 'deck.ly', 'bit.do', 'lc.chat', 'urls.tn', 'soo.gd', 's2r.co', 'clicky.me', 'budurl.com', 'bc.vc', 'branch.io', 'capsulink.com', 'ux9.de', 'fuck.it', 't2m.io', 'shrt.li', 'elbo.in', 'shrtfly.com', 'hiveam.com', 'slink.be', 'plu.sh', 'cutt.ly', 'zii.bz', 'munj.pw', 't.co', 'go.usa.gov', 'on.fb.me', 'j.mp', 'amp.twimg.com', 'ofa.bo', 'ln.is', 'linkis.com']\n"
     ]
    }
   ],
   "source": [
    "known_shorteners = urlexpander.constants.all_short_domains.copy()\n",
    "print(known_shorteners)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VQFpsCdhHrJb"
   },
   "source": [
    "#### Adding Known Shorteners\n",
    "You can make modifications or use your own `list_of_domains` as an argument for the`is_short` function or `is_short_domain` (which is faster and operates on the domain-level)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gPf02z-cHrJc",
    "outputId": "120a5440-8d70-4f59-ecf2-bed86e6a49f5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Now https://www.youtube.com/watch?v=8NwKcfXvGl4 returns: True\n"
     ]
    }
   ],
   "source": [
    "known_shorteners += ['youtube.com']\n",
    "print(f\"Now {urls[2]} returns: {urlexpander.is_short(urls[2], list_of_domains=known_shorteners)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c3HIVOU0HrJc"
   },
   "source": [
    "Now we can shorten our workload:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bRK3vUzLHrJd",
    "outputId": "f96107c9-2d6b-4869-85a1-031a7cbb3c70"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://trib.al/xXI5ruM', 'http://bit.ly/1Sv81cj', 'https://t.co/zNU1eHhQRn']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# filter only domains that need to be shortenened\n",
    "urls_to_shorten = [link for link in urls if urlexpander.is_short(link)]\n",
    "urls_to_shorten"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6eqZyr9MHrJd"
   },
   "source": [
    "By default, urlexpander will expand every URL it is shown. However you can pass a boolean function (one the returns True or False based on an inputted string) to the filter_function parameter. Instead of filtering the inputs before running the `expand` function, you can assign a filter using the `filter_function` argument.<br>\n",
    "Filter functions can be any boolean function that operates on a string. Below is an example function that filters for t.co links:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "U9cO6JF-HrJe"
   },
   "outputs": [],
   "source": [
    "def custom_filter(url):\n",
    "    '''This function returns True if the url is a shortened Twitter URL'''\n",
    "    if urlexpander.get_domain(url) == 't.co':\n",
    "        return True\n",
    "    else:\n",
    "        return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1rgRwX7EHrJf",
    "outputId": "b63fa384-4adf-4f03-ff80-5ea4d86ba52b"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/1 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 1 links to unshorten\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:00<00:00,  3.09it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['https://trib.al/xXI5ruM',\n",
       " 'http://bit.ly/1Sv81cj',\n",
       " 'https://www.youtube.com/watch?v=8NwKcfXvGl4',\n",
       " 'http://www.nfib.com/content/press-release/elections/small-business-endorses-shuster-for-reelection-73730/?utm_campaign=Advocacy&utm_source=Twitter&utm_medium=Social']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resolved_links = urlexpander.expand(urls, \n",
    "                                    filter_function=custom_filter, \n",
    "                                    verbose=1)\n",
    "resolved_links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X9--8zr5HrJf"
   },
   "source": [
    "Although filtering within the `expand` function is convenient, you will see changes in performance time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kkZpDO0EHrJf",
    "outputId": "d559f4ac-0e27-42e9-fc25-8b08d63e7677"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/1 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 3 links to unshorten\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:02<00:00,  2.68s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['https://www.breitbart.com/video/2017/12/31/lindsey-graham-trump-just-cant-tweet-iran/',\n",
       " 'http://www.billshusterforcongress.com/__CONNECTIONPOOL_ERROR__',\n",
       " 'https://www.youtube.com/watch?v=8NwKcfXvGl4',\n",
       " 'http://www.nfib.com/content/press-release/elections/small-business-endorses-shuster-for-reelection-73730/?utm_campaign=Advocacy&utm_source=Twitter&utm_medium=Social']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resolved_links = urlexpander.expand(urls,  \n",
    "                                    filter_function=urlexpander.is_short,\n",
    "                                    verbose=1)\n",
    "resolved_links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mdAkqykuHrJg"
   },
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TujDrj9CHrJg"
   },
   "source": [
    "## Congress Tweets Example\n",
    "\n",
    "But that is a toy example, let's see how this fairs with a larger dataset.<br>\n",
    "This package comes with a [sampled dataset](https://github.com/SMAPPNYU/urlExpander/blob/master/urlexpander/core/datasets.py#L8-L29) of links extracted from Twitter accounts from the 115th Congress. <br>\n",
    "If you work with Twitter data you'll be glad to know there is a function `urlexpander.tweet_utils.get_link` for creating a similar dataset from Tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5LvGZhq4HrJg",
    "outputId": "95e662c3-f9a4-4d9d-8e35-72ac32fb6551"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset has 10000 rows\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>link_domain</th>\n",
       "      <th>link_url_long</th>\n",
       "      <th>link_url_short</th>\n",
       "      <th>tweet_created_at</th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>tweet_text</th>\n",
       "      <th>user_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9998</th>\n",
       "      <td>facebook.com</td>\n",
       "      <td>https://www.facebook.com/theDanRather/posts/10...</td>\n",
       "      <td>https://t.co/VOiuOXFi1P</td>\n",
       "      <td>Tue Jun 20 21:36:04 +0000 2017</td>\n",
       "      <td>877278904846888965</td>\n",
       "      <td>RT @DanRather: Nothing I have ever seen approa...</td>\n",
       "      <td>15808765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9999</th>\n",
       "      <td>bit.ly</td>\n",
       "      <td>http://bit.ly/1YWRIXg</td>\n",
       "      <td>https://t.co/Hz8RojBqOy</td>\n",
       "      <td>Tue Dec 08 19:34:38 +0000 2015</td>\n",
       "      <td>674311141527560197</td>\n",
       "      <td>We need to get people off the sidelines &amp;amp; ...</td>\n",
       "      <td>733751245</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       link_domain                                      link_url_long  \\\n",
       "9998  facebook.com  https://www.facebook.com/theDanRather/posts/10...   \n",
       "9999        bit.ly                              http://bit.ly/1YWRIXg   \n",
       "\n",
       "               link_url_short                tweet_created_at  \\\n",
       "9998  https://t.co/VOiuOXFi1P  Tue Jun 20 21:36:04 +0000 2017   \n",
       "9999  https://t.co/Hz8RojBqOy  Tue Dec 08 19:34:38 +0000 2015   \n",
       "\n",
       "                tweet_id                                         tweet_text  \\\n",
       "9998  877278904846888965  RT @DanRather: Nothing I have ever seen approa...   \n",
       "9999  674311141527560197  We need to get people off the sidelines &amp; ...   \n",
       "\n",
       "        user_id  \n",
       "9998   15808765  \n",
       "9999  733751245  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_congress = urlexpander.datasets.load_congress_twitter_links(nrows=10000)\n",
    "\n",
    "print(f'The dataset has {len(df_congress)} rows')\n",
    "df_congress.tail(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Lgl9pyGeHrJg",
    "outputId": "dfe1da74-3daa-4254-b7a5-32e5763faeaf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2796270302787247"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shortened_urls = df_congress[df_congress.link_domain.apply(urlexpander.is_short)].tweet_id.nunique()\n",
    "all_urls = df_congress.tweet_id.nunique()\n",
    "shortened_urls / all_urls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "COoe7lyLHrJg"
   },
   "source": [
    "About 28% of the links are short!<br>\n",
    "The performance of the next script is dependent on your internet connection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-WwdyEE5HrJh",
    "outputId": "d42295e7-7401-4c45-d13c-1d39da7205f6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieving speedtest.net configuration...\n",
      "Testing from New York University (128.122.215.16)...\n",
      "Retrieving speedtest.net server list...\n",
      "Selecting best server based on ping...\n",
      "Hosted by Speedtest.net (New York City, NY) [2.57 km]: 4.263 ms\n",
      "Testing download speed................................................................................\n",
      "Download: 422.94 Mbit/s\n",
      "Testing upload speed......................................................................................................\n",
      "Upload: 320.82 Mbit/s\n"
     ]
    }
   ],
   "source": [
    "!curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "plMfEqt6HrJh"
   },
   "source": [
    "Let's see how long it takes to expand these 10k links.<br>\n",
    "\n",
    "### Multithreading\n",
    "\n",
    "This is where the optional parameters for `expand` shine.\n",
    "We can created multiple threads for requests (using `n_workers`), cache results into a json file (`cache_file`), and chunk the input into smaller pieces (using `chunksize`). Why does this last part matter? Something I noticed when expanding links in mass is that performance degrades over time. Chunking the input prevents this from happening (not sure why though)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cOVvSI3lHrJh",
    "outputId": "3b1c45b7-f3ed-42f4-96c2-1670ce6ccb76"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/1 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 1020 links to unshorten\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:20<00:00, 20.96s/it]\n"
     ]
    }
   ],
   "source": [
    "resolved_links = urlexpander.expand(df_congress['link_url_long'], \n",
    "                                    chunksize=1280,\n",
    "                                    n_workers=64, \n",
    "                                    cache_file='temp.json', \n",
    "                                    verbose=1,\n",
    "                                    filter_function=urlexpander.is_short)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nTDy8IZgHrJh"
   },
   "source": [
    "At SMaPP, the process of link expansion has been a burden on our research.<br>\n",
    "We hope that this software helps you overcome similar obstacles!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MgXHDCw1HrJh",
    "outputId": "da00f78b-6480-4d56-d841-025557a8f40b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>link_domain</th>\n",
       "      <th>link_url_long</th>\n",
       "      <th>link_url_short</th>\n",
       "      <th>tweet_created_at</th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>tweet_text</th>\n",
       "      <th>user_id</th>\n",
       "      <th>expanded_url</th>\n",
       "      <th>resolved_domain</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9998</th>\n",
       "      <td>facebook.com</td>\n",
       "      <td>https://www.facebook.com/theDanRather/posts/10...</td>\n",
       "      <td>https://t.co/VOiuOXFi1P</td>\n",
       "      <td>Tue Jun 20 21:36:04 +0000 2017</td>\n",
       "      <td>877278904846888965</td>\n",
       "      <td>RT @DanRather: Nothing I have ever seen approa...</td>\n",
       "      <td>15808765</td>\n",
       "      <td>https://www.facebook.com/theDanRather/posts/10...</td>\n",
       "      <td>facebook.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9999</th>\n",
       "      <td>bit.ly</td>\n",
       "      <td>http://bit.ly/1YWRIXg</td>\n",
       "      <td>https://t.co/Hz8RojBqOy</td>\n",
       "      <td>Tue Dec 08 19:34:38 +0000 2015</td>\n",
       "      <td>674311141527560197</td>\n",
       "      <td>We need to get people off the sidelines &amp;amp; ...</td>\n",
       "      <td>733751245</td>\n",
       "      <td>http://speakerryan.com/__CLIENT_ERROR__</td>\n",
       "      <td>speakerryan.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       link_domain                                      link_url_long  \\\n",
       "9998  facebook.com  https://www.facebook.com/theDanRather/posts/10...   \n",
       "9999        bit.ly                              http://bit.ly/1YWRIXg   \n",
       "\n",
       "               link_url_short                tweet_created_at  \\\n",
       "9998  https://t.co/VOiuOXFi1P  Tue Jun 20 21:36:04 +0000 2017   \n",
       "9999  https://t.co/Hz8RojBqOy  Tue Dec 08 19:34:38 +0000 2015   \n",
       "\n",
       "                tweet_id                                         tweet_text  \\\n",
       "9998  877278904846888965  RT @DanRather: Nothing I have ever seen approa...   \n",
       "9999  674311141527560197  We need to get people off the sidelines &amp; ...   \n",
       "\n",
       "        user_id                                       expanded_url  \\\n",
       "9998   15808765  https://www.facebook.com/theDanRather/posts/10...   \n",
       "9999  733751245            http://speakerryan.com/__CLIENT_ERROR__   \n",
       "\n",
       "      resolved_domain  \n",
       "9998     facebook.com  \n",
       "9999  speakerryan.com  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_congress['expanded_url'] = resolved_links\n",
    "df_congress['resolved_domain'] = df_congress['expanded_url'].apply(urlexpander.get_domain)\n",
    "df_congress.tail(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jkqwR-tRHrJi"
   },
   "source": [
    "Here are the top 25 shared domains from this sampled Congress dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_Jpwg-jXHrJi",
    "outputId": "82c0ae24-c4af-4276-c584-eb52a5622d16"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "twitter.com               1517\n",
       "house.gov                 1156\n",
       "youtube.com                580\n",
       "facebook.com               524\n",
       "senate.gov                 441\n",
       "instagram.com              176\n",
       "nytimes.com                165\n",
       "washingtonpost.com         157\n",
       "thehill.com                135\n",
       "politico.com                85\n",
       "foxnews.com                 64\n",
       "cnn.com                     64\n",
       "wsj.com                     64\n",
       "twimg.com                   56\n",
       "usatoday.com                46\n",
       "ow.ly                       46\n",
       "washingtonexaminer.com      46\n",
       "huffingtonpost.com          44\n",
       "medium.com                  43\n",
       "speaker.gov                 34\n",
       "healthcare.gov              33\n",
       "gop.gov                     33\n",
       "c-span.org                  33\n",
       "pscp.tv                     31\n",
       "rollcall.com                31\n",
       "Name: resolved_domain, dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_congress.resolved_domain.value_counts().head(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FizZY_MPHrJi"
   },
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rbt_dpTEHrJi"
   },
   "source": [
    "# Bonus Round!\n",
    "You can count number of `resolved_domain`s for each `user_id ` using `count_matrix()`.<br>\n",
    "You can even choose which domains are counted by modifying the `domain_list` arg:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Mz8nb_vPHrJi",
    "outputId": "f5a7897a-7f10-4d30-bc4e-8c1cdf8ca751"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>facebook.com</th>\n",
       "      <th>youtube.com</th>\n",
       "      <th>twitter.com</th>\n",
       "      <th>google.com</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>911302336307490816</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>941000686275387392</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>948946378939609089</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    facebook.com  youtube.com  twitter.com  google.com\n",
       "user_id                                                               \n",
       "911302336307490816             0            0            1           0\n",
       "941000686275387392             1            0            2           0\n",
       "948946378939609089             0            1            0           0"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_matrix = urlexpander.tweet_utils.count_matrix(df_congress,\n",
    "                                                    user_col='user_id', \n",
    "                                                    domain_col='resolved_domain', \n",
    "                                                    unique_count_col='tweet_id',\n",
    "                                                    domain_list=['youtube.com','facebook.com', 'google.com', 'twitter.com'])\n",
    "\n",
    "count_matrix.tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6Y8wRHH6HrJi"
   },
   "source": [
    "One of the domain lists you might be interested in are US national media outlets -\n",
    "`datasets.load_us_national_media_outlets()` compiled by Gregory Eady (Forthcoming)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gUYi1LgvHrJj",
    "outputId": "bbeef644-fff9-40a0-b3a4-4128796d1b2f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['abcnews.go.com', 'aim.org', 'alternet.org',\n",
       "       'theamericanconservative.com', 'prospect.org'], dtype=object)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urlexpander.datasets.load_us_national_media_outlets()[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rsQEvmPPHrJj"
   },
   "source": [
    "<hr>\n",
    "We also built a one-size-fits-all scraper that returns the title, description, and/or paragraphs from any given URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Fg_uGWkyHrJj",
    "outputId": "7f563b69-b60a-4f00-9554-4eb36af2d5f1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Lindsey Graham to Trump: 'You Just Can't Tweet' About Iran | Breitbart\""
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urlexpander.html_utils.get_webpage_title(urls[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yu1FOqSuHrJj",
    "outputId": "134ad2ff-7c79-43b1-95e3-52998051e074"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Sunday CBS\\'s \"Face the Nation,\" while discussing the last several\\xa0days of protests in Iran over\\xa0government corruption, Sen. Lindsey Graham (R-SC) warned | Breitbart TV'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urlexpander.html_utils.get_webpage_description(urls[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GQlOLRgGHrJj",
    "outputId": "ea0a329a-722f-441f-feb7-9b0600de19ef"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('url', 'https://trib.al/xXI5ruM'),\n",
       "             ('title',\n",
       "              \"Lindsey Graham to Trump: 'You Just Can't Tweet' About Iran | Breitbart\"),\n",
       "             ('description',\n",
       "              'Sunday CBS\\'s \"Face the Nation,\" while discussing the last several\\xa0days of protests in Iran over\\xa0government corruption, Sen. Lindsey Graham (R-SC) warned | Breitbart TV'),\n",
       "             ('paragraphs',\n",
       "              ['Sunday CBS’s “Face the Nation,” while discussing the last several\\xa0days of protests in Iran over\\xa0government corruption, Sen. Lindsey Graham (R-SC) warned President Donald Trump that he couldn’t “just tweet” about the protests.',\n",
       "               'Graham said, “The Iranian people are not our enemy. The Ayatollah is the enemy of the world. Here is what I would do if I were President Trump. I would explain what a better deal would look like. It’s not enough to watch. President Trump is tweeting very sympathetically to the Iranian people. But you just can’t tweet here. You have to lay out a plan.”',\n",
       "               '<em><span>Follow Pam Key on Twitter <a href=\"https://twitter.com/pamkeyNEN\">@pamkeyNEN</a> </span></em>',\n",
       "               '<a href=\"https://www.facebook.com/Breitbart\"></a>.',\n",
       "               '<small>Copyright © 2018 Breitbart</small>'])])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urlexpander.html_utils.get_webpage_meta(urls[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "D9FmUwnNHrJk"
   },
   "source": [
    "## Conclusion\n",
    "Thanks for stumbling upon this package, we hope that it will lead to more research around links.<br>\n",
    "We're working on some projects in thie vein and would love to know if you are too!\n",
    "\n",
    "As an open source package, please feel to reach out about bugs, feature requests, or collaboration!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of quickstart.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}