url_collector.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e067d759",
   "metadata": {},
   "source": [
    "# Olympics Athletes (1896 - 2022) - URL Collector\n",
    "___\n",
    "\n",
    "## About this file\n",
    "Before the web scraping project begins, this file serves as a collector of biographical URLs for athletes from the [Olympedia website](https://www.olympedia.org). The purpose is to assemble a list of URLs in advance to circumvent the need for traversing multiple nested request loops during the scraping process. This strategic approach optimizes the time required for the scraping stage. To gather information about athletes who participated in the Olympics, we will follow these steps:\n",
    "\n",
    "- **STEP 1**: Retrieve the URLs of all countries that competed in the Olympics.\n",
    "\n",
    "- **STEP 2**: Acquire information about all the games in which each country participated (referred to as \"events\" on the Olympedia website).\n",
    "\n",
    "- **STEP 3**: Collect the URLs of all athletes who participated in these events.\n",
    "\n",
    "The list of athlete URLs is saved in a JSON file named `athletes_urls.json`.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "___\n",
    "### Import necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "376a9c89",
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import sleep, perf_counter\n",
    "import concurrent.futures\n",
    "from bs4 import BeautifulSoup \n",
    "import requests\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cacf85fb",
   "metadata": {},
   "source": [
    "### Define Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "400c1ad4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def flatten_list(list_of_lists):\n",
    "    \"\"\"\n",
    "    This function flattens the list of lists, such that [[a1,a2,a3], [b1,b2], ..., [z1,z2,z3]] becomes [a1, a2, ... , z1, z3].\n",
    "    \n",
    "    :param [list_of_lists]: List of lists to be flattened\n",
    "    :type [list_of_lists]: list\n",
    "     \n",
    "    :return : Flattened list\n",
    "    :rtype: list\n",
    "    \"\"\"\n",
    "    flatten_list = []\n",
    "    for the_list in list_of_lists:\n",
    "        for element in the_list:\n",
    "            flatten_list.append(element)\n",
    "    return flatten_list\n",
    "\n",
    "\n",
    "\n",
    "def get_event_urls(country_url):\n",
    "    \"\"\"\n",
    "    This function retrieves the list of urls of all the games in which each country participated \n",
    "    (referred to as \"events\" on the Olympedia website).\n",
    "    \n",
    "    :param [countr_url]: URL of the country that participated\n",
    "    :type [country_url]: String\n",
    "    \n",
    "    :return : list of URLS of all the games \n",
    "    :rtype : list \n",
    "    \"\"\"\n",
    "\n",
    "    # Fetch the url (skip if not found)\n",
    "    response = session_event.get(country_url)\n",
    "    if response.status_code != 200:\n",
    "        print(f\"Error fetching {country_url}\")\n",
    "        return\n",
    "    \n",
    "    \n",
    "    # Get all games in which the country participated\n",
    "    events_urls = []\n",
    "    country_page = BeautifulSoup(response.content, 'lxml')\n",
    "    if country_page.find('tbody') is not None:\n",
    "        table_games = country_page.find('tbody').find_all('tr')\n",
    "        for game in table_games:\n",
    "            game_url = base_url + game.find_all('a')[1]['href']            \n",
    "            events_urls.append(game_url)\n",
    "            \n",
    "    return events_urls\n",
    "\n",
    "\n",
    "\n",
    "def get_athletes_urls(event_url):\n",
    "    \"\"\"\n",
    "    This function retrieves all the URLs of Olympic athletes.\n",
    "    \n",
    "    :param [event_url]: URL of the event\n",
    "    :type [event_url]: str\n",
    "    \n",
    "    :return: List of URLs of all athletes who participated in the event.\n",
    "    :rtype: list\n",
    "    \"\"\"\n",
    "\n",
    "    # Fetch the event URL (skip if not found)\n",
    "    response = requests.get(event_url)\n",
    "    if response.status_code != 200:\n",
    "        print(f\"Error fetching: {event_url}\")\n",
    "        return\n",
    "    \n",
    "                \n",
    "    # Extract the athlele links and append to the list\n",
    "    athletes_urls = []\n",
    "    game_page = BeautifulSoup(response.content, 'lxml')\n",
    "    table_athletes = game_page.find('tbody').find_all('a')\n",
    "    for row in table_athletes:\n",
    "        if 'athlete' in row['href']:\n",
    "            athletes_urls.append(base_url + row['href'])\n",
    "            \n",
    "    sleep(1)\n",
    "    return athletes_urls"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c707d13",
   "metadata": {},
   "source": [
    "### Step 1: Get the URLs of all coutries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c99ad8cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Country URLs successfully fetched\n",
      "Number of participated countries: 232\n"
     ]
    }
   ],
   "source": [
    "athletes_links = []\n",
    "base_url = \"https://www.olympedia.org\"\n",
    "initial_url = \"https://www.olympedia.org/countries\"\n",
    "\n",
    "\n",
    "countries_urls = []\n",
    "response = requests.get(initial_url)\n",
    "if response.status_code == 200:\n",
    "    \n",
    "    page = BeautifulSoup(response.content, \"lxml\")\n",
    "    table_countries = page.find('tbody').find_all('tr')\n",
    "    for country in table_countries:\n",
    "        \n",
    "        # Competed in Modern Olympics\n",
    "        if country.find(attrs = {'class': 'glyphicon glyphicon-ok'}) and country.find('td').text != 'MIX':\n",
    "            country_url = base_url + country.find_all('a')[1]['href'] \n",
    "            countries_urls.append(country_url)\n",
    "            \n",
    "    print('Country URLs successfully fetched')\n",
    "    print('Number of participated countries:', len(countries_urls))\n",
    "        \n",
    "else:\n",
    "    print(response.raise_for_status())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b23c03a",
   "metadata": {},
   "source": [
    "### Step 2: Get events URLs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5e4b3f7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed Time: 38.57 seconds\n"
     ]
    }
   ],
   "source": [
    "start = perf_counter()\n",
    "\n",
    "with requests.Session() as session_event:\n",
    "    with concurrent.futures.ThreadPoolExecutor(max_workers=50) as executor:\n",
    "        all_events_urls = list(executor.map(get_event_urls, countries_urls))\n",
    "\n",
    "end = perf_counter()\n",
    "print(f\"Elapsed Time: {end-start:.02f} seconds\")\n",
    "\n",
    "\n",
    "# Flatten the list of URLs\n",
    "flatten_events_urls = flatten_list(all_events_urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64b5b72e",
   "metadata": {},
   "source": [
    "### Step 3: Get athletes URLs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "58812c9b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed Time: 426.60 seconds\n"
     ]
    }
   ],
   "source": [
    "start = perf_counter()\n",
    "\n",
    "with concurrent.futures.ThreadPoolExecutor(max_workers=50) as executor:\n",
    "    all_athletes_urls = list(executor.map(get_athletes_urls, flatten_events_urls))\n",
    "\n",
    "end = perf_counter()\n",
    "print(f\"Elapsed Time: {end-start:.02f} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6be52a6",
   "metadata": {},
   "source": [
    "Athletes can participate in multiple Olympic Games. Therefore, before crawling athlete webpages to scrape data, it's more practical to remove duplicates in order to optimize computational resources and avoid requesting the same webpages multiple times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "372b5435",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are URLs duplicated - Proceed to remove duplicates.\n",
      "Number of unique athletes:  155665\n"
     ]
    }
   ],
   "source": [
    "# Flatten the list of URLs\n",
    "flatten_athletes_urls = flatten_list(all_athletes_urls)\n",
    "\n",
    "\n",
    "# Check for duplicates\n",
    "if len(flatten_athletes_urls) > len(set(flatten_athletes_urls)):\n",
    "    print('There are URLs duplicated - Proceed to remove duplicates.')\n",
    "    unique_athletes_urls = list(set(flatten_athletes_urls))\n",
    "    print('Number of unique athletes: ', len(unique_athletes_urls))\n",
    "    \n",
    "else:\n",
    "    print('There is no duplicate.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "700ed223",
   "metadata": {},
   "source": [
    "### Step 4: Save all links into JSON file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3b231d57",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('raw_data/athletes_urls.json', 'w') as file:\n",
    "   json.dump(unique_athletes_urls, file)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}