scrape_canyons.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMAPap4O6RNtvecfyvliRAm",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/17jmumford/scrape-canyons/blob/main/scrape_canyons.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Welcome!\n",
        "This is a web scraper designed to collect data from the popular canyoneering website [bluugnome.com](https://www.bluugnome.com). It is currently a work in progress. This was built with assistance from chatGPT.\n",
        "###How to use this file\n",
        "1. Select the 'open in colab' button to open the file in Google Colab. \n",
        "1. Hit the play buttons on all the boxes, in order.\n",
        "1. Click the folder button on the left side.\n",
        "1. You should see 'canyoneering_data.csv' on a list. (hit the folder with a refresh button if it's not popping up right away).\n",
        "1. Hover over the file and three dots should appear. Click on the three dots and select 'download'. \n",
        "\n",
        "Warning: these instructions are for the completed web scraper. Data has errors and requires manual correction. Users should carefully research canyon beta on the source websites and take all precautions before going in a canyon. Canyoneering is dangerous!"
      ],
      "metadata": {
        "id": "w1bFiiI1IAiP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### imports"
      ],
      "metadata": {
        "id": "aiHVM3fe6kc2"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {
        "id": "nHqwZLgbivnt"
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "import re\n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Site Scraper"
      ],
      "metadata": {
        "id": "8T7fDJ9E6q3V"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Scraping the parent page for child pages"
      ],
      "metadata": {
        "id": "mlkoFOFjClic"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "parentSoup = BeautifulSoup(requests.get('http://www.bluugnome.com/cyn_route/canyon-area-list.aspx').text, 'html.parser')\n",
        "routes = parentSoup.find(\"div\", attrs={\"id\": \"div_content\"})\n",
        "\n",
        "regionLinks = []\n",
        "regionStart = 'http://www.bluugnome.com/cyn_route/'\n",
        "\n",
        "for link in routes.find_all('a'):\n",
        "  regionLinks.append((regionStart + link.get('href')))\n"
      ],
      "metadata": {
        "id": "3SU5rbfi_9FV"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Scraping an individual page"
      ],
      "metadata": {
        "id": "KuT753v4LFHm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "hrefStart = 'bluugnome.com/cyn_route/dv/'\n",
        "\n",
        "def addRegion(link, data):\n",
        "  print('Current page: ', link)\n",
        "  pageSoup = BeautifulSoup(requests.get(link).text, 'html.parser')\n",
        "  \n",
        "  title = pageSoup.find('h1', {'class' : 'h_general_main_title'}).text.split(':')[1].split(',')\n",
        "  if len(title) > 1:\n",
        "    state = title[1].strip().replace('.', '').replace('\\n', ' ').replace('\\r', '').replace('\\t', '')\n",
        "    region = title[0].strip().replace('\\n', ' ').replace('\\r', '').replace('\\t', '')\n",
        "  else: \n",
        "    state = 'ERROR'\n",
        "    region = 'ERROR'\n",
        "  area = '-'\n",
        "  canyons = pageSoup.find_all(\"div\", {\"class\": \"div_tr-listing\"})\n",
        "  for canyon in canyons:\n",
        "\n",
        "    data[\"State\"].append(state)\n",
        "    data[\"Region\"].append(region)\n",
        "    data['Canyon'].append(canyon.find('a').text.strip().replace('\\n', ' ').replace('\\r', '').replace('\\t', '')) \n",
        "    data['Link'].append((hrefStart + canyon.find('a').get('href')))\n",
        "    \n",
        "    if canyon.previous_sibling.previous_sibling.name == 'h2':\n",
        "      area = canyon.previous_element.previous_element.strip().replace('\\n', ' ').replace('\\r', '').replace('\\t', '')\n",
        "    data['Area'].append(area)\n",
        "\n",
        "    rating = canyon.find(\"h3\", class_=\"h_tr-listing\").text.strip().replace('\\n', ' ').replace('\\r', '').replace('\\t', '')\n",
        "    index = 0\n",
        "\n",
        "    for start in ['1', '2', '3', '4']:\n",
        "      if (rating.find(start) + 1):\n",
        "        rating = rating[rating.find(start):]\n",
        "        break\n",
        "\n",
        "    appendRating('Difficulty', rating, ['4', '3', '2', '1'])\n",
        "    appendRating('Wetness', rating, ['C', 'B', 'A'])\n",
        "    appendRating('Time', rating, ['IV', 'VI', 'V', 'III', 'II', 'I'])\n",
        "    if 'RK' in rating:\n",
        "      data['Risk'].append('')\n",
        "    elif 'R' in rating:\n",
        "      data['Risk'].append('R')\n",
        "    elif 'X' in rating:\n",
        "      data['Risk'].append('X')\n",
        "    else:\n",
        "      data['Risk'].append('')\n",
        "    trails = canyon.find_all(\"p\", {\"class\": \"p_tr-listing\"})\n",
        "    if len(trails) == 0:\n",
        "      trails = canyon.find_all(\"p\", {\"class\": \"p_general\"})\n",
        " \n",
        "    repeat = False\n",
        "    counter = 1\n",
        "    for trail in trails:\n",
        "      if trail.find('strong'):\n",
        "        if len(trails) > 1:\n",
        "          data['Route'].append(('Route #' + str(counter)))\n",
        "        else:\n",
        "          data['Route'].append('Main route')\n",
        "        if counter > 1:\n",
        "          data[\"State\"].append(data[\"State\"][-1])\n",
        "          data[\"Region\"].append(data[\"Region\"][-1])\n",
        "          data[\"Area\"].append(data[\"Area\"][-1])\n",
        "          data[\"Canyon\"].append(data[\"Canyon\"][-1])\n",
        "          data[\"Link\"].append(data['Link'][-1])\n",
        "          data[\"Difficulty\"].append(data[\"Difficulty\"][-1])\n",
        "          data[\"Wetness\"].append(data[\"Wetness\"][-1])\n",
        "          data[\"Time\"].append(data[\"Time\"][-1])\n",
        "          data[\"Risk\"].append(data[\"Risk\"][-1])\n",
        "\n",
        "        appendSibling('Distance', trail)\n",
        "        appendSibling('Shuttle', trail)\n",
        "        appendSibling('Vehicle', trail)\n",
        "        appendSibling('Permit', trail)\n",
        "        appendSibling('Elevation', trail)\n",
        "\n",
        "        rappel_string = trail.find('strong', string='Rappels -')\n",
        "        if rappel_string:\n",
        "          pattern = re.compile(r'(\\d+).*?(\\d+)')\n",
        "          match = re.search(pattern, rappel_string.next_sibling)\n",
        "          if match:\n",
        "            data['Rappels'].append(int(match.group(1)))      \n",
        "            data['Max Rappel'].append(int(match.group(2)))\n",
        "          else:\n",
        "            data['Rappels'].append(0)\n",
        "            data['Max Rappel'].append(0)\n",
        "        else:\n",
        "          data['Rappels'].append(0)\n",
        "          data['Max Rappel'].append(0)\n",
        "        counter += 1\n",
        "    #troubleshoot(data)\n",
        "\n",
        "\n",
        "def appendSibling(cName, routeInfo):\n",
        "  found = routeInfo.find('strong', string=re.compile(cName + \"(.*)\"))\n",
        "  if found:\n",
        "      try:\n",
        "        data[cName].append(found.next_sibling.strip().replace('\\n', ' ').replace('\\r', '').replace('\\t', ''))\n",
        "      except Exception as e:\n",
        "        print(e)\n",
        "        print(cName)\n",
        "        print(routeInfo)\n",
        "        data[cName].append('ERROR')\n",
        "  else:\n",
        "    data[cName].append('ERROR')\n",
        "\n",
        "def appendRating(rName, rating, options):\n",
        "  optionMissing = True\n",
        "  for option in options:\n",
        "    if option in rating:\n",
        "      data[rName].append(option)\n",
        "      optionMissing = False\n",
        "      break\n",
        "  if optionMissing:\n",
        "    data[rName].append('ERROR')"
      ],
      "metadata": {
        "id": "E-vCaImcPyZ1"
      },
      "execution_count": 53,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Looping through all the child pages and exporting the data"
      ],
      "metadata": {
        "id": "81RLCiSjFzPq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data = {\n",
        "    'State' : [],\n",
        "    'Region' : [],\n",
        "    'Area' : [],\n",
        "    'Canyon' : [],\n",
        "    'Route' : [],\n",
        "    'Difficulty' : [],\n",
        "    'Wetness' : [], \n",
        "    'Time' : [],\n",
        "    'Risk' : [],\n",
        "    'Distance' : [],\n",
        "    'Rappels' : [],\n",
        "    'Max Rappel' : [],\n",
        "    'Elevation' : [],\n",
        "    'Shuttle' : [],\n",
        "    'Vehicle' : [],\n",
        "    'Permit' : [],\n",
        "    'Link' : []\n",
        "}\n",
        "\n",
        "for regionLink in regionLinks:\n",
        "  addRegion(regionLink, data)\n",
        "\n",
        "df = pd.DataFrame(data)\n",
        "df.to_csv(r'canyon_data.csv', index=False)\n"
      ],
      "metadata": {
        "id": "vdnqFhmIFjhc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0105f169-a417-4f31-9e64-7f4a5cb724c9"
      },
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Current page:  http://www.bluugnome.com/cyn_route/coconino-nat-frst/canyon-routes__coconino-nat-frst.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/vr-gorge/canyon-routes__vr-gorge.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/dv/canyon-routes__dv.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/red-rock_nv/canyon-routes__red-rock_nv.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/charleston_nv/canyon-routes__charleston_nv.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/lake-mead/canyon-routes__lake-mead.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/stateline-hills_nv/canyon-routes__stateline-hills_nv.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/valley-of-fire/canyon-routes__valley-of-fire.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/cap-reef/canyon-routes__cap-reef.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/cedar-mesa/canyon-routes__cedar-mesa.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/dirty-devil/canyon-routes__dirty-devil.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/dixie-nat-frst-pine-vly/canyon-routes__dixie-nat-frst-pine-vly_uth.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/dixie-nat-frst-cedar-cty/canyon-routes__dixie-nat-frst-cedar-cty.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/escalante/canyon-routes__escalante.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/goblin-vly/canyon-routes__goblin-vly.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/halls-creek/canyon-routes__halls-creek.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/north-wash/canyon-routes__north-wash.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/poison-springs/canyon-routes__poison-srpings.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/robbers-roost/canyon-routes__robbers-roost.aspx\n",
            "'NoneType' object is not callable\n",
            "Distance\n",
            "<p class=\"p_tr-listing\">\n",
            "<strong>Time Required<br/>    Exit Option 1 - Exiting up the East Fork Exit </strong>- \n",
            "\t\t\t\t5 to 7 hours<br/>    <strong>Exit Option 2 - Exit by \n",
            "\t\t\t\tclimbing up the Main Fork </strong>- 6 to 8 hours<br/><strong>\n",
            "\t\t\t\tDistance</strong><br/><strong>   \n",
            "\t\t\t\tExit Option 1 - Exiting up the East Fork Exit </strong>- 9.4 miles Total, \n",
            "\t\t\t\t2.3 miles Technical<br/>    <strong>Exit Option 2 - Exit by \n",
            "\t\t\t\tclimbing up the Main Fork -</strong> 10.6 miles Total, 2.8 miles \n",
            "\t\t\t\tTechnical<strong><br>\n",
            "\t\t\t\tRappels -</br></strong> 2 Rappels, Longest Rappel 40 feet.  \n",
            "\t\t\t\t(If bypassing rap 1 - longest rap is 25 feet)<br>\n",
            "<strong>Elevation Range -</strong> 5940 - 5230 Feet<br>\n",
            "<strong>Shuttle Required -</strong> No Miles<strong> Vehicle -</strong> High Clearance \n",
            "\t\t\t\trecommended, 4WD if bad road conditions\n",
            "\t\t\t\t<strong>Permit Required -</strong> No</br></br></p>\n",
            "'NoneType' object is not callable\n",
            "Distance\n",
            "<p class=\"p_tr-listing\">\n",
            "<strong>Time Required<br/>    Exit Option 1 - Exiting up the East Fork Exit </strong>- \n",
            "\t\t\t\t4 to 6 hours<br/>    <strong>Exit Option 2 - Exit by \n",
            "\t\t\t\treversing your route climbing back up the Main Fork </strong>- 5 to \n",
            "\t\t\t\t7 hours<br/><strong>\n",
            "\t\t\t\tDistance</strong><br/><strong>   \n",
            "\t\t\t\tExit Option 1 - Exiting up the East Fork Exit </strong>- 9.7 \n",
            "\t\t\t\tmiles Total, 0.5 miles Technical<br/>    <strong>Exit Option 2 - Exit by \n",
            "\t\t\t\treversing your route climbing back up the Main Fork -</strong>  \n",
            "\t\t\t\t9.2 miles Total, 0.5 miles \n",
            "\t\t\t\tTechnical<strong><br>\n",
            "\t\t\t\tRappels -</br></strong> 0 Mandatory rappels, (there are downclimbs \n",
            "\t\t\t\tthat some will want to rappel Bring rope and webbing just in \n",
            "\t\t\t\tcase)<br>\n",
            "<strong>Elevation Range -</strong> 5940 - 5230 Feet<br>\n",
            "<strong>Shuttle Required -</strong> No Miles<strong> Vehicle -</strong> High Clearance \n",
            "\t\t\t\trecommended, 4WD if bad road conditions\n",
            "\t\t\t\t<strong>Permit Required -</strong> No\t\t\t\t</br></br></p>\n",
            "Current page:  http://www.bluugnome.com/cyn_route/san-raf-swell/canyon-routes__san-raf-swell.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/ticaboo/canyon-routes__ticaboo.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/trachyte/canyon-routes__trachyte.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/zion/canyon-routes__zion.aspx\n",
            "Current page:  http://www.bluugnome.com/cyn_route/canyon-routes__misc.aspx\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#troubleshooting code\n",
        "def troubleshoot(data):\n",
        "  print(len(data['State']))\n",
        "  print(len(data['Region']))\n",
        "  print(len(data['Area']))\n",
        "  print(len(data['Canyon']))\n",
        "  print(len(data['Route']))\n",
        "  print(len(data['Difficulty']))\n",
        "  print(len(data['Wetness' ]))\n",
        "  print(len(data['Time']))\n",
        "  print(len(data['Distance']))\n",
        "  print(len(data['Rappels']))\n",
        "  print(len(data['Max Rappel']))\n",
        "  print(len(data['Elevation']))\n",
        "  print(len(data['Shuttle']))\n",
        "  print(len(data['Vehicle']))\n",
        "  print(len(data['Permit']))\n",
        "  print(len(data['Link']))"
      ],
      "metadata": {
        "id": "F7aKWJCtRdVg"
      },
      "execution_count": 159,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Experimentation block\n",
        "(delete before final version)"
      ],
      "metadata": {
        "id": "s3c-aIaO6SNZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "'''\n",
        "<h1 class=\"h_general_main_title\">\n",
        "       Canyoneering Route Description List:\n",
        "       <br/>\n",
        "       Coconino National Forest, Arizona.\n",
        "      </h1>\n",
        "'''\n",
        "title = experimentSoup.find('h1', {'class' : 'h_general_main_title'}).text.split(':')[1].split(',')\n",
        "\n",
        "print(title[1].strip().replace('.', ''))\n",
        "print(title[0].strip())"
      ],
      "metadata": {
        "id": "gclZ7IpAk2BT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d00cfb85-d223-4fc5-d491-52ec17ee8d4a"
      },
      "execution_count": 133,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Arizona\n",
            "Coconino National Forest\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Using beautiful Soup\n",
        "\n",
        "Basic info on beautiful soup for those wishing to build something similar. (points by chatGPT)\n",
        "\n",
        "* Use the find() and find_all() methods to locate the specific elements you want to extract. These methods accept a variety of arguments, such as the name of the tag (e.g. 'p' for paragraphs), the class name (if the element has one), and the id (if the element has one).\n",
        "\n",
        "* Use the string attribute to extract the text content of an element. This attribute returns the text between the opening and closing tags of the element, without the tags themselves.\n",
        "\n",
        "* Use the .contents and .children attributes to navigate the HTML tree and extract information from nested elements. The .contents attribute returns a list of the element's children, while the .children attribute returns an iterator over the element's children.\n",
        "\n",
        "* Use the .parent attribute to access the parent of an element. This can be helpful when you want to extract information from the parent element as well as the element itself."
      ],
      "metadata": {
        "id": "jP-oZXJVk4iO"
      }
    }
  ]
}