part1.Segmenting and Clustering Neighborhoods in Toronto.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {
                "collapsed": true
            },
            "source": "\n## Problem 1\nUse the Notebook to build the code to scrape the following Wikipedia page, https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M, in order to obtain the data that is in the table of postal codes and to transform the data into a pandas dataframe.\n\nTo create the above dataframe:\n\n* The dataframe will consist of three columns: PostalCode, Borough, and Neighborhood\n* Only process the cells that have an assigned borough. Ignore cells with a borough that is Not assigned.\n* More than one neighborhood can exist in one postal code area. For example, in the table on the Wikipedia page, you will notice that M5A is listed twice and has two neighborhoods: Harbourfront and Regent Park. These two rows will be combined into one row with the neighborhoods separated with a comma as shown in row 11  in the above table.\n* If a cell has a borough but a Not assigned  neighborhood, then the neighborhood will be the same as the borough.\n* Clean your Notebook and add Markdown cells to explain your work and any assumptions you are making.\n* In the last cell of your notebook, use the .shape method to print the number of rows of your dataframe."
        },
        {
            "cell_type": "code",
            "execution_count": 127,
            "metadata": {},
            "outputs": [],
            "source": "import pandas as pd # library for data analsysis\npd.set_option('display.max_columns', None)\npd.set_option('display.max_rows', None)\n\nimport json # library to handle JSON files\n\nimport requests # library to handle requests\n\nfrom bs4 import BeautifulSoup\nimport xml"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "\nScrape the List of postal codes of Canada"
        },
        {
            "cell_type": "code",
            "execution_count": 128,
            "metadata": {},
            "outputs": [],
            "source": "List_url = \"https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M\"\nsource = requests.get(List_url).text"
        },
        {
            "cell_type": "code",
            "execution_count": 129,
            "metadata": {},
            "outputs": [],
            "source": "soup = BeautifulSoup(source, 'xml')"
        },
        {
            "cell_type": "code",
            "execution_count": 130,
            "metadata": {},
            "outputs": [],
            "source": "table=soup.find('table')"
        },
        {
            "cell_type": "code",
            "execution_count": 131,
            "metadata": {},
            "outputs": [],
            "source": "#dataframe will consist of three columns: PostalCode, Borough, and Neighborhood\ncolumn_names = ['Postalcode','Borough','Neighborhood']\ndf = pd.DataFrame(columns = column_names)"
        },
        {
            "cell_type": "code",
            "execution_count": 132,
            "metadata": {},
            "outputs": [],
            "source": "# Search all the postcode, borough, neighborhood \ntable_contents=[]\ntable=soup.find('table')\nfor row in table.findAll('td'):\n    cell = {}\n    if row.span.text=='Not assigned':\n        pass\n    else:\n        cell['PostalCode'] = row.p.text[:3]\n        cell['Borough'] = (row.span.text).split('(')[0]\n        cell['Neighborhood'] = (((((row.span.text).split('(')[1]).strip(')')).replace(' /',',')).replace(')',' ')).strip(' ')\n        table_contents.append(cell)\n\n# print(table_contents)\ndf=pd.DataFrame(table_contents)\ndf['Borough']=df['Borough'].replace({'Downtown TorontoStn A PO Boxes25 The Esplanade':'Downtown Toronto Stn A',\n                                             'East TorontoBusiness reply mail Processing Centre969 Eastern':'East Toronto Business',\n                                             'EtobicokeNorthwest':'Etobicoke Northwest','East YorkEast Toronto':'East York/East Toronto',\n                                             'MississaugaCanada Post Gateway Processing Centre':'Mississauga'})"
        },
        {
            "cell_type": "code",
            "execution_count": 133,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>PostalCode</th>\n      <th>Borough</th>\n      <th>Neighborhood</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>M3A</td>\n      <td>North York</td>\n      <td>Parkwoods</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>M4A</td>\n      <td>North York</td>\n      <td>Victoria Village</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>M5A</td>\n      <td>Downtown Toronto</td>\n      <td>Regent Park, Harbourfront</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>M6A</td>\n      <td>North York</td>\n      <td>Lawrence Manor, Lawrence Heights</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>M7A</td>\n      <td>Queen's Park</td>\n      <td>Ontario Provincial Government</td>\n    </tr>\n  </tbody>\n</table>\n</div>",
                        "text/plain": "  PostalCode           Borough                      Neighborhood\n0        M3A        North York                         Parkwoods\n1        M4A        North York                  Victoria Village\n2        M5A  Downtown Toronto         Regent Park, Harbourfront\n3        M6A        North York  Lawrence Manor, Lawrence Heights\n4        M7A      Queen's Park     Ontario Provincial Government"
                    },
                    "execution_count": 133,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": "\ndf.head(5)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Looking to the size of dataframe"
        },
        {
            "cell_type": "code",
            "execution_count": 134,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": "(103, 3)"
                    },
                    "execution_count": 134,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": "df.shape"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": ""
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.7",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.7.10"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 1
}