homework_04-.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from nose.tools import *\n",
    "\n",
    "from mpl_toolkits.basemap import Basemap\n",
    "\n",
    "import networkx as nx\n",
    "# Write your imports here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Spatial Data and Networks Lab\n",
    "## Preparing data, extracting features, visualization and modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab we'll explore terrorism, as described in [this](https://www.kaggle.com/START-UMD/gtd) dataset.\n",
    "\n",
    "The full codebook, describing all variables, their values, the data collection process, and many more details, is located [here](http://start.umd.edu/gtd/downloads/Codebook.pdf).\n",
    "\n",
    "We'll perform our usual process:\n",
    "* Reading and inspecting the data\n",
    "* Validating the data\n",
    "* Exploring the data\n",
    "* Creating inferences and getting to conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 1. Read the dataset (1 point)\n",
    "The dataset is provided in the `data` folder. Read it into Python. You'll likely get an encoding error, so you can use `encoding = \"cp1252\"`.\n",
    "\n",
    "Use the column `eventid` as the index, since this is a good unique identifier and it's used in the dataset (e.g. the column `related` contains the `eventid` of all related crimes).\n",
    "\n",
    "You'll see the dataset has a lot of columns. For this lab, we aren't going to need all of them, only a few. Select these columns (and rename them as described in the tests below):\n",
    "* eventid\n",
    "* iyear, imonth, iday\n",
    "* country_txt\n",
    "* region_txt\n",
    "* multiple (1 if it's a part of a multiple incident and 0 otherwise)\n",
    "* latitude, longitude\n",
    "* targtype1_txt (first target type)\n",
    "* weaptype1_text (weapon type)\n",
    "* gname (perpetrator group name)\n",
    "* nkill (total number of fatalities)\n",
    "* related (IDs of the related incidents)\n",
    "\n",
    "You'll have a total of 12 columns (the eventid which will become an index is not considered a typical column in a dataframe) and just over 170 thousand records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "d40e4c21756633b9aa5581db79f80450",
     "grade": false,
     "grade_id": "cell-1c744200bd2b910d",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "terrorism_data = pd.read_csv(\"data/globalterrorismdb.csv\", encoding=\"cp1252\",index_col=\"eventid\")\n",
    "terrorism_data = terrorism_data[[\"iyear\",\"imonth\",\"iday\",\"country_txt\",\"region_txt\",\"multiple\",\n",
    "                                 \"latitude\",\"longitude\",\"targtype1_txt\",\"weaptype1_txt\",\"gname\", \"nkill\",\"related\"]]\n",
    "\n",
    "terrorism_data.columns = [\"year\",\"month\",\"day\",\"country\",\"region\",\"multiple\",\"latitude\",\"longitude\",\"target_type\",\n",
    "                         \"weapon_type\",\"group_name\",\"fatalities\",\"related\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "2ada1a2c4fc790340bd66f55929fa365",
     "grade": true,
     "grade_id": "cell-5f34e9684b9bb47b",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(terrorism_data)\n",
    "assert_list_equal(terrorism_data.columns.tolist(), [\"year\", \"month\", \"day\", \"country\", \"region\", \"multiple\", \"latitude\", \"longitude\", \"target_type\", \"weapon_type\", \"group_name\", \"fatalities\", \"related\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 2. Convert columns (1 point)\n",
    "Convert the \"year\", \"month\" and \"day\" columns into a single date. Call the new column \"date\".\n",
    "\n",
    "Note that some months and/or days contain 0 as their values. This means that the exact month or day is not present or known. In these cases, write \"not a time\" (NaT) values in the \"date\" column. Do not remove those records!\n",
    "\n",
    "Also, convert the \"multiple\" column into a Boolean one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "45cb70bfd0cb318c5da4eb139d61bae5",
     "grade": false,
     "grade_id": "cell-c415bca15e14b990",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "terrorism_data[\"date\"] = pd.to_datetime(terrorism_data.year * 10000 + terrorism_data.month * 100 + terrorism_data.day,format = \"%Y%m%d\",errors = \"coerce\")\n",
    "terrorism_data.multiple = terrorism_data.multiple.astype(np.bool)\n",
    "terrorism_data = terrorism_data.drop([\"year\", \"month\", \"day\"], axis = 1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "7d9a967ba4907372f8e7d995c68b4e4a",
     "grade": true,
     "grade_id": "cell-8f8c1f0a4e1f54be",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_greater(len(terrorism_data.date), 0)\n",
    "assert_equal(terrorism_data.multiple.dtype, np.bool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 3. Show how the number of attacks evolves with time (1 point)\n",
    "Group all incidents by year. Create a line plot showing how the attacks evolve.\n",
    "\n",
    "Create a function which accepts a dataframe and returns the years with minimum and maximum incidents.\n",
    "\n",
    "Use this function to see the numbers for your dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "4a40c90c814fcd54b021bad2b5c6aabd",
     "grade": false,
     "grade_id": "cell-d95cdafff350daaf",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# Note: This is not autograded\n",
    "attacks_by_year = terrorism_data.groupby(terrorism_data.date.dt.year).date.count()\n",
    "attacks_by_year.plot()\n",
    "plt.title(\"Number of attacks evolves through time\")\n",
    "plt.xlabel(\"year\")\n",
    "plt.ylabel(\"Quantity Attacks\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "7b0a34735fbb5853c98e1174a1bf2e84",
     "grade": false,
     "grade_id": "cell-3c2352bef75801fa",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "def get_max_and_min_incidents_years(dataframe):\n",
    "    # Don't forget to use the dataframe parameter!\n",
    "    min_incidents_year, max_incidents_year = dataframe.groupby(dataframe.date.dt.year).date.count().sort_values(ascending=True).index[0], dataframe.groupby(dataframe.date.dt.year).date.count().sort_values(ascending=False).index[0]\n",
    "    return (min_incidents_year, max_incidents_year)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "d9cf0ea35d0c66c3a3127659731a8965",
     "grade": true,
     "grade_id": "cell-9fab4a50c075077f",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "min_year, max_year = get_max_and_min_incidents_years(terrorism_data)\n",
    "assert_equal(min_year, 1971)\n",
    "assert_equal(max_year, 2014)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 4. Filter terror attacks (1 point)\n",
    "Get only recent attacks (from 1 Jan 2000 onwards, inclusive). Save them in the variable `recent_attacks`.\n",
    "\n",
    "Create another variable which contains attacks before that. Call that `older_attacks`.\n",
    "\n",
    "We'll compare these later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "deb7267a394e92f397bd3241602a5267",
     "grade": false,
     "grade_id": "cell-1c3587b0a671b81e",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "recent_attacks = terrorism_data[terrorism_data.date.dt.year >= 2000]\n",
    "older_attacks = terrorism_data[terrorism_data.date.dt.year < 2000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "1e13a5ebb0c0860922e005052309b1d1",
     "grade": true,
     "grade_id": "cell-1af72cf0b455864f",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(recent_attacks)\n",
    "assert_is_not_none(older_attacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 5. Compare recent to older attacks (1 point)\n",
    "Create a function which accepts a dataframe and returns a `Series`. The index should be a country name, and the values should be the number of  attacks. Sort the values from highest to lowest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "c15c6674d1ffdd3c87cd2d559abd7c5b",
     "grade": false,
     "grade_id": "cell-8e2899086cca6575",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def get_num_attacks_by_country(dataframe):\n",
    "    series = dataframe.groupby(\"country\").size().nlargest(10)\n",
    "    return series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "d42d21df4f382b1bfceaeb6124c4ed9a",
     "grade": true,
     "grade_id": "cell-c51b7c39a69134f9",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "recent_attacks_by_country = get_num_attacks_by_country(recent_attacks)\n",
    "older_attacks_by_country = get_num_attacks_by_country(older_attacks)\n",
    "print(recent_attacks_by_country)\n",
    "print(older_attacks_by_country)\n",
    "assert_equal(recent_attacks_by_country.shape, (10,))\n",
    "assert_equal(older_attacks_by_country.shape, (10,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_graph(num_attacks, title):\n",
    "    plt.bar(range(len(num_attacks)), num_attacks)\n",
    "    plt.xticks(range(len(num_attacks)), num_attacks.index, rotation = \"vertical\")\n",
    "    plt.title(title)\n",
    "    plt.show()\n",
    "\n",
    "show_graph(older_attacks_by_country, \"Older attacks (before 2000)\")\n",
    "show_graph(recent_attacks_by_country, \"Recent attacks (since 2000)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that these attacks definitely changed very much. First, there are many more since 2000, and second, the top targeted countries have changed a lot: only India remains in the top chart (execute the following cell if you don't believe me)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set(recent_attacks_by_country.index).intersection(set(older_attacks_by_country.index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a lot more to explore in the data, but let's focus on the spatial aspect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 6. Create a basic map (2 points)\n",
    "Plot all places where terrorist attacks have occurred.\n",
    "\n",
    "The skeleton below almost does the job. What you need to do is: first, convert the geographical coordinates to horizontal and assign them to the `x` and `y` variables; second, create a scatter plot, using a red circle (`\"o\"`), with marker size 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "cad861d14f9270a8725ebfa7a8d385a0",
     "grade": false,
     "grade_id": "cell-949ee28085e3121b",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize = (12, 10))\n",
    "m = Basemap(projection = \"merc\", llcrnrlat = -73, llcrnrlon = -180, urcrnrlat = 80, urcrnrlon = 180)\n",
    "x, y = m(terrorism_data.longitude.tolist(),terrorism_data.latitude.tolist())\n",
    "m.plot(x,y,\"o\",markersize = 1,color = \"red\")\n",
    "m.drawcoastlines()\n",
    "m.drawcountries()\n",
    "m.fillcontinents(color = \"lightgreen\", lake_color = \"aqua\")\n",
    "m.drawmapboundary(fill_color = \"aqua\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "c60d7fff2a36b55e539d58ee1768cc62",
     "grade": true,
     "grade_id": "cell-db1a0f33e3c947b5",
     "locked": true,
     "points": 2,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(x)\n",
    "assert_is_not_none(y)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 7. Emphasize incidents which are part of a group (0 points)\n",
    "Redraw the map above. This time, filter out the attacks which are part of a series (this is contained in the column `multiple`). Show them on the map with an orange color."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "27bedcbe995f64cb485d9c1f382b8555",
     "grade": false,
     "grade_id": "cell-c92a19d6d8714acc",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# Note: This is not autograded\n",
    "single_attacks = terrorism_data[terrorism_data.multiple == False]\n",
    "multiple_attacks = terrorism_data[terrorism_data.multiple == True]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "09a73870417d67337ba3b7d4f20e50d0",
     "grade": true,
     "grade_id": "cell-0140c2050b753b80",
     "locked": true,
     "points": 0,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize = (12, 10))\n",
    "m = Basemap(projection = \"merc\", llcrnrlat = -73, llcrnrlon = -180, urcrnrlat = 80, urcrnrlon = 180)\n",
    "\n",
    "x, y = m(single_attacks.longitude.tolist(), single_attacks.latitude.tolist())\n",
    "m.plot(x, y, \"o\", markersize = 1, color = \"red\")\n",
    "\n",
    "x, y = m(multiple_attacks.longitude.tolist(), multiple_attacks.latitude.tolist())\n",
    "m.plot(x, y, \"o\", markersize = 1, color = \"orange\")\n",
    "\n",
    "m.drawcoastlines()\n",
    "m.drawcountries()\n",
    "m.fillcontinents(color = \"lightgreen\", lake_color = \"aqua\")\n",
    "m.drawmapboundary(fill_color = \"aqua\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that a lot of the attacks are parts of series, possibly organized. But, remember that we have a point on the map for every single attack. If we see how many multiple attacks there are (as a fraction of the whole), we'll get about 13.4%:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(multiple_attacks) / len(terrorism_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot we made was a little misleading. We're displaying where attacks occur but we omitted the \"density\" of attacks: each location gets a point no matter whether there was one incident, or ten thousand in that point. \n",
    "\n",
    "We can try plotting the densities instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 8. Plot attack densities (1 point)\n",
    "First, we need to filter out attacks with unknown (NaN) location. Do this and save it in the variable `attacks_with_locations`.\n",
    "\n",
    "The following cell prints a 2D histogram (also called a heatmap) over the world map. Execute it, it should run just fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "7d26bf1671643733a7d4fe1f11848ed5",
     "grade": false,
     "grade_id": "cell-e249aca4f843b587",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "attacks_with_locations = terrorism_data[[\"longitude\",\"latitude\"]].dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_attack_densities(attack_data, title = \"Attack densities\"):\n",
    "    plt.figure(figsize = (15, 10))\n",
    "    m = Basemap(projection = \"merc\", llcrnrlat = -73, llcrnrlon = -180, urcrnrlat = 80, urcrnrlon = 180)\n",
    "\n",
    "    # Prepare histogram bins\n",
    "    num_bins = 100\n",
    "    lon_bins = np.linspace(-180, 180, num_bins + 1)\n",
    "    lat_bins = np.linspace(-90, 90, num_bins + 1)\n",
    "\n",
    "    # Create 2D histogram values\n",
    "    density, x_breaks, y_breaks = np.histogram2d(\n",
    "        attack_data.latitude,\n",
    "        attack_data.longitude, \n",
    "        [lat_bins, lon_bins])\n",
    "\n",
    "    # Create the basis of the histogram - the (x, y) value pairs\n",
    "    # and map them to 2D distances\n",
    "    lon_bins_2d, lat_bins_2d = np.meshgrid(lon_bins, lat_bins)\n",
    "    x, y = m(lon_bins_2d, lat_bins_2d)\n",
    "\n",
    "    m.drawcoastlines()\n",
    "    m.drawcountries()\n",
    "    m.pcolormesh(x, y, density)\n",
    "    m.colorbar()\n",
    "    plt.title(title)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_attack_densities(attacks_with_locations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see various \"hot points\". The hottest one, as expected from the data above, is Iraq. Actually, it has so many attacks that it overshadows most of the data in our dataset.\n",
    "\n",
    "Let's remove it and plot the data again. This is a way to estimate **importance** of something (although not a very scientific one :))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "adf4aa6e09e4a50767b0be2c7000f044",
     "grade": false,
     "grade_id": "cell-4696a4495fae2524",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "attacks_without_iraq = attacks_with_locations[attacks_with_locations.country != \"Iraq\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "d0fcbdafdf7e0b072f3a1d37acedb17d",
     "grade": true,
     "grade_id": "cell-dffe18739baef50b",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(attacks_with_locations)\n",
    "assert_is_not_none(attacks_without_iraq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_attack_densities(attacks_without_iraq, \"Attack densities (Iraq omitted)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, the map \"lit up\". Notice that the colorbar now shows completely different values.\n",
    "\n",
    "Based on these visualizations, we can conclude that the number of attacks in Iraq is really significant compared to all others.\n",
    "\n",
    "We're also able to see more hot points now, and we may try to interpret them. However, that's not what we're going to do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 9. Create a graph of connected incidents (1 + 1 points)\n",
    "We already have incidents which are part of groups: the variable `multiple_attacks` holds them.\n",
    "\n",
    "However, it turns out that some of these hold NaNs in the `related` column.\n",
    "\n",
    "Create a new dataset which holds the `eventid` and `related` incidents of every attack in `multiple_attacks`.\n",
    "\n",
    "Filter the dataset so that it contains only valid (i.e. non-empty or NaN) `related` values.\n",
    "\n",
    "Store this dataset into `incident_relations`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "bcba82e0b647a4fd5ab0c3c186dcfbca",
     "grade": false,
     "grade_id": "cell-a2682fbc00a9314f",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "incident_relations = multiple_attacks[[\"related\"]].dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "d64da3f61f8ddd44316148c656ee5e44",
     "grade": true,
     "grade_id": "cell-2760002c8cd2e6fb",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_equal(incident_relations.shape, (22840, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get all relations by splitting the column first, like so:\n",
    "```\n",
    "relations = incident_relations.related.str.split(\", \")\n",
    "```\n",
    "\n",
    "For each relation, if the id of the first and second node aren't the same, add the relation as an edge. An edge has the format `(id_1, id_2)`.\n",
    "\n",
    "The code is not very hard to understand but it definitely IS tricky to write. So I'm giving it to you for free :)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "edges = set()\n",
    "\n",
    "relations = incident_relations.related.str.split(\"\\D+\")\n",
    "for index, related in relations.iteritems():\n",
    "    current_index = int(index)\n",
    "    for related_id in related:\n",
    "        if related_id != \"\":\n",
    "            current_related = int(related_id)\n",
    "            # (id_1, id_2) and (id_2, id_1) should be considered the same\n",
    "            # undirected edge but in the dataset, they are separate\n",
    "            # so we need to avoid double-adding edges\n",
    "            if current_index == current_related:\n",
    "                continue\n",
    "            elif current_index > current_related:\n",
    "                current_index, current_related = current_related, current_index\n",
    "            edges.add((current_index, current_related))\n",
    "print(len(edges))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a graph, `attack_graph` and add all edges that you (or I :D) just calculated to it.\n",
    "\n",
    "Beware: visualization will take a lot of time, so **don't do that** (or just comment it out before submitting). I'm only testing basic properties on the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "fcc7de3f266217fbde311aa5f1f27df6",
     "grade": false,
     "grade_id": "cell-cc347a3a786ca561",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "attack_graph = nx.Graph()\n",
    "attack_graph.add_edges_from(edges)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "ddfbaf0ec6021edc4817b7b3da8e5632",
     "grade": true,
     "grade_id": "cell-a20bcc1a8794b893",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert_is_not_none(attack_graph)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}