ch_09/planet_data_collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Planet Data Collection\n",
    "Using the Open Exoplanet Catalogue database: https://github.com/OpenExoplanetCatalogue/open_exoplanet_catalogue/\n",
    "\n",
    "## Data License\n",
    "Copyright (C) 2012 Hanno Rein\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this database and associated scripts (the \"Database\"), to deal in the Database without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Database, and to permit persons to whom the Database is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Database. A reference to the Database shall be included in all scientific publications that make use of the Database.\n",
    "\n",
    "THE DATABASE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE DATABASE OR THE USE OR OTHER DEALINGS IN THE DATABASE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Follow instructions to get the xml file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xml.etree.ElementTree as ET, urllib.request, gzip, io\n",
    "url = \"https://github.com/OpenExoplanetCatalogue/oec_gzip/raw/master/systems.xml.gz\"\n",
    "oec = ET.parse(gzip.GzipFile(fileobj=io.BytesIO(urllib.request.urlopen(url).read())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parse into Pandas DataFrame\n",
    "Information on what each field means can be found [here](https://github.com/OpenExoplanetCatalogue/open_exoplanet_catalogue/#data-structure)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "def parse(base):\n",
    "    db = oec.findall(f\".//{base}\")\n",
    "    \n",
    "    exclude = ['star', 'videolink', 'binary'] if base in ['system', 'binary'] else ['planet']\n",
    "    \n",
    "    columns = set([attribute.tag for attribute in db[0] if attribute.tag not in exclude])\n",
    "    results = pd.DataFrame(columns=columns)\n",
    "\n",
    "    for entry in db:\n",
    "        data = {col : entry.findtext(col) for col in columns}\n",
    "        if base in ['system', 'binary']:\n",
    "            data['binaries'] = len(entry.findall('.//binary'))\n",
    "            data['stars'] = len(entry.findall('.//star'))\n",
    "        if base in ['system', 'star', 'binary']:\n",
    "            data['planets'] = len(entry.findall('.//planet'))\n",
    "        results = results.append(data, ignore_index=True)\n",
    "\n",
    "    return results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parse planet data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mass</th>\n",
       "      <th>description</th>\n",
       "      <th>periastrontime</th>\n",
       "      <th>semimajoraxis</th>\n",
       "      <th>discoveryyear</th>\n",
       "      <th>list</th>\n",
       "      <th>eccentricity</th>\n",
       "      <th>period</th>\n",
       "      <th>discoverymethod</th>\n",
       "      <th>lastupdate</th>\n",
       "      <th>periastron</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19.4</td>\n",
       "      <td>11 Com b is a brown dwarf-mass companion to th...</td>\n",
       "      <td>2452899.6</td>\n",
       "      <td>1.29</td>\n",
       "      <td>2008</td>\n",
       "      <td>Confirmed planets</td>\n",
       "      <td>0.231</td>\n",
       "      <td>326.03</td>\n",
       "      <td>RV</td>\n",
       "      <td>15/09/20</td>\n",
       "      <td>94.8</td>\n",
       "      <td>11 Com b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11.20</td>\n",
       "      <td>11 Ursae Minoris is a star located in the cons...</td>\n",
       "      <td>2452861.04</td>\n",
       "      <td>1.54</td>\n",
       "      <td>2009</td>\n",
       "      <td>Confirmed planets</td>\n",
       "      <td>0.08</td>\n",
       "      <td>516.22</td>\n",
       "      <td>RV</td>\n",
       "      <td>15/09/20</td>\n",
       "      <td>117.63</td>\n",
       "      <td>11 UMi b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.8</td>\n",
       "      <td>14 Andromedae is an evolved star in the conste...</td>\n",
       "      <td>2452861.4</td>\n",
       "      <td>0.83</td>\n",
       "      <td>2008</td>\n",
       "      <td>Confirmed planets</td>\n",
       "      <td>0</td>\n",
       "      <td>185.84</td>\n",
       "      <td>RV</td>\n",
       "      <td>15/09/20</td>\n",
       "      <td>0</td>\n",
       "      <td>14 And b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.975</td>\n",
       "      <td>The star 14 Herculis is only 59 light years aw...</td>\n",
       "      <td>None</td>\n",
       "      <td>2.864</td>\n",
       "      <td>2002</td>\n",
       "      <td>Confirmed planets</td>\n",
       "      <td>0.359</td>\n",
       "      <td>1766</td>\n",
       "      <td>RV</td>\n",
       "      <td>15/09/21</td>\n",
       "      <td>22.230</td>\n",
       "      <td>14 Her b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7.679</td>\n",
       "      <td>14 Her c is the second companion in the system...</td>\n",
       "      <td>None</td>\n",
       "      <td>9.037</td>\n",
       "      <td>2006</td>\n",
       "      <td>Controversial</td>\n",
       "      <td>0.184</td>\n",
       "      <td>9886</td>\n",
       "      <td>RV</td>\n",
       "      <td>15/09/21</td>\n",
       "      <td>189.076</td>\n",
       "      <td>14 Her c</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mass                                        description periastrontime  \\\n",
       "0   19.4  11 Com b is a brown dwarf-mass companion to th...      2452899.6   \n",
       "1  11.20  11 Ursae Minoris is a star located in the cons...     2452861.04   \n",
       "2    4.8  14 Andromedae is an evolved star in the conste...      2452861.4   \n",
       "3  4.975  The star 14 Herculis is only 59 light years aw...           None   \n",
       "4  7.679  14 Her c is the second companion in the system...           None   \n",
       "\n",
       "  semimajoraxis discoveryyear               list eccentricity  period  \\\n",
       "0          1.29          2008  Confirmed planets        0.231  326.03   \n",
       "1          1.54          2009  Confirmed planets         0.08  516.22   \n",
       "2          0.83          2008  Confirmed planets            0  185.84   \n",
       "3         2.864          2002  Confirmed planets        0.359    1766   \n",
       "4         9.037          2006      Controversial        0.184    9886   \n",
       "\n",
       "  discoverymethod lastupdate periastron      name  \n",
       "0              RV   15/09/20       94.8  11 Com b  \n",
       "1              RV   15/09/20     117.63  11 UMi b  \n",
       "2              RV   15/09/20          0  14 And b  \n",
       "3              RV   15/09/21     22.230  14 Her b  \n",
       "4              RV   15/09/21    189.076  14 Her c  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets = parse('planet')\n",
    "planets.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parse system data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>declination</th>\n",
       "      <th>distance</th>\n",
       "      <th>name</th>\n",
       "      <th>rightascension</th>\n",
       "      <th>binaries</th>\n",
       "      <th>planets</th>\n",
       "      <th>stars</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>+17 47 34</td>\n",
       "      <td>88.9</td>\n",
       "      <td>11 Com</td>\n",
       "      <td>12 20 43</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>+71 49 26.0466</td>\n",
       "      <td>122.1</td>\n",
       "      <td>11 UMi</td>\n",
       "      <td>15 17 05.88899</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>+39 14 10.3092</td>\n",
       "      <td>79.2</td>\n",
       "      <td>14 And</td>\n",
       "      <td>23 31 17.41346</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>+43 49 18</td>\n",
       "      <td>18.1</td>\n",
       "      <td>14 Her</td>\n",
       "      <td>16 10 23</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>+50 31 30.2153</td>\n",
       "      <td>21.146</td>\n",
       "      <td>16 Cygni</td>\n",
       "      <td>19 41 48.95343</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      declination distance      name  rightascension  binaries  planets  stars\n",
       "0       +17 47 34     88.9    11 Com        12 20 43       0.0      1.0    1.0\n",
       "1  +71 49 26.0466    122.1    11 UMi  15 17 05.88899       0.0      1.0    1.0\n",
       "2  +39 14 10.3092     79.2    14 And  23 31 17.41346       0.0      1.0    1.0\n",
       "3       +43 49 18     18.1    14 Her        16 10 23       0.0      2.0    1.0\n",
       "4  +50 31 30.2153   21.146  16 Cygni  19 41 48.95343       2.0      1.0    3.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "systems = parse('system')\n",
    "systems.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parse binary data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>positionangle</th>\n",
       "      <th>separation</th>\n",
       "      <th>name</th>\n",
       "      <th>binaries</th>\n",
       "      <th>planets</th>\n",
       "      <th>stars</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>133.30</td>\n",
       "      <td>39.56</td>\n",
       "      <td>16 Cygni</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>209</td>\n",
       "      <td>3.4</td>\n",
       "      <td>16 Cygni AC</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>237.3</td>\n",
       "      <td>12.37</td>\n",
       "      <td>2M0441+2301</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>79.61</td>\n",
       "      <td>0.2323</td>\n",
       "      <td>2M 044145</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>2M 1938+4603</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  positionangle separation          name  binaries  planets  stars\n",
       "0        133.30      39.56      16 Cygni       1.0      1.0    3.0\n",
       "1           209        3.4   16 Cygni AC       0.0      0.0    2.0\n",
       "2         237.3      12.37   2M0441+2301       1.0      1.0    3.0\n",
       "3         79.61     0.2323     2M 044145       0.0      0.0    2.0\n",
       "4          None       None  2M 1938+4603       0.0      1.0    2.0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "binaries = parse('binary')\n",
    "binaries.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parse star data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>temperature</th>\n",
       "      <th>magV</th>\n",
       "      <th>mass</th>\n",
       "      <th>spectraltype</th>\n",
       "      <th>magJ</th>\n",
       "      <th>radius</th>\n",
       "      <th>magB</th>\n",
       "      <th>magH</th>\n",
       "      <th>magK</th>\n",
       "      <th>metallicity</th>\n",
       "      <th>name</th>\n",
       "      <th>planets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4742</td>\n",
       "      <td>4.74</td>\n",
       "      <td>2.7</td>\n",
       "      <td>G8 III</td>\n",
       "      <td>2.943</td>\n",
       "      <td>19</td>\n",
       "      <td>5.74</td>\n",
       "      <td>2.484</td>\n",
       "      <td>2.282</td>\n",
       "      <td>-0.35</td>\n",
       "      <td>11 Com</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4340</td>\n",
       "      <td>5.024</td>\n",
       "      <td>1.80</td>\n",
       "      <td>K4III</td>\n",
       "      <td>2.876</td>\n",
       "      <td>24.08</td>\n",
       "      <td>6.415</td>\n",
       "      <td>2.091</td>\n",
       "      <td>1.939</td>\n",
       "      <td>0.04</td>\n",
       "      <td>11 UMi</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4813</td>\n",
       "      <td>5.22</td>\n",
       "      <td>2.2</td>\n",
       "      <td>K0III</td>\n",
       "      <td>3.019</td>\n",
       "      <td>11</td>\n",
       "      <td>6.24</td>\n",
       "      <td>2.608</td>\n",
       "      <td>2.331</td>\n",
       "      <td>-0.24</td>\n",
       "      <td>14 And</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5311.0</td>\n",
       "      <td>6.67</td>\n",
       "      <td>1.0</td>\n",
       "      <td>K0 V</td>\n",
       "      <td>5.158</td>\n",
       "      <td>0.708</td>\n",
       "      <td>7.57</td>\n",
       "      <td>4.803</td>\n",
       "      <td>4.714</td>\n",
       "      <td>0.43</td>\n",
       "      <td>14 Her</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5825</td>\n",
       "      <td>5.95</td>\n",
       "      <td>1.11</td>\n",
       "      <td>G2V</td>\n",
       "      <td>5.09</td>\n",
       "      <td>1.243</td>\n",
       "      <td>6.59</td>\n",
       "      <td>4.72</td>\n",
       "      <td>4.43</td>\n",
       "      <td>0.096</td>\n",
       "      <td>16 Cygni A</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  temperature   magV  mass spectraltype   magJ radius   magB   magH   magK  \\\n",
       "0        4742   4.74   2.7       G8 III  2.943     19   5.74  2.484  2.282   \n",
       "1        4340  5.024  1.80        K4III  2.876  24.08  6.415  2.091  1.939   \n",
       "2        4813   5.22   2.2        K0III  3.019     11   6.24  2.608  2.331   \n",
       "3      5311.0   6.67   1.0         K0 V  5.158  0.708   7.57  4.803  4.714   \n",
       "4        5825   5.95  1.11          G2V   5.09  1.243   6.59   4.72   4.43   \n",
       "\n",
       "  metallicity        name  planets  \n",
       "0       -0.35      11 Com      1.0  \n",
       "1        0.04      11 UMi      1.0  \n",
       "2       -0.24      14 And      1.0  \n",
       "3        0.43      14 Her      2.0  \n",
       "4       0.096  16 Cygni A      0.0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stars = parse('star')\n",
    "stars.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save to CSVs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "planets.to_csv('data/planets.csv', index=False)\n",
    "binaries.to_csv('data/binaries.csv', index=False)\n",
    "stars.to_csv('data/stars.csv', index=False)\n",
    "systems.to_csv('data/systems.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "<div style=\"overflow: hidden; margin-bottom: 10px;\">\n",
    "    <div style=\"float: left;\">\n",
    "        <a href=\"../../ch_08/anomaly_detection.ipynb\">\n",
    "        <button>&#8592; Chapter 8</button>\n",
    "    </a>\n",
    "    </div>\n",
    "    <div style=\"float: right;\">\n",
    "        <a href=\"./planets_ml.ipynb\">\n",
    "            <button>Next Notebook &#8594;</button>\n",
    "        </a>\n",
    "    </div>\n",
    "</div>\n",
    "<hr>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}