classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DEBUG: this should be moved outside"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "Progress <- R6::R6Class(\"Progress\",\n",
    "  inherit = KerasCallback,\n",
    "  \n",
    "  public = list(\n",
    "    num_epochs  = NULL,\n",
    "    update_frequency = NULL,\n",
    "    epoch = NULL,\n",
    "    batch = NULL,\n",
    "      \n",
    "    initialize = function(num_epochs, update_frequency) {\n",
    "        self$num_epochs <- num_epochs\n",
    "        self$update_frequency <- update_frequency\n",
    "        self$epoch <- 1\n",
    "    },\n",
    "                        \n",
    "    on_batch_end = function(batch, logs = list()) {\n",
    "        if ((batch + 1) %% self$update_frequency == 0) {\n",
    "            cat('Epoch ', self$epoch + 1, '/', self$num_epochs, ': ', batch + 1,\n",
    "                ' - loss: ', logs[['loss']], '\\r', sep = '')\n",
    "            flush.console()\n",
    "        }\n",
    "        self$batch <- batch\n",
    "    },\n",
    "      \n",
    "    on_epoch_begin = function(epoch, logs = list()) {\n",
    "        self$epoch <- epoch\n",
    "    },\n",
    "                        \n",
    "    on_epoch_end = function(epoch, logs = list()) {\n",
    "        cat('Epoch ', self$epoch + 1, '/', self$num_epochs, ': ', self$batch + 1, ' - loss: ', logs[['loss']], ' - validation loss: ', logs[['val_loss']], '\\n', sep = '')\n",
    "        flush.console()\n",
    "    }\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification\n",
    "In this notebook, we will revisit the Boston housing data set from the last session. It is quite difficult to build a model that performs highly accurate regression. Instead of increasing our model complexity, we can turn the regression problem into a *classification* problem. That way, we do not predict the housing price directly, but predict to a number of *classes* or *labels* that contain certain price ranges.\n",
    "\n",
    "You could argue that we are relaxing our standards a bit in this manner, but it is often a valid approach towards 'solving' a difficult regression problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "── Attaching packages ─────────────────────────────────────── tidyverse 1.2.1 ──\n",
      "✔ tibble  2.0.1     ✔ purrr   0.2.5\n",
      "✔ tidyr   0.8.2     ✔ dplyr   0.7.8\n",
      "✔ readr   1.3.1     ✔ stringr 1.3.1\n",
      "✔ tibble  2.0.1     ✔ forcats 0.3.0\n",
      "── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──\n",
      "✖ dplyr::filter() masks stats::filter()\n",
      "✖ dplyr::lag()    masks stats::lag()\n",
      "\n",
      "Attaching package: ‘reshape2’\n",
      "\n",
      "The following object is masked from ‘package:tidyr’:\n",
      "\n",
      "    smiths\n",
      "\n"
     ]
    }
   ],
   "source": [
    "library(keras)\n",
    "library(ggplot2)\n",
    "library(tidyverse)\n",
    "library(rsample)\n",
    "library(reshape2)\n",
    "\n",
    "library(repr)\n",
    "options(repr.plot.width=6, repr.plot.height=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data <- dataset_boston_housing()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c(c(X_train, y_train), c(X_test, y_test)) %<-% data\n",
    "dim(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist(X_train[1:dim(X_train)[1], 13])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "correlation <- round(cor(X_train), 2)\n",
    "correlation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ggplot(data = melt(correlation), aes(x = Var1, y = Var2, fill = value)) + geom_tile()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# y_train <- y_train %>% cut(breaks = c(1, 25, max(y_train)), labels = FALSE) - 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "breaks <- c(0, 10, 20, 30, 40, max(y_train))\n",
    "y_train_multiclass <- y_train %>% cut(breaks = breaks, labels = FALSE) - 1\n",
    "table(y_train_multiclass)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "1 / table(y_train_multiclass)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "frequencies <- table(y_train_multiclass)\n",
    "class_weights <- vector(mode = \"list\", length = length(frequencies))\n",
    "names(class_weights) <- as.character(seq(0, length(frequencies) - 1))\n",
    "for (i in 1:length(names(class_weights))) {\n",
    "    key <- names(class_weights)[[i]]\n",
    "    class_weights[[key]] <- 1 / frequencies[[key]]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train <- to_categorical(y_train_multiclass)\n",
    "dim(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dim(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalize the data\n",
    "Normalize the data as before, by centering on the mean of the training data, and dividing all values by their standard deviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_mean <- apply(X_train, 2, mean)\n",
    "X_sd <- apply(X_train, 2, sd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train <- scale(X_train, center = X_mean, scale = X_sd)\n",
    "X_test  <- scale(X_test, center = X_mean, scale = X_sd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "split_at <- 0.8 * dim(X_train)[1]\n",
    "X_val <- X_train[-(1:split_at),]\n",
    "y_val <- y_train[-(1:split_at),]\n",
    "X_train <- X_train[1:split_at,]\n",
    "y_train <- y_train[1:split_at,]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dim(y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "    layer_dense(units = 16, input_shape=dim(X_train)[2], activation = \"sigmoid\") %>%\n",
    "#     layer_dense(units = 256, activation = \"sigmoid\") %>%\n",
    "    layer_dense(units = 5, activation = \"softmax\")\n",
    "\n",
    "model %>% compile(\n",
    "    optimizer = optimizer_rmsprop(lr = 0.001),\n",
    "    loss = \"categorical_crossentropy\",\n",
    "    metrics = c(\"accuracy\")\n",
    ")\n",
    "cat(summary(model))\n",
    "num_epochs <- 250\n",
    "history <- model %>% fit(\n",
    "    X_train, y_train,\n",
    "    validation_data = list(X_val, y_val),\n",
    "    epochs = num_epochs,\n",
    "    batch_size = 16,\n",
    "    callbacks=list(Progress$new(num_epochs, 10))\n",
    ")\n",
    "plot(history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred <- model %>% predict(X_val)\n",
    "table(apply(y_pred, 1, which.max))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "    layer_dense(units = 16, input_shape=dim(X_train)[2], activation = \"sigmoid\") %>%\n",
    "#     layer_dense(units = 256, activation = \"sigmoid\") %>%\n",
    "    layer_dense(units = 5, activation = \"softmax\")\n",
    "\n",
    "model %>% compile(\n",
    "    optimizer = optimizer_rmsprop(lr = 0.001),\n",
    "    loss = \"categorical_crossentropy\",\n",
    "    metrics = c(\"accuracy\")\n",
    ")\n",
    "cat(summary(model))\n",
    "num_epochs <- 250\n",
    "history <- model %>% fit(\n",
    "    X_train, y_train,\n",
    "    validation_data = list(X_val, y_val),\n",
    "    epochs = num_epochs,\n",
    "    batch_size = 16,\n",
    "    callbacks = list(Progress$new(num_epochs, 10)),\n",
    "    class_weights = class_weights\n",
    ")\n",
    "plot(history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred <- model %>% predict(X_val)\n",
    "table(apply(y_pred, 1, which.max))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DEBUG: airplane data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observations: 570,118\n",
      "Variables: 33\n",
      "$ YEAR                <int> 2018, 2018, 2018, 2018, 2018, 2018, 2018, 2018, 2…\n",
      "$ MONTH               <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…\n",
      "$ DAY_OF_MONTH        <int> 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 2…\n",
      "$ DAY_OF_WEEK         <int> 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6…\n",
      "$ FL_DATE             <fct> 2018-01-27, 2018-01-27, 2018-01-27, 2018-01-27, 2…\n",
      "$ OP_UNIQUE_CARRIER   <fct> UA, UA, UA, UA, UA, UA, UA, UA, UA, UA, UA, UA, U…\n",
      "$ TAIL_NUM            <fct> N26232, N477UA, N13720, N16217, N33714, N809UA, N…\n",
      "$ OP_CARRIER_FL_NUM   <int> 369, 368, 367, 366, 365, 364, 363, 362, 361, 360,…\n",
      "$ ORIGIN              <fct> FLL, SEA, DCA, LAX, JAX, IAH, EWR, HNL, LAS, IAD,…\n",
      "$ ORIGIN_CITY_NAME    <fct> \"Fort Lauderdale, FL\", \"Seattle, WA\", \"Washington…\n",
      "$ DEST                <fct> IAH, SFO, IAH, ORD, EWR, PHX, HNL, EWR, SFO, TPA,…\n",
      "$ DEST_CITY_NAME      <fct> \"Houston, TX\", \"San Francisco, CA\", \"Houston, TX\"…\n",
      "$ CRS_DEP_TIME        <int> 615, 618, 830, 650, 1824, 1420, 815, 1615, 1520, …\n",
      "$ DEP_TIME            <int> 602, 614, 828, 641, 1810, 1413, 842, 1623, 1515, …\n",
      "$ DEP_DELAY           <dbl> -13, -4, -2, -9, -14, -7, 27, 8, -5, -7, -6, -9, …\n",
      "$ TAXI_OUT            <dbl> 19, 16, 17, 17, 11, 19, 21, 21, 11, 16, 22, 26, 1…\n",
      "$ TAXI_IN             <dbl> 7, 5, 13, 12, 8, 5, 3, 12, 7, 6, 6, 7, 4, 10, 4, …\n",
      "$ CRS_ARR_TIME        <int> 808, 831, 1107, 1250, 2045, 1618, 1438, 653, 1703…\n",
      "$ ARR_TIME            <int> 756, 813, 1108, 1242, 2021, 1559, 1457, 630, 1641…\n",
      "$ ARR_DELAY           <dbl> -12, -18, 1, -8, -24, -19, 19, -23, -22, -18, -14…\n",
      "$ CANCELLED           <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…\n",
      "$ CANCELLATION_CODE   <fct> , , , , , , , , , , , , , , , , , , , , , , , , , \n",
      "$ DIVERTED            <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…\n",
      "$ CRS_ELAPSED_TIME    <dbl> 173, 133, 217, 240, 141, 178, 683, 578, 103, 146,…\n",
      "$ ACTUAL_ELAPSED_TIME <dbl> 174, 119, 220, 241, 131, 166, 675, 547, 86, 135, …\n",
      "$ AIR_TIME            <dbl> 148, 98, 190, 212, 112, 142, 651, 514, 68, 113, 2…\n",
      "$ DISTANCE            <dbl> 966, 679, 1208, 1744, 820, 1009, 4962, 4962, 414,…\n",
      "$ CARRIER_DELAY       <dbl> NA, NA, NA, NA, NA, NA, 0, NA, NA, NA, NA, NA, NA…\n",
      "$ WEATHER_DELAY       <dbl> NA, NA, NA, NA, NA, NA, 0, NA, NA, NA, NA, NA, NA…\n",
      "$ NAS_DELAY           <dbl> NA, NA, NA, NA, NA, NA, 0, NA, NA, NA, NA, NA, NA…\n",
      "$ SECURITY_DELAY      <dbl> NA, NA, NA, NA, NA, NA, 0, NA, NA, NA, NA, NA, NA…\n",
      "$ LATE_AIRCRAFT_DELAY <dbl> NA, NA, NA, NA, NA, NA, 19, NA, NA, NA, NA, NA, N…\n",
      "$ X                   <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…\n"
     ]
    }
   ],
   "source": [
    "data <- read.csv('airplane/2018/airplane-delay-2018-1.csv')\n",
    "glimpse(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>DAY_OF_MONTH</th><th scope=col>DAY_OF_WEEK</th><th scope=col>OP_UNIQUE_CARRIER</th><th scope=col>TAIL_NUM</th><th scope=col>OP_CARRIER_FL_NUM</th><th scope=col>ORIGIN</th><th scope=col>DEST</th><th scope=col>DEP_DELAY</th><th scope=col>CRS_DEP_TIME</th><th scope=col>CRS_ARR_TIME</th><th scope=col>DISTANCE</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>27    </td><td>6     </td><td>UA    </td><td>N26232</td><td>369   </td><td>FLL   </td><td>IAH   </td><td>-13   </td><td> 615  </td><td> 808  </td><td> 966  </td></tr>\n",
       "\t<tr><td>27    </td><td>6     </td><td>UA    </td><td>N477UA</td><td>368   </td><td>SEA   </td><td>SFO   </td><td> -4   </td><td> 618  </td><td> 831  </td><td> 679  </td></tr>\n",
       "\t<tr><td>27    </td><td>6     </td><td>UA    </td><td>N13720</td><td>367   </td><td>DCA   </td><td>IAH   </td><td> -2   </td><td> 830  </td><td>1107  </td><td>1208  </td></tr>\n",
       "\t<tr><td>27    </td><td>6     </td><td>UA    </td><td>N16217</td><td>366   </td><td>LAX   </td><td>ORD   </td><td> -9   </td><td> 650  </td><td>1250  </td><td>1744  </td></tr>\n",
       "\t<tr><td>27    </td><td>6     </td><td>UA    </td><td>N33714</td><td>365   </td><td>JAX   </td><td>EWR   </td><td>-14   </td><td>1824  </td><td>2045  </td><td> 820  </td></tr>\n",
       "\t<tr><td>27    </td><td>6     </td><td>UA    </td><td>N809UA</td><td>364   </td><td>IAH   </td><td>PHX   </td><td> -7   </td><td>1420  </td><td>1618  </td><td>1009  </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllllllll}\n",
       " DAY\\_OF\\_MONTH & DAY\\_OF\\_WEEK & OP\\_UNIQUE\\_CARRIER & TAIL\\_NUM & OP\\_CARRIER\\_FL\\_NUM & ORIGIN & DEST & DEP\\_DELAY & CRS\\_DEP\\_TIME & CRS\\_ARR\\_TIME & DISTANCE\\\\\n",
       "\\hline\n",
       "\t 27     & 6      & UA     & N26232 & 369    & FLL    & IAH    & -13    &  615   &  808   &  966  \\\\\n",
       "\t 27     & 6      & UA     & N477UA & 368    & SEA    & SFO    &  -4    &  618   &  831   &  679  \\\\\n",
       "\t 27     & 6      & UA     & N13720 & 367    & DCA    & IAH    &  -2    &  830   & 1107   & 1208  \\\\\n",
       "\t 27     & 6      & UA     & N16217 & 366    & LAX    & ORD    &  -9    &  650   & 1250   & 1744  \\\\\n",
       "\t 27     & 6      & UA     & N33714 & 365    & JAX    & EWR    & -14    & 1824   & 2045   &  820  \\\\\n",
       "\t 27     & 6      & UA     & N809UA & 364    & IAH    & PHX    &  -7    & 1420   & 1618   & 1009  \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "DAY_OF_MONTH | DAY_OF_WEEK | OP_UNIQUE_CARRIER | TAIL_NUM | OP_CARRIER_FL_NUM | ORIGIN | DEST | DEP_DELAY | CRS_DEP_TIME | CRS_ARR_TIME | DISTANCE | \n",
       "|---|---|---|---|---|---|\n",
       "| 27     | 6      | UA     | N26232 | 369    | FLL    | IAH    | -13    |  615   |  808   |  966   | \n",
       "| 27     | 6      | UA     | N477UA | 368    | SEA    | SFO    |  -4    |  618   |  831   |  679   | \n",
       "| 27     | 6      | UA     | N13720 | 367    | DCA    | IAH    |  -2    |  830   | 1107   | 1208   | \n",
       "| 27     | 6      | UA     | N16217 | 366    | LAX    | ORD    |  -9    |  650   | 1250   | 1744   | \n",
       "| 27     | 6      | UA     | N33714 | 365    | JAX    | EWR    | -14    | 1824   | 2045   |  820   | \n",
       "| 27     | 6      | UA     | N809UA | 364    | IAH    | PHX    |  -7    | 1420   | 1618   | 1009   | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  DAY_OF_MONTH DAY_OF_WEEK OP_UNIQUE_CARRIER TAIL_NUM OP_CARRIER_FL_NUM ORIGIN\n",
       "1 27           6           UA                N26232   369               FLL   \n",
       "2 27           6           UA                N477UA   368               SEA   \n",
       "3 27           6           UA                N13720   367               DCA   \n",
       "4 27           6           UA                N16217   366               LAX   \n",
       "5 27           6           UA                N33714   365               JAX   \n",
       "6 27           6           UA                N809UA   364               IAH   \n",
       "  DEST DEP_DELAY CRS_DEP_TIME CRS_ARR_TIME DISTANCE\n",
       "1 IAH  -13        615          808          966    \n",
       "2 SFO   -4        618          831          679    \n",
       "3 IAH   -2        830         1107         1208    \n",
       "4 ORD   -9        650         1250         1744    \n",
       "5 EWR  -14       1824         2045          820    \n",
       "6 PHX   -7       1420         1618         1009    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "subset <- data[c('DAY_OF_MONTH', 'DAY_OF_WEEK', 'OP_UNIQUE_CARRIER', 'TAIL_NUM', 'OP_CARRIER_FL_NUM', 'ORIGIN', 'DEST', 'DEP_DELAY', 'CRS_DEP_TIME', 'CRS_ARR_TIME', 'DISTANCE')]\n",
    "head(subset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "subset$OP_UNIQUE_CARRIER <- as.numeric(subset$OP_UNIQUE_CARRIER)\n",
    "subset$TAIL_NUM <- as.numeric(subset$TAIL_NUM)\n",
    "subset$OP_CARRIER_FL_NUM <- as.numeric(subset$OP_CARRIER_FL_NUM)\n",
    "subset$ORIGIN <- as.numeric(subset$ORIGIN)\n",
    "subset$DEST <- as.numeric(subset$DEST)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observations: 570,118\n",
      "Variables: 11\n",
      "$ DAY_OF_MONTH      <int> 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,…\n",
      "$ DAY_OF_WEEK       <int> 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, …\n",
      "$ OP_UNIQUE_CARRIER <dbl> 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14,…\n",
      "$ TAIL_NUM          <dbl> 913, 1846, 275, 417, 1257, 3589, 2324, 2324, 1396, …\n",
      "$ OP_CARRIER_FL_NUM <dbl> 369, 368, 367, 366, 365, 364, 363, 362, 361, 360, 3…\n",
      "$ ORIGIN            <dbl> 113, 287, 85, 176, 165, 151, 105, 140, 174, 149, 17…\n",
      "$ DEST              <dbl> 151, 289, 151, 231, 105, 244, 140, 105, 289, 316, 1…\n",
      "$ DEP_DELAY         <dbl> -13, -4, -2, -9, -14, -7, 27, 8, -5, -7, -6, -9, -8…\n",
      "$ CRS_DEP_TIME      <int> 615, 618, 830, 650, 1824, 1420, 815, 1615, 1520, 17…\n",
      "$ CRS_ARR_TIME      <int> 808, 831, 1107, 1250, 2045, 1618, 1438, 653, 1703, …\n",
      "$ DISTANCE          <dbl> 966, 679, 1208, 1744, 820, 1009, 4962, 4962, 414, 8…\n"
     ]
    }
   ],
   "source": [
    "glimpse(subset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>570118</li>\n",
       "\t<li>10</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 570118\n",
       "\\item 10\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 570118\n",
       "2. 10\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 570118     10"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_train <- as.matrix(subset[c('DAY_OF_MONTH', 'DAY_OF_WEEK', 'OP_UNIQUE_CARRIER', 'TAIL_NUM', 'OP_CARRIER_FL_NUM', 'ORIGIN', 'DEST', 'CRS_DEP_TIME', 'CRS_ARR_TIME', 'DISTANCE')], replace = TRUE)\n",
    "dimnames(X_train) <- NULL\n",
    "dim(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "y_train\n",
       "     0      1 \n",
       "537354  32764 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_train <- subset$DEP_DELAY\n",
    "# y_train[y_train < -50] = -50\n",
    "# y_train[y_train > 50] = 50\n",
    "y_train[is.na(y_train)] = 0\n",
    "y_train <- as.numeric(y_train > 60)\n",
    "table(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "y_train\n",
       "        0         1 \n",
       "0.0574688 0.9425312 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "inverse_frequencies <- 1 / table(y_train)\n",
    "class_weights <- inverse_frequencies / sum(inverse_frequencies)\n",
    "class_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tbody>\n",
       "\t<tr><td>1</td><td>0</td></tr>\n",
       "\t<tr><td>1</td><td>0</td></tr>\n",
       "\t<tr><td>1</td><td>0</td></tr>\n",
       "\t<tr><td>1</td><td>0</td></tr>\n",
       "\t<tr><td>1</td><td>0</td></tr>\n",
       "\t<tr><td>1</td><td>0</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{ll}\n",
       "\t 1 & 0\\\\\n",
       "\t 1 & 0\\\\\n",
       "\t 1 & 0\\\\\n",
       "\t 1 & 0\\\\\n",
       "\t 1 & 0\\\\\n",
       "\t 1 & 0\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "| 1 | 0 | \n",
       "| 1 | 0 | \n",
       "| 1 | 0 | \n",
       "| 1 | 0 | \n",
       "| 1 | 0 | \n",
       "| 1 | 0 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "     [,1] [,2]\n",
       "[1,] 1    0   \n",
       "[2,] 1    0   \n",
       "[3,] 1    0   \n",
       "[4,] 1    0   \n",
       "[5,] 1    0   \n",
       "[6,] 1    0   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_train <- to_categorical(y_train)\n",
    "head(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "split_at <- 0.8 * dim(X_train)[1]\n",
    "X_val <- X_train[-(1:split_at),]\n",
    "y_val <- y_train[-(1:split_at),]\n",
    "X_train <- X_train[1:split_at,]\n",
    "y_train <- y_train[1:split_at,]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>114024</li>\n",
       "\t<li>10</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 114024\n",
       "\\item 10\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 114024\n",
       "2. 10\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 114024     10"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dim(X_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>114024</li>\n",
       "\t<li>2</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 114024\n",
       "\\item 2\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 114024\n",
       "2. 2\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 114024      2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dim(y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tbody>\n",
       "\t<tr><td>27  </td><td>6   </td><td>14  </td><td> 913</td><td>369 </td><td>113 </td><td>151 </td><td> 615</td><td> 808</td><td> 966</td></tr>\n",
       "\t<tr><td>27  </td><td>6   </td><td>14  </td><td>1846</td><td>368 </td><td>287 </td><td>289 </td><td> 618</td><td> 831</td><td> 679</td></tr>\n",
       "\t<tr><td>27  </td><td>6   </td><td>14  </td><td> 275</td><td>367 </td><td> 85 </td><td>151 </td><td> 830</td><td>1107</td><td>1208</td></tr>\n",
       "\t<tr><td>27  </td><td>6   </td><td>14  </td><td> 417</td><td>366 </td><td>176 </td><td>231 </td><td> 650</td><td>1250</td><td>1744</td></tr>\n",
       "\t<tr><td>27  </td><td>6   </td><td>14  </td><td>1257</td><td>365 </td><td>165 </td><td>105 </td><td>1824</td><td>2045</td><td> 820</td></tr>\n",
       "\t<tr><td>27  </td><td>6   </td><td>14  </td><td>3589</td><td>364 </td><td>151 </td><td>244 </td><td>1420</td><td>1618</td><td>1009</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{llllllllll}\n",
       "\t 27   & 6    & 14   &  913 & 369  & 113  & 151  &  615 &  808 &  966\\\\\n",
       "\t 27   & 6    & 14   & 1846 & 368  & 287  & 289  &  618 &  831 &  679\\\\\n",
       "\t 27   & 6    & 14   &  275 & 367  &  85  & 151  &  830 & 1107 & 1208\\\\\n",
       "\t 27   & 6    & 14   &  417 & 366  & 176  & 231  &  650 & 1250 & 1744\\\\\n",
       "\t 27   & 6    & 14   & 1257 & 365  & 165  & 105  & 1824 & 2045 &  820\\\\\n",
       "\t 27   & 6    & 14   & 3589 & 364  & 151  & 244  & 1420 & 1618 & 1009\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "| 27   | 6    | 14   |  913 | 369  | 113  | 151  |  615 |  808 |  966 | \n",
       "| 27   | 6    | 14   | 1846 | 368  | 287  | 289  |  618 |  831 |  679 | \n",
       "| 27   | 6    | 14   |  275 | 367  |  85  | 151  |  830 | 1107 | 1208 | \n",
       "| 27   | 6    | 14   |  417 | 366  | 176  | 231  |  650 | 1250 | 1744 | \n",
       "| 27   | 6    | 14   | 1257 | 365  | 165  | 105  | 1824 | 2045 |  820 | \n",
       "| 27   | 6    | 14   | 3589 | 364  | 151  | 244  | 1420 | 1618 | 1009 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]\n",
       "[1,] 27   6    14    913 369  113  151   615  808  966 \n",
       "[2,] 27   6    14   1846 368  287  289   618  831  679 \n",
       "[3,] 27   6    14    275 367   85  151   830 1107 1208 \n",
       "[4,] 27   6    14    417 366  176  231   650 1250 1744 \n",
       "[5,] 27   6    14   1257 365  165  105  1824 2045  820 \n",
       "[6,] 27   6    14   3589 364  151  244  1420 1618 1009 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "ename": "ERROR",
     "evalue": "Error in normalize_shape(input_shape): object 'X_train' not found\n",
     "output_type": "error",
     "traceback": [
      "Error in normalize_shape(input_shape): object 'X_train' not found\nTraceback:\n",
      "1. keras_model_sequential() %>% layer_dense(units = 2, , input_shape = dim(X_train)[2], \n .     activation = \"softmax\")",
      "2. withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))",
      "3. eval(quote(`_fseq`(`_lhs`)), env, env)",
      "4. eval(quote(`_fseq`(`_lhs`)), env, env)",
      "5. `_fseq`(`_lhs`)",
      "6. freduce(value, `_function_list`)",
      "7. withVisible(function_list[[k]](value))",
      "8. function_list[[k]](value)",
      "9. layer_dense(., units = 2, , input_shape = dim(X_train)[2], activation = \"softmax\")",
      "10. create_layer(keras$layers$Dense, object, list(units = as.integer(units), \n  .     activation = activation, use_bias = use_bias, kernel_initializer = kernel_initializer, \n  .     bias_initializer = bias_initializer, kernel_regularizer = kernel_regularizer, \n  .     bias_regularizer = bias_regularizer, activity_regularizer = activity_regularizer, \n  .     kernel_constraint = kernel_constraint, bias_constraint = bias_constraint, \n  .     input_shape = normalize_shape(input_shape), batch_input_shape = normalize_shape(batch_input_shape), \n  .     batch_size = as_nullable_integer(batch_size), dtype = dtype, \n  .     name = name, trainable = trainable, weights = weights))",
      "11. normalize_shape(input_shape)"
     ]
    }
   ],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "#     layer_dense(units = 32, input_shape=dim(X_train)[2], activation = \"relu\") %>%\n",
    "#     layer_dense(units = 256, activation = \"sigmoid\") %>%\n",
    "    layer_dense(units = 2, , input_shape=dim(X_train)[2], activation = \"softmax\")\n",
    "\n",
    "model %>% compile(\n",
    "    optimizer = optimizer_rmsprop(lr = 0.001),\n",
    "    loss = \"binary_crossentropy\",\n",
    "    metrics = c(\"accuracy\")\n",
    ")\n",
    "cat(summary(model))\n",
    "\n",
    "num_epochs <- 10\n",
    "history <- model %>% fit(\n",
    "    X_train, y_train,\n",
    "    validation_data = list(X_val, y_val),\n",
    "    epochs = num_epochs,\n",
    "    batch_size = 1024,\n",
    "    callbacks = list(Progress$new(num_epochs, 10)),\n",
    "    shuffle = TRUE,\n",
    "    class_weight = list(class_weights)\n",
    ")\n",
    "plot(history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>0</li>\n",
       "\t<li>0</li>\n",
       "\t<li>0</li>\n",
       "\t<li>0</li>\n",
       "\t<li>0</li>\n",
       "\t<li>0</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 0\n",
       "\\item 0\n",
       "\\item 0\n",
       "\\item 0\n",
       "\\item 0\n",
       "\\item 0\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 0\n",
       "2. 0\n",
       "3. 0\n",
       "4. 0\n",
       "5. 0\n",
       "6. 0\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 0 0 0 0 0 0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(apply(model %>% predict(X_val), 1, which.max) - 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}