notebooks/02_fastai_DataBunch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Background\n",
    "\n",
    "This notebook walks through the creation of a fastai [DataBunch](https://docs.fast.ai/basic_data.html#DataBunch) object.  This object contains a pytorch dataloader for the train, valid and test sets. From the documentation:\n",
    "\n",
    "```\n",
    "Bind train_dl,valid_dl and test_dl in a data object.\n",
    "\n",
    "It also ensures all the dataloaders are on device and applies to them dl_tfms as batch are drawn (like normalization). path is used internally to store temporary files, collate_fn is passed to the pytorch Dataloader (replacing the one there) to explain how to collate the samples picked for a batch.\n",
    "```\n",
    "\n",
    "Because we are training the language model, we want our dataloader to construct the target variable from the input data.  The target variable for language models are the next word in a sentence.  Furthermore, there are other optimizations with regard to the sequence length and concatenating texts together that avoids wasteful padding.  Luckily the [TextLMDataBunch](https://docs.fast.ai/text.data.html#TextLMDataBunch) does all this work for us (and more) automatically.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text import TextLMDataBunch as lmdb\n",
    "from fastai.text.transform import Tokenizer\n",
    "import pandas as pd\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read in Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can download the above saved dataframes (in pickle format) from Google Cloud Storage:\n",
    "\n",
    "**train_df.pkl (9GB)**: \n",
    "\n",
    "`https://storage.googleapis.com/issue_label_bot/pre_processed_data/2_partitioned_df/train_df.pkl`\n",
    "\n",
    "**valid_df.pkl (1GB)**\n",
    "\n",
    "`https://storage.googleapis.com/issue_label_bot/pre_processed_data/2_partitioned_df/valid_df.pkl`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note: download the data and place in right directory before running this code!\n",
    "\n",
    "valid_df = pd.read_pickle(Path('../data/2_partitioned_df/valid_df.pkl'))\n",
    "train_df = pd.read_pickle(Path('../data/2_partitioned_df/train_df.pkl'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rows in train_df:, 16,762,799\n",
      "rows in valid_df:, 1,858,033\n"
     ]
    }
   ],
   "source": [
    "print(f'rows in train_df:, {train_df.shape[0]:,}')\n",
    "print(f'rows in valid_df:, {valid_df.shape[0]:,}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxxfldtitle Grab excerpt and image using Open ...</td>\n",
       "      <td>https://github.com/markjaquith/page-links-to/i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>xxxfldtitle Gracefully handling Ctrl+C ignores...</td>\n",
       "      <td>https://github.com/dotnet/corefx/issues/32749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>xxxfldtitle GradleAspectJ-Android not working ...</td>\n",
       "      <td>https://github.com/Archinamon/android-gradle-a...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  \\\n",
       "0  xxxfldtitle Grab excerpt and image using Open ...   \n",
       "1  xxxfldtitle Gracefully handling Ctrl+C ignores...   \n",
       "2  xxxfldtitle GradleAspectJ-Android not working ...   \n",
       "\n",
       "                                                 url  \n",
       "0  https://github.com/markjaquith/page-links-to/i...  \n",
       "1      https://github.com/dotnet/corefx/issues/32749  \n",
       "2  https://github.com/Archinamon/android-gradle-a...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create The [DataBunch](https://docs.fast.ai/basic_data.html#DataBunch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instantiate The Tokenizer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pass_through(x):\n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# only thing is we are changing pre_rules to be pass through since we have already done all of the pre-rules.  \n",
    "# you don't want to accidentally apply pre-rules again otherwhise it will corrupt the data.\n",
    "tokenizer = Tokenizer(pre_rules=[pass_through], n_cpus=31)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify path for saving language model artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = Path('../model/lang_model/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create  The Language Model Data Bunch\n",
    "\n",
    "**Warning**: this steps builds the vocabulary and tokenizes the data.  This procedure consumes an incredible amount of memory. This took 1 hour on a machine with 72 cores and 400GB of Memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note you want your own tokenizer, without pre-rules\n",
    "data_lm = lmdb.from_df(path=path,\n",
    "                       train_df=train_df,\n",
    "                       valid_df=valid_df,\n",
    "                       text_cols='text',\n",
    "                       tokenizer=tokenizer,\n",
    "                       chunksize=6000000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm.save() # saves to self.path/data_save.pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Location of Saved DataBunch\n",
    "\n",
    "The databunch object is available here:\n",
    "\n",
    "`https://storage.googleapis.com/issue_label_bot/model/lang_model/data_save.pkl`\n",
    "\n",
    "It is a massive file of 27GB so proceed with caution when downlaoding this file."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}