01_data_representation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Representation\n",
    "\n",
    "The first step is always defining the input and output.\n",
    "\n",
    "We define our input as the title of an incident. This is the first piece of information provided by the responders.\n",
    "\n",
    "Let's first explore our input data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/data.txt\") as f:\n",
    "    data = f.read().strip().split(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Looking at a sample\n",
    "data[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get distribution of data length\n",
    "from collections import Counter\n",
    "\n",
    "word_len_count = Counter([len(x.split()) for x in data])\n",
    "\n",
    "word_len_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# look at longest and shorted data\n",
    "sorted_data = sorted(data, key=lambda x: len(x.split()))\n",
    "print(sorted_data[0])\n",
    "print(sorted_data[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get distribution of words\n",
    "word_count = Counter([word for x in data for word in x.split()])\n",
    "word_count.most_common(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Named Entity Recognition\n",
    "\n",
    "We will frame our task as a Named Entity Recognition (NER) problem.\n",
    "\n",
    "Named Entity Recognition (NER) is a machine learning technique used to identify and classify named entities in text. To put simply the model will be tasked to recognize a few different type of keywords.\n",
    "\n",
    "In this case, we want to recognize services and environments related entities.\n",
    "\n",
    "The input is a string, the output (aka target) is the span where the entities exist.\n",
    "\n",
    "For example:\n",
    "\n",
    "Text:\n",
    "load balancer down in production-3\n",
    "\n",
    "Target:\n",
    "\n",
    "(0, 2, SERVICE)\n",
    "\n",
    "(4, 5, ENVIRONMENT)\n",
    "\n",
    "## Model training\n",
    "\n",
    "How does model training work?\n",
    "\n",
    "- present inputs to the model\n",
    "- ask the model to predict the output for every input\n",
    "- compare the model output to the actual target\n",
    "- repeat until the model is sufficiently good at the task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Representation\n",
    "\n",
    "### How do we represent the text input to feed into the model?\n",
    "\n",
    "1. Tokenization: break a string into \"reasonable\" substring\n",
    "\n",
    "2. Encoding: map a token sequence into a integer id sequence\n",
    "\n",
    "The simplest way to make tokens is to use words, most modern models may break down some words into multiple token. But when we think about tokens, we can roughly think of them as words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# simple tokenizer\n",
    "\n",
    "import re\n",
    "\n",
    "def simple_tokenizer(text):\n",
    "    return re.findall(r'\\b\\w+\\b', text.lower())\n",
    "\n",
    "simple_token = simple_tokenizer(\"Hello, world! Hello, This is a test. Is this a test?\")\n",
    "print(f\"Tokenized input: {simple_token}\")\n",
    "\n",
    "# token to id dict\n",
    "id_to_token = {i: token for i, token in enumerate(set(simple_token))}\n",
    "token_to_id = {token: i for i, token in id_to_token.items()}\n",
    "\n",
    "print(\"Encoding dictionaries:\\n\", id_to_token)\n",
    "print(token_to_id)\n",
    "\n",
    "simple_id = [token_to_id[token] for token in simple_token]\n",
    "print(f\"Final input representation: {simple_id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How do we represent the output for an NER task?\n",
    "\n",
    "We could represent it using the format above. A tuple of the format `(start, end, type)`, e.g. `(0, 2, SERVICE)`\n",
    "\n",
    "This format is great for storage and evaluating correctness.\n",
    "\n",
    "However, we don't actually know how many spans will be in the output at input time. So we need to design the model to accommodate an arbitrary number of outputs.\n",
    "\n",
    "\n",
    "An alternative way is to map each token to a values. This way we can ask the model to just make one prediction for every token.\n",
    "\n",
    "This is often done using the BIO (Beginning Inside Outside) format. We will label the first token of each span of interest with `B-<label>` any subsequent tokens with `I-<label>`. Any irrelevant tokens gets the `O` label\n",
    "\n",
    "In our case, we are trying to predict Services (SERVICE) and Environments (ENVIRONMENT)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Text: load balancer down in production-3\n",
    "\n",
    "label = [\n",
    "    \"B-SERVICE\",  # load\n",
    "    \"I-SERVICE\",  # balancer\n",
    "    \"O\",      # down\n",
    "    \"O\",      # in\n",
    "    \"B-ENVIRONMENT\",  # production-3\n",
    "]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}