03_data_preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data preparation\n",
    "\n",
    "Before we start training the model, we should take a look at the labelled data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "with open(\"data/labelled_data.json\") as f:\n",
    "    data = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize the data\n",
    "\n",
    "text_sample = [\" \".join(d[\"tokens\"]) for d in data]\n",
    "print(f\"Text Sample: {text_sample[:5]}\")\n",
    "label_sample = [d[\"bio_tags\"] for d in data]\n",
    "print(f\"Label Sample: {label_sample[:5]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's see the distribution of the labels\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "label_counter = Counter()\n",
    "for labels in [d[\"bio_tags\"] for d in data]:\n",
    "    label_counter.update(labels)\n",
    "\n",
    "print(label_counter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# look at data with LOC tag\n",
    "\n",
    "[d for d in data if \"B-LOC\" in d[\"bio_tags\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are not enough LOC tags in the data the model won't be able to meaningfully learn to recognize them.\n",
    "\n",
    "It's best to drop these data points and perhaps try location labels when there are more data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [d for d in data if \"B-LOC\" not in d[\"bio_tags\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate the span labels format for the data to help evaluation later\n",
    "\n",
    "from src.utils import extract_spans\n",
    "\n",
    "for d in data:\n",
    "    d[\"labels\"] = extract_spans(d[\"bio_tags\"])\n",
    "\n",
    "data[:2]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at the most common spans and all the unique spans for SVC and ENV\n",
    "\n",
    "span_counter = Counter()\n",
    "for d in data:\n",
    "    for span in d[\"labels\"]:\n",
    "        text = \" \".join(d[\"tokens\"][span[0]:span[1]]).lower()\n",
    "        span_counter.update([f\"{text}, {span[2]}\"])\n",
    "\n",
    "print(span_counter.most_common(5))\n",
    "\n",
    "svc_set = set()\n",
    "env_set = set()\n",
    "\n",
    "for d in data:\n",
    "    for span in d[\"labels\"]:\n",
    "        text = \" \".join(d[\"tokens\"][span[0]:span[1]]).lower()\n",
    "        if span[2] == \"SVC\":\n",
    "            svc_set.add(text)\n",
    "        elif span[2] == \"ENV\":\n",
    "            env_set.add(text)\n",
    "\n",
    "\n",
    "\n",
    "print(f\"SVC: {svc_set}\")\n",
    "print(f\"ENV: {env_set}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and Test Sets in Machine Learning\n",
    "\n",
    "\n",
    "During development of a ML model, data is split into **training** and **test sets** for: model learning and model evaluation.\n",
    "\n",
    "\n",
    "#### **1. Training Set:**\n",
    "\n",
    "- **Purpose:** The training set is used to **train** or fit the model. It includes both the input data and the corresponding correct outputs.\n",
    "\n",
    "- **Why:** The model learns to recognize patterns and make predictions by adjusting its parameters based on the training data.\n",
    "\n",
    "\n",
    "\n",
    "#### **2. Test Set:**\n",
    "\n",
    "- **Purpose:** The testing set is used to **evaluate** the performance of the model. It is separate from the training data and also includes the correct outputs.\n",
    "\n",
    "- **Why:** Using a separate testing set helps to assess how well the model can generalize to new, unseen data. This is crucial for understanding the model's effectiveness and ensuring it isn't just memorizing the training data (a problem known as overfitting).\n",
    "\n",
    "\n",
    "In zero shot learning, you won't need a training set, but you should still have a test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train, test = train_test_split(data, test_size=0.1, random_state=42)\n",
    "\n",
    "# save the split data\n",
    "with open(\"data/train.json\", \"w\") as f:\n",
    "    json.dump(train, f)\n",
    "with open(\"data/test.json\", \"w\") as f:\n",
    "    json.dump(test, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation\n",
    "\n",
    "We want to be able to measure how good our model is at complete the task.\n",
    "\n",
    "The evaluation process will take data that the model has not seen during training to evaluate how well the model is doing. We can decide how good our model is doing by calculating the evaluation metrics.\n",
    "\n",
    "\n",
    "There are typically two types of metrics:\n",
    "\n",
    "Online: to measure the AI/ML system performance in the real world or in production\n",
    "\n",
    "Offline: to measure the AI/ML system performance using existing data while building the model\n",
    "\n",
    "\n",
    "### Offline evaluation\n",
    "\n",
    "Decide important offline metrics we can use to measure the model performance:\n",
    "\n",
    "- Precision, recall, and f1\n",
    "- High recall: costly false negative, e.g. fraud detection\n",
    "- High precision: costly false positive, e.g. legal\n",
    "- Balanced: both important\n",
    "\n",
    "**Question**:  What kind of mix do we want in this case?\n",
    "\n",
    "\n",
    "### Online evaluation\n",
    "In online evaluation, we can measure success in two ways:\n",
    "- Implicit: user click through rate or adding suggestion to incident implies good labels\n",
    "- Explicit: suggest labels for user to accept/reject\n",
    "\n",
    "We can then get a proxy of how good the model is doing in production by uses this signals. (e.g. 5% CTR on suggestions)\n",
    "\n",
    "We compare this value either to an arbitrary goal we set or if we have a existing solution, we can compare the value of new solution vs old solution via A/B testing or shadow scoring.\n",
    "\n",
    "A/B testing: randomly assigns a model to use in production\n",
    "\n",
    "Shadow scoring: running the new model with production inputs, but don't use these inputs to end users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Offline evaluation\n",
    "\n",
    "sample_text = \"google drive service high latency for prod us 1\"\n",
    "\n",
    "prediction_1 = [\n",
    "    (\"google drive\", \"SVC\"),  # correct\n",
    "    (\"prod us 1\", \"ENV\"),     # correct\n",
    "]\n",
    "# 2/2\n",
    "\n",
    "prediction_2 = [\n",
    "    (\"google drive\", \"SVC\"),  # correct\n",
    "    (\"prod us 1\", \"SVC\"),     # incorrect\n",
    "]\n",
    "# 1/2\n",
    "\n",
    "prediction_2 = [\n",
    "    (\"google drive\", \"SVC\"),  # correct\n",
    "]\n",
    "# 1/2\n",
    "\n",
    "prediction_3 = [\n",
    "    (\"service\", \"SVC\"),       # incorrect\n",
    "]\n",
    "# 0/2\n",
    "\n",
    "prediction_4 = [\n",
    "]\n",
    "# 0/2\n",
    "\n",
    "# A prediction is considered correct if both the span and the label are correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.utils import evaluate_ner\n",
    "\n",
    "evaluate_ner(\n",
    "    [{\"tokens\": sample_text.split(), \"labels\": [(0, 2, 'SVC'), (6, 9, 'ENV')]}],\n",
    "    [{\"tokens\": sample_text.split(), \"labels\": [(0, 2, 'SVC'), (6, 9, 'SVC')]}],\n",
    ")\n",
    "# evaluate_ner(prediction_1, sample_text)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "multisignal-rca-K6LZVNnX-py3.11",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}