notebooks/4.0-final_solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final solution\n",
    "by Domrachev Ivan, B20-RO-01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "# My packages\n",
    "# Love you, python <3\n",
    "import sys \n",
    "sys.path.append('../src/')\n",
    "# The packages themselves\n",
    "from models.llama.inference import llama2_predict \n",
    "from models.t5_small.inference import t5_predict\n",
    "# from models.t5_small.train import train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook provides a guide on training and running inference of `t5-small` and `llama2-7b-chat` models. Both of the models are already finetuned, so you can start right from the inference part without specifying the path to the model -- it will automatically pull it from huggingface.co 🤗!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two options:\n",
    "1. Create new instance of the dataset\n",
    "2. Load it from huggingface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the first one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.make_dataset import DetoxDataset\n",
    "\n",
    "# This command would access the tsv file and then save it to the specified folder \n",
    "DetoxDataset(dataset_sv_fname='../data/raw/filtered.tsv', dataset_arrow_fdir='data/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And this is the second:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['reference', 'translation', 'similarity', 'lenght_diff', 'ref_tox', 'trn_tox'],\n",
       "        num_rows: 156516\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['reference', 'translation', 'similarity', 'lenght_diff', 'ref_tox', 'trn_tox'],\n",
       "        num_rows: 17391\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds = load_dataset('domrachev03/toxic_comments_subset')\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## T5-small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> *Note*. Unfortunately, I have messed up my system, and now I'm not able to launch everything, requiring `bitsandbytes` library. Therefore, there is a little possibility that this code would not actually launch. However, one might refer to the notebooks, which are 100% functional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = train(ds)\n",
    "\n",
    "# and then...\n",
    "# model.save_pretrained(\"path/to/save\", from_pt=True)\n",
    "# or\n",
    "# model.push_to_hub(\"where/to/push\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I hate you!', \"I'm busy!\"]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t5_predict([\"I fucking hate you!\", \"Fuck off, I'm busy!\"], device='cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Llama2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modal.com training & inference "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, you have to gain access to the service:\n",
    "1. Regiseter in [modal.com](https://modal.com/) (1 minute, requires GitHub authentication)\n",
    "2. Enter secret from Huggingface (enter the hf token in the `HUGGINGFACE_TOKEN` field and name it `huggingface`), which could be found in the `Settings/API tokens`.\n",
    " \n",
    "The tool is much easier to use via the terminal, because it generates way too much output. Here is the list of commands to launch it in CLI (and corresponding cell with these commands):\n",
    "```bash\n",
    "# Authorization in modal account\n",
    "modal token new   \n",
    "# Launch training process\n",
    "modal run src/models/llama/train_modal.py --dataset llama2_dataset.py --base chat7 --run-id chat7-nontoxic\n",
    "# Copying PEFT pretrained model from modal cloud to local dir\n",
    "modal volume get example-results-vol 'chat7-nontoxic/*' models/llama2 \n",
    "# Running inference for the model in cloud\n",
    "modal run inference.py --base chat7 --run-id chat7-nontoxic --prompt \"[INST]<<SYS>>\\nYou are a Twitch moderator that paraphrases sentences to be non-toxic.\\n<<SYS>> \\n\\nCould you paraphrase this: ...?\\n [/INST]\"\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Local inference "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your PC is powerful enough, then you might try to launch the inference offline using `llama.inference.llama2_predict`. Note, that even with quantified version, execution requires 16 Gb of VRAM. Moreover, it could not launch on the CPU, since the quantization is available only for GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from models.llama.inference import llama2_predict\n",
    "\n",
    "# God bless you\n",
    "llama2_predict([\"I fucking hate you!\", \"Fuck off, I'm busy!\"])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml_courses",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}