Stanford_cs229.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cf8ed19c-c7d8-41de-8316-f0ea9630ea46",
   "metadata": {},
   "source": [
    "https://lena-voita.github.io/nlp_course/language_modeling.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b40dd7ed-83bd-4dd3-b46c-ec5f3f1f6aef",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-21T12:23:20.212534Z",
     "iopub.status.busy": "2024-11-21T12:23:20.211984Z",
     "iopub.status.idle": "2024-11-21T12:23:20.219996Z",
     "shell.execute_reply": "2024-11-21T12:23:20.218621Z",
     "shell.execute_reply.started": "2024-11-21T12:23:20.212490Z"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2959ec59-2af1-4f29-97a7-783ec1d3c966",
   "metadata": {},
   "source": [
    "- what matters when training LLMs\n",
    "    - Architecture\n",
    "    - Training Algorithms/loss\n",
    "        - post-training dataset\n",
    "            - openassistant\n",
    "            - alpaca: use LLMs to scale data collection\n",
    "            - LIMA: you need very little data for SFT! ~few thousands\n",
    "        - pre-training => post-training\n",
    "            - loss 相同，但 dataset (tasks) 不同\n",
    "            - different type of hyperparameters\n",
    "                - at the end of pretraining you essentially end up with a learning rate of 0\n",
    "                - in post-training, you increase your learning rate: $1e-5$\n",
    "    - Data\n",
    "    - Evaluation: PPL\n",
    "        - pretrained benchmark: MMLU\n",
    "        - 单选题：基于的是likelihood，即 likelihood of llm to predict that vs other options\n",
    "            - likelihood of generating A/B/C/D, which is most likely.\n",
    "            - or A/B/C/D which one is the most likely.\n",
    "    - Systems"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5287a220-a66e-4ea1-9d3c-3fb5188af75d",
   "metadata": {},
   "source": [
    "### language modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcbc3b6a-fc85-45ff-87ce-92e85158ef2b",
   "metadata": {},
   "source": [
    "- language modeling\n",
    "    - probability distribution over sequences of tokens/words $p(x_1,\\cdots, p_L)$\n",
    "- LMs are generative models: $x_{1:L}\\sim p(x_1,\\cdots,x_L)$\n",
    "    - 因为语言模型是一个概率分布，生成就是从分布中采样 sampling；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4548a7f-8db9-4351-ab9c-8d1b125a5049",
   "metadata": {},
   "source": [
    "### autoregressive"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fe4e512-5596-4a26-a104-e139eac7d5c4",
   "metadata": {},
   "source": [
    "- Autoregressive (AR) language models\n",
    "    - $p(x_1,\\cdots, x_L)=p(x_1)p(x_2|x_1)p(x_3|x_1,x_2)\\cdots$\n",
    "    - 注意，这个不是近似，而是 chain rule of probability\n",
    "    - you only need a model that can predict the next token given past context.\n",
    "- tasks & steps\n",
    "    - task: predict the next word\n",
    "    - steps:\n",
    "        - tokenize\n",
    "        - forward\n",
    "        - **predict probability of next token**\n",
    "        - sample\n",
    "        - detokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c5f45272-766f-4277-955e-6bc24ae18c47",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-21T12:23:36.146481Z",
     "iopub.status.busy": "2024-11-21T12:23:36.145879Z",
     "iopub.status.idle": "2024-11-21T12:23:36.158230Z",
     "shell.execute_reply": "2024-11-21T12:23:36.155987Z",
     "shell.execute_reply.started": "2024-11-21T12:23:36.146436Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://lena-voita.github.io/resources/lectures/lang_models/neural/nn_lm_idea_linear-min.png\" width=\"500\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://lena-voita.github.io/resources/lectures/lang_models/neural/nn_lm_idea_linear-min.png', width=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c541b123-00f1-42fa-9e5a-f1dcb25b36c6",
   "metadata": {},
   "source": [
    "### loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "24b6fa98-d8d3-481a-bcaf-f3712d72989d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-21T12:24:32.188052Z",
     "iopub.status.busy": "2024-11-21T12:24:32.187477Z",
     "iopub.status.idle": "2024-11-21T12:24:32.199860Z",
     "shell.execute_reply": "2024-11-21T12:24:32.197957Z",
     "shell.execute_reply.started": "2024-11-21T12:24:32.188006Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://lena-voita.github.io/resources/lectures/lang_models/neural/one_step_loss_intuition-min.png\" width=\"500\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://lena-voita.github.io/resources/lectures/lang_models/neural/one_step_loss_intuition-min.png', width=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e59949bf-8be5-481a-9cb5-59f2f2323fda",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-21T12:26:36.199362Z",
     "iopub.status.busy": "2024-11-21T12:26:36.198737Z",
     "iopub.status.idle": "2024-11-21T12:26:36.212057Z",
     "shell.execute_reply": "2024-11-21T12:26:36.209455Z",
     "shell.execute_reply.started": "2024-11-21T12:26:36.199318Z"
    }
   },
   "source": [
    "- maximize text's log-likelihood = minimum the cross entropy loss\n",
    "\n",
    "$$\n",
    "\\max \\prod_{i} p(x_i | x_{1:i-1}) = \\min \\left( - \\sum_{i} \\log p(x_i | x_{1:i-1}) \\right) = \\min \\mathcal{L}(x_{1:L})\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00dbce8b-5364-42fe-9c11-6c34d18db35d",
   "metadata": {},
   "source": [
    "### tokenizer & BPE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecdad42c-473e-43f3-8b33-77ed91a177ef",
   "metadata": {},
   "source": [
    "- Take large corpus of text\n",
    "- Start with one token per character\n",
    "- Merge common pairs of tokens into a token\n",
    "- Repeat until desired vocab size or all merged\n",
    "- 关于空格\n",
    "    - `to` vs. ` to`  vs. `to ` vs. `to\\n`\n",
    "    - 基本每个 token，都会有 `token` 和 ` token` 两个版本\n",
    "- 匹配时是最长匹配；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "58eb9211-0998-4996-b588-57482da2e314",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-21T13:54:53.578372Z",
     "iopub.status.busy": "2024-11-21T13:54:53.577733Z",
     "iopub.status.idle": "2024-11-21T13:54:53.589480Z",
     "shell.execute_reply": "2024-11-21T13:54:53.587841Z",
     "shell.execute_reply.started": "2024-11-21T13:54:53.578326Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  [220]\n",
      "\n",
      " [198]\n",
      "to [935]\n",
      "to  [935, 220]\n",
      " to [316]\n",
      "to\n",
      " [935, 198]\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "tokenizer = tiktoken.get_encoding(\"o200k_base\")\n",
    "for token in [' ', '\\n', 'to', 'to ', ' to', 'to\\n']:\n",
    "    print(token, tokenizer.encode(token))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f118227e-064c-4e9b-ba51-c9d83c2b9204",
   "metadata": {},
   "source": [
    "### PPL for evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b159c639-1a9c-4b2e-bb47-222a4c44e642",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{split}\n",
    "PPL(x_{1:L})=2^{\\frac1L\\mathcal L(x_{1:L})}=\\Pi_{1}^L p(x_i|x_{1:i-1})^{-1/L}\\\\\n",
    "\\mathcal L(x_{1:L})=-\\sum_i\\log p(x_i|x_{1:i-1})\n",
    "\\end{split}\n",
    "$$\n",
    "\n",
    "- PPL: between 1 and |vocab|\n",
    "- intuition: number of tokens that you are hesitating between"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fe42348-1582-4a17-9c20-385ed91e3d45",
   "metadata": {},
   "source": [
    "### Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0645add7-eec5-42ac-941e-b989415181d6",
   "metadata": {},
   "source": [
    "- Note: internet is dirty & not representative of what we want. Practice:\n",
    "\n",
    "1. Download all of internet. Common crawl: 250 billion pages, > 1PB (> 1e6 GB)\n",
    "2. Text extraction from HTML (challenges: math, boiler plate)\n",
    "3. Filter undesirable content (e.g. NSFW, harmful content, PII)\n",
    "4. Deduplicates (url/document/line). E.g. all the headers/footers/menu in forums are always same\n",
    "5. Heuristic filtering. Rm low quality documents (e.g. # words, word length, outlier toks, dirty toks)\n",
    "6. Model based filtering. Predict if page could be references by Wikipedia.\n",
    "7. **Data mix**. Classify data categories (code/books/entertainment). **Reweight domains** using scaling laws to get high downstream performance.\n",
    "\n",
    "- Also: lr annealing on high-quality data, continual pretraining with longer context\n",
    "    - overfitting your model on a very high quality data\n",
    "        - high quality data: (expert) wikipedia \n",
    "        - low quality data: human data \n",
    "    - Learning rate annealing that considers data quality\n",
    "        - Higher learning rate for high quality data, lower for low quality"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86267bbe-a3b4-4f0f-8f80-81041741072b",
   "metadata": {},
   "source": [
    "### SFT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ca748cf-b12f-48d4-bab5-9715a4f87d10",
   "metadata": {},
   "source": [
    "\n",
    "> finetune the LLM with **language modeling**(next token prediction) of the **desired answers**(supervised)\n",
    "\n",
    "- LIMA: you need very little data for SFT! ~few thousands\n",
    "- Just learns the format of desired answers (length, bullet points, ...)\n",
    "    - the knowledge (every user) is already in the pretrained LLM!\n",
    "    - **specializes** to one \"type of user\"\n",
    "- intuition：all you learn is you learn how to format your desired answers.\n",
    "    - your pretrained models, they essentially model the distribution of **every user** on internet,\n",
    "        - one might write bullet points, another one might answer question with an answer\n",
    "    - all you tell your model is like, wait, you should actually be optimizing more for **this type of user than another one**.\n",
    "        - so you're **not actually teaching** it -- you are not teaching anything through this SFT,\n",
    "        - in SFT all you do is tell the model to optimize for one type of user **that is saw already in a pretrained data set**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1eda3f28-0fcf-4691-94a6-6d372351b472",
   "metadata": {},
   "source": [
    "### RLHF"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "709b8f1c-8485-4516-b881-a3d8aba47632",
   "metadata": {},
   "source": [
    "- not purely LLMs, not purely humans.\n",
    "- Problem：SFT is **behavior cloning** of humans (RLHF maximize human preference rather than clone their behavior)\n",
    "    - bound by human abilities: human may prefer things that they are not able to generate\n",
    "        - reading a book vs. writing a book\n",
    "    - hallucination：cloning correct answer teaches LLM to hallucinate if it didn't know about it.\n",
    "    - price: collecting ideal answers is expensive.\n",
    "- reward modeling\n",
    "    - binary reward doesn't have much information\n",
    "    - train a reward model $R(\\cdot)$ using a logistic regression loss the classify preference\n",
    "        - $p(i\\gt j)=\\frac{\\exp(R(x,\\hat y_j))}{\\exp(R(x,\\hat y_i))+\\exp(R(x,\\hat y_j))}$\n",
    "        - Use logits $R(\\cdot)$ as reward => continuous information => information heavy!\n",
    "- optimize $\\mathbb{E}_{\\hat{y} \\sim p_\\theta(\\hat{y} | x)} \\left[ R(x, \\hat{y}) - \\beta \\log \\frac{p_\\theta(\\hat{y} | x)}{p_{\\text{ref}}(\\hat{y} | x)} \\right]$ using PPO\n",
    "    - regularization avoids overoptimization\n",
    "    - LMs are policies (actors) not a model some distribution.\n",
    "- RLHF: challenges of human data\n",
    "    - slow & expensive\n",
    "    - hard to focus on correctness rather than form (eg length)\n",
    "        - https://arxiv.org/pdf/2310.03716\n",
    "        - https://github.com/PrasannS/rlhf-length-biases\n",
    "    - llm opinion\n",
    "        - https://arxiv.org/pdf/2303.17548"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ce26ea9-683a-4e65-bca7-63f18b5cef20",
   "metadata": {},
   "source": [
    "### DPO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e151586b-057e-4e1e-b649-990ce3d75e9f",
   "metadata": {},
   "source": [
    "- idea: maximize probability of **prefered output** ($x, y_w$), minimize **the other** ($x,y_\\ell$)\n",
    "    - DPO can only use the labeled data\n",
    "$$\n",
    "\\mathcal{L}_{\\text{DPO}}(\\pi_\\theta; \\pi_{\\text{ref}}) = -\\mathbb{E}_{(x, y_w, y_l) \\sim \\mathcal{D}} \\left[ \\log \\sigma \\left( \\beta \\log \\frac{\\pi_\\theta(y_w | x)}{\\pi_{\\text{ref}}(y_w | x)} - \\beta \\log \\frac{\\pi_\\theta(y_l | x)}{\\pi_{\\text{ref}}(y_l | x)} \\right) \\right]\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ec75ab9-c496-425c-a166-bbe7b3e2bcf0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}