run_training_dpo_pipeline.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kingzone/kaggle/blob/master/run_training_dpo_pipeline.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Training Pipeline\n",
        "[run_training_dpo_pipeline.ipynb](https://github.com/shibing624/MedicalGPT/blob/main/run_training_dpo_pipeline.ipynb)    | [Open In Colab](https://colab.research.google.com/github/shibing624/MedicalGPT/blob/main/run_training_dpo_pipeline.ipynb)"
      ],
      "metadata": {
        "collapsed": false,
        "id": "tjksQ7bBLy0h"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "Pg5oglenLy0k"
      },
      "source": [
        "# Stage 1: Continue Pretraining\n",
        "\n",
        "第一阶段：PT(Continue PreTraining)增量预训练，在海量领域文本数据上二次预训练GPT模型，以注入领域知识\n",
        "\n",
        "| Stage 1: Continue Pretraining   |  [pretraining.py](https://github.com/shibing624/MedicalGPT/blob/main/pretraining.py) | [run_pt.sh](https://github.com/shibing624/MedicalGPT/blob/main/run_pt.sh)    |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QYj_bwYTLy0l"
      },
      "source": [
        "#### 说明：\n",
        "以下 notebook/colab 代码为了快速验证训练代码可用，我们使用了小size的生成模型和小样本数据集，实际使用时，需要使用更大的模型和数据集，以获得更好的效果。\n",
        "\n",
        "1. 生成模型：使用的是Bloom的`bigscience/bloomz-560m`\n",
        "2. 数据集：PT阶段使用的是中文天龙八部小说部分文本和英文书籍部分文本，位于`data/pretrain`文件夹"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "collapsed": false,
        "id": "GSufR34kLy0l"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HDaTkJnNLy0m"
      },
      "source": [
        "## 配置运行环境\n",
        "\n",
        "本地执行可注释以下配置环境的命令，colab执行要打开注释，用于配置环境\n",
        "\n",
        "colab建议使用T4 GPU训练，设置方式：`代码执行程序 -> 更改运行时类型 -> 运行时类型：Python3，硬件加速器：GPU，GPU类型：T4 -> 保存`\n",
        "\n",
        "步骤：\n",
        "1. 下载最新代码到本地\n",
        "2. 安装依赖包\n",
        "\n",
        "依赖包如下，保证最新版本：\n",
        "\n",
        "```\n",
        "loguru\n",
        "transformers\n",
        "sentencepiece\n",
        "datasets\n",
        "tensorboard\n",
        "tqdm\n",
        "peft\n",
        "trl\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "GbTzZH6SLy0m",
        "outputId": "ac6557cd-42e8-4bdb-b6e9-b532de36b1aa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'MedicalGPT'...\n",
            "remote: Enumerating objects: 58, done.\u001b[K\n",
            "remote: Counting objects: 100% (58/58), done.\u001b[K\n",
            "remote: Compressing objects: 100% (55/55), done.\u001b[K\n",
            "remote: Total 58 (delta 10), reused 31 (delta 0), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (58/58), 7.90 MiB | 21.35 MiB/s, done.\n",
            "Resolving deltas: 100% (10/10), done.\n",
            "/content/MedicalGPT\n",
            "build_domain_tokenizer.py  pretraining.py\n",
            "CITATION.cff               README_EN.md\n",
            "_config.yml                README.md\n",
            "CONTRIBUTING.md            requirements.txt\n",
            "convert_dataset.py         reward_modeling.py\n",
            "\u001b[0m\u001b[01;34mdata\u001b[0m/                      rl_training.py\n",
            "deepspeed_config.json      run_dpo.sh\n",
            "DISCLAIMER                 run_pt.sh\n",
            "\u001b[01;34mdocs\u001b[0m/                      run_rl.sh\n",
            "dpo_training.py            run_rm.sh\n",
            "gradio_demo.py             run_sft.sh\n",
            "inference.py               run_training_dpo_pipeline.ipynb\n",
            "LICENSE                    run_training_pipeline.ipynb\n",
            "merge_peft_adapter.py      supervised_finetuning.py\n",
            "merge_tokenizers.py\n",
            "Collecting loguru (from -r requirements.txt (line 1))\n",
            "  Downloading loguru-0.7.0-py3-none-any.whl (59 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m60.0/60.0 kB\u001b[0m \u001b[31m1.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting transformers>=4.30.1 (from -r requirements.txt (line 2))\n",
            "  Downloading transformers-4.32.0-py3-none-any.whl (7.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.5/7.5 MB\u001b[0m \u001b[31m12.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sentencepiece (from -r requirements.txt (line 3))\n",
            "  Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m27.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting datasets (from -r requirements.txt (line 4))\n",
            "  Downloading datasets-2.14.4-py3-none-any.whl (519 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m519.3/519.3 kB\u001b[0m \u001b[31m24.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 5)) (4.66.1)\n",
            "Requirement already satisfied: tensorboard in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 6)) (2.12.3)\n",
            "Collecting peft>=0.5.0 (from -r requirements.txt (line 8))\n",
            "  Downloading peft-0.5.0-py3-none-any.whl (85 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m85.6/85.6 kB\u001b[0m \u001b[31m10.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting accelerate>=0.20.3 (from -r requirements.txt (line 9))\n",
            "  Downloading accelerate-0.22.0-py3-none-any.whl (251 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m251.2/251.2 kB\u001b[0m \u001b[31m24.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting trl>=0.6.0 (from -r requirements.txt (line 10))\n",
            "  Downloading trl-0.6.0-py3-none-any.whl (110 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.0/110.0 kB\u001b[0m \u001b[31m12.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers>=4.30.1->-r requirements.txt (line 2)) (3.12.2)\n",
            "Collecting huggingface-hub<1.0,>=0.15.1 (from transformers>=4.30.1->-r requirements.txt (line 2))\n",
            "  Downloading huggingface_hub-0.16.4-py3-none-any.whl (268 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m28.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.30.1->-r requirements.txt (line 2)) (1.23.5)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.30.1->-r requirements.txt (line 2)) (23.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.30.1->-r requirements.txt (line 2)) (6.0.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.30.1->-r requirements.txt (line 2)) (2023.6.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers>=4.30.1->-r requirements.txt (line 2)) (2.31.0)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers>=4.30.1->-r requirements.txt (line 2))\n",
            "  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m41.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting safetensors>=0.3.1 (from transformers>=4.30.1->-r requirements.txt (line 2))\n",
            "  Downloading safetensors-0.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m53.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 4)) (9.0.0)\n",
            "Collecting dill<0.3.8,>=0.3.0 (from datasets->-r requirements.txt (line 4))\n",
            "  Downloading dill-0.3.7-py3-none-any.whl (115 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m115.3/115.3 kB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 4)) (1.5.3)\n",
            "Collecting xxhash (from datasets->-r requirements.txt (line 4))\n",
            "  Downloading xxhash-3.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m23.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting multiprocess (from datasets->-r requirements.txt (line 4))\n",
            "  Downloading multiprocess-0.70.15-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m16.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 4)) (2023.6.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 4)) (3.8.5)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 6)) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 6)) (1.57.0)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 6)) (2.17.3)\n",
            "Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 6)) (1.0.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 6)) (3.4.4)\n",
            "Requirement already satisfied: protobuf>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 6)) (3.20.3)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 6)) (67.7.2)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 6)) (0.7.1)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 6)) (2.3.7)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 6)) (0.41.2)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from peft>=0.5.0->-r requirements.txt (line 8)) (5.9.5)\n",
            "Requirement already satisfied: torch>=1.13.0 in /usr/local/lib/python3.10/dist-packages (from peft>=0.5.0->-r requirements.txt (line 8)) (2.0.1+cu118)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (23.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (3.2.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (6.0.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (4.0.3)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (1.9.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (1.4.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (1.3.1)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 6)) (5.3.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 6)) (0.3.0)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 6)) (1.16.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 6)) (4.9)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard->-r requirements.txt (line 6)) (1.3.1)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.15.1->transformers>=4.30.1->-r requirements.txt (line 2)) (4.7.1)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.30.1->-r requirements.txt (line 2)) (3.4)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.30.1->-r requirements.txt (line 2)) (2.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.30.1->-r requirements.txt (line 2)) (2023.7.22)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft>=0.5.0->-r requirements.txt (line 8)) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft>=0.5.0->-r requirements.txt (line 8)) (3.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft>=0.5.0->-r requirements.txt (line 8)) (3.1.2)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft>=0.5.0->-r requirements.txt (line 8)) (2.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.13.0->peft>=0.5.0->-r requirements.txt (line 8)) (3.27.2)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.13.0->peft>=0.5.0->-r requirements.txt (line 8)) (16.0.6)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard->-r requirements.txt (line 6)) (2.1.3)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->-r requirements.txt (line 4)) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->-r requirements.txt (line 4)) (2023.3)\n",
            "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 6)) (0.5.0)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard->-r requirements.txt (line 6)) (3.2.2)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.13.0->peft>=0.5.0->-r requirements.txt (line 8)) (1.3.0)\n",
            "Installing collected packages: tokenizers, sentencepiece, safetensors, xxhash, loguru, dill, multiprocess, huggingface-hub, transformers, datasets, accelerate, trl, peft\n",
            "Successfully installed accelerate-0.22.0 datasets-2.14.4 dill-0.3.7 huggingface-hub-0.16.4 loguru-0.7.0 multiprocess-0.70.15 peft-0.5.0 safetensors-0.3.3 sentencepiece-0.1.99 tokenizers-0.13.3 transformers-4.32.0 trl-0.6.0 xxhash-3.3.0\n"
          ]
        }
      ],
      "source": [
        "!git clone --depth 1 https://github.com/shibing624/MedicalGPT.git\n",
        "%cd MedicalGPT\n",
        "%ls\n",
        "!pip install -r requirements.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o6rIQYn9Ly0n"
      },
      "source": [
        "## Stage1 咱们开始吧\n",
        "\n",
        "训练步骤如下：\n",
        "\n",
        "1. 确认训练集\n",
        "2. 执行训练脚本\n",
        "\n",
        "训练脚本的执行逻辑如下：\n",
        "1. 导入依赖包\n",
        "2. 设置参数\n",
        "3. 定义各函数并加载训练集\n",
        "4. 加载模型和tokenizer\n",
        "5. 开始训练并评估\n",
        "6. 查看训练结果\n",
        "\n",
        "**以下参数可以根据你的GPU实际情况修改，当前参数是根据Colab的T4单卡GPU（16GB显存）配置的**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RegIH3ImLy0o"
      },
      "outputs": [],
      "source": [
        "%ls ./data/pretrain/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "!python pretraining.py \\\n",
        "    --model_type bloom \\\n",
        "    --model_name_or_path bigscience/bloomz-560m \\\n",
        "    --train_file_dir ./data/pretrain \\\n",
        "    --validation_file_dir ./data/pretrain \\\n",
        "    --per_device_train_batch_size 3 \\\n",
        "    --per_device_eval_batch_size 3 \\\n",
        "    --do_train \\\n",
        "    --do_eval \\\n",
        "    --use_peft True \\\n",
        "    --seed 42 \\\n",
        "    --fp16 \\\n",
        "    --max_train_samples 10000 \\\n",
        "    --max_eval_samples 10 \\\n",
        "    --num_train_epochs 1 \\\n",
        "    --learning_rate 2e-4 \\\n",
        "    --warmup_ratio 0.05 \\\n",
        "    --weight_decay 0.01 \\\n",
        "    --logging_strategy steps \\\n",
        "    --logging_steps 10 \\\n",
        "    --eval_steps 50 \\\n",
        "    --evaluation_strategy steps \\\n",
        "    --save_steps 500 \\\n",
        "    --save_strategy steps \\\n",
        "    --save_total_limit 3 \\\n",
        "    --gradient_accumulation_steps 1 \\\n",
        "    --preprocessing_num_workers 1 \\\n",
        "    --block_size 1024 \\\n",
        "    --output_dir outputs-pt-v1 \\\n",
        "    --overwrite_output_dir \\\n",
        "    --ddp_timeout 30000 \\\n",
        "    --logging_first_step True \\\n",
        "    --target_modules all \\\n",
        "    --lora_rank 8 \\\n",
        "    --lora_alpha 16 \\\n",
        "    --lora_dropout 0.05 \\\n",
        "    --torch_dtype float16 \\\n",
        "    --device_map auto \\\n",
        "    --report_to tensorboard \\\n",
        "    --ddp_find_unused_parameters False \\\n",
        "    --gradient_checkpointing True"
      ],
      "metadata": {
        "id": "5us3qzzkLy0p"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6GPceKE5Ly0p"
      },
      "outputs": [],
      "source": [
        "%ls -lh outputs-pt-v1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2N_o6CbSLy0p"
      },
      "source": [
        "模型训练结果：\n",
        "- 使用lora训练模型，则保存的lora权重是`adapter_model.bin`, lora配置文件是`adapter_config.json`，合并到base model的方法见`merge_peft_adapter.py`\n",
        "- 日志保存在`output_dir/runs`目录下，可以使用tensorboard查看，启动tensorboard方式如下：`tensorboard --logdir output_dir/runs --host 0.0.0.0 --port 8009`"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "lora模型权重合并到base model，合并后的模型保存在`--output_dir`目录下，合并方法如下："
      ],
      "metadata": {
        "collapsed": false,
        "id": "wf-7oBa5Ly0q"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "!python merge_peft_adapter.py --model_type bloom \\\n",
        "    --base_model_name_or_path bigscience/bloomz-560m --peft_model_path outputs-pt-v1 --output_dir merged-pt/"
      ],
      "metadata": {
        "id": "uXSzob-pLy0q"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "%ls -lh merged-pt/"
      ],
      "metadata": {
        "id": "SoTlfY2vLy0q"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "%cat merged-pt/config.json"
      ],
      "metadata": {
        "id": "G45URcY6Ly0q"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7cIGekQvLy0q"
      },
      "source": [
        "Stage1 增量预训练完成。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "start_time": "2023-06-15T13:56:17.032821Z",
          "end_time": "2023-06-15T13:56:17.081153Z"
        },
        "id": "A5agusrSLy0r"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Stage 2: Supervised FineTuning\n",
        "\n",
        "第二阶段：SFT(Supervised Fine-tuning)有监督微调，构造指令微调数据集，在预训练模型基础上做指令精调，以对齐指令意图\n",
        "\n",
        "| Stage 2: Supervised Fine-tuning | [supervised_finetuning.py](https://github.com/shibing624/MedicalGPT/blob/main/supervised_finetuning.py) | [run_sft.sh](https://github.com/shibing624/MedicalGPT/blob/main/run_sft.sh)  |"
      ],
      "metadata": {
        "collapsed": false,
        "id": "2bsGZE6zLy0r"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 说明：\n",
        "以下 notebook/colab 代码为了快速验证训练代码可用，我们使用了小size的生成模型和小样本数据集，实际使用时，需要使用更大的模型和数据集，以获得更好的效果。\n",
        "\n",
        "1. 生成模型：使用的是Bloom的`bigscience/bloomz-560m` 或者 Stage1得到的预训练模型\n",
        "2. 数据集：SFT阶段使用的是使用的是Belle的1千条抽样数据，位于`data/finetune`文件夹"
      ],
      "metadata": {
        "collapsed": false,
        "id": "fkXieuLdLy0r"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Stage2 咱们开始吧\n",
        "\n",
        "训练步骤如下：\n",
        "\n",
        "1. 确认训练集\n",
        "2. 执行训练脚本\n",
        "\n",
        "训练脚本的执行逻辑如下：\n",
        "1. 导入依赖包\n",
        "2. 设置参数\n",
        "3. 定义各函数并加载训练集\n",
        "4. 加载模型和tokenizer\n",
        "5. 开始训练并评估\n",
        "6. 查看训练结果"
      ],
      "metadata": {
        "collapsed": false,
        "id": "SK6L9YrFLy0r"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "%ls ./data/finetune"
      ],
      "metadata": {
        "ExecuteTime": {
          "start_time": "2023-06-15T13:58:38.778132Z",
          "end_time": "2023-06-15T13:58:38.966506Z"
        },
        "id": "p15hsXEDLy0r"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "!python supervised_finetuning.py \\\n",
        "    --model_type bloom \\\n",
        "    --model_name_or_path merged-pt \\\n",
        "    --train_file_dir ./data/finetune \\\n",
        "    --validation_file_dir ./data/finetune \\\n",
        "    --per_device_train_batch_size 4 \\\n",
        "    --per_device_eval_batch_size 4 \\\n",
        "    --do_train \\\n",
        "    --do_eval \\\n",
        "    --use_peft True \\\n",
        "    --fp16 \\\n",
        "    --max_train_samples 1000 \\\n",
        "    --max_eval_samples 10 \\\n",
        "    --num_train_epochs 1 \\\n",
        "    --learning_rate 2e-5 \\\n",
        "    --warmup_ratio 0.05 \\\n",
        "    --weight_decay 0.05 \\\n",
        "    --logging_strategy steps \\\n",
        "    --logging_steps 10 \\\n",
        "    --eval_steps 50 \\\n",
        "    --evaluation_strategy steps \\\n",
        "    --save_steps 500 \\\n",
        "    --save_strategy steps \\\n",
        "    --save_total_limit 3 \\\n",
        "    --gradient_accumulation_steps 1 \\\n",
        "    --preprocessing_num_workers 1 \\\n",
        "    --output_dir outputs-sft-v1 \\\n",
        "    --overwrite_output_dir \\\n",
        "    --ddp_timeout 30000 \\\n",
        "    --logging_first_step True \\\n",
        "    --target_modules all \\\n",
        "    --lora_rank 8 \\\n",
        "    --lora_alpha 16 \\\n",
        "    --lora_dropout 0.05 \\\n",
        "    --torch_dtype float16 \\\n",
        "    --device_map auto \\\n",
        "    --report_to tensorboard \\\n",
        "    --ddp_find_unused_parameters False \\\n",
        "    --gradient_checkpointing True"
      ],
      "metadata": {
        "id": "NtpB5CySLy0s"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "%ls -lh outputs-sft-v1"
      ],
      "metadata": {
        "id": "8vuN8buzLy0s"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "模型训练结果：\n",
        "- 使用lora训练模型，则保存的lora权重是`adapter_model.bin`, lora配置文件是`adapter_config.json`，合并到base model的方法见`merge_peft_adapter.py`\n",
        "- 日志保存在`output_dir/runs`目录下，可以使用tensorboard查看，启动tensorboard方式如下：`tensorboard --logdir output_dir/runs --host 0.0.0.0 --port 8009`"
      ],
      "metadata": {
        "collapsed": false,
        "id": "1ySv87UOLy0s"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "lora模型权重合并到base model，合并后的模型保存在`--output_dir`目录下，合并方法如下："
      ],
      "metadata": {
        "collapsed": false,
        "id": "x-bgTJcHLy0s"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "!python merge_peft_adapter.py --model_type bloom \\\n",
        "    --base_model_name_or_path merged-pt --peft_model_path outputs-sft-v1 --output_dir merged-sft/"
      ],
      "metadata": {
        "id": "R82PaWMpLy0t"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "%ls -lh merged-sft/"
      ],
      "metadata": {
        "id": "heIMMDiALy0t"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "%cat merged-sft/config.json"
      ],
      "metadata": {
        "id": "7lKIQj9ILy0t"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Stage2 SFT训练完成。"
      ],
      "metadata": {
        "collapsed": false,
        "id": "sA_0VhUuLy0t"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [],
      "metadata": {
        "ExecuteTime": {
          "start_time": "2023-06-15T14:07:40.731186Z",
          "end_time": "2023-06-15T14:07:40.752635Z"
        },
        "id": "Ld3JhDyZLy0t"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Stage 3: DPO(Direct Preference Optimization)\n",
        "\n",
        "第三阶段：DPO(Direct Preference Optimization)直接偏好优化，DPO通过直接优化语言模型来实现对其行为的精确控制，而无需使用复杂的强化学习，也可以有效学习到人类偏好，DPO相较于RLHF更容易实现且易于训练，效果更好\n",
        "\n",
        "| Stage 3: Direct Preference Optimization        |  [dpo_training.py](https://github.com/shibing624/MedicalGPT/blob/main/dpo_training.py) | [run_dpo.sh](https://github.com/shibing624/MedicalGPT/blob/main/run_dpo.sh)    |"
      ],
      "metadata": {
        "collapsed": false,
        "id": "OpWlKPVSLy0u"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 说明：\n",
        "以下 notebook/colab 代码为了快速验证训练代码可用，我们使用了小size的生成模型和小样本数据集，实际使用时，需要使用更大的模型和数据集，以获得更好的效果。\n",
        "\n",
        "1. 生成模型：使用的是Bloom的`bigscience/bloomz-560m` 或者 Stage2得到的SFT模型\n",
        "2. 数据集：DPO阶段使用的是医疗reward数据，抽样了500条，位于`data/reward`文件夹"
      ],
      "metadata": {
        "collapsed": false,
        "id": "j_PfFvVHLy0v"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Stage3 咱们开始吧\n",
        "\n",
        "训练步骤如下：\n",
        "\n",
        "1. 确认训练集\n",
        "2. 执行训练脚本\n",
        "\n",
        "训练脚本的执行逻辑如下：\n",
        "1. 导入依赖包\n",
        "2. 设置参数\n",
        "3. 定义各函数并加载训练集\n",
        "4. 加载模型和tokenizer\n",
        "5. 开始训练并评估\n",
        "6. 查看训练结果"
      ],
      "metadata": {
        "collapsed": false,
        "id": "WJLAz56pLy0v"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "%ls ./data/reward/"
      ],
      "metadata": {
        "id": "FgFjWqKmLy0v"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "!python dpo_training.py \\\n",
        "    --model_type bloom \\\n",
        "    --model_name_or_path merged-sft \\\n",
        "    --train_file_dir ./data/reward \\\n",
        "    --validation_file_dir ./data/reward \\\n",
        "    --per_device_train_batch_size 3 \\\n",
        "    --per_device_eval_batch_size 1 \\\n",
        "    --do_train \\\n",
        "    --do_eval \\\n",
        "    --use_peft True \\\n",
        "    --max_train_samples 1000 \\\n",
        "    --max_eval_samples 10 \\\n",
        "    --max_steps 100 \\\n",
        "    --eval_steps 10 \\\n",
        "    --save_steps 50 \\\n",
        "    --max_source_length 128 \\\n",
        "    --max_target_length 128 \\\n",
        "    --output_dir outputs-dpo-v1 \\\n",
        "    --target_modules all \\\n",
        "    --lora_rank 8 \\\n",
        "    --lora_alpha 16 \\\n",
        "    --lora_dropout 0.05 \\\n",
        "    --torch_dtype float16 \\\n",
        "    --fp16 True \\\n",
        "    --device_map auto \\\n",
        "    --report_to tensorboard \\\n",
        "    --remove_unused_columns False \\\n",
        "    --gradient_checkpointing True \\\n",
        "    --cache_dir ./cache"
      ],
      "metadata": {
        "id": "kpiorXI9Ly0v"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "%ls -lh outputs-dpo-v1"
      ],
      "metadata": {
        "id": "5mnjDsQ7Ly0v"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "模型训练结果：\n",
        "- 使用lora训练模型，则保存的lora权重是`adapter_model.bin`, lora配置文件是`adapter_config.json`，合并到base model的方法见`merge_peft_adapter.py`\n",
        "- 日志保存在`output_dir/runs`目录下，可以使用tensorboard查看，启动tensorboard方式如下：`tensorboard --logdir output_dir/runs --host 0.0.0.0 --port 8009`"
      ],
      "metadata": {
        "collapsed": false,
        "id": "1GVOn1HsLy0v"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "lora模型权重合并到base model，合并后的模型保存在`--output_dir`目录下，合并方法如下："
      ],
      "metadata": {
        "collapsed": false,
        "id": "o5-tP18yLy0w"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "!python merge_peft_adapter.py --model_type bloom \\\n",
        "    --base_model_name_or_path merged-sft --peft_model_path outputs-dpo-v1 --output_dir merged-dpo/"
      ],
      "metadata": {
        "id": "jjRiF_yFLy0w"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "%ls -lh merged-dpo/"
      ],
      "metadata": {
        "id": "vMnxLd2lLy07"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "%cat merged-dpo/config.json"
      ],
      "metadata": {
        "id": "lfHcVpJoLy07"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Stage3 偏好建模第一次训练完成。"
      ],
      "metadata": {
        "collapsed": false,
        "id": "a8uT4V_cLy07"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**至此一个完整的训练流程演示完成。**"
      ],
      "metadata": {
        "collapsed": false,
        "id": "RymXhi_YLy07"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [],
      "metadata": {
        "ExecuteTime": {
          "start_time": "2023-06-26T12:34:29.620609Z",
          "end_time": "2023-06-26T12:34:29.658428Z"
        },
        "id": "X24h16LILy08"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Test"
      ],
      "metadata": {
        "collapsed": false,
        "id": "Znji0UHELy08"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "!python inference.py --model_type bloom --base_model merged-dpo --interactive"
      ],
      "metadata": {
        "ExecuteTime": {
          "start_time": "2023-06-26T12:34:47.802087Z",
          "end_time": "2023-06-26T12:35:00.864463Z"
        },
        "id": "D6H6aDY2Ly08"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Input:介绍下南京\n",
        "Response:  南京市位于江苏省西南部，是全国首批历史文化名城、国家中心城市和自由贸易试验区。\n",
        "\n",
        "完。\n"
      ],
      "metadata": {
        "collapsed": false,
        "id": "ucJUBYjrLy08"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [],
      "metadata": {
        "id": "SdsTZge0Ly08"
      }
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "language": "python",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.13"
    },
    "vscode": {
      "interpreter": {
        "hash": "f34eed0bebedfc4b6ee51ced43d2c030fe3b92f13c149d072205ca200a67b1ec"
      }
    },
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}