Skip to content

Latest commit

 

History

History
365 lines (310 loc) · 13.5 KB

README_en.md

File metadata and controls

365 lines (310 loc) · 13.5 KB
Raw

GLM-4-9B Chat dialogue model fine-tuning

In this demo, you will experience how to fine-tune the GLM-4-9B-Chat open source model (visual understanding model is not supported). Please strictly follow the steps in the document to avoid unnecessary errors.

Hardware check

The data in this document are tested in the following hardware environment. The actual operating environment requirements and the video memory occupied by the operation are slightly different. Please refer to the actual operating environment. The fine-tuned resource usage is set according to the configuration file in the configs folder

Test hardware information:

  • OS: Ubuntu 22.04
  • Memory: 512GB
  • Python: Python: 3.10.12 / 3.12.3 (Currently, you need to install nltk from the git source code if you use Python 3.12.3)
  • CUDA Version: 12.3
  • GPU Driver: 535.104.05
  • GPU: NVIDIA A100-SXM4-80GB * 8
Fine-tuning Model Fine-tuning solution GPU memory usage Weight save point size
GLM-4-9B-Chat lora (PEFT) 22G 17M
GLM-4-9B-Chat p-tuning v2 (PEFT) 21G 121M
GLM-4-9B-Chat SFT (Zero3 method) 80G (Each GPU, Need 8 GPUs) 20G
GLM-4V-9B lora (PEFT), Include EVA2CLIPModel 75G 37M
GLM-4V-9B SFT Not Support in this Code 28G

GLM-4V-9B fine-tuning cannot work properly with deepspeed, the official fine-tuning script only does the most basic fine-tuning solution, more optimizations require developers to explore on their own

Before starting fine-tuning, please install the dependencies in basic_demo and clone the latest model repos (Hugging Face) first. You also need to install the dependencies in this directory:

pip install -r requirements.txt

NOTE: Some codes in NLTK 3.8.1 might not yet be compatible with Python 3.12. For adaptation methods in such cases, please refer to issues #38.

Multi-round dialogue format

The multi-round dialogue fine-tuning example uses the GLM-4 dialogue format convention, adding different loss_mask to different roles to calculate loss for multiple rounds of replies in one calculation.

For data files, the sample uses the following format:

[
  {
    "messages": [
      {
        "role": "system",
        "content": "<system prompt text>",
        "tools": [
          {
            "name": "<tool name>",
            "args": {
              "<arg name>": "<arg value>"
            }
          }
          // Add more tools if needed
        ]
      },
      {
        "role": "user",
        "content": "<user prompt text>"
      },
      {
        "role": "assistant",
        "content": "<assistant response text>"
      },
      // If Tool Using
      {
        "role": "user",
        "content": "<user prompt text>"
      },
      {
        "role": "assistant",
        "content": "<assistant response text>"
      },
      {
        "role": "observation",
        "content": "<observation prompt text>"
      },
      {
        "role": "assistant",
        "content": "<assistant response observation>"
      },
      // Multi_turns
      {
        "role": "user",
        "content": "<user prompt text>"
      },
      {
        "role": "assistant",
        "content": "<assistant response text>"
      }
    ]
  }
]

This is a sample without tools:

{
  "messages": [
    {
      "role": "user",
      "content": "类型#裤*材质#牛仔布*风格#性感"
    },
    {
      "role": "assistant",
      "content": "3x1的这款牛仔裤采用浅白的牛仔面料为裤身材质，其柔然的手感和细腻的质地，在穿着舒适的同时，透露着清纯甜美的个性气质。除此之外，流畅的裤身剪裁将性感的腿部曲线彰显的淋漓尽致，不失为一款随性出街的必备单品。"
    }
  ]
}

This is a sample with tools:

{
  "messages": [
    {
      "role": "system",
      "content": "",
      "tools": [
        {
          "type": "function",
          "function": {
            "name": "get_recommended_books",
            "description": "Get recommended books based on user's interests",
            "parameters": {
              "type": "object",
              "properties": {
                "interests": {
                  "type": "array",
                  "items": {
                    "type": "string"
                  },
                  "description": "The interests to recommend books for"
                }
              },
              "required": [
                "interests"
              ]
            }
          }
        }
      ]
    },
    {
      "role": "user",
      "content": "Hi, I am looking for some book recommendations. I am interested in history and science fiction."
    },
    {
      "role": "assistant",
      "content": "{\"name\": \"get_recommended_books\", \"arguments\": {\"interests\": [\"history\", \"science fiction\"]}}"
    },
    {
      "role": "observation",
      "content": "{\"books\": [\"Sapiens: A Brief History of Humankind by Yuval Noah Harari\", \"A Brief History of Time by Stephen Hawking\", \"Dune by Frank Herbert\", \"The Martian by Andy Weir\"]}"
    },
    {
      "role": "assistant",
      "content": "Based on your interests in history and science fiction, I would recommend the following books: \"Sapiens: A Brief History of Humankind\" by Yuval Noah Harari, \"A Brief History of Time\" by Stephen Hawking, \"Dune\" by Frank Herbert, and \"The Martian\" by Andy Weir."
    }
  ]
}

This is a sample with VQA Task:

{
  "messages": [
    {
      "role": "user",
      "content": "图片中的动物是什么？",
      "image": "/root/images/0001.jpg"
    },
    {
      "role": "assistant",
      "content": "图片中有一只猫。"
    },
    {
      "role": "user",
      "content": "图片中的猫在做什么？"
    },
    {
      "role": "assistant",
      "content": "这只猫坐在或站在桌子上，桌上有很多食物。"
    }
  ]
}
  • The system role is optional, but if it exists, it must appear before the user role, and the system role can only appear once in a complete conversation (whether it is a single round or a multi-round conversation).
  • The tools field is optional, but if it exists, it must appear after the system role, and the tools field can only appear once in a complete conversation (whether it is a single round or a multi-round conversation). When the tools field exists, the system role must exist and the content field is empty.
  • GLM-4V-9B does not support the tools field and the system field. And image must be placed in the first message. The image field needs to contain the absolute path of the image.

Configuration file

The fine-tuning configuration file is located in the config directory, including the following files:

  1. ds_zereo_2 / ds_zereo_3.json: deepspeed configuration file.

  2. `lora.yaml / ptuning_v2

  3. .yaml / sft.yaml`: Configuration files for different modes of models, including model parameters, optimizer parameters, training parameters, etc. Some important parameters are explained as follows: + data_config section

  • train_file: File path of training dataset.
  • val_file: File path of validation dataset.
  • test_file: File path of test dataset.
  • num_proc: Number of processes to use when loading data.
  • max_input_length: Maximum length of input sequence.
  • max_output_length: Maximum length of output sequence.
  • training_args section
  • output_dir: Directory for saving model and other outputs.
  • max_steps: Maximum number of training steps.
  • per_device_train_batch_size: Training batch size per device (such as GPU).
  • dataloader_num_workers: Number of worker threads to use when loading data.
  • remove_unused_columns: Whether to remove unused columns in data.
  • save_strategy: Model saving strategy (for example, how many steps to save).
  • save_steps: How many steps to save the model.
  • log_level: Log level (such as info).
  • logging_strategy: logging strategy.
  • logging_steps: how many steps to log at.
  • per_device_eval_batch_size: per-device evaluation batch size.
  • evaluation_strategy: evaluation strategy (e.g. how many steps to evaluate at).
  • eval_steps: how many steps to evaluate at.
  • predict_with_generate: whether to use generation mode for prediction.
  • generation_config section
  • max_new_tokens: maximum number of new tokens to generate.
  • peft_config section
  • peft_type: type of parameter tuning to use (supports LORA and PREFIX_TUNING).
  • task_type: task type, here is causal language model (don't change).
  • Lora parameters:
  • r: rank of LoRA.
  • lora_alpha: scaling factor of LoRA.
  • lora_dropout: dropout probability to use in LoRA layer.
  • P-TuningV2 parameters: + num_virtual_tokens: the number of virtual tokens.
  • num_attention_heads: 2: the number of attention heads of P-TuningV2 (do not change).
  • token_dim: 256: the token dimension of P-TuningV2 (do not change).

Start fine-tuning

Execute single machine multi-card/multi-machine multi-card run through the following code, which uses deepspeed as the acceleration solution, and you need to install deepspeed.

OMP_NUM_THREADS=1 torchrun --standalone --nnodes=1 --nproc_per_node=8  finetune.py  data/AdvertiseGen/  THUDM/glm-4-9b-chat  configs/lora.yaml # For Chat Fine-tune
OMP_NUM_THREADS=1 torchrun --standalone --nnodes=1 --nproc_per_node=8  finetune_vision.py  data/CogVLM-311K/  THUDM/glm-4v-9b  configs/lora.yaml  # For VQA Fine-tune

Execute single machine single card run through the following code.

python finetune.py  data/AdvertiseGen/  THUDM/glm-4-9b-chat  configs/lora.yaml # For Chat Fine-tune
python finetune_vision.py  data/CogVLM-311K/  THUDM/glm-4v-9b configs/lora.yaml # For VQA Fine-tune

Fine-tune from a saved point

If you train as described above, each fine-tuning will start from the beginning. If you want to fine-tune from a half-trained model, you can add a fourth parameter, which can be passed in two ways:

  1. yes, automatically start training from the last saved Checkpoint

  2. XX, breakpoint number, for example 600, start training from Checkpoint 600

For example, this is an example code to continue fine-tuning from the last saved point

python finetune.py data/AdvertiseGen/ THUDM/glm-4-9b-chat configs/lora.yaml yes

Use the fine-tuned model

Verify the fine-tuned model in inference.py

You can Use our fine-tuned model in finetune_demo/inference.py, and you can easily test it with just one line of code.

python inference.py your_finetune_path

In this way, the answer you get is the fine-tuned answer.

Use the fine-tuned model in other demos in this repository or external repositories

You can use our LORA and fully fine-tuned models in any demo. This requires you to modify the code yourself according to the following tutorial.

  1. Replace the way to read the model in the demo with the way to read the model in finetune_demo/inference.py.

Please note that for LORA and P-TuningV2, we did not merge the trained models, but recorded the fine-tuned path in adapter_config.json If the location of your original model changes, you should modify the path of base_model_name_or_path in adapter_config.json.

def load_model_and_tokenizer(
        model_dir: Union[str, Path], trust_remote_code: bool = True
) -> tuple[ModelType, TokenizerType]:
    model_dir = _resolve_path(model_dir)


if (model_dir / 'adapter_config.json').exists():
    model = AutoPeftModelForCausalLM.from_pretrained(
        model_dir, trust_remote_code=trust_remote_code, device_map='auto'
    )
tokenizer_dir = model.peft_config['default'].base_model_name_or_path
else:
model = AutoModelForCausalLM.from_pretrained(
    model_dir, trust_remote_code=trust_remote_code, device_map='auto'
)
tokenizer_dir = model_dir
tokenizer = AutoTokenizer.from_pretrained(
    tokenizer_dir, trust_remote_code=trust_remote_code
)
return model, tokenizer
  1. Read the fine-tuned model. Please note that you should use the location of the fine-tuned model. For example, if your model location is /path/to/finetune_adapter_model and the original model address is path/to/base_model, you should use /path/to/finetune_adapter_model as model_dir.
  2. After completing the above operations, you can use the fine-tuned model normally. Other calling methods remain unchanged.
  3. This fine-tuning script has not been tested on long texts of 128K or 1M tokens. Fine-tuning long texts requires GPU devices with larger memory and more efficient fine-tuning solutions, which developers need to handle on their own.

Reference


@inproceedings{liu2022p,
title={P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks},
author={Liu, Xiao and Ji, Kaixuan and Fu, Yicheng and Tam, Weng and Du, Zhengxiao and Yang, Zhilin and Tang, Jie},
booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short
Papers)},
pages={61--68},
year={2022}
}

@misc{tang2023toolalpaca,
title={ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases},
author={Qiaoyu Tang and Ziliang Deng and Hongyu Lin and Xianpei Han and Qiao Liang and Le Sun},
year={2023},
eprint={2306.05301},
archivePrefix={arXiv},
primaryClass={cs.CL}
}