This is a fork of QLoRA
Since I am the creator of the various airoboros models, this fork is fairly well catered to the airoboros instruction/response format, and uses the airoboros prompt.
The instructions.jsonl file (or whatever filename you are using), should be a single JSON string per line, newline separated, with "instruction" and "response" values.
Add: --dataset_format airoboros
Supports fine-tuning MPT base models via --mpt True, but requires a PEFT-compatible base model, e.g.: https://huggingface.co/jondurbin/mpt-30b-qlora-compatible
I prefer using a fixed number of epochs in training rather than trying to stop are a particular step count. I removed the --max_steps parameter in favor of --num_train_epochs (which I usually set to 3)
Try flash_qlora.py instead of qlora.py
MMLU benchmarks, eval, etc.
To fine-tune a llama base model, you should use one of these:
- https://huggingface.co/decapoda-research/llama-7b-hf
- https://huggingface.co/decapoda-research/llama-13b-hf
- https://huggingface.co/decapoda-research/llama-30b-hf
- https://huggingface.co/decapoda-research/llama-65b-hf
Then, you MUST replace the special_tokens_map.json file and tokenizer_config.json file with the ones found in this repo.