Skip to content

dengliangjun/gpt-neo-fine-tuning-example

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

30 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GPT-Neo-2.7B & GPT-J-6B Fine-Tuning Examples Using HuggingFace & DeepSpeed

medium Python3.8.6 PyTorch1.8.1

Installation

cd venv/bin
./pip install -r ../../requirements.txt 
./pip install deepspeed==0.5.9

GPT-Neo

Example with GPT-Neo-1.3B without DeepSpeed
Training and testing log with GPT-Neo-1.3B
Example with GPT-Neo-2.7B with DeepSpeed
DeepSpeed configuration with GPT-Neo-2.7B
Training and testing log with GPT-Neo-2.7B

GPU VRAM load during GPT-Neo-2.7B training

RAM load during GPT-Neo-2.7B training

Results

GPT-J-6B

Example with GPT-J-6B with DeepSpeed
DeepSpeed configuration with GPT-J-6B
Training and testing log with GPT-J-6B

GPU VRAM load during GPT-J-6B training

RAM load during GPT-J-6B training

About

Fine-Tune EleutherAI GPT-Neo And GPT-J-6B To Generate Netflix Movie Descriptions Using Hugginface And DeepSpeed

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%