ifromeast / LLMTrainer Public

Notifications You must be signed in to change notification settings
Fork 3
Star 19

A comparison of pretraining framework for LLM

19 stars 3 forks Branches Tags Activity

Name		Name	Last commit message	Last commit date
Latest commit History 12 Commits
01_base_hf		01_base_hf
02_fsdp		02_fsdp
03_deepspeed		03_deepspeed
04_flash		04_flash
05_colossalai		05_colossalai
06_sophia		06_sophia
config		config
utils		utils
README.md		README.md
requirements.txt		requirements.txt

Repository files navigation

LLMTrainer

对比 LLM 的多种预训练方案的性能，方案包括：

Torch DDP
FSDP
DeepSpeed
FlashAttention V2
ColossalAI
Sophia Optimizer
Lightning Fabric

详细介绍见 https://zhuanlan.zhihu.com/p/648094197?