Skip to content

Commit

Permalink
[Docs] fix typos in sp docs (#821)
Browse files Browse the repository at this point in the history
fix typo
  • Loading branch information
HIT-cwh authored Jul 10, 2024
1 parent 8adc8d4 commit 48df4c8
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/zh_cn/acceleration/train_extreme_long_sequence.rst
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@
- yi-34B
- ZeRO-3
- 16
- OOM
- 227


为解决长序列训练过程中的显存问题,Megatron-LM 团队和 DeepSpeed 团队分别提出了两种序列并行算法,通过对长序列进行切分的方法来降低单 GPU 上计算的序列长度。XTuner 中的序列并行设计思路参考了 DeepSpeed 的工作 `DeepSpeed Ulysses <https://arxiv.org/abs/2309.14509>`_,并加以优化, **以实现一键开启序列并行策略** 。三者的对比如下:
Expand Down

0 comments on commit 48df4c8

Please sign in to comment.