Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for stage3 when setting different communication data type #4540

Merged
merged 12 commits into from
Dec 11, 2023
Merged

Fix for stage3 when setting different communication data type #4540

Changes from 9 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
5af1ecb
Fix for stage3 when setting different communication data type
BacharL Oct 19, 2023
805990c
Update deepspeed/runtime/zero/stage3.py
BacharL Oct 22, 2023
b784f13
Update deepspeed/runtime/zero/stage3.py
BacharL Oct 26, 2023
4686034
Update stage3.py
BacharL Oct 26, 2023
cf7afe2
Merge branch 'master' into comm_dtype
tjruwase Oct 30, 2023
eb64a1a
Merge branch 'master' into comm_dtype
tjruwase Oct 31, 2023
cbdb598
Merge branch 'master' into comm_dtype
BacharL Nov 5, 2023
335b439
Merge branch 'master' into comm_dtype
tjruwase Nov 7, 2023
98dc350
Merge branch 'master' into comm_dtype
BacharL Nov 16, 2023
25470ab
Update deepspeed/runtime/zero/stage3.py
BacharL Dec 4, 2023
7dc149a
Merge branch 'master' into comm_dtype
tjruwase Dec 4, 2023
29ad579
Merge branch 'master' into comm_dtype
tjruwase Dec 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions deepspeed/runtime/zero/stage3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1214,8 +1214,7 @@ def __reduce_and_partition_ipg_grads(self, safe_mode: bool = False) -> None:

@instrument_w_nvtx
def __avg_scatter_contiguous_grads(self, buffer_to_reduce: Tensor) -> List[Tensor]:
dtype = buffer_to_reduce.dtype
if self.communication_data_type == self.dtype:
if self.communication_data_type != self.dtype:
BacharL marked this conversation as resolved.
Show resolved Hide resolved
buffer_to_reduce = buffer_to_reduce.to(self.communication_data_type)
if self.postscale_gradients and self.gradient_predivide_factor != 1.0:
buffer_to_reduce = buffer_to_reduce.div_(self.gradient_predivide_factor)
Expand Down