Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix data_parallel_shard_degree description #659

Merged
merged 1 commit into from
Oct 29, 2024
Merged

Fix data_parallel_shard_degree description #659

merged 1 commit into from
Oct 29, 2024

Conversation

carmocca
Copy link
Contributor

@carmocca carmocca commented Oct 29, 2024 •
edited
Loading

torchtitan/torchtitan/parallelisms/parallel_dims.py

Lines 35 to 36 in 7310abe

for d in (dp_replicate, cp, tp, pp):
assert d >= 1, "Parallelism degree should be >= 1, except for dp_shard"
does not allow data_parallel_replicate_degree to be -1

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 29, 2024
@fegin fegin merged commit dbb0520 into pytorch:main Oct 29, 2024
5 checks passed
mori360 pushed a commit to mori360/torchtitan that referenced this pull request Nov 26, 2024
@carmocca @mori360
Fix data_parallel_shard_degree description (pytorch#659)
85ff524 
https://github.com/pytorch/torchtitan/blob/7310abea8782bbe459b662bc6d8411fe8d55f62c/torchtitan/parallelisms/parallel_dims.py#L35-L36
does not allow `data_parallel_replicate_degree` to be -1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@awgu awgu awgu approved these changes

@fegin fegin fegin approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@carmocca @fegin @awgu @facebook-github-bot