Don't check overflow for bf16 data type #4512
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BacharL
requested review from
jeffra,
tjruwase,
samyam and
mrwyattii
as code owners
|
@hablb, thanks for the PR. While I understand restricting overflow checks to fp16 only, I don't see the benefit of providing a config option to enable overflow checks for bf16/fp32. Can you please explain the need for the config option? I think the restriction part is fine. |
BacharL
force-pushed
the
no_overflow_check
branch
from
43af495 to
75269d6
Compare
|
Thanks @tjruwase |
tjruwase
approved these changes
Oct 19, 2023
baodii
pushed a commit
to baodii/DeepSpeed
that referenced
this pull request
Nov 7, 2023
Always check for fp16 bf16 dynamic range is similar to fp32. don't check overflow by default. Co-authored-by: Olatunji Ruwase <[email protected]>
|
Dear @hablb ,Besides wasting computational resources, are there any other downsides to performing overflow checks on bf16 data types? Would performing overflow checks on bf16 data types cause any other bugs? |
|
No other downsides, however this check significant overhead as it reduces all gradients, communicates and synchronizes with the cpu. |
jon-tow
added a commit
to Stability-AI/DeepSpeed
that referenced
this pull request
Nov 28, 2023
mauryaavinash95
pushed a commit
to mauryaavinash95/DeepSpeed
that referenced
this pull request
Feb 17, 2024
Always check for fp16 bf16 dynamic range is similar to fp32. don't check overflow by default. Co-authored-by: Olatunji Ruwase <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Always check for fp16
bf16 dynamic range is similar to fp32. don't check overflow by default.