Skip to content

Commit

Permalink
Monitor was always enabled causing performance degradation (#5633)
Browse files Browse the repository at this point in the history
The Boolean expression for the monitor to be enabled was incorrect, as
instead of using the `enabled` field, it used the comet configuration
object, making the expression always True.

This caused performance degradation (we've observed ~10% drop) as it
erroneously invoked the events logging flow along with the expensive
calculation of `loss.mean().item()`.

Co-authored-by: Olatunji Ruwase <[email protected]>
Co-authored-by: Logan Adams <[email protected]>
  • Loading branch information
3 people authored Jun 12, 2024
1 parent 91a9986 commit 54f98fd
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion deepspeed/monitor/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -140,5 +140,5 @@ class DeepSpeedMonitorConfig(DeepSpeedConfigModel):
@root_validator
def check_enabled(cls, values):
values["enabled"] = values.get("tensorboard").enabled or values.get("wandb").enabled or values.get(
"csv_monitor").enabled or values.get("comet")
"csv_monitor").enabled or values.get("comet").enabled
return values

0 comments on commit 54f98fd

Please sign in to comment.