ZeRO0 does not handle BF16 gradients properly#5154

Closed

tohtana wants to merge 2 commits intotohtana/fix_fp32_clippingfrom tohtana/fix_bf16_opt_update_hp

+14-14

Commits on Feb 19, 2024

udpate fp32 master params of bf16 optimizer after allreduce
tohtana
committed

Commits on Feb 20, 2024

fix gradient accumulation
tohtana
committed