Skip to content

Commit

Permalink
start optax dox
Browse files Browse the repository at this point in the history
  • Loading branch information
ClashLuke committed Dec 7, 2024
1 parent 0291e6c commit 459d7d6
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ recommended experimental optimizer is `DelayedPSGDKron` ([tuning guide](docs/psg

## Features

* **Optax-like API**: `C = heavyball.chainable; grokfast = C.ChainOpt(p, lr, C.exp_avg, C.scale_by_adam)`
* **Stochastic Rounding**: [FP32 convergence with BF16 parameters](https://github.com/pytorch/pytorch/issues/120376)
* **Inplace EMA**: Same math, but less memory, less compute and higher stability
* **Foreach**: Fast multi-tensor application (turn it off to save memory via `foreach=False`)
Expand Down

0 comments on commit 459d7d6

Please sign in to comment.