paper_baselines

Baseline Submissions from the "Benchmarking Neural Network Training Algorithms" Paper

This directory contains the baseline submissions for the external tuning ruleset as presented in our paper Benchmarking Neural Network Training Algorithms. They are based on eight different update rules:

• Adafactor
• AdamW
• LAMB
• SGD with Momentum
• NadamW
• SGD with Nesterov Momentum
• SAM
• Shampoo

Each update rule has two different tuning search spaces, one where the first momentum parameter (often denoted $\beta_1$) is tuned and one where it is set to a fixed value.