TensorFlow and Pytorch open source implementation for the aProx optimization methods from the paper:
The importance of better models in stochastic optimization
by Hilal Asi and John Duchi.
This repository provides implementation for the aProx optimizatin algorithms (Truncated and Truncated-Adagrad), which improve the robustness of classical optimization algorithms (e.g. SGD and Adagrad) to the stepsize value. The folders Optimizers_tf and Optimizers_pytorch include the implementation for TensorFlow and Pytorch, respectively. Examples of using these optimizers can be found in the files example_tf.py and example_pytorch.py.
The following plots (from the paper) show the time-to-convergence as a function of the stepsize for various methods for CIFAR10 and Stanfrod-dogs datasets. You can reproduce these plots by running the file paper-code/stability_check.py with the desired dataset.
Code author: Hilal Asi
Pull requests and issues: @HilalAsi
If you use this code, please cite our paper:
@article {AsiDu19,
author = {Asi, Hilal and Duchi, John C.},
title = {The importance of better models in stochastic optimization},
year = {2019},
issn = {0027-8424},
eprint = {https://www.pnas.org/content/early/2019/10/29/1908018116.full.pdf},
journal = {Proceedings of the National Academy of Sciences}
}