Feature: Checkpoint saving improvements

[hatanp]

Training at scale requires checkpointing often. To make this feasible the saving needs to be as fast as possible. Various approaches exist and should be evaluated:

• Distributed checkpointing
• Asynchronous checkpointing

PyTorch has a recent blog discussing these:
https://pytorch.org/blog/reducing-checkpointing-times/