Checkpoint selection

[RJ-246]

Added a function that allows you to choose a session to resume training from instead of hard coding it. It chooses the steps zip with the largest number inside the session folder.

[PWhiddy]

Hey, thanks for opening a PR!

So this gives a menu of sessions to start from? Sounds nice. I think it could be factored into it's own file so it doesn't have to be duplicated across files. And it also would could be applied to the run_pretrained script.

This prompts you every time no matter what though? It would be great if by default you weren't required to do this every time, maybe it could be enabled with a flag or cli argument?

[RJ-246]

Good ideas, I'll keep working on it. What would you see as being the default session used if the flag wasn't used?

[PWhiddy]

I think there should still be a hardcoded file_name variable, and if it can't be found or is set to something special like "automatic" or "prompt" it will give you the menu. I didn't get to inspect the code too closely yet but one of the menu options should be starting from scratch of course.

Also make sure to pull this change I just merged in:
#105