[FEATURE] Add Stopping Criteria to Grid Search

[meganjkurka]

🚀 Feature

Add the ability to stop Grid Search if the validation metric exceeds a certain value or if the models are not getting better (i.e. stop if BLEU hasn’t improved by 1 over the 5 best models).

Motivation

Automatic stopping will ensure that the grid search is stopped once the models are "good enough" and there is no unnecessary waste of resources.

[tmostak]

I'd like to upvote this... grid search is great but a lot of time/GPU usage is spent on runs where it is clear within the first 10-30% of the training run that the validation metric is not going to be competitive.

For my use case at least, it would be ideal to specify that the specific run should be terminated if the validation metric exceeds X at Y% into the training run.

[pascal-pfeiffer]

@meganjkurka is that the same that you are looking for? Early stoppinglkilling of an experiment if a threshold validation metric is not met.

[pascal-pfeiffer]

stop if BLEU hasn’t improved by 1 over the 5 best models

@meganjkurka , trying to understand what it expected from the feature: Could you please clarifiy how exactly this is expected to work. Grid search experiments are started in a random order. So, if by chance the first 5 models are all same bad (say all BLEU <1) it would stop? Even if later experiments would be great?