Using old model for validation

[leon-maechler]

Hi,

I have trained a model that I would like to validate but I trained it before the notebook was adapted to include the validation step. So now there is no best_model.pkl in the experiment folder. This is all that is in the folder:

01_tensor_db.pkl 03_tensor_db.pkl 05_tensor_db.pkl 07_tensor_db.pkl 09_tensor_db.pkl 11_tensor_db.pkl 13_tensor_db.pkl 15_tensor_db.pkl 17_tensor_db.pkl state.pkl 02_tensor_db.pkl 04_tensor_db.pkl 06_tensor_db.pkl 08_tensor_db.pkl 10_tensor_db.pkl 12_tensor_db.pkl 14_tensor_db.pkl 16_tensor_db.pkl aggregator_tensor_db.pkl

Is there a way to get it to validate anyhow? If I have to retrain it might get quite close with the deadline...

Best,
Leon

[sarthakpati]

Oh, that's an interesting question. I am not sure if the older version of the notebook could be easily adapted to do validation directly. But perhaps you could load the model in the new notebook and run the inference?

Tagging @psfoley and @alexey-gruzdev for clarity.

[leon-maechler]

Ok so I tried to continue training with the new notebook using as a starting checkpoint the old model but I got this error:

Traceback (most recent call last): File "save.py", line 679, in <module> restore_from_checkpoint_folder = restore_from_checkpoint_folder) File "XXXXX/fets/Challenge/Task_1/fets_challenge/experiment.py", line 493, in run_challenge_experiment raise ValueError(f'Expected temporary model at: checkpoint/{checkpoint_folder}/temp_model.pkl to exist but it was not found.') ValueError: Expected temporary model at: checkpoint/experiment_1/temp_model.pkl to exist but it was not found.
Since I guess this 'temp_model.pkl' was not saved back then.

Training from scratch again is sadly taking too long on my machine to get done before the 26.7 23.59 EST....

Is there any way to get the 'best_model.pkl' out of the "old" .pkl files so that I can run the validation?
(Or can I submit it later?)

In general it looks to me as if this is the only weay to submit and we have some promising results...

[sarthakpati]

Hi,

You don't need to submit the trained model to us, only the notebook containing your algorithm (we will be doing the re-training on our end for validation). If you want, you can continue to optimize the parameters until the 2nd, at which point you will need to submit both the parameters (no algorithmic change shall be permitted) and the paper (the latter on CMT).

[psfoley]

Hi @leon-maechler, I believe you should just be able to create an empty file at checkpoint/experiment_1/temp_model.pkl, start the experiment using your checkpoint files, and then the temp_model.pkl should be overwritten with the model in the native format