-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parity inference task #294
Parity inference task #294
Conversation
Signed-off-by: Kin Long Kelvin Lee <[email protected]>
Signed-off-by: Kin Long Kelvin Lee <[email protected]>
Signed-off-by: Kin Long Kelvin Lee <[email protected]>
Signed-off-by: Kin Long Kelvin Lee <[email protected]>
Signed-off-by: Kin Long Kelvin Lee <[email protected]>
Signed-off-by: Kin Long Kelvin Lee <[email protected]>
Signed-off-by: Kin Long Kelvin Lee <[email protected]>
Signed-off-by: Kin Long Kelvin Lee <[email protected]>
Signed-off-by: Kin Long Kelvin Lee <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good and useful! A lot of failing tests, not sure what is relevant. Double check if these are applicable, then good to merge
I'm pretty certain the tests that fail are the usual suspects. I did pick up something that broke everything (fixed in 2faa68d), but otherwise seems to be fine. I'm going to merge! |
This PR implements a
ParityInferenceTask
(with the lack of a better naming for it), which is used to run a datasplit through a model to evaluate predicted vs. actual, beyond just looking at reduced metrics like MSE/MAE. This task should be pretty straightforward to set up and run, and does not need additional logger configuration. After runningtrainer.predict
, aninference_data.json
is produced in the experiment folder (trainer.log_dir/<name>/inference_data.json
) that can then be reviewed offline.ParityData
structure which is mostly just a helper class for accumulating data.ParityInferenceTask
, which in itself just provides thepredict_step
andon_predict_epoch_end
functions to be called in the PyTorch Lightningpredict
pipeline.