Matbench task model accuracy for xtal2png representation

[sgbaird]

The task is to use a CNN model for a Matbench submission on regressing formation energy using the xtal2png representation (as an image and/or as an array would be fine). This will help with knowing how "good" the xtal2png representation is from a model accuracy perspective, though I don't expect this to set new benchmarks necessarily.

This might look like using skorch with some type of pytorch CNN module (e.g. ResNetUNet, Net) and an MSE loss function. This skorch tutorial looks like it might help with loading images, though this SO answer is probably better for making the actual dataset to pass to skorch.

If regression is too much of a pain (CNNs aren't used as often for property regression in the image-processing domain), an easy fallback is to do the mp_is_metal binary classification task instead of the e_form regression task.

Related:

• Ordinal regression: https://github.com/Raschka-research-group/coral-pytorch
• Ordinal regression tutorial: https://github.com/Raschka-research-group/coral-pytorch/blob/main/docs/tutorials/pytorch_lightning/ordinal-corn_mnist.ipynb
• Predict ages based on images: https://github.com/christianbv/AgeNet#49

Maybe Faris interested in working on this given that he'll be doing some image processing

[sgbaird]

Also

• https://stackoverflow.com/questions/59110844/convolution-neural-network-for-regression-using-pytorch
• https://discuss.pytorch.org/t/pytorch-cnn-for-regression/33436

[sgbaird]

The following tutorial uses grayscale MNIST dataset for classification and might be one of the easiest to adapt to mp_is_metal (or to mp_e_form via adjusting the final layer and the loss function)

• https://medium.datadriveninvestor.com/train-a-cnn-using-skorch-for-mnist-digit-recognition-53d7d2f971c7

[sgbaird]

@faris-k did a classification task on mp_is_metal and is getting the files ready for a Matbench PR. See the notebook.

[sgbaird]

It does leave the question on my mind, why does the regression results are so poor (much worse than dummy), whereas the classification results are OK (a bit better than dummy).

A follow-up computational experiment (that I think we should leave on the back-burner until further notice) is using the classification model, but with bins for the classes (e.g. formation energy between 0 and 0.05). Implementing ordinal classification would be extra work, so first treat it as categorical. I'm putting this here more as a future reference sort of thing as things progress with xtal2png.

It's also interesting in the sense that hyperparameter-tuned XGBoost did a pretty good job on the regression task (~4x better than the CNN regression), and this was with much less information. We'll see if the results still hold when we double-check that data leakage wasn't coming into play. #51 and specifically #51 (comment)