[epic] Implement neural processes emulator

[mastoffel]

see

• https://github.com/wesselb/neuralprocesses
• https://yanndubs.github.io/Neural-Process-Family/text/Intro.html
• https://kasparmartens.rbind.io/post/np/

Now that autoemulate uses the sklearn estimator template, it would be good to use skorch to implement Pytorch models, so that cross-validation and hyperparameter search are compatible.

• Evaluate NeuralProcesses package #137

[kallewesterling]

Leaving a note here that the neuralprocesses package has overlaps with the Turing/BAS collaboration deepsensor, which I have been helping out with. I'd be more than happy to explore interests across these packages, and/or any engagement with the author of 'neuralprocesses' that we can imagine. Dropping comment here to keep it on our radar.

[kallewesterling]

@mastoffel I have noted from last sprint planning meeting that we were thinking about setting up a knowledge exchange with the author of neuralprocesses about the package and how it works. It might be of interest to more people than just our group. Would that be something you think would be beneficial?

[kallewesterling]

Just to note, also, that this might be something we want to do after AI UK so flagging it as not super priority...!

[edaub]

I did some more thorough research on this, and my conclusion is that the utility of neural processes for emulation is still a research question that requires more of a long term investigation. In particular, neural processes don't assume any particular properties of the underlying function to be approximated (other than that they can be approximated by a neural network), but rather attempts to learn them from contextual data before being fitted to a particular model. Gaussian Processes, on the other hand, specify mean and covariance functions to constrain the function space from which the model will be fit. Most other regression models that aren't neural networks have some other kind of constraint like this. Thus to make something that is adaptable to black box emulation, we need to specify example functions or enough simulation-specific data to allow those functions to be learned.

Thus, I don't think it's feasible to implement this in the library in the short term, and my suspicion is that until more is known about how to structure neural process emulation, a neural network is probably going to have similar emulation performance if you want to be purely data driven in how you fit your model. However, I am of the opinion that meta-learning is a potentially rich area to explore for the emulation problem that could overcome some of the existing challenges in black box emulation, and neural processes are the technical methodology with which this would be done.

[mastoffel]

Thanks @edaub, that's super helpful. So it seems like we would need a larger dataset initially, which we usually won't have I think. But I agree that we can just leave neural processes / meta learning as a research issue for the future of the package.

[scotthosking]

tagging @MartinSJRogers for awareness

[mastoffel]

conditional neural processes now implemented in #217