Expand scope of camera simulation

[viljarjf]

Currently, the camera simulation is simply an API simulation, with the image being random noise.
This is limiting, as e.g. the experiments cannot be tested properly with this; they require spatially defined crytstals capable of diffraction.

I started working on this in https://github.com/viljarjf/instamatic/tree/camera-simulation.
There, I added a stage and a random crystal structure, randomly dispersed and oriented on a grid.
I plan to support stage rotation, tilting, correct handling of beam shift/tilt ect. later. However, I am not entirely sure how all the microscope parameters affect the images (real and reciprocal), so to begin with, I plan to keep it simple and allow expansions at a later stage.
Currently, the simulation supports a single crystal phase, random dispersion, any number of circular crystals of any size, and a copper grid. Tilting is mostly implemented, but not tested, so it will display a warning for now.

Expanding this to support more phases, tilting, magnification-induced image rotation and so on should not be too difficult programming-wise, however I'm not too confident that I'll do this correctly without input. Therefore, I'm opening this issue to hopefully have a discussion about what and how things should be simulated.

An improved camera simulation should dramatically expand the features of instamatic that can be properly tested. We can test the experiments, the GUI, calibration (if we implement an accurate simulation). Additionally, it should make further developments much easier, as testing algorithms would not strictly require access to a microscope (at least for initial development).

As for downsides, expanding the scope can easily get carried away. It will necessarily add more dependencies (as is already the case with my branch), which will only get worse with increased accuracy (hopefully! Complex imaging/diffraction simulations should be handled by external libraries). Handling inaccurate alignment, astigmatism ect is a good idea, but the added work of implementing this might not be worth it compared to how basic it is to compensate for by the operator.

[viljarjf]

Timeline-wise, the simple initial implementation is pretty much ready to go. I would like to add some more tests, and try it out some more, but I don't think expanding the initial scope is necessary. As stated, it should be fairly simple to expand, but the scope can be kept simple to start with.

[stefsmeets]

This is great! Being able to set up a simulation to play with instamatic without installing it on the microscope would be very useful.

I can see the dependencies getting out of scope easily. If you are worried about dependency creep, you could place the simulation dependencies as an 'extra' in pyproject.toml, so you would do pip install instamitic[simulation] to install the simulation dependencies (or instamatic[digital-twin] if you want to get on board of that hype train).

I think it makes sense to develop this in a separate module (or possibly package outside of instamatic), having this as separate as possible from the rest of the code. This needs a new interface under microscope and camera to interface with this module.

If you are going this direction, I think the code should be under instamatic/simulation rather than instamatic/camera. Somehow this module needs to maintain the state of both the microscope and camera, right? In the camera module you could have a very shallow wrapper for interfacing with the simulation code. I also foresee that this will also make the code easier to extend and re-use for others.

Configuration would also be something to think about. As a user you will want to be able to control all the parameters of the microscope and store this in some config. So some questions I would have as a user are: how do you initialize a simulated TEM? Can you set up the number of lenses? Can I import the simulation into a notebook? Can I define crystal positions manually? All of this would also be useful for testing.

Just thinking what this could look like:

from instamatic.simulation import DigiTEM

sim = DigiTEM(...)
# alternative
# sim = DigiTEM.from_config('path/to/config.yaml')
sim.add_crystal(x=123, y=321, shape='cube', thickness=10)
sim.add_crystal(...)
...

from instamatic.controller import TEMController

ctrl = TEMController(tem=sim.tem, cam=sim.cam)
ctrl.stage.set_xy(x=123, y=321)
ctrl.get_image()

[viljarjf]

I made a PR with an initial implementation.
but depending on the scope we could migrate this into a seperate repo instead.

how do you initialize a simulated TEM?

Right now, I establish a connection between the camera and the tem members of TEMController after initialization. Very hacky, your proposal looks much cleaner.

Can you set up the number of lenses?

The current simulation does not account for a lot of the instrument geometry apart from the stage. We could expand the scope of simulations to full instruments to account for this, which would be really cool.

Can I import the simulation into a notebook?

You should be able to, notebooks use the same ipython backend I think? It works nicely using e.g. instamatic.controller

Can I define crystal positions manually?

Not yet, but this should be very easy to add. I keep a list of random positions in the Stage-object now.