diff --git a/docs/source/conf.py b/docs/source/conf.py index 9137e53..2022b77 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -59,17 +59,16 @@ Wavefront shaping (WFS) is a technique for controlling the propagation of light. With applications ranging from microscopy to free-space telecommunication, this research field is expanding rapidly. - It stands out that many of the important breakthroughs are made by developing better software that - incorporates increasingly advanced physical models and algorithms. - Typical control software involves individual code for scanning microscopy, image processing, + As the field advances, it stands out that many breakthroughs are driven by the development of better + software that incorporates increasingly advanced physical models and algorithms. + Typical control software involves fragmented implementations for scanning microscopy, image processing, optimization algorithms, low-level hardware control, calibration and troubleshooting, and simulations for testing new algorithms. The complexity of the many different aspects of wavefront shaping software, however, - is becoming a bottleneck for further developments in the field, as well as for end-user adoption. - OpenWFS addresses these challenges by providing a Python module that coherently integrates - all aspects of wavefront shaping code. The module is designed to be modular and easy to expand. - It incorporates elements for hardware control, software simulation, and automated troubleshooting. + is becoming a limiting factor for further developments in the field, as well as for end-user adoption. + OpenWFS addresses these challenges by providing a modular and extensible Python library that + incorporates elements for hardware control, software simulation, and automated troubleshooting. Using these elements, the actual wavefront shaping algorithm and its automated tests can be written in just a few lines of code. } diff --git a/docs/source/core.rst b/docs/source/core.rst index 7b9f956..ede5a49 100644 --- a/docs/source/core.rst +++ b/docs/source/core.rst @@ -71,10 +71,9 @@ Processors ------------ A `Processor` is a `Detector` that takes input from one or more other detectors, and combines/processes this data. We already encountered an example in :numref:`Getting started`, where the `SingleRoiProcessor` was used to average the data from a camera over a region of interest. A block diagram of the data flow of this code is shown in :numref:`hellowfsdiagram`. Since a processor, itself, is a `Detector`, multiple processors can be chained together to combine their functionality. The OpenWFS further includes various processors, such as a `CropProcessor` to crop data to a rectangular region of interest, and a `TransformProcessor` to perform affine image transformations to image produced by a source. - Actuators --------- -Actuators are devices that *move* things in the setup. This can be literal, such as moving a translation stage, or a virtual movement, like an SLM that takes time to switch to a different phase pattern. All actuators and derive from the common :class:`.Actuator` base class. Actuators have no additional methods or properties other than those in the :class:`.Device` base class. +Actuators are devices that *move* things in the setup. This can be literal, such as moving a translation stage, or a virtual movement, like an SLM that takes time to switch to a different phase pattern. All actuators are derived from the common :class:`.Actuator` base class. Actuators have no additional methods or properties other than those in the :class:`.Device` base class. Units and metadata ---------------------------------- @@ -140,4 +139,74 @@ This synchronization is performed automatically. If desired, it is possible to e Finally, devices have a `timeout` attribute, which is the maximum time to wait for a device to become ready. This timeout is used in the state-switching mechanism, and when explicitly waiting for results using :meth:`~.Device.wait()` or :meth:`~.Device.read()`. - +Currently available devices +---------------------------- + +The following devices are currently implemented in OpenWFS: + +.. list-table:: + :header-rows: 1 + + * - Device Name + - Device Type + - Description + * - Camera + - Detector + - Adapter for GenICam/GenTL cameras + * - ScanningMicroscope + - Detector + - Laser scanning microscope using galvo mirrors and NI DAQ + * - StaticSource + - Detector + - Returns pre-set data, simulating a static source + * - NoiseSource + - Detector + - Generates uniform or Gaussian noise as a source + * - SingleRoi + - Processor (Detector) + - Averages signal over a single ROI + * - MultipleRoi + - Processor (Detector) + - Averages signals over multiple regions of interest (ROIs) + * - CropProcessor + - Processor (Detector) + - Crops data from the source to a region of interest + * - TransformProcessor + - Processor (Detector) + - Performs affine transformations on the source data + * - ADCProcessor + - Processor (Detector) + - Simulates an analog-digital converter + * - SimulatedWFS + - Processor + - Simulates wavefront shaping experiment using Fourier transform-based intensity computation at the focal plane + * - Gain + - Actuator + - Controls PMT gain voltage using NI data acquisition card + * - PhaseSLM + - Actuator + - Simulates a phase-only spatial light modulator + * - SLM + - Actuator + - Controls and renders patterns on a Spatial Light Modulator (SLM) using OpenGL + +Available Algorithms +--------------------- + +The following algorithms are available in OpenWFS for wavefront shaping: + +.. list-table:: + :header-rows: 1 + + * - Algorithm Name + - Description + * - FourierDualReference + - A Fourier dual reference algorithm that uses plane waves from a disk in k-space for wavefront shaping :cite:`Mastiani2022`. + * - IterativeDualReference + - A generic iterative dual reference algorithm with the ability to use custom basis functions for non-linear feedback applications. + * - DualReference + - A generic dual reference algorithm with the option for optimized reference, suitable for multi-target optimization and iterative feedback. + * - SimpleGenetic + - A simple genetic algorithm that optimizes wavefronts by selecting elite individuals and introducing mutations for focusing through scattering media :cite:`Piestun2012`. + * - StepwiseSequential + - A stepwise sequential algorithm which systematically modifies the phase pattern of each SLM element :cite:`Vellekoop2007`. \ No newline at end of file diff --git a/docs/source/development.rst b/docs/source/development.rst index 69115c3..078a639 100644 --- a/docs/source/development.rst +++ b/docs/source/development.rst @@ -48,6 +48,11 @@ Reporting bugs and contributing -------------------------------------------------- Bugs can be reported through the GitHub issue tracking system. Better than reporting bugs, we encourage users to *contribute bug fixes, new algorithms, device drivers, and other improvements*. These contributions can be made in the form of a pull request :cite:`zandonellaMassiddaOpenScience2022`, which will be reviewed by the development team and integrated into the package when appropriate. Please contact the current development team through GitHub :cite:`openwfsgithub` to coordinate such contributions. +Implementing new algorithms +-------------------------------------------------- +To implement a new algorithm, the currently existing algorithms can be consulted for a few examples. +Essentially, the algorithm needs to have an execute method, which needs to produce a WFSResult. With OpenWFS, all hardware interactions are abstracted away. Using `slm.set_phases` and `feedback.trigger` the algorithm can be naive to specific hardware. During the execution, different modes are measured, and a transmission matrix is calculated or approached. For most of our algorithms, the same algorithm can be used to analyze a phase stepping experiment. In order to show the versatility of this platform, we implemented the genetic algorithm described in :cite:`Piestun2012` and more recently adapted for a GUI in :cite:`Anderson2024`. + Implementing new devices -------------------------------------------------- @@ -70,7 +75,7 @@ If the detector is created with the flag ``multi_threaded = True``, then `_fetch Implementing a processor ++++++++++++++++++++++++++++++++++ -To implement a data processing step that dynamically processes date from one or more input detectors, implement a custom processor. This is done by deriving from the `Processor` base class and implementing the `__init__` function. This function should pass a list of all upstream nodes, i.e. all detectors which provide the input signals to the processor, the base class constructor. In addition, the :meth"`~Detector._fetch()` method should be implemented to process the data. The framework will wait until the data from all sources is available, and calls `_fetch()` with this data as input. See the implementation of :class:`~.Shutter` or any other processor for an example of how to implement this function. +To implement a data processing step that dynamically processes data from one or more input detectors, implement a custom processor. This is done by deriving from the `Processor` base class and implementing the `__init__` function. This function should pass a list of all upstream nodes, i.e. all detectors which provide the input signals to the processor, the base class constructor. In addition, the :meth"`~Detector._fetch()` method should be implemented to process the data. The framework will wait until the data from all sources is available, and calls `_fetch()` with this data as input. See the implementation of :class:`~.Shutter` or any other processor for an example of how to implement this function. Implementing an actuator +++++++++++++++++++++++++++++++ diff --git a/docs/source/index.rst b/docs/source/index.rst index 0907aa4..c6da15e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -10,5 +10,6 @@ OpenWFS - a library for conducting and simulating wavefront shaping experiments slms simulations development + pydevice api auto_examples/index diff --git a/docs/source/index_latex.rst b/docs/source/index_latex.rst index 198c2b6..5945f93 100644 --- a/docs/source/index_latex.rst +++ b/docs/source/index_latex.rst @@ -9,6 +9,7 @@ OpenWFS - a library for conducting and simulating wavefront shaping experiments core slms simulations + pydevice development conclusion diff --git a/docs/source/pydevice.rst b/docs/source/pydevice.rst new file mode 100644 index 0000000..7eca427 --- /dev/null +++ b/docs/source/pydevice.rst @@ -0,0 +1,6 @@ +.. _section-pydevice: + +OpenWFS in PyDevice +============================================== + +To smoothly enable end-user interaction with wavefront shaping algorithms, the Micro-Manager device adapter PyDevice was developed :cite:`PyDevice`. A more detailed description can be found in the mmCoreAndDevices source tree :cite:`mmCoreAndDevices`. In essence, PyDevice is Micro-Manager adapter that imports objects from a Python script and integrates them as devices, e.g. a camera or stage. OpenWFS was written in compliance with the templates required for PyDevice, which means OpenWFS cameras, scanners and algorithms can be loaded into Micro-Manager as devices. Examples of this are found in the example gallery :cite:`readthedocsOpenWFS`. Further developments due to this seamless connection can be a dedicated Micro-Manager based wavefront shaping GUI. \ No newline at end of file diff --git a/docs/source/readme.rst b/docs/source/readme.rst index 5b8431b..6284454 100644 --- a/docs/source/readme.rst +++ b/docs/source/readme.rst @@ -13,9 +13,9 @@ What is wavefront shaping? -------------------------------- -Wavefront shaping (WFS) is a technique for controlling the propagation of light in arbitrarily complex structures, including strongly scattering materials :cite:`kubby2019`. In WFS, a spatial light modulator (SLM) is used to shape the phase and/or amplitude of the incident light. With a properly constructed wavefront, light can be made to focus through :cite:`Vellekoop2007`, or inside :cite:`vellekoop2008demixing` scattering materials; or light can be shaped to have other desired properties, such as optimal sensitivity for specific measurements :cite:`bouchet2021maximum`, specialized point-spread functions :cite:`boniface2017transmission`, spectral filtering :cite:`Park2012`,, or for functions like optical trapping :cite:`vcivzmar2010situ`. +Wavefront shaping (WFS) is a technique for controlling the propagation of light in arbitrarily complex structures, including strongly scattering materials :cite:`kubby2019`. In WFS, a spatial light modulator (SLM) is used to shape the phase and/or amplitude of the incident light. With a properly constructed wavefront, light can be made to focus through :cite:`Vellekoop2007`, or inside :cite:`vellekoop2008demixing` scattering materials; or light can be shaped to have other desired properties, such as optimal sensitivity for specific measurements :cite:`bouchet2021maximum`, specialized point-spread functions :cite:`boniface2017transmission`, spectral filtering :cite:`Park2012`, or for functions like optical trapping :cite:`vcivzmar2010situ`. -It stands out that an important driving force in WFS is the development of new algorithms, for example to account for sample movement :cite:`valzania2023online`, experimental conditions :cite:`Anderson2016`, to be optimally resilient to noise :cite:`mastiani2021noise`, or to use digital twin models to compute the required correction patterns :cite:`salter2014exploring,ploschner2015seeing,Thendiyammal2020,cox2023model`. Much progress has been made towards developing fast and noise-resilient algorithms, or algorithms designed for specific towards the methodology of wavefront shaping, such as using algorithms based on Hadamard patterns, or Fourier-based approaches :cite:`Mastiani2022`. Fast techniques that enable wavefront shaping in dynamic samples :cite:`Liu2017,Tzang2019`, and many potential applications have been developed and prototyped, including endoscopy :cite:`ploschner2015seeing`, optical trapping :cite:`Cizmar2010`, Raman scattering, :cite:`Thompson2016`, and deep-tissue imaging :cite:`Streich2021`. Applications extend beyond that of microscope imaging such as optimizing photoelectrochemical absorption :cite:`Liew2016` and tuning random lasers :cite:`Bachelard2014`. +It stands out that an important driving force in WFS is the development of new algorithms, for example, to account for sample movement :cite:`valzania2023online`, experimental conditions :cite:`Anderson2016`, to be optimally resilient to noise :cite:`mastiani2021noise`, or to use digital twin models to compute the required correction patterns :cite:`salter2014exploring,ploschner2015seeing,Thendiyammal2020,cox2023model`. Much progress has been made towards developing fast and noise-resilient algorithms, or algorithms designed specifically for the methodology of wavefront shaping, such as using algorithms based on Hadamard patterns or Fourier-based approaches :cite:`Mastiani2022`. Fast techniques that enable wavefront shaping in dynamic samples :cite:`Liu2017,Tzang2019` have also been developed, and many potential applications have been prototyped, including endoscopy :cite:`ploschner2015seeing`, optical trapping :cite:`Cizmar2010`, Raman scattering :cite:`Thompson2016`, and deep-tissue imaging :cite:`Streich2021`. Applications extend beyond that of microscope imaging, such as in optimizing photoelectrochemical absorption :cite:`Liew2016` and tuning random lasers :cite:`Bachelard2014`. With the development of these advanced algorithms, however, the complexity of WFS software is steadily increasing as the field matures, which hinders cooperation as well as end-user adoption. Code for controlling wavefront shaping tends to be complex and setup-specific, and developing this code typically requires detailed technical knowledge and low-level programming. A recent c++ based contribution :cite:`Anderson2024`, highlights the growing need for software based tools that enable use and development. Moreover, since many labs use their own in-house programs to control the experiments, sharing and re-using code between different research groups is troublesome. @@ -50,18 +50,18 @@ OpenWFS is a Python package for performing and for simulating wavefront shaping Getting started ---------------------- -OpenWFS is available on the PyPI repository, and it can be installed with the command ``pip install openwfs``. The latest documentation and the example code can be found on the `Read the Docs `_ website :cite:`openwfsdocumentation`. To use OpenWFS, you need to have Python 3.9 or later installed. At the time of writing, OpenWFS is tested up to Python version 3.11 (not all dependencies were available for Python 3.12 yet). OpenWFS is developed and tested on Windows 11 and Manjaro Linux. +OpenWFS is available on the PyPI repository, and it can be installed with the command ``pip install openwfs``. The latest documentation and the example code can be found on the `Read the Docs `_ website :cite:`openwfsdocumentation`, and the entire repository can be found on :cite:`openwfsgithub`. To use OpenWFS, you need to have Python 3.9 or later installed. At the time of writing, OpenWFS is tested up to Python version 3.11 (not all dependencies were available for Python 3.12 yet). OpenWFS is developed and tested on Windows 11 and Manjaro Linux. Note that for certain hardware components, third party software needs to be installed. This is always mentioned in the documentation and docstrings of these functions. -:numref:`hello-wfs` shows an example of how to use OpenWFS to run a simple wavefront shaping experiment. This example illustrates several of the main concepts of OpenWFS. First, the code initializes objects to control a spatial light modulator (SLM) connected to a video port, and a camera that provides feedback to the wavefront shaping algorithm. +:numref:`hello-wfs` shows an example of how to use OpenWFS to run a simple wavefront shaping experiment. This example illustrates several of the main concepts of OpenWFS. First, the code initializes objects to control a spatial light modulator (SLM) connected to a video port, and a camera that provides feedback to the wavefront shaping algorithm. .. _hello-wfs: .. literalinclude:: ../../examples/hello_wfs.py :language: python :caption: ``hello_wfs.py``. Example of a simple wavefront shaping experiment using OpenWFS. -This example uses the `StepwiseSequential` wavefront shaping algorithm :cite:`vellekoop2008phase`. The algorithm needs access to the SLM for controlling the wavefront. This feedback is obtained from a :class:`~.SingleRoi` object, which takes images from the camera, and averages them over the specified circular region of interest. The algorithm returns the measured transmission matrix in the field `results.t`, which is used to compute the optimal phase pattern to compensate the aberrations. Finally, the code measures the intensity at the detector before and after applying the optimized phase pattern. +This example uses the `~.StepwiseSequential` wavefront shaping algorithm :cite:`vellekoop2008phase`. The algorithm needs access to the SLM for controlling the wavefront. This feedback is obtained from a :class:`~.SingleRoi` object, which takes images from the camera, and averages them over the specified circular region of interest. The algorithm returns the measured transmission matrix in the field `results.t`, which is used to compute the optimal phase pattern to compensate the aberrations. Finally, the code measures the intensity at the detector before and after applying the optimized phase pattern. -This code illustrates how OpenWFS separates the concerns of the hardware control (`SLM` and `Camera`), signal processing (`SingleROIProcessor`) and the algorithm itself (`StepwiseSequential`). A large variety of wavefront shaping experiments can be performed by using different types of feedback signals (such as optimizing multiple foci simultaneously using a :class:`~.MultiRoiProcessor` object), using different algorithms, or different image sources, such as a :class:`~.ScanningMicroscope`. Notably, these objects can be replaced by *mock* objects, that simulate the hardware and allow for rapid prototyping and testing of new algorithms without direct access to wavefront shaping hardware (see :numref:`section-simulations`). +This code illustrates how OpenWFS separates the concerns of the hardware control (:class:`~.SLM` and :class:`~.Camera`), signal processing (:class:`~.SingleRoi(Processor)`) and the algorithm itself (:class:`~.StepwiseSequential`). A large variety of wavefront shaping experiments can be performed by using different types of feedback signals (such as optimizing multiple foci simultaneously using a :class:`~.MultiRoi(Processor)` object), using different algorithms, or different image sources, such as a :class:`~.ScanningMicroscope`. Notably, these objects can be replaced by *mock* objects, that simulate the hardware and allow for rapid prototyping and testing of new algorithms without direct access to wavefront shaping hardware (see :numref:`section-simulations`). Analysis and troubleshooting diff --git a/docs/source/references.bib b/docs/source/references.bib index cb75b43..404a9f5 100644 --- a/docs/source/references.bib +++ b/docs/source/references.bib @@ -6,6 +6,26 @@ @book{goodman2015statistical } + +@article{Piestun2012, + abstract = {We introduce genetic algorithms (GA) for wavefront control to focus light through highly scattering media. We theoretically and experimentally compare GAs to existing phase control algorithms and show that GAs are particularly advantageous in low signal-to-noise environments.}, + author = {Rafael Piestun and Albert N. Brown and Antonio M. Caravaca-Aguirre and Donald B. Conkey}, + doi = {10.1364/OE.20.004840}, + issn = {1094-4087}, + issue = {5}, + journal = {Optics Express, Vol. 20, Issue 5, pp. 4840-4849}, + keywords = {Optical trapping,Phase conjugation,Phase shift,Scattering media,Spatial light modulators,Turbid media}, + month = {2}, + pages = {4840-4849}, + pmid = {22418290}, + publisher = {Optica Publishing Group}, + title = {Genetic algorithm optimization for focusing through turbid media in noisy environments}, + volume = {20}, + url = {https://opg.optica.org/viewmedia.cfm?uri=oe-20-5-4840&seq=0&html=true https://opg.optica.org/abstract.cfm?uri=oe-20-5-4840 https://opg.optica.org/oe/abstract.cfm?uri=oe-20-5-4840}, + year = {2012}, +} + + @misc{Mastiani2024PracticalConsiderations, title = {Practical Considerations for High-Fidelity Wavefront Shaping Experiments}, author = {Mastiani, Bahareh and Cox, Dani{\"e}l W. S. and Ivo M. Vellekoop}, @@ -435,6 +455,11 @@ @article{Anderson2016 } +@misc{mmCoreAndDevices, + author = {Mark Tsuchida and Sam Griffin}, + title = {Micro-Manager mmCoreAndDevices repository}, + url = {https://github.com/micro-manager/mmCoreAndDevices}, +} @misc{MMoverview, author = {Mark Tsuchida and Sam Griffin}, title = {Micro-Manager Project Overview},