Merge remote-tracking branch 'origin/main' into ticket42_sum_detect_m…

…onitor_spectra

.github/workflows/Lint-and-test.yml

@@ -19,7 +19,7 @@ jobs:
       - name: install requirements
         run: pip install -e .[dev]
       - name: run pytest
-        run: python -m pytest
+        run: IBEX_BLUESKY_CORE_LOGS=/tmp/ibex_bluesky_core_logs/ python -m pytest
   results:
     if: ${{ always() }}
     runs-on: ubuntu-latest

.github/workflows/test-against-main.yml

@@ -0,0 +1,38 @@
+name: test-against-main
+on:
+  schedule:
+    - cron: "0 0 * * *"
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        version: ['3.11', "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.version }}
+      - name: install requirements
+        run: pip install -e .[dev]
+      - name: install latest bluesky and ophyd-async
+        run: pip install --upgrade --force-reinstall git+https://github.com/bluesky/bluesky.git@main git+https://github.com/bluesky/ophyd-async.git@main
+      - name: run ruff
+        run: python -m ruff check
+      - name: run pyright
+        run: python -m pyright
+      - name: run pytest
+        run: python -m pytest
+  results:
+    if: ${{ always() }}
+    runs-on: ubuntu-latest
+    name: Final Results
+    needs: [tests]
+    steps:
+    - run: exit 1
+      # see https://stackoverflow.com/a/67532120/4907315
+      if: >-
+          ${{
+               contains(needs.*.result, 'failure')
+            || contains(needs.*.result, 'cancelled')
+          }}

doc/architectural_decisions/005-variance-addition.md

@@ -0,0 +1,58 @@
+# Variance addition to counts data
+
+## Status
+
+Current
+
+## Context
+
+For counts data, the uncertainty on counts is typically defined by poisson counting statistics, i.e. the standard deviation on `N` counts is `sqrt(N)`.
+
+This can be problematic in cases where zero counts have been collected, as the standard deviation will then be zero, which will subsequently lead to "infinite" point weightings in downstream fitting routines for example.
+
+A number of possible approaches were considered:
+
+| Option | Description |
+| --- | --- |
+| A | Reject data with zero counts, i.e. explicitly throw an exception if any data with zero counts is seen as part of a scan. |
+| B | Use a standard deviation of `NaN` for points with zero counts. |
+| C | Define the standard deviation of `N` counts as `1` if counts are zero, otherwise `sqrt(N)`. This is one of the approaches available in mantid for example. |
+| D | Define the standard deviation of `N` counts as `sqrt(N+0.5)` unconditionally - on the basis that "half a count" is smaller than the smallest possible actual measurement which can be taken. |
+| E | No special handling, calculate std. dev. as `sqrt(N)`. |
+
+For clarity, the following table shows the value and associated uncertainty for each option:
+
+| Counts | Std. Dev. (A) | Std. Dev. (B) | Std. Dev. (C) | Std. Dev. (D) | Std. Dev. (E) |
+| ------- | ------ | ------- | ------- | ------- | --- |
+| 0 | raise exception | NaN | 1 | 0.707 | 0 |
+| 1 | 1 | 1 | 1 | 1.224745 | 1 |
+| 2 | 1.414214 | 1.414214 | 1.414214 | 1.581139 | 1.414214 |
+| 3 | 1.732051 | 1.732051 | 1.732051 | 1.870829 | 1.732051 |
+| 4 | 2 | 2 | 2 | 2.12132 | 2 |
+| 5 | 2.236068 | 2.236068 | 2.236068 | 2.345208 | 2.236068 |
+| 10 | 3.162278 | 3.162278 | 3.162278 | 3.24037 | 3.162278 |
+| 50 | 7.071068 | 7.071068 | 7.071068 | 7.106335 | 7.071068 |
+| 100 | 10 | 10 | 10 | 10.02497 | 10 |
+| 500 | 22.36068 | 22.36068 | 22.36068 | 22.37186 | 22.36068 |
+| 1000 | 31.62278 | 31.62278 | 31.62278 | 31.63068 | 31.62278 |
+| 5000 | 70.71068 | 70.71068 | 70.71068 | 70.71421 | 70.71068 |
+| 10000 | 100 | 100 | 100 | 100.0025 | 100 |
+
+## Present
+
+These approaches were discussed in a regular project update meeting including
+- TW & FA (Experiment controls)
+- CK (Reflectometry)
+- JL (Muons)
+- RD (SANS)
+
+## Decision
+
+The consensus was to go with Option D.
+
+## Justification
+
+- Option A will cause real-life scans to crash in low counts regions.
+- Option B involves `NaN`s, which have many surprising floating-point characteristics and are highly likely to be a source of future bugs.
+- Option D was preferred to option C by scientists present.
+- Option E causes surprising results and/or crashes downstream, for example fitting may consider points with zero uncertainty to have "infinite" weight, therefore effectively disregarding all other data.

doc/callbacks/plotting.md

@@ -38,9 +38,12 @@ ax = plt.gca()
 # Set the y-scale to logarithmic
 ax.set_yscale("log")
 # Use the above axes in a LivePlot callback
-plot_callback = LivePlot(y="y_variable", x="x_variable", ax=ax)
+plot_callback = LivePlot(y="y_variable", x="x_variable", ax=ax, yerr="yerr_variable")
+# yerr is the uncertanties of each y value, producing error bars
 ```
 
+By providing a signal name to the `yerr` argument you can pass uncertainties to LivePlot, by not providing anything for this argument means that no errorbars will be drawn. Errorbars are drawn after each point collected, displaying their standard deviation- uncertainty data is collected from Bluesky event documents and errorbars are updated after every new point added.
+
 The `plot_callback` object can then be subscribed to the run engine, using either:
 - An explicit callback when calling the run engine: `RE(some_plan(), plot_callback)`
 - Be subscribed in a plan using `@subs_decorator` from bluesky **(recommended)**

doc/conf.py

@@ -16,10 +16,17 @@
 author = "ISIS Experiment Controls"
 release = "0.1"
 
-
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
 
+nitpicky = True
+nitpick_ignore_regex = [
+    ("py:func", r"^(?!ibex_bluesky_core\.).*$"),
+    ("py:class", r"^(?!ibex_bluesky_core\.).*$"),
+    ("py:class", r"^.*\.T$"),
+    ("py:obj", r"^.*\.T$"),
+]
+
 myst_enable_extensions = ["dollarmath"]
 
 extensions = [
@@ -45,11 +52,9 @@
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
 
 html_theme = "sphinx_rtd_theme"
-html_static_path = ["_static"]
 html_logo = "logo.png"
 html_theme_options = {
     "logo_only": False,
-    "display_version": False,
     "style_nav_header_background": "#343131",
 }
 html_favicon = "favicon.png"

doc/dev/logging.md

@@ -1,10 +1,34 @@
 # Logging
+To invoke the `ibex_bluesky_core` logger, create and use a `logger` object in [the standard way](https://docs.python.org/3/library/logging.html):
 
-To invoke the bluesky logger, import and use it at the desired level:
 ```python
-from ibex_bluesky_core.logger import logger
-logger.blueskylogger.warning("Message to be logged")
+import logging
+logger = logging.getLogger(__name__)
+logger.warning("Message to be logged")
 ```
-The logger utilizes a `TimedRotatingFileHandler` defined in the `logging.conf` file that rolls over the log at midnight.
 
-The default logging level is defined at `INFO`. This means that events of lesser severity will not be logged. To change the default level, change level attribute of logger_blueskycore in the `logging.conf`
+The logger utilizes a `TimedRotatingFileHandler` defined in `src/ibex_bluesky_core/log.py` that rolls over the log at midnight.
+
+By default, the log files will be created in `c:\instrument\var\logs\bluesky`. This can be configured by setting
+the `IBEX_BLUESKY_CORE_LOGS` environment variable.
+
+There are 3 primary logger objects which are "interesting" in the context of `ibex_bluesky_core`:
+- `ibex_bluesky_core` itself
+- `bluesky`, for low-level diagnostic logging from the run engine & plans
+- `ophyd_async` for low-level diagnostic logging from ophyd-async devices
+
+The default logging level for bluesky libraries is defined at `INFO`. This means that events of lesser severity will not be logged. 
+To change the logging level for all bluesky libraries simultaneously, call: 
+
+```python
+from ibex_bluesky_core.log import set_bluesky_log_levels
+set_bluesky_log_levels("DEBUG")
+```
+
+To change the logging level of just a single library (for example, just `opyhyd_async`), use the standard
+python `logging` mechanisms:
+
+```python
+import logging
+logging.getLogger("ophyd_async").setLevel("DEBUG")
+```

doc/fitting/fitting.md

@@ -24,14 +24,17 @@ plt.figure()
 ax = plt.gca() 
 # ax is shared by fit_callback and plot_callback 
 
-plot_callback = LivePlot(y="y_variable", x="x_variable", ax=ax)
-fit_callback = LiveFit(Gaussian.fit(), y="y_variable", x="x_variable", update_every=0.5)
+plot_callback = LivePlot(y="y_signal", x="x_signal", ax=ax, yerr="yerr_signal")
+fit_callback = LiveFit(Gaussian.fit(), y="y_signal", x="x_signal", yerr="yerr_signal", update_every=0.5)
+# Using the yerr parameter allows you to use error bars.
 # update_every = in seconds, how often to recompute the fit. If `None`, do not compute until the end. Default is 1.
 fit_plot_callback = LiveFitPlot(fit_callback, ax=ax, color="r")
 ```
 
 **Note:** that the `LiveFit` callback doesn't directly do the plotting, it will return function parameters of the model its trying to fit to; a `LiveFit` object must be passed to `LiveFitPlot` which can then be subscribed to the `RunEngine`. See the [Bluesky Documentation](https://blueskyproject.io/bluesky/main/callbacks.html#livefitplot) for information on the various arguments that can be passed to the `LiveFitPlot` class.
 
+Using the `yerr` argument allows you to pass uncertainties via a signal to LiveFit, so that the "weight" of each point influences the fit produced. By not providing a signal name you choose not to use uncertainties/weighting in the fitting calculation. Each weight is computed as `1/(standard deviation at point)` and is taken into account to determine how much a point affects the overall fit of the data. Same as the rest of `LiveFit`, the fit will be updated after every new point collected now taking into account the weights of each point. Uncertainty data is collected from Bluesky event documents after each new point.
+
 The `plot_callback` and `fit_plot_callback` objects can then be subscribed to the `RunEngine`, using the same methods as described in [`LivePlot`](../callbacks/plotting.md). See the following example using `@subs_decorator`:
 
 ```py
@@ -79,7 +82,7 @@ from bluesky.callbacks import LiveFitPlot
 from ibex_bluesky_core.callbacks.fitting.fitting_utils import [FIT]
 
 # Pass [FIT].fit() to the first parameter of LiveFit
-lf = LiveFit([FIT].fit(), y="y_variable", x="x_variable", update_every=0.5)
+lf = LiveFit([FIT].fit(), y="y_signal", x="x_signal", update_every=0.5)
 
 # Then subscribe to LiveFitPlot(lf, ...)
 ```
@@ -89,7 +92,7 @@ The `[FIT].fit()` function will pass the `FitMethod` object straight to the `Liv
 **Note:** that for the fits in the above table that require parameters, you will need to pass value(s) to their `.fit` method. For example Polynomial fitting:
 
 ```py
-lf = LiveFit(Polynomial.fit(3),  y="y_variable", x="x_variable", update_every=0.5)
+lf = LiveFit(Polynomial.fit(3),  y="y_signal", x="x_signal", update_every=0.5)
 # For a polynomial of degree 3
 ```
 
@@ -138,7 +141,7 @@ def guess(x: npt.NDArray[np.float64], y: npt.NDArray[np.float64]) -> dict[str, l
 fit_method = FitMethod(model, guess) 
 #Pass the model and guess function to FitMethod
 
-lf = LiveFit(fit_method, y="y_variable", x="x_variable", update_every=0.5)
+lf = LiveFit(fit_method, y="y_signal", x="x_signal", update_every=0.5)
 
 # Then subscribe to LiveFitPlot(lf, ...)
 ```
@@ -163,7 +166,7 @@ def different_model(x: float, c1: float, c0: float) -> float:
 fit_method = FitMethod(different_model, Linear.guess())
 # Uses the user defined model and the standard Guessing. function for linear models
 
-lf = LiveFit(fit_method, y="y_variable", x="x_variable", update_every=0.5)
+lf = LiveFit(fit_method, y="y_signal", x="x_signal", update_every=0.5)
 
 # Then subscribe to LiveFitPlot(lf, ...)
 ```
@@ -188,7 +191,7 @@ def different_guess(x: float, c1: float, c0: float) -> float:
 fit_method = FitMethod(Linear.model(), different_guess)
 # Uses the standard linear model and the user defined Guessing. function
 
-lf = LiveFit(fit_method, y="y_variable", x="x_variable", update_every=0.5)
+lf = LiveFit(fit_method, y="y_signal", x="x_signal", update_every=0.5)
 
 # Then subscribe to LiveFitPlot(lf, ...)
 ```

doc/index.rst

@@ -22,58 +22,15 @@ abstraction library, which allows bluesky to communicate with an underlying cont
 - Bluesky or scanning-related utilities which are useful across multiple beamlines.
 
 
-Overview
-========
+Getting started
+===============
 
-.. note::
-
-    bluesky is a very flexible data acquisition framework. The following example illustrates a minimal scan,
-    not the full extent of bluesky's functionality.
-
-Using ``ibex_bluesky_core``, one can define some simple instrument-specific devices::
-
-    from ibex_bluesky_core.devices.block import block_r, block_mot
-
-    mot = block_mot("mot")  # An IBEX block pointing at a motor
-    det = block_r(float, "p5")  # A readback block
-
-And define a simple step-scan which uses those devices::
-
-    import bluesky.plans as bp
-    from ophyd_async.plan_stubs import ensure_connected
-
-    def my_plan(start: float, stop: float, num: int):
-        yield from ensure_connected(det, mot, force_reconnect=True)
-        yield from bp.scan([det], mot, start, stop, num)
-
-After which, a simple scan can be run by a user::
-
-    from ibex_bluesky_core.run_engine import get_run_engine
-
-    # A bluesky RunEngine instance, already available if using IBEX GUI
-    RE = get_run_engine()
-
-    # Scan "mot" from 0 to 10 in 5 steps, reading "my_detector" at each step.
-    RE(my_plan(0, 10, 5))
-
-That plan may then also use:
-
-- Other `experimental plans <https://blueskyproject.io/bluesky/main/plans.html#summary>`_ or
-  `plan stubs <https://blueskyproject.io/bluesky/main/plans.html#stub-plans>`_, to build up more complex
-  plans.
-- `Callbacks <https://blueskyproject.io/bluesky/main/callbacks.html>`_ provided by either
-  ``ibex_bluesky_core`` or ``bluesky``, which do something with the results of the scan: live
-  fitting, live plotting, file-writing, ...
-- `Simulation facilities <https://blueskyproject.io/bluesky/main/simulation.html>`_ provided by
-  bluesky, to check for problems before the plan is run
-- And a range of other functionality!
-
-See the `manual system tests <https://github.com/ISISComputingGroup/ibex_bluesky_core/tree/main/manual_system_tests>`_ for
-some full runnable examples of complex plans, using a wider range of bluesky functionality.
+.. toctree::
+   :maxdepth: 2
+   :caption: Tutorial
+   :glob:
 
-The reference documentation below lists the functionality that has been implemented in ``ibex_bluesky_core``,
-however the vast majority of `bluesky <https://blueskyproject.io/bluesky/main/index.html>`_ functionality remains
-available, and the advanced user is also encouraged to read that documentation.
+   tutorial/overview.md
 
 Reference documentation
 =======================