cellfinder-core and cellfinder-napari merge update

docs/source/blog/version1/core_and_napari_merge.md

@@ -0,0 +1,44 @@
+---
+blogpost: true
+date: Jan 2, 2024
+author: Will Graham
+location: London, England
+category: BrainGlobe-v1
+language: English
+---
+
+# `cellfinder-core` and `cellfinder-napari` have merged
+
+[BrainGlobe version 1](./version_1_announcement.md) is almost ready, and the next stage of its release journey is the merging of the "backend" `cellfinder-core` and `cellfinder-napari` packages into one.
+We had previously [migrated the `cellfinder` data analysis workflow](./cellfinder_migration_live.md) into the new `brainglobe-workflows` package, as part of our efforts to separate "backend" BrainGlobe tools from common analysis pipelines.
+This means that there is no longer any need to keep the "backend" package (`cellfinder-core`) and nor the visualisation plugin (`cellfinder-napari`) stored in separate, lower-level packages.
+As such;
+
+- [`cellfinder-core`](https://github.com/brainglobe/cellfinder-core) and [`cellfinder-napari`](https://github.com/brainglobe/cellfinder-napari) will be deprecated.
+- [A _package_ called `cellfinder`](https://github.com/brainglobe/cellfinder) will become available as a replacement for this functionality. Note that this will re-use the old "cellfinder" name that the command-line-interface had, [prior to its migration](./cellfinder_migration_live.md).
+- The `cellfinder-napari` plugin is now simply called "cellfinder" internally, and when loaded up in napari.
+- The "cellfinder" name for the whole-brain registration and analysis workflow provided by [`brainglobe-workflows`](documentation/brainglobe-workflows/index) will be deprecated to avoid confusion. This workflow will now be available as "`brainmapper`".
+
+From a user perspective, this is just a restructuring and reorganisation of existing functionality, and the renaming of the cellfinder command-line tool to `brainmapper`.
+If you were using the `cellfinder-core` backend, or the `cellfinder-napari` plugins, you'll need to uninstall those packages and install version `1.0.0` (or later) of the [`cellfinder` _package_](https://pypi.org/project/cellfinder/).
+By merging two packages together, we hope to reduce the complexity of a BrainGlobe install and make the API to the tools more intuitive.
+For developers, this merge again reduces the number of repositories that we have to maintain.
+It also ensures we continue to distinguish between BrainGlobe tools and workflows.
+And finally, we also address a long-standing historical naming issue with the old "cellfinder" (now `brainmapper`) workflow being confused with the corresponding backend packages.
+
+## What do I need to do?
+
+If you were previously using `cellfinder-core` in your scripts, you will need to uninstall it and install `cellfinder` version `1.0.0` or greater.
+Once you've done this, it should just be a case of changing each of your `from cellfinder_core import X` statements to `from cellfinder.core import X`.
+
+If you were previously using `cellfinder-napari`, you'll need to uninstall it and install `cellfinder[napari]`, which will fetch `cellfinder` and its optional napari dependency.
+Any references you have made to the `cellfinder_napari` plugin in your analysis will need to change to the "`cellfinder` plugin" instead.
+
+If you were using the cellfinder command-line tool that was provided by `brainglobe-workflows`, you will need to update your version of `brainglobe-workflows`.
+If you were still using the cellfinder command-line tool provided by the `cellfinder` _package_, with version less than `1.0.0`, you will need to take slightly more involved action - we recommand you look at the instructions on the [full changelog](#full-changelog) for details.
+
+You can take a look at the instructions in the [full changelog](#full-changelog) for more details about updating to the new package.
+
+## Full changelog
+
+You can find the [full changelog on the releases page](../../community/releases/v1/cellfinder-core-and-plugin-merge.md).

docs/source/community/developers/repositories/cellfinder-core/index.md

@@ -1,4 +1,4 @@
-# cellfinder-core
+# cellfinder.core
 
 ## Cell detection
 
@@ -12,12 +12,12 @@ Cell detection in cellfinder has three stages:
 
 #### 2D filtering
 
-Code can be found in `cellfinder_core/detect/filters/plane`.
+Code can be found in `cellfinder/core/detect/filters/plane`.
 Each plane of data is filtered independently, and in parallel across a number of processes.
 
 This part of processing performs two tasks:
 
-1. Applies a filter to enhance peaks in the data (``cellfinder_core/detect/filters/plane/classical_filter.py``).
+1. Applies a filter to enhance peaks in the data (``cellfinder/core/detect/filters/plane/classical_filter.py``).
    This consists of (in order)
     1. a median filter (`scipy.signal.medfilt2d`)
     2. a gaussian filter (`scipy.ndimage.gaussian_filter`)
@@ -38,7 +38,7 @@ Memory usage during 2D filtering, for each plane, is the following:
 
 #### 3D filtering
 
-Code can be found in `cellfinder_core/detect/filters/volume/ball_filter.py`.
+Code can be found in `cellfinder/core/detect/filters/volume/ball_filter.py`.
 Both this step and the structure detection step take place in the main `Python` process, with no parallelism. As the planes are processed in the 2D filtering step, they are passed to this step. When `ball_z_size` planes have been handed over, 3D filtering begins.
 
 The 3D filter stores a 3D array that has depth `ball_z_size`, and contains `ball_z_size` number of planes. This is a small 3D slice of the original data. A spherical kernel runs across the x, y dimensions, and where enough intensity overlaps with the spherical kernel the voxel at the centre of the kernel is marked as being part of a cell. The output of this step is the central plane of the array, with marked cells.
@@ -50,7 +50,7 @@ Memory usage information during 3D filtering:
 
 #### Structure detection
 
-Code can be found in `cellfinder_core/detect/filters/volume/structure_detection.py`.
+Code can be found in `cellfinder/core/detect/filters/volume/structure_detection.py`.
 This step takes the planes output from 3D filtering with marked cell voxels, and detects collections of voxels that are adjacent.
 
 Memory usage information during structure detection:

docs/source/community/releases/v1/cellfinder-core-and-plugin-merge.md

@@ -0,0 +1,82 @@
+# Version 1: changes to the cellfinder backend and plugin
+
+The `cellfinder-core` package and `cellfinder-napari` plugin have now been migrated to a [package called cellfinder](https://github.com/brainglobe/cellfinder).
+Please note that this is enacting the [future warning from the earlier `cellfinder` (CLI) migration](./cellfinder-migration.md#future-warning).
+If you wish to continue using both the `cellfinder` command-line interface *and* the backend Python API, you will need to install the latest version of `brainglobe-workflows`, which will automatically fetch the new version of the `cellfinder` package.
+See the [updating section](#updating) for more details.
+
+## `cellfinder-core`
+
+If you were previously using the Python API to `cellfinder-core`, you will need to uninstall `cellfinder-core` from your environment and install `cellfinder` version `1.0.0` or later.
+
+The internal package structure has not changed in this move, but it is now a submodule of the new `cellfinder` package.
+So if you were using the Python API in your scripts, you will need to change any `from cellfinder_core import X` statements to `from cellfinder.core import X`.
+Once you make this change, everything should work as it was before.
+
+## `cellfinder-napari`
+
+If you were previously using the napari plugin for visualising output data, you will need to uninstall `cellfinder-napari` and install `cellfinder[napari]`, making sure to fetch the optional napari dependency.
+
+The plugin itself has not undergone any interface changes, but is now just called "cellfinder" rather than "cellfinder-napari" when viewed from the napari widget panel.
+
+## `brainglobe-workflows`
+
+Now ships with the new version of `cellfinder`, containing `cellfinder.core` and `cellfinder.napari` submodules.
+`brainglobe-workflows` is now the *only* package that provides the old "cellfinder" command-line interface, or workflow.
+This workflow is now _only_ available under the name `brainmapper` - the name `cellfinder` is reserved for the backend package that contains the merged `cellfinder-core` and `cellfinder-napari` packages mentioned above.
+
+## Updating
+
+The update steps that you will need to perform vary depending on how you currently use cellfinder (CLI) and whether you upgraded to `brainglobe-workflows` previously.
+Regardless, before you begin the process of updating we recommend you uninstall `cellfinder-core` and `cellfinder-napari` from your environment
+
+```bash
+pip uninstall cellfinder-core cellfinder-napari
+```
+
+Alternatively; you can create a new, clean environment to install into.
+
+### I previously updated to `brainglobe-workflows`
+
+If you previously updated from the old cellfinder package to `brainglobe-workflows`, then you should be able to simply update to the latest version of `brainglobe-workflows`, with
+
+```bash
+pip install --upgrade brainglobe-workflows
+```
+
+This will fetch the new `cellfinder` package containing the `cellfinder.core` and `cellfinder.napari` submodules.
+You can then continue to use the CLI to run the workflow as you were doing before, however note that the workflow will now only be available as "`brainmapper`".
+
+### I have not updated to `brainglobe-workflows`, but I don't use the cellfinder CLI
+
+If you don't use the `cellfinder` CLI, then you can just upgrade your version of the `cellfinder` package to version `1.0.0` or later:
+
+```bash
+pip install --upgrade cellfinder
+```
+
+If you do want to make use of the old "cellfinder" workflow, you will need to install `brainglobe-workflows` and use it under its new name, `brainmapper`.
+
+### I have not updated to `brainglobe-workflows`, and I use the cellfinder CLI
+
+In this case, we strongly recommend you either create a new Python environment to install into (and delete your old one), or remove `cellfinder`, `cellfinder-core`, and `cellfinder-napari` from your current environment with
+
+```bash
+pip uninstall cellfinder cellfinder-core cellfinder-napari
+```
+
+You may also wish to use `pip` to uninstall `tensorflow` to make sure that the new install does not encounter potential version conflicts.
+
+Once you have cleaned your environment, you will need to install the latest version of `brainglobe-workflows`:
+
+```bash
+pip install brainglobe-workflows
+```
+
+You will now have access to the `brainmapper` workflow from within your environment.
+This is the same as the old "cellfinder" workflow that you were using previously - but now it is [supplied by the `brainglobe-workflows` package](blog/version1/cellfinder_migration_live).
+You will also have the latest version of the `cellfinder` package (`1.0.0` or later) installed;
+
+- `cellfinder-core` is included as a submodule, `cellfinder.core`.
+- `cellfinder-napari` is included as a submodule, and the plugin has been renamed to just "cellfinder" when viewed in napari.
+- To access the old "cellfinder" command-line tool or workflow, you now need to call `brainmapper`. The interface has not changed, just the name.

docs/source/community/releases/v1/index.md

@@ -10,8 +10,9 @@ You can follow the links provided for more information; including a listing of r
 brainreg and brainreg-napari have been merged into a single package | [Further info](brainreg.md#brainreg-and-brainreg-napari) |
 brainreg-segment has been renamed to brainglobe-segmentation | [Further info](brainreg.md#brainreg-segment) |
 The `cellfinder` command-line-interface has been moved into `brainglobe-workflows` | [Further info](cellfinder-migration.md) |
-The cellfinder package is deprecated - it will later be recycled to merge some backend functionality | [Further info](cellfinder-migration.md#cellfinder-repository)
-The cellfinder Docker image is discontinued | [Further info](cellfinder-migration.md#cellfinder-docker-image)
+The cellfinder package is deprecated - it will later be recycled to merge some backend functionality | [Further info](cellfinder-migration.md#cellfinder-repository) |
+The cellfinder Docker image is discontinued | [Further info](cellfinder-migration.md#cellfinder-docker-image) |
+cellfinder-core and cellfinder-napari merged into "new cellfinder" | [Further info](cellfinder-core-and-plugin-merge.md) |
 
 ## Complete index
 
@@ -21,4 +22,5 @@ The cellfinder Docker image is discontinued | [Further info](cellfinder-migratio
 
 brainreg
 cellfinder-migration
+cellfinder-core-and-plugin-merge
 ```

docs/source/community/roadmaps/index.md

@@ -1,9 +1,8 @@
 # Roadmaps
 
-BrainGlobe roadmaps give you an idea of where the project is heading over the next few months and years. 
-These roadmaps are usually at a high level for the BrainGlobe project as a whole. 
-Plans for individual tools are usually discussed on their respective repos, and 
-over at [Zulip](https://brainglobe.zulipchat.com/).
+BrainGlobe roadmaps give you an idea of where the project is heading over the next few months and years.
+These roadmaps are usually at a high level for the BrainGlobe project as a whole.
+Plans for individual tools are usually discussed on their respective repos, and over at [Zulip](https://brainglobe.zulipchat.com/).
 
 BrainGlobe is a community project, and as such we welcome your feedback. Please [get in touch with us](/contact) if you have
 any questions or suggestions. These roadmaps are living documents and will likely be revisited regularly.
@@ -12,4 +11,4 @@ any questions or suggestions. These roadmaps are living documents and will likel
 :maxdepth: 1
 
 november-2023
-```
+```

docs/source/documentation/cellfinder/user-guide/command-line/candidate-detection.md → docs/source/documentation/brainglobe-workflows/brainmapper/candidate-detection.md

@@ -15,4 +15,3 @@ considered a nucleus pixel. **Default: 0.6**
 Given as a fraction of the soma-diameter. **Default: 0.2**
 * `--threshold` The cell threshold, in multiples of the standard deviation above the mean. **Default: 10**
 * `--soma-spread-factor` Soma size spread factor \(for splitting up cell clusters\). **Default: 1.4**
-

docs/source/documentation/cellfinder/user-guide/command-line/classification.md → docs/source/documentation/brainglobe-workflows/brainmapper/classification.md

@@ -21,4 +21,3 @@ trained on.  Set this to adjust the pixel sizes of the extracted cubes. **Defaul
 * `--cube-depth` The depth \(z\)\) of the cubes to extract in pixels\(must be even\). **Default 20**
 * `--save-empty-cubes` If a cube cannot be extracted \(e.g. to close to the edge of the image\), save an empty cube 
 instead. Useful to keep track of all cell candidates.
-

docs/source/documentation/brainglobe-workflows/brainmapper/cli.md

@@ -0,0 +1,82 @@
+# Command line interface
+
+## Basic usage
+
+To run `brainmapper`, use this general syntax
+
+```bash
+    brainmapper -s signal_channel_images  optional_signal_channel_images -b background_channel_images -o /path/to/output_directory -v 5 2 2 --orientation asl
+```
+
+:::{hint}
+All options can be found by running `brainmapper -h`
+:::
+
+## Arguments
+
+### Mandatory
+
+- `-s` or `--signal-planes-paths` Path to the directory of the signal files. Can also be a text file pointing to the files.
+- **There can be as many signal channels as you like, and each will be treated independently**.
+- `-b` or `--background-planes-path` Path to the directory of the background files. Can also be a text file pointing to the files.
+- **This background channel will be used for all signal channels**
+- `-o` or `--output-dir` Output directory for all intermediate and final results
+
+:::{caution}
+You must also specify the orientation and voxel size of your data, see [Image definition](/documentation/setting-up/image-definition).
+:::
+
+### Optional Arguments
+
+#### Only run parts of `brainmapper`
+
+If for some reason you don't want some parts of `brainmapper` to run, you can use the following options.
+If a part of the pipeline is required by another part it will be run (i.e. `--no-detection` won't do anything unless `--no-classification` is also used).
+`brainmapper` will attempt to work out which parts of the pipeline have already been run (in a given output directory) and not run them again if appropriate.
+
+- `--no-register` Do not run registration
+- `--no-detection` Do not run cell candidate detection
+- `--no-classification` Do not run cell classification
+- `--no-analyse` Do not analyse and export cell positions
+- `--no-figures` Do not create figures (e.g. heatmap)
+
+#### Figure options
+
+Figures cannot be customised much, but the current options are here:
+
+- `--heatmap-smoothing` Gaussian smoothing sigma, in microns.
+- `--no-mask-figs` Don't mask the figures (removing any areas outside the brain, from e.g. smoothing).
+
+#### Performance, debugging and testing
+
+- `--debug` Increase verbosity of statements printed to console and save all intermediate files.
+- `--n-free-cpus` The number of CPU cores on the machine to leave unused by the program to spare resources.
+- `--max-ram` Maximum amount of RAM to use (in GB) — **not currently fully implemented for all parts of `brainmapper`**
+
+Useful for testing or if you know your cells are only in a specific region;
+
+- `--start-plane` The first plane to process in the Z dimension
+- `--end-plane` The last plane to process in the Z dimension
+
+#### Standard space options
+
+- `--transform-all` Transform all cell positions (including artifacts).
+
+## Additional options
+
+```{toctree}
+:maxdepth: 1
+candidate-detection
+classification
+/documentation/brainreg/user-guide/parameters
+```
+
+:::{hint}
+If you're using `brainmapper` at the [Sainsbury Wellcome Centre](https://www.sainsburywellcome.org/web/), you may wish to see the [instructions for using brainmapper on the SWC HPC system](using-brainmapper-at-the-swc).
+:::
+
+```{toctree}
+:maxdepth: 1
+:hidden:
+using-brainmapper-at-the-swc
+```

docs/source/documentation/brainglobe-workflows/brainmapper/data-requirements.md

@@ -0,0 +1,36 @@
+# Data requirements
+
+What kind of data does `brainmapper` support?
+
+## Introduction
+
+`brianmapper` was written to analyse certain kinds of whole brain microscopy datasets (e.g. serial two-photon or lightsheet in cleared tissue).
+Although we are working on supporting different kinds of data, currently, the data must fit these criteria:
+
+### Image channels
+
+For registration, you only need a single channel, but this is ideally a "background" channel, i.e., one with only autofluorescence, and no other strong signal. Typically, we acquire the "signal" channels with red or green filters, and then the "background" channel with blue filters.
+
+For cell detection, you will need two channels, the "signal" channel, and the "background" channel.
+The signal channel should contain brightly labelled cells (e.g. from staining or viral injections).
+The models supplied with `brainmapper` were trained on whole-cell labels, so if you have e.g. a nuclear marker, they will need to be retrained (see [Training the network](training/index).
+However, realistically, the network will need to be retrained for every new application.
+
+### Image structure
+
+Although we hope to support more varied types of data soon, your images must currently:
+
+- Cover the entire brain
+- Be of sufficiently high resolution that cells appear in multiple planes (i.e. 10μm axial spacing)
+- Contain planes that are registered to each other (i.e., this is often not the case with slide scanners or manual acquisition)
+
+### Organisation
+
+`brainmapper` expects that your data will be stored as a series of 2D tiff files.
+These can be in a single directory, or you can generate a text file that points to them.
+Different channels in your dataset must be in different directories or text files.
+
+:::{caution}
+Please ensure that none of the files or folders that you pass to `brainmapper` have a space in them.
+This should be fixed in the future, but for now, please use `/the_path/to/my_data` rather than `/the path/to/my data`
+:::