Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cellfinder-core and cellfinder-napari merge update #104

Merged
merged 17 commits into from
Jan 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions docs/source/blog/version1/core_and_napari_merge.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
blogpost: true
date: Jan 2, 2024
author: Will Graham
location: London, England
category: BrainGlobe-v1
language: English
---

# `cellfinder-core` and `cellfinder-napari` have merged

[BrainGlobe version 1](./version_1_announcement.md) is almost ready, and the next stage of its release journey is the merging of the "backend" `cellfinder-core` and `cellfinder-napari` packages into one.
We had previously [migrated the `cellfinder` data analysis workflow](./cellfinder_migration_live.md) into the new `brainglobe-workflows` package, as part of our efforts to separate "backend" BrainGlobe tools from common analysis pipelines.
This means that there is no longer any need to keep the "backend" package (`cellfinder-core`) and nor the visualisation plugin (`cellfinder-napari`) stored in separate, lower-level packages.
As such;

- [`cellfinder-core`](https://github.com/brainglobe/cellfinder-core) and [`cellfinder-napari`](https://github.com/brainglobe/cellfinder-napari) will be deprecated.
- [A _package_ called `cellfinder`](https://github.com/brainglobe/cellfinder) will become available as a replacement for this functionality. Note that this will re-use the old "cellfinder" name that the command-line-interface had, [prior to its migration](./cellfinder_migration_live.md).
- The `cellfinder-napari` plugin is now simply called "cellfinder" internally, and when loaded up in napari.
- The "cellfinder" name for the whole-brain registration and analysis workflow provided by [`brainglobe-workflows`](/documentation/brainglobe-workflows/index.md) will be deprecated to avoid confusion. This workflow will now be available as "`brainmapper`".

From a user perspective, this is just a restructuring and reorganisation of existing functionality, and the renaming of the cellfinder command-line tool to `brainmapper`.
If you were using the `cellfinder-core` backend, or the `cellfinder-napari` plugins, you'll need to uninstall those packages and install version `1.0.0` (or later) of the [`cellfinder` _package_](https://pypi.org/project/cellfinder/).
By merging two packages together, we hope to reduce the complexity of a BrainGlobe install and make the API to the tools more intuitive.
For developers, this merge again reduces the number of repositories that we have to maintain.
It also ensures we continue to distinguish between BrainGlobe tools and workflows.
And finally, we also address a long-standing historical naming issue with the old "cellfinder" (now `brainmapper`) workflow being confused with the corresponding backend packages.

## What do I need to do?

If you were previously using `cellfinder-core` in your scripts, you will need to uninstall it and install `cellfinder` version `1.0.0` or greater.
Once you've done this, it should just be a case of changing each of your `from cellfinder_core import X` statements to `from cellfinder.core import X`.

If you were previously using `cellfinder-napari`, you'll need to uninstall it and install `cellfinder[napari]`, which will fetch `cellfinder` and its optional napari dependency.
Any references you have made to the `cellfinder_napari` plugin in your analysis will need to change to the "`cellfinder` plugin" instead.

If you were using the cellfinder command-line tool that was provided by `brainglobe-workflows`, you will need to update your version of `brainglobe-workflows`.
If you were still using the cellfinder command-line tool provided by the `cellfinder` _package_, with version less than `1.0.0`, you will need to take slightly more involved action - we recommand you look at the instructions on the [full changelog](#full-changelog) for details.

You can take a look at the instructions in the [full changelog](#full-changelog) for more details about updating to the new package.

## Full changelog

You can find the [full changelog on the releases page](../../community/releases/v1/cellfinder-core-and-plugin-merge.md).
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# cellfinder-core
# cellfinder.core

## Cell detection

Expand All @@ -12,12 +12,12 @@ Cell detection in cellfinder has three stages:

#### 2D filtering

Code can be found in `cellfinder_core/detect/filters/plane`.
Code can be found in `cellfinder/core/detect/filters/plane`.
Each plane of data is filtered independently, and in parallel across a number of processes.

This part of processing performs two tasks:

1. Applies a filter to enhance peaks in the data (``cellfinder_core/detect/filters/plane/classical_filter.py``).
1. Applies a filter to enhance peaks in the data (``cellfinder/core/detect/filters/plane/classical_filter.py``).
This consists of (in order)
1. a median filter (`scipy.signal.medfilt2d`)
2. a gaussian filter (`scipy.ndimage.gaussian_filter`)
Expand All @@ -38,7 +38,7 @@ Memory usage during 2D filtering, for each plane, is the following:

#### 3D filtering

Code can be found in `cellfinder_core/detect/filters/volume/ball_filter.py`.
Code can be found in `cellfinder/core/detect/filters/volume/ball_filter.py`.
Both this step and the structure detection step take place in the main `Python` process, with no parallelism. As the planes are processed in the 2D filtering step, they are passed to this step. When `ball_z_size` planes have been handed over, 3D filtering begins.

The 3D filter stores a 3D array that has depth `ball_z_size`, and contains `ball_z_size` number of planes. This is a small 3D slice of the original data. A spherical kernel runs across the x, y dimensions, and where enough intensity overlaps with the spherical kernel the voxel at the centre of the kernel is marked as being part of a cell. The output of this step is the central plane of the array, with marked cells.
Expand All @@ -50,7 +50,7 @@ Memory usage information during 3D filtering:

#### Structure detection

Code can be found in `cellfinder_core/detect/filters/volume/structure_detection.py`.
Code can be found in `cellfinder/core/detect/filters/volume/structure_detection.py`.
This step takes the planes output from 3D filtering with marked cell voxels, and detects collections of voxels that are adjacent.

Memory usage information during structure detection:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Version 1: changes to the cellfinder backend and plugin

The `cellfinder-core` package and `cellfinder-napari` plugin have now been migrated to a [package called cellfinder](https://github.com/brainglobe/cellfinder).
Please note that this is enacting the [future warning from the earlier `cellfinder` (CLI) migration](./cellfinder-migration.md#future-warning).
If you wish to continue using both the `cellfinder` command-line interface *and* the backend Python API, you will need to install the latest version of `brainglobe-workflows`, which will automatically fetch the new version of the `cellfinder` package.
See the [updating section](#updating) for more details.

## `cellfinder-core`

If you were previously using the Python API to `cellfinder-core`, you will need to uninstall `cellfinder-core` from your environment and install `cellfinder` version `1.0.0` or later.

The internal package structure has not changed in this move, but it is now a submodule of the new `cellfinder` package.
So if you were using the Python API in your scripts, you will need to change any `from cellfinder_core import X` statements to `from cellfinder.core import X`.
Once you make this change, everything should work as it was before.

## `cellfinder-napari`

If you were previously using the napari plugin for visualising output data, you will need to uninstall `cellfinder-napari` and install `cellfinder[napari]`, making sure to fetch the optional napari dependency.

The plugin itself has not undergone any interface changes, but is now just called "cellfinder" rather than "cellfinder-napari" when viewed from the napari widget panel.

## `brainglobe-workflows`

Now ships with the new version of `cellfinder`, containing `cellfinder.core` and `cellfinder.napari` submodules.
`brainglobe-workflows` is now the *only* package that provides the old "cellfinder" command-line interface, or workflow.
This workflow is now _only_ available under the name `brainmapper` - the name `cellfinder` is reserved for the backend package that contains the merged `cellfinder-core` and `cellfinder-napari` packages mentioned above.

## Updating

The update steps that you will need to perform vary depending on how you currently use cellfinder (CLI) and whether you upgraded to `brainglobe-workflows` previously.
Regardless, before you begin the process of updating we recommend you uninstall `cellfinder-core` and `cellfinder-napari` from your environment

```bash
pip uninstall cellfinder-core cellfinder-napari
```

Alternatively; you can create a new, clean environment to install into.

### I previously updated to `brainglobe-workflows`

If you previously updated from the old cellfinder package to `brainglobe-workflows`, then you should be able to simply update to the latest version of `brainglobe-workflows`, with

```bash
pip install --upgrade brainglobe-workflows
```

This will fetch the new `cellfinder` package containing the `cellfinder.core` and `cellfinder.napari` submodules.
You can then continue to use the CLI to run the workflow as you were doing before, however note that the workflow will now only be available as "`brainmapper`".

### I have not updated to `brainglobe-workflows`, but I don't use the cellfinder CLI

If you don't use the `cellfinder` CLI, then you can just upgrade your version of the `cellfinder` package to version `1.0.0` or later:

```bash
pip install --upgrade cellfinder
```

If you do want to make use of the old "cellfinder" workflow, you will need to install `brainglobe-workflows` and use it under its new name, `brainmapper`.

### I have not updated to `brainglobe-workflows`, and I use the cellfinder CLI

In this case, we strongly recommend you either create a new Python environment to install into (and delete your old one), or remove `cellfinder`, `cellfinder-core`, and `cellfinder-napari` from your current environment with

```bash
pip uninstall cellfinder cellfinder-core cellfinder-napari
```

You may also wish to use `pip` to uninstall `tensorflow` to make sure that the new install does not encounter potential version conflicts.

Once you have cleaned your environment, you will need to install the latest version of `brainglobe-workflows`:

```bash
pip install brainglobe-workflows
```

You will now have access to the `brainmapper` workflow from within your environment.
This is the same as the old "cellfinder" workflow that you were using previously - but now it is [supplied by the `brainglobe-workflows` package](/blog/version1/cellfinder_migration_live.md).
You will also have the latest version of the `cellfinder` package (`1.0.0` or later) installed;

- `cellfinder-core` is included as a submodule, `cellfinder.core`.
- `cellfinder-napari` is included as a submodule, and the plugin has been renamed to just "cellfinder" when viewed in napari.
- To access the old "cellfinder" command-line tool or workflow, you now need to call `brainmapper`. The interface has not changed, just the name.
6 changes: 4 additions & 2 deletions docs/source/community/releases/v1/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,9 @@ You can follow the links provided for more information; including a listing of r
brainreg and brainreg-napari have been merged into a single package | [Further info](brainreg.md#brainreg-and-brainreg-napari) |
brainreg-segment has been renamed to brainglobe-segmentation | [Further info](brainreg.md#brainreg-segment) |
The `cellfinder` command-line-interface has been moved into `brainglobe-workflows` | [Further info](cellfinder-migration.md) |
The cellfinder package is deprecated - it will later be recycled to merge some backend functionality | [Further info](cellfinder-migration.md#cellfinder-repository)
The cellfinder Docker image is discontinued | [Further info](cellfinder-migration.md#cellfinder-docker-image)
The cellfinder package is deprecated - it will later be recycled to merge some backend functionality | [Further info](cellfinder-migration.md#cellfinder-repository) |
The cellfinder Docker image is discontinued | [Further info](cellfinder-migration.md#cellfinder-docker-image) |
cellfinder-core and cellfinder-napari merged into "new cellfinder" | [Further info](cellfinder-core-and-plugin-merge.md) |

## Complete index

Expand All @@ -21,4 +22,5 @@ The cellfinder Docker image is discontinued | [Further info](cellfinder-migratio

brainreg
cellfinder-migration
cellfinder-core-and-plugin-merge
```
9 changes: 4 additions & 5 deletions docs/source/community/roadmaps/index.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
# Roadmaps

BrainGlobe roadmaps give you an idea of where the project is heading over the next few months and years.
These roadmaps are usually at a high level for the BrainGlobe project as a whole.
Plans for individual tools are usually discussed on their respective repos, and
over at [Zulip](https://brainglobe.zulipchat.com/).
BrainGlobe roadmaps give you an idea of where the project is heading over the next few months and years.
These roadmaps are usually at a high level for the BrainGlobe project as a whole.
Plans for individual tools are usually discussed on their respective repos, and over at [Zulip](https://brainglobe.zulipchat.com/).

BrainGlobe is a community project, and as such we welcome your feedback. Please [get in touch with us](/contact) if you have
any questions or suggestions. These roadmaps are living documents and will likely be revisited regularly.
Expand All @@ -12,4 +11,4 @@ any questions or suggestions. These roadmaps are living documents and will likel
:maxdepth: 1

november-2023
```
```
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,3 @@ considered a nucleus pixel. **Default: 0.6**
Given as a fraction of the soma-diameter. **Default: 0.2**
* `--threshold` The cell threshold, in multiples of the standard deviation above the mean. **Default: 10**
* `--soma-spread-factor` Soma size spread factor \(for splitting up cell clusters\). **Default: 1.4**

Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,3 @@ trained on. Set this to adjust the pixel sizes of the extracted cubes. **Defaul
* `--cube-depth` The depth \(z\)\) of the cubes to extract in pixels\(must be even\). **Default 20**
* `--save-empty-cubes` If a cube cannot be extracted \(e.g. to close to the edge of the image\), save an empty cube
instead. Useful to keep track of all cell candidates.

82 changes: 82 additions & 0 deletions docs/source/documentation/brainglobe-workflows/brainmapper/cli.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Command line interface

## Basic usage

To run `brainmapper`, use this general syntax

```bash
brainmapper -s signal_channel_images optional_signal_channel_images -b background_channel_images -o /path/to/output_directory -v 5 2 2 --orientation asl
```

:::{hint}
All options can be found by running `brainmapper -h`
:::

## Arguments

### Mandatory

- `-s` or `--signal-planes-paths` Path to the directory of the signal files. Can also be a text file pointing to the files.
- **There can be as many signal channels as you like, and each will be treated independently**.
- `-b` or `--background-planes-path` Path to the directory of the background files. Can also be a text file pointing to the files.
- **This background channel will be used for all signal channels**
- `-o` or `--output-dir` Output directory for all intermediate and final results

:::{caution}
You must also specify the orientation and voxel size of your data, see [Image definition](/documentation/setting-up/image-definition).
:::

### Optional Arguments

#### Only run parts of `brainmapper`

If for some reason you don't want some parts of `brainmapper` to run, you can use the following options.
If a part of the pipeline is required by another part it will be run (i.e. `--no-detection` won't do anything unless `--no-classification` is also used).
`brainmapper` will attempt to work out which parts of the pipeline have already been run (in a given output directory) and not run them again if appropriate.

- `--no-register` Do not run registration
- `--no-detection` Do not run cell candidate detection
- `--no-classification` Do not run cell classification
- `--no-analyse` Do not analyse and export cell positions
- `--no-figures` Do not create figures (e.g. heatmap)

#### Figure options

Figures cannot be customised much, but the current options are here:

- `--heatmap-smoothing` Gaussian smoothing sigma, in microns.
- `--no-mask-figs` Don't mask the figures (removing any areas outside the brain, from e.g. smoothing).

#### Performance, debugging and testing

- `--debug` Increase verbosity of statements printed to console and save all intermediate files.
- `--n-free-cpus` The number of CPU cores on the machine to leave unused by the program to spare resources.
- `--max-ram` Maximum amount of RAM to use (in GB) — **not currently fully implemented for all parts of `brainmapper`**

Useful for testing or if you know your cells are only in a specific region;

- `--start-plane` The first plane to process in the Z dimension
- `--end-plane` The last plane to process in the Z dimension

#### Standard space options

- `--transform-all` Transform all cell positions (including artifacts).

## Additional options

```{toctree}
:maxdepth: 1
candidate-detection
classification
/documentation/brainreg/user-guide/parameters
```

:::{hint}
If you're using `brainmapper` at the [Sainsbury Wellcome Centre](https://www.sainsburywellcome.org/web/), you may wish to see the [instructions for using brainmapper on the SWC HPC system](using-brainmapper-at-the-swc).
:::

```{toctree}
:maxdepth: 1
:hidden:
using-brainmapper-at-the-swc
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Data requirements

What kind of data does `brainmapper` support?

## Introduction

`brianmapper` was written to analyse certain kinds of whole brain microscopy datasets (e.g. serial two-photon or lightsheet in cleared tissue).
Although we are working on supporting different kinds of data, currently, the data must fit these criteria:

### Image channels

For registration, you only need a single channel, but this is ideally a "background" channel, i.e., one with only autofluorescence, and no other strong signal. Typically, we acquire the "signal" channels with red or green filters, and then the "background" channel with blue filters.

For cell detection, you will need two channels, the "signal" channel, and the "background" channel.
The signal channel should contain brightly labelled cells (e.g. from staining or viral injections).
The models supplied with `brainmapper` were trained on whole-cell labels, so if you have e.g. a nuclear marker, they will need to be retrained (see [Training the network](training/index).
However, realistically, the network will need to be retrained for every new application.

### Image structure

Although we hope to support more varied types of data soon, your images must currently:

- Cover the entire brain
- Be of sufficiently high resolution that cells appear in multiple planes (i.e. 10μm axial spacing)
- Contain planes that are registered to each other (i.e., this is often not the case with slide scanners or manual acquisition)

### Organisation

`brainmapper` expects that your data will be stored as a series of 2D tiff files.
These can be in a single directory, or you can generate a text file that points to them.
Different channels in your dataset must be in different directories or text files.

:::{caution}
Please ensure that none of the files or folders that you pass to `brainmapper` have a space in them.
This should be fixed in the future, but for now, please use `/the_path/to/my_data` rather than `/the path/to/my data`
:::
Loading